2024-09-11T09:39:17.018 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-11T09:39:17.035 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-11T09:39:17.119 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-09-10_21:16:03-rbd-squid-distro-default-smithi/7900734 branch: squid description: rbd/device/{base/install cluster/{fixed-3 openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-comp-snappy supported-random-distro$/{centos_latest} thrashers/default thrashosds-health workloads/rbd_nbd} email: ceph-qa@ceph.com first_in_suite: false job_id: '7900734' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi name: teuthology-2024-09-10_21:16:03-rbd-squid-distro-default-smithi no_nested_subset: false openstack: - machine: cpus: 1 disk: 40 ram: 8000 volumes: count: 4 size: 30 os_type: centos os_version: 9.stream overrides: admin_socket: branch: squid ceph: conf: global: mon client directed command retry: 5 mon warn on pool no app: false ms inject socket failures: 5000 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: bluestore block size: 96636764160 bluestore compression algorithm: snappy bluestore compression mode: aggressive bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd blocked scrub grace period: 3600 osd failsafe full ratio: 0.95 osd max markdown count: 1000 osd objectstore: bluestore flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME - but it is still running - objects unfound and apparently lost - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - timeout on replica - late reservation from - MON_DOWN - OSDMAP_FLAGS - OSD_DOWN - PG_DEGRADED - PG_AVAILABILITY - POOL_APP_NOT_ENABLED - mons down - mon down - out of quorum - noscrub - nodeep-scrub sha1: b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: extra_packages: - rbd-nbd flavor: default sha1: b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc selinux: allowlist: - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: squid sha1: daaf78de334647fc0c800f4ac495cc9475a26bae owner: scheduled_teuthology@teuthology priority: 100 repo: https://git.ceph.com/ceph.git roles: - - mon.a - mon.c - osd.0 - osd.1 - osd.2 - - mon.b - mgr.x - osd.3 - osd.4 - osd.5 - - client.0 seed: 6175 sha1: b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc sleep_before_teardown: 0 subset: 88/128 suite: rbd suite_branch: squid suite_path: /home/teuthworker/src/git.ceph.com_ceph_daaf78de334647fc0c800f4ac495cc9475a26bae/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: daaf78de334647fc0c800f4ac495cc9475a26bae targets: smithi017.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHneZ92wLS9wEoVFsEJff3TbxllQaVnFBBpd3aMf/OktTHEsayzyKE9etbZlkwKAIw6ZJcpxOKYD23MNb5RAPCk= smithi096.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNxpPWO64yIr6T4RKgPsPM+OG+4ctP1rBk+N/B/0SvFgl4R4Z4LwFfGE0/chlp8RdvlhhVft6iZMQpNZG6apayA= smithi107.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBApDXY7Q4ol63+pX3d8bnYgsbpILjg7ASGCTyNn+RP4u2qRUH9wCf/c5DvxJr6BO+tMIZskweZhBwboHNAZ9grA= tasks: - install: null - ceph: null - thrashosds: timeout: 1200 - workunit: clients: client.0: - rbd/rbd-nbd.sh teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-10_21:16:03 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856 2024-09-11T09:39:17.119 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_daaf78de334647fc0c800f4ac495cc9475a26bae/qa; will attempt to use it 2024-09-11T09:39:17.120 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_daaf78de334647fc0c800f4ac495cc9475a26bae/qa/tasks 2024-09-11T09:39:17.120 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-11T09:39:17.122 INFO:teuthology.task.internal:Checking packages... 2024-09-11T09:39:17.142 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc' 2024-09-11T09:39:17.142 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-11T09:39:17.142 INFO:teuthology.packaging:ref: None 2024-09-11T09:39:17.142 INFO:teuthology.packaging:tag: None 2024-09-11T09:39:17.142 INFO:teuthology.packaging:branch: squid 2024-09-11T09:39:17.142 INFO:teuthology.packaging:sha1: b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc 2024-09-11T09:39:17.142 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=squid 2024-09-11T09:39:17.387 INFO:teuthology.task.internal:Found packages for ceph version 19.1.1-308.gdaaf78de 2024-09-11T09:39:17.389 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-11T09:39:17.394 INFO:teuthology.task.internal:no buildpackages task found 2024-09-11T09:39:17.395 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-11T09:39:17.424 INFO:teuthology.task.internal:Saving configuration 2024-09-11T09:39:17.436 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-11T09:39:17.442 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-11T09:39:17.465 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi017.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-09-10_21:16:03-rbd-squid-distro-default-smithi/7900734', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-11 09:34:20.063380', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHneZ92wLS9wEoVFsEJff3TbxllQaVnFBBpd3aMf/OktTHEsayzyKE9etbZlkwKAIw6ZJcpxOKYD23MNb5RAPCk='} 2024-09-11T09:39:17.487 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi096.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-09-10_21:16:03-rbd-squid-distro-default-smithi/7900734', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-11 09:34:20.061610', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNxpPWO64yIr6T4RKgPsPM+OG+4ctP1rBk+N/B/0SvFgl4R4Z4LwFfGE0/chlp8RdvlhhVft6iZMQpNZG6apayA='} 2024-09-11T09:39:17.509 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi107.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-09-10_21:16:03-rbd-squid-distro-default-smithi/7900734', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-11 09:34:20.064495', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBApDXY7Q4ol63+pX3d8bnYgsbpILjg7ASGCTyNn+RP4u2qRUH9wCf/c5DvxJr6BO+tMIZskweZhBwboHNAZ9grA='} 2024-09-11T09:39:17.510 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-11T09:39:17.516 INFO:teuthology.task.internal:roles: ubuntu@smithi017.front.sepia.ceph.com - ['mon.a', 'mon.c', 'osd.0', 'osd.1', 'osd.2'] 2024-09-11T09:39:17.516 INFO:teuthology.task.internal:roles: ubuntu@smithi096.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.3', 'osd.4', 'osd.5'] 2024-09-11T09:39:17.516 INFO:teuthology.task.internal:roles: ubuntu@smithi107.front.sepia.ceph.com - ['client.0'] 2024-09-11T09:39:17.516 INFO:teuthology.run_tasks:Running task console_log... 2024-09-11T09:39:17.599 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f88b6e67ac0>, signals=[15]) 2024-09-11T09:39:17.599 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-11T09:39:17.604 INFO:teuthology.task.internal:Opening connections... 2024-09-11T09:39:17.605 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi017.front.sepia.ceph.com 2024-09-11T09:39:17.606 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi017.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-11T09:39:17.682 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi096.front.sepia.ceph.com 2024-09-11T09:39:17.682 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi096.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-11T09:39:17.758 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi107.front.sepia.ceph.com 2024-09-11T09:39:17.759 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-11T09:39:17.832 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-11T09:39:17.838 DEBUG:teuthology.orchestra.run.smithi017:> uname -m 2024-09-11T09:39:17.856 INFO:teuthology.orchestra.run.smithi017.stdout:x86_64 2024-09-11T09:39:17.857 DEBUG:teuthology.orchestra.run.smithi017:> cat /etc/os-release 2024-09-11T09:39:17.914 INFO:teuthology.orchestra.run.smithi017.stdout:NAME="CentOS Stream" 2024-09-11T09:39:17.914 INFO:teuthology.orchestra.run.smithi017.stdout:VERSION="9" 2024-09-11T09:39:17.914 INFO:teuthology.orchestra.run.smithi017.stdout:ID="centos" 2024-09-11T09:39:17.914 INFO:teuthology.orchestra.run.smithi017.stdout:ID_LIKE="rhel fedora" 2024-09-11T09:39:17.914 INFO:teuthology.orchestra.run.smithi017.stdout:VERSION_ID="9" 2024-09-11T09:39:17.914 INFO:teuthology.orchestra.run.smithi017.stdout:PLATFORM_ID="platform:el9" 2024-09-11T09:39:17.915 INFO:teuthology.orchestra.run.smithi017.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-11T09:39:17.915 INFO:teuthology.orchestra.run.smithi017.stdout:ANSI_COLOR="0;31" 2024-09-11T09:39:17.915 INFO:teuthology.orchestra.run.smithi017.stdout:LOGO="fedora-logo-icon" 2024-09-11T09:39:17.915 INFO:teuthology.orchestra.run.smithi017.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-11T09:39:17.915 INFO:teuthology.orchestra.run.smithi017.stdout:HOME_URL="https://centos.org/" 2024-09-11T09:39:17.915 INFO:teuthology.orchestra.run.smithi017.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-11T09:39:17.915 INFO:teuthology.orchestra.run.smithi017.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-11T09:39:17.915 INFO:teuthology.orchestra.run.smithi017.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-11T09:39:17.916 INFO:teuthology.lock.ops:Updating smithi017.front.sepia.ceph.com on lock server 2024-09-11T09:39:17.937 DEBUG:teuthology.orchestra.run.smithi096:> uname -m 2024-09-11T09:39:17.956 INFO:teuthology.orchestra.run.smithi096.stdout:x86_64 2024-09-11T09:39:17.956 DEBUG:teuthology.orchestra.run.smithi096:> cat /etc/os-release 2024-09-11T09:39:18.013 INFO:teuthology.orchestra.run.smithi096.stdout:NAME="CentOS Stream" 2024-09-11T09:39:18.013 INFO:teuthology.orchestra.run.smithi096.stdout:VERSION="9" 2024-09-11T09:39:18.013 INFO:teuthology.orchestra.run.smithi096.stdout:ID="centos" 2024-09-11T09:39:18.013 INFO:teuthology.orchestra.run.smithi096.stdout:ID_LIKE="rhel fedora" 2024-09-11T09:39:18.013 INFO:teuthology.orchestra.run.smithi096.stdout:VERSION_ID="9" 2024-09-11T09:39:18.013 INFO:teuthology.orchestra.run.smithi096.stdout:PLATFORM_ID="platform:el9" 2024-09-11T09:39:18.014 INFO:teuthology.orchestra.run.smithi096.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-11T09:39:18.014 INFO:teuthology.orchestra.run.smithi096.stdout:ANSI_COLOR="0;31" 2024-09-11T09:39:18.014 INFO:teuthology.orchestra.run.smithi096.stdout:LOGO="fedora-logo-icon" 2024-09-11T09:39:18.014 INFO:teuthology.orchestra.run.smithi096.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-11T09:39:18.014 INFO:teuthology.orchestra.run.smithi096.stdout:HOME_URL="https://centos.org/" 2024-09-11T09:39:18.014 INFO:teuthology.orchestra.run.smithi096.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-11T09:39:18.014 INFO:teuthology.orchestra.run.smithi096.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-11T09:39:18.014 INFO:teuthology.orchestra.run.smithi096.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-11T09:39:18.014 INFO:teuthology.lock.ops:Updating smithi096.front.sepia.ceph.com on lock server 2024-09-11T09:39:18.037 DEBUG:teuthology.orchestra.run.smithi107:> uname -m 2024-09-11T09:39:18.056 INFO:teuthology.orchestra.run.smithi107.stdout:x86_64 2024-09-11T09:39:18.056 DEBUG:teuthology.orchestra.run.smithi107:> cat /etc/os-release 2024-09-11T09:39:18.114 INFO:teuthology.orchestra.run.smithi107.stdout:NAME="CentOS Stream" 2024-09-11T09:39:18.114 INFO:teuthology.orchestra.run.smithi107.stdout:VERSION="9" 2024-09-11T09:39:18.114 INFO:teuthology.orchestra.run.smithi107.stdout:ID="centos" 2024-09-11T09:39:18.114 INFO:teuthology.orchestra.run.smithi107.stdout:ID_LIKE="rhel fedora" 2024-09-11T09:39:18.114 INFO:teuthology.orchestra.run.smithi107.stdout:VERSION_ID="9" 2024-09-11T09:39:18.114 INFO:teuthology.orchestra.run.smithi107.stdout:PLATFORM_ID="platform:el9" 2024-09-11T09:39:18.114 INFO:teuthology.orchestra.run.smithi107.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-11T09:39:18.114 INFO:teuthology.orchestra.run.smithi107.stdout:ANSI_COLOR="0;31" 2024-09-11T09:39:18.115 INFO:teuthology.orchestra.run.smithi107.stdout:LOGO="fedora-logo-icon" 2024-09-11T09:39:18.115 INFO:teuthology.orchestra.run.smithi107.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-11T09:39:18.115 INFO:teuthology.orchestra.run.smithi107.stdout:HOME_URL="https://centos.org/" 2024-09-11T09:39:18.115 INFO:teuthology.orchestra.run.smithi107.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-11T09:39:18.115 INFO:teuthology.orchestra.run.smithi107.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-11T09:39:18.115 INFO:teuthology.orchestra.run.smithi107.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-11T09:39:18.115 INFO:teuthology.lock.ops:Updating smithi107.front.sepia.ceph.com on lock server 2024-09-11T09:39:18.137 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-11T09:39:18.146 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-11T09:39:18.150 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-11T09:39:18.150 DEBUG:teuthology.orchestra.run.smithi017:> test '!' -e /home/ubuntu/cephtest 2024-09-11T09:39:18.153 DEBUG:teuthology.orchestra.run.smithi096:> test '!' -e /home/ubuntu/cephtest 2024-09-11T09:39:18.155 DEBUG:teuthology.orchestra.run.smithi107:> test '!' -e /home/ubuntu/cephtest 2024-09-11T09:39:18.172 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-11T09:39:18.177 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-11T09:39:18.177 DEBUG:teuthology.orchestra.run.smithi017:> test -z $(ls -A /var/lib/ceph) 2024-09-11T09:39:18.210 DEBUG:teuthology.orchestra.run.smithi096:> test -z $(ls -A /var/lib/ceph) 2024-09-11T09:39:18.212 DEBUG:teuthology.orchestra.run.smithi107:> test -z $(ls -A /var/lib/ceph) 2024-09-11T09:39:18.262 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-11T09:39:18.342 INFO:teuthology.run_tasks:Running task kernel... 2024-09-11T09:39:18.355 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-09-11T09:39:18.355 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}, 'mon.b': {'kdb': 1, 'sha1': 'distro'}, 'client.0': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-09-11T09:39:18.356 DEBUG:teuthology.orchestra.run.smithi017:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-11T09:39:18.358 DEBUG:teuthology.orchestra.run.smithi096:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-11T09:39:18.358 DEBUG:teuthology.orchestra.run.smithi107:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-11T09:39:18.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T09:39:18.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T09:39:18.376 DEBUG:teuthology.orchestra.run.smithi096:> uname -r 2024-09-11T09:39:18.377 DEBUG:teuthology.orchestra.run.smithi017:> uname -r 2024-09-11T09:39:18.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T09:39:18.378 DEBUG:teuthology.orchestra.run.smithi107:> uname -r 2024-09-11T09:39:18.434 INFO:teuthology.orchestra.run.smithi096.stdout:5.14.0-503.el9.x86_64 2024-09-11T09:39:18.434 INFO:teuthology.orchestra.run.smithi017.stdout:5.14.0-503.el9.x86_64 2024-09-11T09:39:18.435 INFO:teuthology.task.kernel:Running kernel on smithi096: 5.14.0-503.el9.x86_64 2024-09-11T09:39:18.435 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum install -y kernel 2024-09-11T09:39:18.435 INFO:teuthology.task.kernel:Running kernel on smithi017: 5.14.0-503.el9.x86_64 2024-09-11T09:39:18.435 DEBUG:teuthology.orchestra.run.smithi017:> sudo yum install -y kernel 2024-09-11T09:39:18.436 INFO:teuthology.orchestra.run.smithi107.stdout:5.14.0-503.el9.x86_64 2024-09-11T09:39:18.436 INFO:teuthology.task.kernel:Running kernel on smithi107: 5.14.0-503.el9.x86_64 2024-09-11T09:39:18.437 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum install -y kernel 2024-09-11T09:39:20.937 INFO:teuthology.orchestra.run.smithi096.stdout:CentOS Stream 9 - BaseOS 11 MB/s | 8.2 MB 00:00 2024-09-11T09:39:20.949 INFO:teuthology.orchestra.run.smithi107.stdout:CentOS Stream 9 - BaseOS 12 MB/s | 8.2 MB 00:00 2024-09-11T09:39:22.837 INFO:teuthology.orchestra.run.smithi096.stdout:CentOS Stream 9 - AppStream 27 MB/s | 20 MB 00:00 2024-09-11T09:39:23.001 INFO:teuthology.orchestra.run.smithi107.stdout:CentOS Stream 9 - AppStream 22 MB/s | 20 MB 00:00 2024-09-11T09:39:24.819 INFO:teuthology.orchestra.run.smithi017.stdout:CentOS Stream 9 - BaseOS 1.8 MB/s | 8.2 MB 00:04 2024-09-11T09:39:27.451 INFO:teuthology.orchestra.run.smithi107.stdout:CentOS Stream 9 - CRB 10 MB/s | 6.5 MB 00:00 2024-09-11T09:39:28.362 INFO:teuthology.orchestra.run.smithi096.stdout:CentOS Stream 9 - CRB 3.7 MB/s | 6.5 MB 00:01 2024-09-11T09:39:28.935 INFO:teuthology.orchestra.run.smithi107.stdout:CentOS Stream 9 - Extras packages 71 kB/s | 19 kB 00:00 2024-09-11T09:39:29.681 INFO:teuthology.orchestra.run.smithi107.stdout:Extra Packages for Enterprise Linux 37 MB/s | 23 MB 00:00 2024-09-11T09:39:29.880 INFO:teuthology.orchestra.run.smithi096.stdout:CentOS Stream 9 - Extras packages 58 kB/s | 19 kB 00:00 2024-09-11T09:39:30.070 INFO:teuthology.orchestra.run.smithi017.stdout:CentOS Stream 9 - AppStream 5.0 MB/s | 20 MB 00:04 2024-09-11T09:39:30.735 INFO:teuthology.orchestra.run.smithi096.stdout:Extra Packages for Enterprise Linux 31 MB/s | 23 MB 00:00 2024-09-11T09:39:35.044 INFO:teuthology.orchestra.run.smithi107.stdout:lab-extras 41 kB/s | 1.7 kB 00:00 2024-09-11T09:39:35.820 INFO:teuthology.orchestra.run.smithi017.stdout:CentOS Stream 9 - CRB 3.6 MB/s | 6.5 MB 00:01 2024-09-11T09:39:36.089 INFO:teuthology.orchestra.run.smithi096.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-09-11T09:39:37.017 INFO:teuthology.orchestra.run.smithi107.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-11T09:39:37.018 INFO:teuthology.orchestra.run.smithi107.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-11T09:39:37.104 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-09-11T09:39:37.107 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-09-11T09:39:37.107 INFO:teuthology.orchestra.run.smithi107.stdout: Package Architecture Version Repository Size 2024-09-11T09:39:37.107 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-09-11T09:39:37.107 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2024-09-11T09:39:37.107 INFO:teuthology.orchestra.run.smithi107.stdout: kernel x86_64 5.14.0-505.el9 baseos 23 k 2024-09-11T09:39:37.107 INFO:teuthology.orchestra.run.smithi107.stdout:Installing dependencies: 2024-09-11T09:39:37.107 INFO:teuthology.orchestra.run.smithi107.stdout: kernel-core x86_64 5.14.0-505.el9 baseos 16 M 2024-09-11T09:39:37.108 INFO:teuthology.orchestra.run.smithi107.stdout: kernel-modules x86_64 5.14.0-505.el9 baseos 35 M 2024-09-11T09:39:37.108 INFO:teuthology.orchestra.run.smithi107.stdout: kernel-modules-core x86_64 5.14.0-505.el9 baseos 29 M 2024-09-11T09:39:37.108 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-11T09:39:37.108 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2024-09-11T09:39:37.108 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-09-11T09:39:37.108 INFO:teuthology.orchestra.run.smithi107.stdout:Install 4 Packages 2024-09-11T09:39:37.108 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-11T09:39:37.109 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 79 M 2024-09-11T09:39:37.109 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 126 M 2024-09-11T09:39:37.109 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2024-09-11T09:39:37.363 INFO:teuthology.orchestra.run.smithi017.stdout:CentOS Stream 9 - Extras packages 73 kB/s | 19 kB 00:00 2024-09-11T09:39:37.746 INFO:teuthology.orchestra.run.smithi107.stdout:(1/4): kernel-5.14.0-505.el9.x86_64.rpm 42 kB/s | 23 kB 00:00 2024-09-11T09:39:38.078 INFO:teuthology.orchestra.run.smithi096.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-11T09:39:38.079 INFO:teuthology.orchestra.run.smithi096.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-11T09:39:38.160 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2024-09-11T09:39:38.170 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-09-11T09:39:38.170 INFO:teuthology.orchestra.run.smithi096.stdout: Package Architecture Version Repository Size 2024-09-11T09:39:38.170 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-09-11T09:39:38.171 INFO:teuthology.orchestra.run.smithi096.stdout:Installing: 2024-09-11T09:39:38.171 INFO:teuthology.orchestra.run.smithi096.stdout: kernel x86_64 5.14.0-505.el9 baseos 23 k 2024-09-11T09:39:38.171 INFO:teuthology.orchestra.run.smithi096.stdout:Installing dependencies: 2024-09-11T09:39:38.171 INFO:teuthology.orchestra.run.smithi096.stdout: kernel-core x86_64 5.14.0-505.el9 baseos 16 M 2024-09-11T09:39:38.171 INFO:teuthology.orchestra.run.smithi096.stdout: kernel-modules x86_64 5.14.0-505.el9 baseos 35 M 2024-09-11T09:39:38.171 INFO:teuthology.orchestra.run.smithi096.stdout: kernel-modules-core x86_64 5.14.0-505.el9 baseos 29 M 2024-09-11T09:39:38.171 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-11T09:39:38.171 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction Summary 2024-09-11T09:39:38.171 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-09-11T09:39:38.171 INFO:teuthology.orchestra.run.smithi096.stdout:Install 4 Packages 2024-09-11T09:39:38.171 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-11T09:39:38.172 INFO:teuthology.orchestra.run.smithi096.stdout:Total download size: 79 M 2024-09-11T09:39:38.172 INFO:teuthology.orchestra.run.smithi096.stdout:Installed size: 126 M 2024-09-11T09:39:38.172 INFO:teuthology.orchestra.run.smithi096.stdout:Downloading Packages: 2024-09-11T09:39:38.267 INFO:teuthology.orchestra.run.smithi017.stdout:Extra Packages for Enterprise Linux 29 MB/s | 23 MB 00:00 2024-09-11T09:39:38.737 INFO:teuthology.orchestra.run.smithi107.stdout:(2/4): kernel-core-5.14.0-505.el9.x86_64.rpm 10 MB/s | 16 MB 00:01 2024-09-11T09:39:38.740 INFO:teuthology.orchestra.run.smithi096.stdout:(1/4): kernel-5.14.0-505.el9.x86_64.rpm 78 kB/s | 23 kB 00:00 2024-09-11T09:39:39.271 INFO:teuthology.orchestra.run.smithi107.stdout:(3/4): kernel-modules-5.14.0-505.el9.x86_64.rpm 17 MB/s | 35 MB 00:02 2024-09-11T09:39:39.938 INFO:teuthology.orchestra.run.smithi107.stdout:(4/4): kernel-modules-core-5.14.0-505.el9.x86_6 13 MB/s | 29 MB 00:02 2024-09-11T09:39:39.938 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2024-09-11T09:39:39.939 INFO:teuthology.orchestra.run.smithi107.stdout:Total 28 MB/s | 79 MB 00:02 2024-09-11T09:39:40.625 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2024-09-11T09:39:40.984 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2024-09-11T09:39:40.984 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2024-09-11T09:39:40.990 INFO:teuthology.orchestra.run.smithi096.stdout:(2/4): kernel-modules-core-5.14.0-505.el9.x86_6 13 MB/s | 29 MB 00:02 2024-09-11T09:39:41.307 INFO:teuthology.orchestra.run.smithi096.stdout:(3/4): kernel-modules-5.14.0-505.el9.x86_64.rpm 12 MB/s | 35 MB 00:02 2024-09-11T09:39:41.946 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2024-09-11T09:39:41.947 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2024-09-11T09:39:42.241 INFO:teuthology.orchestra.run.smithi096.stdout:(4/4): kernel-core-5.14.0-505.el9.x86_64.rpm 4.1 MB/s | 16 MB 00:03 2024-09-11T09:39:42.241 INFO:teuthology.orchestra.run.smithi096.stdout:-------------------------------------------------------------------------------- 2024-09-11T09:39:42.241 INFO:teuthology.orchestra.run.smithi096.stdout:Total 19 MB/s | 79 MB 00:04 2024-09-11T09:39:42.924 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction check 2024-09-11T09:39:43.290 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction check succeeded. 2024-09-11T09:39:43.290 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction test 2024-09-11T09:39:43.939 INFO:teuthology.orchestra.run.smithi017.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-09-11T09:39:44.270 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction test succeeded. 2024-09-11T09:39:44.271 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction 2024-09-11T09:39:44.404 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2024-09-11T09:39:45.054 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : kernel-modules-core-5.14.0-505.el9.x86_64 1/4 2024-09-11T09:39:45.146 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-11T09:39:45.963 INFO:teuthology.orchestra.run.smithi017.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-11T09:39:45.963 INFO:teuthology.orchestra.run.smithi017.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-11T09:39:46.080 INFO:teuthology.orchestra.run.smithi017.stdout:Dependencies resolved. 2024-09-11T09:39:46.091 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2024-09-11T09:39:46.091 INFO:teuthology.orchestra.run.smithi017.stdout: Package Architecture Version Repository Size 2024-09-11T09:39:46.092 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2024-09-11T09:39:46.092 INFO:teuthology.orchestra.run.smithi017.stdout:Installing: 2024-09-11T09:39:46.092 INFO:teuthology.orchestra.run.smithi017.stdout: kernel x86_64 5.14.0-505.el9 baseos 23 k 2024-09-11T09:39:46.092 INFO:teuthology.orchestra.run.smithi017.stdout:Installing dependencies: 2024-09-11T09:39:46.092 INFO:teuthology.orchestra.run.smithi017.stdout: kernel-core x86_64 5.14.0-505.el9 baseos 16 M 2024-09-11T09:39:46.092 INFO:teuthology.orchestra.run.smithi017.stdout: kernel-modules x86_64 5.14.0-505.el9 baseos 35 M 2024-09-11T09:39:46.092 INFO:teuthology.orchestra.run.smithi017.stdout: kernel-modules-core x86_64 5.14.0-505.el9 baseos 29 M 2024-09-11T09:39:46.092 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-11T09:39:46.092 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction Summary 2024-09-11T09:39:46.092 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2024-09-11T09:39:46.092 INFO:teuthology.orchestra.run.smithi017.stdout:Install 4 Packages 2024-09-11T09:39:46.093 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-11T09:39:46.093 INFO:teuthology.orchestra.run.smithi017.stdout:Total download size: 79 M 2024-09-11T09:39:46.093 INFO:teuthology.orchestra.run.smithi017.stdout:Installed size: 126 M 2024-09-11T09:39:46.093 INFO:teuthology.orchestra.run.smithi017.stdout:Downloading Packages: 2024-09-11T09:39:46.716 INFO:teuthology.orchestra.run.smithi017.stdout:(1/4): kernel-5.14.0-505.el9.x86_64.rpm 40 kB/s | 23 kB 00:00 2024-09-11T09:39:46.725 INFO:teuthology.orchestra.run.smithi096.stdout: Preparing : 1/1 2024-09-11T09:39:47.443 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : kernel-modules-core-5.14.0-505.el9.x86_64 1/4 2024-09-11T09:39:47.525 INFO:teuthology.orchestra.run.smithi017.stdout:(2/4): kernel-core-5.14.0-505.el9.x86_64.rpm 11 MB/s | 16 MB 00:01 2024-09-11T09:39:47.534 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-11T09:39:47.945 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-11T09:39:48.081 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-11T09:39:48.325 INFO:teuthology.orchestra.run.smithi017.stdout:(3/4): kernel-modules-5.14.0-505.el9.x86_64.rpm 16 MB/s | 35 MB 00:02 2024-09-11T09:39:48.825 INFO:teuthology.orchestra.run.smithi017.stdout:(4/4): kernel-modules-core-5.14.0-505.el9.x86_6 14 MB/s | 29 MB 00:02 2024-09-11T09:39:48.825 INFO:teuthology.orchestra.run.smithi017.stdout:-------------------------------------------------------------------------------- 2024-09-11T09:39:48.825 INFO:teuthology.orchestra.run.smithi017.stdout:Total 29 MB/s | 79 MB 00:02 2024-09-11T09:39:49.539 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction check 2024-09-11T09:39:49.928 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction check succeeded. 2024-09-11T09:39:49.928 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction test 2024-09-11T09:39:50.335 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-11T09:39:50.485 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-11T09:39:50.922 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction test succeeded. 2024-09-11T09:39:50.922 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction 2024-09-11T09:39:52.917 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-11T09:39:53.056 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : kernel-5.14.0-505.el9.x86_64 4/4 2024-09-11T09:39:53.492 INFO:teuthology.orchestra.run.smithi017.stdout: Preparing : 1/1 2024-09-11T09:39:54.153 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : kernel-modules-core-5.14.0-505.el9.x86_64 1/4 2024-09-11T09:39:54.310 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-11T09:39:55.323 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-11T09:39:55.369 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : kernel-5.14.0-505.el9.x86_64 4/4 2024-09-11T09:39:57.095 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-11T09:39:57.219 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-11T09:39:57.934 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-11T09:40:00.222 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-11T09:40:02.349 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-11T09:40:02.394 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : kernel-5.14.0-505.el9.x86_64 4/4 2024-09-11T09:40:07.535 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-11T09:40:36.011 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 4/4 2024-09-11T09:40:36.052 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 4/4 2024-09-11T09:40:36.325 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: kernel-5.14.0-505.el9.x86_64 4/4 2024-09-11T09:40:36.325 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : kernel-5.14.0-505.el9.x86_64 1/4 2024-09-11T09:40:36.325 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-11T09:40:36.325 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-11T09:40:37.168 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-11T09:40:37.168 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-11T09:40:37.168 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2024-09-11T09:40:37.168 INFO:teuthology.orchestra.run.smithi107.stdout: kernel-5.14.0-505.el9.x86_64 2024-09-11T09:40:37.168 INFO:teuthology.orchestra.run.smithi107.stdout: kernel-core-5.14.0-505.el9.x86_64 2024-09-11T09:40:37.168 INFO:teuthology.orchestra.run.smithi107.stdout: kernel-modules-5.14.0-505.el9.x86_64 2024-09-11T09:40:37.168 INFO:teuthology.orchestra.run.smithi107.stdout: kernel-modules-core-5.14.0-505.el9.x86_64 2024-09-11T09:40:37.169 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-11T09:40:37.169 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-09-11T09:40:37.523 DEBUG:teuthology.orchestra.run.smithi107:> rpm -q kernel | sort -rV | head -n 1 2024-09-11T09:40:37.612 INFO:teuthology.orchestra.run.smithi107.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-11T09:40:37.613 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-11T09:40:37.613 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-505.el9.x86_64 2024-09-11T09:40:37.613 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-11T09:40:37.613 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-11T09:40:37.613 INFO:teuthology.task.kernel:Installing distro kernel on client.0... 2024-09-11T09:40:37.613 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi107.front.sepia.ceph.com, path=None, version=distro) 2024-09-11T09:40:37.614 DEBUG:teuthology.orchestra.run.smithi107:> rpm -q kernel | sort -rV | head -n 1 2024-09-11T09:40:37.671 INFO:teuthology.orchestra.run.smithi107.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-11T09:40:37.672 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-11T09:40:37.672 DEBUG:teuthology.orchestra.run.smithi107:> sudo rpm -qi grub2-tools 2024-09-11T09:40:37.704 INFO:teuthology.orchestra.run.smithi107.stdout:Name : grub2-tools 2024-09-11T09:40:37.704 INFO:teuthology.orchestra.run.smithi107.stdout:Epoch : 1 2024-09-11T09:40:37.704 INFO:teuthology.orchestra.run.smithi107.stdout:Version : 2.06 2024-09-11T09:40:37.704 INFO:teuthology.orchestra.run.smithi107.stdout:Release : 82.el9 2024-09-11T09:40:37.704 INFO:teuthology.orchestra.run.smithi107.stdout:Architecture: x86_64 2024-09-11T09:40:37.704 INFO:teuthology.orchestra.run.smithi107.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-11T09:40:37.704 INFO:teuthology.orchestra.run.smithi107.stdout:Group : Unspecified 2024-09-11T09:40:37.704 INFO:teuthology.orchestra.run.smithi107.stdout:Size : 8274670 2024-09-11T09:40:37.704 INFO:teuthology.orchestra.run.smithi107.stdout:License : GPLv3+ 2024-09-11T09:40:37.704 INFO:teuthology.orchestra.run.smithi107.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-11T09:40:37.705 INFO:teuthology.orchestra.run.smithi107.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-11T09:40:37.705 INFO:teuthology.orchestra.run.smithi107.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-11T09:40:37.705 INFO:teuthology.orchestra.run.smithi107.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-11T09:40:37.705 INFO:teuthology.orchestra.run.smithi107.stdout:Packager : builder@centos.org 2024-09-11T09:40:37.705 INFO:teuthology.orchestra.run.smithi107.stdout:Vendor : CentOS 2024-09-11T09:40:37.705 INFO:teuthology.orchestra.run.smithi107.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-11T09:40:37.705 INFO:teuthology.orchestra.run.smithi107.stdout:Summary : Support tools for GRUB. 2024-09-11T09:40:37.705 INFO:teuthology.orchestra.run.smithi107.stdout:Description : 2024-09-11T09:40:37.705 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-11T09:40:37.705 INFO:teuthology.orchestra.run.smithi107.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-11T09:40:37.705 INFO:teuthology.orchestra.run.smithi107.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-11T09:40:37.705 INFO:teuthology.orchestra.run.smithi107.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-11T09:40:37.705 INFO:teuthology.orchestra.run.smithi107.stdout:hardware devices. 2024-09-11T09:40:37.705 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-11T09:40:37.705 INFO:teuthology.orchestra.run.smithi107.stdout:This subpackage provides tools for support of all platforms. 2024-09-11T09:40:37.706 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-11T09:40:37.706 INFO:teuthology.task.kernel:Updating grub on smithi107 to boot 5.14.0-505.el9.x86_64 2024-09-11T09:40:37.707 DEBUG:teuthology.orchestra.run.smithi107:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-11T09:40:38.306 INFO:teuthology.orchestra.run.smithi107.stderr:Generating grub configuration file ... 2024-09-11T09:40:38.573 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 4/4 2024-09-11T09:40:38.613 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 4/4 2024-09-11T09:40:39.020 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: kernel-5.14.0-505.el9.x86_64 4/4 2024-09-11T09:40:39.021 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : kernel-5.14.0-505.el9.x86_64 1/4 2024-09-11T09:40:39.021 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-11T09:40:39.021 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-11T09:40:39.639 INFO:teuthology.orchestra.run.smithi107.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-11T09:40:39.668 INFO:teuthology.orchestra.run.smithi107.stderr:done 2024-09-11T09:40:39.677 DEBUG:teuthology.orchestra.run.smithi107:> mktemp 2024-09-11T09:40:39.692 INFO:teuthology.orchestra.run.smithi107.stdout:/tmp/tmp.sPXKs3XEeU 2024-09-11T09:40:39.693 DEBUG:teuthology.orchestra.run.smithi107:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.sPXKs3XEeU 2024-09-11T09:40:39.759 DEBUG:teuthology.orchestra.run.smithi107:> sudo chmod 0666 /tmp/tmp.sPXKs3XEeU 2024-09-11T09:40:39.890 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-11T09:40:39.890 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-11T09:40:39.890 INFO:teuthology.orchestra.run.smithi096.stdout:Installed: 2024-09-11T09:40:39.890 INFO:teuthology.orchestra.run.smithi096.stdout: kernel-5.14.0-505.el9.x86_64 2024-09-11T09:40:39.891 INFO:teuthology.orchestra.run.smithi096.stdout: kernel-core-5.14.0-505.el9.x86_64 2024-09-11T09:40:39.891 INFO:teuthology.orchestra.run.smithi096.stdout: kernel-modules-5.14.0-505.el9.x86_64 2024-09-11T09:40:39.891 INFO:teuthology.orchestra.run.smithi096.stdout: kernel-modules-core-5.14.0-505.el9.x86_64 2024-09-11T09:40:39.891 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-11T09:40:39.891 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2024-09-11T09:40:39.909 DEBUG:teuthology.orchestra.remote:smithi107:/tmp/tmp.sPXKs3XEeU is 6KB 2024-09-11T09:40:39.957 DEBUG:teuthology.orchestra.run.smithi107:> rm -fr /tmp/tmp.sPXKs3XEeU 2024-09-11T09:40:39.972 DEBUG:teuthology.orchestra.run.smithi107:> sudo /bin/ls /boot/loader/entries || true 2024-09-11T09:40:40.034 INFO:teuthology.orchestra.run.smithi107.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-11T09:40:40.035 INFO:teuthology.orchestra.run.smithi107.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-11T09:40:40.035 INFO:teuthology.orchestra.run.smithi107.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-11T09:40:40.035 INFO:teuthology.orchestra.run.smithi107.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64.conf 2024-09-11T09:40:40.036 DEBUG:teuthology.orchestra.run.smithi107:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64 2024-09-11T09:40:40.195 DEBUG:teuthology.orchestra.run.smithi107:> sudo shutdown -r now 2024-09-11T09:40:40.253 DEBUG:teuthology.orchestra.run.smithi096:> rpm -q kernel | sort -rV | head -n 1 2024-09-11T09:40:40.336 INFO:teuthology.orchestra.run.smithi096.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-11T09:40:40.337 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-11T09:40:40.337 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-505.el9.x86_64 2024-09-11T09:40:40.337 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-11T09:40:40.337 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-11T09:40:40.337 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-09-11T09:40:40.337 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi096.front.sepia.ceph.com, path=None, version=distro) 2024-09-11T09:40:40.337 DEBUG:teuthology.orchestra.run.smithi096:> rpm -q kernel | sort -rV | head -n 1 2024-09-11T09:40:40.395 INFO:teuthology.orchestra.run.smithi096.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-11T09:40:40.396 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-11T09:40:40.396 DEBUG:teuthology.orchestra.run.smithi096:> sudo rpm -qi grub2-tools 2024-09-11T09:40:40.428 INFO:teuthology.orchestra.run.smithi096.stdout:Name : grub2-tools 2024-09-11T09:40:40.428 INFO:teuthology.orchestra.run.smithi096.stdout:Epoch : 1 2024-09-11T09:40:40.428 INFO:teuthology.orchestra.run.smithi096.stdout:Version : 2.06 2024-09-11T09:40:40.429 INFO:teuthology.orchestra.run.smithi096.stdout:Release : 82.el9 2024-09-11T09:40:40.429 INFO:teuthology.orchestra.run.smithi096.stdout:Architecture: x86_64 2024-09-11T09:40:40.429 INFO:teuthology.orchestra.run.smithi096.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-11T09:40:40.429 INFO:teuthology.orchestra.run.smithi096.stdout:Group : Unspecified 2024-09-11T09:40:40.429 INFO:teuthology.orchestra.run.smithi096.stdout:Size : 8274670 2024-09-11T09:40:40.429 INFO:teuthology.orchestra.run.smithi096.stdout:License : GPLv3+ 2024-09-11T09:40:40.429 INFO:teuthology.orchestra.run.smithi096.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-11T09:40:40.429 INFO:teuthology.orchestra.run.smithi096.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-11T09:40:40.429 INFO:teuthology.orchestra.run.smithi096.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-11T09:40:40.429 INFO:teuthology.orchestra.run.smithi096.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-11T09:40:40.429 INFO:teuthology.orchestra.run.smithi096.stdout:Packager : builder@centos.org 2024-09-11T09:40:40.430 INFO:teuthology.orchestra.run.smithi096.stdout:Vendor : CentOS 2024-09-11T09:40:40.430 INFO:teuthology.orchestra.run.smithi096.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-11T09:40:40.430 INFO:teuthology.orchestra.run.smithi096.stdout:Summary : Support tools for GRUB. 2024-09-11T09:40:40.430 INFO:teuthology.orchestra.run.smithi096.stdout:Description : 2024-09-11T09:40:40.430 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-11T09:40:40.430 INFO:teuthology.orchestra.run.smithi096.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-11T09:40:40.430 INFO:teuthology.orchestra.run.smithi096.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-11T09:40:40.430 INFO:teuthology.orchestra.run.smithi096.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-11T09:40:40.430 INFO:teuthology.orchestra.run.smithi096.stdout:hardware devices. 2024-09-11T09:40:40.430 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-11T09:40:40.430 INFO:teuthology.orchestra.run.smithi096.stdout:This subpackage provides tools for support of all platforms. 2024-09-11T09:40:40.431 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-11T09:40:40.431 INFO:teuthology.task.kernel:Updating grub on smithi096 to boot 5.14.0-505.el9.x86_64 2024-09-11T09:40:40.432 DEBUG:teuthology.orchestra.run.smithi096:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-11T09:40:41.117 INFO:teuthology.orchestra.run.smithi096.stderr:Generating grub configuration file ... 2024-09-11T09:40:42.492 INFO:teuthology.orchestra.run.smithi096.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-11T09:40:42.535 INFO:teuthology.orchestra.run.smithi096.stderr:done 2024-09-11T09:40:42.537 DEBUG:teuthology.orchestra.run.smithi096:> mktemp 2024-09-11T09:40:42.553 INFO:teuthology.orchestra.run.smithi096.stdout:/tmp/tmp.SouXAq1tRm 2024-09-11T09:40:42.553 DEBUG:teuthology.orchestra.run.smithi096:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.SouXAq1tRm 2024-09-11T09:40:42.624 DEBUG:teuthology.orchestra.run.smithi096:> sudo chmod 0666 /tmp/tmp.SouXAq1tRm 2024-09-11T09:40:42.769 DEBUG:teuthology.orchestra.remote:smithi096:/tmp/tmp.SouXAq1tRm is 6KB 2024-09-11T09:40:42.817 DEBUG:teuthology.orchestra.run.smithi096:> rm -fr /tmp/tmp.SouXAq1tRm 2024-09-11T09:40:42.832 DEBUG:teuthology.orchestra.run.smithi096:> sudo /bin/ls /boot/loader/entries || true 2024-09-11T09:40:42.894 INFO:teuthology.orchestra.run.smithi096.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-11T09:40:42.895 INFO:teuthology.orchestra.run.smithi096.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-11T09:40:42.895 INFO:teuthology.orchestra.run.smithi096.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-11T09:40:42.895 INFO:teuthology.orchestra.run.smithi096.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64.conf 2024-09-11T09:40:42.896 DEBUG:teuthology.orchestra.run.smithi096:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64 2024-09-11T09:40:43.040 DEBUG:teuthology.orchestra.run.smithi096:> sudo shutdown -r now 2024-09-11T09:40:46.239 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 4/4 2024-09-11T09:40:46.280 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 4/4 2024-09-11T09:40:46.651 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: kernel-5.14.0-505.el9.x86_64 4/4 2024-09-11T09:40:46.652 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : kernel-5.14.0-505.el9.x86_64 1/4 2024-09-11T09:40:46.652 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-11T09:40:46.652 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-11T09:40:47.494 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-11T09:40:47.494 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-11T09:40:47.494 INFO:teuthology.orchestra.run.smithi017.stdout:Installed: 2024-09-11T09:40:47.494 INFO:teuthology.orchestra.run.smithi017.stdout: kernel-5.14.0-505.el9.x86_64 2024-09-11T09:40:47.494 INFO:teuthology.orchestra.run.smithi017.stdout: kernel-core-5.14.0-505.el9.x86_64 2024-09-11T09:40:47.494 INFO:teuthology.orchestra.run.smithi017.stdout: kernel-modules-5.14.0-505.el9.x86_64 2024-09-11T09:40:47.494 INFO:teuthology.orchestra.run.smithi017.stdout: kernel-modules-core-5.14.0-505.el9.x86_64 2024-09-11T09:40:47.494 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-11T09:40:47.494 INFO:teuthology.orchestra.run.smithi017.stdout:Complete! 2024-09-11T09:40:47.837 DEBUG:teuthology.orchestra.run.smithi017:> rpm -q kernel | sort -rV | head -n 1 2024-09-11T09:40:47.922 INFO:teuthology.orchestra.run.smithi017.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-11T09:40:47.923 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-11T09:40:47.923 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-505.el9.x86_64 2024-09-11T09:40:47.923 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-11T09:40:47.923 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-11T09:40:47.923 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-09-11T09:40:47.923 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi017.front.sepia.ceph.com, path=None, version=distro) 2024-09-11T09:40:47.923 DEBUG:teuthology.orchestra.run.smithi017:> rpm -q kernel | sort -rV | head -n 1 2024-09-11T09:40:47.982 INFO:teuthology.orchestra.run.smithi017.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-11T09:40:47.982 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-11T09:40:47.982 DEBUG:teuthology.orchestra.run.smithi017:> sudo rpm -qi grub2-tools 2024-09-11T09:40:48.013 INFO:teuthology.orchestra.run.smithi017.stdout:Name : grub2-tools 2024-09-11T09:40:48.013 INFO:teuthology.orchestra.run.smithi017.stdout:Epoch : 1 2024-09-11T09:40:48.013 INFO:teuthology.orchestra.run.smithi017.stdout:Version : 2.06 2024-09-11T09:40:48.014 INFO:teuthology.orchestra.run.smithi017.stdout:Release : 82.el9 2024-09-11T09:40:48.014 INFO:teuthology.orchestra.run.smithi017.stdout:Architecture: x86_64 2024-09-11T09:40:48.014 INFO:teuthology.orchestra.run.smithi017.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-11T09:40:48.014 INFO:teuthology.orchestra.run.smithi017.stdout:Group : Unspecified 2024-09-11T09:40:48.014 INFO:teuthology.orchestra.run.smithi017.stdout:Size : 8274670 2024-09-11T09:40:48.014 INFO:teuthology.orchestra.run.smithi017.stdout:License : GPLv3+ 2024-09-11T09:40:48.014 INFO:teuthology.orchestra.run.smithi017.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-11T09:40:48.014 INFO:teuthology.orchestra.run.smithi017.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-11T09:40:48.014 INFO:teuthology.orchestra.run.smithi017.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-11T09:40:48.014 INFO:teuthology.orchestra.run.smithi017.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-11T09:40:48.014 INFO:teuthology.orchestra.run.smithi017.stdout:Packager : builder@centos.org 2024-09-11T09:40:48.014 INFO:teuthology.orchestra.run.smithi017.stdout:Vendor : CentOS 2024-09-11T09:40:48.014 INFO:teuthology.orchestra.run.smithi017.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-11T09:40:48.015 INFO:teuthology.orchestra.run.smithi017.stdout:Summary : Support tools for GRUB. 2024-09-11T09:40:48.015 INFO:teuthology.orchestra.run.smithi017.stdout:Description : 2024-09-11T09:40:48.015 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-11T09:40:48.015 INFO:teuthology.orchestra.run.smithi017.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-11T09:40:48.015 INFO:teuthology.orchestra.run.smithi017.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-11T09:40:48.015 INFO:teuthology.orchestra.run.smithi017.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-11T09:40:48.015 INFO:teuthology.orchestra.run.smithi017.stdout:hardware devices. 2024-09-11T09:40:48.015 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-11T09:40:48.015 INFO:teuthology.orchestra.run.smithi017.stdout:This subpackage provides tools for support of all platforms. 2024-09-11T09:40:48.016 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-11T09:40:48.016 INFO:teuthology.task.kernel:Updating grub on smithi017 to boot 5.14.0-505.el9.x86_64 2024-09-11T09:40:48.016 DEBUG:teuthology.orchestra.run.smithi017:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-11T09:40:48.692 INFO:teuthology.orchestra.run.smithi017.stderr:Generating grub configuration file ... 2024-09-11T09:40:49.950 INFO:teuthology.orchestra.run.smithi017.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-11T09:40:49.981 INFO:teuthology.orchestra.run.smithi017.stderr:done 2024-09-11T09:40:49.984 DEBUG:teuthology.orchestra.run.smithi017:> mktemp 2024-09-11T09:40:50.023 INFO:teuthology.orchestra.run.smithi017.stdout:/tmp/tmp.kppovzql4Q 2024-09-11T09:40:50.023 DEBUG:teuthology.orchestra.run.smithi017:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.kppovzql4Q 2024-09-11T09:40:50.088 DEBUG:teuthology.orchestra.run.smithi017:> sudo chmod 0666 /tmp/tmp.kppovzql4Q 2024-09-11T09:40:50.224 DEBUG:teuthology.orchestra.remote:smithi017:/tmp/tmp.kppovzql4Q is 6KB 2024-09-11T09:40:50.273 DEBUG:teuthology.orchestra.run.smithi017:> rm -fr /tmp/tmp.kppovzql4Q 2024-09-11T09:40:50.287 DEBUG:teuthology.orchestra.run.smithi017:> sudo /bin/ls /boot/loader/entries || true 2024-09-11T09:40:50.350 INFO:teuthology.orchestra.run.smithi017.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-11T09:40:50.350 INFO:teuthology.orchestra.run.smithi017.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-11T09:40:50.350 INFO:teuthology.orchestra.run.smithi017.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-11T09:40:50.350 INFO:teuthology.orchestra.run.smithi017.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64.conf 2024-09-11T09:40:50.352 DEBUG:teuthology.orchestra.run.smithi017:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64 2024-09-11T09:40:50.502 DEBUG:teuthology.orchestra.run.smithi017:> sudo shutdown -r now 2024-09-11T09:41:10.217 INFO:teuthology.task.kernel:Checking client client.0 for new kernel version... 2024-09-11T09:41:10.218 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi107.front.sepia.ceph.com' 2024-09-11T09:41:10.219 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-11T09:41:13.045 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-09-11T09:41:13.045 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi096.front.sepia.ceph.com' 2024-09-11T09:41:13.046 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi096.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-11T09:41:20.513 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-09-11T09:41:20.513 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi017.front.sepia.ceph.com' 2024-09-11T09:41:20.514 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi017.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-11T09:41:28.769 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.107 2024-09-11T09:41:31.581 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.96 2024-09-11T09:41:37.774 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi107.front.sepia.ceph.com' 2024-09-11T09:41:37.775 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-11T09:41:39.005 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.17 2024-09-11T09:41:40.583 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi096.front.sepia.ceph.com' 2024-09-11T09:41:40.584 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi096.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-11T09:41:40.829 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.107 2024-09-11T09:41:43.649 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.96 2024-09-11T09:41:48.009 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi017.front.sepia.ceph.com' 2024-09-11T09:41:48.010 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi017.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-11T09:41:52.833 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi107.front.sepia.ceph.com' 2024-09-11T09:41:52.834 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-11T09:41:55.652 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi096.front.sepia.ceph.com' 2024-09-11T09:41:55.653 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi096.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-11T09:42:11.265 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.107 2024-09-11T09:42:14.080 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.96 2024-09-11T09:42:26.278 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi107.front.sepia.ceph.com' 2024-09-11T09:42:26.280 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-11T09:42:26.627 DEBUG:teuthology.orchestra.run.smithi107:> true 2024-09-11T09:42:26.962 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi107.front.sepia.ceph.com' 2024-09-11T09:42:26.963 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "5.14.0-505.el9.x86_64"... 2024-09-11T09:42:26.963 DEBUG:teuthology.orchestra.run.smithi107:> uname -r 2024-09-11T09:42:26.977 INFO:teuthology.orchestra.run.smithi107.stdout:5.14.0-505.el9.x86_64 2024-09-11T09:42:26.977 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-505.el9.x86_64 vs 5.14.0-505.el9.x86_64 2024-09-11T09:42:26.977 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-11T09:42:26.978 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-11T09:42:27.978 INFO:teuthology.task.kernel:Enabling kdb on client.0... 2024-09-11T09:42:27.978 DEBUG:teuthology.orchestra.run.smithi107:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-11T09:42:28.084 INFO:teuthology.orchestra.run.smithi107.stdout:ttyS1 2024-09-11T09:42:28.104 DEBUG:teuthology.parallel:result is None 2024-09-11T09:42:29.081 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi096.front.sepia.ceph.com' 2024-09-11T09:42:29.082 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi096.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-11T09:42:29.328 DEBUG:teuthology.orchestra.run.smithi096:> true 2024-09-11T09:42:30.012 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi096.front.sepia.ceph.com' 2024-09-11T09:42:30.012 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-505.el9.x86_64"... 2024-09-11T09:42:30.012 DEBUG:teuthology.orchestra.run.smithi096:> uname -r 2024-09-11T09:42:30.064 INFO:teuthology.orchestra.run.smithi096.stdout:5.14.0-505.el9.x86_64 2024-09-11T09:42:30.064 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-505.el9.x86_64 vs 5.14.0-505.el9.x86_64 2024-09-11T09:42:30.064 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-11T09:42:30.064 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-11T09:42:31.066 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-09-11T09:42:31.066 DEBUG:teuthology.orchestra.run.smithi096:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-11T09:42:31.451 INFO:teuthology.orchestra.run.smithi096.stdout:ttyS1 2024-09-11T09:42:31.484 DEBUG:teuthology.parallel:result is None 2024-09-11T09:42:48.012 DEBUG:teuthology.orchestra.remote:timed out 2024-09-11T09:43:00.013 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi017.front.sepia.ceph.com' 2024-09-11T09:43:00.014 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi017.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-11T09:43:00.339 DEBUG:teuthology.orchestra.run.smithi017:> true 2024-09-11T09:43:00.665 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi017.front.sepia.ceph.com' 2024-09-11T09:43:00.665 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-505.el9.x86_64"... 2024-09-11T09:43:00.666 DEBUG:teuthology.orchestra.run.smithi017:> uname -r 2024-09-11T09:43:00.683 INFO:teuthology.orchestra.run.smithi017.stdout:5.14.0-505.el9.x86_64 2024-09-11T09:43:00.683 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-505.el9.x86_64 vs 5.14.0-505.el9.x86_64 2024-09-11T09:43:00.683 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-11T09:43:00.683 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-11T09:43:01.684 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-09-11T09:43:01.685 DEBUG:teuthology.orchestra.run.smithi017:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-11T09:43:01.769 INFO:teuthology.orchestra.run.smithi017.stdout:ttyS1 2024-09-11T09:43:01.788 DEBUG:teuthology.parallel:result is None 2024-09-11T09:43:01.788 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-11T09:43:01.796 INFO:teuthology.task.internal:Creating test directory... 2024-09-11T09:43:01.796 DEBUG:teuthology.orchestra.run.smithi017:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-11T09:43:01.799 DEBUG:teuthology.orchestra.run.smithi096:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-11T09:43:01.801 DEBUG:teuthology.orchestra.run.smithi107:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-11T09:43:01.820 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-11T09:43:01.827 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-11T09:43:01.833 INFO:teuthology.task.internal:Creating archive directory... 2024-09-11T09:43:01.833 DEBUG:teuthology.orchestra.run.smithi017:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-11T09:43:01.859 DEBUG:teuthology.orchestra.run.smithi096:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-11T09:43:01.861 DEBUG:teuthology.orchestra.run.smithi107:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-11T09:43:01.945 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-11T09:43:01.951 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-11T09:43:01.952 DEBUG:teuthology.orchestra.run.smithi017:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-11T09:43:01.954 DEBUG:teuthology.orchestra.run.smithi096:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-11T09:43:01.957 DEBUG:teuthology.orchestra.run.smithi107:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-11T09:43:01.989 INFO:teuthology.orchestra.run.smithi107.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-11T09:43:01.991 INFO:teuthology.orchestra.run.smithi017.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-11T09:43:01.992 INFO:teuthology.orchestra.run.smithi096.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-11T09:43:02.000 INFO:teuthology.orchestra.run.smithi107.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-11T09:43:02.003 INFO:teuthology.orchestra.run.smithi017.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-11T09:43:02.004 INFO:teuthology.orchestra.run.smithi096.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-11T09:43:02.006 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-11T09:43:02.013 INFO:teuthology.task.internal:Configuring sudo... 2024-09-11T09:43:02.013 DEBUG:teuthology.orchestra.run.smithi017:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-11T09:43:02.048 DEBUG:teuthology.orchestra.run.smithi096:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-11T09:43:02.050 DEBUG:teuthology.orchestra.run.smithi107:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-11T09:43:02.078 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-11T09:43:02.086 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-11T09:43:02.087 DEBUG:teuthology.orchestra.run.smithi017:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-11T09:43:02.118 DEBUG:teuthology.orchestra.run.smithi096:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-11T09:43:02.120 DEBUG:teuthology.orchestra.run.smithi107:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-11T09:43:02.136 DEBUG:teuthology.orchestra.run.smithi017:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-11T09:43:02.220 DEBUG:teuthology.orchestra.run.smithi017:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-11T09:43:02.271 DEBUG:teuthology.orchestra.run.smithi017:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-11T09:43:02.355 DEBUG:teuthology.orchestra.run.smithi017:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-11T09:43:02.397 DEBUG:teuthology.orchestra.run.smithi017:> set -ex 2024-09-11T09:43:02.398 DEBUG:teuthology.orchestra.run.smithi017:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-11T09:43:02.465 DEBUG:teuthology.orchestra.run.smithi096:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-11T09:43:02.512 DEBUG:teuthology.orchestra.run.smithi096:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-11T09:43:02.563 DEBUG:teuthology.orchestra.run.smithi096:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-11T09:43:02.649 DEBUG:teuthology.orchestra.run.smithi096:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-11T09:43:02.691 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-09-11T09:43:02.691 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-11T09:43:02.757 DEBUG:teuthology.orchestra.run.smithi107:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-11T09:43:02.802 DEBUG:teuthology.orchestra.run.smithi107:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-11T09:43:02.862 DEBUG:teuthology.orchestra.run.smithi107:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-11T09:43:02.946 DEBUG:teuthology.orchestra.run.smithi107:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-11T09:43:02.990 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-09-11T09:43:02.990 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-11T09:43:03.057 DEBUG:teuthology.orchestra.run.smithi017:> sudo service rsyslog restart 2024-09-11T09:43:03.059 DEBUG:teuthology.orchestra.run.smithi096:> sudo service rsyslog restart 2024-09-11T09:43:03.062 DEBUG:teuthology.orchestra.run.smithi107:> sudo service rsyslog restart 2024-09-11T09:43:03.110 INFO:teuthology.orchestra.run.smithi096.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-11T09:43:03.111 INFO:teuthology.orchestra.run.smithi017.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-11T09:43:03.147 INFO:teuthology.orchestra.run.smithi107.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-11T09:43:03.615 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-11T09:43:03.623 INFO:teuthology.task.internal:Starting timer... 2024-09-11T09:43:03.623 INFO:teuthology.run_tasks:Running task pcp... 2024-09-11T09:43:03.634 INFO:teuthology.run_tasks:Running task selinux... 2024-09-11T09:43:03.643 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:getty_t:s0']} 2024-09-11T09:43:03.644 DEBUG:teuthology.orchestra.run.smithi017:> sudo service auditd rotate 2024-09-11T09:43:03.702 INFO:teuthology.orchestra.run.smithi017.stdout:Rotating logs: 2024-09-11T09:43:03.703 DEBUG:teuthology.orchestra.run.smithi096:> sudo service auditd rotate 2024-09-11T09:43:03.758 INFO:teuthology.orchestra.run.smithi096.stdout:Rotating logs: 2024-09-11T09:43:03.760 DEBUG:teuthology.orchestra.run.smithi107:> sudo service auditd rotate 2024-09-11T09:43:03.827 INFO:teuthology.orchestra.run.smithi107.stdout:Rotating logs: 2024-09-11T09:43:03.829 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-11T09:43:03.829 DEBUG:teuthology.orchestra.run.smithi017:> /usr/sbin/getenforce 2024-09-11T09:43:03.854 INFO:teuthology.orchestra.run.smithi017.stdout:Permissive 2024-09-11T09:43:03.854 DEBUG:teuthology.orchestra.run.smithi096:> /usr/sbin/getenforce 2024-09-11T09:43:03.880 INFO:teuthology.orchestra.run.smithi096.stdout:Permissive 2024-09-11T09:43:03.881 DEBUG:teuthology.orchestra.run.smithi107:> /usr/sbin/getenforce 2024-09-11T09:43:03.912 INFO:teuthology.orchestra.run.smithi107.stdout:Permissive 2024-09-11T09:43:03.913 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi017.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi096.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi107.front.sepia.ceph.com': 'permissive'} 2024-09-11T09:43:03.913 DEBUG:teuthology.orchestra.run.smithi017:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-09-11T09:43:03.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T09:43:03.944 DEBUG:teuthology.orchestra.run.smithi096:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-09-11T09:43:03.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T09:43:03.976 DEBUG:teuthology.orchestra.run.smithi107:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-09-11T09:43:04.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T09:43:04.005 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-11T09:43:04.005 DEBUG:teuthology.orchestra.run.smithi017:> sudo /usr/sbin/setenforce permissive 2024-09-11T09:43:04.062 DEBUG:teuthology.orchestra.run.smithi096:> sudo /usr/sbin/setenforce permissive 2024-09-11T09:43:04.094 DEBUG:teuthology.orchestra.run.smithi107:> sudo /usr/sbin/setenforce permissive 2024-09-11T09:43:04.124 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-11T09:43:04.134 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-09-11T09:43:04.137 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-09-11T09:43:04.249 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-11T09:43:04.260 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-11T09:43:04.262 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi017.front.sepia.ceph.com,smithi096.front.sepia.ceph.com,smithi107.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-11T09:46:53.767 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi017.front.sepia.ceph.com'), Remote(name='ubuntu@smithi096.front.sepia.ceph.com'), Remote(name='ubuntu@smithi107.front.sepia.ceph.com')] 2024-09-11T09:46:53.769 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi017.front.sepia.ceph.com' 2024-09-11T09:46:53.771 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi017.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-11T09:46:53.847 DEBUG:teuthology.orchestra.run.smithi017:> true 2024-09-11T09:46:53.932 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi017.front.sepia.ceph.com' 2024-09-11T09:46:53.932 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi096.front.sepia.ceph.com' 2024-09-11T09:46:53.933 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi096.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-11T09:46:54.012 DEBUG:teuthology.orchestra.run.smithi096:> true 2024-09-11T09:46:54.093 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi096.front.sepia.ceph.com' 2024-09-11T09:46:54.093 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi107.front.sepia.ceph.com' 2024-09-11T09:46:54.094 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-11T09:46:54.168 DEBUG:teuthology.orchestra.run.smithi107:> true 2024-09-11T09:46:54.244 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi107.front.sepia.ceph.com' 2024-09-11T09:46:54.245 INFO:teuthology.run_tasks:Running task clock... 2024-09-11T09:46:54.256 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-11T09:46:54.256 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-11T09:46:54.257 DEBUG:teuthology.orchestra.run.smithi017:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-11T09:46:54.259 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-11T09:46:54.260 DEBUG:teuthology.orchestra.run.smithi096:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-11T09:46:54.262 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-11T09:46:54.262 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-11T09:46:54.289 INFO:teuthology.orchestra.run.smithi017.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-11T09:46:54.291 INFO:teuthology.orchestra.run.smithi096.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-11T09:46:54.304 INFO:teuthology.orchestra.run.smithi017.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-11T09:46:54.306 INFO:teuthology.orchestra.run.smithi096.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-11T09:46:54.313 INFO:teuthology.orchestra.run.smithi107.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-11T09:46:54.328 INFO:teuthology.orchestra.run.smithi107.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-11T09:46:54.330 INFO:teuthology.orchestra.run.smithi096.stderr:sudo: ntpd: command not found 2024-09-11T09:46:54.331 INFO:teuthology.orchestra.run.smithi017.stderr:sudo: ntpd: command not found 2024-09-11T09:46:54.341 INFO:teuthology.orchestra.run.smithi096.stdout:506 Cannot talk to daemon 2024-09-11T09:46:54.343 INFO:teuthology.orchestra.run.smithi017.stdout:506 Cannot talk to daemon 2024-09-11T09:46:54.352 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: ntpd: command not found 2024-09-11T09:46:54.355 INFO:teuthology.orchestra.run.smithi096.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-11T09:46:54.357 INFO:teuthology.orchestra.run.smithi017.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-11T09:46:54.363 INFO:teuthology.orchestra.run.smithi107.stdout:506 Cannot talk to daemon 2024-09-11T09:46:54.370 INFO:teuthology.orchestra.run.smithi096.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-11T09:46:54.372 INFO:teuthology.orchestra.run.smithi017.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-11T09:46:54.377 INFO:teuthology.orchestra.run.smithi107.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-11T09:46:54.391 INFO:teuthology.orchestra.run.smithi107.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-11T09:46:54.425 INFO:teuthology.orchestra.run.smithi096.stderr:bash: line 1: ntpq: command not found 2024-09-11T09:46:54.429 INFO:teuthology.orchestra.run.smithi096.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-11T09:46:54.429 INFO:teuthology.orchestra.run.smithi096.stdout:=============================================================================== 2024-09-11T09:46:54.429 INFO:teuthology.orchestra.run.smithi096.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-11T09:46:54.429 INFO:teuthology.orchestra.run.smithi096.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-11T09:46:54.429 INFO:teuthology.orchestra.run.smithi096.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-11T09:46:54.429 INFO:teuthology.orchestra.run.smithi096.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-11T09:46:54.433 INFO:teuthology.orchestra.run.smithi017.stderr:bash: line 1: ntpq: command not found 2024-09-11T09:46:54.436 INFO:teuthology.orchestra.run.smithi017.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-11T09:46:54.436 INFO:teuthology.orchestra.run.smithi017.stdout:=============================================================================== 2024-09-11T09:46:54.436 INFO:teuthology.orchestra.run.smithi017.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-11T09:46:54.436 INFO:teuthology.orchestra.run.smithi017.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-11T09:46:54.437 INFO:teuthology.orchestra.run.smithi017.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-11T09:46:54.437 INFO:teuthology.orchestra.run.smithi017.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-11T09:46:54.446 INFO:teuthology.orchestra.run.smithi107.stderr:bash: line 1: ntpq: command not found 2024-09-11T09:46:54.449 INFO:teuthology.orchestra.run.smithi107.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-11T09:46:54.449 INFO:teuthology.orchestra.run.smithi107.stdout:=============================================================================== 2024-09-11T09:46:54.449 INFO:teuthology.orchestra.run.smithi107.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-11T09:46:54.449 INFO:teuthology.orchestra.run.smithi107.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-11T09:46:54.450 INFO:teuthology.orchestra.run.smithi107.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-11T09:46:54.450 INFO:teuthology.orchestra.run.smithi107.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-11T09:46:54.450 INFO:teuthology.run_tasks:Running task install... 2024-09-11T09:46:54.458 DEBUG:teuthology.task.install:project ceph 2024-09-11T09:46:54.458 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'extra_packages': ['rbd-nbd'], 'flavor': 'default', 'sha1': 'b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc'}} 2024-09-11T09:46:54.458 DEBUG:teuthology.task.install:config {'extra_packages': ['rbd-nbd'], 'flavor': 'default', 'sha1': 'b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc'} 2024-09-11T09:46:54.458 INFO:teuthology.task.install:Using flavor: default 2024-09-11T09:46:54.463 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-11T09:46:54.463 INFO:teuthology.task.install:extra packages: ['rbd-nbd'] 2024-09-11T09:46:54.464 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': ['rbd-nbd'], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc', 'tag': None, 'wait_for_package': False} 2024-09-11T09:46:54.464 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc 2024-09-11T09:46:54.466 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': ['rbd-nbd'], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc', 'tag': None, 'wait_for_package': False} 2024-09-11T09:46:54.467 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc 2024-09-11T09:46:54.468 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': ['rbd-nbd'], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc', 'tag': None, 'wait_for_package': False} 2024-09-11T09:46:54.468 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc 2024-09-11T09:46:54.612 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/centos/9/flavors/default/ 2024-09-11T09:46:54.613 INFO:teuthology.task.install.rpm:Package version is 19.1.1-298.gb53b6d3e 2024-09-11T09:46:54.616 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/centos/9/flavors/default/ 2024-09-11T09:46:54.616 INFO:teuthology.task.install.rpm:Package version is 19.1.1-298.gb53b6d3e 2024-09-11T09:46:54.619 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/centos/9/flavors/default/ 2024-09-11T09:46:54.619 INFO:teuthology.task.install.rpm:Package version is 19.1.1-298.gb53b6d3e 2024-09-11T09:46:54.829 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-11T09:46:54.829 DEBUG:teuthology.orchestra.run.smithi017:> set -ex 2024-09-11T09:46:54.829 DEBUG:teuthology.orchestra.run.smithi017:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-11T09:46:54.837 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-11T09:46:54.837 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-09-11T09:46:54.837 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-11T09:46:54.845 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-11T09:46:54.845 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-09-11T09:46:54.845 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-11T09:46:54.857 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, rbd-nbd on remote rpm x86_64 2024-09-11T09:46:54.857 DEBUG:teuthology.orchestra.run.smithi017:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-11T09:46:54.864 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, rbd-nbd on remote rpm x86_64 2024-09-11T09:46:54.864 DEBUG:teuthology.orchestra.run.smithi096:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-11T09:46:54.872 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, rbd-nbd on remote rpm x86_64 2024-09-11T09:46:54.872 DEBUG:teuthology.orchestra.run.smithi107:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-11T09:46:54.925 DEBUG:teuthology.orchestra.run.smithi017:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-11T09:46:54.930 DEBUG:teuthology.orchestra.run.smithi096:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-11T09:46:54.938 DEBUG:teuthology.orchestra.run.smithi107:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-11T09:46:55.002 DEBUG:teuthology.orchestra.run.smithi017:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-11T09:46:55.008 DEBUG:teuthology.orchestra.run.smithi096:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-11T09:46:55.013 DEBUG:teuthology.orchestra.run.smithi107:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-11T09:46:55.067 INFO:teuthology.orchestra.run.smithi017.stdout:check_obsoletes = 1 2024-09-11T09:46:55.069 DEBUG:teuthology.orchestra.run.smithi017:> sudo yum clean all 2024-09-11T09:46:55.072 INFO:teuthology.orchestra.run.smithi096.stdout:check_obsoletes = 1 2024-09-11T09:46:55.074 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum clean all 2024-09-11T09:46:55.077 INFO:teuthology.orchestra.run.smithi107.stdout:check_obsoletes = 1 2024-09-11T09:46:55.078 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum clean all 2024-09-11T09:46:55.303 INFO:teuthology.orchestra.run.smithi096.stdout:45 files removed 2024-09-11T09:46:55.305 INFO:teuthology.orchestra.run.smithi107.stdout:45 files removed 2024-09-11T09:46:55.313 INFO:teuthology.orchestra.run.smithi017.stdout:45 files removed 2024-09-11T09:46:55.330 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum -y install ceph-radosgw 2024-09-11T09:46:55.332 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install ceph-radosgw 2024-09-11T09:46:55.339 DEBUG:teuthology.orchestra.run.smithi017:> sudo yum -y install ceph-radosgw 2024-09-11T09:46:55.875 INFO:teuthology.orchestra.run.smithi107.stdout:ceph packages for x86_64 319 kB/s | 84 kB 00:00 2024-09-11T09:46:55.881 INFO:teuthology.orchestra.run.smithi096.stdout:ceph packages for x86_64 314 kB/s | 84 kB 00:00 2024-09-11T09:46:55.918 INFO:teuthology.orchestra.run.smithi017.stdout:ceph packages for x86_64 297 kB/s | 84 kB 00:00 2024-09-11T09:46:56.167 INFO:teuthology.orchestra.run.smithi107.stdout:ceph noarch packages 47 kB/s | 12 kB 00:00 2024-09-11T09:46:56.172 INFO:teuthology.orchestra.run.smithi096.stdout:ceph noarch packages 47 kB/s | 12 kB 00:00 2024-09-11T09:46:56.203 INFO:teuthology.orchestra.run.smithi017.stdout:ceph noarch packages 49 kB/s | 12 kB 00:00 2024-09-11T09:46:56.449 INFO:teuthology.orchestra.run.smithi096.stdout:ceph source packages 7.9 kB/s | 1.9 kB 00:00 2024-09-11T09:46:56.451 INFO:teuthology.orchestra.run.smithi107.stdout:ceph source packages 7.7 kB/s | 1.9 kB 00:00 2024-09-11T09:46:56.469 INFO:teuthology.orchestra.run.smithi017.stdout:ceph source packages 8.3 kB/s | 1.9 kB 00:00 2024-09-11T09:46:57.123 INFO:teuthology.orchestra.run.smithi096.stdout:CentOS Stream 9 - BaseOS 13 MB/s | 8.2 MB 00:00 2024-09-11T09:46:57.360 INFO:teuthology.orchestra.run.smithi107.stdout:CentOS Stream 9 - BaseOS 9.4 MB/s | 8.2 MB 00:00 2024-09-11T09:46:58.405 INFO:teuthology.orchestra.run.smithi017.stdout:CentOS Stream 9 - BaseOS 4.3 MB/s | 8.2 MB 00:01 2024-09-11T09:46:59.550 INFO:teuthology.orchestra.run.smithi096.stdout:CentOS Stream 9 - AppStream 15 MB/s | 20 MB 00:01 2024-09-11T09:47:00.302 INFO:teuthology.orchestra.run.smithi107.stdout:CentOS Stream 9 - AppStream 11 MB/s | 20 MB 00:01 2024-09-11T09:47:00.823 INFO:teuthology.orchestra.run.smithi017.stdout:CentOS Stream 9 - AppStream 16 MB/s | 20 MB 00:01 2024-09-11T09:47:04.117 INFO:teuthology.orchestra.run.smithi096.stdout:CentOS Stream 9 - CRB 8.7 MB/s | 6.5 MB 00:00 2024-09-11T09:47:04.627 INFO:teuthology.orchestra.run.smithi107.stdout:CentOS Stream 9 - CRB 12 MB/s | 6.5 MB 00:00 2024-09-11T09:47:05.416 INFO:teuthology.orchestra.run.smithi017.stdout:CentOS Stream 9 - CRB 10 MB/s | 6.5 MB 00:00 2024-09-11T09:47:06.318 INFO:teuthology.orchestra.run.smithi096.stdout:CentOS Stream 9 - Extras packages 19 kB/s | 19 kB 00:00 2024-09-11T09:47:06.995 INFO:teuthology.orchestra.run.smithi107.stdout:CentOS Stream 9 - Extras packages 18 kB/s | 19 kB 00:01 2024-09-11T09:47:07.100 INFO:teuthology.orchestra.run.smithi017.stdout:CentOS Stream 9 - Extras packages 45 kB/s | 19 kB 00:00 2024-09-11T09:47:07.405 INFO:teuthology.orchestra.run.smithi096.stdout:Extra Packages for Enterprise Linux 24 MB/s | 23 MB 00:00 2024-09-11T09:47:07.990 INFO:teuthology.orchestra.run.smithi107.stdout:Extra Packages for Enterprise Linux 26 MB/s | 23 MB 00:00 2024-09-11T09:47:08.079 INFO:teuthology.orchestra.run.smithi017.stdout:Extra Packages for Enterprise Linux 27 MB/s | 23 MB 00:00 2024-09-11T09:47:12.869 INFO:teuthology.orchestra.run.smithi096.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2024-09-11T09:47:13.479 INFO:teuthology.orchestra.run.smithi107.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-09-11T09:47:13.685 INFO:teuthology.orchestra.run.smithi017.stdout:lab-extras 41 kB/s | 1.7 kB 00:00 2024-09-11T09:47:15.279 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2024-09-11T09:47:15.295 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-09-11T09:47:15.295 INFO:teuthology.orchestra.run.smithi096.stdout: Package Arch Version Repository Size 2024-09-11T09:47:15.295 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-09-11T09:47:15.295 INFO:teuthology.orchestra.run.smithi096.stdout:Installing: 2024-09-11T09:47:15.295 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-radosgw x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 11 M 2024-09-11T09:47:15.295 INFO:teuthology.orchestra.run.smithi096.stdout:Upgrading: 2024-09-11T09:47:15.296 INFO:teuthology.orchestra.run.smithi096.stdout: librados2 x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 3.4 M 2024-09-11T09:47:15.296 INFO:teuthology.orchestra.run.smithi096.stdout: librbd1 x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 3.2 M 2024-09-11T09:47:15.296 INFO:teuthology.orchestra.run.smithi096.stdout:Installing dependencies: 2024-09-11T09:47:15.296 INFO:teuthology.orchestra.run.smithi096.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-11T09:47:15.296 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-base x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 5.4 M 2024-09-11T09:47:15.296 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-common x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 21 M 2024-09-11T09:47:15.296 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-selinux x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 25 k 2024-09-11T09:47:15.296 INFO:teuthology.orchestra.run.smithi096.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-09-11T09:47:15.296 INFO:teuthology.orchestra.run.smithi096.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-11T09:47:15.296 INFO:teuthology.orchestra.run.smithi096.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-11T09:47:15.297 INFO:teuthology.orchestra.run.smithi096.stdout: libcephfs2 x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 752 k 2024-09-11T09:47:15.297 INFO:teuthology.orchestra.run.smithi096.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-11T09:47:15.297 INFO:teuthology.orchestra.run.smithi096.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-11T09:47:15.297 INFO:teuthology.orchestra.run.smithi096.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-11T09:47:15.297 INFO:teuthology.orchestra.run.smithi096.stdout: libradosstriper1 x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 506 k 2024-09-11T09:47:15.297 INFO:teuthology.orchestra.run.smithi096.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-11T09:47:15.297 INFO:teuthology.orchestra.run.smithi096.stdout: librgw2 x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 5.3 M 2024-09-11T09:47:15.297 INFO:teuthology.orchestra.run.smithi096.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-11T09:47:15.297 INFO:teuthology.orchestra.run.smithi096.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-11T09:47:15.297 INFO:teuthology.orchestra.run.smithi096.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-11T09:47:15.297 INFO:teuthology.orchestra.run.smithi096.stdout: python3-ceph-argparse x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 45 k 2024-09-11T09:47:15.297 INFO:teuthology.orchestra.run.smithi096.stdout: python3-ceph-common x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 136 k 2024-09-11T09:47:15.298 INFO:teuthology.orchestra.run.smithi096.stdout: python3-cephfs x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 162 k 2024-09-11T09:47:15.298 INFO:teuthology.orchestra.run.smithi096.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-11T09:47:15.298 INFO:teuthology.orchestra.run.smithi096.stdout: python3-rados x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 321 k 2024-09-11T09:47:15.298 INFO:teuthology.orchestra.run.smithi096.stdout: python3-rbd x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 301 k 2024-09-11T09:47:15.298 INFO:teuthology.orchestra.run.smithi096.stdout: python3-rgw x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 100 k 2024-09-11T09:47:15.298 INFO:teuthology.orchestra.run.smithi096.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-09-11T09:47:15.298 INFO:teuthology.orchestra.run.smithi096.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-09-11T09:47:15.298 INFO:teuthology.orchestra.run.smithi096.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-11T09:47:15.298 INFO:teuthology.orchestra.run.smithi096.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-11T09:47:15.298 INFO:teuthology.orchestra.run.smithi096.stdout:Installing weak dependencies: 2024-09-11T09:47:15.298 INFO:teuthology.orchestra.run.smithi096.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-09-11T09:47:15.299 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-11T09:47:15.299 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction Summary 2024-09-11T09:47:15.299 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-09-11T09:47:15.299 INFO:teuthology.orchestra.run.smithi096.stdout:Install 30 Packages 2024-09-11T09:47:15.299 INFO:teuthology.orchestra.run.smithi096.stdout:Upgrade 2 Packages 2024-09-11T09:47:15.299 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-11T09:47:15.300 INFO:teuthology.orchestra.run.smithi096.stdout:Total download size: 60 M 2024-09-11T09:47:15.300 INFO:teuthology.orchestra.run.smithi096.stdout:Downloading Packages: 2024-09-11T09:47:15.838 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-09-11T09:47:15.848 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-09-11T09:47:15.848 INFO:teuthology.orchestra.run.smithi107.stdout: Package Arch Version Repository Size 2024-09-11T09:47:15.849 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-09-11T09:47:15.849 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2024-09-11T09:47:15.849 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-radosgw x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 11 M 2024-09-11T09:47:15.849 INFO:teuthology.orchestra.run.smithi107.stdout:Upgrading: 2024-09-11T09:47:15.849 INFO:teuthology.orchestra.run.smithi107.stdout: librados2 x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 3.4 M 2024-09-11T09:47:15.849 INFO:teuthology.orchestra.run.smithi107.stdout: librbd1 x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 3.2 M 2024-09-11T09:47:15.849 INFO:teuthology.orchestra.run.smithi107.stdout:Installing dependencies: 2024-09-11T09:47:15.849 INFO:teuthology.orchestra.run.smithi107.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-11T09:47:15.849 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-base x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 5.4 M 2024-09-11T09:47:15.849 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-common x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 21 M 2024-09-11T09:47:15.849 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-selinux x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 25 k 2024-09-11T09:47:15.849 INFO:teuthology.orchestra.run.smithi107.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-09-11T09:47:15.849 INFO:teuthology.orchestra.run.smithi107.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-11T09:47:15.850 INFO:teuthology.orchestra.run.smithi107.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-11T09:47:15.850 INFO:teuthology.orchestra.run.smithi107.stdout: libcephfs2 x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 752 k 2024-09-11T09:47:15.850 INFO:teuthology.orchestra.run.smithi107.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-11T09:47:15.850 INFO:teuthology.orchestra.run.smithi107.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-11T09:47:15.850 INFO:teuthology.orchestra.run.smithi107.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-11T09:47:15.850 INFO:teuthology.orchestra.run.smithi107.stdout: libradosstriper1 x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 506 k 2024-09-11T09:47:15.850 INFO:teuthology.orchestra.run.smithi107.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-11T09:47:15.850 INFO:teuthology.orchestra.run.smithi107.stdout: librgw2 x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 5.3 M 2024-09-11T09:47:15.850 INFO:teuthology.orchestra.run.smithi107.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-11T09:47:15.850 INFO:teuthology.orchestra.run.smithi107.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-11T09:47:15.850 INFO:teuthology.orchestra.run.smithi107.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-11T09:47:15.850 INFO:teuthology.orchestra.run.smithi107.stdout: python3-ceph-argparse x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 45 k 2024-09-11T09:47:15.851 INFO:teuthology.orchestra.run.smithi107.stdout: python3-ceph-common x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 136 k 2024-09-11T09:47:15.851 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cephfs x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 162 k 2024-09-11T09:47:15.851 INFO:teuthology.orchestra.run.smithi107.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-11T09:47:15.851 INFO:teuthology.orchestra.run.smithi107.stdout: python3-rados x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 321 k 2024-09-11T09:47:15.851 INFO:teuthology.orchestra.run.smithi107.stdout: python3-rbd x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 301 k 2024-09-11T09:47:15.851 INFO:teuthology.orchestra.run.smithi107.stdout: python3-rgw x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 100 k 2024-09-11T09:47:15.851 INFO:teuthology.orchestra.run.smithi107.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-09-11T09:47:15.851 INFO:teuthology.orchestra.run.smithi107.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-09-11T09:47:15.851 INFO:teuthology.orchestra.run.smithi107.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-11T09:47:15.851 INFO:teuthology.orchestra.run.smithi107.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-11T09:47:15.852 INFO:teuthology.orchestra.run.smithi107.stdout:Installing weak dependencies: 2024-09-11T09:47:15.852 INFO:teuthology.orchestra.run.smithi107.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-09-11T09:47:15.852 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-11T09:47:15.852 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2024-09-11T09:47:15.852 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-09-11T09:47:15.852 INFO:teuthology.orchestra.run.smithi107.stdout:Install 30 Packages 2024-09-11T09:47:15.852 INFO:teuthology.orchestra.run.smithi107.stdout:Upgrade 2 Packages 2024-09-11T09:47:15.852 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-11T09:47:15.852 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 60 M 2024-09-11T09:47:15.853 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2024-09-11T09:47:16.123 INFO:teuthology.orchestra.run.smithi017.stdout:Dependencies resolved. 2024-09-11T09:47:16.135 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2024-09-11T09:47:16.136 INFO:teuthology.orchestra.run.smithi017.stdout: Package Arch Version Repository Size 2024-09-11T09:47:16.136 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2024-09-11T09:47:16.136 INFO:teuthology.orchestra.run.smithi017.stdout:Installing: 2024-09-11T09:47:16.136 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-radosgw x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 11 M 2024-09-11T09:47:16.136 INFO:teuthology.orchestra.run.smithi017.stdout:Upgrading: 2024-09-11T09:47:16.136 INFO:teuthology.orchestra.run.smithi017.stdout: librados2 x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 3.4 M 2024-09-11T09:47:16.136 INFO:teuthology.orchestra.run.smithi017.stdout: librbd1 x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 3.2 M 2024-09-11T09:47:16.136 INFO:teuthology.orchestra.run.smithi017.stdout:Installing dependencies: 2024-09-11T09:47:16.136 INFO:teuthology.orchestra.run.smithi017.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-11T09:47:16.136 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-base x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 5.4 M 2024-09-11T09:47:16.136 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-common x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 21 M 2024-09-11T09:47:16.136 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-selinux x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 25 k 2024-09-11T09:47:16.137 INFO:teuthology.orchestra.run.smithi017.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-09-11T09:47:16.137 INFO:teuthology.orchestra.run.smithi017.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-11T09:47:16.137 INFO:teuthology.orchestra.run.smithi017.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-11T09:47:16.137 INFO:teuthology.orchestra.run.smithi017.stdout: libcephfs2 x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 752 k 2024-09-11T09:47:16.137 INFO:teuthology.orchestra.run.smithi017.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-11T09:47:16.137 INFO:teuthology.orchestra.run.smithi017.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-11T09:47:16.137 INFO:teuthology.orchestra.run.smithi017.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-11T09:47:16.137 INFO:teuthology.orchestra.run.smithi017.stdout: libradosstriper1 x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 506 k 2024-09-11T09:47:16.137 INFO:teuthology.orchestra.run.smithi017.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-11T09:47:16.137 INFO:teuthology.orchestra.run.smithi017.stdout: librgw2 x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 5.3 M 2024-09-11T09:47:16.137 INFO:teuthology.orchestra.run.smithi017.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-11T09:47:16.138 INFO:teuthology.orchestra.run.smithi017.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-11T09:47:16.138 INFO:teuthology.orchestra.run.smithi017.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-11T09:47:16.138 INFO:teuthology.orchestra.run.smithi017.stdout: python3-ceph-argparse x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 45 k 2024-09-11T09:47:16.138 INFO:teuthology.orchestra.run.smithi017.stdout: python3-ceph-common x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 136 k 2024-09-11T09:47:16.138 INFO:teuthology.orchestra.run.smithi017.stdout: python3-cephfs x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 162 k 2024-09-11T09:47:16.138 INFO:teuthology.orchestra.run.smithi017.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-11T09:47:16.138 INFO:teuthology.orchestra.run.smithi017.stdout: python3-rados x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 321 k 2024-09-11T09:47:16.138 INFO:teuthology.orchestra.run.smithi017.stdout: python3-rbd x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 301 k 2024-09-11T09:47:16.138 INFO:teuthology.orchestra.run.smithi017.stdout: python3-rgw x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 100 k 2024-09-11T09:47:16.138 INFO:teuthology.orchestra.run.smithi017.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-09-11T09:47:16.138 INFO:teuthology.orchestra.run.smithi017.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-09-11T09:47:16.138 INFO:teuthology.orchestra.run.smithi017.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-11T09:47:16.138 INFO:teuthology.orchestra.run.smithi017.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-11T09:47:16.139 INFO:teuthology.orchestra.run.smithi017.stdout:Installing weak dependencies: 2024-09-11T09:47:16.139 INFO:teuthology.orchestra.run.smithi017.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-09-11T09:47:16.139 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-11T09:47:16.139 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction Summary 2024-09-11T09:47:16.139 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2024-09-11T09:47:16.139 INFO:teuthology.orchestra.run.smithi017.stdout:Install 30 Packages 2024-09-11T09:47:16.139 INFO:teuthology.orchestra.run.smithi017.stdout:Upgrade 2 Packages 2024-09-11T09:47:16.139 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-11T09:47:16.139 INFO:teuthology.orchestra.run.smithi017.stdout:Total download size: 60 M 2024-09-11T09:47:16.139 INFO:teuthology.orchestra.run.smithi017.stdout:Downloading Packages: 2024-09-11T09:47:16.204 INFO:teuthology.orchestra.run.smithi096.stdout:(1/32): ceph-base-19.1.1-298.gb53b6d3e.el9.x86_ 9.2 MB/s | 5.4 MB 00:00 2024-09-11T09:47:16.254 INFO:teuthology.orchestra.run.smithi096.stdout:(2/32): ceph-selinux-19.1.1-298.gb53b6d3e.el9.x 506 kB/s | 25 kB 00:00 2024-09-11T09:47:16.363 INFO:teuthology.orchestra.run.smithi096.stdout:(3/32): libcephfs2-19.1.1-298.gb53b6d3e.el9.x86 6.8 MB/s | 752 kB 00:00 2024-09-11T09:47:16.446 INFO:teuthology.orchestra.run.smithi096.stdout:(4/32): libradosstriper1-19.1.1-298.gb53b6d3e.e 5.9 MB/s | 506 kB 00:00 2024-09-11T09:47:16.597 INFO:teuthology.orchestra.run.smithi096.stdout:(5/32): ceph-radosgw-19.1.1-298.gb53b6d3e.el9.x 11 MB/s | 11 MB 00:00 2024-09-11T09:47:16.813 INFO:teuthology.orchestra.run.smithi096.stdout:(6/32): ceph-common-19.1.1-298.gb53b6d3e.el9.x8 18 MB/s | 21 MB 00:01 2024-09-11T09:47:16.839 INFO:teuthology.orchestra.run.smithi096.stdout:(7/32): python3-ceph-argparse-19.1.1-298.gb53b6 186 kB/s | 45 kB 00:00 2024-09-11T09:47:16.914 INFO:teuthology.orchestra.run.smithi096.stdout:(8/32): python3-ceph-common-19.1.1-298.gb53b6d3 1.3 MB/s | 136 kB 00:00 2024-09-11T09:47:17.128 INFO:teuthology.orchestra.run.smithi017.stdout:(1/32): ceph-base-19.1.1-298.gb53b6d3e.el9.x86_ 8.8 MB/s | 5.4 MB 00:00 2024-09-11T09:47:17.195 INFO:teuthology.orchestra.run.smithi017.stdout:(2/32): ceph-selinux-19.1.1-298.gb53b6d3e.el9.x 379 kB/s | 25 kB 00:00 2024-09-11T09:47:17.354 INFO:teuthology.orchestra.run.smithi017.stdout:(3/32): ceph-radosgw-19.1.1-298.gb53b6d3e.el9.x 13 MB/s | 11 MB 00:00 2024-09-11T09:47:17.373 INFO:teuthology.orchestra.run.smithi096.stdout:(9/32): python3-cephfs-19.1.1-298.gb53b6d3e.el9 304 kB/s | 162 kB 00:00 2024-09-11T09:47:17.387 INFO:teuthology.orchestra.run.smithi017.stdout:(4/32): libcephfs2-19.1.1-298.gb53b6d3e.el9.x86 3.8 MB/s | 752 kB 00:00 2024-09-11T09:47:17.421 INFO:teuthology.orchestra.run.smithi017.stdout:(5/32): libradosstriper1-19.1.1-298.gb53b6d3e.e 7.4 MB/s | 506 kB 00:00 2024-09-11T09:47:17.431 INFO:teuthology.orchestra.run.smithi096.stdout:(10/32): python3-rbd-19.1.1-298.gb53b6d3e.el9.x 5.1 MB/s | 301 kB 00:00 2024-09-11T09:47:17.496 INFO:teuthology.orchestra.run.smithi017.stdout:(6/32): python3-ceph-argparse-19.1.1-298.gb53b6 601 kB/s | 45 kB 00:00 2024-09-11T09:47:17.507 INFO:teuthology.orchestra.run.smithi096.stdout:(11/32): python3-rgw-19.1.1-298.gb53b6d3e.el9.x 1.3 MB/s | 100 kB 00:00 2024-09-11T09:47:17.557 INFO:teuthology.orchestra.run.smithi096.stdout:(12/32): python3-rados-19.1.1-298.gb53b6d3e.el9 500 kB/s | 321 kB 00:00 2024-09-11T09:47:17.571 INFO:teuthology.orchestra.run.smithi017.stdout:(7/32): python3-ceph-common-19.1.1-298.gb53b6d3 1.8 MB/s | 136 kB 00:00 2024-09-11T09:47:17.655 INFO:teuthology.orchestra.run.smithi017.stdout:(8/32): python3-cephfs-19.1.1-298.gb53b6d3e.el9 1.9 MB/s | 162 kB 00:00 2024-09-11T09:47:17.724 INFO:teuthology.orchestra.run.smithi096.stdout:(13/32): librgw2-19.1.1-298.gb53b6d3e.el9.x86_6 4.2 MB/s | 5.3 MB 00:01 2024-09-11T09:47:17.764 INFO:teuthology.orchestra.run.smithi017.stdout:(9/32): librgw2-19.1.1-298.gb53b6d3e.el9.x86_64 14 MB/s | 5.3 MB 00:00 2024-09-11T09:47:17.789 INFO:teuthology.orchestra.run.smithi017.stdout:(10/32): python3-rados-19.1.1-298.gb53b6d3e.el9 2.3 MB/s | 321 kB 00:00 2024-09-11T09:47:17.814 INFO:teuthology.orchestra.run.smithi017.stdout:(11/32): python3-rbd-19.1.1-298.gb53b6d3e.el9.x 5.9 MB/s | 301 kB 00:00 2024-09-11T09:47:17.839 INFO:teuthology.orchestra.run.smithi017.stdout:(12/32): python3-rgw-19.1.1-298.gb53b6d3e.el9.x 1.9 MB/s | 100 kB 00:00 2024-09-11T09:47:17.946 INFO:teuthology.orchestra.run.smithi107.stdout:(1/32): ceph-base-19.1.1-298.gb53b6d3e.el9.x86_ 4.0 MB/s | 5.4 MB 00:01 2024-09-11T09:47:17.996 INFO:teuthology.orchestra.run.smithi107.stdout:(2/32): ceph-selinux-19.1.1-298.gb53b6d3e.el9.x 507 kB/s | 25 kB 00:00 2024-09-11T09:47:18.040 INFO:teuthology.orchestra.run.smithi017.stdout:(13/32): ceph-common-19.1.1-298.gb53b6d3e.el9.x 14 MB/s | 21 MB 00:01 2024-09-11T09:47:18.097 INFO:teuthology.orchestra.run.smithi107.stdout:(3/32): libcephfs2-19.1.1-298.gb53b6d3e.el9.x86 7.4 MB/s | 752 kB 00:00 2024-09-11T09:47:18.098 INFO:teuthology.orchestra.run.smithi017.stdout:(14/32): boost-program-options-1.75.0-8.el9.x86 377 kB/s | 107 kB 00:00 2024-09-11T09:47:18.124 INFO:teuthology.orchestra.run.smithi017.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 543 kB/s | 45 kB 00:00 2024-09-11T09:47:18.172 INFO:teuthology.orchestra.run.smithi107.stdout:(4/32): libradosstriper1-19.1.1-298.gb53b6d3e.e 6.6 MB/s | 506 kB 00:00 2024-09-11T09:47:18.174 INFO:teuthology.orchestra.run.smithi017.stdout:(16/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 480 kB/s | 160 kB 00:00 2024-09-11T09:47:18.191 INFO:teuthology.orchestra.run.smithi096.stdout:(14/32): boost-program-options-1.75.0-8.el9.x86 156 kB/s | 107 kB 00:00 2024-09-11T09:47:18.199 INFO:teuthology.orchestra.run.smithi017.stdout:(17/32): librdkafka-1.6.1-102.el9.x86_64.rpm 6.4 MB/s | 662 kB 00:00 2024-09-11T09:47:18.216 INFO:teuthology.orchestra.run.smithi017.stdout:(18/32): python3-prettytable-0.7.2-27.el9.noarc 1.0 MB/s | 42 kB 00:00 2024-09-11T09:47:18.241 INFO:teuthology.orchestra.run.smithi017.stdout:(19/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.4 MB/s | 292 kB 00:00 2024-09-11T09:47:18.266 INFO:teuthology.orchestra.run.smithi017.stdout:(20/32): qatlib-service-24.02.0-1.el9.x86_64.rp 716 kB/s | 36 kB 00:00 2024-09-11T09:47:18.292 INFO:teuthology.orchestra.run.smithi017.stdout:(21/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 941 kB/s | 47 kB 00:00 2024-09-11T09:47:18.317 INFO:teuthology.orchestra.run.smithi017.stdout:(22/32): qatlib-24.02.0-1.el9.x86_64.rpm 1.8 MB/s | 221 kB 00:00 2024-09-11T09:47:18.334 INFO:teuthology.orchestra.run.smithi096.stdout:(15/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 206 kB/s | 160 kB 00:00 2024-09-11T09:47:18.358 INFO:teuthology.orchestra.run.smithi096.stdout:(16/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 72 kB/s | 45 kB 00:00 2024-09-11T09:47:18.376 INFO:teuthology.orchestra.run.smithi017.stdout:(23/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 2.8 MB/s | 309 kB 00:00 2024-09-11T09:47:18.401 INFO:teuthology.orchestra.run.smithi017.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 2.1 MB/s | 49 kB 00:00 2024-09-11T09:47:18.426 INFO:teuthology.orchestra.run.smithi017.stdout:(25/32): libarrow-doc-9.0.0-11.el9.noarch.rpm 279 kB/s | 26 kB 00:00 2024-09-11T09:47:18.460 INFO:teuthology.orchestra.run.smithi017.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 1.1 MB/s | 67 kB 00:00 2024-09-11T09:47:18.485 INFO:teuthology.orchestra.run.smithi017.stdout:(27/32): re2-20211101-20.el9.x86_64.rpm 7.6 MB/s | 191 kB 00:00 2024-09-11T09:47:18.535 INFO:teuthology.orchestra.run.smithi017.stdout:(28/32): parquet-libs-9.0.0-11.el9.x86_64.rpm 7.5 MB/s | 839 kB 00:00 2024-09-11T09:47:18.572 INFO:teuthology.orchestra.run.smithi107.stdout:(5/32): ceph-common-19.1.1-298.gb53b6d3e.el9.x8 11 MB/s | 21 MB 00:01 2024-09-11T09:47:18.586 INFO:teuthology.orchestra.run.smithi017.stdout:(29/32): thrift-0.15.0-2.el9.x86_64.rpm 16 MB/s | 1.6 MB 00:00 2024-09-11T09:47:18.656 INFO:teuthology.orchestra.run.smithi107.stdout:(6/32): librgw2-19.1.1-298.gb53b6d3e.el9.x86_64 11 MB/s | 5.3 MB 00:00 2024-09-11T09:47:18.669 INFO:teuthology.orchestra.run.smithi017.stdout:(30/32): libarrow-9.0.0-11.el9.x86_64.rpm 12 MB/s | 4.4 MB 00:00 2024-09-11T09:47:18.681 INFO:teuthology.orchestra.run.smithi107.stdout:(7/32): python3-ceph-argparse-19.1.1-298.gb53b6 417 kB/s | 45 kB 00:00 2024-09-11T09:47:18.706 INFO:teuthology.orchestra.run.smithi107.stdout:(8/32): python3-ceph-common-19.1.1-298.gb53b6d3 2.7 MB/s | 136 kB 00:00 2024-09-11T09:47:18.740 INFO:teuthology.orchestra.run.smithi107.stdout:(9/32): python3-cephfs-19.1.1-298.gb53b6d3e.el9 2.7 MB/s | 162 kB 00:00 2024-09-11T09:47:18.774 INFO:teuthology.orchestra.run.smithi107.stdout:(10/32): python3-rados-19.1.1-298.gb53b6d3e.el9 4.7 MB/s | 321 kB 00:00 2024-09-11T09:47:18.799 INFO:teuthology.orchestra.run.smithi107.stdout:(11/32): python3-rbd-19.1.1-298.gb53b6d3e.el9.x 5.0 MB/s | 301 kB 00:00 2024-09-11T09:47:18.803 INFO:teuthology.orchestra.run.smithi017.stdout:(31/32): librados2-19.1.1-298.gb53b6d3e.el9.x86 13 MB/s | 3.4 MB 00:00 2024-09-11T09:47:18.824 INFO:teuthology.orchestra.run.smithi107.stdout:(12/32): python3-rgw-19.1.1-298.gb53b6d3e.el9.x 1.9 MB/s | 100 kB 00:00 2024-09-11T09:47:18.862 INFO:teuthology.orchestra.run.smithi017.stdout:(32/32): librbd1-19.1.1-298.gb53b6d3e.el9.x86_6 11 MB/s | 3.2 MB 00:00 2024-09-11T09:47:18.867 INFO:teuthology.orchestra.run.smithi017.stdout:-------------------------------------------------------------------------------- 2024-09-11T09:47:18.867 INFO:teuthology.orchestra.run.smithi017.stdout:Total 22 MB/s | 60 MB 00:02 2024-09-11T09:47:18.942 INFO:teuthology.orchestra.run.smithi096.stdout:(17/32): python3-prettytable-0.7.2-27.el9.noarc 72 kB/s | 42 kB 00:00 2024-09-11T09:47:18.974 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction check 2024-09-11T09:47:19.014 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction check succeeded. 2024-09-11T09:47:19.014 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction test 2024-09-11T09:47:19.099 INFO:teuthology.orchestra.run.smithi107.stdout:(13/32): boost-program-options-1.75.0-8.el9.x86 356 kB/s | 107 kB 00:00 2024-09-11T09:47:19.134 INFO:teuthology.orchestra.run.smithi096.stdout:(18/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 366 kB/s | 292 kB 00:00 2024-09-11T09:47:19.158 INFO:teuthology.orchestra.run.smithi107.stdout:(14/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 480 kB/s | 160 kB 00:00 2024-09-11T09:47:19.167 INFO:teuthology.orchestra.run.smithi096.stdout:(19/32): librdkafka-1.6.1-102.el9.x86_64.rpm 679 kB/s | 662 kB 00:00 2024-09-11T09:47:19.184 INFO:teuthology.orchestra.run.smithi107.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 541 kB/s | 45 kB 00:00 2024-09-11T09:47:19.483 INFO:teuthology.orchestra.run.smithi107.stdout:(16/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 976 kB/s | 292 kB 00:00 2024-09-11T09:47:19.517 INFO:teuthology.orchestra.run.smithi107.stdout:(17/32): librdkafka-1.6.1-102.el9.x86_64.rpm 1.8 MB/s | 662 kB 00:00 2024-09-11T09:47:19.559 INFO:teuthology.orchestra.run.smithi107.stdout:(18/32): python3-prettytable-0.7.2-27.el9.noarc 561 kB/s | 42 kB 00:00 2024-09-11T09:47:19.626 INFO:teuthology.orchestra.run.smithi107.stdout:(19/32): qatlib-service-24.02.0-1.el9.x86_64.rp 539 kB/s | 36 kB 00:00 2024-09-11T09:47:19.651 INFO:teuthology.orchestra.run.smithi107.stdout:(20/32): qatlib-24.02.0-1.el9.x86_64.rpm 1.6 MB/s | 221 kB 00:00 2024-09-11T09:47:19.699 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction test succeeded. 2024-09-11T09:47:19.699 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction 2024-09-11T09:47:19.709 INFO:teuthology.orchestra.run.smithi096.stdout:(20/32): qatlib-24.02.0-1.el9.x86_64.rpm 288 kB/s | 221 kB 00:00 2024-09-11T09:47:19.734 INFO:teuthology.orchestra.run.smithi096.stdout:(21/32): qatlib-service-24.02.0-1.el9.x86_64.rp 60 kB/s | 36 kB 00:00 2024-09-11T09:47:19.768 INFO:teuthology.orchestra.run.smithi096.stdout:(22/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 80 kB/s | 47 kB 00:00 2024-09-11T09:47:19.793 INFO:teuthology.orchestra.run.smithi107.stdout:(21/32): ceph-radosgw-19.1.1-298.gb53b6d3e.el9. 3.3 MB/s | 11 MB 00:03 2024-09-11T09:47:19.810 INFO:teuthology.orchestra.run.smithi107.stdout:(22/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 258 kB/s | 47 kB 00:00 2024-09-11T09:47:19.827 INFO:teuthology.orchestra.run.smithi096.stdout:(23/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 2.6 MB/s | 309 kB 00:00 2024-09-11T09:47:19.852 INFO:teuthology.orchestra.run.smithi107.stdout:(23/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 1.5 MB/s | 309 kB 00:00 2024-09-11T09:47:19.852 INFO:teuthology.orchestra.run.smithi096.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 2.1 MB/s | 49 kB 00:00 2024-09-11T09:47:19.877 INFO:teuthology.orchestra.run.smithi107.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 2.1 MB/s | 49 kB 00:00 2024-09-11T09:47:19.877 INFO:teuthology.orchestra.run.smithi096.stdout:(25/32): libarrow-doc-9.0.0-11.el9.noarch.rpm 260 kB/s | 26 kB 00:00 2024-09-11T09:47:19.894 INFO:teuthology.orchestra.run.smithi107.stdout:(25/32): libarrow-doc-9.0.0-11.el9.noarch.rpm 342 kB/s | 26 kB 00:00 2024-09-11T09:47:19.903 INFO:teuthology.orchestra.run.smithi096.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-09-11T09:47:19.919 INFO:teuthology.orchestra.run.smithi107.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 1.6 MB/s | 67 kB 00:00 2024-09-11T09:47:19.936 INFO:teuthology.orchestra.run.smithi096.stdout:(27/32): re2-20211101-20.el9.x86_64.rpm 5.6 MB/s | 191 kB 00:00 2024-09-11T09:47:19.969 INFO:teuthology.orchestra.run.smithi107.stdout:(27/32): re2-20211101-20.el9.x86_64.rpm 3.8 MB/s | 191 kB 00:00 2024-09-11T09:47:19.986 INFO:teuthology.orchestra.run.smithi096.stdout:(28/32): parquet-libs-9.0.0-11.el9.x86_64.rpm 7.5 MB/s | 839 kB 00:00 2024-09-11T09:47:20.036 INFO:teuthology.orchestra.run.smithi107.stdout:(28/32): parquet-libs-9.0.0-11.el9.x86_64.rpm 5.8 MB/s | 839 kB 00:00 2024-09-11T09:47:20.045 INFO:teuthology.orchestra.run.smithi096.stdout:(29/32): thrift-0.15.0-2.el9.x86_64.rpm 15 MB/s | 1.6 MB 00:00 2024-09-11T09:47:20.112 INFO:teuthology.orchestra.run.smithi107.stdout:(29/32): libarrow-9.0.0-11.el9.x86_64.rpm 14 MB/s | 4.4 MB 00:00 2024-09-11T09:47:20.129 INFO:teuthology.orchestra.run.smithi096.stdout:(30/32): libarrow-9.0.0-11.el9.x86_64.rpm 11 MB/s | 4.4 MB 00:00 2024-09-11T09:47:20.153 INFO:teuthology.orchestra.run.smithi107.stdout:(30/32): thrift-0.15.0-2.el9.x86_64.rpm 8.6 MB/s | 1.6 MB 00:00 2024-09-11T09:47:20.312 INFO:teuthology.orchestra.run.smithi096.stdout:(31/32): librados2-19.1.1-298.gb53b6d3e.el9.x86 11 MB/s | 3.4 MB 00:00 2024-09-11T09:47:20.387 INFO:teuthology.orchestra.run.smithi107.stdout:(31/32): librados2-19.1.1-298.gb53b6d3e.el9.x86 9.8 MB/s | 3.4 MB 00:00 2024-09-11T09:47:20.437 INFO:teuthology.orchestra.run.smithi096.stdout:(32/32): librbd1-19.1.1-298.gb53b6d3e.el9.x86_6 8.1 MB/s | 3.2 MB 00:00 2024-09-11T09:47:20.443 INFO:teuthology.orchestra.run.smithi096.stdout:-------------------------------------------------------------------------------- 2024-09-11T09:47:20.443 INFO:teuthology.orchestra.run.smithi096.stdout:Total 12 MB/s | 60 MB 00:05 2024-09-11T09:47:20.454 INFO:teuthology.orchestra.run.smithi107.stdout:(32/32): librbd1-19.1.1-298.gb53b6d3e.el9.x86_6 9.2 MB/s | 3.2 MB 00:00 2024-09-11T09:47:20.461 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2024-09-11T09:47:20.461 INFO:teuthology.orchestra.run.smithi107.stdout:Total 13 MB/s | 60 MB 00:04 2024-09-11T09:47:20.549 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction check 2024-09-11T09:47:20.569 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2024-09-11T09:47:20.585 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction check succeeded. 2024-09-11T09:47:20.586 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction test 2024-09-11T09:47:20.605 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2024-09-11T09:47:20.605 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2024-09-11T09:47:21.250 INFO:teuthology.orchestra.run.smithi017.stdout: Preparing : 1/1 2024-09-11T09:47:21.293 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction test succeeded. 2024-09-11T09:47:21.293 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction 2024-09-11T09:47:21.295 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2024-09-11T09:47:21.295 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2024-09-11T09:47:21.329 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-09-11T09:47:21.359 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-09-11T09:47:21.392 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-09-11T09:47:21.415 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-09-11T09:47:21.443 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-09-11T09:47:21.642 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-09-11T09:47:21.696 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-09-11T09:47:22.070 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-09-11T09:47:22.143 INFO:teuthology.orchestra.run.smithi017.stdout: Upgrading : librados2-2:19.1.1-298.gb53b6d3e.el9.x86_64 7/34 2024-09-11T09:47:22.236 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: librados2-2:19.1.1-298.gb53b6d3e.el9.x86_64 7/34 2024-09-11T09:47:22.327 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-rados-2:19.1.1-298.gb53b6d3e.el9.x86_64 8/34 2024-09-11T09:47:22.353 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : libcephfs2-2:19.1.1-298.gb53b6d3e.el9.x86_64 9/34 2024-09-11T09:47:22.419 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: libcephfs2-2:19.1.1-298.gb53b6d3e.el9.x86_64 9/34 2024-09-11T09:47:22.460 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-09-11T09:47:22.506 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-ceph-argparse-2:19.1.1-298.gb53b6d3e.el9.x 11/34 2024-09-11T09:47:22.585 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-cephfs-2:19.1.1-298.gb53b6d3e.el9.x86_64 12/34 2024-09-11T09:47:22.614 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : libradosstriper1-2:19.1.1-298.gb53b6d3e.el9.x86_64 13/34 2024-09-11T09:47:22.679 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: libradosstriper1-2:19.1.1-298.gb53b6d3e.el9.x86_64 13/34 2024-09-11T09:47:22.713 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-09-11T09:47:22.776 INFO:teuthology.orchestra.run.smithi096.stdout: Preparing : 1/1 2024-09-11T09:47:22.801 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2024-09-11T09:47:22.823 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : re2-1:20211101-20.el9.x86_64 15/34 2024-09-11T09:47:22.864 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 16/34 2024-09-11T09:47:22.866 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-09-11T09:47:22.871 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-09-11T09:47:22.901 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-09-11T09:47:22.905 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-09-11T09:47:22.926 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-09-11T09:47:22.926 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-09-11T09:47:22.937 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-09-11T09:47:22.949 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-09-11T09:47:22.950 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-09-11T09:47:22.973 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-09-11T09:47:22.974 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-09-11T09:47:22.993 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-09-11T09:47:22.998 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 19/34 2024-09-11T09:47:23.040 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-09-11T09:47:23.169 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-09-11T09:47:23.192 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-09-11T09:47:23.229 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-09-11T09:47:23.246 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-09-11T09:47:23.384 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-09-11T09:47:23.428 INFO:teuthology.orchestra.run.smithi017.stdout: Upgrading : librbd1-2:19.1.1-298.gb53b6d3e.el9.x86_64 22/34 2024-09-11T09:47:23.520 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: librbd1-2:19.1.1-298.gb53b6d3e.el9.x86_64 22/34 2024-09-11T09:47:23.559 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-rbd-2:19.1.1-298.gb53b6d3e.el9.x86_64 23/34 2024-09-11T09:47:23.593 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-09-11T09:47:23.603 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-09-11T09:47:23.612 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-09-11T09:47:23.670 INFO:teuthology.orchestra.run.smithi096.stdout: Upgrading : librados2-2:19.1.1-298.gb53b6d3e.el9.x86_64 7/34 2024-09-11T09:47:23.702 INFO:teuthology.orchestra.run.smithi107.stdout: Upgrading : librados2-2:19.1.1-298.gb53b6d3e.el9.x86_64 7/34 2024-09-11T09:47:23.763 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: librados2-2:19.1.1-298.gb53b6d3e.el9.x86_64 7/34 2024-09-11T09:47:23.794 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: librados2-2:19.1.1-298.gb53b6d3e.el9.x86_64 7/34 2024-09-11T09:47:23.851 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-rados-2:19.1.1-298.gb53b6d3e.el9.x86_64 8/34 2024-09-11T09:47:23.883 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-rados-2:19.1.1-298.gb53b6d3e.el9.x86_64 8/34 2024-09-11T09:47:23.888 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : libcephfs2-2:19.1.1-298.gb53b6d3e.el9.x86_64 9/34 2024-09-11T09:47:23.911 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libcephfs2-2:19.1.1-298.gb53b6d3e.el9.x86_64 9/34 2024-09-11T09:47:23.943 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: libcephfs2-2:19.1.1-298.gb53b6d3e.el9.x86_64 9/34 2024-09-11T09:47:23.972 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-09-11T09:47:24.014 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: libcephfs2-2:19.1.1-298.gb53b6d3e.el9.x86_64 9/34 2024-09-11T09:47:24.016 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-ceph-argparse-2:19.1.1-298.gb53b6d3e.el9.x 11/34 2024-09-11T09:47:24.086 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-cephfs-2:19.1.1-298.gb53b6d3e.el9.x86_64 12/34 2024-09-11T09:47:24.108 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 25/34 2024-09-11T09:47:24.124 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : libradosstriper1-2:19.1.1-298.gb53b6d3e.el9.x86_64 13/34 2024-09-11T09:47:24.139 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : librgw2-2:19.1.1-298.gb53b6d3e.el9.x86_64 26/34 2024-09-11T09:47:24.153 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-09-11T09:47:24.191 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: libradosstriper1-2:19.1.1-298.gb53b6d3e.el9.x86_64 13/34 2024-09-11T09:47:24.203 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-ceph-argparse-2:19.1.1-298.gb53b6d3e.el9.x 11/34 2024-09-11T09:47:24.210 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: librgw2-2:19.1.1-298.gb53b6d3e.el9.x86_64 26/34 2024-09-11T09:47:24.226 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-09-11T09:47:24.275 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-cephfs-2:19.1.1-298.gb53b6d3e.el9.x86_64 12/34 2024-09-11T09:47:24.291 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-rgw-2:19.1.1-298.gb53b6d3e.el9.x86_64 27/34 2024-09-11T09:47:24.297 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libradosstriper1-2:19.1.1-298.gb53b6d3e.el9.x86_64 13/34 2024-09-11T09:47:24.333 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-ceph-common-2:19.1.1-298.gb53b6d3e.el9.x86 28/34 2024-09-11T09:47:24.349 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : re2-1:20211101-20.el9.x86_64 15/34 2024-09-11T09:47:24.352 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: libradosstriper1-2:19.1.1-298.gb53b6d3e.el9.x86_64 13/34 2024-09-11T09:47:24.382 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 16/34 2024-09-11T09:47:24.395 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-09-11T09:47:24.423 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-09-11T09:47:24.501 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-09-11T09:47:24.502 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : re2-1:20211101-20.el9.x86_64 15/34 2024-09-11T09:47:24.529 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 16/34 2024-09-11T09:47:24.570 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-09-11T09:47:24.603 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-09-11T09:47:24.631 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 19/34 2024-09-11T09:47:24.674 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-09-11T09:47:24.684 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 19/34 2024-09-11T09:47:24.868 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-09-11T09:47:24.983 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-09-11T09:47:25.027 INFO:teuthology.orchestra.run.smithi107.stdout: Upgrading : librbd1-2:19.1.1-298.gb53b6d3e.el9.x86_64 22/34 2024-09-11T09:47:25.118 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: librbd1-2:19.1.1-298.gb53b6d3e.el9.x86_64 22/34 2024-09-11T09:47:25.148 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-rbd-2:19.1.1-298.gb53b6d3e.el9.x86_64 23/34 2024-09-11T09:47:25.184 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-09-11T09:47:25.184 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-09-11T09:47:25.230 INFO:teuthology.orchestra.run.smithi096.stdout: Upgrading : librbd1-2:19.1.1-298.gb53b6d3e.el9.x86_64 22/34 2024-09-11T09:47:25.321 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: librbd1-2:19.1.1-298.gb53b6d3e.el9.x86_64 22/34 2024-09-11T09:47:25.361 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-rbd-2:19.1.1-298.gb53b6d3e.el9.x86_64 23/34 2024-09-11T09:47:25.396 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-09-11T09:47:25.662 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 25/34 2024-09-11T09:47:25.688 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : librgw2-2:19.1.1-298.gb53b6d3e.el9.x86_64 26/34 2024-09-11T09:47:25.758 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: librgw2-2:19.1.1-298.gb53b6d3e.el9.x86_64 26/34 2024-09-11T09:47:25.827 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-rgw-2:19.1.1-298.gb53b6d3e.el9.x86_64 27/34 2024-09-11T09:47:25.857 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-ceph-common-2:19.1.1-298.gb53b6d3e.el9.x86 28/34 2024-09-11T09:47:25.885 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 25/34 2024-09-11T09:47:25.916 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : librgw2-2:19.1.1-298.gb53b6d3e.el9.x86_64 26/34 2024-09-11T09:47:25.987 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: librgw2-2:19.1.1-298.gb53b6d3e.el9.x86_64 26/34 2024-09-11T09:47:26.056 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-rgw-2:19.1.1-298.gb53b6d3e.el9.x86_64 27/34 2024-09-11T09:47:26.086 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-ceph-common-2:19.1.1-298.gb53b6d3e.el9.x86 28/34 2024-09-11T09:47:26.759 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: ceph-common-2:19.1.1-298.gb53b6d3e.el9.x86_64 29/34 2024-09-11T09:47:26.796 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : ceph-common-2:19.1.1-298.gb53b6d3e.el9.x86_64 29/34 2024-09-11T09:47:27.405 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: ceph-common-2:19.1.1-298.gb53b6d3e.el9.x86_64 29/34 2024-09-11T09:47:27.441 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : ceph-base-2:19.1.1-298.gb53b6d3e.el9.x86_64 30/34 2024-09-11T09:47:27.534 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: ceph-base-2:19.1.1-298.gb53b6d3e.el9.x86_64 30/34 2024-09-11T09:47:27.535 INFO:teuthology.orchestra.run.smithi017.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-11T09:47:27.535 INFO:teuthology.orchestra.run.smithi017.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-11T09:47:27.535 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-11T09:47:27.567 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : ceph-selinux-2:19.1.1-298.gb53b6d3e.el9.x86_64 31/34 2024-09-11T09:47:28.188 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-common-2:19.1.1-298.gb53b6d3e.el9.x86_64 29/34 2024-09-11T09:47:28.219 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-common-2:19.1.1-298.gb53b6d3e.el9.x86_64 29/34 2024-09-11T09:47:28.313 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-common-2:19.1.1-298.gb53b6d3e.el9.x86_64 29/34 2024-09-11T09:47:28.358 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-common-2:19.1.1-298.gb53b6d3e.el9.x86_64 29/34 2024-09-11T09:47:28.779 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-common-2:19.1.1-298.gb53b6d3e.el9.x86_64 29/34 2024-09-11T09:47:28.814 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-base-2:19.1.1-298.gb53b6d3e.el9.x86_64 30/34 2024-09-11T09:47:28.905 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-base-2:19.1.1-298.gb53b6d3e.el9.x86_64 30/34 2024-09-11T09:47:28.905 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-11T09:47:28.905 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-11T09:47:28.905 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-11T09:47:28.937 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-common-2:19.1.1-298.gb53b6d3e.el9.x86_64 29/34 2024-09-11T09:47:28.940 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-selinux-2:19.1.1-298.gb53b6d3e.el9.x86_64 31/34 2024-09-11T09:47:28.978 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-base-2:19.1.1-298.gb53b6d3e.el9.x86_64 30/34 2024-09-11T09:47:29.065 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-base-2:19.1.1-298.gb53b6d3e.el9.x86_64 30/34 2024-09-11T09:47:29.065 INFO:teuthology.orchestra.run.smithi096.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-11T09:47:29.065 INFO:teuthology.orchestra.run.smithi096.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-11T09:47:29.065 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-11T09:47:29.113 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-selinux-2:19.1.1-298.gb53b6d3e.el9.x86_64 31/34 2024-09-11T09:47:38.225 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: ceph-selinux-2:19.1.1-298.gb53b6d3e.el9.x86_64 31/34 2024-09-11T09:47:38.226 INFO:teuthology.orchestra.run.smithi017.stdout:skipping the directory /sys 2024-09-11T09:47:38.226 INFO:teuthology.orchestra.run.smithi017.stdout:skipping the directory /proc 2024-09-11T09:47:38.226 INFO:teuthology.orchestra.run.smithi017.stdout:skipping the directory /mnt 2024-09-11T09:47:38.226 INFO:teuthology.orchestra.run.smithi017.stdout:skipping the directory /var/tmp 2024-09-11T09:47:38.226 INFO:teuthology.orchestra.run.smithi017.stdout:skipping the directory /home 2024-09-11T09:47:38.227 INFO:teuthology.orchestra.run.smithi017.stdout:skipping the directory /root 2024-09-11T09:47:38.227 INFO:teuthology.orchestra.run.smithi017.stdout:skipping the directory /tmp 2024-09-11T09:47:38.227 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-11T09:47:38.826 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-selinux-2:19.1.1-298.gb53b6d3e.el9.x86_64 31/34 2024-09-11T09:47:38.826 INFO:teuthology.orchestra.run.smithi107.stdout:skipping the directory /sys 2024-09-11T09:47:38.826 INFO:teuthology.orchestra.run.smithi107.stdout:skipping the directory /proc 2024-09-11T09:47:38.826 INFO:teuthology.orchestra.run.smithi107.stdout:skipping the directory /mnt 2024-09-11T09:47:38.826 INFO:teuthology.orchestra.run.smithi107.stdout:skipping the directory /var/tmp 2024-09-11T09:47:38.826 INFO:teuthology.orchestra.run.smithi107.stdout:skipping the directory /home 2024-09-11T09:47:38.826 INFO:teuthology.orchestra.run.smithi107.stdout:skipping the directory /root 2024-09-11T09:47:38.827 INFO:teuthology.orchestra.run.smithi107.stdout:skipping the directory /tmp 2024-09-11T09:47:38.827 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-11T09:47:39.086 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-selinux-2:19.1.1-298.gb53b6d3e.el9.x86_64 31/34 2024-09-11T09:47:39.086 INFO:teuthology.orchestra.run.smithi096.stdout:skipping the directory /sys 2024-09-11T09:47:39.086 INFO:teuthology.orchestra.run.smithi096.stdout:skipping the directory /proc 2024-09-11T09:47:39.086 INFO:teuthology.orchestra.run.smithi096.stdout:skipping the directory /mnt 2024-09-11T09:47:39.086 INFO:teuthology.orchestra.run.smithi096.stdout:skipping the directory /var/tmp 2024-09-11T09:47:39.086 INFO:teuthology.orchestra.run.smithi096.stdout:skipping the directory /home 2024-09-11T09:47:39.086 INFO:teuthology.orchestra.run.smithi096.stdout:skipping the directory /root 2024-09-11T09:47:39.086 INFO:teuthology.orchestra.run.smithi096.stdout:skipping the directory /tmp 2024-09-11T09:47:39.086 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-11T09:47:39.262 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : ceph-radosgw-2:19.1.1-298.gb53b6d3e.el9.x86_64 32/34 2024-09-11T09:47:39.289 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: ceph-radosgw-2:19.1.1-298.gb53b6d3e.el9.x86_64 32/34 2024-09-11T09:47:39.289 INFO:teuthology.orchestra.run.smithi017.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-11T09:47:39.289 INFO:teuthology.orchestra.run.smithi017.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-11T09:47:39.289 INFO:teuthology.orchestra.run.smithi017.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-11T09:47:39.289 INFO:teuthology.orchestra.run.smithi017.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-11T09:47:39.290 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-11T09:47:39.291 INFO:teuthology.orchestra.run.smithi017.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-09-11T09:47:39.373 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-09-11T09:47:39.373 INFO:teuthology.orchestra.run.smithi017.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-09-11T09:47:39.806 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-radosgw-2:19.1.1-298.gb53b6d3e.el9.x86_64 32/34 2024-09-11T09:47:39.831 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-radosgw-2:19.1.1-298.gb53b6d3e.el9.x86_64 32/34 2024-09-11T09:47:39.832 INFO:teuthology.orchestra.run.smithi107.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-11T09:47:39.832 INFO:teuthology.orchestra.run.smithi107.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-11T09:47:39.832 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-11T09:47:39.832 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-11T09:47:39.832 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-11T09:47:39.833 INFO:teuthology.orchestra.run.smithi107.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-09-11T09:47:39.908 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-09-11T09:47:39.908 INFO:teuthology.orchestra.run.smithi107.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-09-11T09:47:40.068 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-radosgw-2:19.1.1-298.gb53b6d3e.el9.x86_64 32/34 2024-09-11T09:47:40.093 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-radosgw-2:19.1.1-298.gb53b6d3e.el9.x86_64 32/34 2024-09-11T09:47:40.093 INFO:teuthology.orchestra.run.smithi096.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-11T09:47:40.093 INFO:teuthology.orchestra.run.smithi096.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-11T09:47:40.093 INFO:teuthology.orchestra.run.smithi096.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-11T09:47:40.093 INFO:teuthology.orchestra.run.smithi096.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-11T09:47:40.093 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-11T09:47:40.095 INFO:teuthology.orchestra.run.smithi096.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-09-11T09:47:40.179 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-09-11T09:47:40.179 INFO:teuthology.orchestra.run.smithi096.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-09-11T09:47:40.489 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-09-11T09:47:40.489 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : ceph-base-2:19.1.1-298.gb53b6d3e.el9.x86_64 1/34 2024-09-11T09:47:40.489 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : ceph-common-2:19.1.1-298.gb53b6d3e.el9.x86_64 2/34 2024-09-11T09:47:40.489 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : ceph-radosgw-2:19.1.1-298.gb53b6d3e.el9.x86_64 3/34 2024-09-11T09:47:40.489 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : ceph-selinux-2:19.1.1-298.gb53b6d3e.el9.x86_64 4/34 2024-09-11T09:47:40.489 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : libcephfs2-2:19.1.1-298.gb53b6d3e.el9.x86_64 5/34 2024-09-11T09:47:40.489 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : libradosstriper1-2:19.1.1-298.gb53b6d3e.el9.x86_64 6/34 2024-09-11T09:47:40.489 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : librgw2-2:19.1.1-298.gb53b6d3e.el9.x86_64 7/34 2024-09-11T09:47:40.489 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-ceph-argparse-2:19.1.1-298.gb53b6d3e.el9.x 8/34 2024-09-11T09:47:40.490 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-ceph-common-2:19.1.1-298.gb53b6d3e.el9.x86 9/34 2024-09-11T09:47:40.490 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-cephfs-2:19.1.1-298.gb53b6d3e.el9.x86_64 10/34 2024-09-11T09:47:40.490 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-rados-2:19.1.1-298.gb53b6d3e.el9.x86_64 11/34 2024-09-11T09:47:40.490 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-rbd-2:19.1.1-298.gb53b6d3e.el9.x86_64 12/34 2024-09-11T09:47:40.490 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-rgw-2:19.1.1-298.gb53b6d3e.el9.x86_64 13/34 2024-09-11T09:47:40.490 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-09-11T09:47:40.490 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-09-11T09:47:40.490 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-09-11T09:47:40.490 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-09-11T09:47:40.491 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-09-11T09:47:40.491 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-09-11T09:47:40.491 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-09-11T09:47:40.491 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-09-11T09:47:40.491 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-09-11T09:47:40.491 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-09-11T09:47:40.491 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 24/34 2024-09-11T09:47:40.491 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 25/34 2024-09-11T09:47:40.491 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-09-11T09:47:40.491 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-09-11T09:47:40.491 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 28/34 2024-09-11T09:47:40.492 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : re2-1:20211101-20.el9.x86_64 29/34 2024-09-11T09:47:40.492 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-09-11T09:47:40.492 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : librados2-2:19.1.1-298.gb53b6d3e.el9.x86_64 31/34 2024-09-11T09:47:40.492 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-09-11T09:47:40.492 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : librbd1-2:19.1.1-298.gb53b6d3e.el9.x86_64 33/34 2024-09-11T09:47:41.062 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-09-11T09:47:41.063 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-base-2:19.1.1-298.gb53b6d3e.el9.x86_64 1/34 2024-09-11T09:47:41.063 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-common-2:19.1.1-298.gb53b6d3e.el9.x86_64 2/34 2024-09-11T09:47:41.063 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-radosgw-2:19.1.1-298.gb53b6d3e.el9.x86_64 3/34 2024-09-11T09:47:41.063 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-selinux-2:19.1.1-298.gb53b6d3e.el9.x86_64 4/34 2024-09-11T09:47:41.063 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libcephfs2-2:19.1.1-298.gb53b6d3e.el9.x86_64 5/34 2024-09-11T09:47:41.063 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libradosstriper1-2:19.1.1-298.gb53b6d3e.el9.x86_64 6/34 2024-09-11T09:47:41.063 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : librgw2-2:19.1.1-298.gb53b6d3e.el9.x86_64 7/34 2024-09-11T09:47:41.063 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-ceph-argparse-2:19.1.1-298.gb53b6d3e.el9.x 8/34 2024-09-11T09:47:41.063 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-ceph-common-2:19.1.1-298.gb53b6d3e.el9.x86 9/34 2024-09-11T09:47:41.063 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-cephfs-2:19.1.1-298.gb53b6d3e.el9.x86_64 10/34 2024-09-11T09:47:41.063 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-rados-2:19.1.1-298.gb53b6d3e.el9.x86_64 11/34 2024-09-11T09:47:41.063 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-rbd-2:19.1.1-298.gb53b6d3e.el9.x86_64 12/34 2024-09-11T09:47:41.063 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-rgw-2:19.1.1-298.gb53b6d3e.el9.x86_64 13/34 2024-09-11T09:47:41.063 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-09-11T09:47:41.063 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-09-11T09:47:41.064 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-09-11T09:47:41.064 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-09-11T09:47:41.064 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-09-11T09:47:41.064 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-09-11T09:47:41.064 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-09-11T09:47:41.064 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-09-11T09:47:41.064 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-09-11T09:47:41.064 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-09-11T09:47:41.064 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 24/34 2024-09-11T09:47:41.064 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 25/34 2024-09-11T09:47:41.064 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-09-11T09:47:41.064 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-09-11T09:47:41.065 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 28/34 2024-09-11T09:47:41.065 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : re2-1:20211101-20.el9.x86_64 29/34 2024-09-11T09:47:41.065 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-09-11T09:47:41.065 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : librados2-2:19.1.1-298.gb53b6d3e.el9.x86_64 31/34 2024-09-11T09:47:41.065 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-09-11T09:47:41.065 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : librbd1-2:19.1.1-298.gb53b6d3e.el9.x86_64 33/34 2024-09-11T09:47:41.396 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-09-11T09:47:41.396 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-base-2:19.1.1-298.gb53b6d3e.el9.x86_64 1/34 2024-09-11T09:47:41.397 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-common-2:19.1.1-298.gb53b6d3e.el9.x86_64 2/34 2024-09-11T09:47:41.397 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-radosgw-2:19.1.1-298.gb53b6d3e.el9.x86_64 3/34 2024-09-11T09:47:41.397 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-selinux-2:19.1.1-298.gb53b6d3e.el9.x86_64 4/34 2024-09-11T09:47:41.397 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : libcephfs2-2:19.1.1-298.gb53b6d3e.el9.x86_64 5/34 2024-09-11T09:47:41.397 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : libradosstriper1-2:19.1.1-298.gb53b6d3e.el9.x86_64 6/34 2024-09-11T09:47:41.397 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : librgw2-2:19.1.1-298.gb53b6d3e.el9.x86_64 7/34 2024-09-11T09:47:41.397 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-ceph-argparse-2:19.1.1-298.gb53b6d3e.el9.x 8/34 2024-09-11T09:47:41.397 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-ceph-common-2:19.1.1-298.gb53b6d3e.el9.x86 9/34 2024-09-11T09:47:41.397 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-cephfs-2:19.1.1-298.gb53b6d3e.el9.x86_64 10/34 2024-09-11T09:47:41.397 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-rados-2:19.1.1-298.gb53b6d3e.el9.x86_64 11/34 2024-09-11T09:47:41.397 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-rbd-2:19.1.1-298.gb53b6d3e.el9.x86_64 12/34 2024-09-11T09:47:41.397 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-rgw-2:19.1.1-298.gb53b6d3e.el9.x86_64 13/34 2024-09-11T09:47:41.398 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-09-11T09:47:41.398 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-09-11T09:47:41.398 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-09-11T09:47:41.398 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-09-11T09:47:41.398 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-09-11T09:47:41.398 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-09-11T09:47:41.398 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-09-11T09:47:41.398 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-09-11T09:47:41.398 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-09-11T09:47:41.398 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-09-11T09:47:41.398 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 24/34 2024-09-11T09:47:41.398 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 25/34 2024-09-11T09:47:41.398 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-09-11T09:47:41.398 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-09-11T09:47:41.398 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 28/34 2024-09-11T09:47:41.399 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : re2-1:20211101-20.el9.x86_64 29/34 2024-09-11T09:47:41.399 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-09-11T09:47:41.399 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : librados2-2:19.1.1-298.gb53b6d3e.el9.x86_64 31/34 2024-09-11T09:47:41.399 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-09-11T09:47:41.400 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : librbd1-2:19.1.1-298.gb53b6d3e.el9.x86_64 33/34 2024-09-11T09:47:41.486 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-09-11T09:47:41.486 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-11T09:47:41.486 INFO:teuthology.orchestra.run.smithi017.stdout:Upgraded: 2024-09-11T09:47:41.486 INFO:teuthology.orchestra.run.smithi017.stdout: librados2-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-11T09:47:41.486 INFO:teuthology.orchestra.run.smithi017.stdout: librbd1-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-11T09:47:41.486 INFO:teuthology.orchestra.run.smithi017.stdout:Installed: 2024-09-11T09:47:41.486 INFO:teuthology.orchestra.run.smithi017.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-11T09:47:41.486 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-base-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-11T09:47:41.487 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-common-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-11T09:47:41.487 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-radosgw-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-11T09:47:41.487 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-selinux-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-11T09:47:41.487 INFO:teuthology.orchestra.run.smithi017.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-09-11T09:47:41.487 INFO:teuthology.orchestra.run.smithi017.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-11T09:47:41.487 INFO:teuthology.orchestra.run.smithi017.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-11T09:47:41.487 INFO:teuthology.orchestra.run.smithi017.stdout: libcephfs2-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-11T09:47:41.487 INFO:teuthology.orchestra.run.smithi017.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-11T09:47:41.487 INFO:teuthology.orchestra.run.smithi017.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-11T09:47:41.487 INFO:teuthology.orchestra.run.smithi017.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-11T09:47:41.487 INFO:teuthology.orchestra.run.smithi017.stdout: libradosstriper1-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-11T09:47:41.487 INFO:teuthology.orchestra.run.smithi017.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-11T09:47:41.487 INFO:teuthology.orchestra.run.smithi017.stdout: librgw2-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-11T09:47:41.488 INFO:teuthology.orchestra.run.smithi017.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-11T09:47:41.488 INFO:teuthology.orchestra.run.smithi017.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-11T09:47:41.488 INFO:teuthology.orchestra.run.smithi017.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-11T09:47:41.488 INFO:teuthology.orchestra.run.smithi017.stdout: python3-ceph-argparse-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-11T09:47:41.488 INFO:teuthology.orchestra.run.smithi017.stdout: python3-ceph-common-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-11T09:47:41.488 INFO:teuthology.orchestra.run.smithi017.stdout: python3-cephfs-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-11T09:47:41.488 INFO:teuthology.orchestra.run.smithi017.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-11T09:47:41.488 INFO:teuthology.orchestra.run.smithi017.stdout: python3-rados-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-11T09:47:41.488 INFO:teuthology.orchestra.run.smithi017.stdout: python3-rbd-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-11T09:47:41.488 INFO:teuthology.orchestra.run.smithi017.stdout: python3-rgw-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-11T09:47:41.488 INFO:teuthology.orchestra.run.smithi017.stdout: qatlib-24.02.0-1.el9.x86_64 2024-09-11T09:47:41.488 INFO:teuthology.orchestra.run.smithi017.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-09-11T09:47:41.488 INFO:teuthology.orchestra.run.smithi017.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-09-11T09:47:41.489 INFO:teuthology.orchestra.run.smithi017.stdout: re2-1:20211101-20.el9.x86_64 2024-09-11T09:47:41.489 INFO:teuthology.orchestra.run.smithi017.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-11T09:47:41.489 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-11T09:47:41.489 INFO:teuthology.orchestra.run.smithi017.stdout:Complete! 2024-09-11T09:47:41.820 DEBUG:teuthology.orchestra.run.smithi017:> sudo yum -y install ceph-test 2024-09-11T09:47:41.941 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-09-11T09:47:41.941 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-11T09:47:41.941 INFO:teuthology.orchestra.run.smithi107.stdout:Upgraded: 2024-09-11T09:47:41.941 INFO:teuthology.orchestra.run.smithi107.stdout: librados2-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-11T09:47:41.941 INFO:teuthology.orchestra.run.smithi107.stdout: librbd1-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-11T09:47:41.941 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2024-09-11T09:47:41.941 INFO:teuthology.orchestra.run.smithi107.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-11T09:47:41.941 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-base-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-11T09:47:41.941 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-common-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-11T09:47:41.941 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-radosgw-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-11T09:47:41.941 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-selinux-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-11T09:47:41.942 INFO:teuthology.orchestra.run.smithi107.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-09-11T09:47:41.942 INFO:teuthology.orchestra.run.smithi107.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-11T09:47:41.942 INFO:teuthology.orchestra.run.smithi107.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-11T09:47:41.942 INFO:teuthology.orchestra.run.smithi107.stdout: libcephfs2-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-11T09:47:41.942 INFO:teuthology.orchestra.run.smithi107.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-11T09:47:41.942 INFO:teuthology.orchestra.run.smithi107.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-11T09:47:41.942 INFO:teuthology.orchestra.run.smithi107.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-11T09:47:41.942 INFO:teuthology.orchestra.run.smithi107.stdout: libradosstriper1-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-11T09:47:41.942 INFO:teuthology.orchestra.run.smithi107.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-11T09:47:41.942 INFO:teuthology.orchestra.run.smithi107.stdout: librgw2-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-11T09:47:41.942 INFO:teuthology.orchestra.run.smithi107.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-11T09:47:41.942 INFO:teuthology.orchestra.run.smithi107.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-11T09:47:41.942 INFO:teuthology.orchestra.run.smithi107.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-11T09:47:41.942 INFO:teuthology.orchestra.run.smithi107.stdout: python3-ceph-argparse-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-11T09:47:41.942 INFO:teuthology.orchestra.run.smithi107.stdout: python3-ceph-common-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-11T09:47:41.943 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cephfs-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-11T09:47:41.943 INFO:teuthology.orchestra.run.smithi107.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-11T09:47:41.943 INFO:teuthology.orchestra.run.smithi107.stdout: python3-rados-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-11T09:47:41.943 INFO:teuthology.orchestra.run.smithi107.stdout: python3-rbd-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-11T09:47:41.943 INFO:teuthology.orchestra.run.smithi107.stdout: python3-rgw-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-11T09:47:41.943 INFO:teuthology.orchestra.run.smithi107.stdout: qatlib-24.02.0-1.el9.x86_64 2024-09-11T09:47:41.943 INFO:teuthology.orchestra.run.smithi107.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-09-11T09:47:41.943 INFO:teuthology.orchestra.run.smithi107.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-09-11T09:47:41.943 INFO:teuthology.orchestra.run.smithi107.stdout: re2-1:20211101-20.el9.x86_64 2024-09-11T09:47:41.943 INFO:teuthology.orchestra.run.smithi107.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-11T09:47:41.943 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-11T09:47:41.943 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-09-11T09:47:42.254 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install ceph-test 2024-09-11T09:47:42.409 INFO:teuthology.orchestra.run.smithi017.stdout:Last metadata expiration check: 0:00:29 ago on Wed 11 Sep 2024 09:47:13 AM UTC. 2024-09-11T09:47:42.438 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-09-11T09:47:42.438 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-11T09:47:42.438 INFO:teuthology.orchestra.run.smithi096.stdout:Upgraded: 2024-09-11T09:47:42.439 INFO:teuthology.orchestra.run.smithi096.stdout: librados2-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-11T09:47:42.439 INFO:teuthology.orchestra.run.smithi096.stdout: librbd1-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-11T09:47:42.439 INFO:teuthology.orchestra.run.smithi096.stdout:Installed: 2024-09-11T09:47:42.439 INFO:teuthology.orchestra.run.smithi096.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-11T09:47:42.439 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-base-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-11T09:47:42.439 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-common-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-11T09:47:42.439 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-radosgw-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-11T09:47:42.439 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-selinux-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-11T09:47:42.439 INFO:teuthology.orchestra.run.smithi096.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-09-11T09:47:42.439 INFO:teuthology.orchestra.run.smithi096.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-11T09:47:42.439 INFO:teuthology.orchestra.run.smithi096.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-11T09:47:42.439 INFO:teuthology.orchestra.run.smithi096.stdout: libcephfs2-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-11T09:47:42.439 INFO:teuthology.orchestra.run.smithi096.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-11T09:47:42.440 INFO:teuthology.orchestra.run.smithi096.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-11T09:47:42.440 INFO:teuthology.orchestra.run.smithi096.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-11T09:47:42.440 INFO:teuthology.orchestra.run.smithi096.stdout: libradosstriper1-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-11T09:47:42.440 INFO:teuthology.orchestra.run.smithi096.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-11T09:47:42.440 INFO:teuthology.orchestra.run.smithi096.stdout: librgw2-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-11T09:47:42.440 INFO:teuthology.orchestra.run.smithi096.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-11T09:47:42.440 INFO:teuthology.orchestra.run.smithi096.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-11T09:47:42.440 INFO:teuthology.orchestra.run.smithi096.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-11T09:47:42.440 INFO:teuthology.orchestra.run.smithi096.stdout: python3-ceph-argparse-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-11T09:47:42.440 INFO:teuthology.orchestra.run.smithi096.stdout: python3-ceph-common-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-11T09:47:42.440 INFO:teuthology.orchestra.run.smithi096.stdout: python3-cephfs-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-11T09:47:42.440 INFO:teuthology.orchestra.run.smithi096.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-11T09:47:42.440 INFO:teuthology.orchestra.run.smithi096.stdout: python3-rados-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-11T09:47:42.440 INFO:teuthology.orchestra.run.smithi096.stdout: python3-rbd-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-11T09:47:42.441 INFO:teuthology.orchestra.run.smithi096.stdout: python3-rgw-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-11T09:47:42.441 INFO:teuthology.orchestra.run.smithi096.stdout: qatlib-24.02.0-1.el9.x86_64 2024-09-11T09:47:42.441 INFO:teuthology.orchestra.run.smithi096.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-09-11T09:47:42.441 INFO:teuthology.orchestra.run.smithi096.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-09-11T09:47:42.441 INFO:teuthology.orchestra.run.smithi096.stdout: re2-1:20211101-20.el9.x86_64 2024-09-11T09:47:42.441 INFO:teuthology.orchestra.run.smithi096.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-11T09:47:42.441 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-11T09:47:42.441 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2024-09-11T09:47:42.546 INFO:teuthology.orchestra.run.smithi017.stdout:Dependencies resolved. 2024-09-11T09:47:42.546 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2024-09-11T09:47:42.546 INFO:teuthology.orchestra.run.smithi017.stdout: Package Arch Version Repository Size 2024-09-11T09:47:42.547 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2024-09-11T09:47:42.547 INFO:teuthology.orchestra.run.smithi017.stdout:Installing: 2024-09-11T09:47:42.547 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-test x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 49 M 2024-09-11T09:47:42.547 INFO:teuthology.orchestra.run.smithi017.stdout:Installing dependencies: 2024-09-11T09:47:42.547 INFO:teuthology.orchestra.run.smithi017.stdout: libcephsqlite x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 163 k 2024-09-11T09:47:42.547 INFO:teuthology.orchestra.run.smithi017.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-11T09:47:42.547 INFO:teuthology.orchestra.run.smithi017.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-11T09:47:42.547 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-11T09:47:42.547 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction Summary 2024-09-11T09:47:42.547 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2024-09-11T09:47:42.547 INFO:teuthology.orchestra.run.smithi017.stdout:Install 4 Packages 2024-09-11T09:47:42.547 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-11T09:47:42.548 INFO:teuthology.orchestra.run.smithi017.stdout:Total download size: 49 M 2024-09-11T09:47:42.548 INFO:teuthology.orchestra.run.smithi017.stdout:Installed size: 209 M 2024-09-11T09:47:42.548 INFO:teuthology.orchestra.run.smithi017.stdout:Downloading Packages: 2024-09-11T09:47:42.736 INFO:teuthology.orchestra.run.smithi017.stdout:(1/4): socat-1.7.4.1-6.el9.x86_64.rpm 1.8 MB/s | 304 kB 00:00 2024-09-11T09:47:42.787 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum -y install ceph-test 2024-09-11T09:47:42.787 INFO:teuthology.orchestra.run.smithi017.stdout:(2/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 64 kB 00:00 2024-09-11T09:47:42.819 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:00:29 ago on Wed 11 Sep 2024 09:47:13 AM UTC. 2024-09-11T09:47:42.828 INFO:teuthology.orchestra.run.smithi017.stdout:(3/4): libcephsqlite-19.1.1-298.gb53b6d3e.el9.x 637 kB/s | 163 kB 00:00 2024-09-11T09:47:42.951 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-09-11T09:47:42.951 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-09-11T09:47:42.951 INFO:teuthology.orchestra.run.smithi107.stdout: Package Arch Version Repository Size 2024-09-11T09:47:42.951 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-09-11T09:47:42.951 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2024-09-11T09:47:42.951 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-test x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 49 M 2024-09-11T09:47:42.951 INFO:teuthology.orchestra.run.smithi107.stdout:Installing dependencies: 2024-09-11T09:47:42.951 INFO:teuthology.orchestra.run.smithi107.stdout: libcephsqlite x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 163 k 2024-09-11T09:47:42.952 INFO:teuthology.orchestra.run.smithi107.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-11T09:47:42.952 INFO:teuthology.orchestra.run.smithi107.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-11T09:47:42.952 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-11T09:47:42.952 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2024-09-11T09:47:42.952 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-09-11T09:47:42.952 INFO:teuthology.orchestra.run.smithi107.stdout:Install 4 Packages 2024-09-11T09:47:42.952 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-11T09:47:42.953 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 49 M 2024-09-11T09:47:42.953 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 209 M 2024-09-11T09:47:42.953 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2024-09-11T09:47:43.339 INFO:teuthology.orchestra.run.smithi107.stdout:(1/4): socat-1.7.4.1-6.el9.x86_64.rpm 2.2 MB/s | 304 kB 00:00 2024-09-11T09:47:43.349 INFO:teuthology.orchestra.run.smithi096.stdout:Last metadata expiration check: 0:00:31 ago on Wed 11 Sep 2024 09:47:12 AM UTC. 2024-09-11T09:47:43.381 INFO:teuthology.orchestra.run.smithi107.stdout:(2/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.5 MB/s | 64 kB 00:00 2024-09-11T09:47:43.479 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2024-09-11T09:47:43.479 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-09-11T09:47:43.479 INFO:teuthology.orchestra.run.smithi096.stdout: Package Arch Version Repository Size 2024-09-11T09:47:43.480 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-09-11T09:47:43.480 INFO:teuthology.orchestra.run.smithi096.stdout:Installing: 2024-09-11T09:47:43.480 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-test x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 49 M 2024-09-11T09:47:43.480 INFO:teuthology.orchestra.run.smithi096.stdout:Installing dependencies: 2024-09-11T09:47:43.480 INFO:teuthology.orchestra.run.smithi096.stdout: libcephsqlite x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 163 k 2024-09-11T09:47:43.480 INFO:teuthology.orchestra.run.smithi096.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-11T09:47:43.480 INFO:teuthology.orchestra.run.smithi096.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-11T09:47:43.480 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-11T09:47:43.480 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction Summary 2024-09-11T09:47:43.480 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-09-11T09:47:43.480 INFO:teuthology.orchestra.run.smithi096.stdout:Install 4 Packages 2024-09-11T09:47:43.480 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-11T09:47:43.481 INFO:teuthology.orchestra.run.smithi096.stdout:Total download size: 49 M 2024-09-11T09:47:43.481 INFO:teuthology.orchestra.run.smithi096.stdout:Installed size: 209 M 2024-09-11T09:47:43.481 INFO:teuthology.orchestra.run.smithi096.stdout:Downloading Packages: 2024-09-11T09:47:43.606 INFO:teuthology.orchestra.run.smithi107.stdout:(3/4): libcephsqlite-19.1.1-298.gb53b6d3e.el9.x 403 kB/s | 163 kB 00:00 2024-09-11T09:47:43.785 INFO:teuthology.orchestra.run.smithi096.stdout:(1/4): socat-1.7.4.1-6.el9.x86_64.rpm 1.0 MB/s | 304 kB 00:00 2024-09-11T09:47:43.827 INFO:teuthology.orchestra.run.smithi096.stdout:(2/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.5 MB/s | 64 kB 00:00 2024-09-11T09:47:43.852 INFO:teuthology.orchestra.run.smithi096.stdout:(3/4): libcephsqlite-19.1.1-298.gb53b6d3e.el9.x 466 kB/s | 163 kB 00:00 2024-09-11T09:47:44.104 INFO:teuthology.orchestra.run.smithi017.stdout:(4/4): ceph-test-19.1.1-298.gb53b6d3e.el9.x86_6 32 MB/s | 49 MB 00:01 2024-09-11T09:47:44.106 INFO:teuthology.orchestra.run.smithi017.stdout:-------------------------------------------------------------------------------- 2024-09-11T09:47:44.106 INFO:teuthology.orchestra.run.smithi017.stdout:Total 32 MB/s | 49 MB 00:01 2024-09-11T09:47:44.135 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction check 2024-09-11T09:47:44.157 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction check succeeded. 2024-09-11T09:47:44.157 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction test 2024-09-11T09:47:44.553 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction test succeeded. 2024-09-11T09:47:44.553 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction 2024-09-11T09:47:45.126 INFO:teuthology.orchestra.run.smithi017.stdout: Preparing : 1/1 2024-09-11T09:47:45.217 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-09-11T09:47:45.262 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-09-11T09:47:45.277 INFO:teuthology.orchestra.run.smithi096.stdout:(4/4): ceph-test-19.1.1-298.gb53b6d3e.el9.x86_6 27 MB/s | 49 MB 00:01 2024-09-11T09:47:45.280 INFO:teuthology.orchestra.run.smithi096.stdout:-------------------------------------------------------------------------------- 2024-09-11T09:47:45.280 INFO:teuthology.orchestra.run.smithi096.stdout:Total 27 MB/s | 49 MB 00:01 2024-09-11T09:47:45.289 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : libcephsqlite-2:19.1.1-298.gb53b6d3e.el9.x86_64 3/4 2024-09-11T09:47:45.309 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction check 2024-09-11T09:47:45.328 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction check succeeded. 2024-09-11T09:47:45.329 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction test 2024-09-11T09:47:45.709 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction test succeeded. 2024-09-11T09:47:45.709 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction 2024-09-11T09:47:46.106 INFO:teuthology.orchestra.run.smithi107.stdout:(4/4): ceph-test-19.1.1-298.gb53b6d3e.el9.x86_6 17 MB/s | 49 MB 00:02 2024-09-11T09:47:46.108 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2024-09-11T09:47:46.109 INFO:teuthology.orchestra.run.smithi107.stdout:Total 16 MB/s | 49 MB 00:03 2024-09-11T09:47:46.136 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2024-09-11T09:47:46.156 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2024-09-11T09:47:46.156 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2024-09-11T09:47:46.243 INFO:teuthology.orchestra.run.smithi096.stdout: Preparing : 1/1 2024-09-11T09:47:46.317 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-09-11T09:47:46.360 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-09-11T09:47:46.388 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : libcephsqlite-2:19.1.1-298.gb53b6d3e.el9.x86_64 3/4 2024-09-11T09:47:46.534 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2024-09-11T09:47:46.535 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2024-09-11T09:47:47.066 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2024-09-11T09:47:47.142 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-09-11T09:47:47.180 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-09-11T09:47:47.208 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libcephsqlite-2:19.1.1-298.gb53b6d3e.el9.x86_64 3/4 2024-09-11T09:47:50.200 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: libcephsqlite-2:19.1.1-298.gb53b6d3e.el9.x86_64 3/4 2024-09-11T09:47:50.283 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : ceph-test-2:19.1.1-298.gb53b6d3e.el9.x86_64 4/4 2024-09-11T09:47:50.994 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: libcephsqlite-2:19.1.1-298.gb53b6d3e.el9.x86_64 3/4 2024-09-11T09:47:51.069 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-test-2:19.1.1-298.gb53b6d3e.el9.x86_64 4/4 2024-09-11T09:47:51.790 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: libcephsqlite-2:19.1.1-298.gb53b6d3e.el9.x86_64 3/4 2024-09-11T09:47:51.873 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-test-2:19.1.1-298.gb53b6d3e.el9.x86_64 4/4 2024-09-11T09:47:52.632 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: ceph-test-2:19.1.1-298.gb53b6d3e.el9.x86_64 4/4 2024-09-11T09:47:52.633 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : ceph-test-2:19.1.1-298.gb53b6d3e.el9.x86_64 1/4 2024-09-11T09:47:52.633 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : libcephsqlite-2:19.1.1-298.gb53b6d3e.el9.x86_64 2/4 2024-09-11T09:47:52.633 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-09-11T09:47:53.003 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-09-11T09:47:53.003 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-11T09:47:53.003 INFO:teuthology.orchestra.run.smithi017.stdout:Installed: 2024-09-11T09:47:53.003 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-test-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-11T09:47:53.003 INFO:teuthology.orchestra.run.smithi017.stdout: libcephsqlite-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-11T09:47:53.003 INFO:teuthology.orchestra.run.smithi017.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-11T09:47:53.003 INFO:teuthology.orchestra.run.smithi017.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-11T09:47:53.003 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-11T09:47:53.004 INFO:teuthology.orchestra.run.smithi017.stdout:Complete! 2024-09-11T09:47:53.231 DEBUG:teuthology.orchestra.run.smithi017:> sudo yum -y install ceph 2024-09-11T09:47:53.503 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-test-2:19.1.1-298.gb53b6d3e.el9.x86_64 4/4 2024-09-11T09:47:53.503 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-test-2:19.1.1-298.gb53b6d3e.el9.x86_64 1/4 2024-09-11T09:47:53.503 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : libcephsqlite-2:19.1.1-298.gb53b6d3e.el9.x86_64 2/4 2024-09-11T09:47:53.503 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-09-11T09:47:53.819 INFO:teuthology.orchestra.run.smithi017.stdout:Last metadata expiration check: 0:00:40 ago on Wed 11 Sep 2024 09:47:13 AM UTC. 2024-09-11T09:47:53.838 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-09-11T09:47:53.838 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-11T09:47:53.838 INFO:teuthology.orchestra.run.smithi096.stdout:Installed: 2024-09-11T09:47:53.838 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-test-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-11T09:47:53.838 INFO:teuthology.orchestra.run.smithi096.stdout: libcephsqlite-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-11T09:47:53.839 INFO:teuthology.orchestra.run.smithi096.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-11T09:47:53.839 INFO:teuthology.orchestra.run.smithi096.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-11T09:47:53.839 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-11T09:47:53.839 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2024-09-11T09:47:53.967 INFO:teuthology.orchestra.run.smithi017.stdout:Dependencies resolved. 2024-09-11T09:47:53.970 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2024-09-11T09:47:53.970 INFO:teuthology.orchestra.run.smithi017.stdout: Package Arch Version Repository Size 2024-09-11T09:47:53.970 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2024-09-11T09:47:53.970 INFO:teuthology.orchestra.run.smithi017.stdout:Installing: 2024-09-11T09:47:53.970 INFO:teuthology.orchestra.run.smithi017.stdout: ceph x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 6.5 k 2024-09-11T09:47:53.970 INFO:teuthology.orchestra.run.smithi017.stdout:Installing dependencies: 2024-09-11T09:47:53.970 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-mds x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 2.4 M 2024-09-11T09:47:53.971 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-mgr x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 1.5 M 2024-09-11T09:47:53.971 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-mgr-modules-core noarch 2:19.1.1-298.gb53b6d3e.el9 ceph-noarch 249 k 2024-09-11T09:47:53.971 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-mon x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 4.7 M 2024-09-11T09:47:53.971 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-osd x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 17 M 2024-09-11T09:47:53.971 INFO:teuthology.orchestra.run.smithi017.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-09-11T09:47:53.971 INFO:teuthology.orchestra.run.smithi017.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-09-11T09:47:53.971 INFO:teuthology.orchestra.run.smithi017.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-09-11T09:47:53.971 INFO:teuthology.orchestra.run.smithi017.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-11T09:47:53.971 INFO:teuthology.orchestra.run.smithi017.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-11T09:47:53.971 INFO:teuthology.orchestra.run.smithi017.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-11T09:47:53.971 INFO:teuthology.orchestra.run.smithi017.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-11T09:47:53.971 INFO:teuthology.orchestra.run.smithi017.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-11T09:47:53.972 INFO:teuthology.orchestra.run.smithi017.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-11T09:47:53.972 INFO:teuthology.orchestra.run.smithi017.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-11T09:47:53.972 INFO:teuthology.orchestra.run.smithi017.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-09-11T09:47:53.972 INFO:teuthology.orchestra.run.smithi017.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-11T09:47:53.972 INFO:teuthology.orchestra.run.smithi017.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-11T09:47:53.972 INFO:teuthology.orchestra.run.smithi017.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-11T09:47:53.972 INFO:teuthology.orchestra.run.smithi017.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-11T09:47:53.972 INFO:teuthology.orchestra.run.smithi017.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-11T09:47:53.972 INFO:teuthology.orchestra.run.smithi017.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-11T09:47:53.972 INFO:teuthology.orchestra.run.smithi017.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-11T09:47:53.972 INFO:teuthology.orchestra.run.smithi017.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-11T09:47:53.972 INFO:teuthology.orchestra.run.smithi017.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-11T09:47:53.972 INFO:teuthology.orchestra.run.smithi017.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-11T09:47:53.973 INFO:teuthology.orchestra.run.smithi017.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-11T09:47:53.973 INFO:teuthology.orchestra.run.smithi017.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-11T09:47:53.973 INFO:teuthology.orchestra.run.smithi017.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-11T09:47:53.973 INFO:teuthology.orchestra.run.smithi017.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-11T09:47:53.973 INFO:teuthology.orchestra.run.smithi017.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-11T09:47:53.973 INFO:teuthology.orchestra.run.smithi017.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-11T09:47:53.973 INFO:teuthology.orchestra.run.smithi017.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-11T09:47:53.973 INFO:teuthology.orchestra.run.smithi017.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-11T09:47:53.973 INFO:teuthology.orchestra.run.smithi017.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-11T09:47:53.973 INFO:teuthology.orchestra.run.smithi017.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-11T09:47:53.973 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-11T09:47:53.973 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction Summary 2024-09-11T09:47:53.974 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2024-09-11T09:47:53.974 INFO:teuthology.orchestra.run.smithi017.stdout:Install 37 Packages 2024-09-11T09:47:53.974 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-11T09:47:53.974 INFO:teuthology.orchestra.run.smithi017.stdout:Total download size: 30 M 2024-09-11T09:47:53.974 INFO:teuthology.orchestra.run.smithi017.stdout:Installed size: 108 M 2024-09-11T09:47:53.975 INFO:teuthology.orchestra.run.smithi017.stdout:Downloading Packages: 2024-09-11T09:47:54.134 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum -y install ceph 2024-09-11T09:47:54.342 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-test-2:19.1.1-298.gb53b6d3e.el9.x86_64 4/4 2024-09-11T09:47:54.342 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-test-2:19.1.1-298.gb53b6d3e.el9.x86_64 1/4 2024-09-11T09:47:54.342 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libcephsqlite-2:19.1.1-298.gb53b6d3e.el9.x86_64 2/4 2024-09-11T09:47:54.342 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-09-11T09:47:54.659 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-09-11T09:47:54.659 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-11T09:47:54.659 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2024-09-11T09:47:54.659 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-test-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-11T09:47:54.659 INFO:teuthology.orchestra.run.smithi107.stdout: libcephsqlite-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-11T09:47:54.659 INFO:teuthology.orchestra.run.smithi107.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-11T09:47:54.659 INFO:teuthology.orchestra.run.smithi107.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-11T09:47:54.660 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-11T09:47:54.660 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-09-11T09:47:54.716 INFO:teuthology.orchestra.run.smithi096.stdout:Last metadata expiration check: 0:00:42 ago on Wed 11 Sep 2024 09:47:12 AM UTC. 2024-09-11T09:47:54.822 INFO:teuthology.orchestra.run.smithi017.stdout:(1/37): ceph-19.1.1-298.gb53b6d3e.el9.x86_64.rp 49 kB/s | 6.5 kB 00:00 2024-09-11T09:47:54.861 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2024-09-11T09:47:54.864 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-09-11T09:47:54.864 INFO:teuthology.orchestra.run.smithi096.stdout: Package Arch Version Repository Size 2024-09-11T09:47:54.864 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-09-11T09:47:54.864 INFO:teuthology.orchestra.run.smithi096.stdout:Installing: 2024-09-11T09:47:54.864 INFO:teuthology.orchestra.run.smithi096.stdout: ceph x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 6.5 k 2024-09-11T09:47:54.864 INFO:teuthology.orchestra.run.smithi096.stdout:Installing dependencies: 2024-09-11T09:47:54.864 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mds x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 2.4 M 2024-09-11T09:47:54.864 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mgr x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 1.5 M 2024-09-11T09:47:54.864 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mgr-modules-core noarch 2:19.1.1-298.gb53b6d3e.el9 ceph-noarch 249 k 2024-09-11T09:47:54.864 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mon x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 4.7 M 2024-09-11T09:47:54.864 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-osd x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 17 M 2024-09-11T09:47:54.865 INFO:teuthology.orchestra.run.smithi096.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-09-11T09:47:54.865 INFO:teuthology.orchestra.run.smithi096.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-09-11T09:47:54.865 INFO:teuthology.orchestra.run.smithi096.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-09-11T09:47:54.865 INFO:teuthology.orchestra.run.smithi096.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-11T09:47:54.865 INFO:teuthology.orchestra.run.smithi096.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-11T09:47:54.865 INFO:teuthology.orchestra.run.smithi096.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-11T09:47:54.865 INFO:teuthology.orchestra.run.smithi096.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-11T09:47:54.865 INFO:teuthology.orchestra.run.smithi096.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-11T09:47:54.865 INFO:teuthology.orchestra.run.smithi096.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-11T09:47:54.865 INFO:teuthology.orchestra.run.smithi096.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-11T09:47:54.865 INFO:teuthology.orchestra.run.smithi096.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-09-11T09:47:54.865 INFO:teuthology.orchestra.run.smithi096.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-11T09:47:54.865 INFO:teuthology.orchestra.run.smithi096.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-11T09:47:54.866 INFO:teuthology.orchestra.run.smithi096.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-11T09:47:54.866 INFO:teuthology.orchestra.run.smithi096.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-11T09:47:54.866 INFO:teuthology.orchestra.run.smithi096.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-11T09:47:54.866 INFO:teuthology.orchestra.run.smithi096.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-11T09:47:54.866 INFO:teuthology.orchestra.run.smithi096.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-11T09:47:54.866 INFO:teuthology.orchestra.run.smithi096.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-11T09:47:54.866 INFO:teuthology.orchestra.run.smithi096.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-11T09:47:54.866 INFO:teuthology.orchestra.run.smithi096.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-11T09:47:54.866 INFO:teuthology.orchestra.run.smithi096.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-11T09:47:54.866 INFO:teuthology.orchestra.run.smithi096.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-11T09:47:54.866 INFO:teuthology.orchestra.run.smithi096.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-11T09:47:54.866 INFO:teuthology.orchestra.run.smithi096.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-11T09:47:54.866 INFO:teuthology.orchestra.run.smithi096.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-11T09:47:54.867 INFO:teuthology.orchestra.run.smithi096.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-11T09:47:54.867 INFO:teuthology.orchestra.run.smithi096.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-11T09:47:54.867 INFO:teuthology.orchestra.run.smithi096.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-11T09:47:54.867 INFO:teuthology.orchestra.run.smithi096.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-11T09:47:54.867 INFO:teuthology.orchestra.run.smithi096.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-11T09:47:54.867 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-11T09:47:54.867 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction Summary 2024-09-11T09:47:54.867 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-09-11T09:47:54.867 INFO:teuthology.orchestra.run.smithi096.stdout:Install 37 Packages 2024-09-11T09:47:54.867 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-11T09:47:54.868 INFO:teuthology.orchestra.run.smithi096.stdout:Total download size: 30 M 2024-09-11T09:47:54.868 INFO:teuthology.orchestra.run.smithi096.stdout:Installed size: 108 M 2024-09-11T09:47:54.868 INFO:teuthology.orchestra.run.smithi096.stdout:Downloading Packages: 2024-09-11T09:47:54.935 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install ceph 2024-09-11T09:47:54.989 INFO:teuthology.orchestra.run.smithi017.stdout:(2/37): ceph-mgr-19.1.1-298.gb53b6d3e.el9.x86_6 5.1 MB/s | 1.5 MB 00:00 2024-09-11T09:47:55.064 INFO:teuthology.orchestra.run.smithi017.stdout:(3/37): ceph-mds-19.1.1-298.gb53b6d3e.el9.x86_6 6.4 MB/s | 2.4 MB 00:00 2024-09-11T09:47:55.164 INFO:teuthology.orchestra.run.smithi017.stdout:(4/37): ceph-mgr-modules-core-19.1.1-298.gb53b6 2.4 MB/s | 249 kB 00:00 2024-09-11T09:47:55.240 INFO:teuthology.orchestra.run.smithi017.stdout:(5/37): ceph-mon-19.1.1-298.gb53b6d3e.el9.x86_6 11 MB/s | 4.7 MB 00:00 2024-09-11T09:47:55.280 INFO:teuthology.orchestra.run.smithi096.stdout:(1/37): ceph-19.1.1-298.gb53b6d3e.el9.x86_64.rp 49 kB/s | 6.5 kB 00:00 2024-09-11T09:47:55.448 INFO:teuthology.orchestra.run.smithi017.stdout:(6/37): python3-cffi-1.14.5-5.el9.x86_64.rpm 893 kB/s | 253 kB 00:00 2024-09-11T09:47:55.520 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:00:42 ago on Wed 11 Sep 2024 09:47:13 AM UTC. 2024-09-11T09:47:55.615 INFO:teuthology.orchestra.run.smithi017.stdout:(7/37): ceph-osd-19.1.1-298.gb53b6d3e.el9.x86_6 27 MB/s | 17 MB 00:00 2024-09-11T09:47:55.640 INFO:teuthology.orchestra.run.smithi017.stdout:(8/37): python3-ply-3.11-14.el9.noarch.rpm 555 kB/s | 106 kB 00:00 2024-09-11T09:47:55.662 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-09-11T09:47:55.664 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-09-11T09:47:55.664 INFO:teuthology.orchestra.run.smithi107.stdout: Package Arch Version Repository Size 2024-09-11T09:47:55.664 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-09-11T09:47:55.665 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2024-09-11T09:47:55.665 INFO:teuthology.orchestra.run.smithi107.stdout: ceph x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 6.5 k 2024-09-11T09:47:55.665 INFO:teuthology.orchestra.run.smithi107.stdout:Installing dependencies: 2024-09-11T09:47:55.665 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mds x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 2.4 M 2024-09-11T09:47:55.665 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 1.5 M 2024-09-11T09:47:55.665 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-modules-core noarch 2:19.1.1-298.gb53b6d3e.el9 ceph-noarch 249 k 2024-09-11T09:47:55.665 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mon x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 4.7 M 2024-09-11T09:47:55.665 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-osd x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 17 M 2024-09-11T09:47:55.665 INFO:teuthology.orchestra.run.smithi107.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-09-11T09:47:55.665 INFO:teuthology.orchestra.run.smithi107.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-09-11T09:47:55.665 INFO:teuthology.orchestra.run.smithi107.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-09-11T09:47:55.665 INFO:teuthology.orchestra.run.smithi107.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-11T09:47:55.665 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-11T09:47:55.665 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-11T09:47:55.666 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-11T09:47:55.666 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-11T09:47:55.666 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-11T09:47:55.666 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-11T09:47:55.666 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-09-11T09:47:55.666 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-11T09:47:55.666 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-11T09:47:55.666 INFO:teuthology.orchestra.run.smithi107.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-11T09:47:55.666 INFO:teuthology.orchestra.run.smithi107.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-11T09:47:55.666 INFO:teuthology.orchestra.run.smithi107.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-11T09:47:55.666 INFO:teuthology.orchestra.run.smithi107.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-11T09:47:55.666 INFO:teuthology.orchestra.run.smithi107.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-11T09:47:55.667 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-11T09:47:55.667 INFO:teuthology.orchestra.run.smithi107.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-11T09:47:55.667 INFO:teuthology.orchestra.run.smithi107.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-11T09:47:55.667 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-11T09:47:55.667 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-11T09:47:55.667 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-11T09:47:55.667 INFO:teuthology.orchestra.run.smithi107.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-11T09:47:55.667 INFO:teuthology.orchestra.run.smithi107.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-11T09:47:55.667 INFO:teuthology.orchestra.run.smithi107.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-11T09:47:55.667 INFO:teuthology.orchestra.run.smithi107.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-11T09:47:55.667 INFO:teuthology.orchestra.run.smithi107.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-11T09:47:55.667 INFO:teuthology.orchestra.run.smithi107.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-11T09:47:55.667 INFO:teuthology.orchestra.run.smithi107.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-11T09:47:55.668 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-11T09:47:55.668 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2024-09-11T09:47:55.668 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-09-11T09:47:55.668 INFO:teuthology.orchestra.run.smithi107.stdout:Install 37 Packages 2024-09-11T09:47:55.668 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-11T09:47:55.668 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 30 M 2024-09-11T09:47:55.669 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 108 M 2024-09-11T09:47:55.669 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2024-09-11T09:47:55.674 INFO:teuthology.orchestra.run.smithi017.stdout:(9/37): python3-cryptography-36.0.1-4.el9.x86_6 2.8 MB/s | 1.2 MB 00:00 2024-09-11T09:47:55.698 INFO:teuthology.orchestra.run.smithi096.stdout:(2/37): ceph-mgr-19.1.1-298.gb53b6d3e.el9.x86_6 2.8 MB/s | 1.5 MB 00:00 2024-09-11T09:47:55.699 INFO:teuthology.orchestra.run.smithi017.stdout:(10/37): python3-requests-2.25.1-8.el9.noarch.r 2.1 MB/s | 125 kB 00:00 2024-09-11T09:47:55.733 INFO:teuthology.orchestra.run.smithi017.stdout:(11/37): python3-urllib3-1.26.5-6.el9.noarch.rp 3.6 MB/s | 215 kB 00:00 2024-09-11T09:47:55.816 INFO:teuthology.orchestra.run.smithi017.stdout:(12/37): python3-pycparser-2.20-6.el9.noarch.rp 673 kB/s | 135 kB 00:00 2024-09-11T09:47:55.908 INFO:teuthology.orchestra.run.smithi017.stdout:(13/37): lua-5.4.4-4.el9.x86_64.rpm 902 kB/s | 188 kB 00:00 2024-09-11T09:47:55.925 INFO:teuthology.orchestra.run.smithi017.stdout:(14/37): python3-markupsafe-1.1.1-12.el9.x86_64 320 kB/s | 35 kB 00:00 2024-09-11T09:47:55.951 INFO:teuthology.orchestra.run.smithi017.stdout:(15/37): python3-mako-1.1.4-6.el9.noarch.rpm 791 kB/s | 172 kB 00:00 2024-09-11T09:47:55.964 INFO:teuthology.orchestra.run.smithi096.stdout:(3/37): ceph-mds-19.1.1-298.gb53b6d3e.el9.x86_6 2.9 MB/s | 2.4 MB 00:00 2024-09-11T09:47:55.976 INFO:teuthology.orchestra.run.smithi017.stdout:(16/37): python3-packaging-20.9-5.el9.noarch.rp 1.1 MB/s | 77 kB 00:00 2024-09-11T09:47:56.001 INFO:teuthology.orchestra.run.smithi017.stdout:(17/37): python3-toml-0.10.2-6.el9.noarch.rpm 830 kB/s | 42 kB 00:00 2024-09-11T09:47:56.026 INFO:teuthology.orchestra.run.smithi017.stdout:(18/37): python3-pytz-2021.1-5.el9.noarch.rpm 506 kB/s | 51 kB 00:00 2024-09-11T09:47:56.052 INFO:teuthology.orchestra.run.smithi017.stdout:(19/37): lua-devel-5.4.4-4.el9.x86_64.rpm 296 kB/s | 22 kB 00:00 2024-09-11T09:47:56.073 INFO:teuthology.orchestra.run.smithi096.stdout:(4/37): ceph-mon-19.1.1-298.gb53b6d3e.el9.x86_6 5.9 MB/s | 4.7 MB 00:00 2024-09-11T09:47:56.098 INFO:teuthology.orchestra.run.smithi096.stdout:(5/37): ceph-mgr-modules-core-19.1.1-298.gb53b6 1.8 MB/s | 249 kB 00:00 2024-09-11T09:47:56.110 INFO:teuthology.orchestra.run.smithi017.stdout:(20/37): luarocks-3.9.2-5.el9.noarch.rpm 1.4 MB/s | 151 kB 00:00 2024-09-11T09:47:56.144 INFO:teuthology.orchestra.run.smithi017.stdout:(21/37): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 371 kB/s | 43 kB 00:00 2024-09-11T09:47:56.169 INFO:teuthology.orchestra.run.smithi017.stdout:(22/37): python3-jaraco-8.2.1-3.el9.noarch.rpm 429 kB/s | 11 kB 00:00 2024-09-11T09:47:56.195 INFO:teuthology.orchestra.run.smithi017.stdout:(23/37): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.3 MB/s | 172 kB 00:00 2024-09-11T09:47:56.220 INFO:teuthology.orchestra.run.smithi017.stdout:(24/37): python3-jaraco-classes-3.2.1-5.el9.noa 352 kB/s | 18 kB 00:00 2024-09-11T09:47:56.245 INFO:teuthology.orchestra.run.smithi017.stdout:(25/37): python3-jaraco-functools-3.5.0-2.el9.n 793 kB/s | 19 kB 00:00 2024-09-11T09:47:56.249 INFO:teuthology.orchestra.run.smithi107.stdout:(1/37): ceph-19.1.1-298.gb53b6d3e.el9.x86_64.rp 53 kB/s | 6.5 kB 00:00 2024-09-11T09:47:56.270 INFO:teuthology.orchestra.run.smithi017.stdout:(26/37): python3-jaraco-collections-3.0.0-8.el9 464 kB/s | 23 kB 00:00 2024-09-11T09:47:56.296 INFO:teuthology.orchestra.run.smithi017.stdout:(27/37): python3-jaraco-text-3.2.0-6.el9.noarch 391 kB/s | 20 kB 00:00 2024-09-11T09:47:56.321 INFO:teuthology.orchestra.run.smithi017.stdout:(28/37): python3-logutils-0.3.5-21.el9.noarch.r 921 kB/s | 46 kB 00:00 2024-09-11T09:47:56.346 INFO:teuthology.orchestra.run.smithi017.stdout:(29/37): python3-cherrypy-18.6.1-2.el9.noarch.r 1.5 MB/s | 358 kB 00:00 2024-09-11T09:47:56.372 INFO:teuthology.orchestra.run.smithi017.stdout:(30/37): python3-more-itertools-8.12.0-2.el9.no 1.0 MB/s | 79 kB 00:00 2024-09-11T09:47:56.397 INFO:teuthology.orchestra.run.smithi017.stdout:(31/37): python3-portend-3.1.0-2.el9.noarch.rpm 326 kB/s | 16 kB 00:00 2024-09-11T09:47:56.422 INFO:teuthology.orchestra.run.smithi017.stdout:(32/37): python3-tempora-5.0.0-2.el9.noarch.rpm 1.4 MB/s | 36 kB 00:00 2024-09-11T09:47:56.447 INFO:teuthology.orchestra.run.smithi017.stdout:(33/37): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.2 MB/s | 90 kB 00:00 2024-09-11T09:47:56.466 INFO:teuthology.orchestra.run.smithi107.stdout:(2/37): ceph-mgr-19.1.1-298.gb53b6d3e.el9.x86_6 4.5 MB/s | 1.5 MB 00:00 2024-09-11T09:47:56.481 INFO:teuthology.orchestra.run.smithi017.stdout:(34/37): python3-pecan-1.4.2-3.el9.noarch.rpm 1.7 MB/s | 272 kB 00:00 2024-09-11T09:47:56.506 INFO:teuthology.orchestra.run.smithi017.stdout:(35/37): python3-zc-lockfile-2.0-10.el9.noarch. 800 kB/s | 20 kB 00:00 2024-09-11T09:47:56.531 INFO:teuthology.orchestra.run.smithi017.stdout:(36/37): python3-webob-1.8.8-2.el9.noarch.rpm 2.1 MB/s | 230 kB 00:00 2024-09-11T09:47:56.533 INFO:teuthology.orchestra.run.smithi107.stdout:(3/37): ceph-mds-19.1.1-298.gb53b6d3e.el9.x86_6 5.8 MB/s | 2.4 MB 00:00 2024-09-11T09:47:56.556 INFO:teuthology.orchestra.run.smithi017.stdout:(37/37): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.8 MB/s | 427 kB 00:00 2024-09-11T09:47:56.561 INFO:teuthology.orchestra.run.smithi017.stdout:-------------------------------------------------------------------------------- 2024-09-11T09:47:56.561 INFO:teuthology.orchestra.run.smithi017.stdout:Total 12 MB/s | 30 MB 00:02 2024-09-11T09:47:56.608 INFO:teuthology.orchestra.run.smithi107.stdout:(4/37): ceph-mgr-modules-core-19.1.1-298.gb53b6 3.3 MB/s | 249 kB 00:00 2024-09-11T09:47:56.718 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction check 2024-09-11T09:47:56.750 INFO:teuthology.orchestra.run.smithi107.stdout:(5/37): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.7 MB/s | 253 kB 00:00 2024-09-11T09:47:56.763 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction check succeeded. 2024-09-11T09:47:56.764 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction test 2024-09-11T09:47:56.892 INFO:teuthology.orchestra.run.smithi107.stdout:(6/37): ceph-mon-19.1.1-298.gb53b6d3e.el9.x86_6 7.3 MB/s | 4.7 MB 00:00 2024-09-11T09:47:56.925 INFO:teuthology.orchestra.run.smithi107.stdout:(7/37): python3-cryptography-36.0.1-4.el9.x86_6 7.0 MB/s | 1.2 MB 00:00 2024-09-11T09:47:56.940 INFO:teuthology.orchestra.run.smithi096.stdout:(6/37): ceph-osd-19.1.1-298.gb53b6d3e.el9.x86_6 14 MB/s | 17 MB 00:01 2024-09-11T09:47:56.965 INFO:teuthology.orchestra.run.smithi096.stdout:(7/37): python3-cffi-1.14.5-5.el9.x86_64.rpm 284 kB/s | 253 kB 00:00 2024-09-11T09:47:57.143 INFO:teuthology.orchestra.run.smithi107.stdout:(8/37): ceph-osd-19.1.1-298.gb53b6d3e.el9.x86_6 25 MB/s | 17 MB 00:00 2024-09-11T09:47:57.168 INFO:teuthology.orchestra.run.smithi107.stdout:(9/37): python3-pycparser-2.20-6.el9.noarch.rpm 559 kB/s | 135 kB 00:00 2024-09-11T09:47:57.185 INFO:teuthology.orchestra.run.smithi107.stdout:(10/37): python3-ply-3.11-14.el9.noarch.rpm 364 kB/s | 106 kB 00:00 2024-09-11T09:47:57.243 INFO:teuthology.orchestra.run.smithi107.stdout:(11/37): python3-urllib3-1.26.5-6.el9.noarch.rp 2.8 MB/s | 215 kB 00:00 2024-09-11T09:47:57.272 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction test succeeded. 2024-09-11T09:47:57.272 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction 2024-09-11T09:47:57.274 INFO:teuthology.orchestra.run.smithi096.stdout:(8/37): python3-cryptography-36.0.1-4.el9.x86_6 1.0 MB/s | 1.2 MB 00:01 2024-09-11T09:47:57.286 INFO:teuthology.orchestra.run.smithi107.stdout:(12/37): python3-requests-2.25.1-8.el9.noarch.r 875 kB/s | 125 kB 00:00 2024-09-11T09:47:57.328 INFO:teuthology.orchestra.run.smithi107.stdout:(13/37): lua-5.4.4-4.el9.x86_64.rpm 1.3 MB/s | 188 kB 00:00 2024-09-11T09:47:57.369 INFO:teuthology.orchestra.run.smithi107.stdout:(14/37): python3-packaging-20.9-5.el9.noarch.rp 1.8 MB/s | 77 kB 00:00 2024-09-11T09:47:57.386 INFO:teuthology.orchestra.run.smithi107.stdout:(15/37): python3-markupsafe-1.1.1-12.el9.x86_64 346 kB/s | 35 kB 00:00 2024-09-11T09:47:57.412 INFO:teuthology.orchestra.run.smithi107.stdout:(16/37): python3-mako-1.1.4-6.el9.noarch.rpm 1.0 MB/s | 172 kB 00:00 2024-09-11T09:47:57.437 INFO:teuthology.orchestra.run.smithi107.stdout:(17/37): python3-pytz-2021.1-5.el9.noarch.rpm 757 kB/s | 51 kB 00:00 2024-09-11T09:47:57.462 INFO:teuthology.orchestra.run.smithi107.stdout:(18/37): python3-toml-0.10.2-6.el9.noarch.rpm 551 kB/s | 42 kB 00:00 2024-09-11T09:47:57.488 INFO:teuthology.orchestra.run.smithi107.stdout:(19/37): lua-devel-5.4.4-4.el9.x86_64.rpm 295 kB/s | 22 kB 00:00 2024-09-11T09:47:57.538 INFO:teuthology.orchestra.run.smithi107.stdout:(20/37): luarocks-3.9.2-5.el9.noarch.rpm 2.0 MB/s | 151 kB 00:00 2024-09-11T09:47:57.563 INFO:teuthology.orchestra.run.smithi107.stdout:(21/37): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 433 kB/s | 43 kB 00:00 2024-09-11T09:47:57.588 INFO:teuthology.orchestra.run.smithi107.stdout:(22/37): python3-jaraco-8.2.1-3.el9.noarch.rpm 429 kB/s | 11 kB 00:00 2024-09-11T09:47:57.607 INFO:teuthology.orchestra.run.smithi096.stdout:(9/37): python3-ply-3.11-14.el9.noarch.rpm 160 kB/s | 106 kB 00:00 2024-09-11T09:47:57.613 INFO:teuthology.orchestra.run.smithi107.stdout:(23/37): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.3 MB/s | 172 kB 00:00 2024-09-11T09:47:57.639 INFO:teuthology.orchestra.run.smithi107.stdout:(24/37): python3-jaraco-classes-3.2.1-5.el9.noa 351 kB/s | 18 kB 00:00 2024-09-11T09:47:57.664 INFO:teuthology.orchestra.run.smithi107.stdout:(25/37): python3-jaraco-collections-3.0.0-8.el9 461 kB/s | 23 kB 00:00 2024-09-11T09:47:57.689 INFO:teuthology.orchestra.run.smithi107.stdout:(26/37): python3-jaraco-functools-3.5.0-2.el9.n 387 kB/s | 19 kB 00:00 2024-09-11T09:47:57.715 INFO:teuthology.orchestra.run.smithi107.stdout:(27/37): python3-jaraco-text-3.2.0-6.el9.noarch 391 kB/s | 20 kB 00:00 2024-09-11T09:47:57.733 INFO:teuthology.orchestra.run.smithi096.stdout:(10/37): python3-pycparser-2.20-6.el9.noarch.rp 176 kB/s | 135 kB 00:00 2024-09-11T09:47:57.740 INFO:teuthology.orchestra.run.smithi107.stdout:(28/37): python3-cherrypy-18.6.1-2.el9.noarch.r 1.8 MB/s | 358 kB 00:00 2024-09-11T09:47:57.765 INFO:teuthology.orchestra.run.smithi107.stdout:(29/37): python3-logutils-0.3.5-21.el9.noarch.r 612 kB/s | 46 kB 00:00 2024-09-11T09:47:57.790 INFO:teuthology.orchestra.run.smithi107.stdout:(30/37): python3-more-itertools-8.12.0-2.el9.no 1.0 MB/s | 79 kB 00:00 2024-09-11T09:47:57.816 INFO:teuthology.orchestra.run.smithi107.stdout:(31/37): python3-portend-3.1.0-2.el9.noarch.rpm 328 kB/s | 16 kB 00:00 2024-09-11T09:47:57.841 INFO:teuthology.orchestra.run.smithi107.stdout:(32/37): python3-tempora-5.0.0-2.el9.noarch.rpm 1.4 MB/s | 36 kB 00:00 2024-09-11T09:47:57.867 INFO:teuthology.orchestra.run.smithi107.stdout:(33/37): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.2 MB/s | 90 kB 00:00 2024-09-11T09:47:57.875 INFO:teuthology.orchestra.run.smithi096.stdout:(11/37): lua-5.4.4-4.el9.x86_64.rpm 1.3 MB/s | 188 kB 00:00 2024-09-11T09:47:57.891 INFO:teuthology.orchestra.run.smithi107.stdout:(34/37): python3-pecan-1.4.2-3.el9.noarch.rpm 2.1 MB/s | 272 kB 00:00 2024-09-11T09:47:57.917 INFO:teuthology.orchestra.run.smithi107.stdout:(35/37): python3-zc-lockfile-2.0-10.el9.noarch. 796 kB/s | 20 kB 00:00 2024-09-11T09:47:57.942 INFO:teuthology.orchestra.run.smithi107.stdout:(36/37): python3-webob-1.8.8-2.el9.noarch.rpm 2.2 MB/s | 230 kB 00:00 2024-09-11T09:47:57.950 INFO:teuthology.orchestra.run.smithi096.stdout:(12/37): python3-mako-1.1.4-6.el9.noarch.rpm 2.2 MB/s | 172 kB 00:00 2024-09-11T09:47:57.967 INFO:teuthology.orchestra.run.smithi107.stdout:(37/37): python3-werkzeug-2.0.3-3.el9.1.noarch. 4.1 MB/s | 427 kB 00:00 2024-09-11T09:47:57.972 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2024-09-11T09:47:57.972 INFO:teuthology.orchestra.run.smithi107.stdout:Total 13 MB/s | 30 MB 00:02 2024-09-11T09:47:57.975 INFO:teuthology.orchestra.run.smithi096.stdout:(13/37): python3-requests-2.25.1-8.el9.noarch.r 178 kB/s | 125 kB 00:00 2024-09-11T09:47:58.017 INFO:teuthology.orchestra.run.smithi096.stdout:(14/37): python3-markupsafe-1.1.1-12.el9.x86_64 519 kB/s | 35 kB 00:00 2024-09-11T09:47:58.076 INFO:teuthology.orchestra.run.smithi096.stdout:(15/37): python3-packaging-20.9-5.el9.noarch.rp 773 kB/s | 77 kB 00:00 2024-09-11T09:47:58.101 INFO:teuthology.orchestra.run.smithi096.stdout:(16/37): python3-pytz-2021.1-5.el9.noarch.rpm 611 kB/s | 51 kB 00:00 2024-09-11T09:47:58.119 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2024-09-11T09:47:58.143 INFO:teuthology.orchestra.run.smithi096.stdout:(17/37): python3-toml-0.10.2-6.el9.noarch.rpm 625 kB/s | 42 kB 00:00 2024-09-11T09:47:58.158 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2024-09-11T09:47:58.158 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2024-09-11T09:47:58.193 INFO:teuthology.orchestra.run.smithi096.stdout:(18/37): luarocks-3.9.2-5.el9.noarch.rpm 3.0 MB/s | 151 kB 00:00 2024-09-11T09:47:58.219 INFO:teuthology.orchestra.run.smithi096.stdout:(19/37): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 1.7 MB/s | 43 kB 00:00 2024-09-11T09:47:58.244 INFO:teuthology.orchestra.run.smithi096.stdout:(20/37): lua-devel-5.4.4-4.el9.x86_64.rpm 156 kB/s | 22 kB 00:00 2024-09-11T09:47:58.278 INFO:teuthology.orchestra.run.smithi096.stdout:(21/37): python3-cheroot-8.6.0-4.el9.noarch.rpm 2.9 MB/s | 172 kB 00:00 2024-09-11T09:47:58.303 INFO:teuthology.orchestra.run.smithi096.stdout:(22/37): python3-jaraco-8.2.1-3.el9.noarch.rpm 462 kB/s | 11 kB 00:00 2024-09-11T09:47:58.328 INFO:teuthology.orchestra.run.smithi096.stdout:(23/37): python3-jaraco-classes-3.2.1-5.el9.noa 712 kB/s | 18 kB 00:00 2024-09-11T09:47:58.354 INFO:teuthology.orchestra.run.smithi096.stdout:(24/37): python3-urllib3-1.26.5-6.el9.noarch.rp 288 kB/s | 215 kB 00:00 2024-09-11T09:47:58.379 INFO:teuthology.orchestra.run.smithi096.stdout:(25/37): python3-jaraco-collections-3.0.0-8.el9 461 kB/s | 23 kB 00:00 2024-09-11T09:47:58.412 INFO:teuthology.orchestra.run.smithi096.stdout:(26/37): python3-jaraco-text-3.2.0-6.el9.noarch 762 kB/s | 20 kB 00:00 2024-09-11T09:47:58.441 INFO:teuthology.orchestra.run.smithi096.stdout:(27/37): python3-cherrypy-18.6.1-2.el9.noarch.r 1.8 MB/s | 358 kB 00:00 2024-09-11T09:47:58.458 INFO:teuthology.orchestra.run.smithi096.stdout:(28/37): python3-logutils-0.3.5-21.el9.noarch.r 1.0 MB/s | 46 kB 00:00 2024-09-11T09:47:58.475 INFO:teuthology.orchestra.run.smithi096.stdout:(29/37): python3-jaraco-functools-3.5.0-2.el9.n 160 kB/s | 19 kB 00:00 2024-09-11T09:47:58.500 INFO:teuthology.orchestra.run.smithi096.stdout:(30/37): python3-more-itertools-8.12.0-2.el9.no 1.3 MB/s | 79 kB 00:00 2024-09-11T09:47:58.517 INFO:teuthology.orchestra.run.smithi096.stdout:(31/37): python3-portend-3.1.0-2.el9.noarch.rpm 394 kB/s | 16 kB 00:00 2024-09-11T09:47:58.534 INFO:teuthology.orchestra.run.smithi096.stdout:(32/37): python3-pyOpenSSL-21.0.0-1.el9.noarch. 5.3 MB/s | 90 kB 00:00 2024-09-11T09:47:58.551 INFO:teuthology.orchestra.run.smithi096.stdout:(33/37): python3-tempora-5.0.0-2.el9.noarch.rpm 1.0 MB/s | 36 kB 00:00 2024-09-11T09:47:58.576 INFO:teuthology.orchestra.run.smithi096.stdout:(34/37): python3-pecan-1.4.2-3.el9.noarch.rpm 2.3 MB/s | 272 kB 00:00 2024-09-11T09:47:58.593 INFO:teuthology.orchestra.run.smithi096.stdout:(35/37): python3-zc-lockfile-2.0-10.el9.noarch. 1.2 MB/s | 20 kB 00:00 2024-09-11T09:47:58.618 INFO:teuthology.orchestra.run.smithi096.stdout:(36/37): python3-webob-1.8.8-2.el9.noarch.rpm 3.4 MB/s | 230 kB 00:00 2024-09-11T09:47:58.639 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2024-09-11T09:47:58.639 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2024-09-11T09:47:58.644 INFO:teuthology.orchestra.run.smithi096.stdout:(37/37): python3-werkzeug-2.0.3-3.el9.1.noarch. 4.5 MB/s | 427 kB 00:00 2024-09-11T09:47:58.649 INFO:teuthology.orchestra.run.smithi096.stdout:-------------------------------------------------------------------------------- 2024-09-11T09:47:58.649 INFO:teuthology.orchestra.run.smithi096.stdout:Total 8.0 MB/s | 30 MB 00:03 2024-09-11T09:47:58.657 INFO:teuthology.orchestra.run.smithi017.stdout: Preparing : 1/1 2024-09-11T09:47:58.726 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/37 2024-09-11T09:47:58.760 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/37 2024-09-11T09:47:58.796 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/37 2024-09-11T09:47:58.803 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction check 2024-09-11T09:47:58.842 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction check succeeded. 2024-09-11T09:47:58.843 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction test 2024-09-11T09:47:58.871 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/37 2024-09-11T09:47:58.912 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : luarocks-3.9.2-5.el9.noarch 5/37 2024-09-11T09:47:58.944 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/37 2024-09-11T09:47:59.041 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/37 2024-09-11T09:47:59.117 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/37 2024-09-11T09:47:59.164 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 9/37 2024-09-11T09:47:59.199 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/37 2024-09-11T09:47:59.229 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/37 2024-09-11T09:47:59.263 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/37 2024-09-11T09:47:59.305 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/37 2024-09-11T09:47:59.328 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction test succeeded. 2024-09-11T09:47:59.328 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction 2024-09-11T09:47:59.346 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/37 2024-09-11T09:47:59.381 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/37 2024-09-11T09:47:59.414 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/37 2024-09-11T09:47:59.464 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/37 2024-09-11T09:47:59.504 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-packaging-20.9-5.el9.noarch 18/37 2024-09-11T09:47:59.583 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 19/37 2024-09-11T09:47:59.727 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 20/37 2024-09-11T09:47:59.818 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 21/37 2024-09-11T09:47:59.869 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 22/37 2024-09-11T09:47:59.907 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 23/37 2024-09-11T09:47:59.965 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-ply-3.11-14.el9.noarch 24/37 2024-09-11T09:48:00.029 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 25/37 2024-09-11T09:48:00.035 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2024-09-11T09:48:00.104 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/37 2024-09-11T09:48:00.145 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/37 2024-09-11T09:48:00.173 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/37 2024-09-11T09:48:00.225 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 26/37 2024-09-11T09:48:00.244 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/37 2024-09-11T09:48:00.269 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 27/37 2024-09-11T09:48:00.305 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : luarocks-3.9.2-5.el9.noarch 5/37 2024-09-11T09:48:00.338 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/37 2024-09-11T09:48:00.351 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 28/37 2024-09-11T09:48:00.431 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/37 2024-09-11T09:48:00.447 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 29/37 2024-09-11T09:48:00.485 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 30/37 2024-09-11T09:48:00.500 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/37 2024-09-11T09:48:00.541 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 9/37 2024-09-11T09:48:00.576 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/37 2024-09-11T09:48:00.606 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/37 2024-09-11T09:48:00.609 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 31/37 2024-09-11T09:48:00.640 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/37 2024-09-11T09:48:00.674 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/37 2024-09-11T09:48:00.715 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/37 2024-09-11T09:48:00.731 INFO:teuthology.orchestra.run.smithi096.stdout: Preparing : 1/1 2024-09-11T09:48:00.758 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/37 2024-09-11T09:48:00.780 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : ceph-mgr-modules-core-2:19.1.1-298.gb53b6d3e.el9.n 32/37 2024-09-11T09:48:00.792 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/37 2024-09-11T09:48:00.797 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/37 2024-09-11T09:48:00.811 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : ceph-mgr-2:19.1.1-298.gb53b6d3e.el9.x86_64 33/37 2024-09-11T09:48:00.838 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/37 2024-09-11T09:48:00.839 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: ceph-mgr-2:19.1.1-298.gb53b6d3e.el9.x86_64 33/37 2024-09-11T09:48:00.839 INFO:teuthology.orchestra.run.smithi017.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-11T09:48:00.839 INFO:teuthology.orchestra.run.smithi017.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-11T09:48:00.839 INFO:teuthology.orchestra.run.smithi017.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-11T09:48:00.840 INFO:teuthology.orchestra.run.smithi017.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-11T09:48:00.840 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-11T09:48:00.841 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/37 2024-09-11T09:48:00.866 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/37 2024-09-11T09:48:00.879 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-packaging-20.9-5.el9.noarch 18/37 2024-09-11T09:48:00.936 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/37 2024-09-11T09:48:00.958 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 19/37 2024-09-11T09:48:00.981 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : luarocks-3.9.2-5.el9.noarch 5/37 2024-09-11T09:48:01.014 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/37 2024-09-11T09:48:01.105 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/37 2024-09-11T09:48:01.107 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 20/37 2024-09-11T09:48:01.186 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 21/37 2024-09-11T09:48:01.192 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/37 2024-09-11T09:48:01.234 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 9/37 2024-09-11T09:48:01.238 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 22/37 2024-09-11T09:48:01.277 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 23/37 2024-09-11T09:48:01.277 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/37 2024-09-11T09:48:01.307 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/37 2024-09-11T09:48:01.334 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-ply-3.11-14.el9.noarch 24/37 2024-09-11T09:48:01.342 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/37 2024-09-11T09:48:01.375 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/37 2024-09-11T09:48:01.389 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 25/37 2024-09-11T09:48:01.416 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/37 2024-09-11T09:48:01.459 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/37 2024-09-11T09:48:01.494 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/37 2024-09-11T09:48:01.542 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/37 2024-09-11T09:48:01.576 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 26/37 2024-09-11T09:48:01.580 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-packaging-20.9-5.el9.noarch 18/37 2024-09-11T09:48:01.640 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 27/37 2024-09-11T09:48:01.655 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 19/37 2024-09-11T09:48:01.717 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 28/37 2024-09-11T09:48:01.792 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 20/37 2024-09-11T09:48:01.806 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 29/37 2024-09-11T09:48:01.847 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 30/37 2024-09-11T09:48:01.882 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 21/37 2024-09-11T09:48:01.932 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 22/37 2024-09-11T09:48:01.963 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 31/37 2024-09-11T09:48:01.972 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 23/37 2024-09-11T09:48:02.020 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-ply-3.11-14.el9.noarch 24/37 2024-09-11T09:48:02.076 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 25/37 2024-09-11T09:48:02.128 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-mgr-modules-core-2:19.1.1-298.gb53b6d3e.el9.n 32/37 2024-09-11T09:48:02.157 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-mgr-2:19.1.1-298.gb53b6d3e.el9.x86_64 33/37 2024-09-11T09:48:02.184 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-mgr-2:19.1.1-298.gb53b6d3e.el9.x86_64 33/37 2024-09-11T09:48:02.184 INFO:teuthology.orchestra.run.smithi107.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-11T09:48:02.185 INFO:teuthology.orchestra.run.smithi107.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-11T09:48:02.185 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-11T09:48:02.185 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-11T09:48:02.185 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-11T09:48:02.261 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 26/37 2024-09-11T09:48:02.309 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 27/37 2024-09-11T09:48:02.370 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : ceph-osd-2:19.1.1-298.gb53b6d3e.el9.x86_64 34/37 2024-09-11T09:48:02.386 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 28/37 2024-09-11T09:48:02.399 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: ceph-osd-2:19.1.1-298.gb53b6d3e.el9.x86_64 34/37 2024-09-11T09:48:02.399 INFO:teuthology.orchestra.run.smithi017.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-11T09:48:02.400 INFO:teuthology.orchestra.run.smithi017.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-11T09:48:02.400 INFO:teuthology.orchestra.run.smithi017.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-11T09:48:02.400 INFO:teuthology.orchestra.run.smithi017.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-11T09:48:02.400 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-11T09:48:02.475 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 29/37 2024-09-11T09:48:02.517 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 30/37 2024-09-11T09:48:02.633 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 31/37 2024-09-11T09:48:02.788 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-mgr-modules-core-2:19.1.1-298.gb53b6d3e.el9.n 32/37 2024-09-11T09:48:02.819 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-mgr-2:19.1.1-298.gb53b6d3e.el9.x86_64 33/37 2024-09-11T09:48:02.845 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-mgr-2:19.1.1-298.gb53b6d3e.el9.x86_64 33/37 2024-09-11T09:48:02.845 INFO:teuthology.orchestra.run.smithi096.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-11T09:48:02.845 INFO:teuthology.orchestra.run.smithi096.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-11T09:48:02.845 INFO:teuthology.orchestra.run.smithi096.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-11T09:48:02.845 INFO:teuthology.orchestra.run.smithi096.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-11T09:48:02.846 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-11T09:48:02.854 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : ceph-mon-2:19.1.1-298.gb53b6d3e.el9.x86_64 35/37 2024-09-11T09:48:02.880 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: ceph-mon-2:19.1.1-298.gb53b6d3e.el9.x86_64 35/37 2024-09-11T09:48:02.880 INFO:teuthology.orchestra.run.smithi017.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-11T09:48:02.880 INFO:teuthology.orchestra.run.smithi017.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-11T09:48:02.880 INFO:teuthology.orchestra.run.smithi017.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-11T09:48:02.880 INFO:teuthology.orchestra.run.smithi017.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-11T09:48:02.880 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-11T09:48:03.122 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : ceph-mds-2:19.1.1-298.gb53b6d3e.el9.x86_64 36/37 2024-09-11T09:48:03.147 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: ceph-mds-2:19.1.1-298.gb53b6d3e.el9.x86_64 36/37 2024-09-11T09:48:03.147 INFO:teuthology.orchestra.run.smithi017.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-11T09:48:03.147 INFO:teuthology.orchestra.run.smithi017.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-11T09:48:03.147 INFO:teuthology.orchestra.run.smithi017.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-11T09:48:03.147 INFO:teuthology.orchestra.run.smithi017.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-11T09:48:03.148 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-11T09:48:03.214 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : ceph-2:19.1.1-298.gb53b6d3e.el9.x86_64 37/37 2024-09-11T09:48:03.625 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-osd-2:19.1.1-298.gb53b6d3e.el9.x86_64 34/37 2024-09-11T09:48:03.652 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-osd-2:19.1.1-298.gb53b6d3e.el9.x86_64 34/37 2024-09-11T09:48:03.653 INFO:teuthology.orchestra.run.smithi107.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-11T09:48:03.653 INFO:teuthology.orchestra.run.smithi107.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-11T09:48:03.653 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-11T09:48:03.653 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-11T09:48:03.653 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-11T09:48:04.084 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-mon-2:19.1.1-298.gb53b6d3e.el9.x86_64 35/37 2024-09-11T09:48:04.108 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-mon-2:19.1.1-298.gb53b6d3e.el9.x86_64 35/37 2024-09-11T09:48:04.108 INFO:teuthology.orchestra.run.smithi107.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-11T09:48:04.108 INFO:teuthology.orchestra.run.smithi107.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-11T09:48:04.108 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-11T09:48:04.109 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-11T09:48:04.109 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-11T09:48:04.286 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-osd-2:19.1.1-298.gb53b6d3e.el9.x86_64 34/37 2024-09-11T09:48:04.313 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-osd-2:19.1.1-298.gb53b6d3e.el9.x86_64 34/37 2024-09-11T09:48:04.314 INFO:teuthology.orchestra.run.smithi096.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-11T09:48:04.314 INFO:teuthology.orchestra.run.smithi096.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-11T09:48:04.314 INFO:teuthology.orchestra.run.smithi096.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-11T09:48:04.314 INFO:teuthology.orchestra.run.smithi096.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-11T09:48:04.314 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-11T09:48:04.335 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-mds-2:19.1.1-298.gb53b6d3e.el9.x86_64 36/37 2024-09-11T09:48:04.358 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-mds-2:19.1.1-298.gb53b6d3e.el9.x86_64 36/37 2024-09-11T09:48:04.358 INFO:teuthology.orchestra.run.smithi107.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-11T09:48:04.358 INFO:teuthology.orchestra.run.smithi107.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-11T09:48:04.359 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-11T09:48:04.359 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-11T09:48:04.359 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-11T09:48:04.424 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-2:19.1.1-298.gb53b6d3e.el9.x86_64 37/37 2024-09-11T09:48:04.745 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-mon-2:19.1.1-298.gb53b6d3e.el9.x86_64 35/37 2024-09-11T09:48:04.769 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-mon-2:19.1.1-298.gb53b6d3e.el9.x86_64 35/37 2024-09-11T09:48:04.769 INFO:teuthology.orchestra.run.smithi096.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-11T09:48:04.769 INFO:teuthology.orchestra.run.smithi096.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-11T09:48:04.769 INFO:teuthology.orchestra.run.smithi096.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-11T09:48:04.769 INFO:teuthology.orchestra.run.smithi096.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-11T09:48:04.770 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-11T09:48:05.004 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-mds-2:19.1.1-298.gb53b6d3e.el9.x86_64 36/37 2024-09-11T09:48:05.027 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-mds-2:19.1.1-298.gb53b6d3e.el9.x86_64 36/37 2024-09-11T09:48:05.028 INFO:teuthology.orchestra.run.smithi096.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-11T09:48:05.028 INFO:teuthology.orchestra.run.smithi096.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-11T09:48:05.028 INFO:teuthology.orchestra.run.smithi096.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-11T09:48:05.028 INFO:teuthology.orchestra.run.smithi096.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-11T09:48:05.028 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-11T09:48:05.041 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: ceph-2:19.1.1-298.gb53b6d3e.el9.x86_64 37/37 2024-09-11T09:48:05.041 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : ceph-2:19.1.1-298.gb53b6d3e.el9.x86_64 1/37 2024-09-11T09:48:05.041 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : ceph-mds-2:19.1.1-298.gb53b6d3e.el9.x86_64 2/37 2024-09-11T09:48:05.041 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : ceph-mgr-2:19.1.1-298.gb53b6d3e.el9.x86_64 3/37 2024-09-11T09:48:05.042 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : ceph-mon-2:19.1.1-298.gb53b6d3e.el9.x86_64 4/37 2024-09-11T09:48:05.042 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : ceph-osd-2:19.1.1-298.gb53b6d3e.el9.x86_64 5/37 2024-09-11T09:48:05.042 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : ceph-mgr-modules-core-2:19.1.1-298.gb53b6d3e.el9.n 6/37 2024-09-11T09:48:05.042 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/37 2024-09-11T09:48:05.042 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/37 2024-09-11T09:48:05.042 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/37 2024-09-11T09:48:05.042 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/37 2024-09-11T09:48:05.042 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/37 2024-09-11T09:48:05.042 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/37 2024-09-11T09:48:05.042 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/37 2024-09-11T09:48:05.042 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/37 2024-09-11T09:48:05.042 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/37 2024-09-11T09:48:05.043 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 16/37 2024-09-11T09:48:05.043 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 17/37 2024-09-11T09:48:05.043 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 18/37 2024-09-11T09:48:05.043 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 19/37 2024-09-11T09:48:05.043 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 20/37 2024-09-11T09:48:05.043 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 21/37 2024-09-11T09:48:05.043 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 22/37 2024-09-11T09:48:05.043 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 23/37 2024-09-11T09:48:05.043 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 24/37 2024-09-11T09:48:05.043 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 25/37 2024-09-11T09:48:05.043 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 26/37 2024-09-11T09:48:05.043 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 27/37 2024-09-11T09:48:05.044 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 28/37 2024-09-11T09:48:05.045 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 29/37 2024-09-11T09:48:05.045 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 30/37 2024-09-11T09:48:05.045 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 31/37 2024-09-11T09:48:05.045 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 32/37 2024-09-11T09:48:05.045 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 33/37 2024-09-11T09:48:05.045 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 34/37 2024-09-11T09:48:05.045 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 35/37 2024-09-11T09:48:05.045 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 36/37 2024-09-11T09:48:05.093 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-2:19.1.1-298.gb53b6d3e.el9.x86_64 37/37 2024-09-11T09:48:06.145 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 37/37 2024-09-11T09:48:06.145 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-11T09:48:06.145 INFO:teuthology.orchestra.run.smithi017.stdout:Installed: 2024-09-11T09:48:06.146 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-11T09:48:06.146 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-mds-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-11T09:48:06.146 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-mgr-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-11T09:48:06.146 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-mgr-modules-core-2:19.1.1-298.gb53b6d3e.el9.noarch 2024-09-11T09:48:06.146 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-mon-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-11T09:48:06.146 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-osd-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-11T09:48:06.146 INFO:teuthology.orchestra.run.smithi017.stdout: lua-5.4.4-4.el9.x86_64 2024-09-11T09:48:06.146 INFO:teuthology.orchestra.run.smithi017.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-09-11T09:48:06.146 INFO:teuthology.orchestra.run.smithi017.stdout: luarocks-3.9.2-5.el9.noarch 2024-09-11T09:48:06.146 INFO:teuthology.orchestra.run.smithi017.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-11T09:48:06.146 INFO:teuthology.orchestra.run.smithi017.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-11T09:48:06.146 INFO:teuthology.orchestra.run.smithi017.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-11T09:48:06.147 INFO:teuthology.orchestra.run.smithi017.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-11T09:48:06.147 INFO:teuthology.orchestra.run.smithi017.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-11T09:48:06.147 INFO:teuthology.orchestra.run.smithi017.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-11T09:48:06.147 INFO:teuthology.orchestra.run.smithi017.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-11T09:48:06.147 INFO:teuthology.orchestra.run.smithi017.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-11T09:48:06.147 INFO:teuthology.orchestra.run.smithi017.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-11T09:48:06.147 INFO:teuthology.orchestra.run.smithi017.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-11T09:48:06.147 INFO:teuthology.orchestra.run.smithi017.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-11T09:48:06.147 INFO:teuthology.orchestra.run.smithi017.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-11T09:48:06.147 INFO:teuthology.orchestra.run.smithi017.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-11T09:48:06.147 INFO:teuthology.orchestra.run.smithi017.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-11T09:48:06.147 INFO:teuthology.orchestra.run.smithi017.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-11T09:48:06.147 INFO:teuthology.orchestra.run.smithi017.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-11T09:48:06.148 INFO:teuthology.orchestra.run.smithi017.stdout: python3-ply-3.11-14.el9.noarch 2024-09-11T09:48:06.148 INFO:teuthology.orchestra.run.smithi017.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-11T09:48:06.148 INFO:teuthology.orchestra.run.smithi017.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-11T09:48:06.148 INFO:teuthology.orchestra.run.smithi017.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-11T09:48:06.148 INFO:teuthology.orchestra.run.smithi017.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-11T09:48:06.148 INFO:teuthology.orchestra.run.smithi017.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-11T09:48:06.148 INFO:teuthology.orchestra.run.smithi017.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-11T09:48:06.148 INFO:teuthology.orchestra.run.smithi017.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-11T09:48:06.148 INFO:teuthology.orchestra.run.smithi017.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-11T09:48:06.148 INFO:teuthology.orchestra.run.smithi017.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-11T09:48:06.148 INFO:teuthology.orchestra.run.smithi017.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-11T09:48:06.148 INFO:teuthology.orchestra.run.smithi017.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-11T09:48:06.148 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-11T09:48:06.149 INFO:teuthology.orchestra.run.smithi017.stdout:Complete! 2024-09-11T09:48:06.213 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-2:19.1.1-298.gb53b6d3e.el9.x86_64 37/37 2024-09-11T09:48:06.213 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-2:19.1.1-298.gb53b6d3e.el9.x86_64 1/37 2024-09-11T09:48:06.213 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-mds-2:19.1.1-298.gb53b6d3e.el9.x86_64 2/37 2024-09-11T09:48:06.213 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-mgr-2:19.1.1-298.gb53b6d3e.el9.x86_64 3/37 2024-09-11T09:48:06.214 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-mon-2:19.1.1-298.gb53b6d3e.el9.x86_64 4/37 2024-09-11T09:48:06.214 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-osd-2:19.1.1-298.gb53b6d3e.el9.x86_64 5/37 2024-09-11T09:48:06.214 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-mgr-modules-core-2:19.1.1-298.gb53b6d3e.el9.n 6/37 2024-09-11T09:48:06.214 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/37 2024-09-11T09:48:06.214 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/37 2024-09-11T09:48:06.214 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/37 2024-09-11T09:48:06.214 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/37 2024-09-11T09:48:06.214 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/37 2024-09-11T09:48:06.214 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/37 2024-09-11T09:48:06.214 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/37 2024-09-11T09:48:06.214 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/37 2024-09-11T09:48:06.214 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/37 2024-09-11T09:48:06.215 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 16/37 2024-09-11T09:48:06.215 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 17/37 2024-09-11T09:48:06.215 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 18/37 2024-09-11T09:48:06.215 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 19/37 2024-09-11T09:48:06.215 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 20/37 2024-09-11T09:48:06.215 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 21/37 2024-09-11T09:48:06.215 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 22/37 2024-09-11T09:48:06.215 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 23/37 2024-09-11T09:48:06.215 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 24/37 2024-09-11T09:48:06.215 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 25/37 2024-09-11T09:48:06.215 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 26/37 2024-09-11T09:48:06.215 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 27/37 2024-09-11T09:48:06.215 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 28/37 2024-09-11T09:48:06.216 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 29/37 2024-09-11T09:48:06.216 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 30/37 2024-09-11T09:48:06.216 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 31/37 2024-09-11T09:48:06.216 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 32/37 2024-09-11T09:48:06.216 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 33/37 2024-09-11T09:48:06.216 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 34/37 2024-09-11T09:48:06.216 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 35/37 2024-09-11T09:48:06.216 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 36/37 2024-09-11T09:48:06.483 DEBUG:teuthology.orchestra.run.smithi017:> sudo yum -y install ceph-base 2024-09-11T09:48:06.844 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-2:19.1.1-298.gb53b6d3e.el9.x86_64 37/37 2024-09-11T09:48:06.845 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-2:19.1.1-298.gb53b6d3e.el9.x86_64 1/37 2024-09-11T09:48:06.845 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-mds-2:19.1.1-298.gb53b6d3e.el9.x86_64 2/37 2024-09-11T09:48:06.845 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-mgr-2:19.1.1-298.gb53b6d3e.el9.x86_64 3/37 2024-09-11T09:48:06.845 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-mon-2:19.1.1-298.gb53b6d3e.el9.x86_64 4/37 2024-09-11T09:48:06.845 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-osd-2:19.1.1-298.gb53b6d3e.el9.x86_64 5/37 2024-09-11T09:48:06.845 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-mgr-modules-core-2:19.1.1-298.gb53b6d3e.el9.n 6/37 2024-09-11T09:48:06.845 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/37 2024-09-11T09:48:06.845 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/37 2024-09-11T09:48:06.845 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/37 2024-09-11T09:48:06.845 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/37 2024-09-11T09:48:06.845 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/37 2024-09-11T09:48:06.845 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/37 2024-09-11T09:48:06.846 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/37 2024-09-11T09:48:06.846 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/37 2024-09-11T09:48:06.846 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/37 2024-09-11T09:48:06.846 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 16/37 2024-09-11T09:48:06.846 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 17/37 2024-09-11T09:48:06.846 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 18/37 2024-09-11T09:48:06.846 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 19/37 2024-09-11T09:48:06.846 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 20/37 2024-09-11T09:48:06.846 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 21/37 2024-09-11T09:48:06.846 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 22/37 2024-09-11T09:48:06.846 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 23/37 2024-09-11T09:48:06.846 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 24/37 2024-09-11T09:48:06.847 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 25/37 2024-09-11T09:48:06.847 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 26/37 2024-09-11T09:48:06.847 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 27/37 2024-09-11T09:48:06.847 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 28/37 2024-09-11T09:48:06.848 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 29/37 2024-09-11T09:48:06.848 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 30/37 2024-09-11T09:48:06.848 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 31/37 2024-09-11T09:48:06.848 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 32/37 2024-09-11T09:48:06.848 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 33/37 2024-09-11T09:48:06.848 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 34/37 2024-09-11T09:48:06.848 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 35/37 2024-09-11T09:48:06.848 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 36/37 2024-09-11T09:48:07.077 INFO:teuthology.orchestra.run.smithi017.stdout:Last metadata expiration check: 0:00:54 ago on Wed 11 Sep 2024 09:47:13 AM UTC. 2024-09-11T09:48:07.172 INFO:teuthology.orchestra.run.smithi017.stdout:Package ceph-base-2:19.1.1-298.gb53b6d3e.el9.x86_64 is already installed. 2024-09-11T09:48:07.217 INFO:teuthology.orchestra.run.smithi017.stdout:Dependencies resolved. 2024-09-11T09:48:07.218 INFO:teuthology.orchestra.run.smithi017.stdout:Nothing to do. 2024-09-11T09:48:07.218 INFO:teuthology.orchestra.run.smithi017.stdout:Complete! 2024-09-11T09:48:07.293 DEBUG:teuthology.orchestra.run.smithi017:> sudo yum -y install cephadm 2024-09-11T09:48:07.308 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 37/37 2024-09-11T09:48:07.308 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-11T09:48:07.308 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2024-09-11T09:48:07.308 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-11T09:48:07.308 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mds-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-11T09:48:07.308 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-11T09:48:07.309 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-modules-core-2:19.1.1-298.gb53b6d3e.el9.noarch 2024-09-11T09:48:07.309 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mon-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-11T09:48:07.309 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-osd-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-11T09:48:07.309 INFO:teuthology.orchestra.run.smithi107.stdout: lua-5.4.4-4.el9.x86_64 2024-09-11T09:48:07.309 INFO:teuthology.orchestra.run.smithi107.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-09-11T09:48:07.309 INFO:teuthology.orchestra.run.smithi107.stdout: luarocks-3.9.2-5.el9.noarch 2024-09-11T09:48:07.309 INFO:teuthology.orchestra.run.smithi107.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-11T09:48:07.309 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-11T09:48:07.309 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-11T09:48:07.309 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-11T09:48:07.309 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-11T09:48:07.309 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-11T09:48:07.309 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-11T09:48:07.309 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-11T09:48:07.310 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-11T09:48:07.310 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-11T09:48:07.310 INFO:teuthology.orchestra.run.smithi107.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-11T09:48:07.310 INFO:teuthology.orchestra.run.smithi107.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-11T09:48:07.310 INFO:teuthology.orchestra.run.smithi107.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-11T09:48:07.310 INFO:teuthology.orchestra.run.smithi107.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-11T09:48:07.310 INFO:teuthology.orchestra.run.smithi107.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-11T09:48:07.310 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-11T09:48:07.310 INFO:teuthology.orchestra.run.smithi107.stdout: python3-ply-3.11-14.el9.noarch 2024-09-11T09:48:07.310 INFO:teuthology.orchestra.run.smithi107.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-11T09:48:07.310 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-11T09:48:07.310 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-11T09:48:07.310 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-11T09:48:07.310 INFO:teuthology.orchestra.run.smithi107.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-11T09:48:07.311 INFO:teuthology.orchestra.run.smithi107.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-11T09:48:07.311 INFO:teuthology.orchestra.run.smithi107.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-11T09:48:07.311 INFO:teuthology.orchestra.run.smithi107.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-11T09:48:07.311 INFO:teuthology.orchestra.run.smithi107.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-11T09:48:07.311 INFO:teuthology.orchestra.run.smithi107.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-11T09:48:07.311 INFO:teuthology.orchestra.run.smithi107.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-11T09:48:07.311 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-11T09:48:07.311 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-09-11T09:48:07.696 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install ceph-base 2024-09-11T09:48:07.865 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 37/37 2024-09-11T09:48:07.866 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-11T09:48:07.866 INFO:teuthology.orchestra.run.smithi096.stdout:Installed: 2024-09-11T09:48:07.866 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-11T09:48:07.866 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mds-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-11T09:48:07.866 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mgr-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-11T09:48:07.866 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mgr-modules-core-2:19.1.1-298.gb53b6d3e.el9.noarch 2024-09-11T09:48:07.866 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mon-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-11T09:48:07.866 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-osd-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-11T09:48:07.866 INFO:teuthology.orchestra.run.smithi096.stdout: lua-5.4.4-4.el9.x86_64 2024-09-11T09:48:07.866 INFO:teuthology.orchestra.run.smithi096.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-09-11T09:48:07.866 INFO:teuthology.orchestra.run.smithi096.stdout: luarocks-3.9.2-5.el9.noarch 2024-09-11T09:48:07.866 INFO:teuthology.orchestra.run.smithi096.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-11T09:48:07.867 INFO:teuthology.orchestra.run.smithi096.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-11T09:48:07.867 INFO:teuthology.orchestra.run.smithi096.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-11T09:48:07.867 INFO:teuthology.orchestra.run.smithi096.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-11T09:48:07.867 INFO:teuthology.orchestra.run.smithi096.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-11T09:48:07.867 INFO:teuthology.orchestra.run.smithi096.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-11T09:48:07.867 INFO:teuthology.orchestra.run.smithi096.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-11T09:48:07.867 INFO:teuthology.orchestra.run.smithi096.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-11T09:48:07.867 INFO:teuthology.orchestra.run.smithi096.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-11T09:48:07.867 INFO:teuthology.orchestra.run.smithi096.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-11T09:48:07.867 INFO:teuthology.orchestra.run.smithi096.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-11T09:48:07.867 INFO:teuthology.orchestra.run.smithi096.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-11T09:48:07.867 INFO:teuthology.orchestra.run.smithi096.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-11T09:48:07.868 INFO:teuthology.orchestra.run.smithi096.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-11T09:48:07.868 INFO:teuthology.orchestra.run.smithi096.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-11T09:48:07.868 INFO:teuthology.orchestra.run.smithi096.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-11T09:48:07.868 INFO:teuthology.orchestra.run.smithi096.stdout: python3-ply-3.11-14.el9.noarch 2024-09-11T09:48:07.868 INFO:teuthology.orchestra.run.smithi096.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-11T09:48:07.868 INFO:teuthology.orchestra.run.smithi096.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-11T09:48:07.868 INFO:teuthology.orchestra.run.smithi096.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-11T09:48:07.868 INFO:teuthology.orchestra.run.smithi096.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-11T09:48:07.868 INFO:teuthology.orchestra.run.smithi096.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-11T09:48:07.868 INFO:teuthology.orchestra.run.smithi096.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-11T09:48:07.868 INFO:teuthology.orchestra.run.smithi096.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-11T09:48:07.868 INFO:teuthology.orchestra.run.smithi096.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-11T09:48:07.868 INFO:teuthology.orchestra.run.smithi096.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-11T09:48:07.868 INFO:teuthology.orchestra.run.smithi096.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-11T09:48:07.869 INFO:teuthology.orchestra.run.smithi096.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-11T09:48:07.869 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-11T09:48:07.869 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2024-09-11T09:48:07.876 INFO:teuthology.orchestra.run.smithi017.stdout:Last metadata expiration check: 0:00:54 ago on Wed 11 Sep 2024 09:47:13 AM UTC. 2024-09-11T09:48:08.013 INFO:teuthology.orchestra.run.smithi017.stdout:Dependencies resolved. 2024-09-11T09:48:08.014 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2024-09-11T09:48:08.014 INFO:teuthology.orchestra.run.smithi017.stdout: Package Arch Version Repository Size 2024-09-11T09:48:08.014 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2024-09-11T09:48:08.014 INFO:teuthology.orchestra.run.smithi017.stdout:Installing: 2024-09-11T09:48:08.014 INFO:teuthology.orchestra.run.smithi017.stdout: cephadm noarch 2:19.1.1-298.gb53b6d3e.el9 ceph-noarch 765 k 2024-09-11T09:48:08.015 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-11T09:48:08.015 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction Summary 2024-09-11T09:48:08.015 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2024-09-11T09:48:08.015 INFO:teuthology.orchestra.run.smithi017.stdout:Install 1 Package 2024-09-11T09:48:08.015 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-11T09:48:08.015 INFO:teuthology.orchestra.run.smithi017.stdout:Total download size: 765 k 2024-09-11T09:48:08.015 INFO:teuthology.orchestra.run.smithi017.stdout:Installed size: 771 k 2024-09-11T09:48:08.015 INFO:teuthology.orchestra.run.smithi017.stdout:Downloading Packages: 2024-09-11T09:48:08.261 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:00:55 ago on Wed 11 Sep 2024 09:47:13 AM UTC. 2024-09-11T09:48:08.298 INFO:teuthology.orchestra.run.smithi017.stdout:cephadm-19.1.1-298.gb53b6d3e.el9.noarch.rpm 2.6 MB/s | 765 kB 00:00 2024-09-11T09:48:08.299 INFO:teuthology.orchestra.run.smithi017.stdout:-------------------------------------------------------------------------------- 2024-09-11T09:48:08.299 INFO:teuthology.orchestra.run.smithi017.stdout:Total 2.6 MB/s | 765 kB 00:00 2024-09-11T09:48:08.300 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction check 2024-09-11T09:48:08.305 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction check succeeded. 2024-09-11T09:48:08.306 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction test 2024-09-11T09:48:08.308 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum -y install ceph-base 2024-09-11T09:48:08.311 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction test succeeded. 2024-09-11T09:48:08.312 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction 2024-09-11T09:48:08.350 INFO:teuthology.orchestra.run.smithi107.stdout:Package ceph-base-2:19.1.1-298.gb53b6d3e.el9.x86_64 is already installed. 2024-09-11T09:48:08.391 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-09-11T09:48:08.392 INFO:teuthology.orchestra.run.smithi017.stdout: Preparing : 1/1 2024-09-11T09:48:08.392 INFO:teuthology.orchestra.run.smithi107.stdout:Nothing to do. 2024-09-11T09:48:08.392 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-09-11T09:48:08.466 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install cephadm 2024-09-11T09:48:08.851 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: cephadm-2:19.1.1-298.gb53b6d3e.el9.noarch 1/1 2024-09-11T09:48:08.874 INFO:teuthology.orchestra.run.smithi096.stdout:Last metadata expiration check: 0:00:56 ago on Wed 11 Sep 2024 09:47:12 AM UTC. 2024-09-11T09:48:08.940 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : cephadm-2:19.1.1-298.gb53b6d3e.el9.noarch 1/1 2024-09-11T09:48:08.965 INFO:teuthology.orchestra.run.smithi096.stdout:Package ceph-base-2:19.1.1-298.gb53b6d3e.el9.x86_64 is already installed. 2024-09-11T09:48:09.007 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2024-09-11T09:48:09.008 INFO:teuthology.orchestra.run.smithi096.stdout:Nothing to do. 2024-09-11T09:48:09.008 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2024-09-11T09:48:09.023 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:00:56 ago on Wed 11 Sep 2024 09:47:13 AM UTC. 2024-09-11T09:48:09.073 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum -y install cephadm 2024-09-11T09:48:09.154 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-09-11T09:48:09.155 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-09-11T09:48:09.155 INFO:teuthology.orchestra.run.smithi107.stdout: Package Arch Version Repository Size 2024-09-11T09:48:09.156 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-09-11T09:48:09.156 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2024-09-11T09:48:09.156 INFO:teuthology.orchestra.run.smithi107.stdout: cephadm noarch 2:19.1.1-298.gb53b6d3e.el9 ceph-noarch 765 k 2024-09-11T09:48:09.156 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-11T09:48:09.156 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2024-09-11T09:48:09.156 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-09-11T09:48:09.156 INFO:teuthology.orchestra.run.smithi107.stdout:Install 1 Package 2024-09-11T09:48:09.156 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-11T09:48:09.156 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 765 k 2024-09-11T09:48:09.156 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 771 k 2024-09-11T09:48:09.156 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2024-09-11T09:48:09.319 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: cephadm-2:19.1.1-298.gb53b6d3e.el9.noarch 1/1 2024-09-11T09:48:09.439 INFO:teuthology.orchestra.run.smithi107.stdout:cephadm-19.1.1-298.gb53b6d3e.el9.noarch.rpm 2.6 MB/s | 765 kB 00:00 2024-09-11T09:48:09.440 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2024-09-11T09:48:09.440 INFO:teuthology.orchestra.run.smithi107.stdout:Total 2.6 MB/s | 765 kB 00:00 2024-09-11T09:48:09.440 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2024-09-11T09:48:09.445 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2024-09-11T09:48:09.445 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2024-09-11T09:48:09.451 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2024-09-11T09:48:09.451 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2024-09-11T09:48:09.527 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2024-09-11T09:48:09.578 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : cephadm-2:19.1.1-298.gb53b6d3e.el9.noarch 1/1 2024-09-11T09:48:09.578 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-11T09:48:09.578 INFO:teuthology.orchestra.run.smithi017.stdout:Installed: 2024-09-11T09:48:09.578 INFO:teuthology.orchestra.run.smithi017.stdout: cephadm-2:19.1.1-298.gb53b6d3e.el9.noarch 2024-09-11T09:48:09.578 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-11T09:48:09.578 INFO:teuthology.orchestra.run.smithi017.stdout:Complete! 2024-09-11T09:48:09.632 INFO:teuthology.orchestra.run.smithi096.stdout:Last metadata expiration check: 0:00:57 ago on Wed 11 Sep 2024 09:47:12 AM UTC. 2024-09-11T09:48:09.763 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2024-09-11T09:48:09.763 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-09-11T09:48:09.763 INFO:teuthology.orchestra.run.smithi096.stdout: Package Arch Version Repository Size 2024-09-11T09:48:09.763 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-09-11T09:48:09.764 INFO:teuthology.orchestra.run.smithi096.stdout:Installing: 2024-09-11T09:48:09.764 INFO:teuthology.orchestra.run.smithi096.stdout: cephadm noarch 2:19.1.1-298.gb53b6d3e.el9 ceph-noarch 765 k 2024-09-11T09:48:09.764 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-11T09:48:09.764 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction Summary 2024-09-11T09:48:09.764 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-09-11T09:48:09.764 INFO:teuthology.orchestra.run.smithi096.stdout:Install 1 Package 2024-09-11T09:48:09.764 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-11T09:48:09.764 INFO:teuthology.orchestra.run.smithi096.stdout:Total download size: 765 k 2024-09-11T09:48:09.764 INFO:teuthology.orchestra.run.smithi096.stdout:Installed size: 771 k 2024-09-11T09:48:09.764 INFO:teuthology.orchestra.run.smithi096.stdout:Downloading Packages: 2024-09-11T09:48:09.824 DEBUG:teuthology.orchestra.run.smithi017:> sudo yum -y install ceph-immutable-object-cache 2024-09-11T09:48:09.956 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: cephadm-2:19.1.1-298.gb53b6d3e.el9.noarch 1/1 2024-09-11T09:48:10.047 INFO:teuthology.orchestra.run.smithi096.stdout:cephadm-19.1.1-298.gb53b6d3e.el9.noarch.rpm 2.6 MB/s | 765 kB 00:00 2024-09-11T09:48:10.047 INFO:teuthology.orchestra.run.smithi096.stdout:-------------------------------------------------------------------------------- 2024-09-11T09:48:10.048 INFO:teuthology.orchestra.run.smithi096.stdout:Total 2.6 MB/s | 765 kB 00:00 2024-09-11T09:48:10.048 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction check 2024-09-11T09:48:10.050 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : cephadm-2:19.1.1-298.gb53b6d3e.el9.noarch 1/1 2024-09-11T09:48:10.052 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction check succeeded. 2024-09-11T09:48:10.052 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction test 2024-09-11T09:48:10.058 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction test succeeded. 2024-09-11T09:48:10.058 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction 2024-09-11T09:48:10.130 INFO:teuthology.orchestra.run.smithi096.stdout: Preparing : 1/1 2024-09-11T09:48:10.410 INFO:teuthology.orchestra.run.smithi017.stdout:Last metadata expiration check: 0:00:57 ago on Wed 11 Sep 2024 09:47:13 AM UTC. 2024-09-11T09:48:10.417 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: cephadm-2:19.1.1-298.gb53b6d3e.el9.noarch 1/1 2024-09-11T09:48:10.537 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: cephadm-2:19.1.1-298.gb53b6d3e.el9.noarch 1/1 2024-09-11T09:48:10.548 INFO:teuthology.orchestra.run.smithi017.stdout:Dependencies resolved. 2024-09-11T09:48:10.548 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2024-09-11T09:48:10.549 INFO:teuthology.orchestra.run.smithi017.stdout: Package Arch Version Repo Size 2024-09-11T09:48:10.549 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2024-09-11T09:48:10.549 INFO:teuthology.orchestra.run.smithi017.stdout:Installing: 2024-09-11T09:48:10.549 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-immutable-object-cache x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 145 k 2024-09-11T09:48:10.549 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-11T09:48:10.549 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction Summary 2024-09-11T09:48:10.549 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2024-09-11T09:48:10.549 INFO:teuthology.orchestra.run.smithi017.stdout:Install 1 Package 2024-09-11T09:48:10.549 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-11T09:48:10.550 INFO:teuthology.orchestra.run.smithi017.stdout:Total download size: 145 k 2024-09-11T09:48:10.550 INFO:teuthology.orchestra.run.smithi017.stdout:Installed size: 435 k 2024-09-11T09:48:10.550 INFO:teuthology.orchestra.run.smithi017.stdout:Downloading Packages: 2024-09-11T09:48:10.625 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : cephadm-2:19.1.1-298.gb53b6d3e.el9.noarch 1/1 2024-09-11T09:48:10.659 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : cephadm-2:19.1.1-298.gb53b6d3e.el9.noarch 1/1 2024-09-11T09:48:10.659 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-11T09:48:10.659 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2024-09-11T09:48:10.659 INFO:teuthology.orchestra.run.smithi107.stdout: cephadm-2:19.1.1-298.gb53b6d3e.el9.noarch 2024-09-11T09:48:10.659 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-11T09:48:10.659 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-09-11T09:48:10.740 INFO:teuthology.orchestra.run.smithi017.stdout:ceph-immutable-object-cache-19.1.1-298.gb53b6d3 764 kB/s | 145 kB 00:00 2024-09-11T09:48:10.740 INFO:teuthology.orchestra.run.smithi017.stdout:-------------------------------------------------------------------------------- 2024-09-11T09:48:10.741 INFO:teuthology.orchestra.run.smithi017.stdout:Total 758 kB/s | 145 kB 00:00 2024-09-11T09:48:10.741 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction check 2024-09-11T09:48:10.749 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction check succeeded. 2024-09-11T09:48:10.749 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction test 2024-09-11T09:48:10.809 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction test succeeded. 2024-09-11T09:48:10.809 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction 2024-09-11T09:48:10.902 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install ceph-immutable-object-cache 2024-09-11T09:48:10.962 INFO:teuthology.orchestra.run.smithi017.stdout: Preparing : 1/1 2024-09-11T09:48:11.018 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: cephadm-2:19.1.1-298.gb53b6d3e.el9.noarch 1/1 2024-09-11T09:48:11.033 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : ceph-immutable-object-cache-2:19.1.1-298.gb53b6d3e.e 1/1 2024-09-11T09:48:11.060 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.1.1-298.gb53b6d3e.e 1/1 2024-09-11T09:48:11.060 INFO:teuthology.orchestra.run.smithi017.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-11T09:48:11.060 INFO:teuthology.orchestra.run.smithi017.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-11T09:48:11.060 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-11T09:48:11.279 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : cephadm-2:19.1.1-298.gb53b6d3e.el9.noarch 1/1 2024-09-11T09:48:11.279 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-11T09:48:11.279 INFO:teuthology.orchestra.run.smithi096.stdout:Installed: 2024-09-11T09:48:11.279 INFO:teuthology.orchestra.run.smithi096.stdout: cephadm-2:19.1.1-298.gb53b6d3e.el9.noarch 2024-09-11T09:48:11.279 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-11T09:48:11.279 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2024-09-11T09:48:11.468 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:00:58 ago on Wed 11 Sep 2024 09:47:13 AM UTC. 2024-09-11T09:48:11.501 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum -y install ceph-immutable-object-cache 2024-09-11T09:48:11.602 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-09-11T09:48:11.603 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-09-11T09:48:11.603 INFO:teuthology.orchestra.run.smithi107.stdout: Package Arch Version Repo Size 2024-09-11T09:48:11.603 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-09-11T09:48:11.604 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2024-09-11T09:48:11.604 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-immutable-object-cache x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 145 k 2024-09-11T09:48:11.604 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-11T09:48:11.604 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2024-09-11T09:48:11.604 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-09-11T09:48:11.604 INFO:teuthology.orchestra.run.smithi107.stdout:Install 1 Package 2024-09-11T09:48:11.604 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-11T09:48:11.604 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 145 k 2024-09-11T09:48:11.604 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 435 k 2024-09-11T09:48:11.604 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2024-09-11T09:48:11.813 INFO:teuthology.orchestra.run.smithi107.stdout:ceph-immutable-object-cache-19.1.1-298.gb53b6d3 697 kB/s | 145 kB 00:00 2024-09-11T09:48:11.813 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2024-09-11T09:48:11.813 INFO:teuthology.orchestra.run.smithi107.stdout:Total 692 kB/s | 145 kB 00:00 2024-09-11T09:48:11.814 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2024-09-11T09:48:11.821 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2024-09-11T09:48:11.821 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2024-09-11T09:48:11.878 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2024-09-11T09:48:11.879 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2024-09-11T09:48:11.993 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : ceph-immutable-object-cache-2:19.1.1-298.gb53b6d3e.e 1/1 2024-09-11T09:48:12.008 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-11T09:48:12.008 INFO:teuthology.orchestra.run.smithi017.stdout:Installed: 2024-09-11T09:48:12.008 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-immutable-object-cache-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-11T09:48:12.008 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-11T09:48:12.008 INFO:teuthology.orchestra.run.smithi017.stdout:Complete! 2024-09-11T09:48:12.041 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2024-09-11T09:48:12.070 INFO:teuthology.orchestra.run.smithi096.stdout:Last metadata expiration check: 0:01:00 ago on Wed 11 Sep 2024 09:47:12 AM UTC. 2024-09-11T09:48:12.122 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-immutable-object-cache-2:19.1.1-298.gb53b6d3e.e 1/1 2024-09-11T09:48:12.147 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.1.1-298.gb53b6d3e.e 1/1 2024-09-11T09:48:12.147 INFO:teuthology.orchestra.run.smithi107.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-11T09:48:12.147 INFO:teuthology.orchestra.run.smithi107.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-11T09:48:12.147 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-11T09:48:12.205 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2024-09-11T09:48:12.205 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-09-11T09:48:12.205 INFO:teuthology.orchestra.run.smithi096.stdout: Package Arch Version Repo Size 2024-09-11T09:48:12.206 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-09-11T09:48:12.206 INFO:teuthology.orchestra.run.smithi096.stdout:Installing: 2024-09-11T09:48:12.206 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-immutable-object-cache x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 145 k 2024-09-11T09:48:12.206 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-11T09:48:12.206 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction Summary 2024-09-11T09:48:12.206 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-09-11T09:48:12.206 INFO:teuthology.orchestra.run.smithi096.stdout:Install 1 Package 2024-09-11T09:48:12.206 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-11T09:48:12.207 INFO:teuthology.orchestra.run.smithi096.stdout:Total download size: 145 k 2024-09-11T09:48:12.207 INFO:teuthology.orchestra.run.smithi096.stdout:Installed size: 435 k 2024-09-11T09:48:12.207 INFO:teuthology.orchestra.run.smithi096.stdout:Downloading Packages: 2024-09-11T09:48:12.212 DEBUG:teuthology.orchestra.run.smithi017:> sudo yum -y install ceph-mgr 2024-09-11T09:48:12.429 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-immutable-object-cache-19.1.1-298.gb53b6d3 654 kB/s | 145 kB 00:00 2024-09-11T09:48:12.429 INFO:teuthology.orchestra.run.smithi096.stdout:-------------------------------------------------------------------------------- 2024-09-11T09:48:12.430 INFO:teuthology.orchestra.run.smithi096.stdout:Total 649 kB/s | 145 kB 00:00 2024-09-11T09:48:12.430 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction check 2024-09-11T09:48:12.437 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction check succeeded. 2024-09-11T09:48:12.437 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction test 2024-09-11T09:48:12.494 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction test succeeded. 2024-09-11T09:48:12.495 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction 2024-09-11T09:48:12.676 INFO:teuthology.orchestra.run.smithi096.stdout: Preparing : 1/1 2024-09-11T09:48:12.755 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-immutable-object-cache-2:19.1.1-298.gb53b6d3e.e 1/1 2024-09-11T09:48:12.779 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.1.1-298.gb53b6d3e.e 1/1 2024-09-11T09:48:12.779 INFO:teuthology.orchestra.run.smithi096.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-11T09:48:12.779 INFO:teuthology.orchestra.run.smithi096.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-11T09:48:12.779 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-11T09:48:12.805 INFO:teuthology.orchestra.run.smithi017.stdout:Last metadata expiration check: 0:00:59 ago on Wed 11 Sep 2024 09:47:13 AM UTC. 2024-09-11T09:48:12.900 INFO:teuthology.orchestra.run.smithi017.stdout:Package ceph-mgr-2:19.1.1-298.gb53b6d3e.el9.x86_64 is already installed. 2024-09-11T09:48:12.945 INFO:teuthology.orchestra.run.smithi017.stdout:Dependencies resolved. 2024-09-11T09:48:12.946 INFO:teuthology.orchestra.run.smithi017.stdout:Nothing to do. 2024-09-11T09:48:12.946 INFO:teuthology.orchestra.run.smithi017.stdout:Complete! 2024-09-11T09:48:13.017 DEBUG:teuthology.orchestra.run.smithi017:> sudo yum -y install ceph-mgr-dashboard 2024-09-11T09:48:13.097 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-immutable-object-cache-2:19.1.1-298.gb53b6d3e.e 1/1 2024-09-11T09:48:13.097 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-11T09:48:13.097 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2024-09-11T09:48:13.097 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-immutable-object-cache-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-11T09:48:13.097 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-11T09:48:13.098 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-09-11T09:48:13.349 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install ceph-mgr 2024-09-11T09:48:13.600 INFO:teuthology.orchestra.run.smithi017.stdout:Last metadata expiration check: 0:01:00 ago on Wed 11 Sep 2024 09:47:13 AM UTC. 2024-09-11T09:48:13.694 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-immutable-object-cache-2:19.1.1-298.gb53b6d3e.e 1/1 2024-09-11T09:48:13.695 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-11T09:48:13.695 INFO:teuthology.orchestra.run.smithi096.stdout:Installed: 2024-09-11T09:48:13.695 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-immutable-object-cache-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-11T09:48:13.695 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-11T09:48:13.695 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2024-09-11T09:48:13.739 INFO:teuthology.orchestra.run.smithi017.stdout:Dependencies resolved. 2024-09-11T09:48:13.740 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2024-09-11T09:48:13.740 INFO:teuthology.orchestra.run.smithi017.stdout: Package Arch Version Repository Size 2024-09-11T09:48:13.740 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2024-09-11T09:48:13.740 INFO:teuthology.orchestra.run.smithi017.stdout:Installing: 2024-09-11T09:48:13.740 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-mgr-dashboard noarch 2:19.1.1-298.gb53b6d3e.el9 ceph-noarch 3.6 M 2024-09-11T09:48:13.740 INFO:teuthology.orchestra.run.smithi017.stdout:Installing dependencies: 2024-09-11T09:48:13.741 INFO:teuthology.orchestra.run.smithi017.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-09-11T09:48:13.741 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-grafana-dashboards noarch 2:19.1.1-298.gb53b6d3e.el9 ceph-noarch 32 k 2024-09-11T09:48:13.741 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-prometheus-alerts noarch 2:19.1.1-298.gb53b6d3e.el9 ceph-noarch 16 k 2024-09-11T09:48:13.741 INFO:teuthology.orchestra.run.smithi017.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-09-11T09:48:13.741 INFO:teuthology.orchestra.run.smithi017.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-09-11T09:48:13.741 INFO:teuthology.orchestra.run.smithi017.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-09-11T09:48:13.741 INFO:teuthology.orchestra.run.smithi017.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-09-11T09:48:13.741 INFO:teuthology.orchestra.run.smithi017.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-09-11T09:48:13.741 INFO:teuthology.orchestra.run.smithi017.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-09-11T09:48:13.741 INFO:teuthology.orchestra.run.smithi017.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-11T09:48:13.742 INFO:teuthology.orchestra.run.smithi017.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-11T09:48:13.742 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-11T09:48:13.742 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction Summary 2024-09-11T09:48:13.742 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2024-09-11T09:48:13.742 INFO:teuthology.orchestra.run.smithi017.stdout:Install 12 Packages 2024-09-11T09:48:13.742 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-11T09:48:13.742 INFO:teuthology.orchestra.run.smithi017.stdout:Total download size: 8.7 M 2024-09-11T09:48:13.743 INFO:teuthology.orchestra.run.smithi017.stdout:Installed size: 104 M 2024-09-11T09:48:13.743 INFO:teuthology.orchestra.run.smithi017.stdout:Downloading Packages: 2024-09-11T09:48:13.921 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:01:00 ago on Wed 11 Sep 2024 09:47:13 AM UTC. 2024-09-11T09:48:13.934 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum -y install ceph-mgr 2024-09-11T09:48:14.013 INFO:teuthology.orchestra.run.smithi107.stdout:Package ceph-mgr-2:19.1.1-298.gb53b6d3e.el9.x86_64 is already installed. 2024-09-11T09:48:14.057 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-09-11T09:48:14.058 INFO:teuthology.orchestra.run.smithi107.stdout:Nothing to do. 2024-09-11T09:48:14.058 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-09-11T09:48:14.129 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install ceph-mgr-dashboard 2024-09-11T09:48:14.269 INFO:teuthology.orchestra.run.smithi017.stdout:(1/12): ceph-grafana-dashboards-19.1.1-298.gb53 214 kB/s | 32 kB 00:00 2024-09-11T09:48:14.293 INFO:teuthology.orchestra.run.smithi017.stdout:(2/12): ceph-prometheus-alerts-19.1.1-298.gb53b 95 kB/s | 16 kB 00:00 2024-09-11T09:48:14.452 INFO:teuthology.orchestra.run.smithi017.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.7 MB/s | 269 kB 00:00 2024-09-11T09:48:14.505 INFO:teuthology.orchestra.run.smithi096.stdout:Last metadata expiration check: 0:01:02 ago on Wed 11 Sep 2024 09:47:12 AM UTC. 2024-09-11T09:48:14.527 INFO:teuthology.orchestra.run.smithi017.stdout:(4/12): ceph-mgr-dashboard-19.1.1-298.gb53b6d3e 8.7 MB/s | 3.6 MB 00:00 2024-09-11T09:48:14.570 INFO:teuthology.orchestra.run.smithi017.stdout:(5/12): protobuf-3.14.0-13.el9.x86_64.rpm 3.4 MB/s | 1.0 MB 00:00 2024-09-11T09:48:14.600 INFO:teuthology.orchestra.run.smithi096.stdout:Package ceph-mgr-2:19.1.1-298.gb53b6d3e.el9.x86_64 is already installed. 2024-09-11T09:48:14.611 INFO:teuthology.orchestra.run.smithi017.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 581 kB/s | 19 kB 00:00 2024-09-11T09:48:14.644 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2024-09-11T09:48:14.645 INFO:teuthology.orchestra.run.smithi096.stdout:Nothing to do. 2024-09-11T09:48:14.645 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2024-09-11T09:48:14.645 INFO:teuthology.orchestra.run.smithi017.stdout:(7/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 4.6 MB/s | 548 kB 00:00 2024-09-11T09:48:14.671 INFO:teuthology.orchestra.run.smithi017.stdout:(8/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 5.6 MB/s | 144 kB 00:00 2024-09-11T09:48:14.684 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:01:01 ago on Wed 11 Sep 2024 09:47:13 AM UTC. 2024-09-11T09:48:14.695 INFO:teuthology.orchestra.run.smithi017.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 1.2 MB/s | 31 kB 00:00 2024-09-11T09:48:14.717 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum -y install ceph-mgr-dashboard 2024-09-11T09:48:14.721 INFO:teuthology.orchestra.run.smithi017.stdout:(10/12): python3-routes-2.5.1-5.el9.noarch.rpm 7.3 MB/s | 188 kB 00:00 2024-09-11T09:48:14.780 INFO:teuthology.orchestra.run.smithi017.stdout:(11/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 12 MB/s | 2.0 MB 00:00 2024-09-11T09:48:14.816 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-09-11T09:48:14.817 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-09-11T09:48:14.817 INFO:teuthology.orchestra.run.smithi107.stdout: Package Arch Version Repository Size 2024-09-11T09:48:14.818 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-09-11T09:48:14.818 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2024-09-11T09:48:14.818 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-dashboard noarch 2:19.1.1-298.gb53b6d3e.el9 ceph-noarch 3.6 M 2024-09-11T09:48:14.818 INFO:teuthology.orchestra.run.smithi107.stdout:Installing dependencies: 2024-09-11T09:48:14.818 INFO:teuthology.orchestra.run.smithi107.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-09-11T09:48:14.818 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-grafana-dashboards noarch 2:19.1.1-298.gb53b6d3e.el9 ceph-noarch 32 k 2024-09-11T09:48:14.818 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-prometheus-alerts noarch 2:19.1.1-298.gb53b6d3e.el9 ceph-noarch 16 k 2024-09-11T09:48:14.818 INFO:teuthology.orchestra.run.smithi107.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-09-11T09:48:14.818 INFO:teuthology.orchestra.run.smithi107.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-09-11T09:48:14.818 INFO:teuthology.orchestra.run.smithi107.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-09-11T09:48:14.818 INFO:teuthology.orchestra.run.smithi107.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-09-11T09:48:14.819 INFO:teuthology.orchestra.run.smithi107.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-09-11T09:48:14.819 INFO:teuthology.orchestra.run.smithi107.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-09-11T09:48:14.819 INFO:teuthology.orchestra.run.smithi107.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-11T09:48:14.819 INFO:teuthology.orchestra.run.smithi107.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-11T09:48:14.819 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-11T09:48:14.819 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2024-09-11T09:48:14.819 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-09-11T09:48:14.819 INFO:teuthology.orchestra.run.smithi107.stdout:Install 12 Packages 2024-09-11T09:48:14.819 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-11T09:48:14.820 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 8.7 M 2024-09-11T09:48:14.820 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 104 M 2024-09-11T09:48:14.820 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2024-09-11T09:48:15.192 INFO:teuthology.orchestra.run.smithi107.stdout:(1/12): ceph-prometheus-alerts-19.1.1-298.gb53b 110 kB/s | 16 kB 00:00 2024-09-11T09:48:15.217 INFO:teuthology.orchestra.run.smithi107.stdout:(2/12): ceph-grafana-dashboards-19.1.1-298.gb53 182 kB/s | 32 kB 00:00 2024-09-11T09:48:15.274 INFO:teuthology.orchestra.run.smithi096.stdout:Last metadata expiration check: 0:01:03 ago on Wed 11 Sep 2024 09:47:12 AM UTC. 2024-09-11T09:48:15.392 INFO:teuthology.orchestra.run.smithi107.stdout:(3/12): ceph-mgr-dashboard-19.1.1-298.gb53b6d3e 10 MB/s | 3.6 MB 00:00 2024-09-11T09:48:15.406 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2024-09-11T09:48:15.407 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-09-11T09:48:15.407 INFO:teuthology.orchestra.run.smithi096.stdout: Package Arch Version Repository Size 2024-09-11T09:48:15.407 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-09-11T09:48:15.407 INFO:teuthology.orchestra.run.smithi096.stdout:Installing: 2024-09-11T09:48:15.407 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mgr-dashboard noarch 2:19.1.1-298.gb53b6d3e.el9 ceph-noarch 3.6 M 2024-09-11T09:48:15.408 INFO:teuthology.orchestra.run.smithi096.stdout:Installing dependencies: 2024-09-11T09:48:15.408 INFO:teuthology.orchestra.run.smithi096.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-09-11T09:48:15.408 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-grafana-dashboards noarch 2:19.1.1-298.gb53b6d3e.el9 ceph-noarch 32 k 2024-09-11T09:48:15.408 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-prometheus-alerts noarch 2:19.1.1-298.gb53b6d3e.el9 ceph-noarch 16 k 2024-09-11T09:48:15.408 INFO:teuthology.orchestra.run.smithi096.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-09-11T09:48:15.408 INFO:teuthology.orchestra.run.smithi096.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-09-11T09:48:15.408 INFO:teuthology.orchestra.run.smithi096.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-09-11T09:48:15.408 INFO:teuthology.orchestra.run.smithi096.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-09-11T09:48:15.408 INFO:teuthology.orchestra.run.smithi096.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-09-11T09:48:15.408 INFO:teuthology.orchestra.run.smithi096.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-09-11T09:48:15.408 INFO:teuthology.orchestra.run.smithi096.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-11T09:48:15.408 INFO:teuthology.orchestra.run.smithi096.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-11T09:48:15.408 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-11T09:48:15.408 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction Summary 2024-09-11T09:48:15.409 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-09-11T09:48:15.409 INFO:teuthology.orchestra.run.smithi096.stdout:Install 12 Packages 2024-09-11T09:48:15.409 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-11T09:48:15.409 INFO:teuthology.orchestra.run.smithi096.stdout:Total download size: 8.7 M 2024-09-11T09:48:15.409 INFO:teuthology.orchestra.run.smithi096.stdout:Installed size: 104 M 2024-09-11T09:48:15.409 INFO:teuthology.orchestra.run.smithi096.stdout:Downloading Packages: 2024-09-11T09:48:15.471 INFO:teuthology.orchestra.run.smithi017.stdout:(12/12): protobuf-compiler-3.14.0-13.el9.x86_64 847 kB/s | 863 kB 00:01 2024-09-11T09:48:15.477 INFO:teuthology.orchestra.run.smithi017.stdout:-------------------------------------------------------------------------------- 2024-09-11T09:48:15.477 INFO:teuthology.orchestra.run.smithi017.stdout:Total 5.0 MB/s | 8.7 MB 00:01 2024-09-11T09:48:15.531 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction check 2024-09-11T09:48:15.551 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction check succeeded. 2024-09-11T09:48:15.551 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction test 2024-09-11T09:48:15.576 INFO:teuthology.orchestra.run.smithi107.stdout:(4/12): protobuf-compiler-3.14.0-13.el9.x86_64. 4.6 MB/s | 863 kB 00:00 2024-09-11T09:48:15.626 INFO:teuthology.orchestra.run.smithi107.stdout:(5/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 11 MB/s | 548 kB 00:00 2024-09-11T09:48:15.651 INFO:teuthology.orchestra.run.smithi107.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 776 kB/s | 19 kB 00:00 2024-09-11T09:48:15.719 INFO:teuthology.orchestra.run.smithi107.stdout:(7/12): python3-grpcio-1.46.7-10.el9.x86_64.rpm 30 MB/s | 2.0 MB 00:00 2024-09-11T09:48:15.742 INFO:teuthology.orchestra.run.smithi096.stdout:(1/12): ceph-grafana-dashboards-19.1.1-298.gb53 207 kB/s | 32 kB 00:00 2024-09-11T09:48:15.744 INFO:teuthology.orchestra.run.smithi107.stdout:(8/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 5.6 MB/s | 144 kB 00:00 2024-09-11T09:48:15.766 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction test succeeded. 2024-09-11T09:48:15.766 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction 2024-09-11T09:48:15.768 INFO:teuthology.orchestra.run.smithi096.stdout:(2/12): ceph-prometheus-alerts-19.1.1-298.gb53b 92 kB/s | 16 kB 00:00 2024-09-11T09:48:15.769 INFO:teuthology.orchestra.run.smithi107.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 1.2 MB/s | 31 kB 00:00 2024-09-11T09:48:15.795 INFO:teuthology.orchestra.run.smithi107.stdout:(10/12): python3-routes-2.5.1-5.el9.noarch.rpm 7.3 MB/s | 188 kB 00:00 2024-09-11T09:48:15.853 INFO:teuthology.orchestra.run.smithi107.stdout:(11/12): python3-protobuf-3.14.0-13.el9.noarch. 423 kB/s | 269 kB 00:00 2024-09-11T09:48:15.985 INFO:teuthology.orchestra.run.smithi096.stdout:(3/12): ceph-mgr-dashboard-19.1.1-298.gb53b6d3e 9.0 MB/s | 3.6 MB 00:00 2024-09-11T09:48:16.010 INFO:teuthology.orchestra.run.smithi096.stdout:(4/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.1 MB/s | 269 kB 00:00 2024-09-11T09:48:16.037 INFO:teuthology.orchestra.run.smithi107.stdout:(12/12): protobuf-3.14.0-13.el9.x86_64.rpm 1.2 MB/s | 1.0 MB 00:00 2024-09-11T09:48:16.040 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2024-09-11T09:48:16.040 INFO:teuthology.orchestra.run.smithi107.stdout:Total 7.1 MB/s | 8.7 MB 00:01 2024-09-11T09:48:16.060 INFO:teuthology.orchestra.run.smithi096.stdout:(5/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 11 MB/s | 548 kB 00:00 2024-09-11T09:48:16.085 INFO:teuthology.orchestra.run.smithi096.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 784 kB/s | 19 kB 00:00 2024-09-11T09:48:16.087 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2024-09-11T09:48:16.105 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2024-09-11T09:48:16.105 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2024-09-11T09:48:16.119 INFO:teuthology.orchestra.run.smithi096.stdout:(7/12): protobuf-3.14.0-13.el9.x86_64.rpm 2.7 MB/s | 1.0 MB 00:00 2024-09-11T09:48:16.186 INFO:teuthology.orchestra.run.smithi096.stdout:(8/12): python3-grpcio-1.46.7-10.el9.x86_64.rpm 20 MB/s | 2.0 MB 00:00 2024-09-11T09:48:16.211 INFO:teuthology.orchestra.run.smithi096.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 1.2 MB/s | 31 kB 00:00 2024-09-11T09:48:16.236 INFO:teuthology.orchestra.run.smithi096.stdout:(10/12): python3-grpcio-tools-1.46.7-10.el9.x86 1.2 MB/s | 144 kB 00:00 2024-09-11T09:48:16.270 INFO:teuthology.orchestra.run.smithi096.stdout:(11/12): protobuf-compiler-3.14.0-13.el9.x86_64 3.0 MB/s | 863 kB 00:00 2024-09-11T09:48:16.296 INFO:teuthology.orchestra.run.smithi096.stdout:(12/12): python3-routes-2.5.1-5.el9.noarch.rpm 2.2 MB/s | 188 kB 00:00 2024-09-11T09:48:16.299 INFO:teuthology.orchestra.run.smithi096.stdout:-------------------------------------------------------------------------------- 2024-09-11T09:48:16.299 INFO:teuthology.orchestra.run.smithi096.stdout:Total 9.7 MB/s | 8.7 MB 00:00 2024-09-11T09:48:16.315 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2024-09-11T09:48:16.316 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2024-09-11T09:48:16.317 INFO:teuthology.orchestra.run.smithi017.stdout: Preparing : 1/1 2024-09-11T09:48:16.348 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction check 2024-09-11T09:48:16.364 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction check succeeded. 2024-09-11T09:48:16.364 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction test 2024-09-11T09:48:16.398 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-09-11T09:48:16.441 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-09-11T09:48:16.489 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-09-11T09:48:16.526 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-09-11T09:48:16.579 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction test succeeded. 2024-09-11T09:48:16.580 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction 2024-09-11T09:48:16.631 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-09-11T09:48:16.767 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-09-11T09:48:16.844 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2024-09-11T09:48:16.858 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-09-11T09:48:16.912 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-09-11T09:48:16.932 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-09-11T09:48:16.947 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-09-11T09:48:16.960 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-09-11T09:48:16.991 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : ceph-prometheus-alerts-2:19.1.1-298.gb53b6d3e.el9. 10/12 2024-09-11T09:48:16.994 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-09-11T09:48:17.024 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-09-11T09:48:17.112 INFO:teuthology.orchestra.run.smithi096.stdout: Preparing : 1/1 2024-09-11T09:48:17.123 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-09-11T09:48:17.196 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-09-11T09:48:17.239 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-09-11T09:48:17.251 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-09-11T09:48:17.285 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-09-11T09:48:17.316 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-09-11T09:48:17.345 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-09-11T09:48:17.396 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-09-11T09:48:17.407 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-09-11T09:48:17.425 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-09-11T09:48:17.457 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-prometheus-alerts-2:19.1.1-298.gb53b6d3e.el9. 10/12 2024-09-11T09:48:17.533 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-09-11T09:48:17.619 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-09-11T09:48:17.680 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-09-11T09:48:17.709 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-09-11T09:48:17.732 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-prometheus-alerts-2:19.1.1-298.gb53b6d3e.el9. 10/12 2024-09-11T09:48:18.108 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : ceph-grafana-dashboards-2:19.1.1-298.gb53b6d3e.el9 11/12 2024-09-11T09:48:18.142 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : ceph-mgr-dashboard-2:19.1.1-298.gb53b6d3e.el9.noar 12/12 2024-09-11T09:48:18.510 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-grafana-dashboards-2:19.1.1-298.gb53b6d3e.el9 11/12 2024-09-11T09:48:18.550 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-mgr-dashboard-2:19.1.1-298.gb53b6d3e.el9.noar 12/12 2024-09-11T09:48:18.782 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-grafana-dashboards-2:19.1.1-298.gb53b6d3e.el9 11/12 2024-09-11T09:48:18.824 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-mgr-dashboard-2:19.1.1-298.gb53b6d3e.el9.noar 12/12 2024-09-11T09:48:19.476 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.1.1-298.gb53b6d3e.el9.noar 12/12 2024-09-11T09:48:19.476 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : ceph-grafana-dashboards-2:19.1.1-298.gb53b6d3e.el9 1/12 2024-09-11T09:48:19.476 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : ceph-mgr-dashboard-2:19.1.1-298.gb53b6d3e.el9.noar 2/12 2024-09-11T09:48:19.476 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : ceph-prometheus-alerts-2:19.1.1-298.gb53b6d3e.el9. 3/12 2024-09-11T09:48:19.476 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-09-11T09:48:19.477 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-09-11T09:48:19.477 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-09-11T09:48:19.477 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-09-11T09:48:19.477 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-09-11T09:48:19.477 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-09-11T09:48:19.477 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-09-11T09:48:19.477 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-09-11T09:48:19.809 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-09-11T09:48:19.810 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-11T09:48:19.810 INFO:teuthology.orchestra.run.smithi017.stdout:Installed: 2024-09-11T09:48:19.810 INFO:teuthology.orchestra.run.smithi017.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-09-11T09:48:19.810 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-grafana-dashboards-2:19.1.1-298.gb53b6d3e.el9.noarch 2024-09-11T09:48:19.810 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-mgr-dashboard-2:19.1.1-298.gb53b6d3e.el9.noarch 2024-09-11T09:48:19.810 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-prometheus-alerts-2:19.1.1-298.gb53b6d3e.el9.noarch 2024-09-11T09:48:19.810 INFO:teuthology.orchestra.run.smithi017.stdout: grpc-data-1.46.7-10.el9.noarch 2024-09-11T09:48:19.810 INFO:teuthology.orchestra.run.smithi017.stdout: protobuf-3.14.0-13.el9.x86_64 2024-09-11T09:48:19.810 INFO:teuthology.orchestra.run.smithi017.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-09-11T09:48:19.810 INFO:teuthology.orchestra.run.smithi017.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-09-11T09:48:19.810 INFO:teuthology.orchestra.run.smithi017.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-09-11T09:48:19.811 INFO:teuthology.orchestra.run.smithi017.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-09-11T09:48:19.811 INFO:teuthology.orchestra.run.smithi017.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-11T09:48:19.811 INFO:teuthology.orchestra.run.smithi017.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-11T09:48:19.811 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-11T09:48:19.811 INFO:teuthology.orchestra.run.smithi017.stdout:Complete! 2024-09-11T09:48:19.845 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.1.1-298.gb53b6d3e.el9.noar 12/12 2024-09-11T09:48:19.845 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-grafana-dashboards-2:19.1.1-298.gb53b6d3e.el9 1/12 2024-09-11T09:48:19.845 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-mgr-dashboard-2:19.1.1-298.gb53b6d3e.el9.noar 2/12 2024-09-11T09:48:19.845 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-prometheus-alerts-2:19.1.1-298.gb53b6d3e.el9. 3/12 2024-09-11T09:48:19.845 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-09-11T09:48:19.846 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-09-11T09:48:19.846 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-09-11T09:48:19.846 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-09-11T09:48:19.846 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-09-11T09:48:19.846 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-09-11T09:48:19.846 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-09-11T09:48:19.846 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-09-11T09:48:20.040 DEBUG:teuthology.orchestra.run.smithi017:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-11T09:48:20.122 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.1.1-298.gb53b6d3e.el9.noar 12/12 2024-09-11T09:48:20.122 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-grafana-dashboards-2:19.1.1-298.gb53b6d3e.el9 1/12 2024-09-11T09:48:20.122 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-mgr-dashboard-2:19.1.1-298.gb53b6d3e.el9.noar 2/12 2024-09-11T09:48:20.123 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-prometheus-alerts-2:19.1.1-298.gb53b6d3e.el9. 3/12 2024-09-11T09:48:20.123 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-09-11T09:48:20.123 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-09-11T09:48:20.123 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-09-11T09:48:20.123 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-09-11T09:48:20.123 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-09-11T09:48:20.123 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-09-11T09:48:20.123 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-09-11T09:48:20.123 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-09-11T09:48:20.161 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-09-11T09:48:20.161 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-11T09:48:20.161 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2024-09-11T09:48:20.161 INFO:teuthology.orchestra.run.smithi107.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-09-11T09:48:20.161 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-grafana-dashboards-2:19.1.1-298.gb53b6d3e.el9.noarch 2024-09-11T09:48:20.161 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-dashboard-2:19.1.1-298.gb53b6d3e.el9.noarch 2024-09-11T09:48:20.161 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-prometheus-alerts-2:19.1.1-298.gb53b6d3e.el9.noarch 2024-09-11T09:48:20.162 INFO:teuthology.orchestra.run.smithi107.stdout: grpc-data-1.46.7-10.el9.noarch 2024-09-11T09:48:20.162 INFO:teuthology.orchestra.run.smithi107.stdout: protobuf-3.14.0-13.el9.x86_64 2024-09-11T09:48:20.162 INFO:teuthology.orchestra.run.smithi107.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-09-11T09:48:20.162 INFO:teuthology.orchestra.run.smithi107.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-09-11T09:48:20.162 INFO:teuthology.orchestra.run.smithi107.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-09-11T09:48:20.162 INFO:teuthology.orchestra.run.smithi107.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-09-11T09:48:20.162 INFO:teuthology.orchestra.run.smithi107.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-11T09:48:20.162 INFO:teuthology.orchestra.run.smithi107.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-11T09:48:20.162 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-11T09:48:20.162 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-09-11T09:48:20.409 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-11T09:48:20.442 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-09-11T09:48:20.442 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-11T09:48:20.443 INFO:teuthology.orchestra.run.smithi096.stdout:Installed: 2024-09-11T09:48:20.443 INFO:teuthology.orchestra.run.smithi096.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-09-11T09:48:20.443 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-grafana-dashboards-2:19.1.1-298.gb53b6d3e.el9.noarch 2024-09-11T09:48:20.443 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mgr-dashboard-2:19.1.1-298.gb53b6d3e.el9.noarch 2024-09-11T09:48:20.443 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-prometheus-alerts-2:19.1.1-298.gb53b6d3e.el9.noarch 2024-09-11T09:48:20.443 INFO:teuthology.orchestra.run.smithi096.stdout: grpc-data-1.46.7-10.el9.noarch 2024-09-11T09:48:20.443 INFO:teuthology.orchestra.run.smithi096.stdout: protobuf-3.14.0-13.el9.x86_64 2024-09-11T09:48:20.443 INFO:teuthology.orchestra.run.smithi096.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-09-11T09:48:20.443 INFO:teuthology.orchestra.run.smithi096.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-09-11T09:48:20.443 INFO:teuthology.orchestra.run.smithi096.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-09-11T09:48:20.443 INFO:teuthology.orchestra.run.smithi096.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-09-11T09:48:20.443 INFO:teuthology.orchestra.run.smithi096.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-11T09:48:20.443 INFO:teuthology.orchestra.run.smithi096.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-11T09:48:20.444 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-11T09:48:20.444 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2024-09-11T09:48:20.622 INFO:teuthology.orchestra.run.smithi017.stdout:Last metadata expiration check: 0:01:07 ago on Wed 11 Sep 2024 09:47:13 AM UTC. 2024-09-11T09:48:20.690 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-11T09:48:20.762 INFO:teuthology.orchestra.run.smithi017.stdout:Dependencies resolved. 2024-09-11T09:48:20.763 INFO:teuthology.orchestra.run.smithi017.stdout:====================================================================================== 2024-09-11T09:48:20.763 INFO:teuthology.orchestra.run.smithi017.stdout: Package Arch Version Repository Size 2024-09-11T09:48:20.763 INFO:teuthology.orchestra.run.smithi017.stdout:====================================================================================== 2024-09-11T09:48:20.763 INFO:teuthology.orchestra.run.smithi017.stdout:Installing: 2024-09-11T09:48:20.763 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-mgr-diskprediction-local noarch 2:19.1.1-298.gb53b6d3e.el9 ceph-noarch 7.4 M 2024-09-11T09:48:20.763 INFO:teuthology.orchestra.run.smithi017.stdout:Installing dependencies: 2024-09-11T09:48:20.763 INFO:teuthology.orchestra.run.smithi017.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-11T09:48:20.763 INFO:teuthology.orchestra.run.smithi017.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-11T09:48:20.763 INFO:teuthology.orchestra.run.smithi017.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-11T09:48:20.763 INFO:teuthology.orchestra.run.smithi017.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-11T09:48:20.764 INFO:teuthology.orchestra.run.smithi017.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-11T09:48:20.764 INFO:teuthology.orchestra.run.smithi017.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-11T09:48:20.764 INFO:teuthology.orchestra.run.smithi017.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-11T09:48:20.764 INFO:teuthology.orchestra.run.smithi017.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-11T09:48:20.764 INFO:teuthology.orchestra.run.smithi017.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-11T09:48:20.764 INFO:teuthology.orchestra.run.smithi017.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-11T09:48:20.764 INFO:teuthology.orchestra.run.smithi017.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-11T09:48:20.764 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-11T09:48:20.764 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction Summary 2024-09-11T09:48:20.764 INFO:teuthology.orchestra.run.smithi017.stdout:====================================================================================== 2024-09-11T09:48:20.764 INFO:teuthology.orchestra.run.smithi017.stdout:Install 12 Packages 2024-09-11T09:48:20.764 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-11T09:48:20.765 INFO:teuthology.orchestra.run.smithi017.stdout:Total download size: 38 M 2024-09-11T09:48:20.765 INFO:teuthology.orchestra.run.smithi017.stdout:Installed size: 212 M 2024-09-11T09:48:20.765 INFO:teuthology.orchestra.run.smithi017.stdout:Downloading Packages: 2024-09-11T09:48:20.964 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:01:07 ago on Wed 11 Sep 2024 09:47:13 AM UTC. 2024-09-11T09:48:21.098 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-09-11T09:48:21.099 INFO:teuthology.orchestra.run.smithi107.stdout:====================================================================================== 2024-09-11T09:48:21.099 INFO:teuthology.orchestra.run.smithi107.stdout: Package Arch Version Repository Size 2024-09-11T09:48:21.099 INFO:teuthology.orchestra.run.smithi107.stdout:====================================================================================== 2024-09-11T09:48:21.099 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2024-09-11T09:48:21.099 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-diskprediction-local noarch 2:19.1.1-298.gb53b6d3e.el9 ceph-noarch 7.4 M 2024-09-11T09:48:21.099 INFO:teuthology.orchestra.run.smithi107.stdout:Installing dependencies: 2024-09-11T09:48:21.100 INFO:teuthology.orchestra.run.smithi107.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-11T09:48:21.100 INFO:teuthology.orchestra.run.smithi107.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-11T09:48:21.100 INFO:teuthology.orchestra.run.smithi107.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-11T09:48:21.100 INFO:teuthology.orchestra.run.smithi107.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-11T09:48:21.100 INFO:teuthology.orchestra.run.smithi107.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-11T09:48:21.100 INFO:teuthology.orchestra.run.smithi107.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-11T09:48:21.100 INFO:teuthology.orchestra.run.smithi107.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-11T09:48:21.100 INFO:teuthology.orchestra.run.smithi107.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-11T09:48:21.100 INFO:teuthology.orchestra.run.smithi107.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-11T09:48:21.100 INFO:teuthology.orchestra.run.smithi107.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-11T09:48:21.100 INFO:teuthology.orchestra.run.smithi107.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-11T09:48:21.100 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-11T09:48:21.100 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2024-09-11T09:48:21.101 INFO:teuthology.orchestra.run.smithi107.stdout:====================================================================================== 2024-09-11T09:48:21.101 INFO:teuthology.orchestra.run.smithi107.stdout:Install 12 Packages 2024-09-11T09:48:21.101 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-11T09:48:21.101 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 38 M 2024-09-11T09:48:21.101 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 212 M 2024-09-11T09:48:21.101 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2024-09-11T09:48:21.244 INFO:teuthology.orchestra.run.smithi096.stdout:Last metadata expiration check: 0:01:09 ago on Wed 11 Sep 2024 09:47:12 AM UTC. 2024-09-11T09:48:21.281 INFO:teuthology.orchestra.run.smithi107.stdout:(1/12): libquadmath-11.5.0-2.el9.x86_64.rpm 1.4 MB/s | 191 kB 00:00 2024-09-11T09:48:21.331 INFO:teuthology.orchestra.run.smithi107.stdout:(2/12): libgfortran-11.5.0-2.el9.x86_64.rpm 4.2 MB/s | 801 kB 00:00 2024-09-11T09:48:21.356 INFO:teuthology.orchestra.run.smithi107.stdout:(3/12): flexiblas-3.0.4-8.el9.x86_64.rpm 434 kB/s | 33 kB 00:00 2024-09-11T09:48:21.363 INFO:teuthology.orchestra.run.smithi017.stdout:(1/12): libquadmath-11.5.0-2.el9.x86_64.rpm 931 kB/s | 191 kB 00:00 2024-09-11T09:48:21.377 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2024-09-11T09:48:21.378 INFO:teuthology.orchestra.run.smithi096.stdout:====================================================================================== 2024-09-11T09:48:21.378 INFO:teuthology.orchestra.run.smithi096.stdout: Package Arch Version Repository Size 2024-09-11T09:48:21.378 INFO:teuthology.orchestra.run.smithi096.stdout:====================================================================================== 2024-09-11T09:48:21.378 INFO:teuthology.orchestra.run.smithi096.stdout:Installing: 2024-09-11T09:48:21.378 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mgr-diskprediction-local noarch 2:19.1.1-298.gb53b6d3e.el9 ceph-noarch 7.4 M 2024-09-11T09:48:21.378 INFO:teuthology.orchestra.run.smithi096.stdout:Installing dependencies: 2024-09-11T09:48:21.378 INFO:teuthology.orchestra.run.smithi096.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-11T09:48:21.379 INFO:teuthology.orchestra.run.smithi096.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-11T09:48:21.379 INFO:teuthology.orchestra.run.smithi096.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-11T09:48:21.379 INFO:teuthology.orchestra.run.smithi096.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-11T09:48:21.379 INFO:teuthology.orchestra.run.smithi096.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-11T09:48:21.379 INFO:teuthology.orchestra.run.smithi096.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-11T09:48:21.379 INFO:teuthology.orchestra.run.smithi096.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-11T09:48:21.379 INFO:teuthology.orchestra.run.smithi096.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-11T09:48:21.379 INFO:teuthology.orchestra.run.smithi096.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-11T09:48:21.379 INFO:teuthology.orchestra.run.smithi096.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-11T09:48:21.379 INFO:teuthology.orchestra.run.smithi096.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-11T09:48:21.379 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-11T09:48:21.379 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction Summary 2024-09-11T09:48:21.379 INFO:teuthology.orchestra.run.smithi096.stdout:====================================================================================== 2024-09-11T09:48:21.380 INFO:teuthology.orchestra.run.smithi096.stdout:Install 12 Packages 2024-09-11T09:48:21.380 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-11T09:48:21.380 INFO:teuthology.orchestra.run.smithi096.stdout:Total download size: 38 M 2024-09-11T09:48:21.380 INFO:teuthology.orchestra.run.smithi096.stdout:Installed size: 212 M 2024-09-11T09:48:21.380 INFO:teuthology.orchestra.run.smithi096.stdout:Downloading Packages: 2024-09-11T09:48:21.398 INFO:teuthology.orchestra.run.smithi107.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 439 kB/s | 18 kB 00:00 2024-09-11T09:48:21.421 INFO:teuthology.orchestra.run.smithi017.stdout:(2/12): libgfortran-11.5.0-2.el9.x86_64.rpm 3.0 MB/s | 801 kB 00:00 2024-09-11T09:48:21.448 INFO:teuthology.orchestra.run.smithi107.stdout:(5/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 26 MB/s | 3.0 MB 00:00 2024-09-11T09:48:21.474 INFO:teuthology.orchestra.run.smithi107.stdout:(6/12): openblas-0.3.26-2.el9.x86_64.rpm 510 kB/s | 38 kB 00:00 2024-09-11T09:48:21.480 INFO:teuthology.orchestra.run.smithi017.stdout:(3/12): flexiblas-3.0.4-8.el9.x86_64.rpm 279 kB/s | 33 kB 00:00 2024-09-11T09:48:21.555 INFO:teuthology.orchestra.run.smithi017.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 242 kB/s | 18 kB 00:00 2024-09-11T09:48:21.584 INFO:teuthology.orchestra.run.smithi107.stdout:(7/12): openblas-openmp-0.3.26-2.el9.x86_64.rpm 36 MB/s | 4.9 MB 00:00 2024-09-11T09:48:21.616 INFO:teuthology.orchestra.run.smithi107.stdout:(8/12): python3-devel-3.9.19-8.el9.x86_64.rpm 1.7 MB/s | 245 kB 00:00 2024-09-11T09:48:21.656 INFO:teuthology.orchestra.run.smithi017.stdout:(5/12): ceph-mgr-diskprediction-local-19.1.1-29 15 MB/s | 7.4 MB 00:00 2024-09-11T09:48:21.666 INFO:teuthology.orchestra.run.smithi107.stdout:(9/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 4.9 MB/s | 250 kB 00:00 2024-09-11T09:48:21.681 INFO:teuthology.orchestra.run.smithi017.stdout:(6/12): openblas-0.3.26-2.el9.x86_64.rpm 307 kB/s | 38 kB 00:00 2024-09-11T09:48:21.733 INFO:teuthology.orchestra.run.smithi107.stdout:(10/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 34 MB/s | 5.1 MB 00:00 2024-09-11T09:48:21.739 INFO:teuthology.orchestra.run.smithi017.stdout:(7/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 9.4 MB/s | 3.0 MB 00:00 2024-09-11T09:48:21.782 INFO:teuthology.orchestra.run.smithi017.stdout:(8/12): python3-devel-3.9.19-8.el9.x86_64.rpm 2.4 MB/s | 245 kB 00:00 2024-09-11T09:48:21.834 INFO:teuthology.orchestra.run.smithi107.stdout:(11/12): ceph-mgr-diskprediction-local-19.1.1-2 11 MB/s | 7.4 MB 00:00 2024-09-11T09:48:21.891 INFO:teuthology.orchestra.run.smithi017.stdout:(9/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 34 MB/s | 5.1 MB 00:00 2024-09-11T09:48:21.915 INFO:teuthology.orchestra.run.smithi017.stdout:(10/12): python3-numpy-f2py-1.20.1-5.el9.x86_64 1.8 MB/s | 250 kB 00:00 2024-09-11T09:48:21.937 INFO:teuthology.orchestra.run.smithi096.stdout:(1/12): libquadmath-11.5.0-2.el9.x86_64.rpm 922 kB/s | 191 kB 00:00 2024-09-11T09:48:22.037 INFO:teuthology.orchestra.run.smithi096.stdout:(2/12): libgfortran-11.5.0-2.el9.x86_64.rpm 2.5 MB/s | 801 kB 00:00 2024-09-11T09:48:22.049 INFO:teuthology.orchestra.run.smithi017.stdout:(11/12): openblas-openmp-0.3.26-2.el9.x86_64.rp 13 MB/s | 4.9 MB 00:00 2024-09-11T09:48:22.109 INFO:teuthology.orchestra.run.smithi107.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 37 MB/s | 16 MB 00:00 2024-09-11T09:48:22.110 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2024-09-11T09:48:22.110 INFO:teuthology.orchestra.run.smithi107.stdout:Total 38 MB/s | 38 MB 00:01 2024-09-11T09:48:22.188 INFO:teuthology.orchestra.run.smithi096.stdout:(3/12): ceph-mgr-diskprediction-local-19.1.1-29 16 MB/s | 7.4 MB 00:00 2024-09-11T09:48:22.224 INFO:teuthology.orchestra.run.smithi017.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 49 MB/s | 16 MB 00:00 2024-09-11T09:48:22.225 INFO:teuthology.orchestra.run.smithi017.stdout:-------------------------------------------------------------------------------- 2024-09-11T09:48:22.226 INFO:teuthology.orchestra.run.smithi017.stdout:Total 26 MB/s | 38 MB 00:01 2024-09-11T09:48:22.238 INFO:teuthology.orchestra.run.smithi096.stdout:(4/12): flexiblas-3.0.4-8.el9.x86_64.rpm 109 kB/s | 33 kB 00:00 2024-09-11T09:48:22.438 INFO:teuthology.orchestra.run.smithi096.stdout:(5/12): openblas-0.3.26-2.el9.x86_64.rpm 192 kB/s | 38 kB 00:00 2024-09-11T09:48:22.463 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2024-09-11T09:48:22.472 INFO:teuthology.orchestra.run.smithi096.stdout:(6/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 64 kB/s | 18 kB 00:00 2024-09-11T09:48:22.488 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2024-09-11T09:48:22.488 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2024-09-11T09:48:22.598 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction check 2024-09-11T09:48:22.625 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction check succeeded. 2024-09-11T09:48:22.625 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction test 2024-09-11T09:48:22.915 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2024-09-11T09:48:22.915 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2024-09-11T09:48:22.930 INFO:teuthology.orchestra.run.smithi096.stdout:(7/12): python3-devel-3.9.19-8.el9.x86_64.rpm 534 kB/s | 245 kB 00:00 2024-09-11T09:48:23.084 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction test succeeded. 2024-09-11T09:48:23.084 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction 2024-09-11T09:48:23.131 INFO:teuthology.orchestra.run.smithi096.stdout:(8/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 2.7 MB/s | 3.0 MB 00:01 2024-09-11T09:48:23.256 INFO:teuthology.orchestra.run.smithi096.stdout:(9/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.0 MB/s | 250 kB 00:00 2024-09-11T09:48:23.331 INFO:teuthology.orchestra.run.smithi096.stdout:(10/12): openblas-openmp-0.3.26-2.el9.x86_64.rp 5.5 MB/s | 4.9 MB 00:00 2024-09-11T09:48:23.669 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2024-09-11T09:48:23.681 INFO:teuthology.orchestra.run.smithi096.stdout:(11/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 6.8 MB/s | 5.1 MB 00:00 2024-09-11T09:48:23.737 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-11T09:48:23.773 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-11T09:48:23.871 INFO:teuthology.orchestra.run.smithi017.stdout: Preparing : 1/1 2024-09-11T09:48:23.880 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-11T09:48:23.925 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-11T09:48:23.932 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-11T09:48:23.981 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-11T09:48:24.085 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-11T09:48:24.110 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-11T09:48:24.124 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-11T09:48:24.140 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-11T09:48:24.217 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-11T09:48:24.316 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-11T09:48:24.348 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-11T09:48:24.427 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-11T09:48:24.863 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-11T09:48:24.978 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-11T09:48:25.110 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-11T09:48:25.240 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-11T09:48:26.209 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-11T09:48:26.541 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-11T09:48:27.236 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-11T09:48:27.266 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-mgr-diskprediction-local-2:19.1.1-298.gb53b6d 12/12 2024-09-11T09:48:27.637 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-11T09:48:27.664 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : ceph-mgr-diskprediction-local-2:19.1.1-298.gb53b6d 12/12 2024-09-11T09:48:29.842 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.1.1-298.gb53b6d 12/12 2024-09-11T09:48:29.843 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.1.1-298.gb53b6d 1/12 2024-09-11T09:48:29.843 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-11T09:48:29.843 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-11T09:48:29.843 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-11T09:48:29.843 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-11T09:48:29.843 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-11T09:48:29.843 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-11T09:48:29.843 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-11T09:48:29.843 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-11T09:48:29.843 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-11T09:48:29.843 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-11T09:48:30.104 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.1.1-298.gb53b6d 12/12 2024-09-11T09:48:30.104 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.1.1-298.gb53b6d 1/12 2024-09-11T09:48:30.104 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-11T09:48:30.104 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-11T09:48:30.104 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-11T09:48:30.104 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-11T09:48:30.104 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-11T09:48:30.104 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-11T09:48:30.104 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-11T09:48:30.105 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-11T09:48:30.105 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-11T09:48:30.105 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-11T09:48:30.227 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-11T09:48:30.227 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-11T09:48:30.227 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2024-09-11T09:48:30.227 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-diskprediction-local-2:19.1.1-298.gb53b6d3e.el9.noarch 2024-09-11T09:48:30.227 INFO:teuthology.orchestra.run.smithi107.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-11T09:48:30.227 INFO:teuthology.orchestra.run.smithi107.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-11T09:48:30.227 INFO:teuthology.orchestra.run.smithi107.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-11T09:48:30.228 INFO:teuthology.orchestra.run.smithi107.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-11T09:48:30.228 INFO:teuthology.orchestra.run.smithi107.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-11T09:48:30.228 INFO:teuthology.orchestra.run.smithi107.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-11T09:48:30.228 INFO:teuthology.orchestra.run.smithi107.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-11T09:48:30.228 INFO:teuthology.orchestra.run.smithi107.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-11T09:48:30.228 INFO:teuthology.orchestra.run.smithi107.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-11T09:48:30.228 INFO:teuthology.orchestra.run.smithi107.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-11T09:48:30.228 INFO:teuthology.orchestra.run.smithi107.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-11T09:48:30.228 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-11T09:48:30.228 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-09-11T09:48:30.526 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-11T09:48:30.526 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-11T09:48:30.527 INFO:teuthology.orchestra.run.smithi017.stdout:Installed: 2024-09-11T09:48:30.527 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-mgr-diskprediction-local-2:19.1.1-298.gb53b6d3e.el9.noarch 2024-09-11T09:48:30.527 INFO:teuthology.orchestra.run.smithi017.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-11T09:48:30.527 INFO:teuthology.orchestra.run.smithi017.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-11T09:48:30.527 INFO:teuthology.orchestra.run.smithi017.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-11T09:48:30.527 INFO:teuthology.orchestra.run.smithi017.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-11T09:48:30.527 INFO:teuthology.orchestra.run.smithi017.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-11T09:48:30.527 INFO:teuthology.orchestra.run.smithi017.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-11T09:48:30.527 INFO:teuthology.orchestra.run.smithi017.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-11T09:48:30.527 INFO:teuthology.orchestra.run.smithi017.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-11T09:48:30.527 INFO:teuthology.orchestra.run.smithi017.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-11T09:48:30.527 INFO:teuthology.orchestra.run.smithi017.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-11T09:48:30.527 INFO:teuthology.orchestra.run.smithi017.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-11T09:48:30.527 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-11T09:48:30.528 INFO:teuthology.orchestra.run.smithi017.stdout:Complete! 2024-09-11T09:48:30.655 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install ceph-mgr-rook 2024-09-11T09:48:30.892 DEBUG:teuthology.orchestra.run.smithi017:> sudo yum -y install ceph-mgr-rook 2024-09-11T09:48:31.224 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:01:18 ago on Wed 11 Sep 2024 09:47:13 AM UTC. 2024-09-11T09:48:31.357 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-09-11T09:48:31.358 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-09-11T09:48:31.358 INFO:teuthology.orchestra.run.smithi107.stdout: Package Arch Version Repository Size 2024-09-11T09:48:31.359 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-09-11T09:48:31.359 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2024-09-11T09:48:31.359 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-rook noarch 2:19.1.1-298.gb53b6d3e.el9 ceph-noarch 49 k 2024-09-11T09:48:31.359 INFO:teuthology.orchestra.run.smithi107.stdout:Installing dependencies: 2024-09-11T09:48:31.359 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-11T09:48:31.359 INFO:teuthology.orchestra.run.smithi107.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-11T09:48:31.359 INFO:teuthology.orchestra.run.smithi107.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-11T09:48:31.359 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-11T09:48:31.359 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-11T09:48:31.359 INFO:teuthology.orchestra.run.smithi107.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-11T09:48:31.359 INFO:teuthology.orchestra.run.smithi107.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-11T09:48:31.359 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-11T09:48:31.359 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-11T09:48:31.360 INFO:teuthology.orchestra.run.smithi107.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-11T09:48:31.360 INFO:teuthology.orchestra.run.smithi107.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-11T09:48:31.360 INFO:teuthology.orchestra.run.smithi107.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-11T09:48:31.360 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-11T09:48:31.360 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2024-09-11T09:48:31.360 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-09-11T09:48:31.360 INFO:teuthology.orchestra.run.smithi107.stdout:Install 13 Packages 2024-09-11T09:48:31.360 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-11T09:48:31.361 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 2.2 M 2024-09-11T09:48:31.361 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 26 M 2024-09-11T09:48:31.361 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2024-09-11T09:48:31.483 INFO:teuthology.orchestra.run.smithi017.stdout:Last metadata expiration check: 0:01:18 ago on Wed 11 Sep 2024 09:47:13 AM UTC. 2024-09-11T09:48:31.622 INFO:teuthology.orchestra.run.smithi017.stdout:Dependencies resolved. 2024-09-11T09:48:31.624 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2024-09-11T09:48:31.624 INFO:teuthology.orchestra.run.smithi017.stdout: Package Arch Version Repository Size 2024-09-11T09:48:31.624 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2024-09-11T09:48:31.624 INFO:teuthology.orchestra.run.smithi017.stdout:Installing: 2024-09-11T09:48:31.624 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-mgr-rook noarch 2:19.1.1-298.gb53b6d3e.el9 ceph-noarch 49 k 2024-09-11T09:48:31.624 INFO:teuthology.orchestra.run.smithi017.stdout:Installing dependencies: 2024-09-11T09:48:31.624 INFO:teuthology.orchestra.run.smithi017.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-11T09:48:31.624 INFO:teuthology.orchestra.run.smithi017.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-11T09:48:31.624 INFO:teuthology.orchestra.run.smithi017.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-11T09:48:31.624 INFO:teuthology.orchestra.run.smithi017.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-11T09:48:31.624 INFO:teuthology.orchestra.run.smithi017.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-11T09:48:31.624 INFO:teuthology.orchestra.run.smithi017.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-11T09:48:31.625 INFO:teuthology.orchestra.run.smithi017.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-11T09:48:31.625 INFO:teuthology.orchestra.run.smithi017.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-11T09:48:31.625 INFO:teuthology.orchestra.run.smithi017.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-11T09:48:31.625 INFO:teuthology.orchestra.run.smithi017.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-11T09:48:31.625 INFO:teuthology.orchestra.run.smithi017.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-11T09:48:31.625 INFO:teuthology.orchestra.run.smithi017.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-11T09:48:31.625 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-11T09:48:31.625 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction Summary 2024-09-11T09:48:31.625 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2024-09-11T09:48:31.625 INFO:teuthology.orchestra.run.smithi017.stdout:Install 13 Packages 2024-09-11T09:48:31.625 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-11T09:48:31.626 INFO:teuthology.orchestra.run.smithi017.stdout:Total download size: 2.2 M 2024-09-11T09:48:31.626 INFO:teuthology.orchestra.run.smithi017.stdout:Installed size: 26 M 2024-09-11T09:48:31.626 INFO:teuthology.orchestra.run.smithi017.stdout:Downloading Packages: 2024-09-11T09:48:31.643 INFO:teuthology.orchestra.run.smithi107.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 342 kB/s | 26 kB 00:00 2024-09-11T09:48:31.668 INFO:teuthology.orchestra.run.smithi107.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 191 kB/s | 19 kB 00:00 2024-09-11T09:48:31.751 INFO:teuthology.orchestra.run.smithi107.stdout:(3/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.9 MB/s | 159 kB 00:00 2024-09-11T09:48:31.777 INFO:teuthology.orchestra.run.smithi107.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.0 MB/s | 222 kB 00:00 2024-09-11T09:48:31.819 INFO:teuthology.orchestra.run.smithi107.stdout:(5/13): python3-requests-oauthlib-1.3.0-12.el9. 1.3 MB/s | 54 kB 00:00 2024-09-11T09:48:31.853 INFO:teuthology.orchestra.run.smithi107.stdout:(6/13): python3-pyasn1-modules-0.4.8-6.el9.noar 2.7 MB/s | 279 kB 00:00 2024-09-11T09:48:31.895 INFO:teuthology.orchestra.run.smithi107.stdout:(7/13): ceph-mgr-rook-19.1.1-298.gb53b6d3e.el9. 151 kB/s | 49 kB 00:00 2024-09-11T09:48:31.920 INFO:teuthology.orchestra.run.smithi107.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 318 kB/s | 32 kB 00:00 2024-09-11T09:48:31.945 INFO:teuthology.orchestra.run.smithi107.stdout:(9/13): python3-certifi-2023.05.07-4.el9.noarch 153 kB/s | 14 kB 00:00 2024-09-11T09:48:31.970 INFO:teuthology.orchestra.run.smithi107.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 2.5 MB/s | 59 kB 00:00 2024-09-11T09:48:31.982 INFO:teuthology.orchestra.run.smithi017.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 84 kB/s | 26 kB 00:00 2024-09-11T09:48:31.996 INFO:teuthology.orchestra.run.smithi107.stdout:(11/13): python3-google-auth-2.34.0-1.el9.noarc 2.2 MB/s | 220 kB 00:00 2024-09-11T09:48:32.007 INFO:teuthology.orchestra.run.smithi017.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 58 kB/s | 19 kB 00:00 2024-09-11T09:48:32.013 INFO:teuthology.orchestra.run.smithi107.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 2.1 MB/s | 90 kB 00:00 2024-09-11T09:48:32.032 INFO:teuthology.orchestra.run.smithi017.stdout:(3/13): ceph-mgr-rook-19.1.1-298.gb53b6d3e.el9. 139 kB/s | 49 kB 00:00 2024-09-11T09:48:32.054 INFO:teuthology.orchestra.run.smithi107.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 8.0 MB/s | 1.0 MB 00:00 2024-09-11T09:48:32.058 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2024-09-11T09:48:32.058 INFO:teuthology.orchestra.run.smithi107.stdout:Total 3.2 MB/s | 2.2 MB 00:00 2024-09-11T09:48:32.120 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2024-09-11T09:48:32.136 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2024-09-11T09:48:32.136 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2024-09-11T09:48:32.226 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2024-09-11T09:48:32.227 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2024-09-11T09:48:32.441 INFO:teuthology.orchestra.run.smithi017.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 484 kB/s | 222 kB 00:00 2024-09-11T09:48:32.458 INFO:teuthology.orchestra.run.smithi017.stdout:(5/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 352 kB/s | 159 kB 00:00 2024-09-11T09:48:32.499 INFO:teuthology.orchestra.run.smithi017.stdout:(6/13): python3-cachetools-4.2.4-1.el9.noarch.r 774 kB/s | 32 kB 00:00 2024-09-11T09:48:32.525 INFO:teuthology.orchestra.run.smithi017.stdout:(7/13): python3-certifi-2023.05.07-4.el9.noarch 568 kB/s | 14 kB 00:00 2024-09-11T09:48:32.550 INFO:teuthology.orchestra.run.smithi017.stdout:(8/13): python3-requests-oauthlib-1.3.0-12.el9. 491 kB/s | 54 kB 00:00 2024-09-11T09:48:32.575 INFO:teuthology.orchestra.run.smithi017.stdout:(9/13): python3-google-auth-2.34.0-1.el9.noarch 4.3 MB/s | 220 kB 00:00 2024-09-11T09:48:32.609 INFO:teuthology.orchestra.run.smithi017.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 2.3 MB/s | 59 kB 00:00 2024-09-11T09:48:32.634 INFO:teuthology.orchestra.run.smithi017.stdout:(11/13): python3-websocket-client-1.2.3-2.el9.n 3.8 MB/s | 90 kB 00:00 2024-09-11T09:48:32.676 INFO:teuthology.orchestra.run.smithi017.stdout:(12/13): python3-kubernetes-26.1.0-2.el9.noarch 8.1 MB/s | 1.0 MB 00:00 2024-09-11T09:48:32.701 INFO:teuthology.orchestra.run.smithi017.stdout:(13/13): python3-pyasn1-modules-0.4.8-6.el9.noa 417 kB/s | 279 kB 00:00 2024-09-11T09:48:32.703 INFO:teuthology.orchestra.run.smithi017.stdout:-------------------------------------------------------------------------------- 2024-09-11T09:48:32.704 INFO:teuthology.orchestra.run.smithi017.stdout:Total 2.1 MB/s | 2.2 MB 00:01 2024-09-11T09:48:32.737 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2024-09-11T09:48:32.777 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction check 2024-09-11T09:48:32.794 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction check succeeded. 2024-09-11T09:48:32.795 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction test 2024-09-11T09:48:32.893 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction test succeeded. 2024-09-11T09:48:32.894 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction 2024-09-11T09:48:32.901 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-11T09:48:32.948 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-11T09:48:33.002 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-11T09:48:33.040 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-11T09:48:33.081 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-11T09:48:33.183 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-11T09:48:33.285 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-11T09:48:33.342 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-11T09:48:33.441 INFO:teuthology.orchestra.run.smithi017.stdout: Preparing : 1/1 2024-09-11T09:48:33.624 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-11T09:48:33.688 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-11T09:48:33.744 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-11T09:48:33.787 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-11T09:48:33.829 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-11T09:48:33.944 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-11T09:48:34.054 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-11T09:48:34.060 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-11T09:48:34.107 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-11T09:48:34.114 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-11T09:48:34.148 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-11T09:48:34.196 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-11T09:48:34.218 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-mgr-rook-2:19.1.1-298.gb53b6d3e.el9.noarch 13/13 2024-09-11T09:48:34.874 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-11T09:48:34.928 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-11T09:48:34.962 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-11T09:48:35.011 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-11T09:48:35.040 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : ceph-mgr-rook-2:19.1.1-298.gb53b6d3e.el9.noarch 13/13 2024-09-11T09:48:35.042 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-mgr-rook-2:19.1.1-298.gb53b6d3e.el9.noarch 13/13 2024-09-11T09:48:35.042 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-mgr-rook-2:19.1.1-298.gb53b6d3e.el9.noarch 1/13 2024-09-11T09:48:35.042 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-11T09:48:35.042 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-11T09:48:35.042 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-11T09:48:35.042 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-11T09:48:35.042 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-11T09:48:35.043 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-11T09:48:35.043 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-11T09:48:35.043 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-11T09:48:35.043 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-11T09:48:35.043 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-11T09:48:35.043 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-11T09:48:35.375 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-11T09:48:35.376 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-11T09:48:35.376 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2024-09-11T09:48:35.376 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-rook-2:19.1.1-298.gb53b6d3e.el9.noarch 2024-09-11T09:48:35.376 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-11T09:48:35.376 INFO:teuthology.orchestra.run.smithi107.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-11T09:48:35.376 INFO:teuthology.orchestra.run.smithi107.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-11T09:48:35.376 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-11T09:48:35.376 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-11T09:48:35.376 INFO:teuthology.orchestra.run.smithi107.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-11T09:48:35.376 INFO:teuthology.orchestra.run.smithi107.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-11T09:48:35.376 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-11T09:48:35.376 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-11T09:48:35.377 INFO:teuthology.orchestra.run.smithi107.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-11T09:48:35.377 INFO:teuthology.orchestra.run.smithi107.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-11T09:48:35.377 INFO:teuthology.orchestra.run.smithi107.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-11T09:48:35.377 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-11T09:48:35.377 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-09-11T09:48:35.654 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install ceph-mgr-cephadm 2024-09-11T09:48:35.860 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: ceph-mgr-rook-2:19.1.1-298.gb53b6d3e.el9.noarch 13/13 2024-09-11T09:48:35.860 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : ceph-mgr-rook-2:19.1.1-298.gb53b6d3e.el9.noarch 1/13 2024-09-11T09:48:35.860 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-11T09:48:35.860 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-11T09:48:35.860 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-11T09:48:35.860 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-11T09:48:35.860 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-11T09:48:35.860 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-11T09:48:35.860 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-11T09:48:35.860 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-11T09:48:35.861 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-11T09:48:35.861 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-11T09:48:35.861 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-11T09:48:36.210 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-11T09:48:36.210 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-11T09:48:36.210 INFO:teuthology.orchestra.run.smithi017.stdout:Installed: 2024-09-11T09:48:36.210 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-mgr-rook-2:19.1.1-298.gb53b6d3e.el9.noarch 2024-09-11T09:48:36.210 INFO:teuthology.orchestra.run.smithi017.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-11T09:48:36.210 INFO:teuthology.orchestra.run.smithi017.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-11T09:48:36.210 INFO:teuthology.orchestra.run.smithi017.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-11T09:48:36.210 INFO:teuthology.orchestra.run.smithi017.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-11T09:48:36.210 INFO:teuthology.orchestra.run.smithi017.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-11T09:48:36.210 INFO:teuthology.orchestra.run.smithi017.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-11T09:48:36.211 INFO:teuthology.orchestra.run.smithi017.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-11T09:48:36.211 INFO:teuthology.orchestra.run.smithi017.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-11T09:48:36.211 INFO:teuthology.orchestra.run.smithi017.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-11T09:48:36.211 INFO:teuthology.orchestra.run.smithi017.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-11T09:48:36.211 INFO:teuthology.orchestra.run.smithi017.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-11T09:48:36.211 INFO:teuthology.orchestra.run.smithi017.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-11T09:48:36.211 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-11T09:48:36.211 INFO:teuthology.orchestra.run.smithi017.stdout:Complete! 2024-09-11T09:48:36.221 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:01:23 ago on Wed 11 Sep 2024 09:47:13 AM UTC. 2024-09-11T09:48:36.354 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-09-11T09:48:36.355 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-09-11T09:48:36.355 INFO:teuthology.orchestra.run.smithi107.stdout: Package Arch Version Repository Size 2024-09-11T09:48:36.355 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-09-11T09:48:36.355 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2024-09-11T09:48:36.355 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-cephadm noarch 2:19.1.1-298.gb53b6d3e.el9 ceph-noarch 146 k 2024-09-11T09:48:36.356 INFO:teuthology.orchestra.run.smithi107.stdout:Installing dependencies: 2024-09-11T09:48:36.356 INFO:teuthology.orchestra.run.smithi107.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-11T09:48:36.356 INFO:teuthology.orchestra.run.smithi107.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-11T09:48:36.356 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-11T09:48:36.356 INFO:teuthology.orchestra.run.smithi107.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-11T09:48:36.356 INFO:teuthology.orchestra.run.smithi107.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-11T09:48:36.356 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-11T09:48:36.357 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2024-09-11T09:48:36.357 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-09-11T09:48:36.357 INFO:teuthology.orchestra.run.smithi107.stdout:Install 6 Packages 2024-09-11T09:48:36.357 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-11T09:48:36.358 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 7.0 M 2024-09-11T09:48:36.358 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 33 M 2024-09-11T09:48:36.358 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2024-09-11T09:48:36.452 DEBUG:teuthology.orchestra.run.smithi017:> sudo yum -y install ceph-mgr-cephadm 2024-09-11T09:48:36.674 INFO:teuthology.orchestra.run.smithi107.stdout:(1/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 1.8 MB/s | 248 kB 00:00 2024-09-11T09:48:36.724 INFO:teuthology.orchestra.run.smithi107.stdout:(2/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 11 MB/s | 548 kB 00:00 2024-09-11T09:48:36.750 INFO:teuthology.orchestra.run.smithi107.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 2.2 MB/s | 58 kB 00:00 2024-09-11T09:48:36.775 INFO:teuthology.orchestra.run.smithi107.stdout:(4/6): python3-typing-extensions-4.12.2-2.el9.n 2.9 MB/s | 74 kB 00:00 2024-09-11T09:48:36.850 INFO:teuthology.orchestra.run.smithi107.stdout:(5/6): python3-babel-2.9.1-2.el9.noarch.rpm 19 MB/s | 6.0 MB 00:00 2024-09-11T09:48:36.950 INFO:teuthology.orchestra.run.smithi107.stdout:(6/6): ceph-mgr-cephadm-19.1.1-298.gb53b6d3e.el 355 kB/s | 146 kB 00:00 2024-09-11T09:48:36.952 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2024-09-11T09:48:36.952 INFO:teuthology.orchestra.run.smithi107.stdout:Total 12 MB/s | 7.0 MB 00:00 2024-09-11T09:48:37.022 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2024-09-11T09:48:37.034 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2024-09-11T09:48:37.034 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2024-09-11T09:48:37.045 INFO:teuthology.orchestra.run.smithi017.stdout:Last metadata expiration check: 0:01:24 ago on Wed 11 Sep 2024 09:47:13 AM UTC. 2024-09-11T09:48:37.133 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2024-09-11T09:48:37.134 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2024-09-11T09:48:37.185 INFO:teuthology.orchestra.run.smithi017.stdout:Dependencies resolved. 2024-09-11T09:48:37.185 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2024-09-11T09:48:37.185 INFO:teuthology.orchestra.run.smithi017.stdout: Package Arch Version Repository Size 2024-09-11T09:48:37.185 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2024-09-11T09:48:37.186 INFO:teuthology.orchestra.run.smithi017.stdout:Installing: 2024-09-11T09:48:37.186 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-mgr-cephadm noarch 2:19.1.1-298.gb53b6d3e.el9 ceph-noarch 146 k 2024-09-11T09:48:37.186 INFO:teuthology.orchestra.run.smithi017.stdout:Installing dependencies: 2024-09-11T09:48:37.186 INFO:teuthology.orchestra.run.smithi017.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-11T09:48:37.186 INFO:teuthology.orchestra.run.smithi017.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-11T09:48:37.186 INFO:teuthology.orchestra.run.smithi017.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-11T09:48:37.186 INFO:teuthology.orchestra.run.smithi017.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-11T09:48:37.186 INFO:teuthology.orchestra.run.smithi017.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-11T09:48:37.186 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-11T09:48:37.186 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction Summary 2024-09-11T09:48:37.186 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2024-09-11T09:48:37.186 INFO:teuthology.orchestra.run.smithi017.stdout:Install 6 Packages 2024-09-11T09:48:37.186 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-11T09:48:37.187 INFO:teuthology.orchestra.run.smithi017.stdout:Total download size: 7.0 M 2024-09-11T09:48:37.187 INFO:teuthology.orchestra.run.smithi017.stdout:Installed size: 33 M 2024-09-11T09:48:37.187 INFO:teuthology.orchestra.run.smithi017.stdout:Downloading Packages: 2024-09-11T09:48:37.461 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2024-09-11T09:48:37.632 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-11T09:48:37.677 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-11T09:48:37.746 INFO:teuthology.orchestra.run.smithi017.stdout:(1/6): ceph-mgr-cephadm-19.1.1-298.gb53b6d3e.el 386 kB/s | 146 kB 00:00 2024-09-11T09:48:37.796 INFO:teuthology.orchestra.run.smithi017.stdout:(2/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 11 MB/s | 548 kB 00:00 2024-09-11T09:48:37.822 INFO:teuthology.orchestra.run.smithi017.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 2.2 MB/s | 58 kB 00:00 2024-09-11T09:48:37.847 INFO:teuthology.orchestra.run.smithi017.stdout:(4/6): python3-typing-extensions-4.12.2-2.el9.n 2.9 MB/s | 74 kB 00:00 2024-09-11T09:48:38.139 INFO:teuthology.orchestra.run.smithi017.stdout:(5/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 322 kB/s | 248 kB 00:00 2024-09-11T09:48:38.205 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-11T09:48:38.288 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-11T09:48:38.352 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-11T09:48:38.381 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-mgr-cephadm-2:19.1.1-298.gb53b6d3e.el9.noarch 6/6 2024-09-11T09:48:38.523 INFO:teuthology.orchestra.run.smithi017.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 5.2 MB/s | 6.0 MB 00:01 2024-09-11T09:48:38.525 INFO:teuthology.orchestra.run.smithi017.stdout:-------------------------------------------------------------------------------- 2024-09-11T09:48:38.526 INFO:teuthology.orchestra.run.smithi017.stdout:Total 5.2 MB/s | 7.0 MB 00:01 2024-09-11T09:48:38.603 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction check 2024-09-11T09:48:38.616 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction check succeeded. 2024-09-11T09:48:38.616 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction test 2024-09-11T09:48:38.722 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction test succeeded. 2024-09-11T09:48:38.722 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction 2024-09-11T09:48:39.040 INFO:teuthology.orchestra.run.smithi017.stdout: Preparing : 1/1 2024-09-11T09:48:39.142 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.1.1-298.gb53b6d3e.el9.noarch 6/6 2024-09-11T09:48:39.142 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-mgr-cephadm-2:19.1.1-298.gb53b6d3e.el9.noarch 1/6 2024-09-11T09:48:39.142 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-11T09:48:39.142 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-11T09:48:39.142 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-11T09:48:39.142 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-11T09:48:39.215 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-11T09:48:39.267 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-11T09:48:39.424 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-11T09:48:39.424 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-11T09:48:39.424 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2024-09-11T09:48:39.424 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-cephadm-2:19.1.1-298.gb53b6d3e.el9.noarch 2024-09-11T09:48:39.424 INFO:teuthology.orchestra.run.smithi107.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-11T09:48:39.424 INFO:teuthology.orchestra.run.smithi107.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-11T09:48:39.424 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-11T09:48:39.424 INFO:teuthology.orchestra.run.smithi107.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-11T09:48:39.424 INFO:teuthology.orchestra.run.smithi107.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-11T09:48:39.425 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-11T09:48:39.425 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-09-11T09:48:39.682 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install ceph-fuse 2024-09-11T09:48:39.818 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-11T09:48:39.898 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-11T09:48:39.977 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-11T09:48:40.003 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : ceph-mgr-cephadm-2:19.1.1-298.gb53b6d3e.el9.noarch 6/6 2024-09-11T09:48:40.250 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:01:27 ago on Wed 11 Sep 2024 09:47:13 AM UTC. 2024-09-11T09:48:40.385 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-09-11T09:48:40.385 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-09-11T09:48:40.385 INFO:teuthology.orchestra.run.smithi107.stdout: Package Arch Version Repository Size 2024-09-11T09:48:40.386 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-09-11T09:48:40.386 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2024-09-11T09:48:40.386 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-fuse x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 897 k 2024-09-11T09:48:40.386 INFO:teuthology.orchestra.run.smithi107.stdout:Installing dependencies: 2024-09-11T09:48:40.386 INFO:teuthology.orchestra.run.smithi107.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-11T09:48:40.386 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-11T09:48:40.386 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2024-09-11T09:48:40.386 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-09-11T09:48:40.386 INFO:teuthology.orchestra.run.smithi107.stdout:Install 2 Packages 2024-09-11T09:48:40.386 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-11T09:48:40.387 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 977 k 2024-09-11T09:48:40.387 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 2.9 M 2024-09-11T09:48:40.387 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2024-09-11T09:48:40.745 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.1.1-298.gb53b6d3e.el9.noarch 6/6 2024-09-11T09:48:40.745 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : ceph-mgr-cephadm-2:19.1.1-298.gb53b6d3e.el9.noarch 1/6 2024-09-11T09:48:40.745 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-11T09:48:40.745 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-11T09:48:40.745 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-11T09:48:40.745 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-11T09:48:40.853 INFO:teuthology.orchestra.run.smithi107.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 433 kB/s | 80 kB 00:00 2024-09-11T09:48:41.032 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-11T09:48:41.032 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-11T09:48:41.032 INFO:teuthology.orchestra.run.smithi017.stdout:Installed: 2024-09-11T09:48:41.032 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-mgr-cephadm-2:19.1.1-298.gb53b6d3e.el9.noarch 2024-09-11T09:48:41.032 INFO:teuthology.orchestra.run.smithi017.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-11T09:48:41.032 INFO:teuthology.orchestra.run.smithi017.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-11T09:48:41.033 INFO:teuthology.orchestra.run.smithi017.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-11T09:48:41.033 INFO:teuthology.orchestra.run.smithi017.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-11T09:48:41.033 INFO:teuthology.orchestra.run.smithi017.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-11T09:48:41.033 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-11T09:48:41.033 INFO:teuthology.orchestra.run.smithi017.stdout:Complete! 2024-09-11T09:48:41.203 INFO:teuthology.orchestra.run.smithi107.stdout:(2/2): ceph-fuse-19.1.1-298.gb53b6d3e.el9.x86_6 1.6 MB/s | 897 kB 00:00 2024-09-11T09:48:41.203 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2024-09-11T09:48:41.204 INFO:teuthology.orchestra.run.smithi107.stdout:Total 1.2 MB/s | 977 kB 00:00 2024-09-11T09:48:41.216 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2024-09-11T09:48:41.240 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2024-09-11T09:48:41.240 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2024-09-11T09:48:41.280 DEBUG:teuthology.orchestra.run.smithi017:> sudo yum -y install ceph-fuse 2024-09-11T09:48:41.322 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2024-09-11T09:48:41.322 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2024-09-11T09:48:41.519 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2024-09-11T09:48:41.655 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-11T09:48:41.719 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-fuse-2:19.1.1-298.gb53b6d3e.el9.x86_64 2/2 2024-09-11T09:48:41.873 INFO:teuthology.orchestra.run.smithi017.stdout:Last metadata expiration check: 0:01:28 ago on Wed 11 Sep 2024 09:47:13 AM UTC. 2024-09-11T09:48:42.013 INFO:teuthology.orchestra.run.smithi017.stdout:Dependencies resolved. 2024-09-11T09:48:42.014 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2024-09-11T09:48:42.014 INFO:teuthology.orchestra.run.smithi017.stdout: Package Arch Version Repository Size 2024-09-11T09:48:42.014 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2024-09-11T09:48:42.014 INFO:teuthology.orchestra.run.smithi017.stdout:Installing: 2024-09-11T09:48:42.014 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-fuse x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 897 k 2024-09-11T09:48:42.014 INFO:teuthology.orchestra.run.smithi017.stdout:Installing dependencies: 2024-09-11T09:48:42.014 INFO:teuthology.orchestra.run.smithi017.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-11T09:48:42.015 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-11T09:48:42.015 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction Summary 2024-09-11T09:48:42.015 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2024-09-11T09:48:42.015 INFO:teuthology.orchestra.run.smithi017.stdout:Install 2 Packages 2024-09-11T09:48:42.015 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-11T09:48:42.015 INFO:teuthology.orchestra.run.smithi017.stdout:Total download size: 977 k 2024-09-11T09:48:42.015 INFO:teuthology.orchestra.run.smithi017.stdout:Installed size: 2.9 M 2024-09-11T09:48:42.015 INFO:teuthology.orchestra.run.smithi017.stdout:Downloading Packages: 2024-09-11T09:48:42.388 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-fuse-2:19.1.1-298.gb53b6d3e.el9.x86_64 2/2 2024-09-11T09:48:42.388 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-fuse-2:19.1.1-298.gb53b6d3e.el9.x86_64 1/2 2024-09-11T09:48:42.433 INFO:teuthology.orchestra.run.smithi017.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 573 kB/s | 80 kB 00:00 2024-09-11T09:48:42.641 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-11T09:48:42.641 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-11T09:48:42.641 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2024-09-11T09:48:42.641 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-fuse-2:19.1.1-298.gb53b6d3e.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-11T09:48:42.642 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-11T09:48:42.642 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-09-11T09:48:42.817 INFO:teuthology.orchestra.run.smithi017.stdout:(2/2): ceph-fuse-19.1.1-298.gb53b6d3e.el9.x86_6 1.7 MB/s | 897 kB 00:00 2024-09-11T09:48:42.817 INFO:teuthology.orchestra.run.smithi017.stdout:-------------------------------------------------------------------------------- 2024-09-11T09:48:42.817 INFO:teuthology.orchestra.run.smithi017.stdout:Total 1.2 MB/s | 977 kB 00:00 2024-09-11T09:48:42.831 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction check 2024-09-11T09:48:42.855 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction check succeeded. 2024-09-11T09:48:42.855 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction test 2024-09-11T09:48:42.936 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install ceph-volume 2024-09-11T09:48:42.943 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction test succeeded. 2024-09-11T09:48:42.943 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction 2024-09-11T09:48:43.138 INFO:teuthology.orchestra.run.smithi017.stdout: Preparing : 1/1 2024-09-11T09:48:43.290 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-11T09:48:43.369 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : ceph-fuse-2:19.1.1-298.gb53b6d3e.el9.x86_64 2/2 2024-09-11T09:48:43.511 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:01:30 ago on Wed 11 Sep 2024 09:47:13 AM UTC. 2024-09-11T09:48:43.647 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-09-11T09:48:43.648 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-09-11T09:48:43.648 INFO:teuthology.orchestra.run.smithi107.stdout: Package Arch Version Repository Size 2024-09-11T09:48:43.648 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-09-11T09:48:43.648 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2024-09-11T09:48:43.648 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-volume noarch 2:19.1.1-298.gb53b6d3e.el9 ceph-noarch 285 k 2024-09-11T09:48:43.648 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-11T09:48:43.649 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2024-09-11T09:48:43.649 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-09-11T09:48:43.649 INFO:teuthology.orchestra.run.smithi107.stdout:Install 1 Package 2024-09-11T09:48:43.649 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-11T09:48:43.649 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 285 k 2024-09-11T09:48:43.649 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 1.3 M 2024-09-11T09:48:43.649 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2024-09-11T09:48:43.883 INFO:teuthology.orchestra.run.smithi107.stdout:ceph-volume-19.1.1-298.gb53b6d3e.el9.noarch.rpm 1.2 MB/s | 285 kB 00:00 2024-09-11T09:48:43.883 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2024-09-11T09:48:43.884 INFO:teuthology.orchestra.run.smithi107.stdout:Total 1.2 MB/s | 285 kB 00:00 2024-09-11T09:48:43.884 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2024-09-11T09:48:43.892 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2024-09-11T09:48:43.892 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2024-09-11T09:48:43.922 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2024-09-11T09:48:43.923 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2024-09-11T09:48:44.068 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: ceph-fuse-2:19.1.1-298.gb53b6d3e.el9.x86_64 2/2 2024-09-11T09:48:44.068 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : ceph-fuse-2:19.1.1-298.gb53b6d3e.el9.x86_64 1/2 2024-09-11T09:48:44.145 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2024-09-11T09:48:44.227 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-volume-2:19.1.1-298.gb53b6d3e.el9.noarch 1/1 2024-09-11T09:48:44.243 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-volume-2:19.1.1-298.gb53b6d3e.el9.noarch 1/1 2024-09-11T09:48:44.243 INFO:teuthology.orchestra.run.smithi107.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-11T09:48:44.244 INFO:teuthology.orchestra.run.smithi107.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-11T09:48:44.244 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-11T09:48:44.374 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-11T09:48:44.374 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-11T09:48:44.374 INFO:teuthology.orchestra.run.smithi017.stdout:Installed: 2024-09-11T09:48:44.374 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-fuse-2:19.1.1-298.gb53b6d3e.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-11T09:48:44.374 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-11T09:48:44.374 INFO:teuthology.orchestra.run.smithi017.stdout:Complete! 2024-09-11T09:48:44.402 INFO:teuthology.orchestra.run.smithi096.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 787 kB/s | 16 MB 00:21 2024-09-11T09:48:44.403 INFO:teuthology.orchestra.run.smithi096.stdout:-------------------------------------------------------------------------------- 2024-09-11T09:48:44.404 INFO:teuthology.orchestra.run.smithi096.stdout:Total 1.7 MB/s | 38 MB 00:23 2024-09-11T09:48:44.649 DEBUG:teuthology.orchestra.run.smithi017:> sudo yum -y install ceph-volume 2024-09-11T09:48:44.762 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction check 2024-09-11T09:48:44.787 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction check succeeded. 2024-09-11T09:48:44.787 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction test 2024-09-11T09:48:45.223 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction test succeeded. 2024-09-11T09:48:45.224 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction 2024-09-11T09:48:45.247 INFO:teuthology.orchestra.run.smithi017.stdout:Last metadata expiration check: 0:01:32 ago on Wed 11 Sep 2024 09:47:13 AM UTC. 2024-09-11T09:48:45.299 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-volume-2:19.1.1-298.gb53b6d3e.el9.noarch 1/1 2024-09-11T09:48:45.299 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-11T09:48:45.299 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2024-09-11T09:48:45.299 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-volume-2:19.1.1-298.gb53b6d3e.el9.noarch 2024-09-11T09:48:45.299 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-11T09:48:45.299 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-09-11T09:48:45.387 INFO:teuthology.orchestra.run.smithi017.stdout:Dependencies resolved. 2024-09-11T09:48:45.387 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2024-09-11T09:48:45.388 INFO:teuthology.orchestra.run.smithi017.stdout: Package Arch Version Repository Size 2024-09-11T09:48:45.388 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2024-09-11T09:48:45.388 INFO:teuthology.orchestra.run.smithi017.stdout:Installing: 2024-09-11T09:48:45.388 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-volume noarch 2:19.1.1-298.gb53b6d3e.el9 ceph-noarch 285 k 2024-09-11T09:48:45.388 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-11T09:48:45.388 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction Summary 2024-09-11T09:48:45.388 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2024-09-11T09:48:45.388 INFO:teuthology.orchestra.run.smithi017.stdout:Install 1 Package 2024-09-11T09:48:45.388 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-11T09:48:45.388 INFO:teuthology.orchestra.run.smithi017.stdout:Total download size: 285 k 2024-09-11T09:48:45.388 INFO:teuthology.orchestra.run.smithi017.stdout:Installed size: 1.3 M 2024-09-11T09:48:45.388 INFO:teuthology.orchestra.run.smithi017.stdout:Downloading Packages: 2024-09-11T09:48:45.610 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install librados-devel 2024-09-11T09:48:45.630 INFO:teuthology.orchestra.run.smithi017.stdout:ceph-volume-19.1.1-298.gb53b6d3e.el9.noarch.rpm 1.2 MB/s | 285 kB 00:00 2024-09-11T09:48:45.630 INFO:teuthology.orchestra.run.smithi017.stdout:-------------------------------------------------------------------------------- 2024-09-11T09:48:45.631 INFO:teuthology.orchestra.run.smithi017.stdout:Total 1.1 MB/s | 285 kB 00:00 2024-09-11T09:48:45.631 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction check 2024-09-11T09:48:45.640 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction check succeeded. 2024-09-11T09:48:45.640 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction test 2024-09-11T09:48:45.669 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction test succeeded. 2024-09-11T09:48:45.670 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction 2024-09-11T09:48:45.904 INFO:teuthology.orchestra.run.smithi017.stdout: Preparing : 1/1 2024-09-11T09:48:45.965 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : ceph-volume-2:19.1.1-298.gb53b6d3e.el9.noarch 1/1 2024-09-11T09:48:45.982 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: ceph-volume-2:19.1.1-298.gb53b6d3e.el9.noarch 1/1 2024-09-11T09:48:45.982 INFO:teuthology.orchestra.run.smithi017.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-11T09:48:45.982 INFO:teuthology.orchestra.run.smithi017.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-11T09:48:45.982 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-11T09:48:46.020 INFO:teuthology.orchestra.run.smithi096.stdout: Preparing : 1/1 2024-09-11T09:48:46.093 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-11T09:48:46.133 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-11T09:48:46.189 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:01:33 ago on Wed 11 Sep 2024 09:47:13 AM UTC. 2024-09-11T09:48:46.232 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-11T09:48:46.285 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-11T09:48:46.327 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-09-11T09:48:46.327 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-09-11T09:48:46.328 INFO:teuthology.orchestra.run.smithi107.stdout: Package Arch Version Repo Size 2024-09-11T09:48:46.328 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-09-11T09:48:46.328 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2024-09-11T09:48:46.328 INFO:teuthology.orchestra.run.smithi107.stdout: librados-devel x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 127 k 2024-09-11T09:48:46.328 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-11T09:48:46.328 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2024-09-11T09:48:46.328 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-09-11T09:48:46.328 INFO:teuthology.orchestra.run.smithi107.stdout:Install 1 Package 2024-09-11T09:48:46.328 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-11T09:48:46.328 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 127 k 2024-09-11T09:48:46.328 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 456 k 2024-09-11T09:48:46.328 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2024-09-11T09:48:46.477 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-11T09:48:46.506 INFO:teuthology.orchestra.run.smithi107.stdout:librados-devel-19.1.1-298.gb53b6d3e.el9.x86_64. 714 kB/s | 127 kB 00:00 2024-09-11T09:48:46.506 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2024-09-11T09:48:46.507 INFO:teuthology.orchestra.run.smithi107.stdout:Total 708 kB/s | 127 kB 00:00 2024-09-11T09:48:46.507 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2024-09-11T09:48:46.509 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-11T09:48:46.512 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2024-09-11T09:48:46.512 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2024-09-11T09:48:46.575 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2024-09-11T09:48:46.576 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2024-09-11T09:48:46.586 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-11T09:48:46.724 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2024-09-11T09:48:46.819 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : librados-devel-2:19.1.1-298.gb53b6d3e.el9.x86_64 1/1 2024-09-11T09:48:46.990 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : ceph-volume-2:19.1.1-298.gb53b6d3e.el9.noarch 1/1 2024-09-11T09:48:46.990 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-11T09:48:46.990 INFO:teuthology.orchestra.run.smithi017.stdout:Installed: 2024-09-11T09:48:46.990 INFO:teuthology.orchestra.run.smithi017.stdout: ceph-volume-2:19.1.1-298.gb53b6d3e.el9.noarch 2024-09-11T09:48:46.990 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-11T09:48:46.990 INFO:teuthology.orchestra.run.smithi017.stdout:Complete! 2024-09-11T09:48:47.231 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-11T09:48:47.245 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: librados-devel-2:19.1.1-298.gb53b6d3e.el9.x86_64 1/1 2024-09-11T09:48:47.278 DEBUG:teuthology.orchestra.run.smithi017:> sudo yum -y install librados-devel 2024-09-11T09:48:47.355 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-11T09:48:47.490 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : librados-devel-2:19.1.1-298.gb53b6d3e.el9.x86_64 1/1 2024-09-11T09:48:47.490 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-11T09:48:47.490 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2024-09-11T09:48:47.491 INFO:teuthology.orchestra.run.smithi107.stdout: librados-devel-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-11T09:48:47.491 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-11T09:48:47.491 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-09-11T09:48:47.757 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install libcephfs2 2024-09-11T09:48:47.875 INFO:teuthology.orchestra.run.smithi017.stdout:Last metadata expiration check: 0:01:34 ago on Wed 11 Sep 2024 09:47:13 AM UTC. 2024-09-11T09:48:48.015 INFO:teuthology.orchestra.run.smithi017.stdout:Dependencies resolved. 2024-09-11T09:48:48.015 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2024-09-11T09:48:48.015 INFO:teuthology.orchestra.run.smithi017.stdout: Package Arch Version Repo Size 2024-09-11T09:48:48.015 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2024-09-11T09:48:48.015 INFO:teuthology.orchestra.run.smithi017.stdout:Installing: 2024-09-11T09:48:48.015 INFO:teuthology.orchestra.run.smithi017.stdout: librados-devel x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 127 k 2024-09-11T09:48:48.016 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-11T09:48:48.016 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction Summary 2024-09-11T09:48:48.016 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2024-09-11T09:48:48.016 INFO:teuthology.orchestra.run.smithi017.stdout:Install 1 Package 2024-09-11T09:48:48.016 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-11T09:48:48.016 INFO:teuthology.orchestra.run.smithi017.stdout:Total download size: 127 k 2024-09-11T09:48:48.016 INFO:teuthology.orchestra.run.smithi017.stdout:Installed size: 456 k 2024-09-11T09:48:48.017 INFO:teuthology.orchestra.run.smithi017.stdout:Downloading Packages: 2024-09-11T09:48:48.211 INFO:teuthology.orchestra.run.smithi017.stdout:librados-devel-19.1.1-298.gb53b6d3e.el9.x86_64. 653 kB/s | 127 kB 00:00 2024-09-11T09:48:48.211 INFO:teuthology.orchestra.run.smithi017.stdout:-------------------------------------------------------------------------------- 2024-09-11T09:48:48.211 INFO:teuthology.orchestra.run.smithi017.stdout:Total 648 kB/s | 127 kB 00:00 2024-09-11T09:48:48.212 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction check 2024-09-11T09:48:48.217 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction check succeeded. 2024-09-11T09:48:48.217 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction test 2024-09-11T09:48:48.285 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction test succeeded. 2024-09-11T09:48:48.285 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction 2024-09-11T09:48:48.338 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:01:35 ago on Wed 11 Sep 2024 09:47:13 AM UTC. 2024-09-11T09:48:48.430 INFO:teuthology.orchestra.run.smithi107.stdout:Package libcephfs2-2:19.1.1-298.gb53b6d3e.el9.x86_64 is already installed. 2024-09-11T09:48:48.444 INFO:teuthology.orchestra.run.smithi017.stdout: Preparing : 1/1 2024-09-11T09:48:48.475 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-09-11T09:48:48.476 INFO:teuthology.orchestra.run.smithi107.stdout:Nothing to do. 2024-09-11T09:48:48.476 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-09-11T09:48:48.536 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : librados-devel-2:19.1.1-298.gb53b6d3e.el9.x86_64 1/1 2024-09-11T09:48:48.547 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install libcephfs-devel 2024-09-11T09:48:48.597 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-11T09:48:48.981 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: librados-devel-2:19.1.1-298.gb53b6d3e.el9.x86_64 1/1 2024-09-11T09:48:49.114 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:01:36 ago on Wed 11 Sep 2024 09:47:13 AM UTC. 2024-09-11T09:48:49.247 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-09-11T09:48:49.248 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-09-11T09:48:49.248 INFO:teuthology.orchestra.run.smithi107.stdout: Package Arch Version Repo Size 2024-09-11T09:48:49.248 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-09-11T09:48:49.249 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2024-09-11T09:48:49.249 INFO:teuthology.orchestra.run.smithi107.stdout: libcephfs-devel x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 32 k 2024-09-11T09:48:49.249 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-11T09:48:49.249 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2024-09-11T09:48:49.249 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-09-11T09:48:49.249 INFO:teuthology.orchestra.run.smithi107.stdout:Install 1 Package 2024-09-11T09:48:49.249 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-11T09:48:49.249 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 32 k 2024-09-11T09:48:49.249 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 142 k 2024-09-11T09:48:49.249 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2024-09-11T09:48:49.272 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : librados-devel-2:19.1.1-298.gb53b6d3e.el9.x86_64 1/1 2024-09-11T09:48:49.272 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-11T09:48:49.273 INFO:teuthology.orchestra.run.smithi017.stdout:Installed: 2024-09-11T09:48:49.273 INFO:teuthology.orchestra.run.smithi017.stdout: librados-devel-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-11T09:48:49.273 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-11T09:48:49.273 INFO:teuthology.orchestra.run.smithi017.stdout:Complete! 2024-09-11T09:48:49.401 INFO:teuthology.orchestra.run.smithi107.stdout:libcephfs-devel-19.1.1-298.gb53b6d3e.el9.x86_64 208 kB/s | 32 kB 00:00 2024-09-11T09:48:49.402 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2024-09-11T09:48:49.402 INFO:teuthology.orchestra.run.smithi107.stdout:Total 206 kB/s | 32 kB 00:00 2024-09-11T09:48:49.402 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2024-09-11T09:48:49.406 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2024-09-11T09:48:49.406 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2024-09-11T09:48:49.418 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2024-09-11T09:48:49.418 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2024-09-11T09:48:49.507 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2024-09-11T09:48:49.512 DEBUG:teuthology.orchestra.run.smithi017:> sudo yum -y install libcephfs2 2024-09-11T09:48:49.606 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libcephfs-devel-2:19.1.1-298.gb53b6d3e.el9.x86_64 1/1 2024-09-11T09:48:49.773 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-11T09:48:49.809 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-mgr-diskprediction-local-2:19.1.1-298.gb53b6d 12/12 2024-09-11T09:48:49.947 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: libcephfs-devel-2:19.1.1-298.gb53b6d3e.el9.x86_64 1/1 2024-09-11T09:48:50.109 INFO:teuthology.orchestra.run.smithi017.stdout:Last metadata expiration check: 0:01:37 ago on Wed 11 Sep 2024 09:47:13 AM UTC. 2024-09-11T09:48:50.179 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libcephfs-devel-2:19.1.1-298.gb53b6d3e.el9.x86_64 1/1 2024-09-11T09:48:50.179 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-11T09:48:50.180 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2024-09-11T09:48:50.180 INFO:teuthology.orchestra.run.smithi107.stdout: libcephfs-devel-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-11T09:48:50.180 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-11T09:48:50.180 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-09-11T09:48:50.204 INFO:teuthology.orchestra.run.smithi017.stdout:Package libcephfs2-2:19.1.1-298.gb53b6d3e.el9.x86_64 is already installed. 2024-09-11T09:48:50.250 INFO:teuthology.orchestra.run.smithi017.stdout:Dependencies resolved. 2024-09-11T09:48:50.251 INFO:teuthology.orchestra.run.smithi017.stdout:Nothing to do. 2024-09-11T09:48:50.251 INFO:teuthology.orchestra.run.smithi017.stdout:Complete! 2024-09-11T09:48:50.321 DEBUG:teuthology.orchestra.run.smithi017:> sudo yum -y install libcephfs-devel 2024-09-11T09:48:50.390 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install librados2 2024-09-11T09:48:50.906 INFO:teuthology.orchestra.run.smithi017.stdout:Last metadata expiration check: 0:01:37 ago on Wed 11 Sep 2024 09:47:13 AM UTC. 2024-09-11T09:48:50.946 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:01:37 ago on Wed 11 Sep 2024 09:47:13 AM UTC. 2024-09-11T09:48:51.037 INFO:teuthology.orchestra.run.smithi107.stdout:Package librados2-2:19.1.1-298.gb53b6d3e.el9.x86_64 is already installed. 2024-09-11T09:48:51.045 INFO:teuthology.orchestra.run.smithi017.stdout:Dependencies resolved. 2024-09-11T09:48:51.046 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2024-09-11T09:48:51.046 INFO:teuthology.orchestra.run.smithi017.stdout: Package Arch Version Repo Size 2024-09-11T09:48:51.046 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2024-09-11T09:48:51.046 INFO:teuthology.orchestra.run.smithi017.stdout:Installing: 2024-09-11T09:48:51.046 INFO:teuthology.orchestra.run.smithi017.stdout: libcephfs-devel x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 32 k 2024-09-11T09:48:51.046 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-11T09:48:51.046 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction Summary 2024-09-11T09:48:51.047 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2024-09-11T09:48:51.047 INFO:teuthology.orchestra.run.smithi017.stdout:Install 1 Package 2024-09-11T09:48:51.047 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-11T09:48:51.047 INFO:teuthology.orchestra.run.smithi017.stdout:Total download size: 32 k 2024-09-11T09:48:51.047 INFO:teuthology.orchestra.run.smithi017.stdout:Installed size: 142 k 2024-09-11T09:48:51.047 INFO:teuthology.orchestra.run.smithi017.stdout:Downloading Packages: 2024-09-11T09:48:51.080 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-09-11T09:48:51.081 INFO:teuthology.orchestra.run.smithi107.stdout:Nothing to do. 2024-09-11T09:48:51.081 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-09-11T09:48:51.145 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install librbd1 2024-09-11T09:48:51.197 INFO:teuthology.orchestra.run.smithi017.stdout:libcephfs-devel-19.1.1-298.gb53b6d3e.el9.x86_64 211 kB/s | 32 kB 00:00 2024-09-11T09:48:51.198 INFO:teuthology.orchestra.run.smithi017.stdout:-------------------------------------------------------------------------------- 2024-09-11T09:48:51.198 INFO:teuthology.orchestra.run.smithi017.stdout:Total 209 kB/s | 32 kB 00:00 2024-09-11T09:48:51.198 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction check 2024-09-11T09:48:51.202 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction check succeeded. 2024-09-11T09:48:51.202 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction test 2024-09-11T09:48:51.219 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction test succeeded. 2024-09-11T09:48:51.219 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction 2024-09-11T09:48:51.314 INFO:teuthology.orchestra.run.smithi017.stdout: Preparing : 1/1 2024-09-11T09:48:51.423 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : libcephfs-devel-2:19.1.1-298.gb53b6d3e.el9.x86_64 1/1 2024-09-11T09:48:51.704 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:01:38 ago on Wed 11 Sep 2024 09:47:13 AM UTC. 2024-09-11T09:48:51.795 INFO:teuthology.orchestra.run.smithi107.stdout:Package librbd1-2:19.1.1-298.gb53b6d3e.el9.x86_64 is already installed. 2024-09-11T09:48:51.806 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: libcephfs-devel-2:19.1.1-298.gb53b6d3e.el9.x86_64 1/1 2024-09-11T09:48:51.838 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-09-11T09:48:51.839 INFO:teuthology.orchestra.run.smithi107.stdout:Nothing to do. 2024-09-11T09:48:51.839 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-09-11T09:48:51.912 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install python3-rados 2024-09-11T09:48:52.048 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : libcephfs-devel-2:19.1.1-298.gb53b6d3e.el9.x86_64 1/1 2024-09-11T09:48:52.048 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-11T09:48:52.048 INFO:teuthology.orchestra.run.smithi017.stdout:Installed: 2024-09-11T09:48:52.048 INFO:teuthology.orchestra.run.smithi017.stdout: libcephfs-devel-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-11T09:48:52.048 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-11T09:48:52.048 INFO:teuthology.orchestra.run.smithi017.stdout:Complete! 2024-09-11T09:48:52.232 DEBUG:teuthology.orchestra.run.smithi017:> sudo yum -y install librados2 2024-09-11T09:48:52.335 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.1.1-298.gb53b6d 12/12 2024-09-11T09:48:52.335 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.1.1-298.gb53b6d 1/12 2024-09-11T09:48:52.335 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-11T09:48:52.335 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-11T09:48:52.335 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-11T09:48:52.335 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-11T09:48:52.336 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-11T09:48:52.336 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-11T09:48:52.336 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-11T09:48:52.336 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-11T09:48:52.336 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-11T09:48:52.336 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-11T09:48:52.470 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:01:39 ago on Wed 11 Sep 2024 09:47:13 AM UTC. 2024-09-11T09:48:52.561 INFO:teuthology.orchestra.run.smithi107.stdout:Package python3-rados-2:19.1.1-298.gb53b6d3e.el9.x86_64 is already installed. 2024-09-11T09:48:52.603 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-09-11T09:48:52.604 INFO:teuthology.orchestra.run.smithi107.stdout:Nothing to do. 2024-09-11T09:48:52.604 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-09-11T09:48:52.678 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install python3-rgw 2024-09-11T09:48:52.773 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-11T09:48:52.773 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-11T09:48:52.773 INFO:teuthology.orchestra.run.smithi096.stdout:Installed: 2024-09-11T09:48:52.773 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mgr-diskprediction-local-2:19.1.1-298.gb53b6d3e.el9.noarch 2024-09-11T09:48:52.773 INFO:teuthology.orchestra.run.smithi096.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-11T09:48:52.774 INFO:teuthology.orchestra.run.smithi096.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-11T09:48:52.774 INFO:teuthology.orchestra.run.smithi096.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-11T09:48:52.774 INFO:teuthology.orchestra.run.smithi096.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-11T09:48:52.774 INFO:teuthology.orchestra.run.smithi096.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-11T09:48:52.774 INFO:teuthology.orchestra.run.smithi096.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-11T09:48:52.774 INFO:teuthology.orchestra.run.smithi096.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-11T09:48:52.774 INFO:teuthology.orchestra.run.smithi096.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-11T09:48:52.774 INFO:teuthology.orchestra.run.smithi096.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-11T09:48:52.774 INFO:teuthology.orchestra.run.smithi096.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-11T09:48:52.774 INFO:teuthology.orchestra.run.smithi096.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-11T09:48:52.774 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-11T09:48:52.774 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2024-09-11T09:48:52.826 INFO:teuthology.orchestra.run.smithi017.stdout:Last metadata expiration check: 0:01:39 ago on Wed 11 Sep 2024 09:47:13 AM UTC. 2024-09-11T09:48:52.921 INFO:teuthology.orchestra.run.smithi017.stdout:Package librados2-2:19.1.1-298.gb53b6d3e.el9.x86_64 is already installed. 2024-09-11T09:48:52.967 INFO:teuthology.orchestra.run.smithi017.stdout:Dependencies resolved. 2024-09-11T09:48:52.968 INFO:teuthology.orchestra.run.smithi017.stdout:Nothing to do. 2024-09-11T09:48:52.968 INFO:teuthology.orchestra.run.smithi017.stdout:Complete! 2024-09-11T09:48:53.044 DEBUG:teuthology.orchestra.run.smithi017:> sudo yum -y install librbd1 2024-09-11T09:48:53.177 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum -y install ceph-mgr-rook 2024-09-11T09:48:53.242 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:01:40 ago on Wed 11 Sep 2024 09:47:13 AM UTC. 2024-09-11T09:48:53.333 INFO:teuthology.orchestra.run.smithi107.stdout:Package python3-rgw-2:19.1.1-298.gb53b6d3e.el9.x86_64 is already installed. 2024-09-11T09:48:53.376 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-09-11T09:48:53.377 INFO:teuthology.orchestra.run.smithi107.stdout:Nothing to do. 2024-09-11T09:48:53.377 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-09-11T09:48:53.447 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install python3-cephfs 2024-09-11T09:48:53.630 INFO:teuthology.orchestra.run.smithi017.stdout:Last metadata expiration check: 0:01:40 ago on Wed 11 Sep 2024 09:47:13 AM UTC. 2024-09-11T09:48:53.724 INFO:teuthology.orchestra.run.smithi017.stdout:Package librbd1-2:19.1.1-298.gb53b6d3e.el9.x86_64 is already installed. 2024-09-11T09:48:53.749 INFO:teuthology.orchestra.run.smithi096.stdout:Last metadata expiration check: 0:01:41 ago on Wed 11 Sep 2024 09:47:12 AM UTC. 2024-09-11T09:48:53.771 INFO:teuthology.orchestra.run.smithi017.stdout:Dependencies resolved. 2024-09-11T09:48:53.772 INFO:teuthology.orchestra.run.smithi017.stdout:Nothing to do. 2024-09-11T09:48:53.772 INFO:teuthology.orchestra.run.smithi017.stdout:Complete! 2024-09-11T09:48:53.844 DEBUG:teuthology.orchestra.run.smithi017:> sudo yum -y install python3-rados 2024-09-11T09:48:53.883 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2024-09-11T09:48:53.884 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-09-11T09:48:53.884 INFO:teuthology.orchestra.run.smithi096.stdout: Package Arch Version Repository Size 2024-09-11T09:48:53.885 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-09-11T09:48:53.885 INFO:teuthology.orchestra.run.smithi096.stdout:Installing: 2024-09-11T09:48:53.885 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mgr-rook noarch 2:19.1.1-298.gb53b6d3e.el9 ceph-noarch 49 k 2024-09-11T09:48:53.885 INFO:teuthology.orchestra.run.smithi096.stdout:Installing dependencies: 2024-09-11T09:48:53.885 INFO:teuthology.orchestra.run.smithi096.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-11T09:48:53.885 INFO:teuthology.orchestra.run.smithi096.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-11T09:48:53.885 INFO:teuthology.orchestra.run.smithi096.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-11T09:48:53.885 INFO:teuthology.orchestra.run.smithi096.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-11T09:48:53.885 INFO:teuthology.orchestra.run.smithi096.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-11T09:48:53.885 INFO:teuthology.orchestra.run.smithi096.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-11T09:48:53.885 INFO:teuthology.orchestra.run.smithi096.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-11T09:48:53.885 INFO:teuthology.orchestra.run.smithi096.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-11T09:48:53.886 INFO:teuthology.orchestra.run.smithi096.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-11T09:48:53.886 INFO:teuthology.orchestra.run.smithi096.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-11T09:48:53.886 INFO:teuthology.orchestra.run.smithi096.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-11T09:48:53.886 INFO:teuthology.orchestra.run.smithi096.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-11T09:48:53.886 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-11T09:48:53.886 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction Summary 2024-09-11T09:48:53.886 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-09-11T09:48:53.886 INFO:teuthology.orchestra.run.smithi096.stdout:Install 13 Packages 2024-09-11T09:48:53.886 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-11T09:48:53.887 INFO:teuthology.orchestra.run.smithi096.stdout:Total download size: 2.2 M 2024-09-11T09:48:53.887 INFO:teuthology.orchestra.run.smithi096.stdout:Installed size: 26 M 2024-09-11T09:48:53.887 INFO:teuthology.orchestra.run.smithi096.stdout:Downloading Packages: 2024-09-11T09:48:54.008 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:01:41 ago on Wed 11 Sep 2024 09:47:13 AM UTC. 2024-09-11T09:48:54.098 INFO:teuthology.orchestra.run.smithi107.stdout:Package python3-cephfs-2:19.1.1-298.gb53b6d3e.el9.x86_64 is already installed. 2024-09-11T09:48:54.141 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-09-11T09:48:54.141 INFO:teuthology.orchestra.run.smithi107.stdout:Nothing to do. 2024-09-11T09:48:54.141 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-09-11T09:48:54.215 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install python3-rbd 2024-09-11T09:48:54.434 INFO:teuthology.orchestra.run.smithi017.stdout:Last metadata expiration check: 0:01:41 ago on Wed 11 Sep 2024 09:47:13 AM UTC. 2024-09-11T09:48:54.470 INFO:teuthology.orchestra.run.smithi096.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 224 kB/s | 19 kB 00:00 2024-09-11T09:48:54.495 INFO:teuthology.orchestra.run.smithi096.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 231 kB/s | 26 kB 00:00 2024-09-11T09:48:54.531 INFO:teuthology.orchestra.run.smithi017.stdout:Package python3-rados-2:19.1.1-298.gb53b6d3e.el9.x86_64 is already installed. 2024-09-11T09:48:54.557 INFO:teuthology.orchestra.run.smithi096.stdout:(3/13): ceph-mgr-rook-19.1.1-298.gb53b6d3e.el9. 290 kB/s | 49 kB 00:00 2024-09-11T09:48:54.576 INFO:teuthology.orchestra.run.smithi017.stdout:Dependencies resolved. 2024-09-11T09:48:54.577 INFO:teuthology.orchestra.run.smithi017.stdout:Nothing to do. 2024-09-11T09:48:54.577 INFO:teuthology.orchestra.run.smithi017.stdout:Complete! 2024-09-11T09:48:54.579 INFO:teuthology.orchestra.run.smithi096.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.0 MB/s | 222 kB 00:00 2024-09-11T09:48:54.613 INFO:teuthology.orchestra.run.smithi096.stdout:(5/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.3 MB/s | 159 kB 00:00 2024-09-11T09:48:54.655 INFO:teuthology.orchestra.run.smithi096.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 714 kB/s | 54 kB 00:00 2024-09-11T09:48:54.697 INFO:teuthology.orchestra.run.smithi096.stdout:(7/13): python3-cachetools-4.2.4-1.el9.noarch.r 386 kB/s | 32 kB 00:00 2024-09-11T09:48:54.722 INFO:teuthology.orchestra.run.smithi096.stdout:(8/13): python3-certifi-2023.05.07-4.el9.noarch 211 kB/s | 14 kB 00:00 2024-09-11T09:48:54.748 INFO:teuthology.orchestra.run.smithi096.stdout:(9/13): python3-pyasn1-modules-0.4.8-6.el9.noar 1.4 MB/s | 279 kB 00:00 2024-09-11T09:48:54.775 DEBUG:teuthology.orchestra.run.smithi017:> sudo yum -y install python3-rgw 2024-09-11T09:48:54.779 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:01:41 ago on Wed 11 Sep 2024 09:47:13 AM UTC. 2024-09-11T09:48:54.781 INFO:teuthology.orchestra.run.smithi096.stdout:(10/13): python3-google-auth-2.34.0-1.el9.noarc 2.6 MB/s | 220 kB 00:00 2024-09-11T09:48:54.807 INFO:teuthology.orchestra.run.smithi096.stdout:(11/13): python3-websocket-client-1.2.3-2.el9.n 3.8 MB/s | 90 kB 00:00 2024-09-11T09:48:54.832 INFO:teuthology.orchestra.run.smithi096.stdout:(12/13): python3-rsa-4.9-2.el9.noarch.rpm 707 kB/s | 59 kB 00:00 2024-09-11T09:48:54.870 INFO:teuthology.orchestra.run.smithi107.stdout:Package python3-rbd-2:19.1.1-298.gb53b6d3e.el9.x86_64 is already installed. 2024-09-11T09:48:54.873 INFO:teuthology.orchestra.run.smithi096.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 6.8 MB/s | 1.0 MB 00:00 2024-09-11T09:48:54.877 INFO:teuthology.orchestra.run.smithi096.stdout:-------------------------------------------------------------------------------- 2024-09-11T09:48:54.877 INFO:teuthology.orchestra.run.smithi096.stdout:Total 2.2 MB/s | 2.2 MB 00:00 2024-09-11T09:48:54.913 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-09-11T09:48:54.914 INFO:teuthology.orchestra.run.smithi107.stdout:Nothing to do. 2024-09-11T09:48:54.914 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-09-11T09:48:54.948 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction check 2024-09-11T09:48:54.964 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction check succeeded. 2024-09-11T09:48:54.964 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction test 2024-09-11T09:48:54.986 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install rbd-fuse 2024-09-11T09:48:55.055 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction test succeeded. 2024-09-11T09:48:55.055 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction 2024-09-11T09:48:55.373 INFO:teuthology.orchestra.run.smithi017.stdout:Last metadata expiration check: 0:01:42 ago on Wed 11 Sep 2024 09:47:13 AM UTC. 2024-09-11T09:48:55.467 INFO:teuthology.orchestra.run.smithi017.stdout:Package python3-rgw-2:19.1.1-298.gb53b6d3e.el9.x86_64 is already installed. 2024-09-11T09:48:55.513 INFO:teuthology.orchestra.run.smithi017.stdout:Dependencies resolved. 2024-09-11T09:48:55.515 INFO:teuthology.orchestra.run.smithi017.stdout:Nothing to do. 2024-09-11T09:48:55.515 INFO:teuthology.orchestra.run.smithi017.stdout:Complete! 2024-09-11T09:48:55.548 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:01:42 ago on Wed 11 Sep 2024 09:47:13 AM UTC. 2024-09-11T09:48:55.560 INFO:teuthology.orchestra.run.smithi096.stdout: Preparing : 1/1 2024-09-11T09:48:55.586 DEBUG:teuthology.orchestra.run.smithi017:> sudo yum -y install python3-cephfs 2024-09-11T09:48:55.681 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-09-11T09:48:55.682 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-09-11T09:48:55.682 INFO:teuthology.orchestra.run.smithi107.stdout: Package Architecture Version Repository Size 2024-09-11T09:48:55.682 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-09-11T09:48:55.682 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2024-09-11T09:48:55.683 INFO:teuthology.orchestra.run.smithi107.stdout: rbd-fuse x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 85 k 2024-09-11T09:48:55.683 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-11T09:48:55.683 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2024-09-11T09:48:55.683 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-09-11T09:48:55.683 INFO:teuthology.orchestra.run.smithi107.stdout:Install 1 Package 2024-09-11T09:48:55.683 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-11T09:48:55.683 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 85 k 2024-09-11T09:48:55.683 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 231 k 2024-09-11T09:48:55.683 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2024-09-11T09:48:55.736 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-11T09:48:55.791 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-11T09:48:55.855 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-11T09:48:55.862 INFO:teuthology.orchestra.run.smithi107.stdout:rbd-fuse-19.1.1-298.gb53b6d3e.el9.x86_64.rpm 475 kB/s | 85 kB 00:00 2024-09-11T09:48:55.863 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2024-09-11T09:48:55.863 INFO:teuthology.orchestra.run.smithi107.stdout:Total 472 kB/s | 85 kB 00:00 2024-09-11T09:48:55.863 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2024-09-11T09:48:55.870 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2024-09-11T09:48:55.870 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2024-09-11T09:48:55.892 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-11T09:48:55.932 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2024-09-11T09:48:55.933 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2024-09-11T09:48:55.933 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-11T09:48:56.034 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-11T09:48:56.075 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2024-09-11T09:48:56.136 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-11T09:48:56.176 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : rbd-fuse-2:19.1.1-298.gb53b6d3e.el9.x86_64 1/1 2024-09-11T09:48:56.178 INFO:teuthology.orchestra.run.smithi017.stdout:Last metadata expiration check: 0:01:43 ago on Wed 11 Sep 2024 09:47:13 AM UTC. 2024-09-11T09:48:56.192 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-11T09:48:56.273 INFO:teuthology.orchestra.run.smithi017.stdout:Package python3-cephfs-2:19.1.1-298.gb53b6d3e.el9.x86_64 is already installed. 2024-09-11T09:48:56.319 INFO:teuthology.orchestra.run.smithi017.stdout:Dependencies resolved. 2024-09-11T09:48:56.321 INFO:teuthology.orchestra.run.smithi017.stdout:Nothing to do. 2024-09-11T09:48:56.321 INFO:teuthology.orchestra.run.smithi017.stdout:Complete! 2024-09-11T09:48:56.395 DEBUG:teuthology.orchestra.run.smithi017:> sudo yum -y install python3-rbd 2024-09-11T09:48:56.557 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: rbd-fuse-2:19.1.1-298.gb53b6d3e.el9.x86_64 1/1 2024-09-11T09:48:56.867 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : rbd-fuse-2:19.1.1-298.gb53b6d3e.el9.x86_64 1/1 2024-09-11T09:48:56.867 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-11T09:48:56.867 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2024-09-11T09:48:56.867 INFO:teuthology.orchestra.run.smithi107.stdout: rbd-fuse-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-11T09:48:56.867 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-11T09:48:56.867 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-09-11T09:48:56.898 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-11T09:48:56.942 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-11T09:48:56.975 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-11T09:48:56.986 INFO:teuthology.orchestra.run.smithi017.stdout:Last metadata expiration check: 0:01:43 ago on Wed 11 Sep 2024 09:47:13 AM UTC. 2024-09-11T09:48:57.042 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-11T09:48:57.070 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-mgr-rook-2:19.1.1-298.gb53b6d3e.el9.noarch 13/13 2024-09-11T09:48:57.082 INFO:teuthology.orchestra.run.smithi017.stdout:Package python3-rbd-2:19.1.1-298.gb53b6d3e.el9.x86_64 is already installed. 2024-09-11T09:48:57.129 INFO:teuthology.orchestra.run.smithi017.stdout:Dependencies resolved. 2024-09-11T09:48:57.130 INFO:teuthology.orchestra.run.smithi017.stdout:Nothing to do. 2024-09-11T09:48:57.130 INFO:teuthology.orchestra.run.smithi017.stdout:Complete! 2024-09-11T09:48:57.138 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install rbd-mirror 2024-09-11T09:48:57.199 DEBUG:teuthology.orchestra.run.smithi017:> sudo yum -y install rbd-fuse 2024-09-11T09:48:57.728 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:01:44 ago on Wed 11 Sep 2024 09:47:13 AM UTC. 2024-09-11T09:48:57.799 INFO:teuthology.orchestra.run.smithi017.stdout:Last metadata expiration check: 0:01:44 ago on Wed 11 Sep 2024 09:47:13 AM UTC. 2024-09-11T09:48:57.825 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-mgr-rook-2:19.1.1-298.gb53b6d3e.el9.noarch 13/13 2024-09-11T09:48:57.825 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-mgr-rook-2:19.1.1-298.gb53b6d3e.el9.noarch 1/13 2024-09-11T09:48:57.825 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-11T09:48:57.826 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-11T09:48:57.826 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-11T09:48:57.826 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-11T09:48:57.826 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-11T09:48:57.826 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-11T09:48:57.826 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-11T09:48:57.826 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-11T09:48:57.826 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-11T09:48:57.826 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-11T09:48:57.826 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-11T09:48:57.871 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-09-11T09:48:57.872 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-09-11T09:48:57.872 INFO:teuthology.orchestra.run.smithi107.stdout: Package Arch Version Repo Size 2024-09-11T09:48:57.872 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-09-11T09:48:57.872 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2024-09-11T09:48:57.872 INFO:teuthology.orchestra.run.smithi107.stdout: rbd-mirror x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 3.1 M 2024-09-11T09:48:57.872 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-11T09:48:57.873 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2024-09-11T09:48:57.873 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-09-11T09:48:57.873 INFO:teuthology.orchestra.run.smithi107.stdout:Install 1 Package 2024-09-11T09:48:57.873 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-11T09:48:57.873 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 3.1 M 2024-09-11T09:48:57.873 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 13 M 2024-09-11T09:48:57.873 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2024-09-11T09:48:57.941 INFO:teuthology.orchestra.run.smithi017.stdout:Dependencies resolved. 2024-09-11T09:48:57.942 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2024-09-11T09:48:57.942 INFO:teuthology.orchestra.run.smithi017.stdout: Package Architecture Version Repository Size 2024-09-11T09:48:57.942 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2024-09-11T09:48:57.942 INFO:teuthology.orchestra.run.smithi017.stdout:Installing: 2024-09-11T09:48:57.942 INFO:teuthology.orchestra.run.smithi017.stdout: rbd-fuse x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 85 k 2024-09-11T09:48:57.942 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-11T09:48:57.942 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction Summary 2024-09-11T09:48:57.942 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2024-09-11T09:48:57.942 INFO:teuthology.orchestra.run.smithi017.stdout:Install 1 Package 2024-09-11T09:48:57.942 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-11T09:48:57.943 INFO:teuthology.orchestra.run.smithi017.stdout:Total download size: 85 k 2024-09-11T09:48:57.943 INFO:teuthology.orchestra.run.smithi017.stdout:Installed size: 231 k 2024-09-11T09:48:57.943 INFO:teuthology.orchestra.run.smithi017.stdout:Downloading Packages: 2024-09-11T09:48:58.143 INFO:teuthology.orchestra.run.smithi017.stdout:rbd-fuse-19.1.1-298.gb53b6d3e.el9.x86_64.rpm 426 kB/s | 85 kB 00:00 2024-09-11T09:48:58.144 INFO:teuthology.orchestra.run.smithi017.stdout:-------------------------------------------------------------------------------- 2024-09-11T09:48:58.144 INFO:teuthology.orchestra.run.smithi017.stdout:Total 421 kB/s | 85 kB 00:00 2024-09-11T09:48:58.145 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction check 2024-09-11T09:48:58.152 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-11T09:48:58.152 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-11T09:48:58.152 INFO:teuthology.orchestra.run.smithi096.stdout:Installed: 2024-09-11T09:48:58.152 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mgr-rook-2:19.1.1-298.gb53b6d3e.el9.noarch 2024-09-11T09:48:58.152 INFO:teuthology.orchestra.run.smithi096.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-11T09:48:58.152 INFO:teuthology.orchestra.run.smithi096.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-11T09:48:58.152 INFO:teuthology.orchestra.run.smithi096.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-11T09:48:58.152 INFO:teuthology.orchestra.run.smithi096.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-11T09:48:58.152 INFO:teuthology.orchestra.run.smithi096.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-11T09:48:58.152 INFO:teuthology.orchestra.run.smithi096.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-11T09:48:58.152 INFO:teuthology.orchestra.run.smithi096.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-11T09:48:58.153 INFO:teuthology.orchestra.run.smithi096.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-11T09:48:58.153 INFO:teuthology.orchestra.run.smithi096.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-11T09:48:58.153 INFO:teuthology.orchestra.run.smithi096.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-11T09:48:58.153 INFO:teuthology.orchestra.run.smithi096.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-11T09:48:58.153 INFO:teuthology.orchestra.run.smithi096.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-11T09:48:58.153 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-11T09:48:58.153 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2024-09-11T09:48:58.158 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction check succeeded. 2024-09-11T09:48:58.158 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction test 2024-09-11T09:48:58.192 INFO:teuthology.orchestra.run.smithi107.stdout:rbd-mirror-19.1.1-298.gb53b6d3e.el9.x86_64.rpm 9.7 MB/s | 3.1 MB 00:00 2024-09-11T09:48:58.193 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2024-09-11T09:48:58.193 INFO:teuthology.orchestra.run.smithi107.stdout:Total 9.7 MB/s | 3.1 MB 00:00 2024-09-11T09:48:58.193 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2024-09-11T09:48:58.202 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2024-09-11T09:48:58.202 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2024-09-11T09:48:58.230 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction test succeeded. 2024-09-11T09:48:58.231 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction 2024-09-11T09:48:58.303 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2024-09-11T09:48:58.303 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2024-09-11T09:48:58.380 INFO:teuthology.orchestra.run.smithi017.stdout: Preparing : 1/1 2024-09-11T09:48:58.414 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum -y install ceph-mgr-cephadm 2024-09-11T09:48:58.494 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : rbd-fuse-2:19.1.1-298.gb53b6d3e.el9.x86_64 1/1 2024-09-11T09:48:58.739 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2024-09-11T09:48:58.793 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : rbd-mirror-2:19.1.1-298.gb53b6d3e.el9.x86_64 1/1 2024-09-11T09:48:58.819 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: rbd-mirror-2:19.1.1-298.gb53b6d3e.el9.x86_64 1/1 2024-09-11T09:48:58.819 INFO:teuthology.orchestra.run.smithi107.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-11T09:48:58.819 INFO:teuthology.orchestra.run.smithi107.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-11T09:48:58.819 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-11T09:48:58.819 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-11T09:48:58.820 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-11T09:48:58.912 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: rbd-fuse-2:19.1.1-298.gb53b6d3e.el9.x86_64 1/1 2024-09-11T09:48:58.975 INFO:teuthology.orchestra.run.smithi096.stdout:Last metadata expiration check: 0:01:46 ago on Wed 11 Sep 2024 09:47:12 AM UTC. 2024-09-11T09:48:59.107 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2024-09-11T09:48:59.108 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-09-11T09:48:59.108 INFO:teuthology.orchestra.run.smithi096.stdout: Package Arch Version Repository Size 2024-09-11T09:48:59.108 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-09-11T09:48:59.108 INFO:teuthology.orchestra.run.smithi096.stdout:Installing: 2024-09-11T09:48:59.108 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mgr-cephadm noarch 2:19.1.1-298.gb53b6d3e.el9 ceph-noarch 146 k 2024-09-11T09:48:59.108 INFO:teuthology.orchestra.run.smithi096.stdout:Installing dependencies: 2024-09-11T09:48:59.108 INFO:teuthology.orchestra.run.smithi096.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-11T09:48:59.108 INFO:teuthology.orchestra.run.smithi096.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-11T09:48:59.108 INFO:teuthology.orchestra.run.smithi096.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-11T09:48:59.109 INFO:teuthology.orchestra.run.smithi096.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-11T09:48:59.109 INFO:teuthology.orchestra.run.smithi096.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-11T09:48:59.109 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-11T09:48:59.109 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction Summary 2024-09-11T09:48:59.109 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-09-11T09:48:59.109 INFO:teuthology.orchestra.run.smithi096.stdout:Install 6 Packages 2024-09-11T09:48:59.109 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-11T09:48:59.110 INFO:teuthology.orchestra.run.smithi096.stdout:Total download size: 7.0 M 2024-09-11T09:48:59.110 INFO:teuthology.orchestra.run.smithi096.stdout:Installed size: 33 M 2024-09-11T09:48:59.110 INFO:teuthology.orchestra.run.smithi096.stdout:Downloading Packages: 2024-09-11T09:48:59.189 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : rbd-fuse-2:19.1.1-298.gb53b6d3e.el9.x86_64 1/1 2024-09-11T09:48:59.189 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-11T09:48:59.189 INFO:teuthology.orchestra.run.smithi017.stdout:Installed: 2024-09-11T09:48:59.189 INFO:teuthology.orchestra.run.smithi017.stdout: rbd-fuse-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-11T09:48:59.189 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-11T09:48:59.189 INFO:teuthology.orchestra.run.smithi017.stdout:Complete! 2024-09-11T09:48:59.434 DEBUG:teuthology.orchestra.run.smithi017:> sudo yum -y install rbd-mirror 2024-09-11T09:48:59.460 INFO:teuthology.orchestra.run.smithi096.stdout:(1/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 1.6 MB/s | 248 kB 00:00 2024-09-11T09:48:59.510 INFO:teuthology.orchestra.run.smithi096.stdout:(2/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 11 MB/s | 548 kB 00:00 2024-09-11T09:48:59.544 INFO:teuthology.orchestra.run.smithi096.stdout:(3/6): ceph-mgr-cephadm-19.1.1-298.gb53b6d3e.el 610 kB/s | 146 kB 00:00 2024-09-11T09:48:59.569 INFO:teuthology.orchestra.run.smithi096.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 985 kB/s | 58 kB 00:00 2024-09-11T09:48:59.611 INFO:teuthology.orchestra.run.smithi096.stdout:(5/6): python3-typing-extensions-4.12.2-2.el9.n 1.1 MB/s | 74 kB 00:00 2024-09-11T09:48:59.695 INFO:teuthology.orchestra.run.smithi096.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 15 MB/s | 6.0 MB 00:00 2024-09-11T09:48:59.697 INFO:teuthology.orchestra.run.smithi096.stdout:-------------------------------------------------------------------------------- 2024-09-11T09:48:59.697 INFO:teuthology.orchestra.run.smithi096.stdout:Total 12 MB/s | 7.0 MB 00:00 2024-09-11T09:48:59.768 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction check 2024-09-11T09:48:59.778 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction check succeeded. 2024-09-11T09:48:59.779 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction test 2024-09-11T09:48:59.877 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction test succeeded. 2024-09-11T09:48:59.877 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction 2024-09-11T09:48:59.929 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : rbd-mirror-2:19.1.1-298.gb53b6d3e.el9.x86_64 1/1 2024-09-11T09:48:59.929 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-11T09:48:59.929 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2024-09-11T09:48:59.929 INFO:teuthology.orchestra.run.smithi107.stdout: rbd-mirror-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-11T09:48:59.929 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-11T09:48:59.929 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-09-11T09:49:00.035 INFO:teuthology.orchestra.run.smithi017.stdout:Last metadata expiration check: 0:01:47 ago on Wed 11 Sep 2024 09:47:13 AM UTC. 2024-09-11T09:49:00.171 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install rbd-nbd 2024-09-11T09:49:00.177 INFO:teuthology.orchestra.run.smithi017.stdout:Dependencies resolved. 2024-09-11T09:49:00.178 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2024-09-11T09:49:00.178 INFO:teuthology.orchestra.run.smithi017.stdout: Package Arch Version Repo Size 2024-09-11T09:49:00.178 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2024-09-11T09:49:00.178 INFO:teuthology.orchestra.run.smithi017.stdout:Installing: 2024-09-11T09:49:00.178 INFO:teuthology.orchestra.run.smithi017.stdout: rbd-mirror x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 3.1 M 2024-09-11T09:49:00.178 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-11T09:49:00.178 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction Summary 2024-09-11T09:49:00.178 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2024-09-11T09:49:00.178 INFO:teuthology.orchestra.run.smithi017.stdout:Install 1 Package 2024-09-11T09:49:00.178 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-11T09:49:00.179 INFO:teuthology.orchestra.run.smithi017.stdout:Total download size: 3.1 M 2024-09-11T09:49:00.179 INFO:teuthology.orchestra.run.smithi017.stdout:Installed size: 13 M 2024-09-11T09:49:00.179 INFO:teuthology.orchestra.run.smithi017.stdout:Downloading Packages: 2024-09-11T09:49:00.187 INFO:teuthology.orchestra.run.smithi096.stdout: Preparing : 1/1 2024-09-11T09:49:00.350 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-11T09:49:00.396 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-11T09:49:00.522 INFO:teuthology.orchestra.run.smithi017.stdout:rbd-mirror-19.1.1-298.gb53b6d3e.el9.x86_64.rpm 9.1 MB/s | 3.1 MB 00:00 2024-09-11T09:49:00.522 INFO:teuthology.orchestra.run.smithi017.stdout:-------------------------------------------------------------------------------- 2024-09-11T09:49:00.523 INFO:teuthology.orchestra.run.smithi017.stdout:Total 9.0 MB/s | 3.1 MB 00:00 2024-09-11T09:49:00.523 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction check 2024-09-11T09:49:00.533 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction check succeeded. 2024-09-11T09:49:00.533 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction test 2024-09-11T09:49:00.619 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction test succeeded. 2024-09-11T09:49:00.619 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction 2024-09-11T09:49:00.746 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:01:47 ago on Wed 11 Sep 2024 09:47:13 AM UTC. 2024-09-11T09:49:00.880 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-09-11T09:49:00.881 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-09-11T09:49:00.881 INFO:teuthology.orchestra.run.smithi107.stdout: Package Architecture Version Repository Size 2024-09-11T09:49:00.881 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-09-11T09:49:00.881 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2024-09-11T09:49:00.882 INFO:teuthology.orchestra.run.smithi107.stdout: rbd-nbd x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 171 k 2024-09-11T09:49:00.882 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-11T09:49:00.882 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2024-09-11T09:49:00.882 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-09-11T09:49:00.882 INFO:teuthology.orchestra.run.smithi107.stdout:Install 1 Package 2024-09-11T09:49:00.882 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-11T09:49:00.882 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 171 k 2024-09-11T09:49:00.882 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 494 k 2024-09-11T09:49:00.882 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2024-09-11T09:49:00.926 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-11T09:49:01.024 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-11T09:49:01.062 INFO:teuthology.orchestra.run.smithi107.stdout:rbd-nbd-19.1.1-298.gb53b6d3e.el9.x86_64.rpm 948 kB/s | 171 kB 00:00 2024-09-11T09:49:01.063 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2024-09-11T09:49:01.063 INFO:teuthology.orchestra.run.smithi107.stdout:Total 942 kB/s | 171 kB 00:00 2024-09-11T09:49:01.063 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2024-09-11T09:49:01.070 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2024-09-11T09:49:01.071 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2024-09-11T09:49:01.073 INFO:teuthology.orchestra.run.smithi017.stdout: Preparing : 1/1 2024-09-11T09:49:01.097 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-11T09:49:01.126 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-mgr-cephadm-2:19.1.1-298.gb53b6d3e.el9.noarch 6/6 2024-09-11T09:49:01.130 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : rbd-mirror-2:19.1.1-298.gb53b6d3e.el9.x86_64 1/1 2024-09-11T09:49:01.133 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2024-09-11T09:49:01.134 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2024-09-11T09:49:01.158 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: rbd-mirror-2:19.1.1-298.gb53b6d3e.el9.x86_64 1/1 2024-09-11T09:49:01.158 INFO:teuthology.orchestra.run.smithi017.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-11T09:49:01.158 INFO:teuthology.orchestra.run.smithi017.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-11T09:49:01.158 INFO:teuthology.orchestra.run.smithi017.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-11T09:49:01.158 INFO:teuthology.orchestra.run.smithi017.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-11T09:49:01.159 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-11T09:49:01.283 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2024-09-11T09:49:01.393 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : rbd-nbd-2:19.1.1-298.gb53b6d3e.el9.x86_64 1/1 2024-09-11T09:49:01.840 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: rbd-nbd-2:19.1.1-298.gb53b6d3e.el9.x86_64 1/1 2024-09-11T09:49:01.912 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.1.1-298.gb53b6d3e.el9.noarch 6/6 2024-09-11T09:49:01.912 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-mgr-cephadm-2:19.1.1-298.gb53b6d3e.el9.noarch 1/6 2024-09-11T09:49:01.912 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-11T09:49:01.912 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-11T09:49:01.912 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-11T09:49:01.912 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-11T09:49:02.110 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : rbd-nbd-2:19.1.1-298.gb53b6d3e.el9.x86_64 1/1 2024-09-11T09:49:02.110 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-11T09:49:02.110 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2024-09-11T09:49:02.110 INFO:teuthology.orchestra.run.smithi107.stdout: rbd-nbd-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-11T09:49:02.110 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-11T09:49:02.110 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-09-11T09:49:02.183 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-11T09:49:02.183 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-11T09:49:02.183 INFO:teuthology.orchestra.run.smithi096.stdout:Installed: 2024-09-11T09:49:02.183 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-mgr-cephadm-2:19.1.1-298.gb53b6d3e.el9.noarch 2024-09-11T09:49:02.183 INFO:teuthology.orchestra.run.smithi096.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-11T09:49:02.183 INFO:teuthology.orchestra.run.smithi096.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-11T09:49:02.184 INFO:teuthology.orchestra.run.smithi096.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-11T09:49:02.184 INFO:teuthology.orchestra.run.smithi096.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-11T09:49:02.184 INFO:teuthology.orchestra.run.smithi096.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-11T09:49:02.184 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-11T09:49:02.184 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2024-09-11T09:49:02.228 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : rbd-mirror-2:19.1.1-298.gb53b6d3e.el9.x86_64 1/1 2024-09-11T09:49:02.229 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-11T09:49:02.229 INFO:teuthology.orchestra.run.smithi017.stdout:Installed: 2024-09-11T09:49:02.229 INFO:teuthology.orchestra.run.smithi017.stdout: rbd-mirror-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-11T09:49:02.229 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-11T09:49:02.229 INFO:teuthology.orchestra.run.smithi017.stdout:Complete! 2024-09-11T09:49:02.337 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install rbd-nbd 2024-09-11T09:49:02.454 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum -y install ceph-fuse 2024-09-11T09:49:02.468 DEBUG:teuthology.orchestra.run.smithi017:> sudo yum -y install rbd-nbd 2024-09-11T09:49:02.917 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:01:49 ago on Wed 11 Sep 2024 09:47:13 AM UTC. 2024-09-11T09:49:03.009 INFO:teuthology.orchestra.run.smithi107.stdout:Package rbd-nbd-2:19.1.1-298.gb53b6d3e.el9.x86_64 is already installed. 2024-09-11T09:49:03.020 INFO:teuthology.orchestra.run.smithi096.stdout:Last metadata expiration check: 0:01:51 ago on Wed 11 Sep 2024 09:47:12 AM UTC. 2024-09-11T09:49:03.054 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-09-11T09:49:03.055 INFO:teuthology.orchestra.run.smithi107.stdout:Nothing to do. 2024-09-11T09:49:03.055 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-09-11T09:49:03.061 INFO:teuthology.orchestra.run.smithi017.stdout:Last metadata expiration check: 0:01:50 ago on Wed 11 Sep 2024 09:47:13 AM UTC. 2024-09-11T09:49:03.121 DEBUG:teuthology.parallel:result is None 2024-09-11T09:49:03.154 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2024-09-11T09:49:03.155 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-09-11T09:49:03.155 INFO:teuthology.orchestra.run.smithi096.stdout: Package Arch Version Repository Size 2024-09-11T09:49:03.155 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-09-11T09:49:03.155 INFO:teuthology.orchestra.run.smithi096.stdout:Installing: 2024-09-11T09:49:03.155 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-fuse x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 897 k 2024-09-11T09:49:03.155 INFO:teuthology.orchestra.run.smithi096.stdout:Installing dependencies: 2024-09-11T09:49:03.155 INFO:teuthology.orchestra.run.smithi096.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-11T09:49:03.155 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-11T09:49:03.155 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction Summary 2024-09-11T09:49:03.155 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-09-11T09:49:03.155 INFO:teuthology.orchestra.run.smithi096.stdout:Install 2 Packages 2024-09-11T09:49:03.155 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-11T09:49:03.155 INFO:teuthology.orchestra.run.smithi096.stdout:Total download size: 977 k 2024-09-11T09:49:03.156 INFO:teuthology.orchestra.run.smithi096.stdout:Installed size: 2.9 M 2024-09-11T09:49:03.156 INFO:teuthology.orchestra.run.smithi096.stdout:Downloading Packages: 2024-09-11T09:49:03.201 INFO:teuthology.orchestra.run.smithi017.stdout:Dependencies resolved. 2024-09-11T09:49:03.202 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2024-09-11T09:49:03.202 INFO:teuthology.orchestra.run.smithi017.stdout: Package Architecture Version Repository Size 2024-09-11T09:49:03.202 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2024-09-11T09:49:03.202 INFO:teuthology.orchestra.run.smithi017.stdout:Installing: 2024-09-11T09:49:03.203 INFO:teuthology.orchestra.run.smithi017.stdout: rbd-nbd x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 171 k 2024-09-11T09:49:03.203 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-11T09:49:03.203 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction Summary 2024-09-11T09:49:03.203 INFO:teuthology.orchestra.run.smithi017.stdout:================================================================================ 2024-09-11T09:49:03.203 INFO:teuthology.orchestra.run.smithi017.stdout:Install 1 Package 2024-09-11T09:49:03.203 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-11T09:49:03.203 INFO:teuthology.orchestra.run.smithi017.stdout:Total download size: 171 k 2024-09-11T09:49:03.203 INFO:teuthology.orchestra.run.smithi017.stdout:Installed size: 494 k 2024-09-11T09:49:03.203 INFO:teuthology.orchestra.run.smithi017.stdout:Downloading Packages: 2024-09-11T09:49:03.417 INFO:teuthology.orchestra.run.smithi017.stdout:rbd-nbd-19.1.1-298.gb53b6d3e.el9.x86_64.rpm 800 kB/s | 171 kB 00:00 2024-09-11T09:49:03.417 INFO:teuthology.orchestra.run.smithi017.stdout:-------------------------------------------------------------------------------- 2024-09-11T09:49:03.418 INFO:teuthology.orchestra.run.smithi017.stdout:Total 795 kB/s | 171 kB 00:00 2024-09-11T09:49:03.418 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction check 2024-09-11T09:49:03.426 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction check succeeded. 2024-09-11T09:49:03.426 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction test 2024-09-11T09:49:03.494 INFO:teuthology.orchestra.run.smithi017.stdout:Transaction test succeeded. 2024-09-11T09:49:03.494 INFO:teuthology.orchestra.run.smithi017.stdout:Running transaction 2024-09-11T09:49:03.573 INFO:teuthology.orchestra.run.smithi096.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 590 kB/s | 80 kB 00:00 2024-09-11T09:49:03.656 INFO:teuthology.orchestra.run.smithi017.stdout: Preparing : 1/1 2024-09-11T09:49:03.765 INFO:teuthology.orchestra.run.smithi096.stdout:(2/2): ceph-fuse-19.1.1-298.gb53b6d3e.el9.x86_6 2.7 MB/s | 897 kB 00:00 2024-09-11T09:49:03.765 INFO:teuthology.orchestra.run.smithi096.stdout:-------------------------------------------------------------------------------- 2024-09-11T09:49:03.766 INFO:teuthology.orchestra.run.smithi096.stdout:Total 1.6 MB/s | 977 kB 00:00 2024-09-11T09:49:03.767 INFO:teuthology.orchestra.run.smithi017.stdout: Installing : rbd-nbd-2:19.1.1-298.gb53b6d3e.el9.x86_64 1/1 2024-09-11T09:49:03.778 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction check 2024-09-11T09:49:03.802 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction check succeeded. 2024-09-11T09:49:03.802 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction test 2024-09-11T09:49:03.886 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction test succeeded. 2024-09-11T09:49:03.886 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction 2024-09-11T09:49:04.076 INFO:teuthology.orchestra.run.smithi096.stdout: Preparing : 1/1 2024-09-11T09:49:04.139 INFO:teuthology.orchestra.run.smithi017.stdout: Running scriptlet: rbd-nbd-2:19.1.1-298.gb53b6d3e.el9.x86_64 1/1 2024-09-11T09:49:04.217 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-11T09:49:04.282 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-fuse-2:19.1.1-298.gb53b6d3e.el9.x86_64 2/2 2024-09-11T09:49:04.397 INFO:teuthology.orchestra.run.smithi017.stdout: Verifying : rbd-nbd-2:19.1.1-298.gb53b6d3e.el9.x86_64 1/1 2024-09-11T09:49:04.397 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-11T09:49:04.397 INFO:teuthology.orchestra.run.smithi017.stdout:Installed: 2024-09-11T09:49:04.397 INFO:teuthology.orchestra.run.smithi017.stdout: rbd-nbd-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-11T09:49:04.397 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-11T09:49:04.397 INFO:teuthology.orchestra.run.smithi017.stdout:Complete! 2024-09-11T09:49:04.662 DEBUG:teuthology.orchestra.run.smithi017:> sudo yum -y install rbd-nbd 2024-09-11T09:49:05.000 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-fuse-2:19.1.1-298.gb53b6d3e.el9.x86_64 2/2 2024-09-11T09:49:05.000 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-fuse-2:19.1.1-298.gb53b6d3e.el9.x86_64 1/2 2024-09-11T09:49:05.255 INFO:teuthology.orchestra.run.smithi017.stdout:Last metadata expiration check: 0:01:52 ago on Wed 11 Sep 2024 09:47:13 AM UTC. 2024-09-11T09:49:05.258 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-11T09:49:05.258 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-11T09:49:05.258 INFO:teuthology.orchestra.run.smithi096.stdout:Installed: 2024-09-11T09:49:05.258 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-fuse-2:19.1.1-298.gb53b6d3e.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-11T09:49:05.258 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-11T09:49:05.258 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2024-09-11T09:49:05.349 INFO:teuthology.orchestra.run.smithi017.stdout:Package rbd-nbd-2:19.1.1-298.gb53b6d3e.el9.x86_64 is already installed. 2024-09-11T09:49:05.395 INFO:teuthology.orchestra.run.smithi017.stdout:Dependencies resolved. 2024-09-11T09:49:05.396 INFO:teuthology.orchestra.run.smithi017.stdout:Nothing to do. 2024-09-11T09:49:05.396 INFO:teuthology.orchestra.run.smithi017.stdout:Complete! 2024-09-11T09:49:05.530 DEBUG:teuthology.parallel:result is None 2024-09-11T09:49:05.552 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum -y install ceph-volume 2024-09-11T09:49:06.132 INFO:teuthology.orchestra.run.smithi096.stdout:Last metadata expiration check: 0:01:54 ago on Wed 11 Sep 2024 09:47:12 AM UTC. 2024-09-11T09:49:06.270 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2024-09-11T09:49:06.271 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-09-11T09:49:06.271 INFO:teuthology.orchestra.run.smithi096.stdout: Package Arch Version Repository Size 2024-09-11T09:49:06.271 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-09-11T09:49:06.271 INFO:teuthology.orchestra.run.smithi096.stdout:Installing: 2024-09-11T09:49:06.272 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-volume noarch 2:19.1.1-298.gb53b6d3e.el9 ceph-noarch 285 k 2024-09-11T09:49:06.272 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-11T09:49:06.272 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction Summary 2024-09-11T09:49:06.272 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-09-11T09:49:06.272 INFO:teuthology.orchestra.run.smithi096.stdout:Install 1 Package 2024-09-11T09:49:06.272 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-11T09:49:06.272 INFO:teuthology.orchestra.run.smithi096.stdout:Total download size: 285 k 2024-09-11T09:49:06.272 INFO:teuthology.orchestra.run.smithi096.stdout:Installed size: 1.3 M 2024-09-11T09:49:06.272 INFO:teuthology.orchestra.run.smithi096.stdout:Downloading Packages: 2024-09-11T09:49:06.479 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-volume-19.1.1-298.gb53b6d3e.el9.noarch.rpm 1.4 MB/s | 285 kB 00:00 2024-09-11T09:49:06.479 INFO:teuthology.orchestra.run.smithi096.stdout:-------------------------------------------------------------------------------- 2024-09-11T09:49:06.480 INFO:teuthology.orchestra.run.smithi096.stdout:Total 1.3 MB/s | 285 kB 00:00 2024-09-11T09:49:06.480 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction check 2024-09-11T09:49:06.488 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction check succeeded. 2024-09-11T09:49:06.488 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction test 2024-09-11T09:49:06.516 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction test succeeded. 2024-09-11T09:49:06.516 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction 2024-09-11T09:49:06.732 INFO:teuthology.orchestra.run.smithi096.stdout: Preparing : 1/1 2024-09-11T09:49:06.798 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : ceph-volume-2:19.1.1-298.gb53b6d3e.el9.noarch 1/1 2024-09-11T09:49:06.814 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: ceph-volume-2:19.1.1-298.gb53b6d3e.el9.noarch 1/1 2024-09-11T09:49:06.814 INFO:teuthology.orchestra.run.smithi096.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-11T09:49:06.814 INFO:teuthology.orchestra.run.smithi096.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-11T09:49:06.814 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-11T09:49:07.857 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : ceph-volume-2:19.1.1-298.gb53b6d3e.el9.noarch 1/1 2024-09-11T09:49:07.858 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-11T09:49:07.858 INFO:teuthology.orchestra.run.smithi096.stdout:Installed: 2024-09-11T09:49:07.858 INFO:teuthology.orchestra.run.smithi096.stdout: ceph-volume-2:19.1.1-298.gb53b6d3e.el9.noarch 2024-09-11T09:49:07.858 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-11T09:49:07.858 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2024-09-11T09:49:08.169 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum -y install librados-devel 2024-09-11T09:49:08.753 INFO:teuthology.orchestra.run.smithi096.stdout:Last metadata expiration check: 0:01:56 ago on Wed 11 Sep 2024 09:47:12 AM UTC. 2024-09-11T09:49:08.889 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2024-09-11T09:49:08.890 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-09-11T09:49:08.890 INFO:teuthology.orchestra.run.smithi096.stdout: Package Arch Version Repo Size 2024-09-11T09:49:08.890 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-09-11T09:49:08.891 INFO:teuthology.orchestra.run.smithi096.stdout:Installing: 2024-09-11T09:49:08.891 INFO:teuthology.orchestra.run.smithi096.stdout: librados-devel x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 127 k 2024-09-11T09:49:08.891 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-11T09:49:08.891 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction Summary 2024-09-11T09:49:08.891 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-09-11T09:49:08.891 INFO:teuthology.orchestra.run.smithi096.stdout:Install 1 Package 2024-09-11T09:49:08.891 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-11T09:49:08.891 INFO:teuthology.orchestra.run.smithi096.stdout:Total download size: 127 k 2024-09-11T09:49:08.891 INFO:teuthology.orchestra.run.smithi096.stdout:Installed size: 456 k 2024-09-11T09:49:08.891 INFO:teuthology.orchestra.run.smithi096.stdout:Downloading Packages: 2024-09-11T09:49:09.077 INFO:teuthology.orchestra.run.smithi096.stdout:librados-devel-19.1.1-298.gb53b6d3e.el9.x86_64. 682 kB/s | 127 kB 00:00 2024-09-11T09:49:09.077 INFO:teuthology.orchestra.run.smithi096.stdout:-------------------------------------------------------------------------------- 2024-09-11T09:49:09.078 INFO:teuthology.orchestra.run.smithi096.stdout:Total 677 kB/s | 127 kB 00:00 2024-09-11T09:49:09.078 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction check 2024-09-11T09:49:09.083 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction check succeeded. 2024-09-11T09:49:09.083 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction test 2024-09-11T09:49:09.147 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction test succeeded. 2024-09-11T09:49:09.147 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction 2024-09-11T09:49:09.297 INFO:teuthology.orchestra.run.smithi096.stdout: Preparing : 1/1 2024-09-11T09:49:09.399 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : librados-devel-2:19.1.1-298.gb53b6d3e.el9.x86_64 1/1 2024-09-11T09:49:09.856 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: librados-devel-2:19.1.1-298.gb53b6d3e.el9.x86_64 1/1 2024-09-11T09:49:10.126 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : librados-devel-2:19.1.1-298.gb53b6d3e.el9.x86_64 1/1 2024-09-11T09:49:10.126 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-11T09:49:10.126 INFO:teuthology.orchestra.run.smithi096.stdout:Installed: 2024-09-11T09:49:10.126 INFO:teuthology.orchestra.run.smithi096.stdout: librados-devel-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-11T09:49:10.126 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-11T09:49:10.126 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2024-09-11T09:49:10.378 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum -y install libcephfs2 2024-09-11T09:49:10.961 INFO:teuthology.orchestra.run.smithi096.stdout:Last metadata expiration check: 0:01:58 ago on Wed 11 Sep 2024 09:47:12 AM UTC. 2024-09-11T09:49:11.054 INFO:teuthology.orchestra.run.smithi096.stdout:Package libcephfs2-2:19.1.1-298.gb53b6d3e.el9.x86_64 is already installed. 2024-09-11T09:49:11.099 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2024-09-11T09:49:11.100 INFO:teuthology.orchestra.run.smithi096.stdout:Nothing to do. 2024-09-11T09:49:11.100 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2024-09-11T09:49:11.179 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum -y install libcephfs-devel 2024-09-11T09:49:11.746 INFO:teuthology.orchestra.run.smithi096.stdout:Last metadata expiration check: 0:01:59 ago on Wed 11 Sep 2024 09:47:12 AM UTC. 2024-09-11T09:49:11.880 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2024-09-11T09:49:11.881 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-09-11T09:49:11.881 INFO:teuthology.orchestra.run.smithi096.stdout: Package Arch Version Repo Size 2024-09-11T09:49:11.881 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-09-11T09:49:11.881 INFO:teuthology.orchestra.run.smithi096.stdout:Installing: 2024-09-11T09:49:11.881 INFO:teuthology.orchestra.run.smithi096.stdout: libcephfs-devel x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 32 k 2024-09-11T09:49:11.882 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-11T09:49:11.882 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction Summary 2024-09-11T09:49:11.882 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-09-11T09:49:11.882 INFO:teuthology.orchestra.run.smithi096.stdout:Install 1 Package 2024-09-11T09:49:11.882 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-11T09:49:11.882 INFO:teuthology.orchestra.run.smithi096.stdout:Total download size: 32 k 2024-09-11T09:49:11.882 INFO:teuthology.orchestra.run.smithi096.stdout:Installed size: 142 k 2024-09-11T09:49:11.882 INFO:teuthology.orchestra.run.smithi096.stdout:Downloading Packages: 2024-09-11T09:49:12.040 INFO:teuthology.orchestra.run.smithi096.stdout:libcephfs-devel-19.1.1-298.gb53b6d3e.el9.x86_64 201 kB/s | 32 kB 00:00 2024-09-11T09:49:12.041 INFO:teuthology.orchestra.run.smithi096.stdout:-------------------------------------------------------------------------------- 2024-09-11T09:49:12.041 INFO:teuthology.orchestra.run.smithi096.stdout:Total 199 kB/s | 32 kB 00:00 2024-09-11T09:49:12.041 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction check 2024-09-11T09:49:12.044 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction check succeeded. 2024-09-11T09:49:12.044 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction test 2024-09-11T09:49:12.062 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction test succeeded. 2024-09-11T09:49:12.062 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction 2024-09-11T09:49:12.145 INFO:teuthology.orchestra.run.smithi096.stdout: Preparing : 1/1 2024-09-11T09:49:12.254 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : libcephfs-devel-2:19.1.1-298.gb53b6d3e.el9.x86_64 1/1 2024-09-11T09:49:12.704 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: libcephfs-devel-2:19.1.1-298.gb53b6d3e.el9.x86_64 1/1 2024-09-11T09:49:12.940 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : libcephfs-devel-2:19.1.1-298.gb53b6d3e.el9.x86_64 1/1 2024-09-11T09:49:12.940 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-11T09:49:12.940 INFO:teuthology.orchestra.run.smithi096.stdout:Installed: 2024-09-11T09:49:12.940 INFO:teuthology.orchestra.run.smithi096.stdout: libcephfs-devel-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-11T09:49:12.940 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-11T09:49:12.940 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2024-09-11T09:49:13.138 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum -y install librados2 2024-09-11T09:49:13.703 INFO:teuthology.orchestra.run.smithi096.stdout:Last metadata expiration check: 0:02:01 ago on Wed 11 Sep 2024 09:47:12 AM UTC. 2024-09-11T09:49:13.793 INFO:teuthology.orchestra.run.smithi096.stdout:Package librados2-2:19.1.1-298.gb53b6d3e.el9.x86_64 is already installed. 2024-09-11T09:49:13.837 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2024-09-11T09:49:13.838 INFO:teuthology.orchestra.run.smithi096.stdout:Nothing to do. 2024-09-11T09:49:13.838 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2024-09-11T09:49:13.908 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum -y install librbd1 2024-09-11T09:49:14.471 INFO:teuthology.orchestra.run.smithi096.stdout:Last metadata expiration check: 0:02:02 ago on Wed 11 Sep 2024 09:47:12 AM UTC. 2024-09-11T09:49:14.560 INFO:teuthology.orchestra.run.smithi096.stdout:Package librbd1-2:19.1.1-298.gb53b6d3e.el9.x86_64 is already installed. 2024-09-11T09:49:14.604 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2024-09-11T09:49:14.605 INFO:teuthology.orchestra.run.smithi096.stdout:Nothing to do. 2024-09-11T09:49:14.605 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2024-09-11T09:49:14.683 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum -y install python3-rados 2024-09-11T09:49:15.253 INFO:teuthology.orchestra.run.smithi096.stdout:Last metadata expiration check: 0:02:03 ago on Wed 11 Sep 2024 09:47:12 AM UTC. 2024-09-11T09:49:15.343 INFO:teuthology.orchestra.run.smithi096.stdout:Package python3-rados-2:19.1.1-298.gb53b6d3e.el9.x86_64 is already installed. 2024-09-11T09:49:15.385 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2024-09-11T09:49:15.386 INFO:teuthology.orchestra.run.smithi096.stdout:Nothing to do. 2024-09-11T09:49:15.387 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2024-09-11T09:49:15.467 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum -y install python3-rgw 2024-09-11T09:49:16.036 INFO:teuthology.orchestra.run.smithi096.stdout:Last metadata expiration check: 0:02:04 ago on Wed 11 Sep 2024 09:47:12 AM UTC. 2024-09-11T09:49:16.127 INFO:teuthology.orchestra.run.smithi096.stdout:Package python3-rgw-2:19.1.1-298.gb53b6d3e.el9.x86_64 is already installed. 2024-09-11T09:49:16.170 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2024-09-11T09:49:16.172 INFO:teuthology.orchestra.run.smithi096.stdout:Nothing to do. 2024-09-11T09:49:16.172 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2024-09-11T09:49:16.242 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum -y install python3-cephfs 2024-09-11T09:49:16.802 INFO:teuthology.orchestra.run.smithi096.stdout:Last metadata expiration check: 0:02:04 ago on Wed 11 Sep 2024 09:47:12 AM UTC. 2024-09-11T09:49:16.891 INFO:teuthology.orchestra.run.smithi096.stdout:Package python3-cephfs-2:19.1.1-298.gb53b6d3e.el9.x86_64 is already installed. 2024-09-11T09:49:16.935 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2024-09-11T09:49:16.936 INFO:teuthology.orchestra.run.smithi096.stdout:Nothing to do. 2024-09-11T09:49:16.936 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2024-09-11T09:49:17.002 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum -y install python3-rbd 2024-09-11T09:49:17.569 INFO:teuthology.orchestra.run.smithi096.stdout:Last metadata expiration check: 0:02:05 ago on Wed 11 Sep 2024 09:47:12 AM UTC. 2024-09-11T09:49:17.659 INFO:teuthology.orchestra.run.smithi096.stdout:Package python3-rbd-2:19.1.1-298.gb53b6d3e.el9.x86_64 is already installed. 2024-09-11T09:49:17.702 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2024-09-11T09:49:17.703 INFO:teuthology.orchestra.run.smithi096.stdout:Nothing to do. 2024-09-11T09:49:17.703 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2024-09-11T09:49:17.783 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum -y install rbd-fuse 2024-09-11T09:49:18.349 INFO:teuthology.orchestra.run.smithi096.stdout:Last metadata expiration check: 0:02:06 ago on Wed 11 Sep 2024 09:47:12 AM UTC. 2024-09-11T09:49:18.483 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2024-09-11T09:49:18.484 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-09-11T09:49:18.484 INFO:teuthology.orchestra.run.smithi096.stdout: Package Architecture Version Repository Size 2024-09-11T09:49:18.484 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-09-11T09:49:18.484 INFO:teuthology.orchestra.run.smithi096.stdout:Installing: 2024-09-11T09:49:18.484 INFO:teuthology.orchestra.run.smithi096.stdout: rbd-fuse x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 85 k 2024-09-11T09:49:18.484 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-11T09:49:18.484 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction Summary 2024-09-11T09:49:18.485 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-09-11T09:49:18.485 INFO:teuthology.orchestra.run.smithi096.stdout:Install 1 Package 2024-09-11T09:49:18.485 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-11T09:49:18.485 INFO:teuthology.orchestra.run.smithi096.stdout:Total download size: 85 k 2024-09-11T09:49:18.485 INFO:teuthology.orchestra.run.smithi096.stdout:Installed size: 231 k 2024-09-11T09:49:18.485 INFO:teuthology.orchestra.run.smithi096.stdout:Downloading Packages: 2024-09-11T09:49:18.668 INFO:teuthology.orchestra.run.smithi096.stdout:rbd-fuse-19.1.1-298.gb53b6d3e.el9.x86_64.rpm 465 kB/s | 85 kB 00:00 2024-09-11T09:49:18.669 INFO:teuthology.orchestra.run.smithi096.stdout:-------------------------------------------------------------------------------- 2024-09-11T09:49:18.669 INFO:teuthology.orchestra.run.smithi096.stdout:Total 462 kB/s | 85 kB 00:00 2024-09-11T09:49:18.669 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction check 2024-09-11T09:49:18.675 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction check succeeded. 2024-09-11T09:49:18.676 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction test 2024-09-11T09:49:18.739 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction test succeeded. 2024-09-11T09:49:18.740 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction 2024-09-11T09:49:18.890 INFO:teuthology.orchestra.run.smithi096.stdout: Preparing : 1/1 2024-09-11T09:49:18.998 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : rbd-fuse-2:19.1.1-298.gb53b6d3e.el9.x86_64 1/1 2024-09-11T09:49:19.417 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: rbd-fuse-2:19.1.1-298.gb53b6d3e.el9.x86_64 1/1 2024-09-11T09:49:19.707 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : rbd-fuse-2:19.1.1-298.gb53b6d3e.el9.x86_64 1/1 2024-09-11T09:49:19.707 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-11T09:49:19.707 INFO:teuthology.orchestra.run.smithi096.stdout:Installed: 2024-09-11T09:49:19.707 INFO:teuthology.orchestra.run.smithi096.stdout: rbd-fuse-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-11T09:49:19.707 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-11T09:49:19.707 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2024-09-11T09:49:19.982 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum -y install rbd-mirror 2024-09-11T09:49:20.579 INFO:teuthology.orchestra.run.smithi096.stdout:Last metadata expiration check: 0:02:08 ago on Wed 11 Sep 2024 09:47:12 AM UTC. 2024-09-11T09:49:20.719 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2024-09-11T09:49:20.720 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-09-11T09:49:20.720 INFO:teuthology.orchestra.run.smithi096.stdout: Package Arch Version Repo Size 2024-09-11T09:49:20.720 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-09-11T09:49:20.720 INFO:teuthology.orchestra.run.smithi096.stdout:Installing: 2024-09-11T09:49:20.720 INFO:teuthology.orchestra.run.smithi096.stdout: rbd-mirror x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 3.1 M 2024-09-11T09:49:20.720 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-11T09:49:20.720 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction Summary 2024-09-11T09:49:20.720 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-09-11T09:49:20.720 INFO:teuthology.orchestra.run.smithi096.stdout:Install 1 Package 2024-09-11T09:49:20.720 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-11T09:49:20.720 INFO:teuthology.orchestra.run.smithi096.stdout:Total download size: 3.1 M 2024-09-11T09:49:20.720 INFO:teuthology.orchestra.run.smithi096.stdout:Installed size: 13 M 2024-09-11T09:49:20.721 INFO:teuthology.orchestra.run.smithi096.stdout:Downloading Packages: 2024-09-11T09:49:21.065 INFO:teuthology.orchestra.run.smithi096.stdout:rbd-mirror-19.1.1-298.gb53b6d3e.el9.x86_64.rpm 9.0 MB/s | 3.1 MB 00:00 2024-09-11T09:49:21.066 INFO:teuthology.orchestra.run.smithi096.stdout:-------------------------------------------------------------------------------- 2024-09-11T09:49:21.066 INFO:teuthology.orchestra.run.smithi096.stdout:Total 9.0 MB/s | 3.1 MB 00:00 2024-09-11T09:49:21.066 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction check 2024-09-11T09:49:21.075 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction check succeeded. 2024-09-11T09:49:21.075 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction test 2024-09-11T09:49:21.156 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction test succeeded. 2024-09-11T09:49:21.156 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction 2024-09-11T09:49:21.592 INFO:teuthology.orchestra.run.smithi096.stdout: Preparing : 1/1 2024-09-11T09:49:21.666 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : rbd-mirror-2:19.1.1-298.gb53b6d3e.el9.x86_64 1/1 2024-09-11T09:49:21.691 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: rbd-mirror-2:19.1.1-298.gb53b6d3e.el9.x86_64 1/1 2024-09-11T09:49:21.692 INFO:teuthology.orchestra.run.smithi096.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-11T09:49:21.692 INFO:teuthology.orchestra.run.smithi096.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-11T09:49:21.692 INFO:teuthology.orchestra.run.smithi096.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-11T09:49:21.692 INFO:teuthology.orchestra.run.smithi096.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-11T09:49:21.692 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-11T09:49:22.838 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : rbd-mirror-2:19.1.1-298.gb53b6d3e.el9.x86_64 1/1 2024-09-11T09:49:22.838 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-11T09:49:22.839 INFO:teuthology.orchestra.run.smithi096.stdout:Installed: 2024-09-11T09:49:22.839 INFO:teuthology.orchestra.run.smithi096.stdout: rbd-mirror-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-11T09:49:22.839 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-11T09:49:22.839 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2024-09-11T09:49:23.093 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum -y install rbd-nbd 2024-09-11T09:49:23.667 INFO:teuthology.orchestra.run.smithi096.stdout:Last metadata expiration check: 0:02:11 ago on Wed 11 Sep 2024 09:47:12 AM UTC. 2024-09-11T09:49:23.805 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2024-09-11T09:49:23.806 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-09-11T09:49:23.806 INFO:teuthology.orchestra.run.smithi096.stdout: Package Architecture Version Repository Size 2024-09-11T09:49:23.806 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-09-11T09:49:23.806 INFO:teuthology.orchestra.run.smithi096.stdout:Installing: 2024-09-11T09:49:23.806 INFO:teuthology.orchestra.run.smithi096.stdout: rbd-nbd x86_64 2:19.1.1-298.gb53b6d3e.el9 ceph 171 k 2024-09-11T09:49:23.806 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-11T09:49:23.806 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction Summary 2024-09-11T09:49:23.807 INFO:teuthology.orchestra.run.smithi096.stdout:================================================================================ 2024-09-11T09:49:23.807 INFO:teuthology.orchestra.run.smithi096.stdout:Install 1 Package 2024-09-11T09:49:23.807 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-11T09:49:23.807 INFO:teuthology.orchestra.run.smithi096.stdout:Total download size: 171 k 2024-09-11T09:49:23.807 INFO:teuthology.orchestra.run.smithi096.stdout:Installed size: 494 k 2024-09-11T09:49:23.807 INFO:teuthology.orchestra.run.smithi096.stdout:Downloading Packages: 2024-09-11T09:49:24.045 INFO:teuthology.orchestra.run.smithi096.stdout:rbd-nbd-19.1.1-298.gb53b6d3e.el9.x86_64.rpm 720 kB/s | 171 kB 00:00 2024-09-11T09:49:24.046 INFO:teuthology.orchestra.run.smithi096.stdout:-------------------------------------------------------------------------------- 2024-09-11T09:49:24.046 INFO:teuthology.orchestra.run.smithi096.stdout:Total 714 kB/s | 171 kB 00:00 2024-09-11T09:49:24.046 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction check 2024-09-11T09:49:24.054 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction check succeeded. 2024-09-11T09:49:24.054 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction test 2024-09-11T09:49:24.117 INFO:teuthology.orchestra.run.smithi096.stdout:Transaction test succeeded. 2024-09-11T09:49:24.118 INFO:teuthology.orchestra.run.smithi096.stdout:Running transaction 2024-09-11T09:49:24.273 INFO:teuthology.orchestra.run.smithi096.stdout: Preparing : 1/1 2024-09-11T09:49:24.368 INFO:teuthology.orchestra.run.smithi096.stdout: Installing : rbd-nbd-2:19.1.1-298.gb53b6d3e.el9.x86_64 1/1 2024-09-11T09:49:24.810 INFO:teuthology.orchestra.run.smithi096.stdout: Running scriptlet: rbd-nbd-2:19.1.1-298.gb53b6d3e.el9.x86_64 1/1 2024-09-11T09:49:25.112 INFO:teuthology.orchestra.run.smithi096.stdout: Verifying : rbd-nbd-2:19.1.1-298.gb53b6d3e.el9.x86_64 1/1 2024-09-11T09:49:25.113 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-11T09:49:25.113 INFO:teuthology.orchestra.run.smithi096.stdout:Installed: 2024-09-11T09:49:25.113 INFO:teuthology.orchestra.run.smithi096.stdout: rbd-nbd-2:19.1.1-298.gb53b6d3e.el9.x86_64 2024-09-11T09:49:25.113 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-11T09:49:25.113 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2024-09-11T09:49:25.365 DEBUG:teuthology.orchestra.run.smithi096:> sudo yum -y install rbd-nbd 2024-09-11T09:49:25.932 INFO:teuthology.orchestra.run.smithi096.stdout:Last metadata expiration check: 0:02:13 ago on Wed 11 Sep 2024 09:47:12 AM UTC. 2024-09-11T09:49:26.024 INFO:teuthology.orchestra.run.smithi096.stdout:Package rbd-nbd-2:19.1.1-298.gb53b6d3e.el9.x86_64 is already installed. 2024-09-11T09:49:26.068 INFO:teuthology.orchestra.run.smithi096.stdout:Dependencies resolved. 2024-09-11T09:49:26.070 INFO:teuthology.orchestra.run.smithi096.stdout:Nothing to do. 2024-09-11T09:49:26.070 INFO:teuthology.orchestra.run.smithi096.stdout:Complete! 2024-09-11T09:49:26.152 DEBUG:teuthology.parallel:result is None 2024-09-11T09:49:26.153 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc 2024-09-11T09:49:26.295 DEBUG:teuthology.orchestra.run.smithi017:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-11T09:49:26.321 INFO:teuthology.orchestra.run.smithi017.stdout:19.1.1-298.gb53b6d3e.el9 2024-09-11T09:49:26.322 INFO:teuthology.packaging:The installed version of ceph is 19.1.1-298.gb53b6d3e.el9 2024-09-11T09:49:26.322 INFO:teuthology.task.install:The correct ceph version 19.1.1-298.gb53b6d3e is installed. 2024-09-11T09:49:26.324 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc 2024-09-11T09:49:26.461 DEBUG:teuthology.orchestra.run.smithi096:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-11T09:49:26.483 INFO:teuthology.orchestra.run.smithi096.stdout:19.1.1-298.gb53b6d3e.el9 2024-09-11T09:49:26.484 INFO:teuthology.packaging:The installed version of ceph is 19.1.1-298.gb53b6d3e.el9 2024-09-11T09:49:26.484 INFO:teuthology.task.install:The correct ceph version 19.1.1-298.gb53b6d3e is installed. 2024-09-11T09:49:26.485 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc 2024-09-11T09:49:26.618 DEBUG:teuthology.orchestra.run.smithi107:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-11T09:49:26.643 INFO:teuthology.orchestra.run.smithi107.stdout:19.1.1-298.gb53b6d3e.el9 2024-09-11T09:49:26.643 INFO:teuthology.packaging:The installed version of ceph is 19.1.1-298.gb53b6d3e.el9 2024-09-11T09:49:26.644 INFO:teuthology.task.install:The correct ceph version 19.1.1-298.gb53b6d3e is installed. 2024-09-11T09:49:26.645 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-11T09:49:26.646 DEBUG:teuthology.orchestra.run.smithi017:> set -ex 2024-09-11T09:49:26.646 DEBUG:teuthology.orchestra.run.smithi017:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-11T09:49:26.677 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-09-11T09:49:26.677 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-11T09:49:26.703 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-09-11T09:49:26.703 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-11T09:49:26.732 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-11T09:49:26.732 DEBUG:teuthology.orchestra.run.smithi017:> set -ex 2024-09-11T09:49:26.732 DEBUG:teuthology.orchestra.run.smithi017:> sudo dd of=/usr/bin/daemon-helper 2024-09-11T09:49:26.763 DEBUG:teuthology.orchestra.run.smithi017:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-11T09:49:26.833 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-09-11T09:49:26.833 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd of=/usr/bin/daemon-helper 2024-09-11T09:49:26.859 DEBUG:teuthology.orchestra.run.smithi096:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-11T09:49:26.924 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-09-11T09:49:26.925 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd of=/usr/bin/daemon-helper 2024-09-11T09:49:26.954 DEBUG:teuthology.orchestra.run.smithi107:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-11T09:49:27.022 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-11T09:49:27.022 DEBUG:teuthology.orchestra.run.smithi017:> set -ex 2024-09-11T09:49:27.022 DEBUG:teuthology.orchestra.run.smithi017:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-11T09:49:27.054 DEBUG:teuthology.orchestra.run.smithi017:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-11T09:49:27.122 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-09-11T09:49:27.122 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-11T09:49:27.147 DEBUG:teuthology.orchestra.run.smithi096:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-11T09:49:27.212 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-09-11T09:49:27.212 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-11T09:49:27.240 DEBUG:teuthology.orchestra.run.smithi107:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-11T09:49:27.308 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-11T09:49:27.308 DEBUG:teuthology.orchestra.run.smithi017:> set -ex 2024-09-11T09:49:27.308 DEBUG:teuthology.orchestra.run.smithi017:> sudo dd of=/usr/bin/stdin-killer 2024-09-11T09:49:27.334 DEBUG:teuthology.orchestra.run.smithi017:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-11T09:49:27.400 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-09-11T09:49:27.400 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd of=/usr/bin/stdin-killer 2024-09-11T09:49:27.424 DEBUG:teuthology.orchestra.run.smithi096:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-11T09:49:27.488 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-09-11T09:49:27.489 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd of=/usr/bin/stdin-killer 2024-09-11T09:49:27.514 DEBUG:teuthology.orchestra.run.smithi107:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-11T09:49:27.579 INFO:teuthology.run_tasks:Running task ceph... 2024-09-11T09:49:27.653 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-09-11T09:49:27.653 DEBUG:teuthology.orchestra.run.smithi017:> sudo chmod 777 /var/log/ceph 2024-09-11T09:49:27.656 DEBUG:teuthology.orchestra.run.smithi096:> sudo chmod 777 /var/log/ceph 2024-09-11T09:49:27.658 DEBUG:teuthology.orchestra.run.smithi107:> sudo chmod 777 /var/log/ceph 2024-09-11T09:49:27.683 INFO:tasks.ceph:Disabling ceph logrotate... 2024-09-11T09:49:27.683 DEBUG:teuthology.orchestra.run.smithi017:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-11T09:49:27.723 DEBUG:teuthology.orchestra.run.smithi096:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-11T09:49:27.725 DEBUG:teuthology.orchestra.run.smithi107:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-11T09:49:27.750 INFO:tasks.ceph:Creating extra log directories... 2024-09-11T09:49:27.750 DEBUG:teuthology.orchestra.run.smithi017:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-11T09:49:27.789 DEBUG:teuthology.orchestra.run.smithi096:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-11T09:49:27.791 DEBUG:teuthology.orchestra.run.smithi107:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-11T09:49:27.829 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-09-11T09:49:27.829 INFO:tasks.ceph:config {'conf': {'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum', 'noscrub', 'nodeep-scrub'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-09-11T09:49:27.830 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/teuthology-2024-09-10_21:16:03-rbd-squid-distro-default-smithi/7900734', 'branch': 'squid', 'description': 'rbd/device/{base/install cluster/{fixed-3 openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-comp-snappy supported-random-distro$/{centos_latest} thrashers/default thrashosds-health workloads/rbd_nbd}', 'email': 'ceph-qa@ceph.com', 'first_in_suite': False, 'job_id': '7900734', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'teuthology-2024-09-10_21:16:03-rbd-squid-distro-default-smithi', 'no_nested_subset': False, 'openstack': [{'machine': {'cpus': 1, 'disk': 40, 'ram': 8000}, 'volumes': {'count': 4, 'size': 30}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'squid'}, 'ceph': {'conf': {'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum', 'noscrub', 'nodeep-scrub'], 'sha1': 'b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'extra_packages': ['rbd-nbd'], 'flavor': 'default', 'sha1': 'b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc'}}, 'selinux': {'allowlist': ['scontext=system_u:system_r:getty_t:s0']}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'squid', 'sha1': 'daaf78de334647fc0c800f4ac495cc9475a26bae'}}, 'owner': 'scheduled_teuthology@teuthology', 'priority': 100, 'repo': 'https://git.ceph.com/ceph.git', 'roles': [['mon.a', 'mon.c', 'osd.0', 'osd.1', 'osd.2'], ['mon.b', 'mgr.x', 'osd.3', 'osd.4', 'osd.5'], ['client.0']], 'seed': 6175, 'sha1': 'b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc', 'sleep_before_teardown': 0, 'subset': '88/128', 'suite': 'rbd', 'suite_branch': 'squid', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph_daaf78de334647fc0c800f4ac495cc9475a26bae/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph.git', 'suite_sha1': 'daaf78de334647fc0c800f4ac495cc9475a26bae', 'targets': {'smithi017.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHneZ92wLS9wEoVFsEJff3TbxllQaVnFBBpd3aMf/OktTHEsayzyKE9etbZlkwKAIw6ZJcpxOKYD23MNb5RAPCk=', 'smithi096.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNxpPWO64yIr6T4RKgPsPM+OG+4ctP1rBk+N/B/0SvFgl4R4Z4LwFfGE0/chlp8RdvlhhVft6iZMQpNZG6apayA=', 'smithi107.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBApDXY7Q4ol63+pX3d8bnYgsbpILjg7ASGCTyNn+RP4u2qRUH9wCf/c5DvxJr6BO+tMIZskweZhBwboHNAZ9grA='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'thrashosds': {'timeout': 1200}}, {'workunit': {'clients': {'client.0': ['rbd/rbd-nbd.sh']}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a', 'timestamp': '2024-09-10_21:16:03', 'tube': 'smithi', 'user': 'teuthology', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856'} 2024-09-11T09:49:27.830 DEBUG:teuthology.orchestra.run.smithi017:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-11T09:49:27.854 DEBUG:teuthology.orchestra.run.smithi096:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-11T09:49:27.856 DEBUG:teuthology.orchestra.run.smithi107:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-11T09:49:27.873 DEBUG:teuthology.orchestra.run.smithi017:> sudo install -d -m0777 -- /var/run/ceph 2024-09-11T09:49:27.909 DEBUG:teuthology.orchestra.run.smithi096:> sudo install -d -m0777 -- /var/run/ceph 2024-09-11T09:49:27.912 DEBUG:teuthology.orchestra.run.smithi107:> sudo install -d -m0777 -- /var/run/ceph 2024-09-11T09:49:27.939 DEBUG:teuthology.orchestra.run.smithi017:> set -ex 2024-09-11T09:49:27.939 DEBUG:teuthology.orchestra.run.smithi017:> dd if=/scratch_devs of=/dev/stdout 2024-09-11T09:49:27.990 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-11T09:49:27.990 DEBUG:teuthology.orchestra.run.smithi017:> stat /dev/vg_nvme/lv_1 2024-09-11T09:49:28.045 INFO:teuthology.orchestra.run.smithi017.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-11T09:49:28.045 INFO:teuthology.orchestra.run.smithi017.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-11T09:49:28.046 INFO:teuthology.orchestra.run.smithi017.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-09-11T09:49:28.046 INFO:teuthology.orchestra.run.smithi017.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-11T09:49:28.046 INFO:teuthology.orchestra.run.smithi017.stdout:Context: system_u:object_r:device_t:s0 2024-09-11T09:49:28.046 INFO:teuthology.orchestra.run.smithi017.stdout:Access: 2024-09-11 09:49:01.426608125 +0000 2024-09-11T09:49:28.046 INFO:teuthology.orchestra.run.smithi017.stdout:Modify: 2024-09-11 09:46:34.134254066 +0000 2024-09-11T09:49:28.046 INFO:teuthology.orchestra.run.smithi017.stdout:Change: 2024-09-11 09:46:34.134254066 +0000 2024-09-11T09:49:28.046 INFO:teuthology.orchestra.run.smithi017.stdout: Birth: 2024-09-11 09:46:34.134254066 +0000 2024-09-11T09:49:28.046 DEBUG:teuthology.orchestra.run.smithi017:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-11T09:49:28.112 INFO:teuthology.orchestra.run.smithi017.stderr:1+0 records in 2024-09-11T09:49:28.112 INFO:teuthology.orchestra.run.smithi017.stderr:1+0 records out 2024-09-11T09:49:28.112 INFO:teuthology.orchestra.run.smithi017.stderr:512 bytes copied, 0.000109644 s, 4.7 MB/s 2024-09-11T09:49:28.113 DEBUG:teuthology.orchestra.run.smithi017:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-11T09:49:28.170 DEBUG:teuthology.orchestra.run.smithi017:> stat /dev/vg_nvme/lv_2 2024-09-11T09:49:28.226 INFO:teuthology.orchestra.run.smithi017.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-11T09:49:28.227 INFO:teuthology.orchestra.run.smithi017.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-11T09:49:28.227 INFO:teuthology.orchestra.run.smithi017.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-09-11T09:49:28.227 INFO:teuthology.orchestra.run.smithi017.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-11T09:49:28.227 INFO:teuthology.orchestra.run.smithi017.stdout:Context: system_u:object_r:device_t:s0 2024-09-11T09:49:28.227 INFO:teuthology.orchestra.run.smithi017.stdout:Access: 2024-09-11 09:49:01.427608115 +0000 2024-09-11T09:49:28.227 INFO:teuthology.orchestra.run.smithi017.stdout:Modify: 2024-09-11 09:46:34.463250758 +0000 2024-09-11T09:49:28.227 INFO:teuthology.orchestra.run.smithi017.stdout:Change: 2024-09-11 09:46:34.463250758 +0000 2024-09-11T09:49:28.227 INFO:teuthology.orchestra.run.smithi017.stdout: Birth: 2024-09-11 09:46:34.463250758 +0000 2024-09-11T09:49:28.227 DEBUG:teuthology.orchestra.run.smithi017:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-11T09:49:28.291 INFO:teuthology.orchestra.run.smithi017.stderr:1+0 records in 2024-09-11T09:49:28.291 INFO:teuthology.orchestra.run.smithi017.stderr:1+0 records out 2024-09-11T09:49:28.291 INFO:teuthology.orchestra.run.smithi017.stderr:512 bytes copied, 9.1016e-05 s, 5.6 MB/s 2024-09-11T09:49:28.292 DEBUG:teuthology.orchestra.run.smithi017:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-11T09:49:28.349 DEBUG:teuthology.orchestra.run.smithi017:> stat /dev/vg_nvme/lv_3 2024-09-11T09:49:28.405 INFO:teuthology.orchestra.run.smithi017.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-11T09:49:28.405 INFO:teuthology.orchestra.run.smithi017.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-11T09:49:28.405 INFO:teuthology.orchestra.run.smithi017.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-09-11T09:49:28.405 INFO:teuthology.orchestra.run.smithi017.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-11T09:49:28.405 INFO:teuthology.orchestra.run.smithi017.stdout:Context: system_u:object_r:device_t:s0 2024-09-11T09:49:28.405 INFO:teuthology.orchestra.run.smithi017.stdout:Access: 2024-09-11 09:49:01.427608115 +0000 2024-09-11T09:49:28.405 INFO:teuthology.orchestra.run.smithi017.stdout:Modify: 2024-09-11 09:46:34.785247521 +0000 2024-09-11T09:49:28.405 INFO:teuthology.orchestra.run.smithi017.stdout:Change: 2024-09-11 09:46:34.785247521 +0000 2024-09-11T09:49:28.405 INFO:teuthology.orchestra.run.smithi017.stdout: Birth: 2024-09-11 09:46:34.785247521 +0000 2024-09-11T09:49:28.406 DEBUG:teuthology.orchestra.run.smithi017:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-11T09:49:28.469 INFO:teuthology.orchestra.run.smithi017.stderr:1+0 records in 2024-09-11T09:49:28.469 INFO:teuthology.orchestra.run.smithi017.stderr:1+0 records out 2024-09-11T09:49:28.469 INFO:teuthology.orchestra.run.smithi017.stderr:512 bytes copied, 8.341e-05 s, 6.1 MB/s 2024-09-11T09:49:28.470 DEBUG:teuthology.orchestra.run.smithi017:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-11T09:49:28.528 DEBUG:teuthology.orchestra.run.smithi017:> stat /dev/vg_nvme/lv_4 2024-09-11T09:49:28.583 INFO:teuthology.orchestra.run.smithi017.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-11T09:49:28.584 INFO:teuthology.orchestra.run.smithi017.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-11T09:49:28.584 INFO:teuthology.orchestra.run.smithi017.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-09-11T09:49:28.584 INFO:teuthology.orchestra.run.smithi017.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-11T09:49:28.584 INFO:teuthology.orchestra.run.smithi017.stdout:Context: system_u:object_r:device_t:s0 2024-09-11T09:49:28.584 INFO:teuthology.orchestra.run.smithi017.stdout:Access: 2024-09-11 09:49:01.427608115 +0000 2024-09-11T09:49:28.584 INFO:teuthology.orchestra.run.smithi017.stdout:Modify: 2024-09-11 09:46:35.118244172 +0000 2024-09-11T09:49:28.584 INFO:teuthology.orchestra.run.smithi017.stdout:Change: 2024-09-11 09:46:35.118244172 +0000 2024-09-11T09:49:28.584 INFO:teuthology.orchestra.run.smithi017.stdout: Birth: 2024-09-11 09:46:35.118244172 +0000 2024-09-11T09:49:28.584 DEBUG:teuthology.orchestra.run.smithi017:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-11T09:49:28.648 INFO:teuthology.orchestra.run.smithi017.stderr:1+0 records in 2024-09-11T09:49:28.648 INFO:teuthology.orchestra.run.smithi017.stderr:1+0 records out 2024-09-11T09:49:28.648 INFO:teuthology.orchestra.run.smithi017.stderr:512 bytes copied, 9.7054e-05 s, 5.3 MB/s 2024-09-11T09:49:28.650 DEBUG:teuthology.orchestra.run.smithi017:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-11T09:49:28.706 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-11T09:49:28.707 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-09-11T09:49:28.707 DEBUG:teuthology.orchestra.run.smithi096:> dd if=/scratch_devs of=/dev/stdout 2024-09-11T09:49:28.725 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-11T09:49:28.725 DEBUG:teuthology.orchestra.run.smithi096:> stat /dev/vg_nvme/lv_1 2024-09-11T09:49:28.782 INFO:teuthology.orchestra.run.smithi096.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-11T09:49:28.782 INFO:teuthology.orchestra.run.smithi096.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-11T09:49:28.782 INFO:teuthology.orchestra.run.smithi096.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-09-11T09:49:28.782 INFO:teuthology.orchestra.run.smithi096.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-11T09:49:28.782 INFO:teuthology.orchestra.run.smithi096.stdout:Context: system_u:object_r:device_t:s0 2024-09-11T09:49:28.782 INFO:teuthology.orchestra.run.smithi096.stdout:Access: 2024-09-11 09:49:21.950257835 +0000 2024-09-11T09:49:28.782 INFO:teuthology.orchestra.run.smithi096.stdout:Modify: 2024-09-11 09:46:26.990703732 +0000 2024-09-11T09:49:28.782 INFO:teuthology.orchestra.run.smithi096.stdout:Change: 2024-09-11 09:46:26.990703732 +0000 2024-09-11T09:49:28.782 INFO:teuthology.orchestra.run.smithi096.stdout: Birth: 2024-09-11 09:46:26.990703732 +0000 2024-09-11T09:49:28.783 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-11T09:49:28.847 INFO:teuthology.orchestra.run.smithi096.stderr:1+0 records in 2024-09-11T09:49:28.847 INFO:teuthology.orchestra.run.smithi096.stderr:1+0 records out 2024-09-11T09:49:28.847 INFO:teuthology.orchestra.run.smithi096.stderr:512 bytes copied, 0.000267425 s, 1.9 MB/s 2024-09-11T09:49:28.848 DEBUG:teuthology.orchestra.run.smithi096:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-11T09:49:28.905 DEBUG:teuthology.orchestra.run.smithi096:> stat /dev/vg_nvme/lv_2 2024-09-11T09:49:28.961 INFO:teuthology.orchestra.run.smithi096.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-11T09:49:28.961 INFO:teuthology.orchestra.run.smithi096.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-11T09:49:28.962 INFO:teuthology.orchestra.run.smithi096.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-09-11T09:49:28.962 INFO:teuthology.orchestra.run.smithi096.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-11T09:49:28.962 INFO:teuthology.orchestra.run.smithi096.stdout:Context: system_u:object_r:device_t:s0 2024-09-11T09:49:28.962 INFO:teuthology.orchestra.run.smithi096.stdout:Access: 2024-09-11 09:49:21.950257835 +0000 2024-09-11T09:49:28.962 INFO:teuthology.orchestra.run.smithi096.stdout:Modify: 2024-09-11 09:46:27.317695964 +0000 2024-09-11T09:49:28.962 INFO:teuthology.orchestra.run.smithi096.stdout:Change: 2024-09-11 09:46:27.317695964 +0000 2024-09-11T09:49:28.962 INFO:teuthology.orchestra.run.smithi096.stdout: Birth: 2024-09-11 09:46:27.317695964 +0000 2024-09-11T09:49:28.962 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-11T09:49:29.025 INFO:teuthology.orchestra.run.smithi096.stderr:1+0 records in 2024-09-11T09:49:29.025 INFO:teuthology.orchestra.run.smithi096.stderr:1+0 records out 2024-09-11T09:49:29.025 INFO:teuthology.orchestra.run.smithi096.stderr:512 bytes copied, 0.000194925 s, 2.6 MB/s 2024-09-11T09:49:29.026 DEBUG:teuthology.orchestra.run.smithi096:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-11T09:49:29.082 DEBUG:teuthology.orchestra.run.smithi096:> stat /dev/vg_nvme/lv_3 2024-09-11T09:49:29.137 INFO:teuthology.orchestra.run.smithi096.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-11T09:49:29.138 INFO:teuthology.orchestra.run.smithi096.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-11T09:49:29.138 INFO:teuthology.orchestra.run.smithi096.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-09-11T09:49:29.138 INFO:teuthology.orchestra.run.smithi096.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-11T09:49:29.138 INFO:teuthology.orchestra.run.smithi096.stdout:Context: system_u:object_r:device_t:s0 2024-09-11T09:49:29.138 INFO:teuthology.orchestra.run.smithi096.stdout:Access: 2024-09-11 09:49:21.951257812 +0000 2024-09-11T09:49:29.138 INFO:teuthology.orchestra.run.smithi096.stdout:Modify: 2024-09-11 09:46:27.652688006 +0000 2024-09-11T09:49:29.138 INFO:teuthology.orchestra.run.smithi096.stdout:Change: 2024-09-11 09:46:27.652688006 +0000 2024-09-11T09:49:29.138 INFO:teuthology.orchestra.run.smithi096.stdout: Birth: 2024-09-11 09:46:27.652688006 +0000 2024-09-11T09:49:29.138 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-11T09:49:29.201 INFO:teuthology.orchestra.run.smithi096.stderr:1+0 records in 2024-09-11T09:49:29.201 INFO:teuthology.orchestra.run.smithi096.stderr:1+0 records out 2024-09-11T09:49:29.201 INFO:teuthology.orchestra.run.smithi096.stderr:512 bytes copied, 0.000256873 s, 2.0 MB/s 2024-09-11T09:49:29.202 DEBUG:teuthology.orchestra.run.smithi096:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-11T09:49:29.259 DEBUG:teuthology.orchestra.run.smithi096:> stat /dev/vg_nvme/lv_4 2024-09-11T09:49:29.315 INFO:teuthology.orchestra.run.smithi096.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-11T09:49:29.315 INFO:teuthology.orchestra.run.smithi096.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-11T09:49:29.315 INFO:teuthology.orchestra.run.smithi096.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-09-11T09:49:29.316 INFO:teuthology.orchestra.run.smithi096.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-11T09:49:29.316 INFO:teuthology.orchestra.run.smithi096.stdout:Context: system_u:object_r:device_t:s0 2024-09-11T09:49:29.316 INFO:teuthology.orchestra.run.smithi096.stdout:Access: 2024-09-11 09:49:21.951257812 +0000 2024-09-11T09:49:29.316 INFO:teuthology.orchestra.run.smithi096.stdout:Modify: 2024-09-11 09:46:27.970680452 +0000 2024-09-11T09:49:29.316 INFO:teuthology.orchestra.run.smithi096.stdout:Change: 2024-09-11 09:46:27.970680452 +0000 2024-09-11T09:49:29.316 INFO:teuthology.orchestra.run.smithi096.stdout: Birth: 2024-09-11 09:46:27.970680452 +0000 2024-09-11T09:49:29.316 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-11T09:49:29.378 INFO:teuthology.orchestra.run.smithi096.stderr:1+0 records in 2024-09-11T09:49:29.379 INFO:teuthology.orchestra.run.smithi096.stderr:1+0 records out 2024-09-11T09:49:29.379 INFO:teuthology.orchestra.run.smithi096.stderr:512 bytes copied, 0.000214159 s, 2.4 MB/s 2024-09-11T09:49:29.380 DEBUG:teuthology.orchestra.run.smithi096:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-11T09:49:29.436 INFO:tasks.ceph:osd dev map: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2024-09-11T09:49:29.436 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi017.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi096.front.sepia.ceph.com'): {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'}} 2024-09-11T09:49:29.437 INFO:tasks.ceph:Generating config... 2024-09-11T09:49:29.438 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-09-11T09:49:29.438 INFO:tasks.ceph:[global] mon warn on pool no app = False 2024-09-11T09:49:29.438 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-09-11T09:49:29.438 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-09-11T09:49:29.438 INFO:tasks.ceph:[mgr] debug ms = 1 2024-09-11T09:49:29.438 INFO:tasks.ceph:[mon] debug mon = 20 2024-09-11T09:49:29.438 INFO:tasks.ceph:[mon] debug ms = 1 2024-09-11T09:49:29.438 INFO:tasks.ceph:[mon] debug paxos = 20 2024-09-11T09:49:29.438 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-09-11T09:49:29.439 INFO:tasks.ceph:[osd] bluestore compression algorithm = snappy 2024-09-11T09:49:29.439 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2024-09-11T09:49:29.439 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-09-11T09:49:29.439 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2024-09-11T09:49:29.439 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2024-09-11T09:49:29.439 INFO:tasks.ceph:[osd] debug ms = 1 2024-09-11T09:49:29.439 INFO:tasks.ceph:[osd] debug osd = 20 2024-09-11T09:49:29.439 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2024-09-11T09:49:29.439 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-09-11T09:49:29.439 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-09-11T09:49:29.439 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-09-11T09:49:29.439 INFO:tasks.ceph:[osd] osd blocked scrub grace period = 3600 2024-09-11T09:49:29.439 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-09-11T09:49:29.439 INFO:tasks.ceph:[osd] osd max markdown count = 1000 2024-09-11T09:49:29.440 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-09-11T09:49:29.440 INFO:tasks.ceph:Setting up mon.a... 2024-09-11T09:49:29.440 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-09-11T09:49:29.491 INFO:teuthology.orchestra.run.smithi017.stdout:creating /etc/ceph/ceph.keyring 2024-09-11T09:49:29.494 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-09-11T09:49:29.541 DEBUG:teuthology.orchestra.run.smithi017:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-11T09:49:29.568 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.17'), ('mon.c', '[v2:172.21.15.17:3301,v1:172.21.15.17:6790]'), ('mon.b', '172.21.15.96')] 2024-09-11T09:49:29.568 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.17,[v2:172.21.15.17:3301,v1:172.21.15.17:6790],172.21.15.96', 'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok'}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2024-09-11T09:49:29.569 DEBUG:teuthology.orchestra.run.smithi017:> set -ex 2024-09-11T09:49:29.569 DEBUG:teuthology.orchestra.run.smithi017:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-09-11T09:49:29.624 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.17 --addv c '[v2:172.21.15.17:3301,v1:172.21.15.17:6790]' --add b 172.21.15.96 --print /home/ubuntu/cephtest/ceph.monmap 2024-09-11T09:49:29.712 INFO:teuthology.orchestra.run.smithi017.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-09-11T09:49:29.712 INFO:teuthology.orchestra.run.smithi017.stdout:monmaptool: generated fsid a2f1daec-d3b0-4bf5-82c8-72ecbb93d4e4 2024-09-11T09:49:29.712 INFO:teuthology.orchestra.run.smithi017.stdout:setting min_mon_release = quincy 2024-09-11T09:49:29.712 INFO:teuthology.orchestra.run.smithi017.stdout:epoch 0 2024-09-11T09:49:29.712 INFO:teuthology.orchestra.run.smithi017.stdout:fsid a2f1daec-d3b0-4bf5-82c8-72ecbb93d4e4 2024-09-11T09:49:29.713 INFO:teuthology.orchestra.run.smithi017.stdout:last_changed 2024-09-11T09:49:29.713857+0000 2024-09-11T09:49:29.713 INFO:teuthology.orchestra.run.smithi017.stdout:created 2024-09-11T09:49:29.713857+0000 2024-09-11T09:49:29.713 INFO:teuthology.orchestra.run.smithi017.stdout:min_mon_release 17 (quincy) 2024-09-11T09:49:29.713 INFO:teuthology.orchestra.run.smithi017.stdout:election_strategy: 1 2024-09-11T09:49:29.713 INFO:teuthology.orchestra.run.smithi017.stdout:0: [v2:172.21.15.17:3300/0,v1:172.21.15.17:6789/0] mon.a 2024-09-11T09:49:29.713 INFO:teuthology.orchestra.run.smithi017.stdout:1: [v2:172.21.15.96:3300/0,v1:172.21.15.96:6789/0] mon.b 2024-09-11T09:49:29.713 INFO:teuthology.orchestra.run.smithi017.stdout:2: [v2:172.21.15.17:3301/0,v1:172.21.15.17:6790/0] mon.c 2024-09-11T09:49:29.713 INFO:teuthology.orchestra.run.smithi017.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-09-11T09:49:29.714 DEBUG:teuthology.orchestra.run.smithi017:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-09-11T09:49:29.730 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID a2f1daec-d3b0-4bf5-82c8-72ecbb93d4e4... 2024-09-11T09:49:29.730 DEBUG:teuthology.orchestra.run.smithi017:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-11T09:49:29.772 DEBUG:teuthology.orchestra.run.smithi096:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-11T09:49:29.775 DEBUG:teuthology.orchestra.run.smithi107:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-11T09:49:29.814 INFO:teuthology.orchestra.run.smithi017.stdout:[global] 2024-09-11T09:49:29.814 INFO:teuthology.orchestra.run.smithi017.stdout: chdir = "" 2024-09-11T09:49:29.814 INFO:teuthology.orchestra.run.smithi017.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-11T09:49:29.814 INFO:teuthology.orchestra.run.smithi017.stdout: auth supported = cephx 2024-09-11T09:49:29.814 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-11T09:49:29.814 INFO:teuthology.orchestra.run.smithi017.stdout: filestore xattr use omap = true 2024-09-11T09:49:29.814 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-11T09:49:29.814 INFO:teuthology.orchestra.run.smithi017.stdout: mon clock drift allowed = 1.000 2024-09-11T09:49:29.814 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-11T09:49:29.815 INFO:teuthology.orchestra.run.smithi017.stdout: osd crush chooseleaf type = 0 2024-09-11T09:49:29.815 INFO:teuthology.orchestra.run.smithi017.stdout: auth debug = true 2024-09-11T09:49:29.815 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-11T09:49:29.815 INFO:teuthology.orchestra.run.smithi017.stdout: ms die on old message = true 2024-09-11T09:49:29.815 INFO:teuthology.orchestra.run.smithi017.stdout: ms die on bug = true 2024-09-11T09:49:29.815 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-11T09:49:29.815 INFO:teuthology.orchestra.run.smithi017.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-11T09:49:29.815 INFO:teuthology.orchestra.run.smithi017.stdout: mon pg warn max object skew = 0 2024-09-11T09:49:29.815 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-11T09:49:29.815 INFO:teuthology.orchestra.run.smithi017.stdout: # disable pg_autoscaler by default for new pools 2024-09-11T09:49:29.815 INFO:teuthology.orchestra.run.smithi017.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-11T09:49:29.815 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-11T09:49:29.815 INFO:teuthology.orchestra.run.smithi017.stdout: osd pool default size = 2 2024-09-11T09:49:29.816 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-11T09:49:29.816 INFO:teuthology.orchestra.run.smithi017.stdout: mon osd allow primary affinity = true 2024-09-11T09:49:29.816 INFO:teuthology.orchestra.run.smithi017.stdout: mon osd allow pg remap = true 2024-09-11T09:49:29.816 INFO:teuthology.orchestra.run.smithi017.stdout: mon warn on legacy crush tunables = false 2024-09-11T09:49:29.816 INFO:teuthology.orchestra.run.smithi017.stdout: mon warn on crush straw calc version zero = false 2024-09-11T09:49:29.816 INFO:teuthology.orchestra.run.smithi017.stdout: mon warn on no sortbitwise = false 2024-09-11T09:49:29.816 INFO:teuthology.orchestra.run.smithi017.stdout: mon warn on osd down out interval zero = false 2024-09-11T09:49:29.816 INFO:teuthology.orchestra.run.smithi017.stdout: mon warn on too few osds = false 2024-09-11T09:49:29.816 INFO:teuthology.orchestra.run.smithi017.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-11T09:49:29.816 INFO:teuthology.orchestra.run.smithi017.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-11T09:49:29.816 INFO:teuthology.orchestra.run.smithi017.stdout: mon_allow_pool_size_one = true 2024-09-11T09:49:29.816 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-11T09:49:29.816 INFO:teuthology.orchestra.run.smithi017.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-11T09:49:29.817 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-11T09:49:29.817 INFO:teuthology.orchestra.run.smithi017.stdout: osd default data pool replay window = 5 2024-09-11T09:49:29.817 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-11T09:49:29.817 INFO:teuthology.orchestra.run.smithi017.stdout: mon allow pool delete = true 2024-09-11T09:49:29.817 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-11T09:49:29.817 INFO:teuthology.orchestra.run.smithi017.stdout: mon cluster log file level = debug 2024-09-11T09:49:29.817 INFO:teuthology.orchestra.run.smithi017.stdout: debug asserts on shutdown = true 2024-09-11T09:49:29.817 INFO:teuthology.orchestra.run.smithi017.stdout: mon health detail to clog = false 2024-09-11T09:49:29.817 INFO:teuthology.orchestra.run.smithi017.stdout: mon host = "172.21.15.17,[v2:172.21.15.17:3301,v1:172.21.15.17:6790],172.21.15.96" 2024-09-11T09:49:29.817 INFO:teuthology.orchestra.run.smithi017.stdout: mon client directed command retry = 5 2024-09-11T09:49:29.817 INFO:teuthology.orchestra.run.smithi017.stdout: mon warn on pool no app = False 2024-09-11T09:49:29.817 INFO:teuthology.orchestra.run.smithi017.stdout: ms inject socket failures = 5000 2024-09-11T09:49:29.817 INFO:teuthology.orchestra.run.smithi017.stdout: fsid = a2f1daec-d3b0-4bf5-82c8-72ecbb93d4e4 2024-09-11T09:49:29.817 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-11T09:49:29.818 INFO:teuthology.orchestra.run.smithi017.stdout:[osd] 2024-09-11T09:49:29.818 INFO:teuthology.orchestra.run.smithi017.stdout: osd journal size = 100 2024-09-11T09:49:29.818 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-11T09:49:29.818 INFO:teuthology.orchestra.run.smithi017.stdout: osd scrub load threshold = 5.0 2024-09-11T09:49:29.818 INFO:teuthology.orchestra.run.smithi017.stdout: osd scrub max interval = 600 2024-09-11T09:49:29.818 INFO:teuthology.orchestra.run.smithi017.stdout: osd mclock profile = high_recovery_ops 2024-09-11T09:49:29.818 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-11T09:49:29.818 INFO:teuthology.orchestra.run.smithi017.stdout: osd recover clone overlap = true 2024-09-11T09:49:29.818 INFO:teuthology.orchestra.run.smithi017.stdout: osd recovery max chunk = 1048576 2024-09-11T09:49:29.818 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-11T09:49:29.818 INFO:teuthology.orchestra.run.smithi017.stdout: osd debug shutdown = true 2024-09-11T09:49:29.818 INFO:teuthology.orchestra.run.smithi017.stdout: osd debug op order = true 2024-09-11T09:49:29.818 INFO:teuthology.orchestra.run.smithi017.stdout: osd debug verify stray on activate = true 2024-09-11T09:49:29.819 INFO:teuthology.orchestra.run.smithi017.stdout: osd debug trim objects = true 2024-09-11T09:49:29.819 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-11T09:49:29.819 INFO:teuthology.orchestra.run.smithi017.stdout: osd open classes on start = true 2024-09-11T09:49:29.819 INFO:teuthology.orchestra.run.smithi017.stdout: osd debug pg log writeout = true 2024-09-11T09:49:29.819 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-11T09:49:29.819 INFO:teuthology.orchestra.run.smithi017.stdout: osd deep scrub update digest min age = 30 2024-09-11T09:49:29.819 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-11T09:49:29.819 INFO:teuthology.orchestra.run.smithi017.stdout: osd map max advance = 10 2024-09-11T09:49:29.819 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-11T09:49:29.819 INFO:teuthology.orchestra.run.smithi017.stdout: journal zero on create = true 2024-09-11T09:49:29.819 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-11T09:49:29.819 INFO:teuthology.orchestra.run.smithi017.stdout: filestore ondisk finisher threads = 3 2024-09-11T09:49:29.819 INFO:teuthology.orchestra.run.smithi017.stdout: filestore apply finisher threads = 3 2024-09-11T09:49:29.820 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-11T09:49:29.820 INFO:teuthology.orchestra.run.smithi017.stdout: bdev debug aio = true 2024-09-11T09:49:29.820 INFO:teuthology.orchestra.run.smithi017.stdout: osd debug misdirected ops = true 2024-09-11T09:49:29.820 INFO:teuthology.orchestra.run.smithi017.stdout: bluestore block size = 96636764160 2024-09-11T09:49:29.820 INFO:teuthology.orchestra.run.smithi017.stdout: bluestore compression algorithm = snappy 2024-09-11T09:49:29.820 INFO:teuthology.orchestra.run.smithi017.stdout: bluestore compression mode = aggressive 2024-09-11T09:49:29.820 INFO:teuthology.orchestra.run.smithi017.stdout: bluestore fsck on mount = True 2024-09-11T09:49:29.820 INFO:teuthology.orchestra.run.smithi017.stdout: debug bluefs = 1/20 2024-09-11T09:49:29.820 INFO:teuthology.orchestra.run.smithi017.stdout: debug bluestore = 1/20 2024-09-11T09:49:29.820 INFO:teuthology.orchestra.run.smithi017.stdout: debug ms = 1 2024-09-11T09:49:29.820 INFO:teuthology.orchestra.run.smithi017.stdout: debug osd = 20 2024-09-11T09:49:29.820 INFO:teuthology.orchestra.run.smithi017.stdout: debug rocksdb = 4/10 2024-09-11T09:49:29.821 INFO:teuthology.orchestra.run.smithi017.stdout: mon osd backfillfull_ratio = 0.85 2024-09-11T09:49:29.821 INFO:teuthology.orchestra.run.smithi017.stdout: mon osd full ratio = 0.9 2024-09-11T09:49:29.821 INFO:teuthology.orchestra.run.smithi017.stdout: mon osd nearfull ratio = 0.8 2024-09-11T09:49:29.821 INFO:teuthology.orchestra.run.smithi017.stdout: osd blocked scrub grace period = 3600 2024-09-11T09:49:29.821 INFO:teuthology.orchestra.run.smithi017.stdout: osd failsafe full ratio = 0.95 2024-09-11T09:49:29.821 INFO:teuthology.orchestra.run.smithi017.stdout: osd max markdown count = 1000 2024-09-11T09:49:29.821 INFO:teuthology.orchestra.run.smithi017.stdout: osd objectstore = bluestore 2024-09-11T09:49:29.821 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-11T09:49:29.821 INFO:teuthology.orchestra.run.smithi017.stdout:[mgr] 2024-09-11T09:49:29.821 INFO:teuthology.orchestra.run.smithi017.stdout: debug ms = 1 2024-09-11T09:49:29.821 INFO:teuthology.orchestra.run.smithi017.stdout: debug mgr = 20 2024-09-11T09:49:29.821 INFO:teuthology.orchestra.run.smithi017.stdout: debug mon = 20 2024-09-11T09:49:29.821 INFO:teuthology.orchestra.run.smithi017.stdout: debug auth = 20 2024-09-11T09:49:29.822 INFO:teuthology.orchestra.run.smithi017.stdout: mon reweight min pgs per osd = 4 2024-09-11T09:49:29.822 INFO:teuthology.orchestra.run.smithi017.stdout: mon reweight min bytes per osd = 10 2024-09-11T09:49:29.822 INFO:teuthology.orchestra.run.smithi017.stdout: mgr/telemetry/nag = false 2024-09-11T09:49:29.822 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-11T09:49:29.822 INFO:teuthology.orchestra.run.smithi017.stdout:[mon] 2024-09-11T09:49:29.822 INFO:teuthology.orchestra.run.smithi017.stdout: debug ms = 1 2024-09-11T09:49:29.822 INFO:teuthology.orchestra.run.smithi017.stdout: debug mon = 20 2024-09-11T09:49:29.822 INFO:teuthology.orchestra.run.smithi017.stdout: debug paxos = 20 2024-09-11T09:49:29.822 INFO:teuthology.orchestra.run.smithi017.stdout: debug auth = 20 2024-09-11T09:49:29.822 INFO:teuthology.orchestra.run.smithi017.stdout: mon data avail warn = 5 2024-09-11T09:49:29.822 INFO:teuthology.orchestra.run.smithi017.stdout: mon mgr mkfs grace = 240 2024-09-11T09:49:29.822 INFO:teuthology.orchestra.run.smithi017.stdout: mon reweight min pgs per osd = 4 2024-09-11T09:49:29.822 INFO:teuthology.orchestra.run.smithi017.stdout: mon osd reporter subtree level = osd 2024-09-11T09:49:29.823 INFO:teuthology.orchestra.run.smithi017.stdout: mon osd prime pg temp = true 2024-09-11T09:49:29.823 INFO:teuthology.orchestra.run.smithi017.stdout: mon reweight min bytes per osd = 10 2024-09-11T09:49:29.823 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-11T09:49:29.823 INFO:teuthology.orchestra.run.smithi017.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-11T09:49:29.823 INFO:teuthology.orchestra.run.smithi017.stdout: auth mon ticket ttl = 660 # 11m 2024-09-11T09:49:29.823 INFO:teuthology.orchestra.run.smithi017.stdout: auth service ticket ttl = 240 # 4m 2024-09-11T09:49:29.823 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-11T09:49:29.823 INFO:teuthology.orchestra.run.smithi017.stdout: # don't complain about insecure global_id in the test suite 2024-09-11T09:49:29.823 INFO:teuthology.orchestra.run.smithi017.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-11T09:49:29.823 INFO:teuthology.orchestra.run.smithi017.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-11T09:49:29.823 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-11T09:49:29.823 INFO:teuthology.orchestra.run.smithi017.stdout: # 1m isn't quite enough 2024-09-11T09:49:29.823 INFO:teuthology.orchestra.run.smithi017.stdout: mon_down_mkfs_grace = 2m 2024-09-11T09:49:29.823 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-11T09:49:29.824 INFO:teuthology.orchestra.run.smithi017.stdout: mon_warn_on_filestore_osds = false 2024-09-11T09:49:29.824 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-11T09:49:29.824 INFO:teuthology.orchestra.run.smithi017.stdout:[client] 2024-09-11T09:49:29.824 INFO:teuthology.orchestra.run.smithi017.stdout: rgw cache enabled = true 2024-09-11T09:49:29.824 INFO:teuthology.orchestra.run.smithi017.stdout: rgw enable ops log = true 2024-09-11T09:49:29.824 INFO:teuthology.orchestra.run.smithi017.stdout: rgw enable usage log = true 2024-09-11T09:49:29.824 INFO:teuthology.orchestra.run.smithi017.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-11T09:49:29.824 INFO:teuthology.orchestra.run.smithi017.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-11T09:49:29.824 INFO:teuthology.orchestra.run.smithi017.stdout:[mon.a] 2024-09-11T09:49:29.824 INFO:teuthology.orchestra.run.smithi017.stdout:[mon.c] 2024-09-11T09:49:29.824 INFO:teuthology.orchestra.run.smithi017.stdout:[mon.b] 2024-09-11T09:49:29.825 INFO:teuthology.orchestra.run.smithi096.stdout:[global] 2024-09-11T09:49:29.825 INFO:teuthology.orchestra.run.smithi096.stdout: chdir = "" 2024-09-11T09:49:29.825 INFO:teuthology.orchestra.run.smithi096.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-11T09:49:29.825 INFO:teuthology.orchestra.run.smithi096.stdout: auth supported = cephx 2024-09-11T09:49:29.825 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-11T09:49:29.825 INFO:teuthology.orchestra.run.smithi096.stdout: filestore xattr use omap = true 2024-09-11T09:49:29.825 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-11T09:49:29.825 INFO:teuthology.orchestra.run.smithi096.stdout: mon clock drift allowed = 1.000 2024-09-11T09:49:29.825 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-11T09:49:29.825 INFO:teuthology.orchestra.run.smithi096.stdout: osd crush chooseleaf type = 0 2024-09-11T09:49:29.825 INFO:teuthology.orchestra.run.smithi096.stdout: auth debug = true 2024-09-11T09:49:29.826 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-11T09:49:29.826 INFO:teuthology.orchestra.run.smithi096.stdout: ms die on old message = true 2024-09-11T09:49:29.826 INFO:teuthology.orchestra.run.smithi096.stdout: ms die on bug = true 2024-09-11T09:49:29.826 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-11T09:49:29.826 INFO:teuthology.orchestra.run.smithi096.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-11T09:49:29.826 INFO:teuthology.orchestra.run.smithi096.stdout: mon pg warn max object skew = 0 2024-09-11T09:49:29.826 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-11T09:49:29.826 INFO:teuthology.orchestra.run.smithi096.stdout: # disable pg_autoscaler by default for new pools 2024-09-11T09:49:29.826 INFO:teuthology.orchestra.run.smithi096.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-11T09:49:29.826 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-11T09:49:29.826 INFO:teuthology.orchestra.run.smithi096.stdout: osd pool default size = 2 2024-09-11T09:49:29.826 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-11T09:49:29.827 INFO:teuthology.orchestra.run.smithi096.stdout: mon osd allow primary affinity = true 2024-09-11T09:49:29.827 INFO:teuthology.orchestra.run.smithi096.stdout: mon osd allow pg remap = true 2024-09-11T09:49:29.827 INFO:teuthology.orchestra.run.smithi096.stdout: mon warn on legacy crush tunables = false 2024-09-11T09:49:29.827 INFO:teuthology.orchestra.run.smithi096.stdout: mon warn on crush straw calc version zero = false 2024-09-11T09:49:29.827 INFO:teuthology.orchestra.run.smithi096.stdout: mon warn on no sortbitwise = false 2024-09-11T09:49:29.827 INFO:teuthology.orchestra.run.smithi096.stdout: mon warn on osd down out interval zero = false 2024-09-11T09:49:29.827 INFO:teuthology.orchestra.run.smithi096.stdout: mon warn on too few osds = false 2024-09-11T09:49:29.827 INFO:teuthology.orchestra.run.smithi096.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-11T09:49:29.827 INFO:teuthology.orchestra.run.smithi096.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-11T09:49:29.827 INFO:teuthology.orchestra.run.smithi096.stdout: mon_allow_pool_size_one = true 2024-09-11T09:49:29.827 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-11T09:49:29.827 INFO:teuthology.orchestra.run.smithi096.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-11T09:49:29.827 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-11T09:49:29.827 INFO:teuthology.orchestra.run.smithi096.stdout: osd default data pool replay window = 5 2024-09-11T09:49:29.828 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-11T09:49:29.828 INFO:teuthology.orchestra.run.smithi096.stdout: mon allow pool delete = true 2024-09-11T09:49:29.828 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-11T09:49:29.828 INFO:teuthology.orchestra.run.smithi096.stdout: mon cluster log file level = debug 2024-09-11T09:49:29.828 INFO:teuthology.orchestra.run.smithi096.stdout: debug asserts on shutdown = true 2024-09-11T09:49:29.828 INFO:teuthology.orchestra.run.smithi096.stdout: mon health detail to clog = false 2024-09-11T09:49:29.828 INFO:teuthology.orchestra.run.smithi096.stdout: mon host = "172.21.15.17,[v2:172.21.15.17:3301,v1:172.21.15.17:6790],172.21.15.96" 2024-09-11T09:49:29.828 INFO:teuthology.orchestra.run.smithi096.stdout: mon client directed command retry = 5 2024-09-11T09:49:29.828 INFO:teuthology.orchestra.run.smithi096.stdout: mon warn on pool no app = False 2024-09-11T09:49:29.828 INFO:teuthology.orchestra.run.smithi096.stdout: ms inject socket failures = 5000 2024-09-11T09:49:29.828 INFO:teuthology.orchestra.run.smithi096.stdout: fsid = a2f1daec-d3b0-4bf5-82c8-72ecbb93d4e4 2024-09-11T09:49:29.828 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-11T09:49:29.829 INFO:teuthology.orchestra.run.smithi096.stdout:[osd] 2024-09-11T09:49:29.829 INFO:teuthology.orchestra.run.smithi096.stdout: osd journal size = 100 2024-09-11T09:49:29.829 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-11T09:49:29.829 INFO:teuthology.orchestra.run.smithi096.stdout: osd scrub load threshold = 5.0 2024-09-11T09:49:29.829 INFO:teuthology.orchestra.run.smithi096.stdout: osd scrub max interval = 600 2024-09-11T09:49:29.829 INFO:teuthology.orchestra.run.smithi096.stdout: osd mclock profile = high_recovery_ops 2024-09-11T09:49:29.829 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-11T09:49:29.829 INFO:teuthology.orchestra.run.smithi096.stdout: osd recover clone overlap = true 2024-09-11T09:49:29.829 INFO:teuthology.orchestra.run.smithi096.stdout: osd recovery max chunk = 1048576 2024-09-11T09:49:29.829 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-11T09:49:29.829 INFO:teuthology.orchestra.run.smithi096.stdout: osd debug shutdown = true 2024-09-11T09:49:29.829 INFO:teuthology.orchestra.run.smithi096.stdout: osd debug op order = true 2024-09-11T09:49:29.830 INFO:teuthology.orchestra.run.smithi096.stdout: osd debug verify stray on activate = true 2024-09-11T09:49:29.830 INFO:teuthology.orchestra.run.smithi096.stdout: osd debug trim objects = true 2024-09-11T09:49:29.830 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-11T09:49:29.830 INFO:teuthology.orchestra.run.smithi096.stdout: osd open classes on start = true 2024-09-11T09:49:29.830 INFO:teuthology.orchestra.run.smithi096.stdout: osd debug pg log writeout = true 2024-09-11T09:49:29.830 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-11T09:49:29.830 INFO:teuthology.orchestra.run.smithi096.stdout: osd deep scrub update digest min age = 30 2024-09-11T09:49:29.830 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-11T09:49:29.830 INFO:teuthology.orchestra.run.smithi096.stdout: osd map max advance = 10 2024-09-11T09:49:29.830 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-11T09:49:29.830 INFO:teuthology.orchestra.run.smithi096.stdout: journal zero on create = true 2024-09-11T09:49:29.830 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-11T09:49:29.830 INFO:teuthology.orchestra.run.smithi096.stdout: filestore ondisk finisher threads = 3 2024-09-11T09:49:29.831 INFO:teuthology.orchestra.run.smithi096.stdout: filestore apply finisher threads = 3 2024-09-11T09:49:29.831 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-11T09:49:29.831 INFO:teuthology.orchestra.run.smithi096.stdout: bdev debug aio = true 2024-09-11T09:49:29.831 INFO:teuthology.orchestra.run.smithi096.stdout: osd debug misdirected ops = true 2024-09-11T09:49:29.831 INFO:teuthology.orchestra.run.smithi096.stdout: bluestore block size = 96636764160 2024-09-11T09:49:29.831 INFO:teuthology.orchestra.run.smithi096.stdout: bluestore compression algorithm = snappy 2024-09-11T09:49:29.831 INFO:teuthology.orchestra.run.smithi096.stdout: bluestore compression mode = aggressive 2024-09-11T09:49:29.831 INFO:teuthology.orchestra.run.smithi096.stdout: bluestore fsck on mount = True 2024-09-11T09:49:29.831 INFO:teuthology.orchestra.run.smithi096.stdout: debug bluefs = 1/20 2024-09-11T09:49:29.831 INFO:teuthology.orchestra.run.smithi096.stdout: debug bluestore = 1/20 2024-09-11T09:49:29.831 INFO:teuthology.orchestra.run.smithi096.stdout: debug ms = 1 2024-09-11T09:49:29.831 INFO:teuthology.orchestra.run.smithi096.stdout: debug osd = 20 2024-09-11T09:49:29.832 INFO:teuthology.orchestra.run.smithi096.stdout: debug rocksdb = 4/10 2024-09-11T09:49:29.832 INFO:teuthology.orchestra.run.smithi096.stdout: mon osd backfillfull_ratio = 0.85 2024-09-11T09:49:29.832 INFO:teuthology.orchestra.run.smithi096.stdout: mon osd full ratio = 0.9 2024-09-11T09:49:29.832 INFO:teuthology.orchestra.run.smithi096.stdout: mon osd nearfull ratio = 0.8 2024-09-11T09:49:29.832 INFO:teuthology.orchestra.run.smithi096.stdout: osd blocked scrub grace period = 3600 2024-09-11T09:49:29.832 INFO:teuthology.orchestra.run.smithi096.stdout: osd failsafe full ratio = 0.95 2024-09-11T09:49:29.832 INFO:teuthology.orchestra.run.smithi096.stdout: osd max markdown count = 1000 2024-09-11T09:49:29.832 INFO:teuthology.orchestra.run.smithi096.stdout: osd objectstore = bluestore 2024-09-11T09:49:29.832 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-11T09:49:29.832 INFO:teuthology.orchestra.run.smithi096.stdout:[mgr] 2024-09-11T09:49:29.832 INFO:teuthology.orchestra.run.smithi096.stdout: debug ms = 1 2024-09-11T09:49:29.832 INFO:teuthology.orchestra.run.smithi096.stdout: debug mgr = 20 2024-09-11T09:49:29.832 INFO:teuthology.orchestra.run.smithi096.stdout: debug mon = 20 2024-09-11T09:49:29.833 INFO:teuthology.orchestra.run.smithi096.stdout: debug auth = 20 2024-09-11T09:49:29.833 INFO:teuthology.orchestra.run.smithi096.stdout: mon reweight min pgs per osd = 4 2024-09-11T09:49:29.833 INFO:teuthology.orchestra.run.smithi096.stdout: mon reweight min bytes per osd = 10 2024-09-11T09:49:29.833 INFO:teuthology.orchestra.run.smithi096.stdout: mgr/telemetry/nag = false 2024-09-11T09:49:29.833 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-11T09:49:29.833 INFO:teuthology.orchestra.run.smithi096.stdout:[mon] 2024-09-11T09:49:29.833 INFO:teuthology.orchestra.run.smithi096.stdout: debug ms = 1 2024-09-11T09:49:29.833 INFO:teuthology.orchestra.run.smithi096.stdout: debug mon = 20 2024-09-11T09:49:29.833 INFO:teuthology.orchestra.run.smithi096.stdout: debug paxos = 20 2024-09-11T09:49:29.833 INFO:teuthology.orchestra.run.smithi096.stdout: debug auth = 20 2024-09-11T09:49:29.833 INFO:teuthology.orchestra.run.smithi096.stdout: mon data avail warn = 5 2024-09-11T09:49:29.833 INFO:teuthology.orchestra.run.smithi096.stdout: mon mgr mkfs grace = 240 2024-09-11T09:49:29.833 INFO:teuthology.orchestra.run.smithi096.stdout: mon reweight min pgs per osd = 4 2024-09-11T09:49:29.834 INFO:teuthology.orchestra.run.smithi096.stdout: mon osd reporter subtree level = osd 2024-09-11T09:49:29.834 INFO:teuthology.orchestra.run.smithi096.stdout: mon osd prime pg temp = true 2024-09-11T09:49:29.834 INFO:teuthology.orchestra.run.smithi096.stdout: mon reweight min bytes per osd = 10 2024-09-11T09:49:29.834 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-11T09:49:29.834 INFO:teuthology.orchestra.run.smithi096.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-11T09:49:29.834 INFO:teuthology.orchestra.run.smithi096.stdout: auth mon ticket ttl = 660 # 11m 2024-09-11T09:49:29.834 INFO:teuthology.orchestra.run.smithi096.stdout: auth service ticket ttl = 240 # 4m 2024-09-11T09:49:29.834 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-11T09:49:29.834 INFO:teuthology.orchestra.run.smithi096.stdout: # don't complain about insecure global_id in the test suite 2024-09-11T09:49:29.834 INFO:teuthology.orchestra.run.smithi096.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-11T09:49:29.834 INFO:teuthology.orchestra.run.smithi096.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-11T09:49:29.834 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-11T09:49:29.834 INFO:teuthology.orchestra.run.smithi096.stdout: # 1m isn't quite enough 2024-09-11T09:49:29.835 INFO:teuthology.orchestra.run.smithi096.stdout: mon_down_mkfs_grace = 2m 2024-09-11T09:49:29.835 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-11T09:49:29.835 INFO:teuthology.orchestra.run.smithi096.stdout: mon_warn_on_filestore_osds = false 2024-09-11T09:49:29.835 INFO:teuthology.orchestra.run.smithi096.stdout: 2024-09-11T09:49:29.835 INFO:teuthology.orchestra.run.smithi096.stdout:[client] 2024-09-11T09:49:29.835 INFO:teuthology.orchestra.run.smithi096.stdout: rgw cache enabled = true 2024-09-11T09:49:29.835 INFO:teuthology.orchestra.run.smithi096.stdout: rgw enable ops log = true 2024-09-11T09:49:29.835 INFO:teuthology.orchestra.run.smithi096.stdout: rgw enable usage log = true 2024-09-11T09:49:29.835 INFO:teuthology.orchestra.run.smithi096.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-11T09:49:29.835 INFO:teuthology.orchestra.run.smithi096.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-11T09:49:29.835 INFO:teuthology.orchestra.run.smithi096.stdout:[mon.a] 2024-09-11T09:49:29.835 INFO:teuthology.orchestra.run.smithi096.stdout:[mon.c] 2024-09-11T09:49:29.835 INFO:teuthology.orchestra.run.smithi096.stdout:[mon.b] 2024-09-11T09:49:29.836 INFO:teuthology.orchestra.run.smithi107.stdout:[global] 2024-09-11T09:49:29.836 INFO:teuthology.orchestra.run.smithi107.stdout: chdir = "" 2024-09-11T09:49:29.836 INFO:teuthology.orchestra.run.smithi107.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-11T09:49:29.836 INFO:teuthology.orchestra.run.smithi107.stdout: auth supported = cephx 2024-09-11T09:49:29.837 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-11T09:49:29.837 INFO:teuthology.orchestra.run.smithi107.stdout: filestore xattr use omap = true 2024-09-11T09:49:29.837 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-11T09:49:29.837 INFO:teuthology.orchestra.run.smithi107.stdout: mon clock drift allowed = 1.000 2024-09-11T09:49:29.837 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-11T09:49:29.837 INFO:teuthology.orchestra.run.smithi107.stdout: osd crush chooseleaf type = 0 2024-09-11T09:49:29.837 INFO:teuthology.orchestra.run.smithi107.stdout: auth debug = true 2024-09-11T09:49:29.837 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-11T09:49:29.837 INFO:teuthology.orchestra.run.smithi107.stdout: ms die on old message = true 2024-09-11T09:49:29.837 INFO:teuthology.orchestra.run.smithi107.stdout: ms die on bug = true 2024-09-11T09:49:29.837 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-11T09:49:29.837 INFO:teuthology.orchestra.run.smithi107.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-11T09:49:29.838 INFO:teuthology.orchestra.run.smithi107.stdout: mon pg warn max object skew = 0 2024-09-11T09:49:29.838 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-11T09:49:29.838 INFO:teuthology.orchestra.run.smithi107.stdout: # disable pg_autoscaler by default for new pools 2024-09-11T09:49:29.838 INFO:teuthology.orchestra.run.smithi107.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-11T09:49:29.838 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-11T09:49:29.838 INFO:teuthology.orchestra.run.smithi107.stdout: osd pool default size = 2 2024-09-11T09:49:29.838 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-11T09:49:29.838 INFO:teuthology.orchestra.run.smithi107.stdout: mon osd allow primary affinity = true 2024-09-11T09:49:29.838 INFO:teuthology.orchestra.run.smithi107.stdout: mon osd allow pg remap = true 2024-09-11T09:49:29.838 INFO:teuthology.orchestra.run.smithi107.stdout: mon warn on legacy crush tunables = false 2024-09-11T09:49:29.838 INFO:teuthology.orchestra.run.smithi107.stdout: mon warn on crush straw calc version zero = false 2024-09-11T09:49:29.838 INFO:teuthology.orchestra.run.smithi107.stdout: mon warn on no sortbitwise = false 2024-09-11T09:49:29.838 INFO:teuthology.orchestra.run.smithi107.stdout: mon warn on osd down out interval zero = false 2024-09-11T09:49:29.838 INFO:teuthology.orchestra.run.smithi107.stdout: mon warn on too few osds = false 2024-09-11T09:49:29.839 INFO:teuthology.orchestra.run.smithi107.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-11T09:49:29.839 INFO:teuthology.orchestra.run.smithi107.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-11T09:49:29.839 INFO:teuthology.orchestra.run.smithi107.stdout: mon_allow_pool_size_one = true 2024-09-11T09:49:29.839 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-11T09:49:29.839 INFO:teuthology.orchestra.run.smithi107.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-11T09:49:29.839 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-11T09:49:29.839 INFO:teuthology.orchestra.run.smithi107.stdout: osd default data pool replay window = 5 2024-09-11T09:49:29.839 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-11T09:49:29.839 INFO:teuthology.orchestra.run.smithi107.stdout: mon allow pool delete = true 2024-09-11T09:49:29.839 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-11T09:49:29.839 INFO:teuthology.orchestra.run.smithi107.stdout: mon cluster log file level = debug 2024-09-11T09:49:29.839 INFO:teuthology.orchestra.run.smithi107.stdout: debug asserts on shutdown = true 2024-09-11T09:49:29.839 INFO:teuthology.orchestra.run.smithi107.stdout: mon health detail to clog = false 2024-09-11T09:49:29.839 INFO:teuthology.orchestra.run.smithi107.stdout: mon host = "172.21.15.17,[v2:172.21.15.17:3301,v1:172.21.15.17:6790],172.21.15.96" 2024-09-11T09:49:29.840 INFO:teuthology.orchestra.run.smithi107.stdout: mon client directed command retry = 5 2024-09-11T09:49:29.840 INFO:teuthology.orchestra.run.smithi107.stdout: mon warn on pool no app = False 2024-09-11T09:49:29.840 INFO:teuthology.orchestra.run.smithi107.stdout: ms inject socket failures = 5000 2024-09-11T09:49:29.840 INFO:teuthology.orchestra.run.smithi107.stdout: fsid = a2f1daec-d3b0-4bf5-82c8-72ecbb93d4e4 2024-09-11T09:49:29.840 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-11T09:49:29.840 INFO:teuthology.orchestra.run.smithi107.stdout:[osd] 2024-09-11T09:49:29.840 INFO:teuthology.orchestra.run.smithi107.stdout: osd journal size = 100 2024-09-11T09:49:29.840 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-11T09:49:29.840 INFO:teuthology.orchestra.run.smithi107.stdout: osd scrub load threshold = 5.0 2024-09-11T09:49:29.840 INFO:teuthology.orchestra.run.smithi107.stdout: osd scrub max interval = 600 2024-09-11T09:49:29.840 INFO:teuthology.orchestra.run.smithi107.stdout: osd mclock profile = high_recovery_ops 2024-09-11T09:49:29.840 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-11T09:49:29.840 INFO:teuthology.orchestra.run.smithi107.stdout: osd recover clone overlap = true 2024-09-11T09:49:29.840 INFO:teuthology.orchestra.run.smithi107.stdout: osd recovery max chunk = 1048576 2024-09-11T09:49:29.841 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-11T09:49:29.841 INFO:teuthology.orchestra.run.smithi107.stdout: osd debug shutdown = true 2024-09-11T09:49:29.841 INFO:teuthology.orchestra.run.smithi107.stdout: osd debug op order = true 2024-09-11T09:49:29.841 INFO:teuthology.orchestra.run.smithi107.stdout: osd debug verify stray on activate = true 2024-09-11T09:49:29.841 INFO:teuthology.orchestra.run.smithi107.stdout: osd debug trim objects = true 2024-09-11T09:49:29.841 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-11T09:49:29.841 INFO:teuthology.orchestra.run.smithi107.stdout: osd open classes on start = true 2024-09-11T09:49:29.841 INFO:teuthology.orchestra.run.smithi107.stdout: osd debug pg log writeout = true 2024-09-11T09:49:29.841 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-11T09:49:29.841 INFO:teuthology.orchestra.run.smithi107.stdout: osd deep scrub update digest min age = 30 2024-09-11T09:49:29.841 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-11T09:49:29.841 INFO:teuthology.orchestra.run.smithi107.stdout: osd map max advance = 10 2024-09-11T09:49:29.841 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-11T09:49:29.841 INFO:teuthology.orchestra.run.smithi107.stdout: journal zero on create = true 2024-09-11T09:49:29.842 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-11T09:49:29.842 INFO:teuthology.orchestra.run.smithi107.stdout: filestore ondisk finisher threads = 3 2024-09-11T09:49:29.842 INFO:teuthology.orchestra.run.smithi107.stdout: filestore apply finisher threads = 3 2024-09-11T09:49:29.842 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-11T09:49:29.842 INFO:teuthology.orchestra.run.smithi107.stdout: bdev debug aio = true 2024-09-11T09:49:29.842 INFO:teuthology.orchestra.run.smithi107.stdout: osd debug misdirected ops = true 2024-09-11T09:49:29.842 INFO:teuthology.orchestra.run.smithi107.stdout: bluestore block size = 96636764160 2024-09-11T09:49:29.842 INFO:teuthology.orchestra.run.smithi107.stdout: bluestore compression algorithm = snappy 2024-09-11T09:49:29.842 INFO:teuthology.orchestra.run.smithi107.stdout: bluestore compression mode = aggressive 2024-09-11T09:49:29.842 INFO:teuthology.orchestra.run.smithi107.stdout: bluestore fsck on mount = True 2024-09-11T09:49:29.842 INFO:teuthology.orchestra.run.smithi107.stdout: debug bluefs = 1/20 2024-09-11T09:49:29.842 INFO:teuthology.orchestra.run.smithi107.stdout: debug bluestore = 1/20 2024-09-11T09:49:29.842 INFO:teuthology.orchestra.run.smithi107.stdout: debug ms = 1 2024-09-11T09:49:29.842 INFO:teuthology.orchestra.run.smithi107.stdout: debug osd = 20 2024-09-11T09:49:29.842 INFO:teuthology.orchestra.run.smithi107.stdout: debug rocksdb = 4/10 2024-09-11T09:49:29.843 INFO:teuthology.orchestra.run.smithi107.stdout: mon osd backfillfull_ratio = 0.85 2024-09-11T09:49:29.843 INFO:teuthology.orchestra.run.smithi107.stdout: mon osd full ratio = 0.9 2024-09-11T09:49:29.843 INFO:teuthology.orchestra.run.smithi107.stdout: mon osd nearfull ratio = 0.8 2024-09-11T09:49:29.843 INFO:teuthology.orchestra.run.smithi107.stdout: osd blocked scrub grace period = 3600 2024-09-11T09:49:29.843 INFO:teuthology.orchestra.run.smithi107.stdout: osd failsafe full ratio = 0.95 2024-09-11T09:49:29.843 INFO:teuthology.orchestra.run.smithi107.stdout: osd max markdown count = 1000 2024-09-11T09:49:29.843 INFO:teuthology.orchestra.run.smithi107.stdout: osd objectstore = bluestore 2024-09-11T09:49:29.843 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-11T09:49:29.843 INFO:teuthology.orchestra.run.smithi107.stdout:[mgr] 2024-09-11T09:49:29.843 INFO:teuthology.orchestra.run.smithi107.stdout: debug ms = 1 2024-09-11T09:49:29.843 INFO:teuthology.orchestra.run.smithi107.stdout: debug mgr = 20 2024-09-11T09:49:29.843 INFO:teuthology.orchestra.run.smithi107.stdout: debug mon = 20 2024-09-11T09:49:29.843 INFO:teuthology.orchestra.run.smithi107.stdout: debug auth = 20 2024-09-11T09:49:29.843 INFO:teuthology.orchestra.run.smithi107.stdout: mon reweight min pgs per osd = 4 2024-09-11T09:49:29.843 INFO:teuthology.orchestra.run.smithi107.stdout: mon reweight min bytes per osd = 10 2024-09-11T09:49:29.844 INFO:teuthology.orchestra.run.smithi107.stdout: mgr/telemetry/nag = false 2024-09-11T09:49:29.844 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-11T09:49:29.844 INFO:teuthology.orchestra.run.smithi107.stdout:[mon] 2024-09-11T09:49:29.844 INFO:teuthology.orchestra.run.smithi107.stdout: debug ms = 1 2024-09-11T09:49:29.844 INFO:teuthology.orchestra.run.smithi107.stdout: debug mon = 20 2024-09-11T09:49:29.844 INFO:teuthology.orchestra.run.smithi107.stdout: debug paxos = 20 2024-09-11T09:49:29.844 INFO:teuthology.orchestra.run.smithi107.stdout: debug auth = 20 2024-09-11T09:49:29.844 INFO:teuthology.orchestra.run.smithi107.stdout: mon data avail warn = 5 2024-09-11T09:49:29.844 INFO:teuthology.orchestra.run.smithi107.stdout: mon mgr mkfs grace = 240 2024-09-11T09:49:29.844 INFO:teuthology.orchestra.run.smithi107.stdout: mon reweight min pgs per osd = 4 2024-09-11T09:49:29.844 INFO:teuthology.orchestra.run.smithi107.stdout: mon osd reporter subtree level = osd 2024-09-11T09:49:29.844 INFO:teuthology.orchestra.run.smithi107.stdout: mon osd prime pg temp = true 2024-09-11T09:49:29.844 INFO:teuthology.orchestra.run.smithi107.stdout: mon reweight min bytes per osd = 10 2024-09-11T09:49:29.844 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-11T09:49:29.845 INFO:teuthology.orchestra.run.smithi107.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-11T09:49:29.845 INFO:teuthology.orchestra.run.smithi107.stdout: auth mon ticket ttl = 660 # 11m 2024-09-11T09:49:29.845 INFO:teuthology.orchestra.run.smithi107.stdout: auth service ticket ttl = 240 # 4m 2024-09-11T09:49:29.845 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-11T09:49:29.845 INFO:teuthology.orchestra.run.smithi107.stdout: # don't complain about insecure global_id in the test suite 2024-09-11T09:49:29.845 INFO:teuthology.orchestra.run.smithi107.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-11T09:49:29.845 INFO:teuthology.orchestra.run.smithi107.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-11T09:49:29.845 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-11T09:49:29.845 INFO:teuthology.orchestra.run.smithi107.stdout: # 1m isn't quite enough 2024-09-11T09:49:29.845 INFO:teuthology.orchestra.run.smithi107.stdout: mon_down_mkfs_grace = 2m 2024-09-11T09:49:29.845 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-11T09:49:29.845 INFO:teuthology.orchestra.run.smithi107.stdout: mon_warn_on_filestore_osds = false 2024-09-11T09:49:29.845 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-09-11T09:49:29.845 INFO:teuthology.orchestra.run.smithi107.stdout:[client] 2024-09-11T09:49:29.846 INFO:teuthology.orchestra.run.smithi107.stdout: rgw cache enabled = true 2024-09-11T09:49:29.846 INFO:teuthology.orchestra.run.smithi107.stdout: rgw enable ops log = true 2024-09-11T09:49:29.846 INFO:teuthology.orchestra.run.smithi107.stdout: rgw enable usage log = true 2024-09-11T09:49:29.846 INFO:teuthology.orchestra.run.smithi107.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-11T09:49:29.846 INFO:teuthology.orchestra.run.smithi107.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-11T09:49:29.846 INFO:teuthology.orchestra.run.smithi107.stdout:[mon.a] 2024-09-11T09:49:29.846 INFO:teuthology.orchestra.run.smithi107.stdout:[mon.c] 2024-09-11T09:49:29.846 INFO:teuthology.orchestra.run.smithi107.stdout:[mon.b] 2024-09-11T09:49:29.848 INFO:tasks.ceph:Creating admin key on mon.a... 2024-09-11T09:49:29.848 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-09-11T09:49:29.943 INFO:tasks.ceph:Copying monmap to all nodes... 2024-09-11T09:49:29.943 DEBUG:teuthology.orchestra.run.smithi017:> set -ex 2024-09-11T09:49:29.943 DEBUG:teuthology.orchestra.run.smithi017:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-09-11T09:49:29.959 DEBUG:teuthology.orchestra.run.smithi017:> set -ex 2024-09-11T09:49:29.959 DEBUG:teuthology.orchestra.run.smithi017:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-09-11T09:49:30.014 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi017.front.sepia.ceph.com 2024-09-11T09:49:30.014 DEBUG:teuthology.orchestra.run.smithi017:> set -ex 2024-09-11T09:49:30.015 DEBUG:teuthology.orchestra.run.smithi017:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-11T09:49:30.015 DEBUG:teuthology.orchestra.run.smithi017:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-11T09:49:30.090 DEBUG:teuthology.orchestra.run.smithi017:> set -ex 2024-09-11T09:49:30.090 DEBUG:teuthology.orchestra.run.smithi017:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-11T09:49:30.146 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi096.front.sepia.ceph.com 2024-09-11T09:49:30.146 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-09-11T09:49:30.146 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-11T09:49:30.146 DEBUG:teuthology.orchestra.run.smithi096:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-11T09:49:30.180 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-09-11T09:49:30.181 DEBUG:teuthology.orchestra.run.smithi096:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-11T09:49:30.235 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi107.front.sepia.ceph.com 2024-09-11T09:49:30.236 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-09-11T09:49:30.236 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-11T09:49:30.236 DEBUG:teuthology.orchestra.run.smithi107:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-11T09:49:30.277 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-09-11T09:49:30.277 DEBUG:teuthology.orchestra.run.smithi107:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-11T09:49:30.335 INFO:tasks.ceph:Setting up mon nodes... 2024-09-11T09:49:30.336 INFO:tasks.ceph:Setting up mgr nodes... 2024-09-11T09:49:30.336 DEBUG:teuthology.orchestra.run.smithi096:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-09-11T09:49:30.395 INFO:teuthology.orchestra.run.smithi096.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-09-11T09:49:30.397 INFO:tasks.ceph:Setting up mds nodes... 2024-09-11T09:49:30.398 INFO:tasks.ceph_client:Setting up client nodes... 2024-09-11T09:49:30.398 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-09-11T09:49:30.465 INFO:teuthology.orchestra.run.smithi107.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-09-11T09:49:30.478 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-09-11T09:49:30.479 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi017.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi096.front.sepia.ceph.com'): {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'}} 2024-09-11T09:49:30.479 DEBUG:teuthology.orchestra.run.smithi017:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-09-11T09:49:30.506 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-11T09:49:30.506 INFO:tasks.ceph:role: osd.0 2024-09-11T09:49:30.506 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi017.front.sepia.ceph.com 2024-09-11T09:49:30.506 DEBUG:teuthology.orchestra.run.smithi017:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-11T09:49:30.572 INFO:teuthology.orchestra.run.smithi017.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-11T09:49:30.572 INFO:teuthology.orchestra.run.smithi017.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-11T09:49:30.572 INFO:teuthology.orchestra.run.smithi017.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-11T09:49:30.573 INFO:teuthology.orchestra.run.smithi017.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-11T09:49:30.573 INFO:teuthology.orchestra.run.smithi017.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-11T09:49:30.573 INFO:teuthology.orchestra.run.smithi017.stdout: = sunit=0 swidth=0 blks 2024-09-11T09:49:30.573 INFO:teuthology.orchestra.run.smithi017.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-11T09:49:30.573 INFO:teuthology.orchestra.run.smithi017.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-11T09:49:30.573 INFO:teuthology.orchestra.run.smithi017.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-11T09:49:30.573 INFO:teuthology.orchestra.run.smithi017.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-11T09:49:30.605 INFO:teuthology.orchestra.run.smithi017.stdout:Discarding blocks...Done. 2024-09-11T09:49:30.607 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi017.front.sepia.ceph.com -o noatime 2024-09-11T09:49:30.607 DEBUG:teuthology.orchestra.run.smithi017:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-09-11T09:49:30.677 DEBUG:teuthology.orchestra.run.smithi017:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-09-11T09:49:30.746 DEBUG:teuthology.orchestra.run.smithi017:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-09-11T09:49:30.814 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-11T09:49:30.814 INFO:tasks.ceph:role: osd.1 2024-09-11T09:49:30.814 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi017.front.sepia.ceph.com 2024-09-11T09:49:30.814 DEBUG:teuthology.orchestra.run.smithi017:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-11T09:49:30.879 INFO:teuthology.orchestra.run.smithi017.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-11T09:49:30.879 INFO:teuthology.orchestra.run.smithi017.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-11T09:49:30.879 INFO:teuthology.orchestra.run.smithi017.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-11T09:49:30.879 INFO:teuthology.orchestra.run.smithi017.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-11T09:49:30.879 INFO:teuthology.orchestra.run.smithi017.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-11T09:49:30.879 INFO:teuthology.orchestra.run.smithi017.stdout: = sunit=0 swidth=0 blks 2024-09-11T09:49:30.879 INFO:teuthology.orchestra.run.smithi017.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-11T09:49:30.880 INFO:teuthology.orchestra.run.smithi017.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-11T09:49:30.880 INFO:teuthology.orchestra.run.smithi017.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-11T09:49:30.880 INFO:teuthology.orchestra.run.smithi017.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-11T09:49:30.912 INFO:teuthology.orchestra.run.smithi017.stdout:Discarding blocks...Done. 2024-09-11T09:49:30.914 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi017.front.sepia.ceph.com -o noatime 2024-09-11T09:49:30.914 DEBUG:teuthology.orchestra.run.smithi017:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-09-11T09:49:30.984 DEBUG:teuthology.orchestra.run.smithi017:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-09-11T09:49:31.054 DEBUG:teuthology.orchestra.run.smithi017:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-09-11T09:49:31.119 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-11T09:49:31.120 INFO:tasks.ceph:role: osd.2 2024-09-11T09:49:31.120 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi017.front.sepia.ceph.com 2024-09-11T09:49:31.120 DEBUG:teuthology.orchestra.run.smithi017:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-11T09:49:31.185 INFO:teuthology.orchestra.run.smithi017.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-11T09:49:31.185 INFO:teuthology.orchestra.run.smithi017.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-11T09:49:31.185 INFO:teuthology.orchestra.run.smithi017.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-11T09:49:31.185 INFO:teuthology.orchestra.run.smithi017.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-11T09:49:31.185 INFO:teuthology.orchestra.run.smithi017.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-11T09:49:31.185 INFO:teuthology.orchestra.run.smithi017.stdout: = sunit=0 swidth=0 blks 2024-09-11T09:49:31.185 INFO:teuthology.orchestra.run.smithi017.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-11T09:49:31.185 INFO:teuthology.orchestra.run.smithi017.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-11T09:49:31.185 INFO:teuthology.orchestra.run.smithi017.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-11T09:49:31.185 INFO:teuthology.orchestra.run.smithi017.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-11T09:49:31.217 INFO:teuthology.orchestra.run.smithi017.stdout:Discarding blocks...Done. 2024-09-11T09:49:31.218 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi017.front.sepia.ceph.com -o noatime 2024-09-11T09:49:31.219 DEBUG:teuthology.orchestra.run.smithi017:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-09-11T09:49:31.288 DEBUG:teuthology.orchestra.run.smithi017:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-09-11T09:49:31.358 DEBUG:teuthology.orchestra.run.smithi017:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-11T09:49:31.451 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-11T09:49:31.451+0000 7f48017cc640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-09-11T09:49:31.451 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-11T09:49:31.451+0000 7f48017cc640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-09-11T09:49:31.451 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-11T09:49:31.451+0000 7f48017cc640 -1 bdev(0x55f8a4b37800 /var/lib/ceph/osd/ceph-0/block) open stat got: (1) Operation not permitted 2024-09-11T09:49:31.451 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-11T09:49:31.451+0000 7f48017cc640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-09-11T09:49:34.616 DEBUG:teuthology.orchestra.run.smithi017:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-11T09:49:34.646 DEBUG:teuthology.orchestra.run.smithi017:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-11T09:49:34.739 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-11T09:49:34.740+0000 7fd38280d640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-09-11T09:49:34.740 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-11T09:49:34.740+0000 7fd38280d640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-09-11T09:49:34.740 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-11T09:49:34.740+0000 7fd38280d640 -1 bdev(0x558888b83800 /var/lib/ceph/osd/ceph-1/block) open stat got: (1) Operation not permitted 2024-09-11T09:49:34.740 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-11T09:49:34.740+0000 7fd38280d640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-09-11T09:49:38.143 DEBUG:teuthology.orchestra.run.smithi017:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-11T09:49:38.174 DEBUG:teuthology.orchestra.run.smithi017:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-11T09:49:38.267 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-11T09:49:38.268+0000 7f45d03fd640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-09-11T09:49:38.268 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-11T09:49:38.268+0000 7f45d03fd640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-09-11T09:49:38.268 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-11T09:49:38.268+0000 7f45d03fd640 -1 bdev(0x5627eb293800 /var/lib/ceph/osd/ceph-2/block) open stat got: (1) Operation not permitted 2024-09-11T09:49:38.268 INFO:teuthology.orchestra.run.smithi017.stderr:2024-09-11T09:49:38.268+0000 7f45d03fd640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-09-11T09:49:41.179 DEBUG:teuthology.orchestra.run.smithi017:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-09-11T09:49:41.213 DEBUG:teuthology.orchestra.run.smithi096:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-09-11T09:49:41.244 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2024-09-11T09:49:41.244 INFO:tasks.ceph:role: osd.3 2024-09-11T09:49:41.244 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi096.front.sepia.ceph.com 2024-09-11T09:49:41.244 DEBUG:teuthology.orchestra.run.smithi096:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-11T09:49:41.321 INFO:teuthology.orchestra.run.smithi096.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-11T09:49:41.321 INFO:teuthology.orchestra.run.smithi096.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-11T09:49:41.321 INFO:teuthology.orchestra.run.smithi096.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-11T09:49:41.321 INFO:teuthology.orchestra.run.smithi096.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-11T09:49:41.321 INFO:teuthology.orchestra.run.smithi096.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-11T09:49:41.321 INFO:teuthology.orchestra.run.smithi096.stdout: = sunit=0 swidth=0 blks 2024-09-11T09:49:41.322 INFO:teuthology.orchestra.run.smithi096.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-11T09:49:41.322 INFO:teuthology.orchestra.run.smithi096.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-11T09:49:41.322 INFO:teuthology.orchestra.run.smithi096.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-11T09:49:41.322 INFO:teuthology.orchestra.run.smithi096.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-11T09:49:41.836 INFO:teuthology.orchestra.run.smithi096.stdout:Discarding blocks...Done. 2024-09-11T09:49:41.839 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi096.front.sepia.ceph.com -o noatime 2024-09-11T09:49:41.839 DEBUG:teuthology.orchestra.run.smithi096:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-3 2024-09-11T09:49:41.952 DEBUG:teuthology.orchestra.run.smithi096:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-09-11T09:49:41.986 DEBUG:teuthology.orchestra.run.smithi096:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-09-11T09:49:42.053 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2024-09-11T09:49:42.053 INFO:tasks.ceph:role: osd.4 2024-09-11T09:49:42.053 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi096.front.sepia.ceph.com 2024-09-11T09:49:42.053 DEBUG:teuthology.orchestra.run.smithi096:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-11T09:49:42.127 INFO:teuthology.orchestra.run.smithi096.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-11T09:49:42.127 INFO:teuthology.orchestra.run.smithi096.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-11T09:49:42.127 INFO:teuthology.orchestra.run.smithi096.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-11T09:49:42.128 INFO:teuthology.orchestra.run.smithi096.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-11T09:49:42.128 INFO:teuthology.orchestra.run.smithi096.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-11T09:49:42.128 INFO:teuthology.orchestra.run.smithi096.stdout: = sunit=0 swidth=0 blks 2024-09-11T09:49:42.128 INFO:teuthology.orchestra.run.smithi096.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-11T09:49:42.128 INFO:teuthology.orchestra.run.smithi096.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-11T09:49:42.128 INFO:teuthology.orchestra.run.smithi096.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-11T09:49:42.128 INFO:teuthology.orchestra.run.smithi096.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-11T09:49:42.581 INFO:teuthology.orchestra.run.smithi096.stdout:Discarding blocks...Done. 2024-09-11T09:49:42.584 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi096.front.sepia.ceph.com -o noatime 2024-09-11T09:49:42.584 DEBUG:teuthology.orchestra.run.smithi096:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-4 2024-09-11T09:49:42.677 DEBUG:teuthology.orchestra.run.smithi096:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-09-11T09:49:42.706 DEBUG:teuthology.orchestra.run.smithi096:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-09-11T09:49:42.771 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2024-09-11T09:49:42.771 INFO:tasks.ceph:role: osd.5 2024-09-11T09:49:42.771 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi096.front.sepia.ceph.com 2024-09-11T09:49:42.771 DEBUG:teuthology.orchestra.run.smithi096:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-11T09:49:42.844 INFO:teuthology.orchestra.run.smithi096.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-11T09:49:42.844 INFO:teuthology.orchestra.run.smithi096.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-11T09:49:42.844 INFO:teuthology.orchestra.run.smithi096.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-11T09:49:42.844 INFO:teuthology.orchestra.run.smithi096.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-11T09:49:42.844 INFO:teuthology.orchestra.run.smithi096.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-11T09:49:42.844 INFO:teuthology.orchestra.run.smithi096.stdout: = sunit=0 swidth=0 blks 2024-09-11T09:49:42.844 INFO:teuthology.orchestra.run.smithi096.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-11T09:49:42.844 INFO:teuthology.orchestra.run.smithi096.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-11T09:49:42.844 INFO:teuthology.orchestra.run.smithi096.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-11T09:49:42.844 INFO:teuthology.orchestra.run.smithi096.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-11T09:49:43.328 INFO:teuthology.orchestra.run.smithi096.stdout:Discarding blocks...Done. 2024-09-11T09:49:43.330 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi096.front.sepia.ceph.com -o noatime 2024-09-11T09:49:43.330 DEBUG:teuthology.orchestra.run.smithi096:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-5 2024-09-11T09:49:43.480 DEBUG:teuthology.orchestra.run.smithi096:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-09-11T09:49:43.510 DEBUG:teuthology.orchestra.run.smithi096:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-11T09:49:43.602 INFO:teuthology.orchestra.run.smithi096.stderr:2024-09-11T09:49:43.602+0000 7fe00d287640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-09-11T09:49:43.605 INFO:teuthology.orchestra.run.smithi096.stderr:2024-09-11T09:49:43.606+0000 7fe00d287640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-09-11T09:49:43.606 INFO:teuthology.orchestra.run.smithi096.stderr:2024-09-11T09:49:43.606+0000 7fe00d287640 -1 bdev(0x560073bc3800 /var/lib/ceph/osd/ceph-3/block) open stat got: (1) Operation not permitted 2024-09-11T09:49:43.606 INFO:teuthology.orchestra.run.smithi096.stderr:2024-09-11T09:49:43.606+0000 7fe00d287640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-09-11T09:49:46.787 DEBUG:teuthology.orchestra.run.smithi096:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-09-11T09:49:46.816 DEBUG:teuthology.orchestra.run.smithi096:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-11T09:49:46.907 INFO:teuthology.orchestra.run.smithi096.stderr:2024-09-11T09:49:46.908+0000 7f6f59219640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-09-11T09:49:46.908 INFO:teuthology.orchestra.run.smithi096.stderr:2024-09-11T09:49:46.908+0000 7f6f59219640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-09-11T09:49:46.908 INFO:teuthology.orchestra.run.smithi096.stderr:2024-09-11T09:49:46.908+0000 7f6f59219640 -1 bdev(0x56158bf11800 /var/lib/ceph/osd/ceph-4/block) open stat got: (1) Operation not permitted 2024-09-11T09:49:46.908 INFO:teuthology.orchestra.run.smithi096.stderr:2024-09-11T09:49:46.908+0000 7f6f59219640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-09-11T09:49:49.821 DEBUG:teuthology.orchestra.run.smithi096:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-09-11T09:49:49.849 DEBUG:teuthology.orchestra.run.smithi096:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-11T09:49:49.936 INFO:teuthology.orchestra.run.smithi096.stderr:2024-09-11T09:49:49.937+0000 7f080522c640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-09-11T09:49:49.937 INFO:teuthology.orchestra.run.smithi096.stderr:2024-09-11T09:49:49.937+0000 7f080522c640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-09-11T09:49:49.937 INFO:teuthology.orchestra.run.smithi096.stderr:2024-09-11T09:49:49.937+0000 7f080522c640 -1 bdev(0x5653e6a39800 /var/lib/ceph/osd/ceph-5/block) open stat got: (1) Operation not permitted 2024-09-11T09:49:49.937 INFO:teuthology.orchestra.run.smithi096.stderr:2024-09-11T09:49:49.937+0000 7f080522c640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-09-11T09:49:53.352 DEBUG:teuthology.orchestra.run.smithi096:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-09-11T09:49:53.383 INFO:tasks.ceph:Reading keys from all nodes... 2024-09-11T09:49:53.383 DEBUG:teuthology.orchestra.run.smithi017:> set -ex 2024-09-11T09:49:53.384 DEBUG:teuthology.orchestra.run.smithi017:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-09-11T09:49:53.415 DEBUG:teuthology.orchestra.run.smithi017:> set -ex 2024-09-11T09:49:53.415 DEBUG:teuthology.orchestra.run.smithi017:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-09-11T09:49:53.486 DEBUG:teuthology.orchestra.run.smithi017:> set -ex 2024-09-11T09:49:53.486 DEBUG:teuthology.orchestra.run.smithi017:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-09-11T09:49:53.556 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-09-11T09:49:53.557 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-09-11T09:49:53.585 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-09-11T09:49:53.585 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-09-11T09:49:53.650 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-09-11T09:49:53.650 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-09-11T09:49:53.717 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-09-11T09:49:53.717 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-09-11T09:49:53.782 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-09-11T09:49:53.782 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-09-11T09:49:53.802 INFO:tasks.ceph:Adding keys to all mons... 2024-09-11T09:49:53.802 DEBUG:teuthology.orchestra.run.smithi017:> sudo tee -a /etc/ceph/ceph.keyring 2024-09-11T09:49:53.804 DEBUG:teuthology.orchestra.run.smithi096:> sudo tee -a /etc/ceph/ceph.keyring 2024-09-11T09:49:53.831 INFO:teuthology.orchestra.run.smithi017.stdout:[osd.0] 2024-09-11T09:49:53.831 INFO:teuthology.orchestra.run.smithi017.stdout: key = AQCrZ+FmsUb2GhAA4/S1DkMQUQ1DAoitgrwoVQ== 2024-09-11T09:49:53.831 INFO:teuthology.orchestra.run.smithi017.stdout:[osd.1] 2024-09-11T09:49:53.831 INFO:teuthology.orchestra.run.smithi017.stdout: key = AQCuZ+FmJzwuLBAA3PqjJEasjKu+i8/19gpkxQ== 2024-09-11T09:49:53.831 INFO:teuthology.orchestra.run.smithi017.stdout:[osd.2] 2024-09-11T09:49:53.831 INFO:teuthology.orchestra.run.smithi017.stdout: key = AQCyZ+Fm8FoKEBAAL/t46coDLLBp1VQn7+X1+Q== 2024-09-11T09:49:53.831 INFO:teuthology.orchestra.run.smithi017.stdout:[mgr.x] 2024-09-11T09:49:53.831 INFO:teuthology.orchestra.run.smithi017.stdout: key = AQCqZ+FmEeWpFxAA+uaYDzEBRRY4yVfPhpv+UQ== 2024-09-11T09:49:53.832 INFO:teuthology.orchestra.run.smithi017.stdout:[osd.3] 2024-09-11T09:49:53.832 INFO:teuthology.orchestra.run.smithi017.stdout: key = AQC3Z+FmVhv7IxAAMPwfTtb1iUhsdDjGtLx/nQ== 2024-09-11T09:49:53.832 INFO:teuthology.orchestra.run.smithi017.stdout:[osd.4] 2024-09-11T09:49:53.832 INFO:teuthology.orchestra.run.smithi017.stdout: key = AQC6Z+Fm65MxNhAAV8NYGXxD+lOubQwqqdJW3Q== 2024-09-11T09:49:53.832 INFO:teuthology.orchestra.run.smithi017.stdout:[osd.5] 2024-09-11T09:49:53.832 INFO:teuthology.orchestra.run.smithi017.stdout: key = AQC9Z+FmLZLuNxAAz6MD0eeBeJ9SJvGxfwaclw== 2024-09-11T09:49:53.832 INFO:teuthology.orchestra.run.smithi017.stdout:[client.0] 2024-09-11T09:49:53.832 INFO:teuthology.orchestra.run.smithi017.stdout: key = AQCqZ+FmPkvQGxAAXsHLRyx/ZVnufXx3Sx6N6A== 2024-09-11T09:49:53.846 INFO:teuthology.orchestra.run.smithi096.stdout:[osd.0] 2024-09-11T09:49:53.846 INFO:teuthology.orchestra.run.smithi096.stdout: key = AQCrZ+FmsUb2GhAA4/S1DkMQUQ1DAoitgrwoVQ== 2024-09-11T09:49:53.846 INFO:teuthology.orchestra.run.smithi096.stdout:[osd.1] 2024-09-11T09:49:53.846 INFO:teuthology.orchestra.run.smithi096.stdout: key = AQCuZ+FmJzwuLBAA3PqjJEasjKu+i8/19gpkxQ== 2024-09-11T09:49:53.847 INFO:teuthology.orchestra.run.smithi096.stdout:[osd.2] 2024-09-11T09:49:53.847 INFO:teuthology.orchestra.run.smithi096.stdout: key = AQCyZ+Fm8FoKEBAAL/t46coDLLBp1VQn7+X1+Q== 2024-09-11T09:49:53.847 INFO:teuthology.orchestra.run.smithi096.stdout:[mgr.x] 2024-09-11T09:49:53.847 INFO:teuthology.orchestra.run.smithi096.stdout: key = AQCqZ+FmEeWpFxAA+uaYDzEBRRY4yVfPhpv+UQ== 2024-09-11T09:49:53.847 INFO:teuthology.orchestra.run.smithi096.stdout:[osd.3] 2024-09-11T09:49:53.847 INFO:teuthology.orchestra.run.smithi096.stdout: key = AQC3Z+FmVhv7IxAAMPwfTtb1iUhsdDjGtLx/nQ== 2024-09-11T09:49:53.847 INFO:teuthology.orchestra.run.smithi096.stdout:[osd.4] 2024-09-11T09:49:53.847 INFO:teuthology.orchestra.run.smithi096.stdout: key = AQC6Z+Fm65MxNhAAV8NYGXxD+lOubQwqqdJW3Q== 2024-09-11T09:49:53.847 INFO:teuthology.orchestra.run.smithi096.stdout:[osd.5] 2024-09-11T09:49:53.847 INFO:teuthology.orchestra.run.smithi096.stdout: key = AQC9Z+FmLZLuNxAAz6MD0eeBeJ9SJvGxfwaclw== 2024-09-11T09:49:53.847 INFO:teuthology.orchestra.run.smithi096.stdout:[client.0] 2024-09-11T09:49:53.847 INFO:teuthology.orchestra.run.smithi096.stdout: key = AQCqZ+FmPkvQGxAAXsHLRyx/ZVnufXx3Sx6N6A== 2024-09-11T09:49:53.848 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-11T09:49:53.876 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-11T09:49:53.940 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-11T09:49:53.942 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-11T09:49:53.994 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-11T09:49:53.996 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-11T09:49:54.048 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-11T09:49:54.050 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-11T09:49:54.101 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-11T09:49:54.104 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-11T09:49:54.154 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-11T09:49:54.156 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-11T09:49:54.207 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-11T09:49:54.209 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-11T09:49:54.260 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-11T09:49:54.262 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-11T09:49:54.316 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-09-11T09:49:54.316 DEBUG:teuthology.orchestra.run.smithi017:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-09-11T09:49:54.342 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-11T09:49:54.439 DEBUG:teuthology.orchestra.run.smithi017:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-09-11T09:49:54.464 DEBUG:teuthology.orchestra.run.smithi017:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-09-11T09:49:54.531 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-11T09:49:54.627 DEBUG:teuthology.orchestra.run.smithi017:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-09-11T09:49:54.653 DEBUG:teuthology.orchestra.run.smithi096:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-09-11T09:49:54.679 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-11T09:49:54.777 DEBUG:teuthology.orchestra.run.smithi096:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-09-11T09:49:54.802 DEBUG:teuthology.orchestra.run.smithi017:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-09-11T09:49:54.804 DEBUG:teuthology.orchestra.run.smithi096:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-09-11T09:49:54.857 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-09-11T09:49:54.858 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-11T09:49:54.858 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-11T09:49:54.860 INFO:tasks.ceph.mon.a:Started 2024-09-11T09:49:54.860 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-11T09:49:54.861 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-11T09:49:54.862 INFO:tasks.ceph.mon.c:Started 2024-09-11T09:49:54.863 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-11T09:49:54.863 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-11T09:49:54.899 INFO:tasks.ceph.mon.b:Started 2024-09-11T09:49:54.900 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-09-11T09:49:54.900 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-09-11T09:49:54.900 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-09-11T09:49:54.902 INFO:tasks.ceph.mgr.x:Started 2024-09-11T09:49:54.902 INFO:tasks.ceph:Setting crush tunables to default 2024-09-11T09:49:54.903 DEBUG:teuthology.orchestra.run.smithi017:> sudo ceph --cluster ceph osd crush tunables default 2024-09-11T09:50:00.257 INFO:teuthology.orchestra.run.smithi017.stderr:adjusted tunables profile to default 2024-09-11T09:50:00.269 INFO:tasks.ceph:check_enable_crimson: False 2024-09-11T09:50:00.269 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-09-11T09:50:00.269 DEBUG:teuthology.orchestra.run.smithi017:> set -ex 2024-09-11T09:50:00.270 DEBUG:teuthology.orchestra.run.smithi017:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-09-11T09:50:00.297 DEBUG:teuthology.orchestra.run.smithi017:> set -ex 2024-09-11T09:50:00.297 DEBUG:teuthology.orchestra.run.smithi017:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-09-11T09:50:00.363 DEBUG:teuthology.orchestra.run.smithi017:> set -ex 2024-09-11T09:50:00.364 DEBUG:teuthology.orchestra.run.smithi017:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-09-11T09:50:00.430 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-09-11T09:50:00.430 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-09-11T09:50:00.459 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-09-11T09:50:00.459 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-09-11T09:50:00.497 INFO:tasks.ceph.mgr.x.smithi096.stderr:2024-09-11T09:50:00.497+0000 7f4857c77040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-11T09:50:00.524 DEBUG:teuthology.orchestra.run.smithi096:> set -ex 2024-09-11T09:50:00.524 DEBUG:teuthology.orchestra.run.smithi096:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-09-11T09:50:00.542 INFO:tasks.ceph.mgr.x.smithi096.stderr:2024-09-11T09:50:00.543+0000 7f4857c77040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-11T09:50:00.566 DEBUG:teuthology.orchestra.run.smithi096:> sudo ceph --cluster ceph osd new 4829b7a1-d607-4a56-b6d2-55036cb8acf8 0 2024-09-11T09:50:00.692 INFO:tasks.ceph.mgr.x.smithi096.stderr:2024-09-11T09:50:00.693+0000 7f4857c77040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-11T09:50:00.745 INFO:tasks.ceph.mgr.x.smithi096.stderr:2024-09-11T09:50:00.746+0000 7f4857c77040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-11T09:50:00.788 INFO:teuthology.orchestra.run.smithi096.stdout:0 2024-09-11T09:50:00.799 DEBUG:teuthology.orchestra.run.smithi096:> sudo ceph --cluster ceph osd new ab1d5df0-3c0e-44cd-88a0-96b07964148b 1 2024-09-11T09:50:00.915 INFO:tasks.ceph.mgr.x.smithi096.stderr:2024-09-11T09:50:00.916+0000 7f4857c77040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-11T09:50:00.981 INFO:teuthology.orchestra.run.smithi096.stdout:1 2024-09-11T09:50:00.993 DEBUG:teuthology.orchestra.run.smithi096:> sudo ceph --cluster ceph osd new 4c8e50db-b49c-4d44-a913-47b5b30f8699 2 2024-09-11T09:50:01.031 INFO:tasks.ceph.mgr.x.smithi096.stderr:2024-09-11T09:50:01.032+0000 7f4857c77040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-11T09:50:01.187 INFO:teuthology.orchestra.run.smithi096.stdout:2 2024-09-11T09:50:01.198 DEBUG:teuthology.orchestra.run.smithi096:> sudo ceph --cluster ceph osd new 37abc55b-a1eb-49d5-b1ff-b507ee2cb5a6 3 2024-09-11T09:50:01.366 INFO:tasks.ceph.mgr.x.smithi096.stderr:2024-09-11T09:50:01.366+0000 7f4857c77040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-11T09:50:01.418 INFO:teuthology.orchestra.run.smithi096.stdout:3 2024-09-11T09:50:01.422 INFO:tasks.ceph.mgr.x.smithi096.stderr:2024-09-11T09:50:01.423+0000 7f4857c77040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-11T09:50:01.429 DEBUG:teuthology.orchestra.run.smithi096:> sudo ceph --cluster ceph osd new 067d9586-4f5a-4d17-96c5-86a7b8bcd114 4 2024-09-11T09:50:01.580 INFO:tasks.ceph.mgr.x.smithi096.stderr:2024-09-11T09:50:01.581+0000 7f4857c77040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-11T09:50:01.612 INFO:teuthology.orchestra.run.smithi096.stdout:4 2024-09-11T09:50:01.623 DEBUG:teuthology.orchestra.run.smithi096:> sudo ceph --cluster ceph osd new 8ff7a724-0a3a-4aa2-baf1-fd52f88d476c 5 2024-09-11T09:50:01.807 INFO:teuthology.orchestra.run.smithi096.stdout:5 2024-09-11T09:50:01.818 INFO:tasks.ceph.osd.0:Restarting daemon 2024-09-11T09:50:01.818 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-09-11T09:50:01.821 INFO:tasks.ceph.osd.0:Started 2024-09-11T09:50:01.821 INFO:tasks.ceph.osd.1:Restarting daemon 2024-09-11T09:50:01.821 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-09-11T09:50:01.823 INFO:tasks.ceph.osd.1:Started 2024-09-11T09:50:01.823 INFO:tasks.ceph.osd.2:Restarting daemon 2024-09-11T09:50:01.824 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-09-11T09:50:01.826 INFO:tasks.ceph.osd.2:Started 2024-09-11T09:50:01.826 INFO:tasks.ceph.osd.3:Restarting daemon 2024-09-11T09:50:01.826 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-09-11T09:50:01.828 INFO:tasks.ceph.osd.3:Started 2024-09-11T09:50:01.828 INFO:tasks.ceph.osd.4:Restarting daemon 2024-09-11T09:50:01.828 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-09-11T09:50:01.830 INFO:tasks.ceph.osd.4:Started 2024-09-11T09:50:01.830 INFO:tasks.ceph.osd.5:Restarting daemon 2024-09-11T09:50:01.831 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-09-11T09:50:01.832 INFO:tasks.ceph.osd.5:Started 2024-09-11T09:50:01.833 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-11T09:50:02.018 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-11T09:50:02.018 INFO:teuthology.orchestra.run.smithi017.stdout:{"epoch":8,"fsid":"a2f1daec-d3b0-4bf5-82c8-72ecbb93d4e4","created":"2024-09-11T09:50:00.165804+0000","modified":"2024-09-11T09:50:01.801190+0000","last_up_change":"0.000000","last_in_change":"2024-09-11T09:50:01.801190+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4829b7a1-d607-4a56-b6d2-55036cb8acf8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ab1d5df0-3c0e-44cd-88a0-96b07964148b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"4c8e50db-b49c-4d44-a913-47b5b30f8699","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"37abc55b-a1eb-49d5-b1ff-b507ee2cb5a6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"067d9586-4f5a-4d17-96c5-86a7b8bcd114","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"8ff7a724-0a3a-4aa2-baf1-fd52f88d476c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-11T09:50:02.029 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-09-11T09:50:02.030 INFO:tasks.ceph:Waiting for OSDs to come up 2024-09-11T09:50:02.534 INFO:tasks.ceph.mgr.x.smithi096.stderr:2024-09-11T09:50:02.534+0000 7f4857c77040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-11T09:50:02.587 INFO:tasks.ceph.mgr.x.smithi096.stderr:2024-09-11T09:50:02.588+0000 7f4857c77040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-11T09:50:02.634 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-11T09:50:02.782 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:50:02.782+0000 7f61b0a0d640 -1 Falling back to public interface 2024-09-11T09:50:02.809 INFO:teuthology.misc.health.smithi017.stdout: 2024-09-11T09:50:02.809 INFO:teuthology.misc.health.smithi017.stdout:{"epoch":8,"fsid":"a2f1daec-d3b0-4bf5-82c8-72ecbb93d4e4","created":"2024-09-11T09:50:00.165804+0000","modified":"2024-09-11T09:50:01.801190+0000","last_up_change":"0.000000","last_in_change":"2024-09-11T09:50:01.801190+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4829b7a1-d607-4a56-b6d2-55036cb8acf8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ab1d5df0-3c0e-44cd-88a0-96b07964148b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"4c8e50db-b49c-4d44-a913-47b5b30f8699","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"37abc55b-a1eb-49d5-b1ff-b507ee2cb5a6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"067d9586-4f5a-4d17-96c5-86a7b8bcd114","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"8ff7a724-0a3a-4aa2-baf1-fd52f88d476c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-11T09:50:02.818 DEBUG:teuthology.misc:0 of 6 OSDs are up 2024-09-11T09:50:02.950 INFO:tasks.ceph.mgr.x.smithi096.stderr:2024-09-11T09:50:02.951+0000 7f4857c77040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-11T09:50:03.033 INFO:tasks.ceph.mgr.x.smithi096.stderr:2024-09-11T09:50:03.034+0000 7f4857c77040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-11T09:50:03.142 INFO:tasks.ceph.mgr.x.smithi096.stderr:2024-09-11T09:50:03.142+0000 7f4857c77040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-11T09:50:03.189 INFO:tasks.ceph.mgr.x.smithi096.stderr:2024-09-11T09:50:03.190+0000 7f4857c77040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-11T09:50:03.260 INFO:tasks.ceph.osd.0.smithi017.stderr:2024-09-11T09:50:03.260+0000 7fb73f14d640 -1 Falling back to public interface 2024-09-11T09:50:03.300 INFO:tasks.ceph.mgr.x.smithi096.stderr:2024-09-11T09:50:03.300+0000 7f4857c77040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-11T09:50:03.334 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:50:03.334+0000 7fdec88a5640 -1 Falling back to public interface 2024-09-11T09:50:03.340 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:50:03.340+0000 7f3c99f25640 -1 Falling back to public interface 2024-09-11T09:50:03.354 INFO:tasks.ceph.mgr.x.smithi096.stderr:2024-09-11T09:50:03.355+0000 7f4857c77040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-11T09:50:03.400 INFO:tasks.ceph.mgr.x.smithi096.stderr:2024-09-11T09:50:03.401+0000 7f4857c77040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-11T09:50:03.509 INFO:tasks.ceph.mgr.x.smithi096.stderr:2024-09-11T09:50:03.510+0000 7f4857c77040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-11T09:50:03.536 INFO:tasks.ceph.osd.2.smithi017.stderr:2024-09-11T09:50:03.535+0000 7fd20e1cc640 -1 Falling back to public interface 2024-09-11T09:50:03.558 INFO:tasks.ceph.mgr.x.smithi096.stderr:2024-09-11T09:50:03.558+0000 7f4857c77040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-11T09:50:03.609 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-09-11T09:50:03.609+0000 7f7b6f82c640 -1 Falling back to public interface 2024-09-11T09:50:03.609 INFO:tasks.ceph.mgr.x.smithi096.stderr:2024-09-11T09:50:03.610+0000 7f4857c77040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-11T09:50:03.676 INFO:tasks.ceph.mgr.x.smithi096.stderr:2024-09-11T09:50:03.677+0000 7f4857c77040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-11T09:50:03.720 INFO:tasks.ceph.mgr.x.smithi096.stderr:2024-09-11T09:50:03.721+0000 7f4857c77040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-11T09:50:03.763 INFO:tasks.ceph.mgr.x.smithi096.stderr:2024-09-11T09:50:03.764+0000 7f4857c77040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-11T09:50:04.404 INFO:tasks.ceph.mgr.x.smithi096.stderr:2024-09-11T09:50:04.405+0000 7f4857c77040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-11T09:50:05.418 INFO:tasks.ceph.mgr.x.smithi096.stderr:2024-09-11T09:50:05.419+0000 7f47f8549640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-09-11T09:50:05.419 INFO:tasks.ceph.mgr.x.smithi096.stderr:2024-09-11T09:50:05.419+0000 7f47f8549640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-09-11T09:50:06.562 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:50:06.562+0000 7f61b0a0d640 -1 osd.1 0 log_to_monitors true 2024-09-11T09:50:07.116 INFO:tasks.ceph.osd.2.smithi017.stderr:2024-09-11T09:50:07.116+0000 7fd20e1cc640 -1 osd.2 0 log_to_monitors true 2024-09-11T09:50:07.130 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:50:07.131+0000 7f3c99f25640 -1 osd.5 0 log_to_monitors true 2024-09-11T09:50:07.182 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:50:07.183+0000 7fdec88a5640 -1 osd.4 0 log_to_monitors true 2024-09-11T09:50:07.537 INFO:tasks.ceph.osd.0.smithi017.stderr:2024-09-11T09:50:07.537+0000 7fb73f14d640 -1 osd.0 0 log_to_monitors true 2024-09-11T09:50:07.667 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-09-11T09:50:07.668+0000 7f7b6f82c640 -1 osd.3 0 log_to_monitors true 2024-09-11T09:50:09.259 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:50:09.259+0000 7f3c966c8640 -1 osd.5 0 waiting for initial osdmap 2024-09-11T09:50:09.271 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:50:09.272+0000 7f3c8dcd4640 -1 osd.5 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-11T09:50:09.317 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:50:09.318+0000 7fdec5048640 -1 osd.4 0 waiting for initial osdmap 2024-09-11T09:50:09.328 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:50:09.329+0000 7fdebc654640 -1 osd.4 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-11T09:50:09.345 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:50:09.344+0000 7f61ac44b640 -1 osd.1 0 waiting for initial osdmap 2024-09-11T09:50:09.358 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:50:09.358+0000 7f61a426b640 -1 osd.1 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-11T09:50:09.388 INFO:tasks.ceph.osd.2.smithi017.stderr:2024-09-11T09:50:09.388+0000 7fd20a96f640 -1 osd.2 0 waiting for initial osdmap 2024-09-11T09:50:09.402 INFO:tasks.ceph.osd.2.smithi017.stderr:2024-09-11T09:50:09.402+0000 7fd201f7b640 -1 osd.2 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-11T09:50:09.425 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-11T09:50:09.744 INFO:teuthology.misc.health.smithi017.stdout: 2024-09-11T09:50:09.744 INFO:teuthology.misc.health.smithi017.stdout:{"epoch":11,"fsid":"a2f1daec-d3b0-4bf5-82c8-72ecbb93d4e4","created":"2024-09-11T09:50:00.165804+0000","modified":"2024-09-11T09:50:09.426642+0000","last_up_change":"2024-09-11T09:50:09.426642+0000","last_in_change":"2024-09-11T09:50:01.801190+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4829b7a1-d607-4a56-b6d2-55036cb8acf8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"ab1d5df0-3c0e-44cd-88a0-96b07964148b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6800","nonce":4139511644},{"type":"v1","addr":"172.21.15.17:6801","nonce":4139511644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6802","nonce":4139511644},{"type":"v1","addr":"172.21.15.17:6803","nonce":4139511644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6806","nonce":4139511644},{"type":"v1","addr":"172.21.15.17:6807","nonce":4139511644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6804","nonce":4139511644},{"type":"v1","addr":"172.21.15.17:6805","nonce":4139511644}]},"public_addr":"172.21.15.17:6801/4139511644","cluster_addr":"172.21.15.17:6803/4139511644","heartbeat_back_addr":"172.21.15.17:6807/4139511644","heartbeat_front_addr":"172.21.15.17:6805/4139511644","state":["exists","up"]},{"osd":2,"uuid":"4c8e50db-b49c-4d44-a913-47b5b30f8699","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6816","nonce":1151282506},{"type":"v1","addr":"172.21.15.17:6817","nonce":1151282506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6818","nonce":1151282506},{"type":"v1","addr":"172.21.15.17:6819","nonce":1151282506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6822","nonce":1151282506},{"type":"v1","addr":"172.21.15.17:6823","nonce":1151282506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6820","nonce":1151282506},{"type":"v1","addr":"172.21.15.17:6821","nonce":1151282506}]},"public_addr":"172.21.15.17:6817/1151282506","cluster_addr":"172.21.15.17:6819/1151282506","heartbeat_back_addr":"172.21.15.17:6823/1151282506","heartbeat_front_addr":"172.21.15.17:6821/1151282506","state":["exists","up"]},{"osd":3,"uuid":"37abc55b-a1eb-49d5-b1ff-b507ee2cb5a6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"067d9586-4f5a-4d17-96c5-86a7b8bcd114","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":2785176937},{"type":"v1","addr":"172.21.15.96:6801","nonce":2785176937}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":2785176937},{"type":"v1","addr":"172.21.15.96:6803","nonce":2785176937}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":2785176937},{"type":"v1","addr":"172.21.15.96:6807","nonce":2785176937}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":2785176937},{"type":"v1","addr":"172.21.15.96:6805","nonce":2785176937}]},"public_addr":"172.21.15.96:6801/2785176937","cluster_addr":"172.21.15.96:6803/2785176937","heartbeat_back_addr":"172.21.15.96:6807/2785176937","heartbeat_front_addr":"172.21.15.96:6805/2785176937","state":["exists","up"]},{"osd":5,"uuid":"8ff7a724-0a3a-4aa2-baf1-fd52f88d476c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":509935370},{"type":"v1","addr":"172.21.15.96:6809","nonce":509935370}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6810","nonce":509935370},{"type":"v1","addr":"172.21.15.96:6811","nonce":509935370}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6814","nonce":509935370},{"type":"v1","addr":"172.21.15.96:6815","nonce":509935370}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":509935370},{"type":"v1","addr":"172.21.15.96:6813","nonce":509935370}]},"public_addr":"172.21.15.96:6809/509935370","cluster_addr":"172.21.15.96:6811/509935370","heartbeat_back_addr":"172.21.15.96:6815/509935370","heartbeat_front_addr":"172.21.15.96:6813/509935370","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-11T09:50:09.760 DEBUG:teuthology.misc:4 of 6 OSDs are up 2024-09-11T09:50:10.122 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-09-11T09:50:10.122+0000 7f7b6b7bb640 -1 osd.3 0 waiting for initial osdmap 2024-09-11T09:50:10.135 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-09-11T09:50:10.136+0000 7f7b635db640 -1 osd.3 11 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-11T09:50:10.197 INFO:tasks.ceph.osd.0.smithi017.stderr:2024-09-11T09:50:10.197+0000 7fb73b0dc640 -1 osd.0 0 waiting for initial osdmap 2024-09-11T09:50:10.213 INFO:tasks.ceph.osd.0.smithi017.stderr:2024-09-11T09:50:10.213+0000 7fb732efc640 -1 osd.0 11 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-11T09:50:16.366 DEBUG:teuthology.orchestra.run.smithi017:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-11T09:50:16.662 INFO:teuthology.misc.health.smithi017.stdout: 2024-09-11T09:50:16.662 INFO:teuthology.misc.health.smithi017.stdout:{"epoch":14,"fsid":"a2f1daec-d3b0-4bf5-82c8-72ecbb93d4e4","created":"2024-09-11T09:50:00.165804+0000","modified":"2024-09-11T09:50:12.468441+0000","last_up_change":"2024-09-11T09:50:10.457262+0000","last_in_change":"2024-09-11T09:50:01.801190+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-11T09:50:10.457184+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4829b7a1-d607-4a56-b6d2-55036cb8acf8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6808","nonce":4154449929},{"type":"v1","addr":"172.21.15.17:6809","nonce":4154449929}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6810","nonce":4154449929},{"type":"v1","addr":"172.21.15.17:6811","nonce":4154449929}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6814","nonce":4154449929},{"type":"v1","addr":"172.21.15.17:6815","nonce":4154449929}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6812","nonce":4154449929},{"type":"v1","addr":"172.21.15.17:6813","nonce":4154449929}]},"public_addr":"172.21.15.17:6809/4154449929","cluster_addr":"172.21.15.17:6811/4154449929","heartbeat_back_addr":"172.21.15.17:6815/4154449929","heartbeat_front_addr":"172.21.15.17:6813/4154449929","state":["exists","up"]},{"osd":1,"uuid":"ab1d5df0-3c0e-44cd-88a0-96b07964148b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6800","nonce":4139511644},{"type":"v1","addr":"172.21.15.17:6801","nonce":4139511644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6802","nonce":4139511644},{"type":"v1","addr":"172.21.15.17:6803","nonce":4139511644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6806","nonce":4139511644},{"type":"v1","addr":"172.21.15.17:6807","nonce":4139511644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6804","nonce":4139511644},{"type":"v1","addr":"172.21.15.17:6805","nonce":4139511644}]},"public_addr":"172.21.15.17:6801/4139511644","cluster_addr":"172.21.15.17:6803/4139511644","heartbeat_back_addr":"172.21.15.17:6807/4139511644","heartbeat_front_addr":"172.21.15.17:6805/4139511644","state":["exists","up"]},{"osd":2,"uuid":"4c8e50db-b49c-4d44-a913-47b5b30f8699","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6816","nonce":1151282506},{"type":"v1","addr":"172.21.15.17:6817","nonce":1151282506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6818","nonce":1151282506},{"type":"v1","addr":"172.21.15.17:6819","nonce":1151282506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6822","nonce":1151282506},{"type":"v1","addr":"172.21.15.17:6823","nonce":1151282506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6820","nonce":1151282506},{"type":"v1","addr":"172.21.15.17:6821","nonce":1151282506}]},"public_addr":"172.21.15.17:6817/1151282506","cluster_addr":"172.21.15.17:6819/1151282506","heartbeat_back_addr":"172.21.15.17:6823/1151282506","heartbeat_front_addr":"172.21.15.17:6821/1151282506","state":["exists","up"]},{"osd":3,"uuid":"37abc55b-a1eb-49d5-b1ff-b507ee2cb5a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":332854807},{"type":"v1","addr":"172.21.15.96:6817","nonce":332854807}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6818","nonce":332854807},{"type":"v1","addr":"172.21.15.96:6819","nonce":332854807}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":332854807},{"type":"v1","addr":"172.21.15.96:6823","nonce":332854807}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":332854807},{"type":"v1","addr":"172.21.15.96:6821","nonce":332854807}]},"public_addr":"172.21.15.96:6817/332854807","cluster_addr":"172.21.15.96:6819/332854807","heartbeat_back_addr":"172.21.15.96:6823/332854807","heartbeat_front_addr":"172.21.15.96:6821/332854807","state":["exists","up"]},{"osd":4,"uuid":"067d9586-4f5a-4d17-96c5-86a7b8bcd114","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":2785176937},{"type":"v1","addr":"172.21.15.96:6801","nonce":2785176937}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":2785176937},{"type":"v1","addr":"172.21.15.96:6803","nonce":2785176937}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":2785176937},{"type":"v1","addr":"172.21.15.96:6807","nonce":2785176937}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":2785176937},{"type":"v1","addr":"172.21.15.96:6805","nonce":2785176937}]},"public_addr":"172.21.15.96:6801/2785176937","cluster_addr":"172.21.15.96:6803/2785176937","heartbeat_back_addr":"172.21.15.96:6807/2785176937","heartbeat_front_addr":"172.21.15.96:6805/2785176937","state":["exists","up"]},{"osd":5,"uuid":"8ff7a724-0a3a-4aa2-baf1-fd52f88d476c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":509935370},{"type":"v1","addr":"172.21.15.96:6809","nonce":509935370}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6810","nonce":509935370},{"type":"v1","addr":"172.21.15.96:6811","nonce":509935370}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6814","nonce":509935370},{"type":"v1","addr":"172.21.15.96:6815","nonce":509935370}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":509935370},{"type":"v1","addr":"172.21.15.96:6813","nonce":509935370}]},"public_addr":"172.21.15.96:6809/509935370","cluster_addr":"172.21.15.96:6811/509935370","heartbeat_back_addr":"172.21.15.96:6815/509935370","heartbeat_front_addr":"172.21.15.96:6813/509935370","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T09:50:08.513270+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T09:50:07.522863+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T09:50:08.134364+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T09:50:08.622366+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T09:50:08.156142+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T09:50:08.108116+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-11T09:50:16.674 DEBUG:teuthology.misc:6 of 6 OSDs are up 2024-09-11T09:50:16.674 INFO:tasks.ceph:Creating RBD pool 2024-09-11T09:50:16.674 DEBUG:teuthology.orchestra.run.smithi017:> sudo ceph --cluster ceph osd pool create rbd 8 2024-09-11T09:50:17.480 INFO:teuthology.orchestra.run.smithi017.stderr:pool 'rbd' created 2024-09-11T09:50:17.495 DEBUG:teuthology.orchestra.run.smithi017:> rbd --cluster ceph pool init rbd 2024-09-11T09:50:20.507 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-09-11T09:50:20.507 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-09-11T09:50:20.508 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-09-11T09:50:20.508 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-09-11T09:50:20.508 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-09-11T09:50:20.856 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-11T09:50:20.867 INFO:teuthology.orchestra.run.smithi017.stdout:{"epoch":5,"flags":0,"active_gid":4102,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6824","nonce":3355344653},{"type":"v1","addr":"172.21.15.96:6825","nonce":3355344653}]},"active_addr":"172.21.15.96:6825/3355344653","active_change":"2024-09-11T09:50:04.408307+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.5","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.96:0","nonce":2118898407}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.96:0","nonce":1227819331}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.96:0","nonce":1773604767}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.96:0","nonce":3402717183}]}]} 2024-09-11T09:50:20.869 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-09-11T09:50:20.869 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-09-11T09:50:20.869 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-11T09:50:21.159 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-11T09:50:21.159 INFO:teuthology.orchestra.run.smithi017.stdout:{"epoch":18,"fsid":"a2f1daec-d3b0-4bf5-82c8-72ecbb93d4e4","created":"2024-09-11T09:50:00.165804+0000","modified":"2024-09-11T09:50:20.490609+0000","last_up_change":"2024-09-11T09:50:10.457262+0000","last_in_change":"2024-09-11T09:50:01.801190+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-11T09:50:10.457184+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-11T09:50:16.993202+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":18,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4829b7a1-d607-4a56-b6d2-55036cb8acf8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6808","nonce":4154449929},{"type":"v1","addr":"172.21.15.17:6809","nonce":4154449929}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6810","nonce":4154449929},{"type":"v1","addr":"172.21.15.17:6811","nonce":4154449929}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6814","nonce":4154449929},{"type":"v1","addr":"172.21.15.17:6815","nonce":4154449929}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6812","nonce":4154449929},{"type":"v1","addr":"172.21.15.17:6813","nonce":4154449929}]},"public_addr":"172.21.15.17:6809/4154449929","cluster_addr":"172.21.15.17:6811/4154449929","heartbeat_back_addr":"172.21.15.17:6815/4154449929","heartbeat_front_addr":"172.21.15.17:6813/4154449929","state":["exists","up"]},{"osd":1,"uuid":"ab1d5df0-3c0e-44cd-88a0-96b07964148b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6800","nonce":4139511644},{"type":"v1","addr":"172.21.15.17:6801","nonce":4139511644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6802","nonce":4139511644},{"type":"v1","addr":"172.21.15.17:6803","nonce":4139511644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6806","nonce":4139511644},{"type":"v1","addr":"172.21.15.17:6807","nonce":4139511644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6804","nonce":4139511644},{"type":"v1","addr":"172.21.15.17:6805","nonce":4139511644}]},"public_addr":"172.21.15.17:6801/4139511644","cluster_addr":"172.21.15.17:6803/4139511644","heartbeat_back_addr":"172.21.15.17:6807/4139511644","heartbeat_front_addr":"172.21.15.17:6805/4139511644","state":["exists","up"]},{"osd":2,"uuid":"4c8e50db-b49c-4d44-a913-47b5b30f8699","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6816","nonce":1151282506},{"type":"v1","addr":"172.21.15.17:6817","nonce":1151282506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6818","nonce":1151282506},{"type":"v1","addr":"172.21.15.17:6819","nonce":1151282506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6822","nonce":1151282506},{"type":"v1","addr":"172.21.15.17:6823","nonce":1151282506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6820","nonce":1151282506},{"type":"v1","addr":"172.21.15.17:6821","nonce":1151282506}]},"public_addr":"172.21.15.17:6817/1151282506","cluster_addr":"172.21.15.17:6819/1151282506","heartbeat_back_addr":"172.21.15.17:6823/1151282506","heartbeat_front_addr":"172.21.15.17:6821/1151282506","state":["exists","up"]},{"osd":3,"uuid":"37abc55b-a1eb-49d5-b1ff-b507ee2cb5a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":332854807},{"type":"v1","addr":"172.21.15.96:6817","nonce":332854807}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6818","nonce":332854807},{"type":"v1","addr":"172.21.15.96:6819","nonce":332854807}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":332854807},{"type":"v1","addr":"172.21.15.96:6823","nonce":332854807}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":332854807},{"type":"v1","addr":"172.21.15.96:6821","nonce":332854807}]},"public_addr":"172.21.15.96:6817/332854807","cluster_addr":"172.21.15.96:6819/332854807","heartbeat_back_addr":"172.21.15.96:6823/332854807","heartbeat_front_addr":"172.21.15.96:6821/332854807","state":["exists","up"]},{"osd":4,"uuid":"067d9586-4f5a-4d17-96c5-86a7b8bcd114","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":2785176937},{"type":"v1","addr":"172.21.15.96:6801","nonce":2785176937}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":2785176937},{"type":"v1","addr":"172.21.15.96:6803","nonce":2785176937}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":2785176937},{"type":"v1","addr":"172.21.15.96:6807","nonce":2785176937}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":2785176937},{"type":"v1","addr":"172.21.15.96:6805","nonce":2785176937}]},"public_addr":"172.21.15.96:6801/2785176937","cluster_addr":"172.21.15.96:6803/2785176937","heartbeat_back_addr":"172.21.15.96:6807/2785176937","heartbeat_front_addr":"172.21.15.96:6805/2785176937","state":["exists","up"]},{"osd":5,"uuid":"8ff7a724-0a3a-4aa2-baf1-fd52f88d476c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":509935370},{"type":"v1","addr":"172.21.15.96:6809","nonce":509935370}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6810","nonce":509935370},{"type":"v1","addr":"172.21.15.96:6811","nonce":509935370}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6814","nonce":509935370},{"type":"v1","addr":"172.21.15.96:6815","nonce":509935370}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":509935370},{"type":"v1","addr":"172.21.15.96:6813","nonce":509935370}]},"public_addr":"172.21.15.96:6809/509935370","cluster_addr":"172.21.15.96:6811/509935370","heartbeat_back_addr":"172.21.15.96:6815/509935370","heartbeat_front_addr":"172.21.15.96:6813/509935370","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T09:50:08.513270+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T09:50:07.522863+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T09:50:08.134364+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T09:50:08.622366+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T09:50:08.156142+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T09:50:08.108116+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-11T09:50:21.172 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-09-11T09:50:21.172 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-11T09:50:21.460 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-11T09:50:21.461 INFO:teuthology.orchestra.run.smithi017.stdout:{"epoch":18,"fsid":"a2f1daec-d3b0-4bf5-82c8-72ecbb93d4e4","created":"2024-09-11T09:50:00.165804+0000","modified":"2024-09-11T09:50:20.490609+0000","last_up_change":"2024-09-11T09:50:10.457262+0000","last_in_change":"2024-09-11T09:50:01.801190+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-11T09:50:10.457184+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-11T09:50:16.993202+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":18,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4829b7a1-d607-4a56-b6d2-55036cb8acf8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6808","nonce":4154449929},{"type":"v1","addr":"172.21.15.17:6809","nonce":4154449929}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6810","nonce":4154449929},{"type":"v1","addr":"172.21.15.17:6811","nonce":4154449929}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6814","nonce":4154449929},{"type":"v1","addr":"172.21.15.17:6815","nonce":4154449929}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6812","nonce":4154449929},{"type":"v1","addr":"172.21.15.17:6813","nonce":4154449929}]},"public_addr":"172.21.15.17:6809/4154449929","cluster_addr":"172.21.15.17:6811/4154449929","heartbeat_back_addr":"172.21.15.17:6815/4154449929","heartbeat_front_addr":"172.21.15.17:6813/4154449929","state":["exists","up"]},{"osd":1,"uuid":"ab1d5df0-3c0e-44cd-88a0-96b07964148b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6800","nonce":4139511644},{"type":"v1","addr":"172.21.15.17:6801","nonce":4139511644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6802","nonce":4139511644},{"type":"v1","addr":"172.21.15.17:6803","nonce":4139511644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6806","nonce":4139511644},{"type":"v1","addr":"172.21.15.17:6807","nonce":4139511644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6804","nonce":4139511644},{"type":"v1","addr":"172.21.15.17:6805","nonce":4139511644}]},"public_addr":"172.21.15.17:6801/4139511644","cluster_addr":"172.21.15.17:6803/4139511644","heartbeat_back_addr":"172.21.15.17:6807/4139511644","heartbeat_front_addr":"172.21.15.17:6805/4139511644","state":["exists","up"]},{"osd":2,"uuid":"4c8e50db-b49c-4d44-a913-47b5b30f8699","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6816","nonce":1151282506},{"type":"v1","addr":"172.21.15.17:6817","nonce":1151282506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6818","nonce":1151282506},{"type":"v1","addr":"172.21.15.17:6819","nonce":1151282506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6822","nonce":1151282506},{"type":"v1","addr":"172.21.15.17:6823","nonce":1151282506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6820","nonce":1151282506},{"type":"v1","addr":"172.21.15.17:6821","nonce":1151282506}]},"public_addr":"172.21.15.17:6817/1151282506","cluster_addr":"172.21.15.17:6819/1151282506","heartbeat_back_addr":"172.21.15.17:6823/1151282506","heartbeat_front_addr":"172.21.15.17:6821/1151282506","state":["exists","up"]},{"osd":3,"uuid":"37abc55b-a1eb-49d5-b1ff-b507ee2cb5a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":332854807},{"type":"v1","addr":"172.21.15.96:6817","nonce":332854807}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6818","nonce":332854807},{"type":"v1","addr":"172.21.15.96:6819","nonce":332854807}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":332854807},{"type":"v1","addr":"172.21.15.96:6823","nonce":332854807}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":332854807},{"type":"v1","addr":"172.21.15.96:6821","nonce":332854807}]},"public_addr":"172.21.15.96:6817/332854807","cluster_addr":"172.21.15.96:6819/332854807","heartbeat_back_addr":"172.21.15.96:6823/332854807","heartbeat_front_addr":"172.21.15.96:6821/332854807","state":["exists","up"]},{"osd":4,"uuid":"067d9586-4f5a-4d17-96c5-86a7b8bcd114","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":2785176937},{"type":"v1","addr":"172.21.15.96:6801","nonce":2785176937}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":2785176937},{"type":"v1","addr":"172.21.15.96:6803","nonce":2785176937}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":2785176937},{"type":"v1","addr":"172.21.15.96:6807","nonce":2785176937}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":2785176937},{"type":"v1","addr":"172.21.15.96:6805","nonce":2785176937}]},"public_addr":"172.21.15.96:6801/2785176937","cluster_addr":"172.21.15.96:6803/2785176937","heartbeat_back_addr":"172.21.15.96:6807/2785176937","heartbeat_front_addr":"172.21.15.96:6805/2785176937","state":["exists","up"]},{"osd":5,"uuid":"8ff7a724-0a3a-4aa2-baf1-fd52f88d476c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":509935370},{"type":"v1","addr":"172.21.15.96:6809","nonce":509935370}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6810","nonce":509935370},{"type":"v1","addr":"172.21.15.96:6811","nonce":509935370}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6814","nonce":509935370},{"type":"v1","addr":"172.21.15.96:6815","nonce":509935370}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":509935370},{"type":"v1","addr":"172.21.15.96:6813","nonce":509935370}]},"public_addr":"172.21.15.96:6809/509935370","cluster_addr":"172.21.15.96:6811/509935370","heartbeat_back_addr":"172.21.15.96:6815/509935370","heartbeat_front_addr":"172.21.15.96:6813/509935370","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T09:50:08.513270+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T09:50:07.522863+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T09:50:08.134364+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T09:50:08.622366+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T09:50:08.156142+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T09:50:08.108116+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-11T09:50:21.473 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-09-11T09:50:21.474 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-09-11T09:50:21.474 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-09-11T09:50:21.475 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-09-11T09:50:21.475 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-09-11T09:50:21.476 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-09-11T09:50:21.654 INFO:teuthology.orchestra.run.smithi017.stdout:47244640261 2024-09-11T09:50:21.654 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-11T09:50:21.657 INFO:teuthology.orchestra.run.smithi017.stdout:47244640261 2024-09-11T09:50:21.658 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-11T09:50:21.686 INFO:teuthology.orchestra.run.smithi017.stdout:47244640262 2024-09-11T09:50:21.686 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-09-11T09:50:21.687 INFO:teuthology.orchestra.run.smithi017.stdout:51539607558 2024-09-11T09:50:21.688 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-11T09:50:21.703 INFO:teuthology.orchestra.run.smithi017.stdout:51539607557 2024-09-11T09:50:21.703 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-11T09:50:21.704 INFO:teuthology.orchestra.run.smithi017.stdout:47244640262 2024-09-11T09:50:21.704 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-09-11T09:50:22.027 INFO:teuthology.orchestra.run.smithi017.stdout:47244640259 2024-09-11T09:50:22.044 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640261 got 47244640259 for osd.1 2024-09-11T09:50:22.084 INFO:teuthology.orchestra.run.smithi017.stdout:47244640260 2024-09-11T09:50:22.098 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640262 got 47244640260 for osd.4 2024-09-11T09:50:22.114 INFO:teuthology.orchestra.run.smithi017.stdout:47244640259 2024-09-11T09:50:22.127 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640261 got 47244640259 for osd.2 2024-09-11T09:50:22.169 INFO:teuthology.orchestra.run.smithi017.stdout:47244640260 2024-09-11T09:50:22.174 INFO:teuthology.orchestra.run.smithi017.stdout:51539607556 2024-09-11T09:50:22.186 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640262 got 47244640260 for osd.5 2024-09-11T09:50:22.187 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607558 got 51539607556 for osd.3 2024-09-11T09:50:22.241 INFO:teuthology.orchestra.run.smithi017.stdout:51539607555 2024-09-11T09:50:22.254 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607557 got 51539607555 for osd.0 2024-09-11T09:50:23.045 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-11T09:50:23.099 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-09-11T09:50:23.128 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-11T09:50:23.187 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-09-11T09:50:23.187 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-11T09:50:23.256 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-11T09:50:23.519 INFO:teuthology.orchestra.run.smithi017.stdout:47244640261 2024-09-11T09:50:23.537 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640261 got 47244640261 for osd.1 2024-09-11T09:50:23.537 DEBUG:teuthology.parallel:result is None 2024-09-11T09:50:23.552 INFO:teuthology.orchestra.run.smithi017.stdout:47244640261 2024-09-11T09:50:23.571 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640261 got 47244640261 for osd.2 2024-09-11T09:50:23.571 DEBUG:teuthology.parallel:result is None 2024-09-11T09:50:23.579 INFO:teuthology.orchestra.run.smithi017.stdout:47244640262 2024-09-11T09:50:23.586 INFO:teuthology.orchestra.run.smithi017.stdout:47244640262 2024-09-11T09:50:23.593 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640262 got 47244640262 for osd.5 2024-09-11T09:50:23.594 DEBUG:teuthology.parallel:result is None 2024-09-11T09:50:23.601 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640262 got 47244640262 for osd.4 2024-09-11T09:50:23.601 DEBUG:teuthology.parallel:result is None 2024-09-11T09:50:23.633 INFO:teuthology.orchestra.run.smithi017.stdout:51539607558 2024-09-11T09:50:23.646 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607558 got 51539607558 for osd.3 2024-09-11T09:50:23.646 DEBUG:teuthology.parallel:result is None 2024-09-11T09:50:23.682 INFO:teuthology.orchestra.run.smithi017.stdout:51539607557 2024-09-11T09:50:23.695 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607557 got 51539607557 for osd.0 2024-09-11T09:50:23.695 DEBUG:teuthology.parallel:result is None 2024-09-11T09:50:23.695 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-11T09:50:23.695 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T09:50:24.021 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-11T09:50:24.021 INFO:teuthology.orchestra.run.smithi017.stderr:dumped all 2024-09-11T09:50:24.033 INFO:teuthology.orchestra.run.smithi017.stdout:{"pg_ready":true,"pg_map":{"version":19,"stamp":"2024-09-11T09:50:22.417041+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":161,"num_write_kb":2418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120,"ondisk_log_size":120,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":175800,"kb_used_data":1120,"kb_used_omap":9,"kb_used_meta":174518,"kb_avail":566055240,"statfs":{"total":579820584960,"available":579640565760,"internally_reserved":0,"allocated":1146880,"data_stored":1512556,"data_compressed":64992,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":9566,"internal_metadata":178707106},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.934806"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":19,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-11T09:50:20.497924+0000","last_change":"2024-09-11T09:50:20.497924+0000","last_active":"2024-09-11T09:50:20.497924+0000","last_peered":"2024-09-11T09:50:20.497924+0000","last_clean":"2024-09-11T09:50:20.497924+0000","last_became_active":"2024-09-11T09:50:18.544390+0000","last_became_peered":"2024-09-11T09:50:18.544390+0000","last_unstale":"2024-09-11T09:50:20.497924+0000","last_undegraded":"2024-09-11T09:50:20.497924+0000","last_fullsized":"2024-09-11T09:50:20.497924+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_clean_scrub_stamp":"2024-09-11T09:50:17.477890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T17:32:48.479885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00046094899999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":19,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-11T09:50:20.497127+0000","last_change":"2024-09-11T09:50:20.497446+0000","last_active":"2024-09-11T09:50:20.497127+0000","last_peered":"2024-09-11T09:50:20.497127+0000","last_clean":"2024-09-11T09:50:20.497127+0000","last_became_active":"2024-09-11T09:50:18.497481+0000","last_became_peered":"2024-09-11T09:50:18.497481+0000","last_unstale":"2024-09-11T09:50:20.497127+0000","last_undegraded":"2024-09-11T09:50:20.497127+0000","last_fullsized":"2024-09-11T09:50:20.497127+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_clean_scrub_stamp":"2024-09-11T09:50:17.477890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T14:12:09.857529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00050697600000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":19,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-11T09:50:20.497924+0000","last_change":"2024-09-11T09:50:20.497924+0000","last_active":"2024-09-11T09:50:20.497924+0000","last_peered":"2024-09-11T09:50:20.497924+0000","last_clean":"2024-09-11T09:50:20.497924+0000","last_became_active":"2024-09-11T09:50:18.489881+0000","last_became_peered":"2024-09-11T09:50:18.489881+0000","last_unstale":"2024-09-11T09:50:20.497924+0000","last_undegraded":"2024-09-11T09:50:20.497924+0000","last_fullsized":"2024-09-11T09:50:20.497924+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_clean_scrub_stamp":"2024-09-11T09:50:17.477890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T20:33:05.471570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045546699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":19,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-11T09:50:20.497074+0000","last_change":"2024-09-11T09:50:20.497435+0000","last_active":"2024-09-11T09:50:20.497074+0000","last_peered":"2024-09-11T09:50:20.497074+0000","last_clean":"2024-09-11T09:50:20.497074+0000","last_became_active":"2024-09-11T09:50:18.492880+0000","last_became_peered":"2024-09-11T09:50:18.492880+0000","last_unstale":"2024-09-11T09:50:20.497074+0000","last_undegraded":"2024-09-11T09:50:20.497074+0000","last_fullsized":"2024-09-11T09:50:20.497074+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_clean_scrub_stamp":"2024-09-11T09:50:17.477890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T17:22:43.514126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00050337500000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"18'2","reported_seq":21,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-11T09:50:20.501218+0000","last_change":"2024-09-11T09:50:20.501218+0000","last_active":"2024-09-11T09:50:20.501218+0000","last_peered":"2024-09-11T09:50:20.501218+0000","last_clean":"2024-09-11T09:50:20.501218+0000","last_became_active":"2024-09-11T09:50:18.493165+0000","last_became_peered":"2024-09-11T09:50:18.493165+0000","last_unstale":"2024-09-11T09:50:20.501218+0000","last_undegraded":"2024-09-11T09:50:20.501218+0000","last_fullsized":"2024-09-11T09:50:20.501218+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_clean_scrub_stamp":"2024-09-11T09:50:17.477890+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T18:04:31.250555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00094463900000000005,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":19,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-11T09:50:21.030263+0000","last_change":"2024-09-11T09:50:21.030566+0000","last_active":"2024-09-11T09:50:21.030263+0000","last_peered":"2024-09-11T09:50:21.030263+0000","last_clean":"2024-09-11T09:50:21.030263+0000","last_became_active":"2024-09-11T09:50:18.493109+0000","last_became_peered":"2024-09-11T09:50:18.493109+0000","last_unstale":"2024-09-11T09:50:21.030263+0000","last_undegraded":"2024-09-11T09:50:21.030263+0000","last_fullsized":"2024-09-11T09:50:21.030263+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_clean_scrub_stamp":"2024-09-11T09:50:17.477890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:37:16.328368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041008499999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":19,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-11T09:50:20.498284+0000","last_change":"2024-09-11T09:50:20.498284+0000","last_active":"2024-09-11T09:50:20.498284+0000","last_peered":"2024-09-11T09:50:20.498284+0000","last_clean":"2024-09-11T09:50:20.498284+0000","last_became_active":"2024-09-11T09:50:18.493210+0000","last_became_peered":"2024-09-11T09:50:18.493210+0000","last_unstale":"2024-09-11T09:50:20.498284+0000","last_undegraded":"2024-09-11T09:50:20.498284+0000","last_fullsized":"2024-09-11T09:50:20.498284+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_clean_scrub_stamp":"2024-09-11T09:50:17.477890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T18:16:13.887681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079749499999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"16'1","reported_seq":20,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-11T09:50:20.500435+0000","last_change":"2024-09-11T09:50:20.500687+0000","last_active":"2024-09-11T09:50:20.500435+0000","last_peered":"2024-09-11T09:50:20.500435+0000","last_clean":"2024-09-11T09:50:20.500435+0000","last_became_active":"2024-09-11T09:50:18.493650+0000","last_became_peered":"2024-09-11T09:50:18.493650+0000","last_unstale":"2024-09-11T09:50:20.500435+0000","last_undegraded":"2024-09-11T09:50:20.500435+0000","last_fullsized":"2024-09-11T09:50:20.500435+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_clean_scrub_stamp":"2024-09-11T09:50:17.477890+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T14:26:32.838143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041261100000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"13'117","reported_seq":170,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-11T09:50:20.497413+0000","last_change":"2024-09-11T09:50:11.472613+0000","last_active":"2024-09-11T09:50:20.497413+0000","last_peered":"2024-09-11T09:50:20.497413+0000","last_clean":"2024-09-11T09:50:20.497413+0000","last_became_active":"2024-09-11T09:50:11.472306+0000","last_became_peered":"2024-09-11T09:50:11.472306+0000","last_unstale":"2024-09-11T09:50:20.497413+0000","last_undegraded":"2024-09-11T09:50:20.497413+0000","last_fullsized":"2024-09-11T09:50:20.497413+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:50:10.457262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:50:10.457262+0000","last_clean_scrub_stamp":"2024-09-11T09:50:10.457262+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T20:43:33.725287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":64992,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":12,"seq":51539607558,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27264,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344576,"statfs":{"total":96636764160,"available":96608845824,"internally_reserved":0,"allocated":237568,"data_stored":647321,"data_compressed":32496,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":12,"seq":51539607557,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31424,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340416,"statfs":{"total":96636764160,"available":96604585984,"internally_reserved":0,"allocated":237568,"data_stored":647321,"data_compressed":32496,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":11,"seq":47244640262,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27208,"kb_used_data":172,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344632,"statfs":{"total":96636764160,"available":96608903168,"internally_reserved":0,"allocated":176128,"data_stored":56972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":11,"seq":47244640262,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27188,"kb_used_data":156,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344652,"statfs":{"total":96636764160,"available":96608923648,"internally_reserved":0,"allocated":159744,"data_stored":51985,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640261,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31348,"kb_used_data":156,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340492,"statfs":{"total":96636764160,"available":96604663808,"internally_reserved":0,"allocated":159744,"data_stored":51985,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640261,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31368,"kb_used_data":172,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340472,"statfs":{"total":96636764160,"available":96604643328,"internally_reserved":0,"allocated":176128,"data_stored":56972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32496,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32496,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T09:50:24.034 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T09:50:24.321 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-11T09:50:24.322 INFO:teuthology.orchestra.run.smithi017.stderr:dumped all 2024-09-11T09:50:24.333 INFO:teuthology.orchestra.run.smithi017.stdout:{"pg_ready":true,"pg_map":{"version":19,"stamp":"2024-09-11T09:50:22.417041+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":161,"num_write_kb":2418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120,"ondisk_log_size":120,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":175800,"kb_used_data":1120,"kb_used_omap":9,"kb_used_meta":174518,"kb_avail":566055240,"statfs":{"total":579820584960,"available":579640565760,"internally_reserved":0,"allocated":1146880,"data_stored":1512556,"data_compressed":64992,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":9566,"internal_metadata":178707106},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.934806"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":19,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-11T09:50:20.497924+0000","last_change":"2024-09-11T09:50:20.497924+0000","last_active":"2024-09-11T09:50:20.497924+0000","last_peered":"2024-09-11T09:50:20.497924+0000","last_clean":"2024-09-11T09:50:20.497924+0000","last_became_active":"2024-09-11T09:50:18.544390+0000","last_became_peered":"2024-09-11T09:50:18.544390+0000","last_unstale":"2024-09-11T09:50:20.497924+0000","last_undegraded":"2024-09-11T09:50:20.497924+0000","last_fullsized":"2024-09-11T09:50:20.497924+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_clean_scrub_stamp":"2024-09-11T09:50:17.477890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T17:32:48.479885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00046094899999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":19,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-11T09:50:20.497127+0000","last_change":"2024-09-11T09:50:20.497446+0000","last_active":"2024-09-11T09:50:20.497127+0000","last_peered":"2024-09-11T09:50:20.497127+0000","last_clean":"2024-09-11T09:50:20.497127+0000","last_became_active":"2024-09-11T09:50:18.497481+0000","last_became_peered":"2024-09-11T09:50:18.497481+0000","last_unstale":"2024-09-11T09:50:20.497127+0000","last_undegraded":"2024-09-11T09:50:20.497127+0000","last_fullsized":"2024-09-11T09:50:20.497127+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_clean_scrub_stamp":"2024-09-11T09:50:17.477890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T14:12:09.857529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00050697600000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":19,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-11T09:50:20.497924+0000","last_change":"2024-09-11T09:50:20.497924+0000","last_active":"2024-09-11T09:50:20.497924+0000","last_peered":"2024-09-11T09:50:20.497924+0000","last_clean":"2024-09-11T09:50:20.497924+0000","last_became_active":"2024-09-11T09:50:18.489881+0000","last_became_peered":"2024-09-11T09:50:18.489881+0000","last_unstale":"2024-09-11T09:50:20.497924+0000","last_undegraded":"2024-09-11T09:50:20.497924+0000","last_fullsized":"2024-09-11T09:50:20.497924+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_clean_scrub_stamp":"2024-09-11T09:50:17.477890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T20:33:05.471570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045546699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":19,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-11T09:50:20.497074+0000","last_change":"2024-09-11T09:50:20.497435+0000","last_active":"2024-09-11T09:50:20.497074+0000","last_peered":"2024-09-11T09:50:20.497074+0000","last_clean":"2024-09-11T09:50:20.497074+0000","last_became_active":"2024-09-11T09:50:18.492880+0000","last_became_peered":"2024-09-11T09:50:18.492880+0000","last_unstale":"2024-09-11T09:50:20.497074+0000","last_undegraded":"2024-09-11T09:50:20.497074+0000","last_fullsized":"2024-09-11T09:50:20.497074+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_clean_scrub_stamp":"2024-09-11T09:50:17.477890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T17:22:43.514126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00050337500000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"18'2","reported_seq":21,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-11T09:50:20.501218+0000","last_change":"2024-09-11T09:50:20.501218+0000","last_active":"2024-09-11T09:50:20.501218+0000","last_peered":"2024-09-11T09:50:20.501218+0000","last_clean":"2024-09-11T09:50:20.501218+0000","last_became_active":"2024-09-11T09:50:18.493165+0000","last_became_peered":"2024-09-11T09:50:18.493165+0000","last_unstale":"2024-09-11T09:50:20.501218+0000","last_undegraded":"2024-09-11T09:50:20.501218+0000","last_fullsized":"2024-09-11T09:50:20.501218+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_clean_scrub_stamp":"2024-09-11T09:50:17.477890+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T18:04:31.250555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00094463900000000005,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":19,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-11T09:50:21.030263+0000","last_change":"2024-09-11T09:50:21.030566+0000","last_active":"2024-09-11T09:50:21.030263+0000","last_peered":"2024-09-11T09:50:21.030263+0000","last_clean":"2024-09-11T09:50:21.030263+0000","last_became_active":"2024-09-11T09:50:18.493109+0000","last_became_peered":"2024-09-11T09:50:18.493109+0000","last_unstale":"2024-09-11T09:50:21.030263+0000","last_undegraded":"2024-09-11T09:50:21.030263+0000","last_fullsized":"2024-09-11T09:50:21.030263+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_clean_scrub_stamp":"2024-09-11T09:50:17.477890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:37:16.328368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041008499999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":19,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-11T09:50:20.498284+0000","last_change":"2024-09-11T09:50:20.498284+0000","last_active":"2024-09-11T09:50:20.498284+0000","last_peered":"2024-09-11T09:50:20.498284+0000","last_clean":"2024-09-11T09:50:20.498284+0000","last_became_active":"2024-09-11T09:50:18.493210+0000","last_became_peered":"2024-09-11T09:50:18.493210+0000","last_unstale":"2024-09-11T09:50:20.498284+0000","last_undegraded":"2024-09-11T09:50:20.498284+0000","last_fullsized":"2024-09-11T09:50:20.498284+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_clean_scrub_stamp":"2024-09-11T09:50:17.477890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T18:16:13.887681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079749499999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"16'1","reported_seq":20,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-11T09:50:20.500435+0000","last_change":"2024-09-11T09:50:20.500687+0000","last_active":"2024-09-11T09:50:20.500435+0000","last_peered":"2024-09-11T09:50:20.500435+0000","last_clean":"2024-09-11T09:50:20.500435+0000","last_became_active":"2024-09-11T09:50:18.493650+0000","last_became_peered":"2024-09-11T09:50:18.493650+0000","last_unstale":"2024-09-11T09:50:20.500435+0000","last_undegraded":"2024-09-11T09:50:20.500435+0000","last_fullsized":"2024-09-11T09:50:20.500435+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_clean_scrub_stamp":"2024-09-11T09:50:17.477890+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T14:26:32.838143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041261100000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"13'117","reported_seq":170,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-11T09:50:20.497413+0000","last_change":"2024-09-11T09:50:11.472613+0000","last_active":"2024-09-11T09:50:20.497413+0000","last_peered":"2024-09-11T09:50:20.497413+0000","last_clean":"2024-09-11T09:50:20.497413+0000","last_became_active":"2024-09-11T09:50:11.472306+0000","last_became_peered":"2024-09-11T09:50:11.472306+0000","last_unstale":"2024-09-11T09:50:20.497413+0000","last_undegraded":"2024-09-11T09:50:20.497413+0000","last_fullsized":"2024-09-11T09:50:20.497413+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:50:10.457262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:50:10.457262+0000","last_clean_scrub_stamp":"2024-09-11T09:50:10.457262+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T20:43:33.725287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":64992,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":12,"seq":51539607558,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27264,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344576,"statfs":{"total":96636764160,"available":96608845824,"internally_reserved":0,"allocated":237568,"data_stored":647321,"data_compressed":32496,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":12,"seq":51539607557,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31424,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340416,"statfs":{"total":96636764160,"available":96604585984,"internally_reserved":0,"allocated":237568,"data_stored":647321,"data_compressed":32496,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":11,"seq":47244640262,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27208,"kb_used_data":172,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344632,"statfs":{"total":96636764160,"available":96608903168,"internally_reserved":0,"allocated":176128,"data_stored":56972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":11,"seq":47244640262,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27188,"kb_used_data":156,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344652,"statfs":{"total":96636764160,"available":96608923648,"internally_reserved":0,"allocated":159744,"data_stored":51985,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640261,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31348,"kb_used_data":156,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340492,"statfs":{"total":96636764160,"available":96604663808,"internally_reserved":0,"allocated":159744,"data_stored":51985,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640261,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31368,"kb_used_data":172,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340472,"statfs":{"total":96636764160,"available":96604643328,"internally_reserved":0,"allocated":176128,"data_stored":56972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32496,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32496,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T09:50:24.334 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-11T09:50:24.334 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-09-11T09:50:24.335 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-09-11T09:50:24.335 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-09-11T09:50:24.660 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-11T09:50:24.660 INFO:teuthology.orchestra.run.smithi017.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-11T09:50:24.673 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-09-11T09:50:24.673 INFO:teuthology.run_tasks:Running task thrashosds... 2024-09-11T09:50:24.684 INFO:tasks.thrashosds:config is {'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0} 2024-09-11T09:50:24.684 INFO:tasks.thrashosds:overrides is {'admin_socket': {'branch': 'squid'}, 'ceph': {'conf': {'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum', 'noscrub', 'nodeep-scrub'], 'sha1': 'b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'extra_packages': ['rbd-nbd'], 'flavor': 'default', 'sha1': 'b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc'}}, 'selinux': {'allowlist': ['scontext=system_u:system_r:getty_t:s0']}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'squid', 'sha1': 'daaf78de334647fc0c800f4ac495cc9475a26bae'}} 2024-09-11T09:50:24.684 INFO:tasks.thrashosds:config is {'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0, 'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5} 2024-09-11T09:50:24.685 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr_debug_aggressive_pg_num_changes true 2024-09-11T09:50:24.994 INFO:tasks.thrashosds:Beginning thrashosds... 2024-09-11T09:50:24.994 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-11T09:50:24.994 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T09:50:25.282 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-11T09:50:25.282 INFO:teuthology.orchestra.run.smithi017.stderr:dumped all 2024-09-11T09:50:25.294 INFO:teuthology.orchestra.run.smithi017.stdout:{"pg_ready":true,"pg_map":{"version":21,"stamp":"2024-09-11T09:50:24.949095+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":161,"num_write_kb":2418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120,"ondisk_log_size":120,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":175800,"kb_used_data":1120,"kb_used_omap":9,"kb_used_meta":174518,"kb_avail":566055240,"statfs":{"total":579820584960,"available":579640565760,"internally_reserved":0,"allocated":1146880,"data_stored":1512556,"data_compressed":64992,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":9566,"internal_metadata":178707106},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.463027"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":19,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-11T09:50:20.497924+0000","last_change":"2024-09-11T09:50:20.497924+0000","last_active":"2024-09-11T09:50:20.497924+0000","last_peered":"2024-09-11T09:50:20.497924+0000","last_clean":"2024-09-11T09:50:20.497924+0000","last_became_active":"2024-09-11T09:50:18.544390+0000","last_became_peered":"2024-09-11T09:50:18.544390+0000","last_unstale":"2024-09-11T09:50:20.497924+0000","last_undegraded":"2024-09-11T09:50:20.497924+0000","last_fullsized":"2024-09-11T09:50:20.497924+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_clean_scrub_stamp":"2024-09-11T09:50:17.477890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T17:32:48.479885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00046094899999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":19,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-11T09:50:20.497127+0000","last_change":"2024-09-11T09:50:20.497446+0000","last_active":"2024-09-11T09:50:20.497127+0000","last_peered":"2024-09-11T09:50:20.497127+0000","last_clean":"2024-09-11T09:50:20.497127+0000","last_became_active":"2024-09-11T09:50:18.497481+0000","last_became_peered":"2024-09-11T09:50:18.497481+0000","last_unstale":"2024-09-11T09:50:20.497127+0000","last_undegraded":"2024-09-11T09:50:20.497127+0000","last_fullsized":"2024-09-11T09:50:20.497127+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_clean_scrub_stamp":"2024-09-11T09:50:17.477890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T14:12:09.857529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00050697600000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":19,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-11T09:50:20.497924+0000","last_change":"2024-09-11T09:50:20.497924+0000","last_active":"2024-09-11T09:50:20.497924+0000","last_peered":"2024-09-11T09:50:20.497924+0000","last_clean":"2024-09-11T09:50:20.497924+0000","last_became_active":"2024-09-11T09:50:18.489881+0000","last_became_peered":"2024-09-11T09:50:18.489881+0000","last_unstale":"2024-09-11T09:50:20.497924+0000","last_undegraded":"2024-09-11T09:50:20.497924+0000","last_fullsized":"2024-09-11T09:50:20.497924+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_clean_scrub_stamp":"2024-09-11T09:50:17.477890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T20:33:05.471570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045546699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":19,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-11T09:50:20.497074+0000","last_change":"2024-09-11T09:50:20.497435+0000","last_active":"2024-09-11T09:50:20.497074+0000","last_peered":"2024-09-11T09:50:20.497074+0000","last_clean":"2024-09-11T09:50:20.497074+0000","last_became_active":"2024-09-11T09:50:18.492880+0000","last_became_peered":"2024-09-11T09:50:18.492880+0000","last_unstale":"2024-09-11T09:50:20.497074+0000","last_undegraded":"2024-09-11T09:50:20.497074+0000","last_fullsized":"2024-09-11T09:50:20.497074+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_clean_scrub_stamp":"2024-09-11T09:50:17.477890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T17:22:43.514126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00050337500000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"18'2","reported_seq":21,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-11T09:50:20.501218+0000","last_change":"2024-09-11T09:50:20.501218+0000","last_active":"2024-09-11T09:50:20.501218+0000","last_peered":"2024-09-11T09:50:20.501218+0000","last_clean":"2024-09-11T09:50:20.501218+0000","last_became_active":"2024-09-11T09:50:18.493165+0000","last_became_peered":"2024-09-11T09:50:18.493165+0000","last_unstale":"2024-09-11T09:50:20.501218+0000","last_undegraded":"2024-09-11T09:50:20.501218+0000","last_fullsized":"2024-09-11T09:50:20.501218+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_clean_scrub_stamp":"2024-09-11T09:50:17.477890+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T18:04:31.250555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00094463900000000005,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":19,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-11T09:50:21.030263+0000","last_change":"2024-09-11T09:50:21.030566+0000","last_active":"2024-09-11T09:50:21.030263+0000","last_peered":"2024-09-11T09:50:21.030263+0000","last_clean":"2024-09-11T09:50:21.030263+0000","last_became_active":"2024-09-11T09:50:18.493109+0000","last_became_peered":"2024-09-11T09:50:18.493109+0000","last_unstale":"2024-09-11T09:50:21.030263+0000","last_undegraded":"2024-09-11T09:50:21.030263+0000","last_fullsized":"2024-09-11T09:50:21.030263+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_clean_scrub_stamp":"2024-09-11T09:50:17.477890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:37:16.328368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041008499999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":19,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-11T09:50:20.498284+0000","last_change":"2024-09-11T09:50:20.498284+0000","last_active":"2024-09-11T09:50:20.498284+0000","last_peered":"2024-09-11T09:50:20.498284+0000","last_clean":"2024-09-11T09:50:20.498284+0000","last_became_active":"2024-09-11T09:50:18.493210+0000","last_became_peered":"2024-09-11T09:50:18.493210+0000","last_unstale":"2024-09-11T09:50:20.498284+0000","last_undegraded":"2024-09-11T09:50:20.498284+0000","last_fullsized":"2024-09-11T09:50:20.498284+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_clean_scrub_stamp":"2024-09-11T09:50:17.477890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T18:16:13.887681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079749499999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"16'1","reported_seq":20,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-11T09:50:20.500435+0000","last_change":"2024-09-11T09:50:20.500687+0000","last_active":"2024-09-11T09:50:20.500435+0000","last_peered":"2024-09-11T09:50:20.500435+0000","last_clean":"2024-09-11T09:50:20.500435+0000","last_became_active":"2024-09-11T09:50:18.493650+0000","last_became_peered":"2024-09-11T09:50:18.493650+0000","last_unstale":"2024-09-11T09:50:20.500435+0000","last_undegraded":"2024-09-11T09:50:20.500435+0000","last_fullsized":"2024-09-11T09:50:20.500435+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_clean_scrub_stamp":"2024-09-11T09:50:17.477890+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T14:26:32.838143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041261100000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"13'117","reported_seq":170,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-11T09:50:20.497413+0000","last_change":"2024-09-11T09:50:11.472613+0000","last_active":"2024-09-11T09:50:20.497413+0000","last_peered":"2024-09-11T09:50:20.497413+0000","last_clean":"2024-09-11T09:50:20.497413+0000","last_became_active":"2024-09-11T09:50:11.472306+0000","last_became_peered":"2024-09-11T09:50:11.472306+0000","last_unstale":"2024-09-11T09:50:20.497413+0000","last_undegraded":"2024-09-11T09:50:20.497413+0000","last_fullsized":"2024-09-11T09:50:20.497413+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:50:10.457262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:50:10.457262+0000","last_clean_scrub_stamp":"2024-09-11T09:50:10.457262+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T20:43:33.725287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":64992,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":12,"seq":51539607558,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27264,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344576,"statfs":{"total":96636764160,"available":96608845824,"internally_reserved":0,"allocated":237568,"data_stored":647321,"data_compressed":32496,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":12,"seq":51539607557,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31424,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340416,"statfs":{"total":96636764160,"available":96604585984,"internally_reserved":0,"allocated":237568,"data_stored":647321,"data_compressed":32496,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":11,"seq":47244640262,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27208,"kb_used_data":172,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344632,"statfs":{"total":96636764160,"available":96608903168,"internally_reserved":0,"allocated":176128,"data_stored":56972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":11,"seq":47244640262,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27188,"kb_used_data":156,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344652,"statfs":{"total":96636764160,"available":96608923648,"internally_reserved":0,"allocated":159744,"data_stored":51985,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640261,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31348,"kb_used_data":156,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340492,"statfs":{"total":96636764160,"available":96604663808,"internally_reserved":0,"allocated":159744,"data_stored":51985,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640261,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31368,"kb_used_data":172,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340472,"statfs":{"total":96636764160,"available":96604643328,"internally_reserved":0,"allocated":176128,"data_stored":56972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32496,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32496,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T09:50:25.295 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T09:50:25.584 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-11T09:50:25.584 INFO:teuthology.orchestra.run.smithi017.stderr:dumped all 2024-09-11T09:50:25.596 INFO:teuthology.orchestra.run.smithi017.stdout:{"pg_ready":true,"pg_map":{"version":21,"stamp":"2024-09-11T09:50:24.949095+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":161,"num_write_kb":2418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120,"ondisk_log_size":120,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":175800,"kb_used_data":1120,"kb_used_omap":9,"kb_used_meta":174518,"kb_avail":566055240,"statfs":{"total":579820584960,"available":579640565760,"internally_reserved":0,"allocated":1146880,"data_stored":1512556,"data_compressed":64992,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":9566,"internal_metadata":178707106},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.463027"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":19,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-11T09:50:20.497924+0000","last_change":"2024-09-11T09:50:20.497924+0000","last_active":"2024-09-11T09:50:20.497924+0000","last_peered":"2024-09-11T09:50:20.497924+0000","last_clean":"2024-09-11T09:50:20.497924+0000","last_became_active":"2024-09-11T09:50:18.544390+0000","last_became_peered":"2024-09-11T09:50:18.544390+0000","last_unstale":"2024-09-11T09:50:20.497924+0000","last_undegraded":"2024-09-11T09:50:20.497924+0000","last_fullsized":"2024-09-11T09:50:20.497924+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_clean_scrub_stamp":"2024-09-11T09:50:17.477890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T17:32:48.479885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00046094899999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":19,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-11T09:50:20.497127+0000","last_change":"2024-09-11T09:50:20.497446+0000","last_active":"2024-09-11T09:50:20.497127+0000","last_peered":"2024-09-11T09:50:20.497127+0000","last_clean":"2024-09-11T09:50:20.497127+0000","last_became_active":"2024-09-11T09:50:18.497481+0000","last_became_peered":"2024-09-11T09:50:18.497481+0000","last_unstale":"2024-09-11T09:50:20.497127+0000","last_undegraded":"2024-09-11T09:50:20.497127+0000","last_fullsized":"2024-09-11T09:50:20.497127+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_clean_scrub_stamp":"2024-09-11T09:50:17.477890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T14:12:09.857529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00050697600000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":19,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-11T09:50:20.497924+0000","last_change":"2024-09-11T09:50:20.497924+0000","last_active":"2024-09-11T09:50:20.497924+0000","last_peered":"2024-09-11T09:50:20.497924+0000","last_clean":"2024-09-11T09:50:20.497924+0000","last_became_active":"2024-09-11T09:50:18.489881+0000","last_became_peered":"2024-09-11T09:50:18.489881+0000","last_unstale":"2024-09-11T09:50:20.497924+0000","last_undegraded":"2024-09-11T09:50:20.497924+0000","last_fullsized":"2024-09-11T09:50:20.497924+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_clean_scrub_stamp":"2024-09-11T09:50:17.477890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T20:33:05.471570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045546699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":19,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-11T09:50:20.497074+0000","last_change":"2024-09-11T09:50:20.497435+0000","last_active":"2024-09-11T09:50:20.497074+0000","last_peered":"2024-09-11T09:50:20.497074+0000","last_clean":"2024-09-11T09:50:20.497074+0000","last_became_active":"2024-09-11T09:50:18.492880+0000","last_became_peered":"2024-09-11T09:50:18.492880+0000","last_unstale":"2024-09-11T09:50:20.497074+0000","last_undegraded":"2024-09-11T09:50:20.497074+0000","last_fullsized":"2024-09-11T09:50:20.497074+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_clean_scrub_stamp":"2024-09-11T09:50:17.477890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T17:22:43.514126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00050337500000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"18'2","reported_seq":21,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-11T09:50:20.501218+0000","last_change":"2024-09-11T09:50:20.501218+0000","last_active":"2024-09-11T09:50:20.501218+0000","last_peered":"2024-09-11T09:50:20.501218+0000","last_clean":"2024-09-11T09:50:20.501218+0000","last_became_active":"2024-09-11T09:50:18.493165+0000","last_became_peered":"2024-09-11T09:50:18.493165+0000","last_unstale":"2024-09-11T09:50:20.501218+0000","last_undegraded":"2024-09-11T09:50:20.501218+0000","last_fullsized":"2024-09-11T09:50:20.501218+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_clean_scrub_stamp":"2024-09-11T09:50:17.477890+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T18:04:31.250555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00094463900000000005,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":19,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-11T09:50:21.030263+0000","last_change":"2024-09-11T09:50:21.030566+0000","last_active":"2024-09-11T09:50:21.030263+0000","last_peered":"2024-09-11T09:50:21.030263+0000","last_clean":"2024-09-11T09:50:21.030263+0000","last_became_active":"2024-09-11T09:50:18.493109+0000","last_became_peered":"2024-09-11T09:50:18.493109+0000","last_unstale":"2024-09-11T09:50:21.030263+0000","last_undegraded":"2024-09-11T09:50:21.030263+0000","last_fullsized":"2024-09-11T09:50:21.030263+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_clean_scrub_stamp":"2024-09-11T09:50:17.477890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:37:16.328368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041008499999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":19,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-11T09:50:20.498284+0000","last_change":"2024-09-11T09:50:20.498284+0000","last_active":"2024-09-11T09:50:20.498284+0000","last_peered":"2024-09-11T09:50:20.498284+0000","last_clean":"2024-09-11T09:50:20.498284+0000","last_became_active":"2024-09-11T09:50:18.493210+0000","last_became_peered":"2024-09-11T09:50:18.493210+0000","last_unstale":"2024-09-11T09:50:20.498284+0000","last_undegraded":"2024-09-11T09:50:20.498284+0000","last_fullsized":"2024-09-11T09:50:20.498284+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_clean_scrub_stamp":"2024-09-11T09:50:17.477890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T18:16:13.887681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079749499999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"16'1","reported_seq":20,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-11T09:50:20.500435+0000","last_change":"2024-09-11T09:50:20.500687+0000","last_active":"2024-09-11T09:50:20.500435+0000","last_peered":"2024-09-11T09:50:20.500435+0000","last_clean":"2024-09-11T09:50:20.500435+0000","last_became_active":"2024-09-11T09:50:18.493650+0000","last_became_peered":"2024-09-11T09:50:18.493650+0000","last_unstale":"2024-09-11T09:50:20.500435+0000","last_undegraded":"2024-09-11T09:50:20.500435+0000","last_fullsized":"2024-09-11T09:50:20.500435+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_clean_scrub_stamp":"2024-09-11T09:50:17.477890+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T14:26:32.838143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041261100000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"13'117","reported_seq":170,"reported_epoch":18,"state":"active+clean","last_fresh":"2024-09-11T09:50:20.497413+0000","last_change":"2024-09-11T09:50:11.472613+0000","last_active":"2024-09-11T09:50:20.497413+0000","last_peered":"2024-09-11T09:50:20.497413+0000","last_clean":"2024-09-11T09:50:20.497413+0000","last_became_active":"2024-09-11T09:50:11.472306+0000","last_became_peered":"2024-09-11T09:50:11.472306+0000","last_unstale":"2024-09-11T09:50:20.497413+0000","last_undegraded":"2024-09-11T09:50:20.497413+0000","last_fullsized":"2024-09-11T09:50:20.497413+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:50:10.457262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:50:10.457262+0000","last_clean_scrub_stamp":"2024-09-11T09:50:10.457262+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T20:43:33.725287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":1180736,"data_compressed":64992,"data_compressed_allocated":90112,"data_compressed_original":1146880,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":3,"up_from":12,"seq":51539607558,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27264,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344576,"statfs":{"total":96636764160,"available":96608845824,"internally_reserved":0,"allocated":237568,"data_stored":647321,"data_compressed":32496,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":12,"seq":51539607557,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31424,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340416,"statfs":{"total":96636764160,"available":96604585984,"internally_reserved":0,"allocated":237568,"data_stored":647321,"data_compressed":32496,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":11,"seq":47244640262,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27208,"kb_used_data":172,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344632,"statfs":{"total":96636764160,"available":96608903168,"internally_reserved":0,"allocated":176128,"data_stored":56972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":11,"seq":47244640262,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27188,"kb_used_data":156,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344652,"statfs":{"total":96636764160,"available":96608923648,"internally_reserved":0,"allocated":159744,"data_stored":51985,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640261,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31348,"kb_used_data":156,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340492,"statfs":{"total":96636764160,"available":96604663808,"internally_reserved":0,"allocated":159744,"data_stored":51985,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640261,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31368,"kb_used_data":172,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340472,"statfs":{"total":96636764160,"available":96604643328,"internally_reserved":0,"allocated":176128,"data_stored":56972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32496,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32496,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T09:50:25.597 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-11T09:50:25.597 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump 2024-09-11T09:50:25.885 INFO:teuthology.orchestra.run.smithi017.stdout:epoch 19 2024-09-11T09:50:25.886 INFO:teuthology.orchestra.run.smithi017.stdout:fsid a2f1daec-d3b0-4bf5-82c8-72ecbb93d4e4 2024-09-11T09:50:25.886 INFO:teuthology.orchestra.run.smithi017.stdout:created 2024-09-11T09:50:00.165804+0000 2024-09-11T09:50:25.886 INFO:teuthology.orchestra.run.smithi017.stdout:modified 2024-09-11T09:50:24.947183+0000 2024-09-11T09:50:25.886 INFO:teuthology.orchestra.run.smithi017.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-09-11T09:50:25.886 INFO:teuthology.orchestra.run.smithi017.stdout:crush_version 5 2024-09-11T09:50:25.886 INFO:teuthology.orchestra.run.smithi017.stdout:full_ratio 0.95 2024-09-11T09:50:25.886 INFO:teuthology.orchestra.run.smithi017.stdout:backfillfull_ratio 0.9 2024-09-11T09:50:25.886 INFO:teuthology.orchestra.run.smithi017.stdout:nearfull_ratio 0.85 2024-09-11T09:50:25.886 INFO:teuthology.orchestra.run.smithi017.stdout:require_min_compat_client luminous 2024-09-11T09:50:25.886 INFO:teuthology.orchestra.run.smithi017.stdout:min_compat_client jewel 2024-09-11T09:50:25.886 INFO:teuthology.orchestra.run.smithi017.stdout:require_osd_release squid 2024-09-11T09:50:25.887 INFO:teuthology.orchestra.run.smithi017.stdout:stretch_mode_enabled false 2024-09-11T09:50:25.887 INFO:teuthology.orchestra.run.smithi017.stdout:pool 1 '.mgr' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 14 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 6.06 2024-09-11T09:50:25.887 INFO:teuthology.orchestra.run.smithi017.stdout:pool 2 'rbd' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 autoscale_mode off last_change 18 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd read_balance_score 2.25 2024-09-11T09:50:25.887 INFO:teuthology.orchestra.run.smithi017.stdout:max_osd 6 2024-09-11T09:50:25.887 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0 up in weight 1 up_from 12 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.17:6808/4154449929,v1:172.21.15.17:6809/4154449929] [v2:172.21.15.17:6810/4154449929,v1:172.21.15.17:6811/4154449929] exists,up 4829b7a1-d607-4a56-b6d2-55036cb8acf8 2024-09-11T09:50:25.887 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1 up in weight 1 up_from 11 up_thru 15 down_at 0 last_clean_interval [0,0) [v2:172.21.15.17:6800/4139511644,v1:172.21.15.17:6801/4139511644] [v2:172.21.15.17:6802/4139511644,v1:172.21.15.17:6803/4139511644] exists,up ab1d5df0-3c0e-44cd-88a0-96b07964148b 2024-09-11T09:50:25.887 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2 up in weight 1 up_from 11 up_thru 15 down_at 0 last_clean_interval [0,0) [v2:172.21.15.17:6816/1151282506,v1:172.21.15.17:6817/1151282506] [v2:172.21.15.17:6818/1151282506,v1:172.21.15.17:6819/1151282506] exists,up 4c8e50db-b49c-4d44-a913-47b5b30f8699 2024-09-11T09:50:25.887 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3 up in weight 1 up_from 12 up_thru 15 down_at 0 last_clean_interval [0,0) [v2:172.21.15.96:6816/332854807,v1:172.21.15.96:6817/332854807] [v2:172.21.15.96:6818/332854807,v1:172.21.15.96:6819/332854807] exists,up 37abc55b-a1eb-49d5-b1ff-b507ee2cb5a6 2024-09-11T09:50:25.887 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4 up in weight 1 up_from 11 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.96:6800/2785176937,v1:172.21.15.96:6801/2785176937] [v2:172.21.15.96:6802/2785176937,v1:172.21.15.96:6803/2785176937] exists,up 067d9586-4f5a-4d17-96c5-86a7b8bcd114 2024-09-11T09:50:25.887 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5 up in weight 1 up_from 11 up_thru 15 down_at 0 last_clean_interval [0,0) [v2:172.21.15.96:6808/509935370,v1:172.21.15.96:6809/509935370] [v2:172.21.15.96:6810/509935370,v1:172.21.15.96:6811/509935370] exists,up 8ff7a724-0a3a-4aa2-baf1-fd52f88d476c 2024-09-11T09:50:25.901 INFO:tasks.ceph.ceph_manager.ceph:['osd.0 up in weight 1 up_from 12 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.17:6808/4154449929,v1:172.21.15.17:6809/4154449929] [v2:172.21.15.17:6810/4154449929,v1:172.21.15.17:6811/4154449929] exists,up 4829b7a1-d607-4a56-b6d2-55036cb8acf8', 'osd.1 up in weight 1 up_from 11 up_thru 15 down_at 0 last_clean_interval [0,0) [v2:172.21.15.17:6800/4139511644,v1:172.21.15.17:6801/4139511644] [v2:172.21.15.17:6802/4139511644,v1:172.21.15.17:6803/4139511644] exists,up ab1d5df0-3c0e-44cd-88a0-96b07964148b', 'osd.2 up in weight 1 up_from 11 up_thru 15 down_at 0 last_clean_interval [0,0) [v2:172.21.15.17:6816/1151282506,v1:172.21.15.17:6817/1151282506] [v2:172.21.15.17:6818/1151282506,v1:172.21.15.17:6819/1151282506] exists,up 4c8e50db-b49c-4d44-a913-47b5b30f8699', 'osd.3 up in weight 1 up_from 12 up_thru 15 down_at 0 last_clean_interval [0,0) [v2:172.21.15.96:6816/332854807,v1:172.21.15.96:6817/332854807] [v2:172.21.15.96:6818/332854807,v1:172.21.15.96:6819/332854807] exists,up 37abc55b-a1eb-49d5-b1ff-b507ee2cb5a6', 'osd.4 up in weight 1 up_from 11 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.96:6800/2785176937,v1:172.21.15.96:6801/2785176937] [v2:172.21.15.96:6802/2785176937,v1:172.21.15.96:6803/2785176937] exists,up 067d9586-4f5a-4d17-96c5-86a7b8bcd114', 'osd.5 up in weight 1 up_from 11 up_thru 15 down_at 0 last_clean_interval [0,0) [v2:172.21.15.96:6808/509935370,v1:172.21.15.96:6809/509935370] [v2:172.21.15.96:6810/509935370,v1:172.21.15.96:6811/509935370] exists,up 8ff7a724-0a3a-4aa2-baf1-fd52f88d476c'] 2024-09-11T09:50:25.902 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mon.a.asok config show 2024-09-11T09:50:26.004 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:50:26.004 INFO:teuthology.orchestra.run.smithi017.stdout: "name": "mon.a", 2024-09-11T09:50:26.004 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster": "ceph", 2024-09-11T09:50:26.004 INFO:teuthology.orchestra.run.smithi017.stdout: "admin_socket": "/var/run/ceph/ceph-mon.a.asok", 2024-09-11T09:50:26.004 INFO:teuthology.orchestra.run.smithi017.stdout: "admin_socket_mode": "", 2024-09-11T09:50:26.004 INFO:teuthology.orchestra.run.smithi017.stdout: "auth_allow_insecure_global_id_reclaim": "true", 2024-09-11T09:50:26.004 INFO:teuthology.orchestra.run.smithi017.stdout: "auth_client_required": "cephx, none", 2024-09-11T09:50:26.004 INFO:teuthology.orchestra.run.smithi017.stdout: "auth_cluster_required": "cephx", 2024-09-11T09:50:26.004 INFO:teuthology.orchestra.run.smithi017.stdout: "auth_debug": "true", 2024-09-11T09:50:26.004 INFO:teuthology.orchestra.run.smithi017.stdout: "auth_expose_insecure_global_id_reclaim": "true", 2024-09-11T09:50:26.004 INFO:teuthology.orchestra.run.smithi017.stdout: "auth_mon_ticket_ttl": "660.000000", 2024-09-11T09:50:26.005 INFO:teuthology.orchestra.run.smithi017.stdout: "auth_service_required": "cephx", 2024-09-11T09:50:26.005 INFO:teuthology.orchestra.run.smithi017.stdout: "auth_service_ticket_ttl": "240.000000", 2024-09-11T09:50:26.005 INFO:teuthology.orchestra.run.smithi017.stdout: "auth_supported": "cephx", 2024-09-11T09:50:26.005 INFO:teuthology.orchestra.run.smithi017.stdout: "bdev_aio": "true", 2024-09-11T09:50:26.005 INFO:teuthology.orchestra.run.smithi017.stdout: "bdev_aio_max_queue_depth": "1024", 2024-09-11T09:50:26.005 INFO:teuthology.orchestra.run.smithi017.stdout: "bdev_aio_poll_ms": "250", 2024-09-11T09:50:26.005 INFO:teuthology.orchestra.run.smithi017.stdout: "bdev_aio_reap_max": "16", 2024-09-11T09:50:26.005 INFO:teuthology.orchestra.run.smithi017.stdout: "bdev_async_discard": "false", 2024-09-11T09:50:26.005 INFO:teuthology.orchestra.run.smithi017.stdout: "bdev_block_size": "4096", 2024-09-11T09:50:26.005 INFO:teuthology.orchestra.run.smithi017.stdout: "bdev_debug_aio": "false", 2024-09-11T09:50:26.005 INFO:teuthology.orchestra.run.smithi017.stdout: "bdev_debug_aio_log_age": "5.000000", 2024-09-11T09:50:26.005 INFO:teuthology.orchestra.run.smithi017.stdout: "bdev_debug_aio_suicide_timeout": "60.000000", 2024-09-11T09:50:26.005 INFO:teuthology.orchestra.run.smithi017.stdout: "bdev_debug_inflight_ios": "false", 2024-09-11T09:50:26.006 INFO:teuthology.orchestra.run.smithi017.stdout: "bdev_enable_discard": "false", 2024-09-11T09:50:26.006 INFO:teuthology.orchestra.run.smithi017.stdout: "bdev_flock_retry": "3", 2024-09-11T09:50:26.006 INFO:teuthology.orchestra.run.smithi017.stdout: "bdev_flock_retry_interval": "0.100000", 2024-09-11T09:50:26.006 INFO:teuthology.orchestra.run.smithi017.stdout: "bdev_inject_crash": "0", 2024-09-11T09:50:26.006 INFO:teuthology.orchestra.run.smithi017.stdout: "bdev_inject_crash_flush_delay": "2", 2024-09-11T09:50:26.006 INFO:teuthology.orchestra.run.smithi017.stdout: "bdev_ioring": "false", 2024-09-11T09:50:26.006 INFO:teuthology.orchestra.run.smithi017.stdout: "bdev_ioring_hipri": "false", 2024-09-11T09:50:26.006 INFO:teuthology.orchestra.run.smithi017.stdout: "bdev_ioring_sqthread_poll": "false", 2024-09-11T09:50:26.006 INFO:teuthology.orchestra.run.smithi017.stdout: "bdev_nvme_unbind_from_kernel": "false", 2024-09-11T09:50:26.006 INFO:teuthology.orchestra.run.smithi017.stdout: "bdev_read_buffer_alignment": "4096", 2024-09-11T09:50:26.006 INFO:teuthology.orchestra.run.smithi017.stdout: "bdev_read_preallocated_huge_buffers": "", 2024-09-11T09:50:26.006 INFO:teuthology.orchestra.run.smithi017.stdout: "bdev_stalled_read_warn_lifetime": "86400", 2024-09-11T09:50:26.006 INFO:teuthology.orchestra.run.smithi017.stdout: "bdev_stalled_read_warn_threshold": "1", 2024-09-11T09:50:26.006 INFO:teuthology.orchestra.run.smithi017.stdout: "bdev_type": "", 2024-09-11T09:50:26.006 INFO:teuthology.orchestra.run.smithi017.stdout: "bluefs_alloc_size": "1048576", 2024-09-11T09:50:26.007 INFO:teuthology.orchestra.run.smithi017.stdout: "bluefs_allocator": "hybrid", 2024-09-11T09:50:26.007 INFO:teuthology.orchestra.run.smithi017.stdout: "bluefs_buffered_io": "true", 2024-09-11T09:50:26.007 INFO:teuthology.orchestra.run.smithi017.stdout: "bluefs_check_for_zeros": "false", 2024-09-11T09:50:26.007 INFO:teuthology.orchestra.run.smithi017.stdout: "bluefs_check_volume_selector_often": "false", 2024-09-11T09:50:26.007 INFO:teuthology.orchestra.run.smithi017.stdout: "bluefs_check_volume_selector_on_umount": "false", 2024-09-11T09:50:26.007 INFO:teuthology.orchestra.run.smithi017.stdout: "bluefs_compact_log_sync": "false", 2024-09-11T09:50:26.007 INFO:teuthology.orchestra.run.smithi017.stdout: "bluefs_failed_shared_alloc_cooldown": "600.000000", 2024-09-11T09:50:26.007 INFO:teuthology.orchestra.run.smithi017.stdout: "bluefs_log_compact_min_ratio": "5.000000", 2024-09-11T09:50:26.007 INFO:teuthology.orchestra.run.smithi017.stdout: "bluefs_log_compact_min_size": "16777216", 2024-09-11T09:50:26.007 INFO:teuthology.orchestra.run.smithi017.stdout: "bluefs_log_replay_check_allocations": "true", 2024-09-11T09:50:26.007 INFO:teuthology.orchestra.run.smithi017.stdout: "bluefs_max_log_runway": "4194304", 2024-09-11T09:50:26.007 INFO:teuthology.orchestra.run.smithi017.stdout: "bluefs_max_prefetch": "1048576", 2024-09-11T09:50:26.007 INFO:teuthology.orchestra.run.smithi017.stdout: "bluefs_min_flush_size": "524288", 2024-09-11T09:50:26.007 INFO:teuthology.orchestra.run.smithi017.stdout: "bluefs_min_log_runway": "1048576", 2024-09-11T09:50:26.008 INFO:teuthology.orchestra.run.smithi017.stdout: "bluefs_replay_recovery": "false", 2024-09-11T09:50:26.008 INFO:teuthology.orchestra.run.smithi017.stdout: "bluefs_replay_recovery_disable_compact": "false", 2024-09-11T09:50:26.008 INFO:teuthology.orchestra.run.smithi017.stdout: "bluefs_shared_alloc_size": "65536", 2024-09-11T09:50:26.008 INFO:teuthology.orchestra.run.smithi017.stdout: "bluefs_sync_write": "false", 2024-09-11T09:50:26.008 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_2q_cache_kin_ratio": "0.500000", 2024-09-11T09:50:26.008 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_2q_cache_kout_ratio": "0.500000", 2024-09-11T09:50:26.008 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_alloc_stats_dump_interval": "86400.000000", 2024-09-11T09:50:26.008 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_allocation_from_file": "true", 2024-09-11T09:50:26.008 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_allocator": "hybrid", 2024-09-11T09:50:26.008 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_avl_alloc_bf_free_pct": "4", 2024-09-11T09:50:26.008 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_avl_alloc_bf_threshold": "131072", 2024-09-11T09:50:26.008 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_avl_alloc_ff_max_search_bytes": "16777216", 2024-09-11T09:50:26.008 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_avl_alloc_ff_max_search_count": "100", 2024-09-11T09:50:26.009 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_bdev_label_multi": "true", 2024-09-11T09:50:26.009 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_bdev_label_multi_upgrade": "false", 2024-09-11T09:50:26.009 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_bdev_label_require_all": "true", 2024-09-11T09:50:26.009 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_bitmapallocator_blocks_per_zone": "1024", 2024-09-11T09:50:26.009 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_bitmapallocator_span_size": "1024", 2024-09-11T09:50:26.009 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_blobid_prealloc": "10240", 2024-09-11T09:50:26.009 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_block_create": "true", 2024-09-11T09:50:26.009 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_block_db_create": "false", 2024-09-11T09:50:26.009 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_block_db_path": "", 2024-09-11T09:50:26.009 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_block_db_size": "0", 2024-09-11T09:50:26.009 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_block_path": "", 2024-09-11T09:50:26.009 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_block_preallocate_file": "false", 2024-09-11T09:50:26.009 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_block_size": "107374182400", 2024-09-11T09:50:26.010 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_block_wal_create": "false", 2024-09-11T09:50:26.010 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_block_wal_path": "", 2024-09-11T09:50:26.010 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_block_wal_size": "100663296", 2024-09-11T09:50:26.010 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_bluefs": "true", 2024-09-11T09:50:26.010 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_bluefs_alloc_failure_dump_interval": "0.000000", 2024-09-11T09:50:26.010 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_bluefs_env_mirror": "false", 2024-09-11T09:50:26.010 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_bluefs_max_free": "10737418240", 2024-09-11T09:50:26.010 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_cache_age_bin_interval": "1.000000", 2024-09-11T09:50:26.010 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_cache_age_bins_data": "1 2 6 24 120 720 0 0 0 0", 2024-09-11T09:50:26.010 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_cache_age_bins_kv": "1 2 6 24 120 720 0 0 0 0", 2024-09-11T09:50:26.010 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_cache_age_bins_kv_onode": "0 0 0 0 0 0 0 0 0 720", 2024-09-11T09:50:26.010 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_cache_age_bins_meta": "1 2 6 24 120 720 0 0 0 0", 2024-09-11T09:50:26.010 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_cache_autotune": "true", 2024-09-11T09:50:26.010 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_cache_autotune_interval": "5.000000", 2024-09-11T09:50:26.011 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_cache_kv_onode_ratio": "0.040000", 2024-09-11T09:50:26.011 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_cache_kv_ratio": "0.450000", 2024-09-11T09:50:26.011 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_cache_meta_ratio": "0.450000", 2024-09-11T09:50:26.011 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_cache_size": "0", 2024-09-11T09:50:26.011 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_cache_size_hdd": "1073741824", 2024-09-11T09:50:26.011 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_cache_size_ssd": "3221225472", 2024-09-11T09:50:26.011 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_cache_trim_interval": "0.050000", 2024-09-11T09:50:26.011 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_cache_trim_max_skip_pinned": "1000", 2024-09-11T09:50:26.011 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_cache_type": "2q", 2024-09-11T09:50:26.011 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_cleaner_sleep_interval": "5.000000", 2024-09-11T09:50:26.011 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_clone_cow": "true", 2024-09-11T09:50:26.011 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_compression_algorithm": "snappy", 2024-09-11T09:50:26.012 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_compression_max_blob_size": "0", 2024-09-11T09:50:26.012 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_compression_max_blob_size_hdd": "65536", 2024-09-11T09:50:26.012 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_compression_max_blob_size_ssd": "65536", 2024-09-11T09:50:26.012 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_compression_min_blob_size": "0", 2024-09-11T09:50:26.012 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_compression_min_blob_size_hdd": "8192", 2024-09-11T09:50:26.012 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_compression_min_blob_size_ssd": "65536", 2024-09-11T09:50:26.012 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_compression_mode": "none", 2024-09-11T09:50:26.012 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_compression_required_ratio": "0.875000", 2024-09-11T09:50:26.012 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_csum_type": "crc32c", 2024-09-11T09:50:26.012 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_debug_enforce_settings": "default", 2024-09-11T09:50:26.012 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_debug_freelist": "false", 2024-09-11T09:50:26.012 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_debug_fsck_abort": "false", 2024-09-11T09:50:26.013 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_debug_inject_allocation_from_file_failure": "0.000000", 2024-09-11T09:50:26.013 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_debug_inject_csum_err_probability": "0.000000", 2024-09-11T09:50:26.013 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_debug_inject_read_err": "false", 2024-09-11T09:50:26.013 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_debug_legacy_omap": "false", 2024-09-11T09:50:26.013 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_debug_no_reuse_blocks": "false", 2024-09-11T09:50:26.013 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_debug_omit_block_device_write": "false", 2024-09-11T09:50:26.013 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_debug_omit_kv_commit": "false", 2024-09-11T09:50:26.013 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_debug_permit_any_bdev_label": "false", 2024-09-11T09:50:26.013 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_debug_prefragment_max": "1048576", 2024-09-11T09:50:26.013 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_debug_random_read_err": "0.000000", 2024-09-11T09:50:26.013 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_debug_randomize_serial_transaction": "0", 2024-09-11T09:50:26.014 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_debug_small_allocations": "0", 2024-09-11T09:50:26.014 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_debug_too_many_blobs_threshold": "24576", 2024-09-11T09:50:26.014 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_default_buffered_read": "true", 2024-09-11T09:50:26.014 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_default_buffered_write": "false", 2024-09-11T09:50:26.014 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_deferred_batch_ops": "0", 2024-09-11T09:50:26.014 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_deferred_batch_ops_hdd": "64", 2024-09-11T09:50:26.014 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_deferred_batch_ops_ssd": "16", 2024-09-11T09:50:26.014 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_elastic_shared_blobs": "true", 2024-09-11T09:50:26.014 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_extent_map_inline_shard_prealloc_size": "256", 2024-09-11T09:50:26.014 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_extent_map_shard_max_size": "1200", 2024-09-11T09:50:26.014 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_extent_map_shard_min_size": "150", 2024-09-11T09:50:26.014 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_extent_map_shard_target_size": "500", 2024-09-11T09:50:26.015 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_extent_map_shard_target_size_slop": "0.200000", 2024-09-11T09:50:26.015 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_fail_eio": "false", 2024-09-11T09:50:26.015 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_freelist_blocks_per_key": "128", 2024-09-11T09:50:26.015 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_fsck_error_on_no_per_pg_omap": "false", 2024-09-11T09:50:26.015 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_fsck_error_on_no_per_pool_omap": "false", 2024-09-11T09:50:26.015 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_fsck_error_on_no_per_pool_stats": "false", 2024-09-11T09:50:26.015 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_fsck_on_mkfs": "true", 2024-09-11T09:50:26.015 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_fsck_on_mkfs_deep": "false", 2024-09-11T09:50:26.015 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_fsck_on_mount": "false", 2024-09-11T09:50:26.015 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_fsck_on_mount_deep": "false", 2024-09-11T09:50:26.015 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_fsck_on_umount": "false", 2024-09-11T09:50:26.015 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_fsck_on_umount_deep": "false", 2024-09-11T09:50:26.015 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_fsck_quick_fix_on_mount": "false", 2024-09-11T09:50:26.015 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_fsck_quick_fix_threads": "2", 2024-09-11T09:50:26.016 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_fsck_read_bytes_cap": "67108864", 2024-09-11T09:50:26.016 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_fsck_shared_blob_tracker_size": "0.031250", 2024-09-11T09:50:26.016 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_gc_enable_blob_threshold": "0", 2024-09-11T09:50:26.016 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_gc_enable_total_threshold": "0", 2024-09-11T09:50:26.016 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_hybrid_alloc_mem_cap": "67108864", 2024-09-11T09:50:26.016 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_ignore_data_csum": "false", 2024-09-11T09:50:26.016 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_kv_sync_util_logging_s": "10.000000", 2024-09-11T09:50:26.016 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_kvbackend": "rocksdb", 2024-09-11T09:50:26.016 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_log_collection_list_age": "60.000000", 2024-09-11T09:50:26.016 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_log_omap_iterator_age": "5.000000", 2024-09-11T09:50:26.016 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_log_op_age": "5.000000", 2024-09-11T09:50:26.016 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_max_alloc_size": "0", 2024-09-11T09:50:26.017 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_max_blob_size": "0", 2024-09-11T09:50:26.017 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_max_blob_size_hdd": "65536", 2024-09-11T09:50:26.017 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_max_blob_size_ssd": "65536", 2024-09-11T09:50:26.017 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_max_defer_interval": "3.000000", 2024-09-11T09:50:26.017 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_max_deferred_txc": "32", 2024-09-11T09:50:26.017 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_min_alloc_size": "0", 2024-09-11T09:50:26.017 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_min_alloc_size_hdd": "4096", 2024-09-11T09:50:26.017 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_min_alloc_size_ssd": "4096", 2024-09-11T09:50:26.017 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_nid_prealloc": "1024", 2024-09-11T09:50:26.017 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_prefer_deferred_size": "0", 2024-09-11T09:50:26.017 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_prefer_deferred_size_hdd": "65536", 2024-09-11T09:50:26.017 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_prefer_deferred_size_ssd": "0", 2024-09-11T09:50:26.017 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_qfsck_on_mount": "true", 2024-09-11T09:50:26.017 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_retry_disk_reads": "3", 2024-09-11T09:50:26.018 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_rocksdb_cf": "true", 2024-09-11T09:50:26.018 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_rocksdb_cfs": "m(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L=min_write_buffer_number_to_merge=32 P=min_write_buffer_number_to_merge=32", 2024-09-11T09:50:26.018 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_rocksdb_options": "compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0", 2024-09-11T09:50:26.018 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_rocksdb_options_annex": "", 2024-09-11T09:50:26.018 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_slow_ops_warn_lifetime": "86400", 2024-09-11T09:50:26.018 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_slow_ops_warn_threshold": "1", 2024-09-11T09:50:26.018 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_spdk_coremask": "0x1", 2024-09-11T09:50:26.018 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_spdk_io_sleep": "5", 2024-09-11T09:50:26.018 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_spdk_max_io_completion": "0", 2024-09-11T09:50:26.018 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_spdk_mem": "512", 2024-09-11T09:50:26.018 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_sync_submit_transaction": "false", 2024-09-11T09:50:26.018 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_throttle_bytes": "67108864", 2024-09-11T09:50:26.018 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_throttle_cost_per_io": "0", 2024-09-11T09:50:26.018 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_throttle_cost_per_io_hdd": "670000", 2024-09-11T09:50:26.019 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_throttle_cost_per_io_ssd": "4000", 2024-09-11T09:50:26.019 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_throttle_deferred_bytes": "134217728", 2024-09-11T09:50:26.019 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_throttle_trace_rate": "0.000000", 2024-09-11T09:50:26.019 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_tracing": "false", 2024-09-11T09:50:26.019 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_use_optimal_io_size_for_min_alloc_size": "false", 2024-09-11T09:50:26.019 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_volume_selection_policy": "use_some_extra", 2024-09-11T09:50:26.019 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_volume_selection_reserved": "0", 2024-09-11T09:50:26.019 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_volume_selection_reserved_factor": "2.000000", 2024-09-11T09:50:26.019 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_warn_on_bluefs_spillover": "true", 2024-09-11T09:50:26.019 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_warn_on_legacy_statfs": "true", 2024-09-11T09:50:26.019 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_warn_on_no_per_pg_omap": "false", 2024-09-11T09:50:26.019 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_warn_on_no_per_pool_omap": "true", 2024-09-11T09:50:26.019 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_warn_on_spurious_read_errors": "true", 2024-09-11T09:50:26.019 INFO:teuthology.orchestra.run.smithi017.stdout: "bluestore_zero_block_detection": "false", 2024-09-11T09:50:26.019 INFO:teuthology.orchestra.run.smithi017.stdout: "cephadm_path": "/usr/sbin/cephadm", 2024-09-11T09:50:26.020 INFO:teuthology.orchestra.run.smithi017.stdout: "cephfs_mirror_action_update_interval": "2", 2024-09-11T09:50:26.020 INFO:teuthology.orchestra.run.smithi017.stdout: "cephfs_mirror_directory_scan_interval": "10", 2024-09-11T09:50:26.020 INFO:teuthology.orchestra.run.smithi017.stdout: "cephfs_mirror_max_concurrent_directory_syncs": "3", 2024-09-11T09:50:26.020 INFO:teuthology.orchestra.run.smithi017.stdout: "cephfs_mirror_max_consecutive_failures_per_directory": "10", 2024-09-11T09:50:26.020 INFO:teuthology.orchestra.run.smithi017.stdout: "cephfs_mirror_max_snapshot_sync_per_cycle": "3", 2024-09-11T09:50:26.020 INFO:teuthology.orchestra.run.smithi017.stdout: "cephfs_mirror_mount_timeout": "10", 2024-09-11T09:50:26.020 INFO:teuthology.orchestra.run.smithi017.stdout: "cephfs_mirror_perf_stats_prio": "5", 2024-09-11T09:50:26.020 INFO:teuthology.orchestra.run.smithi017.stdout: "cephfs_mirror_restart_mirror_on_blocklist_interval": "30", 2024-09-11T09:50:26.020 INFO:teuthology.orchestra.run.smithi017.stdout: "cephfs_mirror_restart_mirror_on_failure_interval": "20", 2024-09-11T09:50:26.020 INFO:teuthology.orchestra.run.smithi017.stdout: "cephfs_mirror_retry_failed_directories_interval": "60", 2024-09-11T09:50:26.020 INFO:teuthology.orchestra.run.smithi017.stdout: "cephsqlite_blocklist_dead_locker": "true", 2024-09-11T09:50:26.020 INFO:teuthology.orchestra.run.smithi017.stdout: "cephsqlite_lock_renewal_interval": "2000", 2024-09-11T09:50:26.020 INFO:teuthology.orchestra.run.smithi017.stdout: "cephsqlite_lock_renewal_timeout": "30000", 2024-09-11T09:50:26.020 INFO:teuthology.orchestra.run.smithi017.stdout: "cephx_cluster_require_signatures": "false", 2024-09-11T09:50:26.021 INFO:teuthology.orchestra.run.smithi017.stdout: "cephx_cluster_require_version": "2", 2024-09-11T09:50:26.021 INFO:teuthology.orchestra.run.smithi017.stdout: "cephx_require_signatures": "false", 2024-09-11T09:50:26.021 INFO:teuthology.orchestra.run.smithi017.stdout: "cephx_require_version": "2", 2024-09-11T09:50:26.021 INFO:teuthology.orchestra.run.smithi017.stdout: "cephx_service_require_signatures": "false", 2024-09-11T09:50:26.021 INFO:teuthology.orchestra.run.smithi017.stdout: "cephx_service_require_version": "2", 2024-09-11T09:50:26.021 INFO:teuthology.orchestra.run.smithi017.stdout: "cephx_sign_messages": "true", 2024-09-11T09:50:26.021 INFO:teuthology.orchestra.run.smithi017.stdout: "chdir": "", 2024-09-11T09:50:26.021 INFO:teuthology.orchestra.run.smithi017.stdout: "client_acl_type": "", 2024-09-11T09:50:26.021 INFO:teuthology.orchestra.run.smithi017.stdout: "client_asio_thread_count": "2", 2024-09-11T09:50:26.021 INFO:teuthology.orchestra.run.smithi017.stdout: "client_cache_mid": "0.750000", 2024-09-11T09:50:26.021 INFO:teuthology.orchestra.run.smithi017.stdout: "client_cache_size": "16384", 2024-09-11T09:50:26.021 INFO:teuthology.orchestra.run.smithi017.stdout: "client_caps_release_delay": "5", 2024-09-11T09:50:26.021 INFO:teuthology.orchestra.run.smithi017.stdout: "client_check_pool_perm": "true", 2024-09-11T09:50:26.021 INFO:teuthology.orchestra.run.smithi017.stdout: "client_collect_and_send_global_metrics": "false", 2024-09-11T09:50:26.022 INFO:teuthology.orchestra.run.smithi017.stdout: "client_debug_force_sync_read": "false", 2024-09-11T09:50:26.022 INFO:teuthology.orchestra.run.smithi017.stdout: "client_debug_getattr_caps": "false", 2024-09-11T09:50:26.022 INFO:teuthology.orchestra.run.smithi017.stdout: "client_debug_inject_features": "", 2024-09-11T09:50:26.022 INFO:teuthology.orchestra.run.smithi017.stdout: "client_debug_inject_tick_delay": "0", 2024-09-11T09:50:26.022 INFO:teuthology.orchestra.run.smithi017.stdout: "client_die_on_failed_dentry_invalidate": "true", 2024-09-11T09:50:26.022 INFO:teuthology.orchestra.run.smithi017.stdout: "client_die_on_failed_remount": "false", 2024-09-11T09:50:26.022 INFO:teuthology.orchestra.run.smithi017.stdout: "client_dirsize_rbytes": "true", 2024-09-11T09:50:26.022 INFO:teuthology.orchestra.run.smithi017.stdout: "client_force_lazyio": "false", 2024-09-11T09:50:26.022 INFO:teuthology.orchestra.run.smithi017.stdout: "client_fs": "", 2024-09-11T09:50:26.022 INFO:teuthology.orchestra.run.smithi017.stdout: "client_inject_fixed_oldest_tid": "false", 2024-09-11T09:50:26.022 INFO:teuthology.orchestra.run.smithi017.stdout: "client_inject_release_failure": "false", 2024-09-11T09:50:26.022 INFO:teuthology.orchestra.run.smithi017.stdout: "client_max_inline_size": "4096", 2024-09-11T09:50:26.022 INFO:teuthology.orchestra.run.smithi017.stdout: "client_max_retries_on_remount_failure": "5", 2024-09-11T09:50:26.022 INFO:teuthology.orchestra.run.smithi017.stdout: "client_mds_namespace": "", 2024-09-11T09:50:26.023 INFO:teuthology.orchestra.run.smithi017.stdout: "client_metadata": "", 2024-09-11T09:50:26.023 INFO:teuthology.orchestra.run.smithi017.stdout: "client_mount_gid": "-1", 2024-09-11T09:50:26.023 INFO:teuthology.orchestra.run.smithi017.stdout: "client_mount_timeout": "300", 2024-09-11T09:50:26.023 INFO:teuthology.orchestra.run.smithi017.stdout: "client_mount_uid": "-1", 2024-09-11T09:50:26.023 INFO:teuthology.orchestra.run.smithi017.stdout: "client_mountpoint": "/", 2024-09-11T09:50:26.023 INFO:teuthology.orchestra.run.smithi017.stdout: "client_notify_timeout": "10", 2024-09-11T09:50:26.023 INFO:teuthology.orchestra.run.smithi017.stdout: "client_oc": "true", 2024-09-11T09:50:26.023 INFO:teuthology.orchestra.run.smithi017.stdout: "client_oc_max_dirty": "104857600", 2024-09-11T09:50:26.023 INFO:teuthology.orchestra.run.smithi017.stdout: "client_oc_max_dirty_age": "5.000000", 2024-09-11T09:50:26.023 INFO:teuthology.orchestra.run.smithi017.stdout: "client_oc_max_objects": "1000", 2024-09-11T09:50:26.023 INFO:teuthology.orchestra.run.smithi017.stdout: "client_oc_size": "209715200", 2024-09-11T09:50:26.023 INFO:teuthology.orchestra.run.smithi017.stdout: "client_oc_target_dirty": "8388608", 2024-09-11T09:50:26.023 INFO:teuthology.orchestra.run.smithi017.stdout: "client_permissions": "true", 2024-09-11T09:50:26.023 INFO:teuthology.orchestra.run.smithi017.stdout: "client_quota": "true", 2024-09-11T09:50:26.024 INFO:teuthology.orchestra.run.smithi017.stdout: "client_quota_df": "true", 2024-09-11T09:50:26.024 INFO:teuthology.orchestra.run.smithi017.stdout: "client_readahead_max_bytes": "0", 2024-09-11T09:50:26.024 INFO:teuthology.orchestra.run.smithi017.stdout: "client_readahead_max_periods": "4", 2024-09-11T09:50:26.024 INFO:teuthology.orchestra.run.smithi017.stdout: "client_readahead_min": "131072", 2024-09-11T09:50:26.024 INFO:teuthology.orchestra.run.smithi017.stdout: "client_reconnect_stale": "false", 2024-09-11T09:50:26.024 INFO:teuthology.orchestra.run.smithi017.stdout: "client_shutdown_timeout": "30", 2024-09-11T09:50:26.024 INFO:teuthology.orchestra.run.smithi017.stdout: "client_snapdir": ".snap", 2024-09-11T09:50:26.024 INFO:teuthology.orchestra.run.smithi017.stdout: "client_tick_interval": "1", 2024-09-11T09:50:26.024 INFO:teuthology.orchestra.run.smithi017.stdout: "client_trace": "", 2024-09-11T09:50:26.024 INFO:teuthology.orchestra.run.smithi017.stdout: "client_try_dentry_invalidate": "false", 2024-09-11T09:50:26.024 INFO:teuthology.orchestra.run.smithi017.stdout: "client_use_faked_inos": "false", 2024-09-11T09:50:26.024 INFO:teuthology.orchestra.run.smithi017.stdout: "client_use_random_mds": "false", 2024-09-11T09:50:26.024 INFO:teuthology.orchestra.run.smithi017.stdout: "clog_to_graylog": "false", 2024-09-11T09:50:26.024 INFO:teuthology.orchestra.run.smithi017.stdout: "clog_to_graylog_host": "127.0.0.1", 2024-09-11T09:50:26.024 INFO:teuthology.orchestra.run.smithi017.stdout: "clog_to_graylog_port": "12201", 2024-09-11T09:50:26.025 INFO:teuthology.orchestra.run.smithi017.stdout: "clog_to_monitors": "default=true", 2024-09-11T09:50:26.025 INFO:teuthology.orchestra.run.smithi017.stdout: "clog_to_syslog": "false", 2024-09-11T09:50:26.025 INFO:teuthology.orchestra.run.smithi017.stdout: "clog_to_syslog_facility": "default=daemon audit=local0", 2024-09-11T09:50:26.025 INFO:teuthology.orchestra.run.smithi017.stdout: "clog_to_syslog_level": "info", 2024-09-11T09:50:26.025 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addr": "-", 2024-09-11T09:50:26.025 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_network": "", 2024-09-11T09:50:26.025 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_network_interface": "", 2024-09-11T09:50:26.025 INFO:teuthology.orchestra.run.smithi017.stdout: "compressor_zlib_isal": "false", 2024-09-11T09:50:26.025 INFO:teuthology.orchestra.run.smithi017.stdout: "compressor_zlib_level": "5", 2024-09-11T09:50:26.025 INFO:teuthology.orchestra.run.smithi017.stdout: "compressor_zlib_winsize": "-15", 2024-09-11T09:50:26.025 INFO:teuthology.orchestra.run.smithi017.stdout: "compressor_zstd_level": "1", 2024-09-11T09:50:26.025 INFO:teuthology.orchestra.run.smithi017.stdout: "container_image": "docker.io/ceph/daemon-base:latest-master-devel", 2024-09-11T09:50:26.025 INFO:teuthology.orchestra.run.smithi017.stdout: "crash_dir": "/var/lib/ceph/crash", 2024-09-11T09:50:26.025 INFO:teuthology.orchestra.run.smithi017.stdout: "crimson_alien_op_num_threads": "6", 2024-09-11T09:50:26.025 INFO:teuthology.orchestra.run.smithi017.stdout: "crimson_alien_thread_cpu_cores": "", 2024-09-11T09:50:26.026 INFO:teuthology.orchestra.run.smithi017.stdout: "crimson_osd_obc_lru_size": "512", 2024-09-11T09:50:26.026 INFO:teuthology.orchestra.run.smithi017.stdout: "crimson_osd_scheduler_concurrency": "0", 2024-09-11T09:50:26.026 INFO:teuthology.orchestra.run.smithi017.stdout: "crimson_osd_stat_interval": "0", 2024-09-11T09:50:26.026 INFO:teuthology.orchestra.run.smithi017.stdout: "crimson_seastar_cpu_cores": "", 2024-09-11T09:50:26.026 INFO:teuthology.orchestra.run.smithi017.stdout: "crimson_seastar_num_threads": "0", 2024-09-11T09:50:26.026 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location": "", 2024-09-11T09:50:26.026 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location_hook": "", 2024-09-11T09:50:26.026 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_location_hook_timeout": "10", 2024-09-11T09:50:26.026 INFO:teuthology.orchestra.run.smithi017.stdout: "daemonize": "false", 2024-09-11T09:50:26.026 INFO:teuthology.orchestra.run.smithi017.stdout: "daos_pool": "tank", 2024-09-11T09:50:26.026 INFO:teuthology.orchestra.run.smithi017.stdout: "dbstore_config_uri": "file:/var/lib/ceph/radosgw/dbstore-config.db", 2024-09-11T09:50:26.026 INFO:teuthology.orchestra.run.smithi017.stdout: "dbstore_db_dir": "/var/lib/ceph/radosgw", 2024-09-11T09:50:26.026 INFO:teuthology.orchestra.run.smithi017.stdout: "dbstore_db_name_prefix": "dbstore", 2024-09-11T09:50:26.026 INFO:teuthology.orchestra.run.smithi017.stdout: "debug_alienstore": "0/5", 2024-09-11T09:50:26.027 INFO:teuthology.orchestra.run.smithi017.stdout: "debug_allow_any_pool_priority": "false", 2024-09-11T09:50:26.027 INFO:teuthology.orchestra.run.smithi017.stdout: "debug_asok": "1/5", 2024-09-11T09:50:26.027 INFO:teuthology.orchestra.run.smithi017.stdout: "debug_asok_assert_abort": "false", 2024-09-11T09:50:26.027 INFO:teuthology.orchestra.run.smithi017.stdout: "debug_asserts_on_shutdown": "true", 2024-09-11T09:50:26.027 INFO:teuthology.orchestra.run.smithi017.stdout: "debug_auth": "20/20", 2024-09-11T09:50:26.027 INFO:teuthology.orchestra.run.smithi017.stdout: "debug_bdev": "1/3", 2024-09-11T09:50:26.027 INFO:teuthology.orchestra.run.smithi017.stdout: "debug_bluefs": "1/5", 2024-09-11T09:50:26.027 INFO:teuthology.orchestra.run.smithi017.stdout: "debug_bluestore": "1/5", 2024-09-11T09:50:26.027 INFO:teuthology.orchestra.run.smithi017.stdout: "debug_buffer": "0/1", 2024-09-11T09:50:26.027 INFO:teuthology.orchestra.run.smithi017.stdout: "debug_ceph_exporter": "1/5", 2024-09-11T09:50:26.027 INFO:teuthology.orchestra.run.smithi017.stdout: "debug_cephfs_mirror": "0/5", 2024-09-11T09:50:26.027 INFO:teuthology.orchestra.run.smithi017.stdout: "debug_cephsqlite": "0/5", 2024-09-11T09:50:26.027 INFO:teuthology.orchestra.run.smithi017.stdout: "debug_client": "0/5", 2024-09-11T09:50:26.028 INFO:teuthology.orchestra.run.smithi017.stdout: "debug_compressor": "1/5", 2024-09-11T09:50:26.028 INFO:teuthology.orchestra.run.smithi017.stdout: "debug_context": "0/1", 2024-09-11T09:50:26.028 INFO:teuthology.orchestra.run.smithi017.stdout: "debug_crimson_interrupt": "0/5", 2024-09-11T09:50:26.028 INFO:teuthology.orchestra.run.smithi017.stdout: "debug_crush": "1/1", 2024-09-11T09:50:26.028 INFO:teuthology.orchestra.run.smithi017.stdout: "debug_crypto": "1/5", 2024-09-11T09:50:26.028 INFO:teuthology.orchestra.run.smithi017.stdout: "debug_cyanstore": "0/5", 2024-09-11T09:50:26.028 INFO:teuthology.orchestra.run.smithi017.stdout: "debug_deliberately_leak_memory": "false", 2024-09-11T09:50:26.028 INFO:teuthology.orchestra.run.smithi017.stdout: "debug_disable_randomized_ping": "false", 2024-09-11T09:50:26.028 INFO:teuthology.orchestra.run.smithi017.stdout: "debug_dpdk": "1/5", 2024-09-11T09:50:26.028 INFO:teuthology.orchestra.run.smithi017.stdout: "debug_eventtrace": "1/5", 2024-09-11T09:50:26.028 INFO:teuthology.orchestra.run.smithi017.stdout: "debug_filer": "0/1", 2024-09-11T09:50:26.028 INFO:teuthology.orchestra.run.smithi017.stdout: "debug_filestore": "1/3", 2024-09-11T09:50:26.028 INFO:teuthology.orchestra.run.smithi017.stdout: "debug_finisher": "1/1", 2024-09-11T09:50:26.028 INFO:teuthology.orchestra.run.smithi017.stdout: "debug_fuse": "1/5", 2024-09-11T09:50:26.029 INFO:teuthology.orchestra.run.smithi017.stdout: "debug_heartbeat_testing_span": "0", 2024-09-11T09:50:26.029 INFO:teuthology.orchestra.run.smithi017.stdout: "debug_heartbeatmap": "1/5", 2024-09-11T09:50:26.029 INFO:teuthology.orchestra.run.smithi017.stdout: "debug_immutable_obj_cache": "0/5", 2024-09-11T09:50:26.029 INFO:teuthology.orchestra.run.smithi017.stdout: "debug_javaclient": "1/5", 2024-09-11T09:50:26.029 INFO:teuthology.orchestra.run.smithi017.stdout: "debug_journal": "1/3", 2024-09-11T09:50:26.029 INFO:teuthology.orchestra.run.smithi017.stdout: "debug_journaler": "0/5", 2024-09-11T09:50:26.029 INFO:teuthology.orchestra.run.smithi017.stdout: "debug_kstore": "1/5", 2024-09-11T09:50:26.029 INFO:teuthology.orchestra.run.smithi017.stdout: "debug_lockdep": "0/1", 2024-09-11T09:50:26.029 INFO:teuthology.orchestra.run.smithi017.stdout: "debug_mclock": "1/5", 2024-09-11T09:50:26.029 INFO:teuthology.orchestra.run.smithi017.stdout: "debug_mds": "1/5", 2024-09-11T09:50:26.029 INFO:teuthology.orchestra.run.smithi017.stdout: "debug_mds_balancer": "1/5", 2024-09-11T09:50:26.029 INFO:teuthology.orchestra.run.smithi017.stdout: "debug_mds_locker": "1/5", 2024-09-11T09:50:26.029 INFO:teuthology.orchestra.run.smithi017.stdout: "debug_mds_log": "1/5", 2024-09-11T09:50:26.029 INFO:teuthology.orchestra.run.smithi017.stdout: "debug_mds_log_expire": "1/5", 2024-09-11T09:50:26.029 INFO:teuthology.orchestra.run.smithi017.stdout: "debug_mds_migrator": "1/5", 2024-09-11T09:50:26.030 INFO:teuthology.orchestra.run.smithi017.stdout: "debug_mds_quiesce": "3/5", 2024-09-11T09:50:26.030 INFO:teuthology.orchestra.run.smithi017.stdout: "debug_memstore": "1/5", 2024-09-11T09:50:26.030 INFO:teuthology.orchestra.run.smithi017.stdout: "debug_mgr": "2/5", 2024-09-11T09:50:26.030 INFO:teuthology.orchestra.run.smithi017.stdout: "debug_mgrc": "1/5", 2024-09-11T09:50:26.030 INFO:teuthology.orchestra.run.smithi017.stdout: "debug_mon": "20/20", 2024-09-11T09:50:26.030 INFO:teuthology.orchestra.run.smithi017.stdout: "debug_monc": "0/10", 2024-09-11T09:50:26.030 INFO:teuthology.orchestra.run.smithi017.stdout: "debug_ms": "1/1", 2024-09-11T09:50:26.030 INFO:teuthology.orchestra.run.smithi017.stdout: "debug_none": "0/5", 2024-09-11T09:50:26.030 INFO:teuthology.orchestra.run.smithi017.stdout: "debug_objclass": "0/5", 2024-09-11T09:50:26.030 INFO:teuthology.orchestra.run.smithi017.stdout: "debug_objectcacher": "0/5", 2024-09-11T09:50:26.030 INFO:teuthology.orchestra.run.smithi017.stdout: "debug_objecter": "0/1", 2024-09-11T09:50:26.030 INFO:teuthology.orchestra.run.smithi017.stdout: "debug_optracker": "0/5", 2024-09-11T09:50:26.030 INFO:teuthology.orchestra.run.smithi017.stdout: "debug_osd": "1/5", 2024-09-11T09:50:26.030 INFO:teuthology.orchestra.run.smithi017.stdout: "debug_paxos": "20/20", 2024-09-11T09:50:26.031 INFO:teuthology.orchestra.run.smithi017.stdout: "debug_perfcounter": "1/5", 2024-09-11T09:50:26.031 INFO:teuthology.orchestra.run.smithi017.stdout: "debug_prioritycache": "1/5", 2024-09-11T09:50:26.031 INFO:teuthology.orchestra.run.smithi017.stdout: "debug_rados": "0/5", 2024-09-11T09:50:26.031 INFO:teuthology.orchestra.run.smithi017.stdout: "debug_rbd": "0/5", 2024-09-11T09:50:26.031 INFO:teuthology.orchestra.run.smithi017.stdout: "debug_rbd_mirror": "0/5", 2024-09-11T09:50:26.031 INFO:teuthology.orchestra.run.smithi017.stdout: "debug_rbd_pwl": "0/5", 2024-09-11T09:50:26.031 INFO:teuthology.orchestra.run.smithi017.stdout: "debug_rbd_replay": "0/5", 2024-09-11T09:50:26.031 INFO:teuthology.orchestra.run.smithi017.stdout: "debug_refs": "0/0", 2024-09-11T09:50:26.031 INFO:teuthology.orchestra.run.smithi017.stdout: "debug_reserver": "1/1", 2024-09-11T09:50:26.031 INFO:teuthology.orchestra.run.smithi017.stdout: "debug_rgw": "1/5", 2024-09-11T09:50:26.031 INFO:teuthology.orchestra.run.smithi017.stdout: "debug_rgw_access": "1/5", 2024-09-11T09:50:26.031 INFO:teuthology.orchestra.run.smithi017.stdout: "debug_rgw_datacache": "1/5", 2024-09-11T09:50:26.031 INFO:teuthology.orchestra.run.smithi017.stdout: "debug_rgw_dbstore": "1/5", 2024-09-11T09:50:26.031 INFO:teuthology.orchestra.run.smithi017.stdout: "debug_rgw_flight": "1/5", 2024-09-11T09:50:26.031 INFO:teuthology.orchestra.run.smithi017.stdout: "debug_rgw_lifecycle": "1/5", 2024-09-11T09:50:26.031 INFO:teuthology.orchestra.run.smithi017.stdout: "debug_rgw_sync": "1/5", 2024-09-11T09:50:26.032 INFO:teuthology.orchestra.run.smithi017.stdout: "debug_rocksdb": "4/5", 2024-09-11T09:50:26.032 INFO:teuthology.orchestra.run.smithi017.stdout: "debug_seastore": "0/5", 2024-09-11T09:50:26.032 INFO:teuthology.orchestra.run.smithi017.stdout: "debug_seastore_backref": "0/5", 2024-09-11T09:50:26.032 INFO:teuthology.orchestra.run.smithi017.stdout: "debug_seastore_cache": "0/5", 2024-09-11T09:50:26.032 INFO:teuthology.orchestra.run.smithi017.stdout: "debug_seastore_cleaner": "0/5", 2024-09-11T09:50:26.032 INFO:teuthology.orchestra.run.smithi017.stdout: "debug_seastore_device": "0/5", 2024-09-11T09:50:26.032 INFO:teuthology.orchestra.run.smithi017.stdout: "debug_seastore_epm": "0/5", 2024-09-11T09:50:26.032 INFO:teuthology.orchestra.run.smithi017.stdout: "debug_seastore_fixedkv_tree": "0/5", 2024-09-11T09:50:26.032 INFO:teuthology.orchestra.run.smithi017.stdout: "debug_seastore_journal": "0/5", 2024-09-11T09:50:26.032 INFO:teuthology.orchestra.run.smithi017.stdout: "debug_seastore_lba": "0/5", 2024-09-11T09:50:26.032 INFO:teuthology.orchestra.run.smithi017.stdout: "debug_seastore_odata": "0/5", 2024-09-11T09:50:26.033 INFO:teuthology.orchestra.run.smithi017.stdout: "debug_seastore_omap": "0/5", 2024-09-11T09:50:26.033 INFO:teuthology.orchestra.run.smithi017.stdout: "debug_seastore_onode": "0/5", 2024-09-11T09:50:26.033 INFO:teuthology.orchestra.run.smithi017.stdout: "debug_seastore_t": "0/5", 2024-09-11T09:50:26.033 INFO:teuthology.orchestra.run.smithi017.stdout: "debug_seastore_tm": "0/5", 2024-09-11T09:50:26.033 INFO:teuthology.orchestra.run.smithi017.stdout: "debug_striper": "0/1", 2024-09-11T09:50:26.033 INFO:teuthology.orchestra.run.smithi017.stdout: "debug_test": "0/5", 2024-09-11T09:50:26.033 INFO:teuthology.orchestra.run.smithi017.stdout: "debug_throttle": "1/1", 2024-09-11T09:50:26.033 INFO:teuthology.orchestra.run.smithi017.stdout: "debug_timer": "0/1", 2024-09-11T09:50:26.033 INFO:teuthology.orchestra.run.smithi017.stdout: "debug_tp": "0/5", 2024-09-11T09:50:26.033 INFO:teuthology.orchestra.run.smithi017.stdout: "debug_trace": "1/5", 2024-09-11T09:50:26.033 INFO:teuthology.orchestra.run.smithi017.stdout: "defer_client_eviction_on_laggy_osds": "false", 2024-09-11T09:50:26.033 INFO:teuthology.orchestra.run.smithi017.stdout: "device_failure_prediction_mode": "none", 2024-09-11T09:50:26.033 INFO:teuthology.orchestra.run.smithi017.stdout: "enable_experimental_unrecoverable_data_corrupting_features": "", 2024-09-11T09:50:26.034 INFO:teuthology.orchestra.run.smithi017.stdout: "erasure_code_dir": "/usr/lib64/ceph/erasure-code", 2024-09-11T09:50:26.034 INFO:teuthology.orchestra.run.smithi017.stdout: "err_to_graylog": "false", 2024-09-11T09:50:26.034 INFO:teuthology.orchestra.run.smithi017.stdout: "err_to_journald": "false", 2024-09-11T09:50:26.034 INFO:teuthology.orchestra.run.smithi017.stdout: "err_to_stderr": "true", 2024-09-11T09:50:26.034 INFO:teuthology.orchestra.run.smithi017.stdout: "err_to_syslog": "false", 2024-09-11T09:50:26.034 INFO:teuthology.orchestra.run.smithi017.stdout: "event_tracing": "false", 2024-09-11T09:50:26.034 INFO:teuthology.orchestra.run.smithi017.stdout: "exporter_addr": "0.0.0.0", 2024-09-11T09:50:26.034 INFO:teuthology.orchestra.run.smithi017.stdout: "exporter_http_port": "9926", 2024-09-11T09:50:26.034 INFO:teuthology.orchestra.run.smithi017.stdout: "exporter_prio_limit": "5", 2024-09-11T09:50:26.034 INFO:teuthology.orchestra.run.smithi017.stdout: "exporter_sock_dir": "/var/run/ceph/", 2024-09-11T09:50:26.034 INFO:teuthology.orchestra.run.smithi017.stdout: "exporter_sort_metrics": "true", 2024-09-11T09:50:26.034 INFO:teuthology.orchestra.run.smithi017.stdout: "exporter_stats_period": "5", 2024-09-11T09:50:26.034 INFO:teuthology.orchestra.run.smithi017.stdout: "fake_statfs_for_testing": "0", 2024-09-11T09:50:26.034 INFO:teuthology.orchestra.run.smithi017.stdout: "fatal_signal_handlers": "true", 2024-09-11T09:50:26.034 INFO:teuthology.orchestra.run.smithi017.stdout: "filer_max_purge_ops": "10", 2024-09-11T09:50:26.035 INFO:teuthology.orchestra.run.smithi017.stdout: "filer_max_truncate_ops": "128", 2024-09-11T09:50:26.035 INFO:teuthology.orchestra.run.smithi017.stdout: "filestore_apply_finisher_threads": "1", 2024-09-11T09:50:26.035 INFO:teuthology.orchestra.run.smithi017.stdout: "filestore_blackhole": "false", 2024-09-11T09:50:26.035 INFO:teuthology.orchestra.run.smithi017.stdout: "filestore_btrfs_clone_range": "true", 2024-09-11T09:50:26.035 INFO:teuthology.orchestra.run.smithi017.stdout: "filestore_btrfs_snap": "true", 2024-09-11T09:50:26.035 INFO:teuthology.orchestra.run.smithi017.stdout: "filestore_caller_concurrency": "10", 2024-09-11T09:50:26.035 INFO:teuthology.orchestra.run.smithi017.stdout: "filestore_collect_device_partition_information": "true", 2024-09-11T09:50:26.035 INFO:teuthology.orchestra.run.smithi017.stdout: "filestore_commit_timeout": "600.000000", 2024-09-11T09:50:26.035 INFO:teuthology.orchestra.run.smithi017.stdout: "filestore_debug_inject_read_err": "false", 2024-09-11T09:50:26.035 INFO:teuthology.orchestra.run.smithi017.stdout: "filestore_debug_omap_check": "false", 2024-09-11T09:50:26.035 INFO:teuthology.orchestra.run.smithi017.stdout: "filestore_debug_random_read_err": "0.000000", 2024-09-11T09:50:26.035 INFO:teuthology.orchestra.run.smithi017.stdout: "filestore_debug_verify_split": "false", 2024-09-11T09:50:26.035 INFO:teuthology.orchestra.run.smithi017.stdout: "filestore_dump_file": "", 2024-09-11T09:50:26.035 INFO:teuthology.orchestra.run.smithi017.stdout: "filestore_expected_throughput_bytes": "209715200.000000", 2024-09-11T09:50:26.035 INFO:teuthology.orchestra.run.smithi017.stdout: "filestore_expected_throughput_ops": "200.000000", 2024-09-11T09:50:26.036 INFO:teuthology.orchestra.run.smithi017.stdout: "filestore_fadvise": "true", 2024-09-11T09:50:26.036 INFO:teuthology.orchestra.run.smithi017.stdout: "filestore_fail_eio": "true", 2024-09-11T09:50:26.036 INFO:teuthology.orchestra.run.smithi017.stdout: "filestore_fd_cache_shards": "16", 2024-09-11T09:50:26.036 INFO:teuthology.orchestra.run.smithi017.stdout: "filestore_fd_cache_size": "128", 2024-09-11T09:50:26.036 INFO:teuthology.orchestra.run.smithi017.stdout: "filestore_fiemap": "false", 2024-09-11T09:50:26.036 INFO:teuthology.orchestra.run.smithi017.stdout: "filestore_fiemap_threshold": "4096", 2024-09-11T09:50:26.036 INFO:teuthology.orchestra.run.smithi017.stdout: "filestore_fsync_flushes_journal_data": "false", 2024-09-11T09:50:26.036 INFO:teuthology.orchestra.run.smithi017.stdout: "filestore_index_retry_probability": "0.000000", 2024-09-11T09:50:26.036 INFO:teuthology.orchestra.run.smithi017.stdout: "filestore_inject_stall": "0", 2024-09-11T09:50:26.036 INFO:teuthology.orchestra.run.smithi017.stdout: "filestore_journal_parallel": "false", 2024-09-11T09:50:26.036 INFO:teuthology.orchestra.run.smithi017.stdout: "filestore_journal_trailing": "false", 2024-09-11T09:50:26.036 INFO:teuthology.orchestra.run.smithi017.stdout: "filestore_journal_writeahead": "false", 2024-09-11T09:50:26.036 INFO:teuthology.orchestra.run.smithi017.stdout: "filestore_kill_at": "0", 2024-09-11T09:50:26.036 INFO:teuthology.orchestra.run.smithi017.stdout: "filestore_max_alloc_hint_size": "1048576", 2024-09-11T09:50:26.036 INFO:teuthology.orchestra.run.smithi017.stdout: "filestore_max_inline_xattr_size": "0", 2024-09-11T09:50:26.037 INFO:teuthology.orchestra.run.smithi017.stdout: "filestore_max_inline_xattr_size_btrfs": "2048", 2024-09-11T09:50:26.037 INFO:teuthology.orchestra.run.smithi017.stdout: "filestore_max_inline_xattr_size_other": "512", 2024-09-11T09:50:26.037 INFO:teuthology.orchestra.run.smithi017.stdout: "filestore_max_inline_xattr_size_xfs": "65536", 2024-09-11T09:50:26.037 INFO:teuthology.orchestra.run.smithi017.stdout: "filestore_max_inline_xattrs": "0", 2024-09-11T09:50:26.037 INFO:teuthology.orchestra.run.smithi017.stdout: "filestore_max_inline_xattrs_btrfs": "10", 2024-09-11T09:50:26.037 INFO:teuthology.orchestra.run.smithi017.stdout: "filestore_max_inline_xattrs_other": "2", 2024-09-11T09:50:26.037 INFO:teuthology.orchestra.run.smithi017.stdout: "filestore_max_inline_xattrs_xfs": "10", 2024-09-11T09:50:26.037 INFO:teuthology.orchestra.run.smithi017.stdout: "filestore_max_sync_interval": "5.000000", 2024-09-11T09:50:26.037 INFO:teuthology.orchestra.run.smithi017.stdout: "filestore_max_xattr_value_size": "0", 2024-09-11T09:50:26.037 INFO:teuthology.orchestra.run.smithi017.stdout: "filestore_max_xattr_value_size_btrfs": "65536", 2024-09-11T09:50:26.037 INFO:teuthology.orchestra.run.smithi017.stdout: "filestore_max_xattr_value_size_other": "1024", 2024-09-11T09:50:26.037 INFO:teuthology.orchestra.run.smithi017.stdout: "filestore_max_xattr_value_size_xfs": "65536", 2024-09-11T09:50:26.037 INFO:teuthology.orchestra.run.smithi017.stdout: "filestore_merge_threshold": "-10", 2024-09-11T09:50:26.037 INFO:teuthology.orchestra.run.smithi017.stdout: "filestore_min_sync_interval": "0.010000", 2024-09-11T09:50:26.037 INFO:teuthology.orchestra.run.smithi017.stdout: "filestore_odsync_write": "false", 2024-09-11T09:50:26.038 INFO:teuthology.orchestra.run.smithi017.stdout: "filestore_omap_backend": "rocksdb", 2024-09-11T09:50:26.038 INFO:teuthology.orchestra.run.smithi017.stdout: "filestore_omap_backend_path": "", 2024-09-11T09:50:26.038 INFO:teuthology.orchestra.run.smithi017.stdout: "filestore_omap_header_cache_size": "1024", 2024-09-11T09:50:26.038 INFO:teuthology.orchestra.run.smithi017.stdout: "filestore_ondisk_finisher_threads": "1", 2024-09-11T09:50:26.038 INFO:teuthology.orchestra.run.smithi017.stdout: "filestore_op_thread_suicide_timeout": "180", 2024-09-11T09:50:26.038 INFO:teuthology.orchestra.run.smithi017.stdout: "filestore_op_thread_timeout": "60", 2024-09-11T09:50:26.038 INFO:teuthology.orchestra.run.smithi017.stdout: "filestore_op_threads": "2", 2024-09-11T09:50:26.038 INFO:teuthology.orchestra.run.smithi017.stdout: "filestore_punch_hole": "false", 2024-09-11T09:50:26.038 INFO:teuthology.orchestra.run.smithi017.stdout: "filestore_queue_high_delay_multiple": "0.000000", 2024-09-11T09:50:26.038 INFO:teuthology.orchestra.run.smithi017.stdout: "filestore_queue_high_delay_multiple_bytes": "0.000000", 2024-09-11T09:50:26.038 INFO:teuthology.orchestra.run.smithi017.stdout: "filestore_queue_high_delay_multiple_ops": "0.000000", 2024-09-11T09:50:26.038 INFO:teuthology.orchestra.run.smithi017.stdout: "filestore_queue_high_threshhold": "0.900000", 2024-09-11T09:50:26.038 INFO:teuthology.orchestra.run.smithi017.stdout: "filestore_queue_low_threshhold": "0.300000", 2024-09-11T09:50:26.038 INFO:teuthology.orchestra.run.smithi017.stdout: "filestore_queue_max_bytes": "104857600", 2024-09-11T09:50:26.039 INFO:teuthology.orchestra.run.smithi017.stdout: "filestore_queue_max_delay_multiple": "0.000000", 2024-09-11T09:50:26.039 INFO:teuthology.orchestra.run.smithi017.stdout: "filestore_queue_max_delay_multiple_bytes": "0.000000", 2024-09-11T09:50:26.039 INFO:teuthology.orchestra.run.smithi017.stdout: "filestore_queue_max_delay_multiple_ops": "0.000000", 2024-09-11T09:50:26.039 INFO:teuthology.orchestra.run.smithi017.stdout: "filestore_queue_max_ops": "50", 2024-09-11T09:50:26.039 INFO:teuthology.orchestra.run.smithi017.stdout: "filestore_rocksdb_options": "max_background_jobs=10,compaction_readahead_size=2097152,compression=kNoCompression", 2024-09-11T09:50:26.039 INFO:teuthology.orchestra.run.smithi017.stdout: "filestore_seek_data_hole": "false", 2024-09-11T09:50:26.039 INFO:teuthology.orchestra.run.smithi017.stdout: "filestore_sloppy_crc": "false", 2024-09-11T09:50:26.039 INFO:teuthology.orchestra.run.smithi017.stdout: "filestore_sloppy_crc_block_size": "65536", 2024-09-11T09:50:26.039 INFO:teuthology.orchestra.run.smithi017.stdout: "filestore_splice": "false", 2024-09-11T09:50:26.039 INFO:teuthology.orchestra.run.smithi017.stdout: "filestore_split_multiple": "2", 2024-09-11T09:50:26.039 INFO:teuthology.orchestra.run.smithi017.stdout: "filestore_split_rand_factor": "20", 2024-09-11T09:50:26.039 INFO:teuthology.orchestra.run.smithi017.stdout: "filestore_update_to": "1000", 2024-09-11T09:50:26.039 INFO:teuthology.orchestra.run.smithi017.stdout: "filestore_wbthrottle_btrfs_bytes_hard_limit": "419430400", 2024-09-11T09:50:26.039 INFO:teuthology.orchestra.run.smithi017.stdout: "filestore_wbthrottle_btrfs_bytes_start_flusher": "41943040", 2024-09-11T09:50:26.039 INFO:teuthology.orchestra.run.smithi017.stdout: "filestore_wbthrottle_btrfs_inodes_hard_limit": "5000", 2024-09-11T09:50:26.040 INFO:teuthology.orchestra.run.smithi017.stdout: "filestore_wbthrottle_btrfs_inodes_start_flusher": "500", 2024-09-11T09:50:26.040 INFO:teuthology.orchestra.run.smithi017.stdout: "filestore_wbthrottle_btrfs_ios_hard_limit": "5000", 2024-09-11T09:50:26.040 INFO:teuthology.orchestra.run.smithi017.stdout: "filestore_wbthrottle_btrfs_ios_start_flusher": "500", 2024-09-11T09:50:26.040 INFO:teuthology.orchestra.run.smithi017.stdout: "filestore_wbthrottle_enable": "true", 2024-09-11T09:50:26.040 INFO:teuthology.orchestra.run.smithi017.stdout: "filestore_wbthrottle_xfs_bytes_hard_limit": "419430400", 2024-09-11T09:50:26.040 INFO:teuthology.orchestra.run.smithi017.stdout: "filestore_wbthrottle_xfs_bytes_start_flusher": "41943040", 2024-09-11T09:50:26.040 INFO:teuthology.orchestra.run.smithi017.stdout: "filestore_wbthrottle_xfs_inodes_hard_limit": "5000", 2024-09-11T09:50:26.040 INFO:teuthology.orchestra.run.smithi017.stdout: "filestore_wbthrottle_xfs_inodes_start_flusher": "500", 2024-09-11T09:50:26.040 INFO:teuthology.orchestra.run.smithi017.stdout: "filestore_wbthrottle_xfs_ios_hard_limit": "5000", 2024-09-11T09:50:26.040 INFO:teuthology.orchestra.run.smithi017.stdout: "filestore_wbthrottle_xfs_ios_start_flusher": "500", 2024-09-11T09:50:26.040 INFO:teuthology.orchestra.run.smithi017.stdout: "filestore_xfs_extsize": "false", 2024-09-11T09:50:26.040 INFO:teuthology.orchestra.run.smithi017.stdout: "filestore_zfs_snap": "false", 2024-09-11T09:50:26.040 INFO:teuthology.orchestra.run.smithi017.stdout: "fio_dir": "/tmp/fio", 2024-09-11T09:50:26.040 INFO:teuthology.orchestra.run.smithi017.stdout: "fsid": "a2f1daec-d3b0-4bf5-82c8-72ecbb93d4e4", 2024-09-11T09:50:26.041 INFO:teuthology.orchestra.run.smithi017.stdout: "fuse_allow_other": "true", 2024-09-11T09:50:26.041 INFO:teuthology.orchestra.run.smithi017.stdout: "fuse_atomic_o_trunc": "true", 2024-09-11T09:50:26.041 INFO:teuthology.orchestra.run.smithi017.stdout: "fuse_big_writes": "true", 2024-09-11T09:50:26.041 INFO:teuthology.orchestra.run.smithi017.stdout: "fuse_debug": "false", 2024-09-11T09:50:26.041 INFO:teuthology.orchestra.run.smithi017.stdout: "fuse_default_permissions": "false", 2024-09-11T09:50:26.041 INFO:teuthology.orchestra.run.smithi017.stdout: "fuse_disable_pagecache": "false", 2024-09-11T09:50:26.041 INFO:teuthology.orchestra.run.smithi017.stdout: "fuse_max_write": "0", 2024-09-11T09:50:26.041 INFO:teuthology.orchestra.run.smithi017.stdout: "fuse_multithreaded": "true", 2024-09-11T09:50:26.041 INFO:teuthology.orchestra.run.smithi017.stdout: "fuse_require_active_mds": "true", 2024-09-11T09:50:26.041 INFO:teuthology.orchestra.run.smithi017.stdout: "fuse_set_user_groups": "true", 2024-09-11T09:50:26.041 INFO:teuthology.orchestra.run.smithi017.stdout: "fuse_splice_move": "true", 2024-09-11T09:50:26.041 INFO:teuthology.orchestra.run.smithi017.stdout: "fuse_splice_read": "true", 2024-09-11T09:50:26.041 INFO:teuthology.orchestra.run.smithi017.stdout: "fuse_splice_write": "true", 2024-09-11T09:50:26.041 INFO:teuthology.orchestra.run.smithi017.stdout: "fuse_syncfs_on_mksnap": "true", 2024-09-11T09:50:26.041 INFO:teuthology.orchestra.run.smithi017.stdout: "fuse_use_invalidate_cb": "true", 2024-09-11T09:50:26.042 INFO:teuthology.orchestra.run.smithi017.stdout: "gss_ktab_client_file": "/var/lib/ceph/mon.a/gss_client_mon.a.ktab", 2024-09-11T09:50:26.042 INFO:teuthology.orchestra.run.smithi017.stdout: "gss_target_name": "ceph", 2024-09-11T09:50:26.042 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_file": "", 2024-09-11T09:50:26.042 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_inject_failure": "0", 2024-09-11T09:50:26.042 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_interval": "5", 2024-09-11T09:50:26.042 INFO:teuthology.orchestra.run.smithi017.stdout: "host": "smithi017", 2024-09-11T09:50:26.042 INFO:teuthology.orchestra.run.smithi017.stdout: "immutable_object_cache_client_dedicated_thread_num": "2", 2024-09-11T09:50:26.042 INFO:teuthology.orchestra.run.smithi017.stdout: "immutable_object_cache_max_inflight_ops": "128", 2024-09-11T09:50:26.042 INFO:teuthology.orchestra.run.smithi017.stdout: "immutable_object_cache_max_size": "1073741824", 2024-09-11T09:50:26.042 INFO:teuthology.orchestra.run.smithi017.stdout: "immutable_object_cache_path": "/tmp/ceph_immutable_object_cache", 2024-09-11T09:50:26.042 INFO:teuthology.orchestra.run.smithi017.stdout: "immutable_object_cache_qos_bps_burst": "0", 2024-09-11T09:50:26.042 INFO:teuthology.orchestra.run.smithi017.stdout: "immutable_object_cache_qos_bps_burst_seconds": "1", 2024-09-11T09:50:26.042 INFO:teuthology.orchestra.run.smithi017.stdout: "immutable_object_cache_qos_bps_limit": "0", 2024-09-11T09:50:26.042 INFO:teuthology.orchestra.run.smithi017.stdout: "immutable_object_cache_qos_iops_burst": "0", 2024-09-11T09:50:26.043 INFO:teuthology.orchestra.run.smithi017.stdout: "immutable_object_cache_qos_iops_burst_seconds": "1", 2024-09-11T09:50:26.043 INFO:teuthology.orchestra.run.smithi017.stdout: "immutable_object_cache_qos_iops_limit": "0", 2024-09-11T09:50:26.043 INFO:teuthology.orchestra.run.smithi017.stdout: "immutable_object_cache_qos_schedule_tick_min": "50", 2024-09-11T09:50:26.043 INFO:teuthology.orchestra.run.smithi017.stdout: "immutable_object_cache_sock": "/var/run/ceph/immutable_object_cache_sock", 2024-09-11T09:50:26.043 INFO:teuthology.orchestra.run.smithi017.stdout: "immutable_object_cache_watermark": "0.900000", 2024-09-11T09:50:26.043 INFO:teuthology.orchestra.run.smithi017.stdout: "inject_early_sigterm": "false", 2024-09-11T09:50:26.043 INFO:teuthology.orchestra.run.smithi017.stdout: "jaeger_agent_port": "6799", 2024-09-11T09:50:26.043 INFO:teuthology.orchestra.run.smithi017.stdout: "jaeger_tracing_enable": "false", 2024-09-11T09:50:26.043 INFO:teuthology.orchestra.run.smithi017.stdout: "journal_aio": "true", 2024-09-11T09:50:26.043 INFO:teuthology.orchestra.run.smithi017.stdout: "journal_align_min_size": "65536", 2024-09-11T09:50:26.043 INFO:teuthology.orchestra.run.smithi017.stdout: "journal_block_align": "true", 2024-09-11T09:50:26.043 INFO:teuthology.orchestra.run.smithi017.stdout: "journal_block_size": "4096", 2024-09-11T09:50:26.043 INFO:teuthology.orchestra.run.smithi017.stdout: "journal_dio": "true", 2024-09-11T09:50:26.043 INFO:teuthology.orchestra.run.smithi017.stdout: "journal_discard": "false", 2024-09-11T09:50:26.043 INFO:teuthology.orchestra.run.smithi017.stdout: "journal_force_aio": "false", 2024-09-11T09:50:26.044 INFO:teuthology.orchestra.run.smithi017.stdout: "journal_ignore_corruption": "false", 2024-09-11T09:50:26.044 INFO:teuthology.orchestra.run.smithi017.stdout: "journal_max_write_bytes": "10485760", 2024-09-11T09:50:26.044 INFO:teuthology.orchestra.run.smithi017.stdout: "journal_max_write_entries": "100", 2024-09-11T09:50:26.044 INFO:teuthology.orchestra.run.smithi017.stdout: "journal_replay_from": "0", 2024-09-11T09:50:26.044 INFO:teuthology.orchestra.run.smithi017.stdout: "journal_throttle_high_multiple": "0.000000", 2024-09-11T09:50:26.044 INFO:teuthology.orchestra.run.smithi017.stdout: "journal_throttle_high_threshhold": "0.900000", 2024-09-11T09:50:26.044 INFO:teuthology.orchestra.run.smithi017.stdout: "journal_throttle_low_threshhold": "0.600000", 2024-09-11T09:50:26.044 INFO:teuthology.orchestra.run.smithi017.stdout: "journal_throttle_max_multiple": "0.000000", 2024-09-11T09:50:26.044 INFO:teuthology.orchestra.run.smithi017.stdout: "journal_write_header_frequency": "0", 2024-09-11T09:50:26.044 INFO:teuthology.orchestra.run.smithi017.stdout: "journal_zero_on_create": "false", 2024-09-11T09:50:26.044 INFO:teuthology.orchestra.run.smithi017.stdout: "journaler_prefetch_periods": "10", 2024-09-11T09:50:26.044 INFO:teuthology.orchestra.run.smithi017.stdout: "journaler_prezero_periods": "5", 2024-09-11T09:50:26.044 INFO:teuthology.orchestra.run.smithi017.stdout: "journaler_write_head_interval": "15", 2024-09-11T09:50:26.044 INFO:teuthology.orchestra.run.smithi017.stdout: "key": "", 2024-09-11T09:50:26.044 INFO:teuthology.orchestra.run.smithi017.stdout: "keyfile": "", 2024-09-11T09:50:26.045 INFO:teuthology.orchestra.run.smithi017.stdout: "keyring": "/var/lib/ceph/mon/ceph-a/keyring", 2024-09-11T09:50:26.045 INFO:teuthology.orchestra.run.smithi017.stdout: "kstore_backend": "rocksdb", 2024-09-11T09:50:26.045 INFO:teuthology.orchestra.run.smithi017.stdout: "kstore_default_stripe_size": "65536", 2024-09-11T09:50:26.045 INFO:teuthology.orchestra.run.smithi017.stdout: "kstore_fsck_on_mount": "false", 2024-09-11T09:50:26.045 INFO:teuthology.orchestra.run.smithi017.stdout: "kstore_fsck_on_mount_deep": "true", 2024-09-11T09:50:26.045 INFO:teuthology.orchestra.run.smithi017.stdout: "kstore_max_bytes": "67108864", 2024-09-11T09:50:26.045 INFO:teuthology.orchestra.run.smithi017.stdout: "kstore_max_ops": "512", 2024-09-11T09:50:26.045 INFO:teuthology.orchestra.run.smithi017.stdout: "kstore_nid_prealloc": "1024", 2024-09-11T09:50:26.045 INFO:teuthology.orchestra.run.smithi017.stdout: "kstore_onode_map_size": "1024", 2024-09-11T09:50:26.045 INFO:teuthology.orchestra.run.smithi017.stdout: "kstore_rocksdb_options": "compression=kNoCompression", 2024-09-11T09:50:26.045 INFO:teuthology.orchestra.run.smithi017.stdout: "kstore_sync_submit_transaction": "false", 2024-09-11T09:50:26.045 INFO:teuthology.orchestra.run.smithi017.stdout: "kstore_sync_transaction": "false", 2024-09-11T09:50:26.045 INFO:teuthology.orchestra.run.smithi017.stdout: "librados_thread_count": "2", 2024-09-11T09:50:26.045 INFO:teuthology.orchestra.run.smithi017.stdout: "lockdep": "false", 2024-09-11T09:50:26.045 INFO:teuthology.orchestra.run.smithi017.stdout: "lockdep_force_backtrace": "false", 2024-09-11T09:50:26.046 INFO:teuthology.orchestra.run.smithi017.stdout: "log_coarse_timestamps": "true", 2024-09-11T09:50:26.046 INFO:teuthology.orchestra.run.smithi017.stdout: "log_file": "/var/log/ceph/ceph-mon.a.log", 2024-09-11T09:50:26.046 INFO:teuthology.orchestra.run.smithi017.stdout: "log_flush_on_exit": "false", 2024-09-11T09:50:26.046 INFO:teuthology.orchestra.run.smithi017.stdout: "log_graylog_host": "127.0.0.1", 2024-09-11T09:50:26.046 INFO:teuthology.orchestra.run.smithi017.stdout: "log_graylog_port": "12201", 2024-09-11T09:50:26.046 INFO:teuthology.orchestra.run.smithi017.stdout: "log_max_new": "1000", 2024-09-11T09:50:26.046 INFO:teuthology.orchestra.run.smithi017.stdout: "log_max_recent": "10000", 2024-09-11T09:50:26.046 INFO:teuthology.orchestra.run.smithi017.stdout: "log_stderr_prefix": "", 2024-09-11T09:50:26.046 INFO:teuthology.orchestra.run.smithi017.stdout: "log_stop_at_utilization": "0.970000", 2024-09-11T09:50:26.046 INFO:teuthology.orchestra.run.smithi017.stdout: "log_to_file": "true", 2024-09-11T09:50:26.046 INFO:teuthology.orchestra.run.smithi017.stdout: "log_to_graylog": "false", 2024-09-11T09:50:26.046 INFO:teuthology.orchestra.run.smithi017.stdout: "log_to_journald": "false", 2024-09-11T09:50:26.046 INFO:teuthology.orchestra.run.smithi017.stdout: "log_to_stderr": "false", 2024-09-11T09:50:26.046 INFO:teuthology.orchestra.run.smithi017.stdout: "log_to_syslog": "false", 2024-09-11T09:50:26.046 INFO:teuthology.orchestra.run.smithi017.stdout: "max_rotating_auth_attempts": "10", 2024-09-11T09:50:26.047 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_abort_on_newly_corrupt_dentry": "true", 2024-09-11T09:50:26.047 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_action_on_write_error": "1", 2024-09-11T09:50:26.047 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_alternate_name_max": "8192", 2024-09-11T09:50:26.047 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_asio_thread_count": "2", 2024-09-11T09:50:26.047 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_bal_export_pin": "true", 2024-09-11T09:50:26.047 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_bal_fragment_dirs": "true", 2024-09-11T09:50:26.047 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_bal_fragment_fast_factor": "1.500000", 2024-09-11T09:50:26.047 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_bal_fragment_interval": "5", 2024-09-11T09:50:26.047 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_bal_fragment_size_max": "100000", 2024-09-11T09:50:26.047 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_bal_idle_threshold": "0.000000", 2024-09-11T09:50:26.047 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_bal_interval": "10", 2024-09-11T09:50:26.047 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_bal_max": "-1", 2024-09-11T09:50:26.047 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_bal_max_until": "-1", 2024-09-11T09:50:26.047 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_bal_merge_size": "50", 2024-09-11T09:50:26.047 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_bal_midchunk": "0.300000", 2024-09-11T09:50:26.048 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_bal_min_rebalance": "0.100000", 2024-09-11T09:50:26.048 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_bal_min_start": "0.200000", 2024-09-11T09:50:26.048 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_bal_minchunk": "0.001000", 2024-09-11T09:50:26.048 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_bal_mode": "0", 2024-09-11T09:50:26.048 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_bal_need_max": "1.200000", 2024-09-11T09:50:26.048 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_bal_need_min": "0.800000", 2024-09-11T09:50:26.048 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_bal_overload_epochs": "2", 2024-09-11T09:50:26.048 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_bal_replicate_threshold": "8000.000000", 2024-09-11T09:50:26.048 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_bal_sample_interval": "3.000000", 2024-09-11T09:50:26.048 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_bal_split_bits": "3", 2024-09-11T09:50:26.048 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_bal_split_rd": "25000.000000", 2024-09-11T09:50:26.048 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_bal_split_size": "10000", 2024-09-11T09:50:26.048 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_bal_split_wr": "10000.000000", 2024-09-11T09:50:26.048 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_bal_target_decay": "10.000000", 2024-09-11T09:50:26.048 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_bal_unreplicate_threshold": "0.000000", 2024-09-11T09:50:26.049 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_beacon_grace": "15.000000", 2024-09-11T09:50:26.049 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_beacon_interval": "4.000000", 2024-09-11T09:50:26.049 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_beacon_mon_down_grace": "60", 2024-09-11T09:50:26.049 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_cache_memory_limit": "4294967296", 2024-09-11T09:50:26.049 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_cache_mid": "0.700000", 2024-09-11T09:50:26.049 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_cache_quiesce_decay_rate": "1.000000", 2024-09-11T09:50:26.049 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_cache_quiesce_delay": "0", 2024-09-11T09:50:26.049 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_cache_quiesce_sleep": "200", 2024-09-11T09:50:26.049 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_cache_quiesce_splitauth": "true", 2024-09-11T09:50:26.049 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_cache_quiesce_threshold": "524288", 2024-09-11T09:50:26.049 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_cache_release_free_interval": "10", 2024-09-11T09:50:26.049 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_cache_reservation": "0.050000", 2024-09-11T09:50:26.049 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_cache_trim_decay_rate": "1.000000", 2024-09-11T09:50:26.049 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_cache_trim_interval": "1", 2024-09-11T09:50:26.049 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_cache_trim_threshold": "262144", 2024-09-11T09:50:26.050 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_cap_acquisition_throttle_retry_request_timeout": "0.500000", 2024-09-11T09:50:26.050 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_cap_revoke_eviction_timeout": "0.000000", 2024-09-11T09:50:26.050 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_client_delegate_inos_pct": "50", 2024-09-11T09:50:26.050 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_client_prealloc_inos": "1000", 2024-09-11T09:50:26.050 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_client_writeable_range_max_inc_objs": "1024", 2024-09-11T09:50:26.050 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_connect_bootstrapping": "false", 2024-09-11T09:50:26.050 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_damage_table_max_entries": "10000", 2024-09-11T09:50:26.050 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_data": "/var/lib/ceph/mds/ceph-a", 2024-09-11T09:50:26.050 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_debug_auth_pins": "false", 2024-09-11T09:50:26.050 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_debug_frag": "false", 2024-09-11T09:50:26.050 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_debug_scatterstat": "false", 2024-09-11T09:50:26.050 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_debug_subtrees": "false", 2024-09-11T09:50:26.050 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_decay_halflife": "5.000000", 2024-09-11T09:50:26.050 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_default_dir_hash": "2", 2024-09-11T09:50:26.050 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_defer_session_stale": "true", 2024-09-11T09:50:26.051 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_deny_all_reconnect": "false", 2024-09-11T09:50:26.051 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_dir_keys_per_op": "16384", 2024-09-11T09:50:26.051 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_dir_max_commit_size": "10", 2024-09-11T09:50:26.051 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_dir_max_entries": "0", 2024-09-11T09:50:26.051 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_dir_prefetch": "true", 2024-09-11T09:50:26.051 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_dirstat_min_interval": "1.000000", 2024-09-11T09:50:26.051 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_dump_cache_after_rejoin": "false", 2024-09-11T09:50:26.051 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_dump_cache_on_map": "false", 2024-09-11T09:50:26.051 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_dump_cache_threshold_file": "0", 2024-09-11T09:50:26.051 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_dump_cache_threshold_formatter": "1073741824", 2024-09-11T09:50:26.051 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_early_reply": "true", 2024-09-11T09:50:26.051 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_enable_op_tracker": "true", 2024-09-11T09:50:26.051 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_enforce_unique_name": "true", 2024-09-11T09:50:26.051 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_export_ephemeral_distributed": "true", 2024-09-11T09:50:26.051 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_export_ephemeral_distributed_factor": "2.000000", 2024-09-11T09:50:26.052 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_export_ephemeral_random": "true", 2024-09-11T09:50:26.052 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_export_ephemeral_random_max": "0.010000", 2024-09-11T09:50:26.052 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_extraordinary_events_dump_interval": "0", 2024-09-11T09:50:26.052 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_forward_all_requests_to_auth": "false", 2024-09-11T09:50:26.052 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_freeze_tree_timeout": "30.000000", 2024-09-11T09:50:26.052 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_fscrypt_last_block_max_size": "4096", 2024-09-11T09:50:26.052 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_go_bad_corrupt_dentry": "true", 2024-09-11T09:50:26.052 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_hack_allow_loading_invalid_metadata": "false", 2024-09-11T09:50:26.052 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_health_cache_threshold": "1.500000", 2024-09-11T09:50:26.052 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_health_summarize_threshold": "10", 2024-09-11T09:50:26.052 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_heartbeat_grace": "15.000000", 2024-09-11T09:50:26.052 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_heartbeat_reset_grace": "1000", 2024-09-11T09:50:26.052 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_inject_health_dummy": "false", 2024-09-11T09:50:26.052 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_inject_journal_corrupt_dentry_first": "0.000000", 2024-09-11T09:50:26.053 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_inject_migrator_session_race": "false", 2024-09-11T09:50:26.053 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_inject_rename_corrupt_dentry_first": "0.000000", 2024-09-11T09:50:26.053 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_inject_skip_replaying_inotable": "false", 2024-09-11T09:50:26.053 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_inject_traceless_reply_probability": "0.000000", 2024-09-11T09:50:26.053 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_join_fs": "", 2024-09-11T09:50:26.053 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_journal_format": "1", 2024-09-11T09:50:26.053 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_kill_after_journal_logs_flushed": "false", 2024-09-11T09:50:26.053 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_kill_create_at": "0", 2024-09-11T09:50:26.053 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_kill_export_at": "0", 2024-09-11T09:50:26.053 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_kill_import_at": "0", 2024-09-11T09:50:26.053 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_kill_journal_at": "0", 2024-09-11T09:50:26.053 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_kill_journal_expire_at": "0", 2024-09-11T09:50:26.053 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_kill_journal_replay_at": "0", 2024-09-11T09:50:26.053 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_kill_link_at": "0", 2024-09-11T09:50:26.054 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_kill_mdstable_at": "0", 2024-09-11T09:50:26.054 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_kill_openc_at": "0", 2024-09-11T09:50:26.054 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_kill_rename_at": "0", 2024-09-11T09:50:26.054 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_kill_shutdown_at": "0", 2024-09-11T09:50:26.054 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_log_event_large_threshold": "524288", 2024-09-11T09:50:26.054 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_log_events_per_segment": "1024", 2024-09-11T09:50:26.054 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_log_major_segment_event_ratio": "12", 2024-09-11T09:50:26.054 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_log_max_events": "-1", 2024-09-11T09:50:26.054 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_log_max_segments": "128", 2024-09-11T09:50:26.054 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_log_pause": "false", 2024-09-11T09:50:26.054 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_log_segment_size": "0", 2024-09-11T09:50:26.054 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_log_skip_corrupt_events": "false", 2024-09-11T09:50:26.054 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_log_skip_unbounded_events": "false", 2024-09-11T09:50:26.054 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_log_trim_decay_rate": "1.000000", 2024-09-11T09:50:26.054 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_log_trim_threshold": "128", 2024-09-11T09:50:26.055 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_log_trim_upkeep_interval": "1000", 2024-09-11T09:50:26.055 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_log_warn_factor": "2.000000", 2024-09-11T09:50:26.055 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_max_caps_per_client": "1048576", 2024-09-11T09:50:26.055 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_max_completed_flushes": "100000", 2024-09-11T09:50:26.055 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_max_completed_requests": "100000", 2024-09-11T09:50:26.055 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_max_export_size": "20971520", 2024-09-11T09:50:26.055 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_max_file_recover": "32", 2024-09-11T09:50:26.055 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_max_purge_files": "64", 2024-09-11T09:50:26.055 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_max_purge_ops": "8192", 2024-09-11T09:50:26.055 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_max_purge_ops_per_pg": "0.500000", 2024-09-11T09:50:26.055 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_max_scrub_ops_in_progress": "5", 2024-09-11T09:50:26.055 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_max_snaps_per_dir": "100", 2024-09-11T09:50:26.055 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_metrics_update_interval": "2", 2024-09-11T09:50:26.055 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_min_caps_per_client": "100", 2024-09-11T09:50:26.055 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_min_caps_working_set": "10000", 2024-09-11T09:50:26.056 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_mon_shutdown_timeout": "5.000000", 2024-09-11T09:50:26.056 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_numa_node": "-1", 2024-09-11T09:50:26.056 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_oft_prefetch_dirfrags": "false", 2024-09-11T09:50:26.056 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_op_complaint_time": "30.000000", 2024-09-11T09:50:26.056 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_op_history_duration": "600", 2024-09-11T09:50:26.056 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_op_history_size": "20", 2024-09-11T09:50:26.056 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_op_history_slow_op_size": "20", 2024-09-11T09:50:26.056 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_op_history_slow_op_threshold": "10", 2024-09-11T09:50:26.056 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_op_log_threshold": "5", 2024-09-11T09:50:26.056 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_ping_grace": "15", 2024-09-11T09:50:26.056 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_ping_interval": "5", 2024-09-11T09:50:26.056 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_purge_queue_busy_flush_period": "1.000000", 2024-09-11T09:50:26.056 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_recall_global_max_decay_threshold": "131072", 2024-09-11T09:50:26.056 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_recall_max_caps": "30000", 2024-09-11T09:50:26.056 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_recall_max_decay_rate": "1.500000", 2024-09-11T09:50:26.057 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_recall_max_decay_threshold": "131072", 2024-09-11T09:50:26.057 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_recall_warning_decay_rate": "60.000000", 2024-09-11T09:50:26.057 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_recall_warning_threshold": "262144", 2024-09-11T09:50:26.057 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_reconnect_timeout": "45.000000", 2024-09-11T09:50:26.057 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_replay_interval": "1.000000", 2024-09-11T09:50:26.057 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_replay_unsafe_with_closed_session": "false", 2024-09-11T09:50:26.057 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_request_load_average_decay_rate": "60.000000", 2024-09-11T09:50:26.057 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_root_ino_gid": "0", 2024-09-11T09:50:26.057 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_root_ino_uid": "0", 2024-09-11T09:50:26.057 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_scatter_nudge_interval": "5.000000", 2024-09-11T09:50:26.057 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_session_blocklist_on_evict": "true", 2024-09-11T09:50:26.057 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_session_blocklist_on_timeout": "true", 2024-09-11T09:50:26.057 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_session_cache_liveness_decay_rate": "300.000000", 2024-09-11T09:50:26.057 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_session_cache_liveness_magnitude": "10", 2024-09-11T09:50:26.058 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_session_cap_acquisition_decay_rate": "30.000000", 2024-09-11T09:50:26.058 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_session_cap_acquisition_throttle": "100000", 2024-09-11T09:50:26.058 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_session_max_caps_throttle_ratio": "1.100000", 2024-09-11T09:50:26.058 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_session_metadata_threshold": "16777216", 2024-09-11T09:50:26.058 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_sessionmap_keys_per_op": "1024", 2024-09-11T09:50:26.058 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_shutdown_check": "0", 2024-09-11T09:50:26.058 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_skip_ino": "0", 2024-09-11T09:50:26.058 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_sleep_rank_change": "0.000000", 2024-09-11T09:50:26.058 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_snap_max_uid": "4294967294", 2024-09-11T09:50:26.058 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_snap_min_uid": "0", 2024-09-11T09:50:26.058 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_snap_rstat": "false", 2024-09-11T09:50:26.058 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_standby_replay_damaged": "false", 2024-09-11T09:50:26.058 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_symlink_recovery": "true", 2024-09-11T09:50:26.058 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_task_status_update_interval": "2.000000", 2024-09-11T09:50:26.059 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_thrash_exports": "0", 2024-09-11T09:50:26.059 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_thrash_fragments": "0", 2024-09-11T09:50:26.059 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_tick_interval": "5.000000", 2024-09-11T09:50:26.059 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_valgrind_exit": "false", 2024-09-11T09:50:26.059 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_verify_backtrace": "1", 2024-09-11T09:50:26.059 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_verify_scatter": "false", 2024-09-11T09:50:26.059 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_wipe_ino_prealloc": "false", 2024-09-11T09:50:26.059 INFO:teuthology.orchestra.run.smithi017.stdout: "mds_wipe_sessions": "false", 2024-09-11T09:50:26.059 INFO:teuthology.orchestra.run.smithi017.stdout: "mempool_debug": "false", 2024-09-11T09:50:26.059 INFO:teuthology.orchestra.run.smithi017.stdout: "memstore_debug_omit_block_device_write": "false", 2024-09-11T09:50:26.059 INFO:teuthology.orchestra.run.smithi017.stdout: "memstore_device_bytes": "1073741824", 2024-09-11T09:50:26.059 INFO:teuthology.orchestra.run.smithi017.stdout: "memstore_page_set": "false", 2024-09-11T09:50:26.059 INFO:teuthology.orchestra.run.smithi017.stdout: "memstore_page_size": "65536", 2024-09-11T09:50:26.060 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr_client_bytes": "134217728", 2024-09-11T09:50:26.060 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr_client_messages": "512", 2024-09-11T09:50:26.060 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr_client_service_daemon_unregister_timeout": "1.000000", 2024-09-11T09:50:26.060 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr_connect_retry_interval": "1.000000", 2024-09-11T09:50:26.060 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr_data": "/var/lib/ceph/mgr/ceph-a", 2024-09-11T09:50:26.060 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr_debug_aggressive_pg_num_changes": "false", 2024-09-11T09:50:26.060 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr_disabled_modules": "", 2024-09-11T09:50:26.060 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr_enable_op_tracker": "true", 2024-09-11T09:50:26.060 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr_initial_modules": "restful iostat nfs", 2024-09-11T09:50:26.060 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr_max_pg_num_change": "128", 2024-09-11T09:50:26.060 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr_mds_bytes": "134217728", 2024-09-11T09:50:26.060 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr_mds_messages": "128", 2024-09-11T09:50:26.060 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr_module_path": "/usr/share/ceph/mgr", 2024-09-11T09:50:26.060 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr_mon_bytes": "134217728", 2024-09-11T09:50:26.061 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr_mon_messages": "128", 2024-09-11T09:50:26.061 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr_num_op_tracker_shard": "32", 2024-09-11T09:50:26.061 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr_op_complaint_time": "30.000000", 2024-09-11T09:50:26.061 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr_op_history_duration": "600", 2024-09-11T09:50:26.061 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr_op_history_size": "20", 2024-09-11T09:50:26.061 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr_op_history_slow_op_size": "20", 2024-09-11T09:50:26.061 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr_op_history_slow_op_threshold": "10.000000", 2024-09-11T09:50:26.061 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr_op_log_threshold": "5", 2024-09-11T09:50:26.061 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr_osd_bytes": "536870912", 2024-09-11T09:50:26.061 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr_osd_messages": "8192", 2024-09-11T09:50:26.061 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr_pool": "true", 2024-09-11T09:50:26.061 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr_service_beacon_grace": "60.000000", 2024-09-11T09:50:26.061 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr_standby_modules": "true", 2024-09-11T09:50:26.061 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr_stats_period": "5", 2024-09-11T09:50:26.062 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr_stats_threshold": "5", 2024-09-11T09:50:26.062 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr_tick_period": "2", 2024-09-11T09:50:26.062 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr_ttl_cache_expire_seconds": "0", 2024-09-11T09:50:26.062 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_accept_timeout_factor": "2.000000", 2024-09-11T09:50:26.062 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_allow_pool_delete": "true", 2024-09-11T09:50:26.062 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_allow_pool_size_one": "true", 2024-09-11T09:50:26.062 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_auth_validate_all_caps": "true", 2024-09-11T09:50:26.062 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_cache_target_full_warn_ratio": "0.660000", 2024-09-11T09:50:26.062 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_clean_pg_upmaps_per_chunk": "256", 2024-09-11T09:50:26.062 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_client_bytes": "104857600", 2024-09-11T09:50:26.062 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_client_directed_command_retry": "5", 2024-09-11T09:50:26.062 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_client_hunt_interval": "3.000000", 2024-09-11T09:50:26.063 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_client_hunt_interval_backoff": "1.500000", 2024-09-11T09:50:26.063 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_client_hunt_interval_max_multiple": "10.000000", 2024-09-11T09:50:26.063 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_client_hunt_interval_min_multiple": "1.000000", 2024-09-11T09:50:26.063 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_client_hunt_parallel": "3", 2024-09-11T09:50:26.063 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_client_log_interval": "1.000000", 2024-09-11T09:50:26.063 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_client_max_log_entries_per_message": "1000", 2024-09-11T09:50:26.063 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_client_ping_interval": "10.000000", 2024-09-11T09:50:26.063 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_client_ping_timeout": "30.000000", 2024-09-11T09:50:26.063 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_client_target_rank": "-1", 2024-09-11T09:50:26.063 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_clock_drift_allowed": "1.000000", 2024-09-11T09:50:26.063 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_clock_drift_warn_backoff": "5.000000", 2024-09-11T09:50:26.063 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_cluster_log_file": "default=/var/log/ceph/ceph.$channel.log cluster=/var/log/ceph/ceph.log", 2024-09-11T09:50:26.063 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_cluster_log_file_level": "debug", 2024-09-11T09:50:26.064 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_cluster_log_to_file": "true", 2024-09-11T09:50:26.064 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_cluster_log_to_graylog": "false", 2024-09-11T09:50:26.064 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_cluster_log_to_graylog_host": "127.0.0.1", 2024-09-11T09:50:26.064 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_cluster_log_to_graylog_port": "12201", 2024-09-11T09:50:26.064 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_cluster_log_to_journald": "false", 2024-09-11T09:50:26.064 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_cluster_log_to_stderr": "false", 2024-09-11T09:50:26.064 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_cluster_log_to_syslog": "default=false", 2024-09-11T09:50:26.064 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_cluster_log_to_syslog_facility": "daemon", 2024-09-11T09:50:26.065 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_cluster_log_to_syslog_level": "info", 2024-09-11T09:50:26.065 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_compact_on_bootstrap": "false", 2024-09-11T09:50:26.065 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_compact_on_start": "false", 2024-09-11T09:50:26.065 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_compact_on_trim": "true", 2024-09-11T09:50:26.065 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_con_tracker_persist_interval": "10", 2024-09-11T09:50:26.065 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_con_tracker_score_halflife": "43200", 2024-09-11T09:50:26.065 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_config_key_max_entry_size": "65536", 2024-09-11T09:50:26.065 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_cpu_threads": "4", 2024-09-11T09:50:26.065 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_crush_min_required_version": "hammer", 2024-09-11T09:50:26.065 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_daemon_bytes": "419430400", 2024-09-11T09:50:26.065 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_data": "/var/lib/ceph/mon/ceph-a", 2024-09-11T09:50:26.065 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_data_avail_crit": "5", 2024-09-11T09:50:26.065 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_data_avail_warn": "5", 2024-09-11T09:50:26.066 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_data_size_warn": "16106127360", 2024-09-11T09:50:26.066 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_debug_block_osdmap_trim": "false", 2024-09-11T09:50:26.066 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_debug_deprecated_as_obsolete": "false", 2024-09-11T09:50:26.066 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_debug_dump_json": "false", 2024-09-11T09:50:26.066 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_debug_dump_location": "/var/log/ceph/ceph-mon.a.tdump", 2024-09-11T09:50:26.066 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_debug_dump_transactions": "false", 2024-09-11T09:50:26.066 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_debug_extra_checks": "false", 2024-09-11T09:50:26.066 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_debug_no_initial_persistent_features": "false", 2024-09-11T09:50:26.066 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_debug_no_require_bluestore_for_ec_overwrites": "false", 2024-09-11T09:50:26.066 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_debug_no_require_reef": "false", 2024-09-11T09:50:26.066 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_debug_no_require_squid": "false", 2024-09-11T09:50:26.066 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_debug_unsafe_allow_tier_with_nonempty_snaps": "false", 2024-09-11T09:50:26.066 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_delta_reset_interval": "10.000000", 2024-09-11T09:50:26.066 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_dns_srv_name": "ceph-mon", 2024-09-11T09:50:26.067 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_down_mkfs_grace": "120", 2024-09-11T09:50:26.067 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_down_uptime_grace": "60", 2024-09-11T09:50:26.067 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_election_default_strategy": "1", 2024-09-11T09:50:26.067 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_election_timeout": "5.000000", 2024-09-11T09:50:26.067 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_elector_ignore_propose_margin": "0.000500", 2024-09-11T09:50:26.067 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_elector_ping_divisor": "2", 2024-09-11T09:50:26.067 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_elector_ping_timeout": "2.000000", 2024-09-11T09:50:26.067 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_enable_op_tracker": "true", 2024-09-11T09:50:26.067 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_fake_pool_delete": "false", 2024-09-11T09:50:26.067 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_force_quorum_join": "false", 2024-09-11T09:50:26.067 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_fsmap_prune_threshold": "300", 2024-09-11T09:50:26.067 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_globalid_prealloc": "10000", 2024-09-11T09:50:26.067 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_health_detail_to_clog": "false", 2024-09-11T09:50:26.067 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_health_log_update_period": "5", 2024-09-11T09:50:26.067 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_health_max_detail": "50", 2024-09-11T09:50:26.068 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_health_to_clog": "true", 2024-09-11T09:50:26.068 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_health_to_clog_interval": "600", 2024-09-11T09:50:26.068 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_health_to_clog_tick_interval": "60.000000", 2024-09-11T09:50:26.068 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_host": "172.21.15.17,[v2:172.21.15.17:3301,v1:172.21.15.17:6790],172.21.15.96", 2024-09-11T09:50:26.068 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_host_override": "", 2024-09-11T09:50:26.068 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_initial_members": "", 2024-09-11T09:50:26.068 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_inject_pg_merge_bounce_probability": "0.000000", 2024-09-11T09:50:26.068 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_inject_sync_get_chunk_delay": "0.000000", 2024-09-11T09:50:26.068 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_inject_transaction_delay_max": "10.000000", 2024-09-11T09:50:26.068 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_inject_transaction_delay_probability": "0.000000", 2024-09-11T09:50:26.068 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_keyvaluedb": "rocksdb", 2024-09-11T09:50:26.068 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_lease": "5.000000", 2024-09-11T09:50:26.068 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_lease_ack_timeout_factor": "2.000000", 2024-09-11T09:50:26.068 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_lease_renew_interval_factor": "0.600000", 2024-09-11T09:50:26.069 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_log_full_interval": "50", 2024-09-11T09:50:26.069 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_log_max": "10000", 2024-09-11T09:50:26.069 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_log_max_summary": "50", 2024-09-11T09:50:26.069 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_max_log_entries_per_event": "4096", 2024-09-11T09:50:26.069 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_max_log_epochs": "500", 2024-09-11T09:50:26.069 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_max_mdsmap_epochs": "500", 2024-09-11T09:50:26.069 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_max_mgrmap_epochs": "500", 2024-09-11T09:50:26.069 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_max_osd": "10000", 2024-09-11T09:50:26.069 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_max_pg_per_osd": "10000", 2024-09-11T09:50:26.069 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_max_pool_pg_num": "65536", 2024-09-11T09:50:26.069 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_max_snap_prune_per_epoch": "100", 2024-09-11T09:50:26.069 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_mds_blocklist_interval": "86400.000000", 2024-09-11T09:50:26.069 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_mds_force_trim_to": "0", 2024-09-11T09:50:26.069 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_mds_skip_sanity": "false", 2024-09-11T09:50:26.070 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_memory_autotune": "true", 2024-09-11T09:50:26.070 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_memory_target": "2147483648", 2024-09-11T09:50:26.070 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_mgr_beacon_grace": "30", 2024-09-11T09:50:26.070 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_mgr_blocklist_interval": "86400.000000", 2024-09-11T09:50:26.070 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_mgr_digest_period": "5", 2024-09-11T09:50:26.070 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_mgr_inactive_grace": "60", 2024-09-11T09:50:26.070 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_mgr_mkfs_grace": "240", 2024-09-11T09:50:26.070 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_mgr_proxy_client_bytes_ratio": "0.300000", 2024-09-11T09:50:26.070 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_min_osdmap_epochs": "500", 2024-09-11T09:50:26.070 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_op_complaint_time": "30", 2024-09-11T09:50:26.070 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_op_history_duration": "600", 2024-09-11T09:50:26.070 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_op_history_size": "20", 2024-09-11T09:50:26.070 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_op_history_slow_op_size": "20", 2024-09-11T09:50:26.070 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_op_history_slow_op_threshold": "10", 2024-09-11T09:50:26.070 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_op_log_threshold": "5", 2024-09-11T09:50:26.071 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_osd_adjust_down_out_interval": "true", 2024-09-11T09:50:26.071 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_osd_adjust_heartbeat_grace": "true", 2024-09-11T09:50:26.071 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_osd_auto_mark_auto_out_in": "true", 2024-09-11T09:50:26.071 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_osd_auto_mark_in": "false", 2024-09-11T09:50:26.071 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_osd_auto_mark_new_in": "true", 2024-09-11T09:50:26.071 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_osd_backfillfull_ratio": "0.900000", 2024-09-11T09:50:26.071 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_osd_blocklist_default_expire": "3600.000000", 2024-09-11T09:50:26.071 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_osd_cache_size": "500", 2024-09-11T09:50:26.071 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_osd_cache_size_min": "134217728", 2024-09-11T09:50:26.071 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_osd_crush_smoke_test": "true", 2024-09-11T09:50:26.071 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_osd_destroyed_out_interval": "600", 2024-09-11T09:50:26.071 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_osd_down_out_interval": "600", 2024-09-11T09:50:26.071 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_osd_down_out_subtree_limit": "rack", 2024-09-11T09:50:26.072 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_osd_err_op_age_ratio": "128.000000", 2024-09-11T09:50:26.072 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_osd_force_trim_to": "0", 2024-09-11T09:50:26.072 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_osd_full_ratio": "0.950000", 2024-09-11T09:50:26.072 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_osd_initial_require_min_compat_client": "luminous", 2024-09-11T09:50:26.072 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_osd_laggy_halflife": "3600", 2024-09-11T09:50:26.072 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_osd_laggy_max_interval": "300", 2024-09-11T09:50:26.072 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_osd_laggy_weight": "0.300000", 2024-09-11T09:50:26.072 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_osd_mapping_pgs_per_chunk": "4096", 2024-09-11T09:50:26.072 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_osd_max_creating_pgs": "1024", 2024-09-11T09:50:26.072 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_osd_max_initial_pgs": "1024", 2024-09-11T09:50:26.072 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_osd_min_down_reporters": "2", 2024-09-11T09:50:26.072 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_osd_min_in_ratio": "0.750000", 2024-09-11T09:50:26.072 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_osd_min_up_ratio": "0.300000", 2024-09-11T09:50:26.072 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_osd_nearfull_ratio": "0.850000", 2024-09-11T09:50:26.073 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_osd_prime_pg_temp": "true", 2024-09-11T09:50:26.073 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_osd_prime_pg_temp_max_estimate": "0.250000", 2024-09-11T09:50:26.073 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_osd_prime_pg_temp_max_time": "0.500000", 2024-09-11T09:50:26.073 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_osd_report_timeout": "900", 2024-09-11T09:50:26.073 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_osd_reporter_subtree_level": "osd", 2024-09-11T09:50:26.073 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_osd_snap_trim_queue_warn_on": "32768", 2024-09-11T09:50:26.073 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_osd_warn_num_repaired": "10", 2024-09-11T09:50:26.073 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_osd_warn_op_age": "32.000000", 2024-09-11T09:50:26.073 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_osdmap_full_prune_enabled": "true", 2024-09-11T09:50:26.073 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_osdmap_full_prune_interval": "10", 2024-09-11T09:50:26.073 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_osdmap_full_prune_min": "10000", 2024-09-11T09:50:26.073 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_osdmap_full_prune_txsize": "100", 2024-09-11T09:50:26.073 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_pg_check_down_all_threshold": "0.500000", 2024-09-11T09:50:26.074 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_pg_stuck_threshold": "60", 2024-09-11T09:50:26.074 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_pg_warn_max_object_skew": "0.000000", 2024-09-11T09:50:26.074 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_pg_warn_min_objects": "10000", 2024-09-11T09:50:26.074 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_pg_warn_min_per_osd": "0", 2024-09-11T09:50:26.074 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_pg_warn_min_pool_objects": "1000", 2024-09-11T09:50:26.074 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_pool_quota_crit_threshold": "0", 2024-09-11T09:50:26.074 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_pool_quota_warn_threshold": "0", 2024-09-11T09:50:26.074 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_probe_timeout": "2.000000", 2024-09-11T09:50:26.074 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_reweight_max_change": "0.050000", 2024-09-11T09:50:26.074 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_reweight_max_osds": "4", 2024-09-11T09:50:26.074 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_reweight_min_bytes_per_osd": "10", 2024-09-11T09:50:26.074 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_reweight_min_pgs_per_osd": "4", 2024-09-11T09:50:26.074 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_rocksdb_options": "write_buffer_size=33554432,compression=kNoCompression,level_compaction_dynamic_level_bytes=true", 2024-09-11T09:50:26.074 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_scrub_inject_crc_mismatch": "0.000000", 2024-09-11T09:50:26.074 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_scrub_inject_missing_keys": "0.000000", 2024-09-11T09:50:26.075 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_scrub_interval": "86400", 2024-09-11T09:50:26.075 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_scrub_max_keys": "100", 2024-09-11T09:50:26.075 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_scrub_timeout": "300", 2024-09-11T09:50:26.075 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_session_timeout": "300", 2024-09-11T09:50:26.075 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_smart_report_timeout": "5", 2024-09-11T09:50:26.075 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_stat_smooth_intervals": "6", 2024-09-11T09:50:26.075 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_stretch_cluster_recovery_ratio": "0.600000", 2024-09-11T09:50:26.075 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_stretch_pool_min_size": "2", 2024-09-11T09:50:26.075 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_stretch_pool_size": "4", 2024-09-11T09:50:26.075 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_stretch_recovery_min_wait": "15.000000", 2024-09-11T09:50:26.075 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_subscribe_interval": "86400.000000", 2024-09-11T09:50:26.075 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_sync_debug": "false", 2024-09-11T09:50:26.075 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_sync_max_payload_keys": "2000", 2024-09-11T09:50:26.075 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_sync_max_payload_size": "1048576", 2024-09-11T09:50:26.075 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_sync_provider_kill_at": "0", 2024-09-11T09:50:26.076 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_sync_requester_kill_at": "0", 2024-09-11T09:50:26.076 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_sync_timeout": "60.000000", 2024-09-11T09:50:26.076 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_target_pg_per_osd": "100", 2024-09-11T09:50:26.076 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_tick_interval": "5", 2024-09-11T09:50:26.076 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_timecheck_interval": "300.000000", 2024-09-11T09:50:26.076 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_timecheck_skew_interval": "30.000000", 2024-09-11T09:50:26.076 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_use_min_delay_socket": "false", 2024-09-11T09:50:26.076 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_warn_older_version_delay": "604800", 2024-09-11T09:50:26.076 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_warn_on_cache_pools_without_hit_sets": "true", 2024-09-11T09:50:26.076 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_warn_on_crush_straw_calc_version_zero": "false", 2024-09-11T09:50:26.076 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_warn_on_degraded_stretch_mode": "true", 2024-09-11T09:50:26.076 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_warn_on_filestore_osds": "false", 2024-09-11T09:50:26.076 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_warn_on_insecure_global_id_reclaim": "false", 2024-09-11T09:50:26.077 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_warn_on_insecure_global_id_reclaim_allowed": "false", 2024-09-11T09:50:26.077 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_warn_on_legacy_crush_tunables": "false", 2024-09-11T09:50:26.077 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_warn_on_misplaced": "false", 2024-09-11T09:50:26.077 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_warn_on_msgr2_not_enabled": "true", 2024-09-11T09:50:26.077 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_warn_on_older_version": "true", 2024-09-11T09:50:26.077 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_warn_on_osd_down_out_interval_zero": "false", 2024-09-11T09:50:26.077 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_warn_on_pool_no_app": "false", 2024-09-11T09:50:26.077 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_warn_on_pool_no_redundancy": "false", 2024-09-11T09:50:26.077 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_warn_on_pool_pg_num_not_power_of_two": "false", 2024-09-11T09:50:26.077 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_warn_on_slow_ping_ratio": "0.050000", 2024-09-11T09:50:26.077 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_warn_on_slow_ping_time": "0.000000", 2024-09-11T09:50:26.077 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_warn_on_too_few_osds": "false", 2024-09-11T09:50:26.077 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_warn_pg_not_deep_scrubbed_ratio": "0.750000", 2024-09-11T09:50:26.077 INFO:teuthology.orchestra.run.smithi017.stdout: "mon_warn_pg_not_scrubbed_ratio": "0.500000", 2024-09-11T09:50:26.078 INFO:teuthology.orchestra.run.smithi017.stdout: "monmap": "", 2024-09-11T09:50:26.078 INFO:teuthology.orchestra.run.smithi017.stdout: "motr_admin_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-09-11T09:50:26.078 INFO:teuthology.orchestra.run.smithi017.stdout: "motr_admin_fid": "0x7200000000000001:0x0", 2024-09-11T09:50:26.078 INFO:teuthology.orchestra.run.smithi017.stdout: "motr_ha_endpoint": "192.168.180.182@tcp:12345:1:1", 2024-09-11T09:50:26.078 INFO:teuthology.orchestra.run.smithi017.stdout: "motr_my_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-09-11T09:50:26.078 INFO:teuthology.orchestra.run.smithi017.stdout: "motr_my_fid": "0x7200000000000001:0x0", 2024-09-11T09:50:26.078 INFO:teuthology.orchestra.run.smithi017.stdout: "motr_profile_fid": "0x7000000000000001:0x0", 2024-09-11T09:50:26.078 INFO:teuthology.orchestra.run.smithi017.stdout: "motr_tracing_enabled": "false", 2024-09-11T09:50:26.078 INFO:teuthology.orchestra.run.smithi017.stdout: "ms_async_op_threads": "3", 2024-09-11T09:50:26.078 INFO:teuthology.orchestra.run.smithi017.stdout: "ms_async_rdma_buffer_size": "131072", 2024-09-11T09:50:26.078 INFO:teuthology.orchestra.run.smithi017.stdout: "ms_async_rdma_cm": "false", 2024-09-11T09:50:26.078 INFO:teuthology.orchestra.run.smithi017.stdout: "ms_async_rdma_device_name": "", 2024-09-11T09:50:26.078 INFO:teuthology.orchestra.run.smithi017.stdout: "ms_async_rdma_dscp": "96", 2024-09-11T09:50:26.078 INFO:teuthology.orchestra.run.smithi017.stdout: "ms_async_rdma_enable_hugepage": "false", 2024-09-11T09:50:26.079 INFO:teuthology.orchestra.run.smithi017.stdout: "ms_async_rdma_gid_idx": "0", 2024-09-11T09:50:26.079 INFO:teuthology.orchestra.run.smithi017.stdout: "ms_async_rdma_local_gid": "", 2024-09-11T09:50:26.079 INFO:teuthology.orchestra.run.smithi017.stdout: "ms_async_rdma_polling_us": "1000", 2024-09-11T09:50:26.079 INFO:teuthology.orchestra.run.smithi017.stdout: "ms_async_rdma_port_num": "1", 2024-09-11T09:50:26.079 INFO:teuthology.orchestra.run.smithi017.stdout: "ms_async_rdma_receive_buffers": "32768", 2024-09-11T09:50:26.079 INFO:teuthology.orchestra.run.smithi017.stdout: "ms_async_rdma_receive_queue_len": "4096", 2024-09-11T09:50:26.079 INFO:teuthology.orchestra.run.smithi017.stdout: "ms_async_rdma_roce_ver": "1", 2024-09-11T09:50:26.079 INFO:teuthology.orchestra.run.smithi017.stdout: "ms_async_rdma_send_buffers": "1024", 2024-09-11T09:50:26.079 INFO:teuthology.orchestra.run.smithi017.stdout: "ms_async_rdma_sl": "3", 2024-09-11T09:50:26.079 INFO:teuthology.orchestra.run.smithi017.stdout: "ms_async_rdma_support_srq": "true", 2024-09-11T09:50:26.079 INFO:teuthology.orchestra.run.smithi017.stdout: "ms_async_rdma_type": "ib", 2024-09-11T09:50:26.079 INFO:teuthology.orchestra.run.smithi017.stdout: "ms_async_reap_threshold": "5", 2024-09-11T09:50:26.079 INFO:teuthology.orchestra.run.smithi017.stdout: "ms_bind_before_connect": "false", 2024-09-11T09:50:26.079 INFO:teuthology.orchestra.run.smithi017.stdout: "ms_bind_ipv4": "true", 2024-09-11T09:50:26.080 INFO:teuthology.orchestra.run.smithi017.stdout: "ms_bind_ipv6": "false", 2024-09-11T09:50:26.080 INFO:teuthology.orchestra.run.smithi017.stdout: "ms_bind_msgr1": "true", 2024-09-11T09:50:26.080 INFO:teuthology.orchestra.run.smithi017.stdout: "ms_bind_msgr2": "true", 2024-09-11T09:50:26.080 INFO:teuthology.orchestra.run.smithi017.stdout: "ms_bind_port_max": "7568", 2024-09-11T09:50:26.080 INFO:teuthology.orchestra.run.smithi017.stdout: "ms_bind_port_min": "6800", 2024-09-11T09:50:26.080 INFO:teuthology.orchestra.run.smithi017.stdout: "ms_bind_prefer_ipv4": "false", 2024-09-11T09:50:26.080 INFO:teuthology.orchestra.run.smithi017.stdout: "ms_bind_retry_count": "3", 2024-09-11T09:50:26.080 INFO:teuthology.orchestra.run.smithi017.stdout: "ms_bind_retry_delay": "5", 2024-09-11T09:50:26.080 INFO:teuthology.orchestra.run.smithi017.stdout: "ms_blackhole_client": "false", 2024-09-11T09:50:26.080 INFO:teuthology.orchestra.run.smithi017.stdout: "ms_blackhole_mds": "false", 2024-09-11T09:50:26.080 INFO:teuthology.orchestra.run.smithi017.stdout: "ms_blackhole_mgr": "false", 2024-09-11T09:50:26.080 INFO:teuthology.orchestra.run.smithi017.stdout: "ms_blackhole_mon": "false", 2024-09-11T09:50:26.080 INFO:teuthology.orchestra.run.smithi017.stdout: "ms_blackhole_osd": "false", 2024-09-11T09:50:26.080 INFO:teuthology.orchestra.run.smithi017.stdout: "ms_client_mode": "crc secure", 2024-09-11T09:50:26.080 INFO:teuthology.orchestra.run.smithi017.stdout: "ms_cluster_mode": "crc secure", 2024-09-11T09:50:26.081 INFO:teuthology.orchestra.run.smithi017.stdout: "ms_cluster_type": "", 2024-09-11T09:50:26.081 INFO:teuthology.orchestra.run.smithi017.stdout: "ms_compress_secure": "false", 2024-09-11T09:50:26.081 INFO:teuthology.orchestra.run.smithi017.stdout: "ms_connection_idle_timeout": "900", 2024-09-11T09:50:26.081 INFO:teuthology.orchestra.run.smithi017.stdout: "ms_connection_ready_timeout": "10", 2024-09-11T09:50:26.081 INFO:teuthology.orchestra.run.smithi017.stdout: "ms_crc_data": "true", 2024-09-11T09:50:26.081 INFO:teuthology.orchestra.run.smithi017.stdout: "ms_crc_header": "true", 2024-09-11T09:50:26.081 INFO:teuthology.orchestra.run.smithi017.stdout: "ms_die_on_bad_msg": "false", 2024-09-11T09:50:26.081 INFO:teuthology.orchestra.run.smithi017.stdout: "ms_die_on_bug": "true", 2024-09-11T09:50:26.081 INFO:teuthology.orchestra.run.smithi017.stdout: "ms_die_on_old_message": "true", 2024-09-11T09:50:26.081 INFO:teuthology.orchestra.run.smithi017.stdout: "ms_die_on_skipped_message": "false", 2024-09-11T09:50:26.081 INFO:teuthology.orchestra.run.smithi017.stdout: "ms_die_on_unhandled_msg": "false", 2024-09-11T09:50:26.081 INFO:teuthology.orchestra.run.smithi017.stdout: "ms_dispatch_throttle_bytes": "104857600", 2024-09-11T09:50:26.081 INFO:teuthology.orchestra.run.smithi017.stdout: "ms_dpdk_coremask": "0xF", 2024-09-11T09:50:26.082 INFO:teuthology.orchestra.run.smithi017.stdout: "ms_dpdk_debug_allow_loopback": "false", 2024-09-11T09:50:26.082 INFO:teuthology.orchestra.run.smithi017.stdout: "ms_dpdk_devs_allowlist": "", 2024-09-11T09:50:26.082 INFO:teuthology.orchestra.run.smithi017.stdout: "ms_dpdk_enable_tso": "true", 2024-09-11T09:50:26.082 INFO:teuthology.orchestra.run.smithi017.stdout: "ms_dpdk_gateway_ipv4_addr": "", 2024-09-11T09:50:26.082 INFO:teuthology.orchestra.run.smithi017.stdout: "ms_dpdk_host_ipv4_addr": "", 2024-09-11T09:50:26.082 INFO:teuthology.orchestra.run.smithi017.stdout: "ms_dpdk_hugepages": "", 2024-09-11T09:50:26.082 INFO:teuthology.orchestra.run.smithi017.stdout: "ms_dpdk_hw_flow_control": "true", 2024-09-11T09:50:26.082 INFO:teuthology.orchestra.run.smithi017.stdout: "ms_dpdk_hw_queue_weight": "1.000000", 2024-09-11T09:50:26.082 INFO:teuthology.orchestra.run.smithi017.stdout: "ms_dpdk_lro": "true", 2024-09-11T09:50:26.082 INFO:teuthology.orchestra.run.smithi017.stdout: "ms_dpdk_memory_channel": "4", 2024-09-11T09:50:26.082 INFO:teuthology.orchestra.run.smithi017.stdout: "ms_dpdk_netmask_ipv4_addr": "", 2024-09-11T09:50:26.082 INFO:teuthology.orchestra.run.smithi017.stdout: "ms_dpdk_pmd": "", 2024-09-11T09:50:26.082 INFO:teuthology.orchestra.run.smithi017.stdout: "ms_dpdk_port_id": "0", 2024-09-11T09:50:26.083 INFO:teuthology.orchestra.run.smithi017.stdout: "ms_dpdk_rx_buffer_count_per_core": "8192", 2024-09-11T09:50:26.083 INFO:teuthology.orchestra.run.smithi017.stdout: "ms_dump_corrupt_message_level": "1", 2024-09-11T09:50:26.083 INFO:teuthology.orchestra.run.smithi017.stdout: "ms_dump_on_send": "false", 2024-09-11T09:50:26.083 INFO:teuthology.orchestra.run.smithi017.stdout: "ms_initial_backoff": "0.200000", 2024-09-11T09:50:26.083 INFO:teuthology.orchestra.run.smithi017.stdout: "ms_inject_delay_max": "1.000000", 2024-09-11T09:50:26.083 INFO:teuthology.orchestra.run.smithi017.stdout: "ms_inject_delay_probability": "0.000000", 2024-09-11T09:50:26.083 INFO:teuthology.orchestra.run.smithi017.stdout: "ms_inject_delay_type": "", 2024-09-11T09:50:26.083 INFO:teuthology.orchestra.run.smithi017.stdout: "ms_inject_internal_delays": "0.000000", 2024-09-11T09:50:26.083 INFO:teuthology.orchestra.run.smithi017.stdout: "ms_inject_network_congestion": "0", 2024-09-11T09:50:26.083 INFO:teuthology.orchestra.run.smithi017.stdout: "ms_inject_socket_failures": "5000", 2024-09-11T09:50:26.083 INFO:teuthology.orchestra.run.smithi017.stdout: "ms_learn_addr_from_peer": "true", 2024-09-11T09:50:26.083 INFO:teuthology.orchestra.run.smithi017.stdout: "ms_max_accept_failures": "4", 2024-09-11T09:50:26.083 INFO:teuthology.orchestra.run.smithi017.stdout: "ms_max_backoff": "15.000000", 2024-09-11T09:50:26.084 INFO:teuthology.orchestra.run.smithi017.stdout: "ms_mon_client_mode": "secure crc", 2024-09-11T09:50:26.084 INFO:teuthology.orchestra.run.smithi017.stdout: "ms_mon_cluster_mode": "secure crc", 2024-09-11T09:50:26.084 INFO:teuthology.orchestra.run.smithi017.stdout: "ms_mon_service_mode": "secure crc", 2024-09-11T09:50:26.084 INFO:teuthology.orchestra.run.smithi017.stdout: "ms_osd_compress_min_size": "1024", 2024-09-11T09:50:26.084 INFO:teuthology.orchestra.run.smithi017.stdout: "ms_osd_compress_mode": "none", 2024-09-11T09:50:26.084 INFO:teuthology.orchestra.run.smithi017.stdout: "ms_osd_compression_algorithm": "snappy", 2024-09-11T09:50:26.084 INFO:teuthology.orchestra.run.smithi017.stdout: "ms_pq_max_tokens_per_priority": "16777216", 2024-09-11T09:50:26.084 INFO:teuthology.orchestra.run.smithi017.stdout: "ms_pq_min_cost": "65536", 2024-09-11T09:50:26.084 INFO:teuthology.orchestra.run.smithi017.stdout: "ms_public_type": "", 2024-09-11T09:50:26.084 INFO:teuthology.orchestra.run.smithi017.stdout: "ms_service_mode": "crc secure", 2024-09-11T09:50:26.084 INFO:teuthology.orchestra.run.smithi017.stdout: "ms_tcp_listen_backlog": "512", 2024-09-11T09:50:26.084 INFO:teuthology.orchestra.run.smithi017.stdout: "ms_tcp_nodelay": "true", 2024-09-11T09:50:26.084 INFO:teuthology.orchestra.run.smithi017.stdout: "ms_tcp_prefetch_max_size": "4096", 2024-09-11T09:50:26.085 INFO:teuthology.orchestra.run.smithi017.stdout: "ms_tcp_rcvbuf": "0", 2024-09-11T09:50:26.085 INFO:teuthology.orchestra.run.smithi017.stdout: "ms_type": "async+posix", 2024-09-11T09:50:26.085 INFO:teuthology.orchestra.run.smithi017.stdout: "no_config_file": "false", 2024-09-11T09:50:26.085 INFO:teuthology.orchestra.run.smithi017.stdout: "objecter_completion_locks_per_session": "32", 2024-09-11T09:50:26.085 INFO:teuthology.orchestra.run.smithi017.stdout: "objecter_debug_inject_relock_delay": "false", 2024-09-11T09:50:26.085 INFO:teuthology.orchestra.run.smithi017.stdout: "objecter_inflight_op_bytes": "104857600", 2024-09-11T09:50:26.085 INFO:teuthology.orchestra.run.smithi017.stdout: "objecter_inflight_ops": "1024", 2024-09-11T09:50:26.085 INFO:teuthology.orchestra.run.smithi017.stdout: "objecter_inject_no_watch_ping": "false", 2024-09-11T09:50:26.085 INFO:teuthology.orchestra.run.smithi017.stdout: "objecter_retry_writes_after_first_reply": "false", 2024-09-11T09:50:26.085 INFO:teuthology.orchestra.run.smithi017.stdout: "objecter_tick_interval": "5.000000", 2024-09-11T09:50:26.085 INFO:teuthology.orchestra.run.smithi017.stdout: "objecter_timeout": "10.000000", 2024-09-11T09:50:26.085 INFO:teuthology.orchestra.run.smithi017.stdout: "objectstore_blackhole": "false", 2024-09-11T09:50:26.085 INFO:teuthology.orchestra.run.smithi017.stdout: "objectstore_debug_throw_on_failed_txc": "false", 2024-09-11T09:50:26.086 INFO:teuthology.orchestra.run.smithi017.stdout: "openssl_engine_opts": "", 2024-09-11T09:50:26.086 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_agent_delay_time": "5.000000", 2024-09-11T09:50:26.086 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_agent_hist_halflife": "1000", 2024-09-11T09:50:26.086 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_agent_max_low_ops": "2", 2024-09-11T09:50:26.086 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_agent_max_ops": "4", 2024-09-11T09:50:26.086 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_agent_min_evict_effort": "0.100000", 2024-09-11T09:50:26.086 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_agent_quantize_effort": "0.100000", 2024-09-11T09:50:26.086 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_agent_slop": "0.020000", 2024-09-11T09:50:26.086 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_aggregated_slow_ops_logging": "true", 2024-09-11T09:50:26.086 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_allow_recovery_below_min_size": "true", 2024-09-11T09:50:26.086 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_asio_thread_count": "2", 2024-09-11T09:50:26.086 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_async_recovery_min_cost": "100", 2024-09-11T09:50:26.086 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_auto_mark_unfound_lost": "false", 2024-09-11T09:50:26.087 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_backfill_retry_interval": "30.000000", 2024-09-11T09:50:26.087 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_backfill_scan_max": "512", 2024-09-11T09:50:26.087 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_backfill_scan_min": "64", 2024-09-11T09:50:26.087 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_backoff_on_degraded": "false", 2024-09-11T09:50:26.087 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_backoff_on_peering": "false", 2024-09-11T09:50:26.087 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_backoff_on_unfound": "true", 2024-09-11T09:50:26.087 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_beacon_report_interval": "300", 2024-09-11T09:50:26.087 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_bench_duration": "30", 2024-09-11T09:50:26.087 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_bench_large_size_max_throughput": "104857600", 2024-09-11T09:50:26.087 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_bench_max_block_size": "67108864", 2024-09-11T09:50:26.087 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_bench_small_size_max_iops": "100", 2024-09-11T09:50:26.087 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_blkin_trace_all": "false", 2024-09-11T09:50:26.087 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_blocked_scrub_grace_period": "120", 2024-09-11T09:50:26.087 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_calc_pg_upmaps_aggressively": "true", 2024-09-11T09:50:26.087 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_calc_pg_upmaps_aggressively_fast": "true", 2024-09-11T09:50:26.088 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_calc_pg_upmaps_local_fallback_retries": "100", 2024-09-11T09:50:26.088 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_check_for_log_corruption": "false", 2024-09-11T09:50:26.088 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_check_max_object_name_len_on_startup": "true", 2024-09-11T09:50:26.088 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_class_default_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-09-11T09:50:26.088 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_class_dir": "/usr/lib64/rados-classes", 2024-09-11T09:50:26.088 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_class_load_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-09-11T09:50:26.088 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_class_update_on_start": "true", 2024-09-11T09:50:26.088 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_client_message_cap": "256", 2024-09-11T09:50:26.088 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_client_message_size_cap": "524288000", 2024-09-11T09:50:26.088 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_client_op_priority": "63", 2024-09-11T09:50:26.088 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_client_watch_timeout": "30", 2024-09-11T09:50:26.088 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_command_max_records": "256", 2024-09-11T09:50:26.088 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_compact_on_start": "false", 2024-09-11T09:50:26.089 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_copyfrom_max_chunk": "8388608", 2024-09-11T09:50:26.089 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_crush_chooseleaf_type": "0", 2024-09-11T09:50:26.089 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_crush_initial_weight": "-1.000000", 2024-09-11T09:50:26.089 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_crush_update_on_start": "true", 2024-09-11T09:50:26.089 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_crush_update_weight_set": "true", 2024-09-11T09:50:26.089 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_data": "/var/lib/ceph/osd/ceph-a", 2024-09-11T09:50:26.089 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_debug_crash_on_ignored_backoff": "false", 2024-09-11T09:50:26.089 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_debug_deep_scrub_sleep": "0.000000", 2024-09-11T09:50:26.089 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_debug_drop_ping_duration": "0", 2024-09-11T09:50:26.089 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_debug_drop_ping_probability": "0.000000", 2024-09-11T09:50:26.089 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_debug_feed_pullee": "-1", 2024-09-11T09:50:26.089 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_debug_inject_copyfrom_error": "false", 2024-09-11T09:50:26.089 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_debug_inject_dispatch_delay_duration": "0.100000", 2024-09-11T09:50:26.089 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_debug_inject_dispatch_delay_probability": "0.000000", 2024-09-11T09:50:26.090 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_debug_misdirected_ops": "false", 2024-09-11T09:50:26.090 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_debug_no_acting_change": "false", 2024-09-11T09:50:26.090 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_debug_no_purge_strays": "false", 2024-09-11T09:50:26.090 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_debug_op_order": "false", 2024-09-11T09:50:26.090 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_debug_pg_log_writeout": "false", 2024-09-11T09:50:26.090 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_debug_pretend_recovery_active": "false", 2024-09-11T09:50:26.090 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_debug_random_push_read_error": "0.000000", 2024-09-11T09:50:26.090 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_debug_reject_backfill_probability": "0.000000", 2024-09-11T09:50:26.090 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_debug_shutdown": "false", 2024-09-11T09:50:26.090 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_debug_skip_full_check_in_backfill_reservation": "false", 2024-09-11T09:50:26.090 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_debug_skip_full_check_in_recovery": "false", 2024-09-11T09:50:26.090 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_debug_trim_objects": "false", 2024-09-11T09:50:26.090 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_debug_verify_cached_snaps": "false", 2024-09-11T09:50:26.090 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_debug_verify_missing_on_start": "false", 2024-09-11T09:50:26.091 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_debug_verify_snaps": "false", 2024-09-11T09:50:26.091 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_debug_verify_stray_on_activate": "false", 2024-09-11T09:50:26.091 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_deep_scrub_interval": "604800.000000", 2024-09-11T09:50:26.091 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_deep_scrub_keys": "1024", 2024-09-11T09:50:26.091 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_deep_scrub_large_omap_object_key_threshold": "200000", 2024-09-11T09:50:26.091 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_deep_scrub_large_omap_object_value_sum_threshold": "1073741824", 2024-09-11T09:50:26.091 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_deep_scrub_randomize_ratio": "0.150000", 2024-09-11T09:50:26.091 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_deep_scrub_stride": "524288", 2024-09-11T09:50:26.091 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_deep_scrub_update_digest_min_age": "7200", 2024-09-11T09:50:26.091 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_default_data_pool_replay_window": "5", 2024-09-11T09:50:26.091 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_default_notify_timeout": "30", 2024-09-11T09:50:26.091 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_delete_sleep": "0.000000", 2024-09-11T09:50:26.092 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_delete_sleep_hdd": "5.000000", 2024-09-11T09:50:26.092 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_delete_sleep_hybrid": "1.000000", 2024-09-11T09:50:26.092 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_delete_sleep_ssd": "1.000000", 2024-09-11T09:50:26.092 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_discard_disconnected_ops": "true", 2024-09-11T09:50:26.092 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_enable_op_tracker": "true", 2024-09-11T09:50:26.092 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_erasure_code_plugins": "jerasure lrc", 2024-09-11T09:50:26.092 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_extblkdev_plugins": "vdo", 2024-09-11T09:50:26.092 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_failsafe_full_ratio": "0.970000", 2024-09-11T09:50:26.092 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_fast_fail_on_connection_refused": "true", 2024-09-11T09:50:26.092 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_fast_info": "true", 2024-09-11T09:50:26.092 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_fast_shutdown": "true", 2024-09-11T09:50:26.092 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_fast_shutdown_notify_mon": "true", 2024-09-11T09:50:26.092 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_fast_shutdown_timeout": "15", 2024-09-11T09:50:26.092 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_find_best_info_ignore_history_les": "false", 2024-09-11T09:50:26.093 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_force_auth_primary_missing_objects": "100", 2024-09-11T09:50:26.093 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_force_recovery_pg_log_entries_factor": "1.300000", 2024-09-11T09:50:26.093 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_function_tracing": "false", 2024-09-11T09:50:26.093 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_heartbeat_grace": "20", 2024-09-11T09:50:26.093 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_heartbeat_interval": "6", 2024-09-11T09:50:26.093 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_heartbeat_min_healthy_ratio": "0.330000", 2024-09-11T09:50:26.093 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_heartbeat_min_peers": "10", 2024-09-11T09:50:26.093 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_heartbeat_min_size": "2000", 2024-09-11T09:50:26.093 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_heartbeat_stale": "600", 2024-09-11T09:50:26.093 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_heartbeat_use_min_delay_socket": "false", 2024-09-11T09:50:26.093 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_hit_set_max_size": "100000", 2024-09-11T09:50:26.093 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_hit_set_min_size": "1000", 2024-09-11T09:50:26.093 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_hit_set_namespace": ".ceph-internal", 2024-09-11T09:50:26.094 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_ignore_stale_divergent_priors": "false", 2024-09-11T09:50:26.094 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_inject_bad_map_crc_probability": "0.000000", 2024-09-11T09:50:26.094 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_inject_failure_on_pg_removal": "false", 2024-09-11T09:50:26.094 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_journal": "/var/lib/ceph/osd/ceph-a/journal", 2024-09-11T09:50:26.094 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_journal_flush_on_shutdown": "true", 2024-09-11T09:50:26.094 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_journal_size": "5120", 2024-09-11T09:50:26.094 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_kill_backfill_at": "0", 2024-09-11T09:50:26.094 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_loop_before_reset_tphandle": "64", 2024-09-11T09:50:26.094 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_map_cache_size": "50", 2024-09-11T09:50:26.094 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_map_dedup": "true", 2024-09-11T09:50:26.094 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_map_message_max": "40", 2024-09-11T09:50:26.094 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_map_message_max_bytes": "10485760", 2024-09-11T09:50:26.094 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_map_share_max_epochs": "40", 2024-09-11T09:50:26.094 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_max_attr_name_len": "100", 2024-09-11T09:50:26.095 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_max_attr_size": "0", 2024-09-11T09:50:26.095 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_max_backfills": "1", 2024-09-11T09:50:26.095 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_max_markdown_count": "5", 2024-09-11T09:50:26.095 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_max_markdown_period": "600", 2024-09-11T09:50:26.095 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_max_object_name_len": "2048", 2024-09-11T09:50:26.095 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_max_object_namespace_len": "256", 2024-09-11T09:50:26.095 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_max_object_size": "134217728", 2024-09-11T09:50:26.095 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_max_omap_bytes_per_request": "1073741824", 2024-09-11T09:50:26.095 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_max_omap_entries_per_request": "1024", 2024-09-11T09:50:26.095 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_max_pg_blocked_by": "16", 2024-09-11T09:50:26.095 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_max_pg_log_entries": "10000", 2024-09-11T09:50:26.095 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_max_pg_per_osd_hard_ratio": "3.000000", 2024-09-11T09:50:26.095 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_max_pgls": "1024", 2024-09-11T09:50:26.095 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_max_push_cost": "8388608", 2024-09-11T09:50:26.095 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_max_push_objects": "10", 2024-09-11T09:50:26.096 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_max_scrubs": "3", 2024-09-11T09:50:26.096 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_max_snap_prune_intervals_per_epoch": "512", 2024-09-11T09:50:26.096 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_max_trimming_pgs": "2", 2024-09-11T09:50:26.096 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_max_write_op_reply_len": "64", 2024-09-11T09:50:26.096 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_max_write_size": "90", 2024-09-11T09:50:26.096 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_mclock_force_run_benchmark_on_init": "false", 2024-09-11T09:50:26.096 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_mclock_iops_capacity_threshold_hdd": "500.000000", 2024-09-11T09:50:26.096 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_mclock_iops_capacity_threshold_ssd": "80000.000000", 2024-09-11T09:50:26.096 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_mclock_max_capacity_iops_hdd": "315.000000", 2024-09-11T09:50:26.096 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_mclock_max_capacity_iops_ssd": "21500.000000", 2024-09-11T09:50:26.096 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_mclock_max_sequential_bandwidth_hdd": "157286400", 2024-09-11T09:50:26.096 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_mclock_max_sequential_bandwidth_ssd": "1258291200", 2024-09-11T09:50:26.096 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_mclock_override_recovery_settings": "false", 2024-09-11T09:50:26.096 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_mclock_profile": "balanced", 2024-09-11T09:50:26.097 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_mclock_scheduler_anticipation_timeout": "0.000000", 2024-09-11T09:50:26.097 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_mclock_scheduler_background_best_effort_lim": "0.000000", 2024-09-11T09:50:26.097 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_mclock_scheduler_background_best_effort_res": "0.000000", 2024-09-11T09:50:26.097 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_mclock_scheduler_background_best_effort_wgt": "1", 2024-09-11T09:50:26.097 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_mclock_scheduler_background_recovery_lim": "0.000000", 2024-09-11T09:50:26.097 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_mclock_scheduler_background_recovery_res": "0.000000", 2024-09-11T09:50:26.097 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_mclock_scheduler_background_recovery_wgt": "1", 2024-09-11T09:50:26.097 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_mclock_scheduler_client_lim": "0.000000", 2024-09-11T09:50:26.097 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_mclock_scheduler_client_res": "0.000000", 2024-09-11T09:50:26.097 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_mclock_scheduler_client_wgt": "1", 2024-09-11T09:50:26.097 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_mclock_skip_benchmark": "false", 2024-09-11T09:50:26.097 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_memory_base": "805306368", 2024-09-11T09:50:26.097 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_memory_cache_min": "134217728", 2024-09-11T09:50:26.097 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_memory_cache_resize_interval": "1.000000", 2024-09-11T09:50:26.098 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_memory_expected_fragmentation": "0.150000", 2024-09-11T09:50:26.098 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_memory_target": "4294967296", 2024-09-11T09:50:26.098 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_memory_target_autotune": "false", 2024-09-11T09:50:26.098 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_memory_target_cgroup_limit_ratio": "0.800000", 2024-09-11T09:50:26.098 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_min_pg_log_entries": "250", 2024-09-11T09:50:26.098 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_min_recovery_priority": "0", 2024-09-11T09:50:26.098 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_mon_heartbeat_interval": "30", 2024-09-11T09:50:26.098 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_mon_heartbeat_stat_stale": "3600", 2024-09-11T09:50:26.098 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_mon_report_interval": "5", 2024-09-11T09:50:26.098 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_mon_report_max_in_flight": "2", 2024-09-11T09:50:26.098 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_mon_shutdown_timeout": "5.000000", 2024-09-11T09:50:26.098 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_num_cache_shards": "32", 2024-09-11T09:50:26.098 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_num_op_tracker_shard": "32", 2024-09-11T09:50:26.098 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_numa_auto_affinity": "true", 2024-09-11T09:50:26.098 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_numa_node": "-1", 2024-09-11T09:50:26.099 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_numa_prefer_iface": "true", 2024-09-11T09:50:26.099 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_object_clean_region_max_num_intervals": "10", 2024-09-11T09:50:26.099 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_objecter_finishers": "1", 2024-09-11T09:50:26.099 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_objectstore": "bluestore", 2024-09-11T09:50:26.099 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_objectstore_fuse": "false", 2024-09-11T09:50:26.099 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_objectstore_tracing": "false", 2024-09-11T09:50:26.099 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_op_complaint_time": "30.000000", 2024-09-11T09:50:26.099 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_op_history_duration": "600", 2024-09-11T09:50:26.099 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_op_history_size": "20", 2024-09-11T09:50:26.099 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_op_history_slow_op_size": "20", 2024-09-11T09:50:26.099 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_op_history_slow_op_threshold": "10.000000", 2024-09-11T09:50:26.099 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_op_log_threshold": "5", 2024-09-11T09:50:26.099 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_op_num_shards": "0", 2024-09-11T09:50:26.099 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_op_num_shards_hdd": "5", 2024-09-11T09:50:26.100 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_op_num_shards_ssd": "8", 2024-09-11T09:50:26.100 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_op_num_threads_per_shard": "0", 2024-09-11T09:50:26.100 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_op_num_threads_per_shard_hdd": "1", 2024-09-11T09:50:26.100 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_op_num_threads_per_shard_ssd": "2", 2024-09-11T09:50:26.100 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_op_pq_max_tokens_per_priority": "4194304", 2024-09-11T09:50:26.100 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_op_pq_min_cost": "65536", 2024-09-11T09:50:26.100 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_op_queue": "mclock_scheduler", 2024-09-11T09:50:26.100 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_op_queue_cut_off": "high", 2024-09-11T09:50:26.100 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_op_thread_suicide_timeout": "150", 2024-09-11T09:50:26.100 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_op_thread_timeout": "15", 2024-09-11T09:50:26.100 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_open_classes_on_start": "true", 2024-09-11T09:50:26.100 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_os_flags": "0", 2024-09-11T09:50:26.100 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_peering_op_priority": "255", 2024-09-11T09:50:26.100 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_pg_delete_cost": "1048576", 2024-09-11T09:50:26.100 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_pg_delete_priority": "5", 2024-09-11T09:50:26.100 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_pg_epoch_max_lag_factor": "2.000000", 2024-09-11T09:50:26.101 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_pg_epoch_persisted_max_stale": "40", 2024-09-11T09:50:26.101 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_pg_log_dups_tracked": "3000", 2024-09-11T09:50:26.101 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_pg_log_trim_max": "10000", 2024-09-11T09:50:26.101 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_pg_log_trim_min": "100", 2024-09-11T09:50:26.101 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_pg_max_concurrent_snap_trims": "2", 2024-09-11T09:50:26.101 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_pg_object_context_cache_count": "64", 2024-09-11T09:50:26.101 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_pg_stat_report_interval_max_epochs": "500", 2024-09-11T09:50:26.101 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_pg_stat_report_interval_max_seconds": "5", 2024-09-11T09:50:26.101 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_pool_default_cache_max_evict_check_size": "10", 2024-09-11T09:50:26.101 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_pool_default_cache_min_evict_age": "0", 2024-09-11T09:50:26.101 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_pool_default_cache_min_flush_age": "0", 2024-09-11T09:50:26.101 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_pool_default_cache_target_dirty_high_ratio": "0.600000", 2024-09-11T09:50:26.101 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_pool_default_cache_target_dirty_ratio": "0.400000", 2024-09-11T09:50:26.101 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_pool_default_cache_target_full_ratio": "0.800000", 2024-09-11T09:50:26.102 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_pool_default_crimson": "false", 2024-09-11T09:50:26.102 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_pool_default_crush_rule": "-1", 2024-09-11T09:50:26.102 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_pool_default_ec_fast_read": "false", 2024-09-11T09:50:26.102 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_pool_default_erasure_code_profile": "plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd", 2024-09-11T09:50:26.102 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_pool_default_flag_bulk": "false", 2024-09-11T09:50:26.102 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_pool_default_flag_hashpspool": "true", 2024-09-11T09:50:26.102 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_pool_default_flag_nodelete": "false", 2024-09-11T09:50:26.102 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_pool_default_flag_nopgchange": "false", 2024-09-11T09:50:26.102 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_pool_default_flag_nosizechange": "false", 2024-09-11T09:50:26.102 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_pool_default_flags": "0", 2024-09-11T09:50:26.102 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_pool_default_hit_set_bloom_fpp": "0.050000", 2024-09-11T09:50:26.102 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_pool_default_min_size": "0", 2024-09-11T09:50:26.103 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_pool_default_pg_autoscale_mode": "off", 2024-09-11T09:50:26.103 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_pool_default_pg_num": "32", 2024-09-11T09:50:26.103 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_pool_default_pgp_num": "0", 2024-09-11T09:50:26.103 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_pool_default_read_lease_ratio": "0.800000", 2024-09-11T09:50:26.103 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_pool_default_read_ratio": "70", 2024-09-11T09:50:26.103 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_pool_default_size": "2", 2024-09-11T09:50:26.103 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_pool_default_type": "replicated", 2024-09-11T09:50:26.103 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_pool_erasure_code_stripe_unit": "4096", 2024-09-11T09:50:26.103 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_pool_use_gmt_hitset": "true", 2024-09-11T09:50:26.103 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_push_per_object_cost": "1000", 2024-09-11T09:50:26.103 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_read_ec_check_for_errors": "false", 2024-09-11T09:50:26.103 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_recover_clone_overlap": "true", 2024-09-11T09:50:26.103 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_recover_clone_overlap_limit": "10", 2024-09-11T09:50:26.103 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_recovery_cost": "20971520", 2024-09-11T09:50:26.103 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_recovery_delay_start": "0.000000", 2024-09-11T09:50:26.104 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_recovery_max_active": "0", 2024-09-11T09:50:26.104 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_recovery_max_active_hdd": "3", 2024-09-11T09:50:26.104 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_recovery_max_active_ssd": "10", 2024-09-11T09:50:26.104 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_recovery_max_chunk": "8388608", 2024-09-11T09:50:26.104 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_recovery_max_omap_entries_per_chunk": "8096", 2024-09-11T09:50:26.104 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_recovery_max_single_start": "1", 2024-09-11T09:50:26.104 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_recovery_op_priority": "3", 2024-09-11T09:50:26.104 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_recovery_op_warn_multiple": "16", 2024-09-11T09:50:26.104 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_recovery_priority": "5", 2024-09-11T09:50:26.104 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_recovery_retry_interval": "30.000000", 2024-09-11T09:50:26.104 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_recovery_sleep": "0.000000", 2024-09-11T09:50:26.104 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_recovery_sleep_hdd": "0.100000", 2024-09-11T09:50:26.104 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_recovery_sleep_hybrid": "0.025000", 2024-09-11T09:50:26.104 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_recovery_sleep_ssd": "0.000000", 2024-09-11T09:50:26.104 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_repair_during_recovery": "false", 2024-09-11T09:50:26.105 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_requested_scrub_priority": "120", 2024-09-11T09:50:26.105 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_rocksdb_iterator_bounds_enabled": "true", 2024-09-11T09:50:26.105 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_rollback_to_cluster_snap": "", 2024-09-11T09:50:26.105 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_scrub_auto_repair": "false", 2024-09-11T09:50:26.105 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_scrub_auto_repair_num_errors": "5", 2024-09-11T09:50:26.105 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_scrub_backoff_ratio": "0.660000", 2024-09-11T09:50:26.105 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_scrub_begin_hour": "0", 2024-09-11T09:50:26.105 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_scrub_begin_week_day": "0", 2024-09-11T09:50:26.105 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_scrub_chunk_max": "25", 2024-09-11T09:50:26.105 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_scrub_chunk_min": "5", 2024-09-11T09:50:26.105 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_scrub_cost": "52428800", 2024-09-11T09:50:26.105 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_scrub_disable_reservation_queuing": "false", 2024-09-11T09:50:26.105 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_scrub_during_recovery": "false", 2024-09-11T09:50:26.105 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_scrub_end_hour": "0", 2024-09-11T09:50:26.105 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_scrub_end_week_day": "0", 2024-09-11T09:50:26.106 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_scrub_event_cost": "4096", 2024-09-11T09:50:26.106 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_scrub_extended_sleep": "0.000000", 2024-09-11T09:50:26.106 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_scrub_interval_randomize_ratio": "0.500000", 2024-09-11T09:50:26.106 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_scrub_invalid_stats": "true", 2024-09-11T09:50:26.106 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_scrub_load_threshold": "0.500000", 2024-09-11T09:50:26.106 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_scrub_max_interval": "604800.000000", 2024-09-11T09:50:26.106 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_scrub_max_preemptions": "5", 2024-09-11T09:50:26.106 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_scrub_min_interval": "86400.000000", 2024-09-11T09:50:26.106 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_scrub_priority": "5", 2024-09-11T09:50:26.106 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_scrub_sleep": "0.000000", 2024-09-11T09:50:26.106 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_shallow_scrub_chunk_max": "100", 2024-09-11T09:50:26.106 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_shallow_scrub_chunk_min": "50", 2024-09-11T09:50:26.106 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_shutdown_pgref_assert": "false", 2024-09-11T09:50:26.106 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_skip_data_digest": "false", 2024-09-11T09:50:26.106 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_smart_report_timeout": "5", 2024-09-11T09:50:26.107 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_snap_trim_cost": "1048576", 2024-09-11T09:50:26.107 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_snap_trim_priority": "5", 2024-09-11T09:50:26.107 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_snap_trim_sleep": "0.000000", 2024-09-11T09:50:26.107 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_snap_trim_sleep_hdd": "5.000000", 2024-09-11T09:50:26.107 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_snap_trim_sleep_hybrid": "2.000000", 2024-09-11T09:50:26.107 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_snap_trim_sleep_ssd": "0.000000", 2024-09-11T09:50:26.107 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_stats_update_period_not_scrubbing": "120", 2024-09-11T09:50:26.107 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_stats_update_period_scrubbing": "15", 2024-09-11T09:50:26.107 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_target_pg_log_entries_per_osd": "300000", 2024-09-11T09:50:26.107 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_target_transaction_size": "30", 2024-09-11T09:50:26.107 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_tier_default_cache_hit_set_count": "4", 2024-09-11T09:50:26.107 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_tier_default_cache_hit_set_grade_decay_rate": "20", 2024-09-11T09:50:26.107 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_tier_default_cache_hit_set_period": "1200", 2024-09-11T09:50:26.107 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_tier_default_cache_hit_set_search_last_n": "1", 2024-09-11T09:50:26.107 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_tier_default_cache_hit_set_type": "bloom", 2024-09-11T09:50:26.108 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_tier_default_cache_min_read_recency_for_promote": "1", 2024-09-11T09:50:26.108 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_tier_default_cache_min_write_recency_for_promote": "1", 2024-09-11T09:50:26.108 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_tier_default_cache_mode": "writeback", 2024-09-11T09:50:26.108 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_tier_promote_max_bytes_sec": "5242880", 2024-09-11T09:50:26.108 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_tier_promote_max_objects_sec": "25", 2024-09-11T09:50:26.108 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_tracing": "false", 2024-09-11T09:50:26.108 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_use_stale_snap": "false", 2024-09-11T09:50:26.108 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_uuid": "00000000-0000-0000-0000-000000000000", 2024-09-11T09:50:26.108 INFO:teuthology.orchestra.run.smithi017.stdout: "osdc_blkin_trace_all": "false", 2024-09-11T09:50:26.108 INFO:teuthology.orchestra.run.smithi017.stdout: "paxos_kill_at": "0", 2024-09-11T09:50:26.108 INFO:teuthology.orchestra.run.smithi017.stdout: "paxos_max_join_drift": "10", 2024-09-11T09:50:26.108 INFO:teuthology.orchestra.run.smithi017.stdout: "paxos_min": "500", 2024-09-11T09:50:26.108 INFO:teuthology.orchestra.run.smithi017.stdout: "paxos_min_wait": "0.050000", 2024-09-11T09:50:26.108 INFO:teuthology.orchestra.run.smithi017.stdout: "paxos_propose_interval": "1.000000", 2024-09-11T09:50:26.108 INFO:teuthology.orchestra.run.smithi017.stdout: "paxos_service_trim_max": "500", 2024-09-11T09:50:26.109 INFO:teuthology.orchestra.run.smithi017.stdout: "paxos_service_trim_max_multiplier": "20", 2024-09-11T09:50:26.109 INFO:teuthology.orchestra.run.smithi017.stdout: "paxos_service_trim_min": "250", 2024-09-11T09:50:26.109 INFO:teuthology.orchestra.run.smithi017.stdout: "paxos_stash_full_interval": "25", 2024-09-11T09:50:26.109 INFO:teuthology.orchestra.run.smithi017.stdout: "paxos_trim_max": "500", 2024-09-11T09:50:26.109 INFO:teuthology.orchestra.run.smithi017.stdout: "paxos_trim_min": "250", 2024-09-11T09:50:26.109 INFO:teuthology.orchestra.run.smithi017.stdout: "perf": "true", 2024-09-11T09:50:26.109 INFO:teuthology.orchestra.run.smithi017.stdout: "pid_file": "/var/run/ceph/ceph-mon.a.pid", 2024-09-11T09:50:26.109 INFO:teuthology.orchestra.run.smithi017.stdout: "plugin_crypto_accelerator": "crypto_isal", 2024-09-11T09:50:26.109 INFO:teuthology.orchestra.run.smithi017.stdout: "plugin_dir": "/usr/lib64/ceph", 2024-09-11T09:50:26.109 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "-", 2024-09-11T09:50:26.109 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrv": "", 2024-09-11T09:50:26.109 INFO:teuthology.orchestra.run.smithi017.stdout: "public_bind_addr": "-", 2024-09-11T09:50:26.109 INFO:teuthology.orchestra.run.smithi017.stdout: "public_network": "", 2024-09-11T09:50:26.109 INFO:teuthology.orchestra.run.smithi017.stdout: "public_network_interface": "", 2024-09-11T09:50:26.109 INFO:teuthology.orchestra.run.smithi017.stdout: "qat_compressor_busy_polling": "false", 2024-09-11T09:50:26.110 INFO:teuthology.orchestra.run.smithi017.stdout: "qat_compressor_enabled": "false", 2024-09-11T09:50:26.110 INFO:teuthology.orchestra.run.smithi017.stdout: "qat_compressor_session_max_number": "256", 2024-09-11T09:50:26.110 INFO:teuthology.orchestra.run.smithi017.stdout: "rados_mon_op_timeout": "0", 2024-09-11T09:50:26.110 INFO:teuthology.orchestra.run.smithi017.stdout: "rados_osd_op_timeout": "0", 2024-09-11T09:50:26.110 INFO:teuthology.orchestra.run.smithi017.stdout: "rados_tracing": "false", 2024-09-11T09:50:26.110 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_atime_update_interval": "60", 2024-09-11T09:50:26.110 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_auto_exclusive_lock_until_manual_request": "true", 2024-09-11T09:50:26.110 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_balance_parent_reads": "false", 2024-09-11T09:50:26.110 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_balance_snap_reads": "false", 2024-09-11T09:50:26.110 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_blkin_trace_all": "false", 2024-09-11T09:50:26.110 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_blocklist_expire_seconds": "0", 2024-09-11T09:50:26.110 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_blocklist_on_break_lock": "true", 2024-09-11T09:50:26.110 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_cache": "true", 2024-09-11T09:50:26.110 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_cache_block_writes_upfront": "false", 2024-09-11T09:50:26.110 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_cache_max_dirty": "25165824", 2024-09-11T09:50:26.111 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_cache_max_dirty_age": "1.000000", 2024-09-11T09:50:26.111 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_cache_max_dirty_object": "0", 2024-09-11T09:50:26.111 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_cache_policy": "writearound", 2024-09-11T09:50:26.111 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_cache_size": "33554432", 2024-09-11T09:50:26.111 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_cache_target_dirty": "16777216", 2024-09-11T09:50:26.111 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_cache_writethrough_until_flush": "true", 2024-09-11T09:50:26.111 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_clone_copy_on_read": "false", 2024-09-11T09:50:26.111 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_compression_hint": "none", 2024-09-11T09:50:26.111 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_concurrent_management_ops": "10", 2024-09-11T09:50:26.111 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_config_pool_override_update_timestamp": "0", 2024-09-11T09:50:26.111 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_default_clone_format": "auto", 2024-09-11T09:50:26.111 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_default_data_pool": "", 2024-09-11T09:50:26.111 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_default_features": "61", 2024-09-11T09:50:26.111 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_default_format": "2", 2024-09-11T09:50:26.111 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_default_map_options": "", 2024-09-11T09:50:26.112 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_default_order": "22", 2024-09-11T09:50:26.112 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_default_pool": "rbd", 2024-09-11T09:50:26.112 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_default_snapshot_quiesce_mode": "required", 2024-09-11T09:50:26.112 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_default_stripe_count": "0", 2024-09-11T09:50:26.112 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_default_stripe_unit": "0", 2024-09-11T09:50:26.112 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_disable_zero_copy_writes": "true", 2024-09-11T09:50:26.112 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_discard_granularity_bytes": "65536", 2024-09-11T09:50:26.112 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_discard_on_zeroed_write_same": "true", 2024-09-11T09:50:26.112 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_enable_alloc_hint": "true", 2024-09-11T09:50:26.112 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_invalidate_object_map_on_timeout": "true", 2024-09-11T09:50:26.112 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_io_scheduler": "simple", 2024-09-11T09:50:26.112 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_io_scheduler_simple_max_delay": "0", 2024-09-11T09:50:26.112 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_journal_commit_age": "5.000000", 2024-09-11T09:50:26.112 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_journal_max_concurrent_object_sets": "0", 2024-09-11T09:50:26.113 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_journal_max_payload_bytes": "16384", 2024-09-11T09:50:26.113 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_journal_object_flush_age": "0.000000", 2024-09-11T09:50:26.113 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_journal_object_flush_bytes": "1048576", 2024-09-11T09:50:26.113 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_journal_object_flush_interval": "0", 2024-09-11T09:50:26.113 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_journal_object_max_in_flight_appends": "0", 2024-09-11T09:50:26.113 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_journal_object_writethrough_until_flush": "true", 2024-09-11T09:50:26.113 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_journal_order": "24", 2024-09-11T09:50:26.113 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_journal_pool": "", 2024-09-11T09:50:26.113 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_journal_splay_width": "4", 2024-09-11T09:50:26.113 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_localize_parent_reads": "false", 2024-09-11T09:50:26.113 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_localize_snap_reads": "false", 2024-09-11T09:50:26.114 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_mirror_concurrent_image_deletions": "1", 2024-09-11T09:50:26.114 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_mirror_concurrent_image_syncs": "5", 2024-09-11T09:50:26.114 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_mirror_delete_retry_interval": "30.000000", 2024-09-11T09:50:26.114 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_mirror_image_perf_stats_prio": "5", 2024-09-11T09:50:26.114 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_mirror_image_policy_migration_throttle": "300", 2024-09-11T09:50:26.114 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_mirror_image_policy_rebalance_timeout": "0.000000", 2024-09-11T09:50:26.114 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_mirror_image_policy_type": "simple", 2024-09-11T09:50:26.114 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_mirror_image_policy_update_throttle_interval": "1.000000", 2024-09-11T09:50:26.114 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_mirror_image_state_check_interval": "30", 2024-09-11T09:50:26.114 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_mirror_journal_commit_age": "5.000000", 2024-09-11T09:50:26.114 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_mirror_journal_poll_age": "5.000000", 2024-09-11T09:50:26.114 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_mirror_leader_heartbeat_interval": "5", 2024-09-11T09:50:26.114 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_mirror_leader_max_acquire_attempts_before_break": "3", 2024-09-11T09:50:26.115 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_mirror_leader_max_missed_heartbeats": "2", 2024-09-11T09:50:26.115 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_mirror_memory_autotune": "true", 2024-09-11T09:50:26.115 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_mirror_memory_base": "805306368", 2024-09-11T09:50:26.115 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_mirror_memory_cache_autotune_interval": "30.000000", 2024-09-11T09:50:26.115 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_mirror_memory_cache_min": "134217728", 2024-09-11T09:50:26.115 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_mirror_memory_cache_resize_interval": "5.000000", 2024-09-11T09:50:26.115 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_mirror_memory_expected_fragmentation": "0.150000", 2024-09-11T09:50:26.115 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_mirror_memory_target": "4294967296", 2024-09-11T09:50:26.115 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_mirror_perf_stats_prio": "5", 2024-09-11T09:50:26.115 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_mirror_pool_replayers_refresh_interval": "30", 2024-09-11T09:50:26.115 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_mirror_sync_point_update_age": "30.000000", 2024-09-11T09:50:26.115 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_mirroring_delete_delay": "0", 2024-09-11T09:50:26.115 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_mirroring_max_mirroring_snapshots": "5", 2024-09-11T09:50:26.115 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_mirroring_replay_delay": "0", 2024-09-11T09:50:26.116 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_mirroring_resync_after_disconnect": "false", 2024-09-11T09:50:26.116 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_move_parent_to_trash_on_remove": "false", 2024-09-11T09:50:26.116 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_move_to_trash_on_remove": "false", 2024-09-11T09:50:26.116 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_move_to_trash_on_remove_expire_seconds": "0", 2024-09-11T09:50:26.116 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_mtime_update_interval": "60", 2024-09-11T09:50:26.116 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_non_blocking_aio": "true", 2024-09-11T09:50:26.116 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_op_thread_timeout": "60", 2024-09-11T09:50:26.116 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_op_threads": "1", 2024-09-11T09:50:26.116 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_parent_cache_enabled": "false", 2024-09-11T09:50:26.116 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_persistent_cache_mode": "disabled", 2024-09-11T09:50:26.116 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_persistent_cache_path": "/tmp", 2024-09-11T09:50:26.116 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_persistent_cache_size": "1073741824", 2024-09-11T09:50:26.116 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_plugins": "", 2024-09-11T09:50:26.116 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_qos_bps_burst": "0", 2024-09-11T09:50:26.117 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_qos_bps_burst_seconds": "1", 2024-09-11T09:50:26.117 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_qos_bps_limit": "0", 2024-09-11T09:50:26.117 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_qos_exclude_ops": "0", 2024-09-11T09:50:26.117 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_qos_iops_burst": "0", 2024-09-11T09:50:26.117 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_qos_iops_burst_seconds": "1", 2024-09-11T09:50:26.117 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_qos_iops_limit": "0", 2024-09-11T09:50:26.117 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_qos_read_bps_burst": "0", 2024-09-11T09:50:26.117 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_qos_read_bps_burst_seconds": "1", 2024-09-11T09:50:26.117 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_qos_read_bps_limit": "0", 2024-09-11T09:50:26.117 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_qos_read_iops_burst": "0", 2024-09-11T09:50:26.117 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_qos_read_iops_burst_seconds": "1", 2024-09-11T09:50:26.117 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_qos_read_iops_limit": "0", 2024-09-11T09:50:26.117 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_qos_schedule_tick_min": "50", 2024-09-11T09:50:26.118 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_qos_write_bps_burst": "0", 2024-09-11T09:50:26.118 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_qos_write_bps_burst_seconds": "1", 2024-09-11T09:50:26.118 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_qos_write_bps_limit": "0", 2024-09-11T09:50:26.118 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_qos_write_iops_burst": "0", 2024-09-11T09:50:26.118 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_qos_write_iops_burst_seconds": "1", 2024-09-11T09:50:26.118 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_qos_write_iops_limit": "0", 2024-09-11T09:50:26.118 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_quiesce_notification_attempts": "10", 2024-09-11T09:50:26.118 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_read_from_replica_policy": "default", 2024-09-11T09:50:26.118 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_readahead_disable_after_bytes": "52428800", 2024-09-11T09:50:26.118 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_readahead_max_bytes": "524288", 2024-09-11T09:50:26.118 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_readahead_trigger_requests": "10", 2024-09-11T09:50:26.118 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_request_timed_out_seconds": "30", 2024-09-11T09:50:26.118 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_skip_partial_discard": "true", 2024-09-11T09:50:26.118 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_sparse_read_threshold_bytes": "65536", 2024-09-11T09:50:26.119 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_tracing": "false", 2024-09-11T09:50:26.119 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_validate_names": "true", 2024-09-11T09:50:26.119 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd_validate_pool": "true", 2024-09-11T09:50:26.119 INFO:teuthology.orchestra.run.smithi017.stdout: "restapi_base_url": "", 2024-09-11T09:50:26.119 INFO:teuthology.orchestra.run.smithi017.stdout: "restapi_log_level": "", 2024-09-11T09:50:26.119 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_account_default_quota_max_objects": "-1", 2024-09-11T09:50:26.119 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_account_default_quota_max_size": "-1", 2024-09-11T09:50:26.119 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_acl_grants_max_num": "100", 2024-09-11T09:50:26.119 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_admin_entry": "admin", 2024-09-11T09:50:26.119 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_allow_notification_secrets_in_cleartext": "false", 2024-09-11T09:50:26.119 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_backend_store": "rados", 2024-09-11T09:50:26.119 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_barbican_url": "", 2024-09-11T09:50:26.119 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_beast_enable_async": "true", 2024-09-11T09:50:26.119 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_bucket_counters_cache": "false", 2024-09-11T09:50:26.120 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_bucket_counters_cache_size": "10000", 2024-09-11T09:50:26.120 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_bucket_default_quota_max_objects": "-1", 2024-09-11T09:50:26.120 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_bucket_default_quota_max_size": "-1", 2024-09-11T09:50:26.120 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_bucket_index_max_aio": "128", 2024-09-11T09:50:26.120 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_bucket_index_transaction_instrumentation": "false", 2024-09-11T09:50:26.120 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_bucket_quota_cache_size": "10000", 2024-09-11T09:50:26.120 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_bucket_quota_ttl": "600", 2024-09-11T09:50:26.120 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_bucket_sync_spawn_window": "20", 2024-09-11T09:50:26.120 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_cache_enabled": "true", 2024-09-11T09:50:26.120 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_cache_expiry_interval": "900", 2024-09-11T09:50:26.120 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_cache_lru_size": "25000", 2024-09-11T09:50:26.120 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_config_store": "rados", 2024-09-11T09:50:26.120 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_content_length_compat": "false", 2024-09-11T09:50:26.120 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_copy_obj_progress": "true", 2024-09-11T09:50:26.121 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_copy_obj_progress_every_bytes": "1048576", 2024-09-11T09:50:26.121 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_cors_rules_max_num": "100", 2024-09-11T09:50:26.121 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_cross_domain_policy": "", 2024-09-11T09:50:26.121 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_crypt_default_encryption_key": "", 2024-09-11T09:50:26.121 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_crypt_kmip_addr": "", 2024-09-11T09:50:26.121 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_crypt_kmip_ca_path": "", 2024-09-11T09:50:26.121 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_crypt_kmip_client_cert": "", 2024-09-11T09:50:26.121 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_crypt_kmip_client_key": "", 2024-09-11T09:50:26.121 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_crypt_kmip_kms_key_template": "", 2024-09-11T09:50:26.121 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_crypt_kmip_password": "", 2024-09-11T09:50:26.121 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_crypt_kmip_s3_key_template": "$keyid", 2024-09-11T09:50:26.121 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_crypt_kmip_username": "", 2024-09-11T09:50:26.121 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_crypt_require_ssl": "true", 2024-09-11T09:50:26.122 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_crypt_s3_kms_backend": "barbican", 2024-09-11T09:50:26.122 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_crypt_s3_kms_encryption_keys": "", 2024-09-11T09:50:26.122 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_crypt_sse_s3_backend": "vault", 2024-09-11T09:50:26.122 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_crypt_sse_s3_key_template": "%bucket_id", 2024-09-11T09:50:26.122 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_crypt_sse_s3_vault_addr": "", 2024-09-11T09:50:26.122 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_crypt_sse_s3_vault_auth": "token", 2024-09-11T09:50:26.122 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_crypt_sse_s3_vault_namespace": "", 2024-09-11T09:50:26.122 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_crypt_sse_s3_vault_prefix": "", 2024-09-11T09:50:26.122 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_crypt_sse_s3_vault_secret_engine": "transit", 2024-09-11T09:50:26.122 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_crypt_sse_s3_vault_ssl_cacert": "", 2024-09-11T09:50:26.122 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_crypt_sse_s3_vault_ssl_clientcert": "", 2024-09-11T09:50:26.122 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_crypt_sse_s3_vault_ssl_clientkey": "", 2024-09-11T09:50:26.122 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_crypt_sse_s3_vault_token_file": "", 2024-09-11T09:50:26.122 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_crypt_sse_s3_vault_verify_ssl": "true", 2024-09-11T09:50:26.123 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_crypt_suppress_logs": "true", 2024-09-11T09:50:26.123 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_crypt_vault_addr": "", 2024-09-11T09:50:26.123 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_crypt_vault_auth": "token", 2024-09-11T09:50:26.123 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_crypt_vault_namespace": "", 2024-09-11T09:50:26.123 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_crypt_vault_prefix": "", 2024-09-11T09:50:26.123 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_crypt_vault_secret_engine": "transit", 2024-09-11T09:50:26.123 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_crypt_vault_ssl_cacert": "", 2024-09-11T09:50:26.123 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_crypt_vault_ssl_clientcert": "", 2024-09-11T09:50:26.123 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_crypt_vault_ssl_clientkey": "", 2024-09-11T09:50:26.123 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_crypt_vault_token_file": "", 2024-09-11T09:50:26.123 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_crypt_vault_verify_ssl": "true", 2024-09-11T09:50:26.123 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_curl_buffersize": "524288", 2024-09-11T09:50:26.123 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_curl_low_speed_limit": "1024", 2024-09-11T09:50:26.123 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_curl_low_speed_time": "300", 2024-09-11T09:50:26.124 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_curl_tcp_keepalive": "0", 2024-09-11T09:50:26.124 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_curl_wait_timeout_ms": "1000", 2024-09-11T09:50:26.124 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_d3n_l1_datacache_persistent_path": "/tmp/rgw_datacache/", 2024-09-11T09:50:26.124 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_d3n_l1_datacache_size": "1073741824", 2024-09-11T09:50:26.124 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_d3n_l1_evict_cache_on_start": "true", 2024-09-11T09:50:26.124 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_d3n_l1_eviction_policy": "lru", 2024-09-11T09:50:26.124 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_d3n_l1_fadvise": "4", 2024-09-11T09:50:26.124 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_d3n_l1_local_datacache_enabled": "false", 2024-09-11T09:50:26.124 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_d3n_libaio_aio_num": "64", 2024-09-11T09:50:26.124 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_d3n_libaio_aio_threads": "20", 2024-09-11T09:50:26.124 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_d4n_host": "127.0.0.1", 2024-09-11T09:50:26.124 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_d4n_port": "6379", 2024-09-11T09:50:26.125 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_data": "/var/lib/ceph/radosgw/ceph-a", 2024-09-11T09:50:26.125 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_data_log_changes_size": "1000", 2024-09-11T09:50:26.125 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_data_log_num_shards": "128", 2024-09-11T09:50:26.125 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_data_log_obj_prefix": "data_log", 2024-09-11T09:50:26.125 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_data_log_window": "30", 2024-09-11T09:50:26.125 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_data_notify_interval_msec": "0", 2024-09-11T09:50:26.125 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_data_sync_poll_interval": "20", 2024-09-11T09:50:26.125 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_data_sync_spawn_window": "20", 2024-09-11T09:50:26.125 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_debug_inject_latency_bi_unlink": "0", 2024-09-11T09:50:26.125 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_debug_inject_olh_cancel_modification_err": "false", 2024-09-11T09:50:26.125 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_debug_inject_set_olh_err": "0", 2024-09-11T09:50:26.125 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_default_data_log_backing": "fifo", 2024-09-11T09:50:26.126 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_default_realm_info_oid": "default.realm", 2024-09-11T09:50:26.126 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_default_region_info_oid": "default.region", 2024-09-11T09:50:26.126 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_default_zone_info_oid": "default.zone", 2024-09-11T09:50:26.126 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_default_zonegroup_info_oid": "default.zonegroup", 2024-09-11T09:50:26.126 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_defer_to_bucket_acls": "", 2024-09-11T09:50:26.126 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_delete_multi_obj_max_num": "1000", 2024-09-11T09:50:26.126 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_dmclock_admin_lim": "0.000000", 2024-09-11T09:50:26.126 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_dmclock_admin_res": "100.000000", 2024-09-11T09:50:26.126 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_dmclock_admin_wgt": "100.000000", 2024-09-11T09:50:26.126 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_dmclock_auth_lim": "0.000000", 2024-09-11T09:50:26.126 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_dmclock_auth_res": "200.000000", 2024-09-11T09:50:26.126 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_dmclock_auth_wgt": "100.000000", 2024-09-11T09:50:26.126 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_dmclock_data_lim": "0.000000", 2024-09-11T09:50:26.126 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_dmclock_data_res": "500.000000", 2024-09-11T09:50:26.127 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_dmclock_data_wgt": "500.000000", 2024-09-11T09:50:26.127 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_dmclock_metadata_lim": "0.000000", 2024-09-11T09:50:26.127 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_dmclock_metadata_res": "500.000000", 2024-09-11T09:50:26.127 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_dmclock_metadata_wgt": "500.000000", 2024-09-11T09:50:26.127 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_dns_name": "", 2024-09-11T09:50:26.127 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_dns_s3website_name": "", 2024-09-11T09:50:26.127 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_dynamic_resharding": "true", 2024-09-11T09:50:26.127 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_enable_apis": "s3, s3website, swift, swift_auth, admin, sts, iam, notifications", 2024-09-11T09:50:26.127 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_enable_gc_threads": "true", 2024-09-11T09:50:26.127 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_enable_lc_threads": "true", 2024-09-11T09:50:26.127 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_enable_ops_log": "false", 2024-09-11T09:50:26.127 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_enable_quota_threads": "true", 2024-09-11T09:50:26.127 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_enable_static_website": "false", 2024-09-11T09:50:26.127 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_enable_usage_log": "false", 2024-09-11T09:50:26.128 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_enforce_swift_acls": "true", 2024-09-11T09:50:26.128 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_exit_timeout_secs": "120", 2024-09-11T09:50:26.128 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_expose_bucket": "false", 2024-09-11T09:50:26.128 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_extended_http_attrs": "", 2024-09-11T09:50:26.128 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_filter": "none", 2024-09-11T09:50:26.128 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_frontend_defaults": "beast ssl_certificate=config://rgw/cert/$realm/$zone.crt ssl_private_key=config://rgw/cert/$realm/$zone.key", 2024-09-11T09:50:26.128 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_frontends": "beast port=7480", 2024-09-11T09:50:26.128 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_gc_max_concurrent_io": "10", 2024-09-11T09:50:26.128 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_gc_max_deferred": "50", 2024-09-11T09:50:26.128 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_gc_max_deferred_entries_size": "3072", 2024-09-11T09:50:26.128 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_gc_max_objs": "32", 2024-09-11T09:50:26.128 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_gc_max_queue_size": "134213632", 2024-09-11T09:50:26.129 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_gc_max_trim_chunk": "16", 2024-09-11T09:50:26.129 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_gc_obj_min_wait": "7200", 2024-09-11T09:50:26.129 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_gc_processor_max_time": "3600", 2024-09-11T09:50:26.129 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_gc_processor_period": "3600", 2024-09-11T09:50:26.129 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_get_obj_max_req_size": "4194304", 2024-09-11T09:50:26.129 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_get_obj_window_size": "16777216", 2024-09-11T09:50:26.129 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_healthcheck_disabling_path": "", 2024-09-11T09:50:26.129 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_ignore_get_invalid_range": "false", 2024-09-11T09:50:26.129 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_init_timeout": "300", 2024-09-11T09:50:26.129 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_inject_notify_timeout_probability": "0.000000", 2024-09-11T09:50:26.129 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_json_config": "/var/lib/ceph/radosgw/config.json", 2024-09-11T09:50:26.129 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_kafka_connection_idle": "300", 2024-09-11T09:50:26.129 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_kafka_message_timeout": "5000", 2024-09-11T09:50:26.130 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_kafka_sleep_timeout": "10", 2024-09-11T09:50:26.130 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_keystone_accepted_admin_roles": "", 2024-09-11T09:50:26.130 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_keystone_accepted_reader_roles": "", 2024-09-11T09:50:26.130 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_keystone_accepted_roles": "Member, admin", 2024-09-11T09:50:26.130 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_keystone_admin_domain": "", 2024-09-11T09:50:26.130 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_keystone_admin_password": "", 2024-09-11T09:50:26.130 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_keystone_admin_password_path": "", 2024-09-11T09:50:26.130 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_keystone_admin_project": "", 2024-09-11T09:50:26.130 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_keystone_admin_tenant": "", 2024-09-11T09:50:26.130 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_keystone_admin_token": "", 2024-09-11T09:50:26.130 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_keystone_admin_token_path": "", 2024-09-11T09:50:26.130 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_keystone_admin_user": "", 2024-09-11T09:50:26.130 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_keystone_api_version": "2", 2024-09-11T09:50:26.131 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_keystone_barbican_domain": "", 2024-09-11T09:50:26.131 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_keystone_barbican_password": "", 2024-09-11T09:50:26.131 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_keystone_barbican_project": "", 2024-09-11T09:50:26.131 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_keystone_barbican_tenant": "", 2024-09-11T09:50:26.131 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_keystone_barbican_user": "", 2024-09-11T09:50:26.131 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_keystone_expired_token_cache_expiration": "3600", 2024-09-11T09:50:26.131 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_keystone_implicit_tenants": "false", 2024-09-11T09:50:26.131 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_keystone_service_token_accepted_roles": "admin", 2024-09-11T09:50:26.131 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_keystone_service_token_enabled": "false", 2024-09-11T09:50:26.131 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_keystone_token_cache_size": "10000", 2024-09-11T09:50:26.131 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_keystone_url": "", 2024-09-11T09:50:26.131 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_keystone_verify_ssl": "true", 2024-09-11T09:50:26.131 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_lc_debug_interval": "-1", 2024-09-11T09:50:26.132 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_lc_lock_max_time": "90", 2024-09-11T09:50:26.132 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_lc_max_objs": "32", 2024-09-11T09:50:26.132 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_lc_max_rules": "1000", 2024-09-11T09:50:26.132 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_lc_max_worker": "3", 2024-09-11T09:50:26.132 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_lc_max_wp_worker": "3", 2024-09-11T09:50:26.132 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_lc_thread_delay": "0", 2024-09-11T09:50:26.132 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_ldap_binddn": "uid=admin,cn=users,dc=example,dc=com", 2024-09-11T09:50:26.132 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_ldap_dnattr": "uid", 2024-09-11T09:50:26.132 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_ldap_searchdn": "cn=users,cn=accounts,dc=example,dc=com", 2024-09-11T09:50:26.132 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_ldap_searchfilter": "", 2024-09-11T09:50:26.132 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_ldap_secret": "/etc/openldap/secret", 2024-09-11T09:50:26.132 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_ldap_uri": "", 2024-09-11T09:50:26.132 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_lifecycle_work_time": "00:00-06:00", 2024-09-11T09:50:26.133 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_list_bucket_min_readahead": "1000", 2024-09-11T09:50:26.133 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_list_buckets_max_chunk": "1000", 2024-09-11T09:50:26.133 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_log_http_headers": "", 2024-09-11T09:50:26.133 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_log_nonexistent_bucket": "false", 2024-09-11T09:50:26.133 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_log_object_name": "%Y-%m-%d-%H-%i-%n", 2024-09-11T09:50:26.133 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_log_object_name_utc": "false", 2024-09-11T09:50:26.133 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_lua_max_memory_per_state": "128000", 2024-09-11T09:50:26.133 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_luarocks_location": "/tmp/rgw_luarocks/mon.a", 2024-09-11T09:50:26.133 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_max_attr_name_len": "0", 2024-09-11T09:50:26.133 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_max_attr_size": "0", 2024-09-11T09:50:26.133 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_max_attrs_num_in_req": "0", 2024-09-11T09:50:26.133 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_max_chunk_size": "4194304", 2024-09-11T09:50:26.134 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_max_concurrent_requests": "1024", 2024-09-11T09:50:26.134 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_max_copy_obj_concurrent_io": "10", 2024-09-11T09:50:26.134 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_max_dynamic_shards": "1999", 2024-09-11T09:50:26.134 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_max_listing_results": "1000", 2024-09-11T09:50:26.134 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_max_notify_retries": "10", 2024-09-11T09:50:26.134 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_max_objs_per_shard": "100000", 2024-09-11T09:50:26.134 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_max_put_param_size": "1048576", 2024-09-11T09:50:26.134 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_max_put_size": "5368709120", 2024-09-11T09:50:26.134 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_max_slo_entries": "1000", 2024-09-11T09:50:26.134 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_md_log_max_shards": "64", 2024-09-11T09:50:26.134 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_md_notify_interval_msec": "200", 2024-09-11T09:50:26.134 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_meta_sync_poll_interval": "20", 2024-09-11T09:50:26.135 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_meta_sync_spawn_window": "20", 2024-09-11T09:50:26.135 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_mime_types_file": "/etc/mime.types", 2024-09-11T09:50:26.135 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_mp_lock_max_time": "600", 2024-09-11T09:50:26.135 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_multi_obj_del_max_aio": "16", 2024-09-11T09:50:26.135 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_multipart_min_part_size": "5242880", 2024-09-11T09:50:26.135 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_multipart_part_upload_limit": "10000", 2024-09-11T09:50:26.135 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_nfs_fhcache_partitions": "3", 2024-09-11T09:50:26.135 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_nfs_fhcache_size": "2017", 2024-09-11T09:50:26.135 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_nfs_frontends": "rgw-nfs", 2024-09-11T09:50:26.135 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_nfs_lru_lane_hiwat": "911", 2024-09-11T09:50:26.135 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_nfs_lru_lanes": "5", 2024-09-11T09:50:26.135 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_nfs_max_gc": "300", 2024-09-11T09:50:26.135 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_nfs_namespace_expire_secs": "300", 2024-09-11T09:50:26.135 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_nfs_run_gc_threads": "false", 2024-09-11T09:50:26.136 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_nfs_run_lc_threads": "false", 2024-09-11T09:50:26.136 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_nfs_run_quota_threads": "false", 2024-09-11T09:50:26.136 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_nfs_run_sync_thread": "false", 2024-09-11T09:50:26.136 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_nfs_s3_fast_attrs": "false", 2024-09-11T09:50:26.136 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_nfs_write_completion_interval_s": "10", 2024-09-11T09:50:26.136 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_num_async_rados_threads": "32", 2024-09-11T09:50:26.136 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_num_control_oids": "8", 2024-09-11T09:50:26.136 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_numa_node": "-1", 2024-09-11T09:50:26.136 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_obj_stripe_size": "4194304", 2024-09-11T09:50:26.136 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_obj_tombstone_cache_size": "1000", 2024-09-11T09:50:26.136 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_objexp_chunk_size": "100", 2024-09-11T09:50:26.136 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_objexp_gc_interval": "600", 2024-09-11T09:50:26.136 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_objexp_hints_num_shards": "127", 2024-09-11T09:50:26.137 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_olh_pending_timeout_sec": "3600", 2024-09-11T09:50:26.137 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_op_thread_suicide_timeout": "0", 2024-09-11T09:50:26.137 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_op_thread_timeout": "600", 2024-09-11T09:50:26.137 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_op_tracing": "false", 2024-09-11T09:50:26.137 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_opa_token": "", 2024-09-11T09:50:26.137 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_opa_url": "", 2024-09-11T09:50:26.137 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_opa_verify_ssl": "true", 2024-09-11T09:50:26.137 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_ops_log_data_backlog": "5242880", 2024-09-11T09:50:26.137 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_ops_log_file_path": "/var/log/ceph/ops-log-ceph-mon.a.log", 2024-09-11T09:50:26.137 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_ops_log_rados": "false", 2024-09-11T09:50:26.137 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_ops_log_socket_path": "", 2024-09-11T09:50:26.137 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_override_bucket_index_max_shards": "0", 2024-09-11T09:50:26.137 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_pending_bucket_index_op_expiration": "120", 2024-09-11T09:50:26.138 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_period_latest_epoch_info_oid": ".latest_epoch", 2024-09-11T09:50:26.138 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_period_push_interval": "2.000000", 2024-09-11T09:50:26.138 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_period_push_interval_max": "30.000000", 2024-09-11T09:50:26.138 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_period_root_pool": ".rgw.root", 2024-09-11T09:50:26.138 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_policy_reject_invalid_principals": "true", 2024-09-11T09:50:26.138 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_posix_base_path": "/tmp/rgw_posix_driver", 2024-09-11T09:50:26.138 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_posix_cache_lanes": "3", 2024-09-11T09:50:26.138 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_posix_cache_lmdb_count": "3", 2024-09-11T09:50:26.138 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_posix_cache_max_buckets": "100", 2024-09-11T09:50:26.138 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_posix_cache_partitions": "3", 2024-09-11T09:50:26.138 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_posix_database_root": "/var/lib/ceph/radosgw", 2024-09-11T09:50:26.138 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_print_continue": "true", 2024-09-11T09:50:26.138 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_print_prohibited_content_length": "false", 2024-09-11T09:50:26.138 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_put_obj_max_window_size": "67108864", 2024-09-11T09:50:26.138 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_put_obj_min_window_size": "16777216", 2024-09-11T09:50:26.139 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_rados_pool_autoscale_bias": "4.000000", 2024-09-11T09:50:26.139 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_rados_pool_recovery_priority": "5", 2024-09-11T09:50:26.139 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_rados_tracing": "false", 2024-09-11T09:50:26.139 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_realm": "", 2024-09-11T09:50:26.139 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_realm_id": "", 2024-09-11T09:50:26.139 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_realm_root_pool": ".rgw.root", 2024-09-11T09:50:26.139 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_region": "", 2024-09-11T09:50:26.139 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_region_root_pool": ".rgw.root", 2024-09-11T09:50:26.139 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_relaxed_region_enforcement": "false", 2024-09-11T09:50:26.139 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_relaxed_s3_bucket_names": "false", 2024-09-11T09:50:26.139 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_remote_addr_param": "REMOTE_ADDR", 2024-09-11T09:50:26.139 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_request_uri": "", 2024-09-11T09:50:26.140 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_reshard_batch_size": "64", 2024-09-11T09:50:26.140 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_reshard_bucket_lock_duration": "360", 2024-09-11T09:50:26.140 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_reshard_max_aio": "128", 2024-09-11T09:50:26.140 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_reshard_num_logs": "16", 2024-09-11T09:50:26.140 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_reshard_thread_interval": "600", 2024-09-11T09:50:26.140 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_resolve_cname": "false", 2024-09-11T09:50:26.140 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_rest_getusage_op_compat": "false", 2024-09-11T09:50:26.140 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_run_sync_thread": "true", 2024-09-11T09:50:26.140 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_s3_auth_disable_signature_url": "false", 2024-09-11T09:50:26.140 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_s3_auth_order": "sts, external, local", 2024-09-11T09:50:26.140 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_s3_auth_use_keystone": "false", 2024-09-11T09:50:26.140 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_s3_auth_use_ldap": "false", 2024-09-11T09:50:26.140 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_s3_auth_use_rados": "true", 2024-09-11T09:50:26.140 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_s3_auth_use_sts": "false", 2024-09-11T09:50:26.141 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_s3_client_max_sig_ver": "-1", 2024-09-11T09:50:26.141 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_s3_success_create_obj_status": "0", 2024-09-11T09:50:26.141 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_safe_max_objects_per_shard": "102400", 2024-09-11T09:50:26.141 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_scheduler_type": "throttler", 2024-09-11T09:50:26.141 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_script_uri": "", 2024-09-11T09:50:26.141 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_service_provider_name": "", 2024-09-11T09:50:26.141 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_shard_warning_threshold": "90.000000", 2024-09-11T09:50:26.141 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_sts_client_id": "", 2024-09-11T09:50:26.141 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_sts_client_secret": "", 2024-09-11T09:50:26.141 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_sts_entry": "sts", 2024-09-11T09:50:26.141 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_sts_key": "sts", 2024-09-11T09:50:26.141 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_sts_max_session_duration": "43200", 2024-09-11T09:50:26.141 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_sts_min_session_duration": "900", 2024-09-11T09:50:26.142 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_sts_token_introspection_url": "", 2024-09-11T09:50:26.142 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_swift_account_in_url": "false", 2024-09-11T09:50:26.142 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_swift_auth_entry": "auth", 2024-09-11T09:50:26.142 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_swift_auth_url": "", 2024-09-11T09:50:26.142 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_swift_custom_header": "", 2024-09-11T09:50:26.142 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_swift_enforce_content_length": "false", 2024-09-11T09:50:26.142 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_swift_need_stats": "true", 2024-09-11T09:50:26.142 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_swift_tenant_name": "", 2024-09-11T09:50:26.142 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_swift_token_expiration": "86400", 2024-09-11T09:50:26.143 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_swift_url": "", 2024-09-11T09:50:26.143 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_swift_url_prefix": "swift", 2024-09-11T09:50:26.143 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_swift_versioning_enabled": "false", 2024-09-11T09:50:26.143 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_sync_data_full_inject_err_probability": "0.000000", 2024-09-11T09:50:26.143 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_sync_data_inject_err_probability": "0.000000", 2024-09-11T09:50:26.143 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_sync_lease_period": "120", 2024-09-11T09:50:26.143 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_sync_log_trim_concurrent_buckets": "4", 2024-09-11T09:50:26.143 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_sync_log_trim_interval": "1200", 2024-09-11T09:50:26.143 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_sync_log_trim_max_buckets": "16", 2024-09-11T09:50:26.143 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_sync_log_trim_min_cold_buckets": "4", 2024-09-11T09:50:26.143 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_sync_meta_inject_err_probability": "0.000000", 2024-09-11T09:50:26.143 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_sync_obj_etag_verify": "false", 2024-09-11T09:50:26.144 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_sync_trace_history_size": "4096", 2024-09-11T09:50:26.144 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_sync_trace_per_node_log_size": "32", 2024-09-11T09:50:26.144 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_sync_trace_servicemap_update_interval": "10", 2024-09-11T09:50:26.144 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_thread_pool_size": "512", 2024-09-11T09:50:26.144 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_topic_persistency_max_retries": "0", 2024-09-11T09:50:26.144 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_topic_persistency_sleep_duration": "0", 2024-09-11T09:50:26.144 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_topic_persistency_time_to_live": "0", 2024-09-11T09:50:26.144 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_topic_require_publish_policy": "false", 2024-09-11T09:50:26.144 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_torrent_comment": "", 2024-09-11T09:50:26.144 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_torrent_createby": "", 2024-09-11T09:50:26.144 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_torrent_encoding": "", 2024-09-11T09:50:26.144 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_torrent_flag": "false", 2024-09-11T09:50:26.144 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_torrent_max_size": "5368709120", 2024-09-11T09:50:26.145 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_torrent_origin": "", 2024-09-11T09:50:26.145 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_torrent_sha_unit": "524288", 2024-09-11T09:50:26.145 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_torrent_tracker": "", 2024-09-11T09:50:26.145 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_trust_forwarded_https": "false", 2024-09-11T09:50:26.145 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_usage_log_flush_threshold": "1024", 2024-09-11T09:50:26.145 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_usage_log_tick_interval": "30", 2024-09-11T09:50:26.145 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_usage_max_shards": "32", 2024-09-11T09:50:26.145 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_usage_max_user_shards": "1", 2024-09-11T09:50:26.145 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_use_opa_authz": "false", 2024-09-11T09:50:26.145 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_user_counters_cache": "false", 2024-09-11T09:50:26.145 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_user_counters_cache_size": "10000", 2024-09-11T09:50:26.145 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_user_default_quota_max_objects": "-1", 2024-09-11T09:50:26.146 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_user_default_quota_max_size": "-1", 2024-09-11T09:50:26.146 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_user_max_buckets": "1000", 2024-09-11T09:50:26.146 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_user_policies_max_num": "100", 2024-09-11T09:50:26.146 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_user_quota_bucket_sync_interval": "180", 2024-09-11T09:50:26.146 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_user_quota_sync_idle_users": "false", 2024-09-11T09:50:26.146 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_user_quota_sync_interval": "86400", 2024-09-11T09:50:26.146 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_user_quota_sync_wait_time": "86400", 2024-09-11T09:50:26.146 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_user_unique_email": "true", 2024-09-11T09:50:26.146 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_verify_ssl": "true", 2024-09-11T09:50:26.146 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_website_routing_rules_max_num": "50", 2024-09-11T09:50:26.146 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_zone": "", 2024-09-11T09:50:26.146 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_zone_id": "", 2024-09-11T09:50:26.146 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_zone_root_pool": ".rgw.root", 2024-09-11T09:50:26.147 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_zonegroup": "", 2024-09-11T09:50:26.147 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_zonegroup_id": "", 2024-09-11T09:50:26.147 INFO:teuthology.orchestra.run.smithi017.stdout: "rgw_zonegroup_root_pool": ".rgw.root", 2024-09-11T09:50:26.147 INFO:teuthology.orchestra.run.smithi017.stdout: "rgwlc_auto_session_clear": "true", 2024-09-11T09:50:26.147 INFO:teuthology.orchestra.run.smithi017.stdout: "rgwlc_skip_bucket_step": "false", 2024-09-11T09:50:26.147 INFO:teuthology.orchestra.run.smithi017.stdout: "rocksdb_block_size": "4096", 2024-09-11T09:50:26.147 INFO:teuthology.orchestra.run.smithi017.stdout: "rocksdb_bloom_bits_per_key": "20", 2024-09-11T09:50:26.147 INFO:teuthology.orchestra.run.smithi017.stdout: "rocksdb_cache_index_and_filter_blocks": "true", 2024-09-11T09:50:26.147 INFO:teuthology.orchestra.run.smithi017.stdout: "rocksdb_cache_index_and_filter_blocks_with_high_priority": "false", 2024-09-11T09:50:26.147 INFO:teuthology.orchestra.run.smithi017.stdout: "rocksdb_cache_row_ratio": "0.000000", 2024-09-11T09:50:26.147 INFO:teuthology.orchestra.run.smithi017.stdout: "rocksdb_cache_shard_bits": "4", 2024-09-11T09:50:26.147 INFO:teuthology.orchestra.run.smithi017.stdout: "rocksdb_cache_size": "536870912", 2024-09-11T09:50:26.148 INFO:teuthology.orchestra.run.smithi017.stdout: "rocksdb_cache_type": "binned_lru", 2024-09-11T09:50:26.148 INFO:teuthology.orchestra.run.smithi017.stdout: "rocksdb_cf_compact_on_deletion": "true", 2024-09-11T09:50:26.148 INFO:teuthology.orchestra.run.smithi017.stdout: "rocksdb_cf_compact_on_deletion_sliding_window": "32768", 2024-09-11T09:50:26.148 INFO:teuthology.orchestra.run.smithi017.stdout: "rocksdb_cf_compact_on_deletion_trigger": "16384", 2024-09-11T09:50:26.148 INFO:teuthology.orchestra.run.smithi017.stdout: "rocksdb_collect_compaction_stats": "false", 2024-09-11T09:50:26.148 INFO:teuthology.orchestra.run.smithi017.stdout: "rocksdb_collect_extended_stats": "false", 2024-09-11T09:50:26.148 INFO:teuthology.orchestra.run.smithi017.stdout: "rocksdb_collect_memory_stats": "false", 2024-09-11T09:50:26.148 INFO:teuthology.orchestra.run.smithi017.stdout: "rocksdb_delete_range_threshold": "1048576", 2024-09-11T09:50:26.148 INFO:teuthology.orchestra.run.smithi017.stdout: "rocksdb_index_type": "binary_search", 2024-09-11T09:50:26.148 INFO:teuthology.orchestra.run.smithi017.stdout: "rocksdb_log_to_ceph_log": "true", 2024-09-11T09:50:26.148 INFO:teuthology.orchestra.run.smithi017.stdout: "rocksdb_metadata_block_size": "4096", 2024-09-11T09:50:26.148 INFO:teuthology.orchestra.run.smithi017.stdout: "rocksdb_partition_filters": "false", 2024-09-11T09:50:26.148 INFO:teuthology.orchestra.run.smithi017.stdout: "rocksdb_perf": "false", 2024-09-11T09:50:26.149 INFO:teuthology.orchestra.run.smithi017.stdout: "rocksdb_pin_l0_filter_and_index_blocks_in_cache": "false", 2024-09-11T09:50:26.149 INFO:teuthology.orchestra.run.smithi017.stdout: "rotating_keys_bootstrap_timeout": "30", 2024-09-11T09:50:26.149 INFO:teuthology.orchestra.run.smithi017.stdout: "rotating_keys_renewal_timeout": "10", 2024-09-11T09:50:26.149 INFO:teuthology.orchestra.run.smithi017.stdout: "run_dir": "/var/run/ceph", 2024-09-11T09:50:26.149 INFO:teuthology.orchestra.run.smithi017.stdout: "seastore_block_create": "true", 2024-09-11T09:50:26.149 INFO:teuthology.orchestra.run.smithi017.stdout: "seastore_cache_lru_size": "67108864", 2024-09-11T09:50:26.149 INFO:teuthology.orchestra.run.smithi017.stdout: "seastore_cbjournal_size": "5368709120", 2024-09-11T09:50:26.149 INFO:teuthology.orchestra.run.smithi017.stdout: "seastore_data_delta_based_overwrite": "0", 2024-09-11T09:50:26.149 INFO:teuthology.orchestra.run.smithi017.stdout: "seastore_default_max_object_size": "16777216", 2024-09-11T09:50:26.149 INFO:teuthology.orchestra.run.smithi017.stdout: "seastore_default_object_metadata_reservation": "16777216", 2024-09-11T09:50:26.149 INFO:teuthology.orchestra.run.smithi017.stdout: "seastore_device_size": "53687091200", 2024-09-11T09:50:26.149 INFO:teuthology.orchestra.run.smithi017.stdout: "seastore_disable_end_to_end_data_protection": "true", 2024-09-11T09:50:26.149 INFO:teuthology.orchestra.run.smithi017.stdout: "seastore_full_integrity_check": "false", 2024-09-11T09:50:26.149 INFO:teuthology.orchestra.run.smithi017.stdout: "seastore_journal_batch_capacity": "16", 2024-09-11T09:50:26.150 INFO:teuthology.orchestra.run.smithi017.stdout: "seastore_journal_batch_flush_size": "16777216", 2024-09-11T09:50:26.150 INFO:teuthology.orchestra.run.smithi017.stdout: "seastore_journal_batch_preferred_fullness": "0.950000", 2024-09-11T09:50:26.150 INFO:teuthology.orchestra.run.smithi017.stdout: "seastore_journal_iodepth_limit": "5", 2024-09-11T09:50:26.150 INFO:teuthology.orchestra.run.smithi017.stdout: "seastore_main_device_type": "SSD", 2024-09-11T09:50:26.150 INFO:teuthology.orchestra.run.smithi017.stdout: "seastore_max_concurrent_transactions": "8", 2024-09-11T09:50:26.150 INFO:teuthology.orchestra.run.smithi017.stdout: "seastore_max_data_allocation_size": "32768", 2024-09-11T09:50:26.150 INFO:teuthology.orchestra.run.smithi017.stdout: "seastore_multiple_tiers_default_evict_ratio": "0.600000", 2024-09-11T09:50:26.150 INFO:teuthology.orchestra.run.smithi017.stdout: "seastore_multiple_tiers_fast_evict_ratio": "0.700000", 2024-09-11T09:50:26.150 INFO:teuthology.orchestra.run.smithi017.stdout: "seastore_multiple_tiers_stop_evict_ratio": "0.500000", 2024-09-11T09:50:26.150 INFO:teuthology.orchestra.run.smithi017.stdout: "seastore_obj_data_write_amplification": "1.250000", 2024-09-11T09:50:26.150 INFO:teuthology.orchestra.run.smithi017.stdout: "seastore_segment_size": "67108864", 2024-09-11T09:50:26.150 INFO:teuthology.orchestra.run.smithi017.stdout: "set_keepcaps": "false", 2024-09-11T09:50:26.150 INFO:teuthology.orchestra.run.smithi017.stdout: "setgroup": "", 2024-09-11T09:50:26.150 INFO:teuthology.orchestra.run.smithi017.stdout: "setuser": "", 2024-09-11T09:50:26.150 INFO:teuthology.orchestra.run.smithi017.stdout: "setuser_match_path": "", 2024-09-11T09:50:26.151 INFO:teuthology.orchestra.run.smithi017.stdout: "target_max_misplaced_ratio": "0.050000", 2024-09-11T09:50:26.151 INFO:teuthology.orchestra.run.smithi017.stdout: "thp": "false", 2024-09-11T09:50:26.151 INFO:teuthology.orchestra.run.smithi017.stdout: "threadpool_default_timeout": "60", 2024-09-11T09:50:26.151 INFO:teuthology.orchestra.run.smithi017.stdout: "threadpool_empty_queue_max_wait": "2", 2024-09-11T09:50:26.151 INFO:teuthology.orchestra.run.smithi017.stdout: "throttler_perf_counter": "true" 2024-09-11T09:50:26.151 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:50:26.153 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=0 2024-09-11T09:50:26.268 INFO:teuthology.orchestra.run.smithi017.stdout:mon.a: {} 2024-09-11T09:50:26.268 INFO:teuthology.orchestra.run.smithi017.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-11T09:50:26.278 INFO:teuthology.orchestra.run.smithi017.stdout:mon.b: {} 2024-09-11T09:50:26.278 INFO:teuthology.orchestra.run.smithi017.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-11T09:50:26.287 INFO:teuthology.orchestra.run.smithi017.stdout:mon.c: {} 2024-09-11T09:50:26.287 INFO:teuthology.orchestra.run.smithi017.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-11T09:50:26.297 DEBUG:teuthology.orchestra.run.smithi017:> type ceph-objectstore-tool 2024-09-11T09:50:26.313 INFO:teuthology.orchestra.run.smithi017.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-09-11T09:50:26.313 DEBUG:teuthology.orchestra.run.smithi096:> type ceph-objectstore-tool 2024-09-11T09:50:26.331 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-09-11T09:50:26.332 INFO:teuthology.run_tasks:Running task workunit... 2024-09-11T09:50:26.349 INFO:tasks.workunit:Pulling workunits from ref daaf78de334647fc0c800f4ac495cc9475a26bae 2024-09-11T09:50:26.349 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-09-11T09:50:26.349 DEBUG:teuthology.orchestra.run.smithi107:> stat -- /home/ubuntu/cephtest/mnt.0 2024-09-11T09:50:26.350 INFO:tasks.thrashosds.thrasher:starting do_thrash 2024-09-11T09:50:26.350 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5] 2024-09-11T09:50:26.350 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-11T09:50:26.350 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-09-11T09:50:26.350 INFO:tasks.thrashosds.thrasher:Setting osd 4 primary_affinity to 1.000000 2024-09-11T09:50:26.351 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 4 1 2024-09-11T09:50:26.351 INFO:tasks.thrashosds.thrasher:starting do_sighup with a delay of 0.1 2024-09-11T09:50:26.351 INFO:tasks.thrashosds.thrasher:starting do_optrack_toggle with a delay of 2.0 2024-09-11T09:50:26.351 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T09:50:26.352 INFO:tasks.thrashosds.thrasher:starting do_dump_ops 2024-09-11T09:50:26.352 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-11T09:50:26.352 INFO:tasks.thrashosds.thrasher:starting do_noscrub_toggle with a delay of 2.0 2024-09-11T09:50:26.352 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-11T09:50:26.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T09:50:26.371 INFO:teuthology.orchestra.run.smithi107.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-09-11T09:50:26.371 DEBUG:teuthology.orchestra.run.smithi107:> mkdir -- /home/ubuntu/cephtest/mnt.0 2024-09-11T09:50:26.372 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:50:26.373+0000 7f3c96ec9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31909) UID: 0 2024-09-11T09:50:26.430 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2024-09-11T09:50:26.430 DEBUG:teuthology.orchestra.run.smithi107:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2024-09-11T09:50:26.452 INFO:tasks.ceph.osd.2.smithi017.stderr:2024-09-11T09:50:26.453+0000 7fd20b170640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31672) UID: 0 2024-09-11T09:50:26.476 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:50:26.476 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [], 2024-09-11T09:50:26.476 INFO:teuthology.orchestra.run.smithi017.stdout: "num_ops": 0 2024-09-11T09:50:26.476 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:50:26.487 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-11T09:50:26.488 INFO:tasks.workunit:timeout=3h 2024-09-11T09:50:26.488 INFO:tasks.workunit:cleanup=True 2024-09-11T09:50:26.489 DEBUG:teuthology.orchestra.run.smithi107:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout daaf78de334647fc0c800f4ac495cc9475a26bae 2024-09-11T09:50:26.490 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-11T09:50:26.491 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T09:50:26.512 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-11T09:50:26.512 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T09:50:26.531 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-11T09:50:26.531 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T09:50:26.546 INFO:tasks.workunit.client.0.smithi107.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-09-11T09:50:26.549 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-11T09:50:26.549 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T09:50:26.553 INFO:tasks.ceph.osd.2.smithi017.stderr:2024-09-11T09:50:26.554+0000 7fd20b170640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31672) UID: 0 2024-09-11T09:50:26.566 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-11T09:50:26.567 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T09:50:26.584 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-11T09:50:26.585 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T09:50:26.601 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:50:26.602 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:50:26.611 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-11T09:50:26.654 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:50:26.655+0000 7f3c96ec9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31909) UID: 0 2024-09-11T09:50:26.755 INFO:tasks.ceph.osd.0.smithi017.stderr:2024-09-11T09:50:26.756+0000 7fb73c0f1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31665) UID: 0 2024-09-11T09:50:26.758 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:50:26.759 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:50:26.767 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-11T09:50:26.856 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:50:26.856+0000 7fdec5849640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31907) UID: 0 2024-09-11T09:50:26.916 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:50:26.916 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:50:26.925 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-11T09:50:26.970 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:50:26.971+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:50:27.032 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:50:27.032 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:50:27.041 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-11T09:50:27.058 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:50:27.058+0000 7fdec5849640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31907) UID: 0 2024-09-11T09:50:27.158 INFO:tasks.ceph.osd.0.smithi017.stderr:2024-09-11T09:50:27.159+0000 7fb73c0f1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31665) UID: 0 2024-09-11T09:50:27.189 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:50:27.189 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:50:27.198 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-11T09:50:27.282 INFO:tasks.ceph.osd.2.smithi017.stderr:2024-09-11T09:50:27.283+0000 7fd20b170640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31672) UID: 0 2024-09-11T09:50:27.345 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:50:27.346 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:50:27.354 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-11T09:50:27.359 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:50:27.360+0000 7fdec5849640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31907) UID: 0 2024-09-11T09:50:27.460 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:50:27.461+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:50:27.504 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:50:27.504 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:50:27.512 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-11T09:50:27.517 INFO:teuthology.orchestra.run.smithi017.stderr:set osd.4 primary-affinity to 1 (8655362) 2024-09-11T09:50:27.519 INFO:teuthology.orchestra.run.smithi017.stderr:noscrub is set 2024-09-11T09:50:27.575 INFO:tasks.ceph.osd.0.smithi017.stderr:2024-09-11T09:50:27.576+0000 7fb73c0f1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31665) UID: 0 2024-09-11T09:50:27.627 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:50:27.627 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:50:27.635 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-11T09:50:27.679 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-09-11T09:50:27.680+0000 7f7b6c7d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31904) UID: 0 2024-09-11T09:50:27.749 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:50:27.750 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:50:27.758 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-11T09:50:27.800 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:50:27.800+0000 7fdec5849640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31907) UID: 0 2024-09-11T09:50:27.864 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:50:27.865+0000 7fdec5849640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31907) UID: 0 2024-09-11T09:50:27.903 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:50:27.903 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:50:27.912 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-11T09:50:27.996 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-09-11T09:50:27.997+0000 7f7b6c7d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31904) UID: 0 2024-09-11T09:50:28.057 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:50:28.057 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:50:28.066 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-11T09:50:28.067 INFO:tasks.ceph.osd.0.smithi017.stderr:2024-09-11T09:50:28.068+0000 7fb73c0f1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31665) UID: 0 2024-09-11T09:50:28.168 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-09-11T09:50:28.168+0000 7f7b6c7d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31904) UID: 0 2024-09-11T09:50:28.212 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:50:28.212 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:50:28.221 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-11T09:50:28.305 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:50:28.306+0000 7fdec5849640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31907) UID: 0 2024-09-11T09:50:28.369 INFO:tasks.ceph.osd.0.smithi017.stderr:2024-09-11T09:50:28.370+0000 7fb73c0f1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31665) UID: 0 2024-09-11T09:50:28.380 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:50:28.380 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:50:28.389 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-11T09:50:28.473 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:50:28.474+0000 7fdec5849640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31907) UID: 0 2024-09-11T09:50:28.534 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:50:28.534 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:50:28.543 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-11T09:50:28.571 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:50:28.572+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:50:28.600 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T09:50:28.671 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:50:28.672+0000 7f3c96ec9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31909) UID: 0 2024-09-11T09:50:28.689 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:50:28.689 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:50:28.698 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-11T09:50:28.736 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-11T09:50:28.736 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T09:50:28.753 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-11T09:50:28.753 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T09:50:28.769 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-11T09:50:28.769 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T09:50:28.781 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:50:28.782+0000 7f3c96ec9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31909) UID: 0 2024-09-11T09:50:28.786 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-11T09:50:28.786 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T09:50:28.803 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-11T09:50:28.804 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T09:50:28.820 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-11T09:50:28.820 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T09:50:28.841 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-11T09:50:28.842 INFO:teuthology.orchestra.run.smithi096.stdout: "ops": [], 2024-09-11T09:50:28.842 INFO:teuthology.orchestra.run.smithi096.stdout: "complaint_time": 30, 2024-09-11T09:50:28.842 INFO:teuthology.orchestra.run.smithi096.stdout: "num_blocked_ops": 0 2024-09-11T09:50:28.842 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-11T09:50:28.850 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-11T09:50:28.891 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:50:28.892+0000 7f3c96ec9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31909) UID: 0 2024-09-11T09:50:28.974 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:50:28.975+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:50:28.997 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-11T09:50:28.997 INFO:teuthology.orchestra.run.smithi096.stdout: "size": 20, 2024-09-11T09:50:28.997 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 600, 2024-09-11T09:50:28.997 INFO:teuthology.orchestra.run.smithi096.stdout: "ops": [ 2024-09-11T09:50:28.997 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:28.997 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4230.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-11T09:50:28.997 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:18.490736+0000", 2024-09-11T09:50:28.997 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 10.506563793, 2024-09-11T09:50:28.997 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0059097569999999999, 2024-09-11T09:50:28.998 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:50:28.998 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:50:28.998 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:28.998 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:50:28.998 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4230", 2024-09-11T09:50:28.998 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.17:0/4101109576", 2024-09-11T09:50:28.998 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 1 2024-09-11T09:50:28.998 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:28.998 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:50:28.998 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:28.998 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:50:28.999 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.490736+0000", 2024-09-11T09:50:28.999 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:28.999 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:28.999 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:28.999 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:50:28.999 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.490736+0000", 2024-09-11T09:50:28.999 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:28.999 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:28.999 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:28.999 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:50:28.999 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.490742+0000", 2024-09-11T09:50:28.999 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.48e-06 2024-09-11T09:50:28.999 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:29.000 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:29.000 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:50:29.000 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.490749+0000", 2024-09-11T09:50:29.000 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.4569999999999999e-06 2024-09-11T09:50:29.000 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:29.000 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:29.000 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:50:29.000 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.490762+0000", 2024-09-11T09:50:29.000 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.2721000000000001e-05 2024-09-11T09:50:29.000 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:29.000 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:29.000 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:50:29.000 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.490774+0000", 2024-09-11T09:50:29.001 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1823e-05 2024-09-11T09:50:29.001 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:29.001 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:29.001 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:50:29.001 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.490846+0000", 2024-09-11T09:50:29.001 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.1878000000000005e-05 2024-09-11T09:50:29.001 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:29.001 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:29.001 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for peered", 2024-09-11T09:50:29.001 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.490857+0000", 2024-09-11T09:50:29.001 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1017e-05 2024-09-11T09:50:29.001 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:29.002 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:29.002 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:50:29.002 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.493856+0000", 2024-09-11T09:50:29.002 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00299885 2024-09-11T09:50:29.002 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:29.002 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:29.002 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:50:29.002 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.494020+0000", 2024-09-11T09:50:29.002 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00016436599999999999 2024-09-11T09:50:29.002 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:29.002 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:29.002 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 2", 2024-09-11T09:50:29.002 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.494186+0000", 2024-09-11T09:50:29.003 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00016562799999999999 2024-09-11T09:50:29.003 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:29.003 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:29.003 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:50:29.003 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.495407+0000", 2024-09-11T09:50:29.003 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0012217420000000001 2024-09-11T09:50:29.003 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:29.003 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:29.003 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:50:29.003 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.496579+0000", 2024-09-11T09:50:29.003 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001171844 2024-09-11T09:50:29.003 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:29.004 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:29.004 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:50:29.004 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.496620+0000", 2024-09-11T09:50:29.004 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.0479000000000003e-05 2024-09-11T09:50:29.004 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:29.004 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:29.004 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:50:29.004 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.496646+0000", 2024-09-11T09:50:29.004 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.6472e-05 2024-09-11T09:50:29.004 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:29.004 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:50:29.005 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:29.005 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:29.005 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:29.005 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_repop_reply(client.4230.0:1 2.3 e16/15 ondisk, result = 0)", 2024-09-11T09:50:29.005 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:18.495321+0000", 2024-09-11T09:50:29.005 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 10.501979128, 2024-09-11T09:50:29.005 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.4965999999999998e-05, 2024-09-11T09:50:29.005 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:50:29.005 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:50:29.005 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "started", 2024-09-11T09:50:29.005 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:50:29.006 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:29.006 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:50:29.006 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.495321+0000", 2024-09-11T09:50:29.006 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:29.006 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:29.006 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:29.006 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:50:29.006 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.495321+0000", 2024-09-11T09:50:29.006 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:29.006 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:29.006 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:29.007 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:50:29.007 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.495322+0000", 2024-09-11T09:50:29.007 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.7700000000000004e-07 2024-09-11T09:50:29.007 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:29.007 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:29.007 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:50:29.007 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.495325+0000", 2024-09-11T09:50:29.007 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.7780000000000001e-06 2024-09-11T09:50:29.007 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:29.007 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:29.007 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:50:29.007 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.495338+0000", 2024-09-11T09:50:29.008 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.2315e-05 2024-09-11T09:50:29.008 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:29.008 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:29.008 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:50:29.008 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.495345+0000", 2024-09-11T09:50:29.008 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.8889999999999996e-06 2024-09-11T09:50:29.008 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:29.008 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:29.008 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:50:29.008 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.495392+0000", 2024-09-11T09:50:29.008 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.7524000000000001e-05 2024-09-11T09:50:29.008 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:29.008 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:29.008 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:50:29.009 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.495402+0000", 2024-09-11T09:50:29.009 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.5840000000000001e-06 2024-09-11T09:50:29.009 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:29.009 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:29.009 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:50:29.009 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.495416+0000", 2024-09-11T09:50:29.009 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.4199000000000001e-05 2024-09-11T09:50:29.009 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:29.009 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:50:29.009 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:29.009 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:29.009 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:29.010 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4230.0:2 2.2 2:5cea7035:::rbd_info:head [call rbd.metadata_list in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e16)", 2024-09-11T09:50:29.010 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:18.499576+0000", 2024-09-11T09:50:29.010 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 10.497724621, 2024-09-11T09:50:29.010 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00018163000000000001, 2024-09-11T09:50:29.010 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:50:29.010 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:50:29.010 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "reached pg", 2024-09-11T09:50:29.010 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:50:29.010 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4230", 2024-09-11T09:50:29.010 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.17:0/4101109576", 2024-09-11T09:50:29.010 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 2 2024-09-11T09:50:29.010 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:29.010 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:50:29.010 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:29.011 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:50:29.011 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.499576+0000", 2024-09-11T09:50:29.011 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:29.011 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:29.011 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:29.011 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:50:29.011 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.499576+0000", 2024-09-11T09:50:29.011 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:29.011 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:29.011 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:29.011 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:50:29.011 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.499577+0000", 2024-09-11T09:50:29.011 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.124e-06 2024-09-11T09:50:29.011 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:29.012 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:29.012 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:50:29.012 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.499584+0000", 2024-09-11T09:50:29.012 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.1949999999999997e-06 2024-09-11T09:50:29.012 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:29.012 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:29.012 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:50:29.012 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.499597+0000", 2024-09-11T09:50:29.012 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.366e-05 2024-09-11T09:50:29.012 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:29.012 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:29.012 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:50:29.012 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.499604+0000", 2024-09-11T09:50:29.012 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.2709999999999998e-06 2024-09-11T09:50:29.013 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:29.013 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:29.013 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:50:29.013 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.499660+0000", 2024-09-11T09:50:29.013 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.6270999999999997e-05 2024-09-11T09:50:29.013 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:29.013 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:29.013 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:50:29.013 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.499757+0000", 2024-09-11T09:50:29.013 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.7108999999999995e-05 2024-09-11T09:50:29.013 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:29.013 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:50:29.013 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:29.013 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:29.014 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:29.014 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4230.0:3 2.2 2:5cea7035:::rbd_info:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e16)", 2024-09-11T09:50:29.014 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:18.499937+0000", 2024-09-11T09:50:29.014 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 10.497362832, 2024-09-11T09:50:29.014 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000143318, 2024-09-11T09:50:29.014 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:50:29.014 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:50:29.014 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "reached pg", 2024-09-11T09:50:29.014 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:50:29.014 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4230", 2024-09-11T09:50:29.014 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.17:0/4101109576", 2024-09-11T09:50:29.014 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 3 2024-09-11T09:50:29.014 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:29.014 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:50:29.014 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:29.015 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:50:29.015 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.499937+0000", 2024-09-11T09:50:29.015 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:29.015 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:29.015 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:29.015 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:50:29.015 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.499937+0000", 2024-09-11T09:50:29.015 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:29.015 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:29.015 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:29.015 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:50:29.015 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.499938+0000", 2024-09-11T09:50:29.015 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.195e-06 2024-09-11T09:50:29.015 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:29.015 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:29.016 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:50:29.016 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.499943+0000", 2024-09-11T09:50:29.016 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.9080000000000003e-06 2024-09-11T09:50:29.016 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:29.016 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:29.016 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:50:29.016 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.499956+0000", 2024-09-11T09:50:29.016 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.2728000000000001e-05 2024-09-11T09:50:29.016 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:29.016 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:29.016 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:50:29.017 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.499964+0000", 2024-09-11T09:50:29.017 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.8220000000000007e-06 2024-09-11T09:50:29.017 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:29.017 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:29.017 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:50:29.017 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.500017+0000", 2024-09-11T09:50:29.017 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.2809000000000002e-05 2024-09-11T09:50:29.017 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:29.017 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:29.017 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:50:29.017 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.500081+0000", 2024-09-11T09:50:29.017 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.3856000000000007e-05 2024-09-11T09:50:29.018 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:29.018 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:50:29.018 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:29.018 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:29.018 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:29.018 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4230.0:5 2.2 2:5cea7035:::rbd_info:head [create,write 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-11T09:50:29.018 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:19.492662+0000", 2024-09-11T09:50:29.018 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 9.5046381719999999, 2024-09-11T09:50:29.018 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0043824759999999997, 2024-09-11T09:50:29.018 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:50:29.018 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:50:29.018 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:29.018 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:50:29.018 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4230", 2024-09-11T09:50:29.019 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.17:0/4101109576", 2024-09-11T09:50:29.019 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 5 2024-09-11T09:50:29.019 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:29.019 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:50:29.019 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:29.019 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:50:29.019 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.492662+0000", 2024-09-11T09:50:29.019 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:29.019 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:29.019 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:29.019 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:50:29.019 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.492662+0000", 2024-09-11T09:50:29.020 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:29.020 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:29.020 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:29.020 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:50:29.020 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.492664+0000", 2024-09-11T09:50:29.020 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.3599999999999999e-06 2024-09-11T09:50:29.020 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:29.020 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:29.020 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:50:29.020 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.492675+0000", 2024-09-11T09:50:29.020 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1182000000000001e-05 2024-09-11T09:50:29.020 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:29.020 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:29.020 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:50:29.021 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.492693+0000", 2024-09-11T09:50:29.021 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.7031999999999999e-05 2024-09-11T09:50:29.021 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:29.021 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:29.021 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:50:29.021 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.492702+0000", 2024-09-11T09:50:29.021 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.5540000000000003e-06 2024-09-11T09:50:29.021 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:29.021 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:29.021 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:50:29.021 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.492767+0000", 2024-09-11T09:50:29.021 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.5411000000000001e-05 2024-09-11T09:50:29.022 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:29.022 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:29.022 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op must wait for map", 2024-09-11T09:50:29.022 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.492773+0000", 2024-09-11T09:50:29.022 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.1039999999999998e-06 2024-09-11T09:50:29.022 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:29.022 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:29.022 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:50:29.022 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.494354+0000", 2024-09-11T09:50:29.022 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0015814049999999999 2024-09-11T09:50:29.022 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:29.022 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:29.022 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:50:29.022 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.494476+0000", 2024-09-11T09:50:29.023 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000122089 2024-09-11T09:50:29.023 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:29.023 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:29.023 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 1", 2024-09-11T09:50:29.023 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.494577+0000", 2024-09-11T09:50:29.023 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000100536 2024-09-11T09:50:29.023 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:29.023 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:29.023 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:50:29.023 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.495770+0000", 2024-09-11T09:50:29.023 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0011932469999999999 2024-09-11T09:50:29.023 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:29.023 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:29.023 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:50:29.024 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.496966+0000", 2024-09-11T09:50:29.024 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001196003 2024-09-11T09:50:29.024 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:29.024 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:29.024 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:50:29.024 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.497011+0000", 2024-09-11T09:50:29.024 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.5229999999999999e-05 2024-09-11T09:50:29.024 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:29.024 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:29.024 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:50:29.024 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.497044+0000", 2024-09-11T09:50:29.024 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.3322999999999999e-05 2024-09-11T09:50:29.024 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:29.024 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:50:29.024 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:29.025 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:29.025 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:29.025 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_repop_reply(client.4230.0:5 2.2 e17/15 ondisk, result = 0)", 2024-09-11T09:50:29.025 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:19.495669+0000", 2024-09-11T09:50:29.025 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 9.5016313839999995, 2024-09-11T09:50:29.025 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000106431, 2024-09-11T09:50:29.025 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:50:29.025 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:50:29.025 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "started", 2024-09-11T09:50:29.025 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:50:29.025 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:29.025 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:50:29.025 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.495669+0000", 2024-09-11T09:50:29.025 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:29.026 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:29.026 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:29.026 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:50:29.026 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.495669+0000", 2024-09-11T09:50:29.026 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:29.026 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:29.026 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:29.026 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:50:29.026 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.495670+0000", 2024-09-11T09:50:29.027 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.218e-06 2024-09-11T09:50:29.027 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:29.027 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:29.027 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:50:29.027 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.495676+0000", 2024-09-11T09:50:29.027 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.5119999999999997e-06 2024-09-11T09:50:29.027 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:29.027 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:29.027 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:50:29.027 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.495690+0000", 2024-09-11T09:50:29.027 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.3312000000000001e-05 2024-09-11T09:50:29.027 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:29.027 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:29.027 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:50:29.028 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.495706+0000", 2024-09-11T09:50:29.028 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.6560000000000001e-05 2024-09-11T09:50:29.028 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:29.028 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:29.028 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:50:29.028 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.495759+0000", 2024-09-11T09:50:29.028 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.2828000000000003e-05 2024-09-11T09:50:29.028 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:29.028 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:29.028 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:50:29.028 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.495765+0000", 2024-09-11T09:50:29.028 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.7869999999999998e-06 2024-09-11T09:50:29.028 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:29.028 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:29.029 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:50:29.029 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.495775+0000", 2024-09-11T09:50:29.029 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.0214000000000001e-05 2024-09-11T09:50:29.029 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:29.029 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:50:29.029 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:29.029 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:29.029 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:29.029 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4230.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-09-11T09:50:29.029 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:20.498634+0000", 2024-09-11T09:50:29.029 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 8.4986657720000007, 2024-09-11T09:50:29.029 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0050648539999999997, 2024-09-11T09:50:29.029 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:50:29.029 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:50:29.030 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:29.030 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:50:29.030 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4230", 2024-09-11T09:50:29.030 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.17:0/4101109576", 2024-09-11T09:50:29.030 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 7 2024-09-11T09:50:29.030 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:29.030 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:50:29.030 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:29.030 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:50:29.030 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.498634+0000", 2024-09-11T09:50:29.030 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:29.030 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:29.030 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:29.030 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:50:29.030 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.498634+0000", 2024-09-11T09:50:29.031 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:29.031 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:29.031 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:29.031 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:50:29.031 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.498637+0000", 2024-09-11T09:50:29.031 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.1919999999999999e-06 2024-09-11T09:50:29.031 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:29.031 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:29.031 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:50:29.031 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.498648+0000", 2024-09-11T09:50:29.031 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1116e-05 2024-09-11T09:50:29.031 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:29.031 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:29.031 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:50:29.032 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.498666+0000", 2024-09-11T09:50:29.032 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.8411999999999999e-05 2024-09-11T09:50:29.032 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:29.032 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:29.032 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:50:29.032 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.498675+0000", 2024-09-11T09:50:29.032 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.4059999999999994e-06 2024-09-11T09:50:29.032 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:29.032 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:29.032 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:50:29.032 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.498745+0000", 2024-09-11T09:50:29.032 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.9764000000000005e-05 2024-09-11T09:50:29.033 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:29.033 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:29.033 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op must wait for map", 2024-09-11T09:50:29.033 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.498754+0000", 2024-09-11T09:50:29.033 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.5969999999999992e-06 2024-09-11T09:50:29.033 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:29.033 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:29.033 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:50:29.033 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.500385+0000", 2024-09-11T09:50:29.033 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0016308360000000001 2024-09-11T09:50:29.033 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:29.033 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:29.033 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:50:29.033 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.500473+0000", 2024-09-11T09:50:29.034 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.7966000000000004e-05 2024-09-11T09:50:29.034 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:29.034 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:29.034 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 1", 2024-09-11T09:50:29.034 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.500591+0000", 2024-09-11T09:50:29.034 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000117895 2024-09-11T09:50:29.034 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:29.034 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:29.034 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:50:29.034 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.501685+0000", 2024-09-11T09:50:29.034 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001094886 2024-09-11T09:50:29.034 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:29.034 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:29.034 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:50:29.035 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.503627+0000", 2024-09-11T09:50:29.035 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001941199 2024-09-11T09:50:29.035 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:29.035 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:29.035 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:50:29.035 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.503667+0000", 2024-09-11T09:50:29.035 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.0645000000000001e-05 2024-09-11T09:50:29.035 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:29.035 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:29.035 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:50:29.035 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.503699+0000", 2024-09-11T09:50:29.035 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.1940000000000003e-05 2024-09-11T09:50:29.035 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:29.035 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:50:29.035 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:29.036 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:29.036 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:29.036 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_repop_reply(client.4230.0:7 2.2 e18/15 ondisk, result = 0)", 2024-09-11T09:50:29.036 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:20.501608+0000", 2024-09-11T09:50:29.036 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 8.4956922499999994, 2024-09-11T09:50:29.036 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.2459000000000005e-05, 2024-09-11T09:50:29.036 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:50:29.036 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:50:29.036 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "started", 2024-09-11T09:50:29.036 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:50:29.036 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:29.036 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:50:29.036 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.501608+0000", 2024-09-11T09:50:29.036 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:29.037 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:29.037 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:29.037 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:50:29.037 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.501608+0000", 2024-09-11T09:50:29.037 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:29.037 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:29.037 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:29.037 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:50:29.037 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.501609+0000", 2024-09-11T09:50:29.037 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.0160000000000001e-06 2024-09-11T09:50:29.037 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:29.037 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:29.037 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:50:29.037 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.501613+0000", 2024-09-11T09:50:29.038 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.5959999999999997e-06 2024-09-11T09:50:29.038 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:29.038 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:29.038 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:50:29.038 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.501625+0000", 2024-09-11T09:50:29.038 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1888e-05 2024-09-11T09:50:29.038 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:29.038 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:29.038 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:50:29.038 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.501634+0000", 2024-09-11T09:50:29.038 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.6139999999999992e-06 2024-09-11T09:50:29.038 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:29.038 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:29.038 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:50:29.038 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.501676+0000", 2024-09-11T09:50:29.038 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.1953999999999998e-05 2024-09-11T09:50:29.039 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:29.039 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:29.039 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:50:29.039 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.501682+0000", 2024-09-11T09:50:29.039 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.8000000000000004e-06 2024-09-11T09:50:29.039 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:29.039 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:29.039 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:50:29.039 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.501690+0000", 2024-09-11T09:50:29.039 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.5909999999999996e-06 2024-09-11T09:50:29.039 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:29.039 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:50:29.039 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:29.039 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:29.039 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:50:29.040 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-11T09:50:29.041 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-11T09:50:29.084 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:50:29.085+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:50:29.149 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:50:29.149 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [], 2024-09-11T09:50:29.149 INFO:teuthology.orchestra.run.smithi017.stdout: "num_ops": 0 2024-09-11T09:50:29.149 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:50:29.158 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-11T09:50:29.176 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:50:29.177+0000 7fdec5849640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31907) UID: 0 2024-09-11T09:50:29.277 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:50:29.278+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:50:29.305 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:50:29.305 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [], 2024-09-11T09:50:29.305 INFO:teuthology.orchestra.run.smithi017.stdout: "complaint_time": 30, 2024-09-11T09:50:29.305 INFO:teuthology.orchestra.run.smithi017.stdout: "num_blocked_ops": 0 2024-09-11T09:50:29.305 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:50:29.314 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-11T09:50:29.398 INFO:tasks.ceph.osd.2.smithi017.stderr:2024-09-11T09:50:29.399+0000 7fd20b170640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31672) UID: 0 2024-09-11T09:50:29.463 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:50:29.463 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 20, 2024-09-11T09:50:29.463 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 600, 2024-09-11T09:50:29.463 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [ 2024-09-11T09:50:29.464 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.464 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4183.0:21 1.0 e13/12 1:30306672:devicehealth::main.db.0000000000000000:head v 13'11, mlcod=13'8)", 2024-09-11T09:50:29.464 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:11.491728+0000", 2024-09-11T09:50:29.464 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 17.971471959999999, 2024-09-11T09:50:29.464 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00093269800000000003, 2024-09-11T09:50:29.464 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:50:29.464 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:50:29.464 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:29.464 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:50:29.464 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.464 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:50:29.464 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.491728+0000", 2024-09-11T09:50:29.464 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:29.465 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.465 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.465 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:50:29.465 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.491728+0000", 2024-09-11T09:50:29.465 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:29.465 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.465 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.465 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:50:29.465 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.491729+0000", 2024-09-11T09:50:29.465 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.8599999999999997e-07 2024-09-11T09:50:29.465 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.465 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.465 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:50:29.466 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.491820+0000", 2024-09-11T09:50:29.466 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.0812999999999999e-05 2024-09-11T09:50:29.466 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.466 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.466 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:50:29.466 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.491833+0000", 2024-09-11T09:50:29.466 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.341e-05 2024-09-11T09:50:29.466 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.466 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.466 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:50:29.466 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.491841+0000", 2024-09-11T09:50:29.466 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.8730000000000007e-06 2024-09-11T09:50:29.467 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.467 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.467 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:50:29.467 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.491916+0000", 2024-09-11T09:50:29.467 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.4881000000000004e-05 2024-09-11T09:50:29.467 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.467 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.467 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:50:29.467 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.491931+0000", 2024-09-11T09:50:29.467 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.4825e-05 2024-09-11T09:50:29.467 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.467 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.467 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:50:29.468 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.492633+0000", 2024-09-11T09:50:29.468 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00070191600000000002 2024-09-11T09:50:29.468 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.468 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.468 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:50:29.468 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.492661+0000", 2024-09-11T09:50:29.468 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.8093999999999999e-05 2024-09-11T09:50:29.468 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:29.468 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:50:29.468 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:29.468 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.468 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.469 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4183.0:30 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'16, mlcod=13'14)", 2024-09-11T09:50:29.469 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:11.498866+0000", 2024-09-11T09:50:29.469 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 17.964334236999999, 2024-09-11T09:50:29.469 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00089294000000000005, 2024-09-11T09:50:29.469 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:50:29.469 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:50:29.469 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:29.469 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:50:29.469 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.469 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:50:29.469 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.498866+0000", 2024-09-11T09:50:29.469 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:29.469 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.470 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.470 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:50:29.470 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.498866+0000", 2024-09-11T09:50:29.470 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:29.470 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.470 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.470 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:50:29.470 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.498867+0000", 2024-09-11T09:50:29.470 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.7000000000000003e-07 2024-09-11T09:50:29.471 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.471 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.471 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:50:29.471 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.498873+0000", 2024-09-11T09:50:29.471 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.0140000000000004e-06 2024-09-11T09:50:29.471 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.471 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.471 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:50:29.471 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.498886+0000", 2024-09-11T09:50:29.472 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2745000000000001e-05 2024-09-11T09:50:29.472 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.472 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.472 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:50:29.472 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.498894+0000", 2024-09-11T09:50:29.472 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.2349999999999999e-06 2024-09-11T09:50:29.472 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.472 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.472 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:50:29.472 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.498941+0000", 2024-09-11T09:50:29.472 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.7679999999999998e-05 2024-09-11T09:50:29.472 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.473 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.473 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:50:29.473 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.498955+0000", 2024-09-11T09:50:29.473 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3446e-05 2024-09-11T09:50:29.473 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.473 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.473 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:50:29.473 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.499717+0000", 2024-09-11T09:50:29.473 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00076248199999999998 2024-09-11T09:50:29.473 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.473 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.473 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:50:29.473 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.499759+0000", 2024-09-11T09:50:29.473 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.1467999999999997e-05 2024-09-11T09:50:29.474 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:29.474 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:50:29.474 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:29.474 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.474 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.474 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4183.0:31 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'17, mlcod=13'14)", 2024-09-11T09:50:29.474 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:11.499319+0000", 2024-09-11T09:50:29.474 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 17.963881291, 2024-09-11T09:50:29.474 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0010073879999999999, 2024-09-11T09:50:29.474 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:50:29.474 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:50:29.474 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:29.474 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:50:29.474 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.475 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:50:29.475 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.499319+0000", 2024-09-11T09:50:29.475 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:29.475 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.475 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.475 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:50:29.475 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.499319+0000", 2024-09-11T09:50:29.475 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:29.475 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.475 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.475 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:50:29.475 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.499320+0000", 2024-09-11T09:50:29.475 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.7700000000000004e-07 2024-09-11T09:50:29.476 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.476 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.476 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:50:29.476 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.499326+0000", 2024-09-11T09:50:29.476 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.6420000000000001e-06 2024-09-11T09:50:29.476 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.476 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.476 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:50:29.476 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.499339+0000", 2024-09-11T09:50:29.476 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2537999999999999e-05 2024-09-11T09:50:29.476 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.476 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.476 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:50:29.477 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.499345+0000", 2024-09-11T09:50:29.477 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.8220000000000003e-06 2024-09-11T09:50:29.477 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.477 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.477 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:50:29.477 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.499403+0000", 2024-09-11T09:50:29.477 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.8814000000000003e-05 2024-09-11T09:50:29.477 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.477 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.477 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:50:29.477 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.499427+0000", 2024-09-11T09:50:29.477 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.3853999999999999e-05 2024-09-11T09:50:29.477 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.477 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.478 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:50:29.478 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.500294+0000", 2024-09-11T09:50:29.478 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00086622399999999997 2024-09-11T09:50:29.478 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.478 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.478 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:50:29.478 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.500326+0000", 2024-09-11T09:50:29.478 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.2716999999999999e-05 2024-09-11T09:50:29.478 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:29.478 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:50:29.478 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:29.478 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.478 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.479 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4183.0:32 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'18, mlcod=13'14)", 2024-09-11T09:50:29.479 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:11.499796+0000", 2024-09-11T09:50:29.479 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 17.963404085000001, 2024-09-11T09:50:29.479 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00092915100000000002, 2024-09-11T09:50:29.479 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:50:29.479 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:50:29.479 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:29.479 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:50:29.479 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.479 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:50:29.479 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.499796+0000", 2024-09-11T09:50:29.479 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:29.480 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.480 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.480 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:50:29.480 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.499796+0000", 2024-09-11T09:50:29.480 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:29.480 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.480 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.480 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:50:29.480 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.499797+0000", 2024-09-11T09:50:29.480 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.9400000000000004e-07 2024-09-11T09:50:29.480 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.480 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.480 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:50:29.481 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.499841+0000", 2024-09-11T09:50:29.481 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.3900000000000003e-05 2024-09-11T09:50:29.481 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.481 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.481 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:50:29.481 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.499852+0000", 2024-09-11T09:50:29.481 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1462e-05 2024-09-11T09:50:29.481 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.481 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.481 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:50:29.481 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.499858+0000", 2024-09-11T09:50:29.481 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.8320000000000002e-06 2024-09-11T09:50:29.481 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.482 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.482 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:50:29.482 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.499915+0000", 2024-09-11T09:50:29.482 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.7187e-05 2024-09-11T09:50:29.482 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.482 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.482 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:50:29.482 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.499930+0000", 2024-09-11T09:50:29.482 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.4928e-05 2024-09-11T09:50:29.482 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.482 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.482 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:50:29.482 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.500697+0000", 2024-09-11T09:50:29.483 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00076658399999999995 2024-09-11T09:50:29.483 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.483 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.483 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:50:29.483 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.500725+0000", 2024-09-11T09:50:29.483 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.8364000000000002e-05 2024-09-11T09:50:29.483 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:29.483 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:50:29.483 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:29.483 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.483 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.483 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4183.0:34 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'20, mlcod=13'15)", 2024-09-11T09:50:29.483 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:11.501016+0000", 2024-09-11T09:50:29.483 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 17.962184567000001, 2024-09-11T09:50:29.484 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00097815399999999996, 2024-09-11T09:50:29.484 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:50:29.484 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:50:29.484 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:29.484 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:50:29.484 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.484 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:50:29.484 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.501016+0000", 2024-09-11T09:50:29.484 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:29.484 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.484 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.484 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:50:29.484 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.501016+0000", 2024-09-11T09:50:29.484 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:29.485 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.485 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.485 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:50:29.485 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.501016+0000", 2024-09-11T09:50:29.485 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.3799999999999997e-07 2024-09-11T09:50:29.485 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.485 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.485 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:50:29.485 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.501023+0000", 2024-09-11T09:50:29.485 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.2180000000000002e-06 2024-09-11T09:50:29.485 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.485 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.485 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:50:29.486 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.501031+0000", 2024-09-11T09:50:29.486 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.5379999999999997e-06 2024-09-11T09:50:29.486 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.486 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.486 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:50:29.486 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.501035+0000", 2024-09-11T09:50:29.486 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.216e-06 2024-09-11T09:50:29.486 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.486 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.486 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:50:29.486 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.501105+0000", 2024-09-11T09:50:29.486 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.9757000000000003e-05 2024-09-11T09:50:29.486 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.486 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.487 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:50:29.487 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.501120+0000", 2024-09-11T09:50:29.487 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.5358000000000001e-05 2024-09-11T09:50:29.487 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.487 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.487 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:50:29.487 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.501955+0000", 2024-09-11T09:50:29.487 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00083505000000000005 2024-09-11T09:50:29.487 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.487 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.487 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:50:29.487 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.501994+0000", 2024-09-11T09:50:29.487 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.8479000000000002e-05 2024-09-11T09:50:29.487 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:29.488 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:50:29.488 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:29.488 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.488 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.488 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4183.0:38 1.0 e13/12 1:30306672:devicehealth::main.db.0000000000000000:head v 13'24, mlcod=13'22)", 2024-09-11T09:50:29.488 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:11.503891+0000", 2024-09-11T09:50:29.488 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 17.959309125000001, 2024-09-11T09:50:29.488 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000959351, 2024-09-11T09:50:29.488 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:50:29.488 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:50:29.488 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:29.488 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:50:29.488 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.489 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:50:29.489 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.503891+0000", 2024-09-11T09:50:29.489 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:29.489 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.489 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.489 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:50:29.489 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.503891+0000", 2024-09-11T09:50:29.489 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:29.489 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.489 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.489 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:50:29.489 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.503892+0000", 2024-09-11T09:50:29.489 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.5499999999999998e-07 2024-09-11T09:50:29.489 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.490 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.490 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:50:29.490 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.503967+0000", 2024-09-11T09:50:29.490 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.4993999999999994e-05 2024-09-11T09:50:29.490 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.490 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.490 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:50:29.490 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.503976+0000", 2024-09-11T09:50:29.490 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.4709999999999996e-06 2024-09-11T09:50:29.490 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.490 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.490 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:50:29.490 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.503981+0000", 2024-09-11T09:50:29.491 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.6789999999999996e-06 2024-09-11T09:50:29.491 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.491 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.491 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:50:29.491 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.504021+0000", 2024-09-11T09:50:29.491 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.0065999999999999e-05 2024-09-11T09:50:29.491 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.491 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.491 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:50:29.491 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.504036+0000", 2024-09-11T09:50:29.491 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.468e-05 2024-09-11T09:50:29.491 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.491 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.491 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:50:29.492 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.504813+0000", 2024-09-11T09:50:29.492 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00077718299999999998 2024-09-11T09:50:29.492 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.492 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.492 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:50:29.492 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.504850+0000", 2024-09-11T09:50:29.492 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.7722999999999998e-05 2024-09-11T09:50:29.492 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:29.492 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:50:29.492 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:29.492 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.492 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.493 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4183.0:54 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'32, mlcod=13'29)", 2024-09-11T09:50:29.493 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:11.515404+0000", 2024-09-11T09:50:29.493 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 17.947796520000001, 2024-09-11T09:50:29.493 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00089471799999999997, 2024-09-11T09:50:29.493 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:50:29.493 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:50:29.493 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:29.493 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:50:29.493 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.493 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:50:29.493 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.515404+0000", 2024-09-11T09:50:29.493 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:29.493 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.493 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.494 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:50:29.494 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.515404+0000", 2024-09-11T09:50:29.494 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:29.494 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.494 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.494 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:50:29.494 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.515404+0000", 2024-09-11T09:50:29.494 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.3799999999999996e-07 2024-09-11T09:50:29.494 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.494 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.494 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:50:29.494 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.515410+0000", 2024-09-11T09:50:29.494 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.1789999999999999e-06 2024-09-11T09:50:29.495 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.495 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.495 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:50:29.495 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.515422+0000", 2024-09-11T09:50:29.495 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1929999999999999e-05 2024-09-11T09:50:29.495 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.495 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.495 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:50:29.495 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.515427+0000", 2024-09-11T09:50:29.495 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.6060000000000002e-06 2024-09-11T09:50:29.495 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.495 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.495 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:50:29.496 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.515482+0000", 2024-09-11T09:50:29.496 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.4743000000000001e-05 2024-09-11T09:50:29.496 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.496 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.496 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:50:29.496 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.515498+0000", 2024-09-11T09:50:29.496 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.5778999999999999e-05 2024-09-11T09:50:29.496 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.496 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.496 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:50:29.496 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.516271+0000", 2024-09-11T09:50:29.496 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00077312599999999996 2024-09-11T09:50:29.496 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.496 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.497 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:50:29.497 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.516298+0000", 2024-09-11T09:50:29.497 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.7617e-05 2024-09-11T09:50:29.497 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:29.497 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:50:29.497 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:29.497 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.497 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.497 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4183.0:57 1.0 e13/12 1:30306672:devicehealth::main.db.0000000000000000:head v 13'35, mlcod=13'33)", 2024-09-11T09:50:29.497 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:11.519823+0000", 2024-09-11T09:50:29.497 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 17.943377463000001, 2024-09-11T09:50:29.497 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00090876100000000001, 2024-09-11T09:50:29.498 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:50:29.498 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:50:29.498 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:29.498 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:50:29.498 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.498 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:50:29.498 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.519823+0000", 2024-09-11T09:50:29.498 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:29.498 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.498 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.498 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:50:29.498 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.519823+0000", 2024-09-11T09:50:29.498 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:29.498 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.499 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.499 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:50:29.499 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.519823+0000", 2024-09-11T09:50:29.499 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.7000000000000004e-07 2024-09-11T09:50:29.499 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.499 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.499 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:50:29.499 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.519859+0000", 2024-09-11T09:50:29.499 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.5508e-05 2024-09-11T09:50:29.499 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.499 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.499 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:50:29.499 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.519871+0000", 2024-09-11T09:50:29.500 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.221e-05 2024-09-11T09:50:29.500 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.500 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.500 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:50:29.500 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.519877+0000", 2024-09-11T09:50:29.500 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.0410000000000003e-06 2024-09-11T09:50:29.500 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.500 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.500 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:50:29.500 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.519955+0000", 2024-09-11T09:50:29.500 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.8169000000000003e-05 2024-09-11T09:50:29.500 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.500 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.500 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:50:29.501 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.519972+0000", 2024-09-11T09:50:29.501 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6595999999999998e-05 2024-09-11T09:50:29.501 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.501 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.501 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:50:29.501 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.520691+0000", 2024-09-11T09:50:29.501 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000719405 2024-09-11T09:50:29.501 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.501 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.501 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:50:29.501 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.520731+0000", 2024-09-11T09:50:29.501 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.0061999999999997e-05 2024-09-11T09:50:29.501 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:29.501 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:50:29.502 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:29.502 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.502 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.502 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4183.0:58 1.0 e13/12 1:30306672:devicehealth::main.db.0000000000000000:head v 13'36, mlcod=13'33)", 2024-09-11T09:50:29.502 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:11.520285+0000", 2024-09-11T09:50:29.502 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 17.942915055, 2024-09-11T09:50:29.502 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00091014799999999995, 2024-09-11T09:50:29.502 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:50:29.502 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:50:29.502 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:29.502 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:50:29.502 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.502 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:50:29.503 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.520285+0000", 2024-09-11T09:50:29.503 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:29.503 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.503 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.503 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:50:29.503 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.520285+0000", 2024-09-11T09:50:29.503 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:29.503 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.503 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.503 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:50:29.503 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.520286+0000", 2024-09-11T09:50:29.503 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.4399999999999999e-07 2024-09-11T09:50:29.503 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.503 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.504 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:50:29.504 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.520348+0000", 2024-09-11T09:50:29.504 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.1760000000000005e-05 2024-09-11T09:50:29.504 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.504 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.504 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:50:29.504 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.520359+0000", 2024-09-11T09:50:29.504 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1362999999999999e-05 2024-09-11T09:50:29.504 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.504 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.504 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:50:29.504 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.520364+0000", 2024-09-11T09:50:29.505 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.3600000000000004e-06 2024-09-11T09:50:29.505 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.505 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.505 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:50:29.505 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.520413+0000", 2024-09-11T09:50:29.505 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.8226000000000002e-05 2024-09-11T09:50:29.505 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.505 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.505 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:50:29.505 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.520426+0000", 2024-09-11T09:50:29.505 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3560000000000001e-05 2024-09-11T09:50:29.505 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.505 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.505 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:50:29.506 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.521159+0000", 2024-09-11T09:50:29.506 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00073255499999999997 2024-09-11T09:50:29.506 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.506 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.506 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:50:29.506 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.521195+0000", 2024-09-11T09:50:29.506 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.6480000000000003e-05 2024-09-11T09:50:29.506 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:29.506 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:50:29.506 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:29.506 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.506 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.506 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4183.0:68 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'42, mlcod=13'39)", 2024-09-11T09:50:29.507 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:11.534957+0000", 2024-09-11T09:50:29.507 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 17.928243364, 2024-09-11T09:50:29.507 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.001065351, 2024-09-11T09:50:29.507 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:50:29.507 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:50:29.507 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:29.507 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:50:29.507 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.507 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:50:29.507 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.534957+0000", 2024-09-11T09:50:29.507 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:29.507 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.507 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.508 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:50:29.508 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.534957+0000", 2024-09-11T09:50:29.508 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:29.508 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.508 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.508 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:50:29.508 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.534957+0000", 2024-09-11T09:50:29.508 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.3799999999999998e-07 2024-09-11T09:50:29.508 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.508 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.508 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:50:29.508 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.534964+0000", 2024-09-11T09:50:29.508 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.9809999999999997e-06 2024-09-11T09:50:29.509 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.509 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.509 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:50:29.509 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.534976+0000", 2024-09-11T09:50:29.509 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2119999999999999e-05 2024-09-11T09:50:29.509 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.509 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.509 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:50:29.509 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.534980+0000", 2024-09-11T09:50:29.509 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.9890000000000003e-06 2024-09-11T09:50:29.509 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.509 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.509 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:50:29.509 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.535030+0000", 2024-09-11T09:50:29.510 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.9258000000000002e-05 2024-09-11T09:50:29.510 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.510 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.510 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:50:29.510 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.535044+0000", 2024-09-11T09:50:29.510 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3981e-05 2024-09-11T09:50:29.510 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.510 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.510 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:50:29.510 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.535995+0000", 2024-09-11T09:50:29.510 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000951804 2024-09-11T09:50:29.510 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.510 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.511 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:50:29.511 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.536022+0000", 2024-09-11T09:50:29.511 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.6780000000000001e-05 2024-09-11T09:50:29.511 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:29.511 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:50:29.511 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:29.511 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.511 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.511 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4183.0:69 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'43, mlcod=13'39)", 2024-09-11T09:50:29.511 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:11.535331+0000", 2024-09-11T09:50:29.511 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 17.927869041000001, 2024-09-11T09:50:29.511 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0011256090000000001, 2024-09-11T09:50:29.511 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:50:29.511 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:50:29.512 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:29.512 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:50:29.512 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.512 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:50:29.512 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.535331+0000", 2024-09-11T09:50:29.512 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:29.512 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.512 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.512 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:50:29.512 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.535331+0000", 2024-09-11T09:50:29.512 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:29.512 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.512 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.513 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:50:29.513 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.535332+0000", 2024-09-11T09:50:29.513 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.1600000000000001e-07 2024-09-11T09:50:29.513 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.513 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.513 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:50:29.513 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.535393+0000", 2024-09-11T09:50:29.513 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.1452999999999996e-05 2024-09-11T09:50:29.513 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.513 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.513 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:50:29.513 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.535402+0000", 2024-09-11T09:50:29.514 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.4300000000000006e-06 2024-09-11T09:50:29.514 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.514 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.514 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:50:29.514 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.535405+0000", 2024-09-11T09:50:29.514 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.512e-06 2024-09-11T09:50:29.514 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.514 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.514 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:50:29.514 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.535482+0000", 2024-09-11T09:50:29.514 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.7132999999999995e-05 2024-09-11T09:50:29.514 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.514 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.514 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:50:29.515 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.535506+0000", 2024-09-11T09:50:29.515 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.4070000000000002e-05 2024-09-11T09:50:29.515 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.515 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.515 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:50:29.515 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.536424+0000", 2024-09-11T09:50:29.515 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00091766700000000005 2024-09-11T09:50:29.515 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.515 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.515 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:50:29.515 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.536457+0000", 2024-09-11T09:50:29.515 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.2827999999999998e-05 2024-09-11T09:50:29.515 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:29.516 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:50:29.516 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:29.516 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.516 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.516 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4183.0:83 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'53, mlcod=13'50)", 2024-09-11T09:50:29.516 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:11.558836+0000", 2024-09-11T09:50:29.516 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 17.904363786000001, 2024-09-11T09:50:29.516 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00090313100000000005, 2024-09-11T09:50:29.516 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:50:29.516 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:50:29.516 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:29.516 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:50:29.516 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.517 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:50:29.517 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.558836+0000", 2024-09-11T09:50:29.517 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:29.517 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.517 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.517 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:50:29.517 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.558836+0000", 2024-09-11T09:50:29.517 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:29.517 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.517 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.517 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:50:29.517 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.558837+0000", 2024-09-11T09:50:29.517 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.4900000000000005e-07 2024-09-11T09:50:29.518 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.518 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.518 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:50:29.518 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.558857+0000", 2024-09-11T09:50:29.518 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.9502999999999999e-05 2024-09-11T09:50:29.518 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.518 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.518 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:50:29.518 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.558870+0000", 2024-09-11T09:50:29.518 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3020000000000001e-05 2024-09-11T09:50:29.518 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.518 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.518 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:50:29.518 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.558876+0000", 2024-09-11T09:50:29.519 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.7270000000000004e-06 2024-09-11T09:50:29.519 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.519 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.519 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:50:29.519 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.558934+0000", 2024-09-11T09:50:29.519 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.8100999999999998e-05 2024-09-11T09:50:29.519 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.519 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.519 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:50:29.519 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.558947+0000", 2024-09-11T09:50:29.519 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3269e-05 2024-09-11T09:50:29.519 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.519 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.519 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:50:29.520 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.559720+0000", 2024-09-11T09:50:29.520 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00077287600000000003 2024-09-11T09:50:29.520 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.520 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.520 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:50:29.520 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.559740+0000", 2024-09-11T09:50:29.520 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.9786e-05 2024-09-11T09:50:29.520 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:29.520 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:50:29.520 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:29.520 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.520 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.520 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4183.0:84 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'54, mlcod=13'50)", 2024-09-11T09:50:29.521 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:11.559267+0000", 2024-09-11T09:50:29.521 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 17.903933359, 2024-09-11T09:50:29.521 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00093990199999999995, 2024-09-11T09:50:29.521 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:50:29.521 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:50:29.521 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:29.521 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:50:29.521 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.521 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:50:29.522 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.559267+0000", 2024-09-11T09:50:29.522 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:29.522 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.522 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.522 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:50:29.522 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.559267+0000", 2024-09-11T09:50:29.522 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:29.522 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.522 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.522 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:50:29.522 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.559267+0000", 2024-09-11T09:50:29.522 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.2900000000000003e-07 2024-09-11T09:50:29.522 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.523 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.523 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:50:29.523 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.559322+0000", 2024-09-11T09:50:29.523 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.4429999999999999e-05 2024-09-11T09:50:29.523 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.523 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.523 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:50:29.523 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.559332+0000", 2024-09-11T09:50:29.523 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0473e-05 2024-09-11T09:50:29.523 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.523 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.523 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:50:29.524 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.559338+0000", 2024-09-11T09:50:29.524 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.0730000000000001e-06 2024-09-11T09:50:29.524 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.524 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.524 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:50:29.524 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.559373+0000", 2024-09-11T09:50:29.524 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.4626999999999999e-05 2024-09-11T09:50:29.524 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.524 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.524 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:50:29.524 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.559384+0000", 2024-09-11T09:50:29.524 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1307e-05 2024-09-11T09:50:29.524 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.525 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.525 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:50:29.525 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.560175+0000", 2024-09-11T09:50:29.525 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00079016399999999999 2024-09-11T09:50:29.525 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.525 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.525 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:50:29.525 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.560207+0000", 2024-09-11T09:50:29.525 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.2199000000000001e-05 2024-09-11T09:50:29.525 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:29.525 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:50:29.525 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:29.525 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.526 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.526 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4183.0:89 1.0 e13/12 1:30306672:devicehealth::main.db.0000000000000000:head v 13'59, mlcod=13'57)", 2024-09-11T09:50:29.526 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:11.563614+0000", 2024-09-11T09:50:29.526 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 17.899585738999999, 2024-09-11T09:50:29.526 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00091281399999999996, 2024-09-11T09:50:29.526 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:50:29.526 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:50:29.526 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:29.526 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:50:29.526 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.526 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:50:29.526 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.563614+0000", 2024-09-11T09:50:29.526 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:29.526 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.527 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.527 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:50:29.527 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.563614+0000", 2024-09-11T09:50:29.527 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:29.527 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.527 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.527 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:50:29.527 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.563615+0000", 2024-09-11T09:50:29.527 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.6899999999999998e-07 2024-09-11T09:50:29.527 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.527 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.527 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:50:29.527 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.563681+0000", 2024-09-11T09:50:29.528 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.5769e-05 2024-09-11T09:50:29.528 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.528 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.528 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:50:29.528 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.563690+0000", 2024-09-11T09:50:29.528 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.6800000000000005e-06 2024-09-11T09:50:29.528 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.528 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.528 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:50:29.528 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.563699+0000", 2024-09-11T09:50:29.528 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.3170000000000002e-06 2024-09-11T09:50:29.528 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.528 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.529 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:50:29.529 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.563734+0000", 2024-09-11T09:50:29.529 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.5023e-05 2024-09-11T09:50:29.529 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.529 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.529 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:50:29.529 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.563746+0000", 2024-09-11T09:50:29.529 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2661e-05 2024-09-11T09:50:29.529 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.529 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.529 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:50:29.529 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.564500+0000", 2024-09-11T09:50:29.529 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00075323100000000004 2024-09-11T09:50:29.529 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.530 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.530 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:50:29.530 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.564527+0000", 2024-09-11T09:50:29.530 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.7664000000000001e-05 2024-09-11T09:50:29.530 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:29.530 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:50:29.530 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:29.530 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.530 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.530 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4183.0:105 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'67, mlcod=13'64)", 2024-09-11T09:50:29.530 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:11.712146+0000", 2024-09-11T09:50:29.530 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 17.751053988999999, 2024-09-11T09:50:29.530 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0010916859999999999, 2024-09-11T09:50:29.531 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:50:29.531 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:50:29.531 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:29.531 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:50:29.531 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.531 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:50:29.531 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.712146+0000", 2024-09-11T09:50:29.531 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:29.531 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.531 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.531 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:50:29.531 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.712146+0000", 2024-09-11T09:50:29.531 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:29.531 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.532 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.532 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:50:29.532 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.712147+0000", 2024-09-11T09:50:29.532 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.8200000000000001e-07 2024-09-11T09:50:29.532 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.532 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.532 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:50:29.532 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.712155+0000", 2024-09-11T09:50:29.532 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.0309999999999998e-06 2024-09-11T09:50:29.532 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.532 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.532 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:50:29.532 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.712163+0000", 2024-09-11T09:50:29.533 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.5920000000000001e-06 2024-09-11T09:50:29.533 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.533 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.533 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:50:29.533 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.712169+0000", 2024-09-11T09:50:29.533 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.4299999999999997e-06 2024-09-11T09:50:29.533 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.533 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.533 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:50:29.533 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.712418+0000", 2024-09-11T09:50:29.533 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00024978499999999998 2024-09-11T09:50:29.533 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.533 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.533 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:50:29.534 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.712432+0000", 2024-09-11T09:50:29.534 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3190000000000001e-05 2024-09-11T09:50:29.534 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.534 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.534 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:50:29.534 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.713206+0000", 2024-09-11T09:50:29.534 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00077420199999999998 2024-09-11T09:50:29.534 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.534 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.534 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:50:29.534 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.713238+0000", 2024-09-11T09:50:29.534 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.2073999999999998e-05 2024-09-11T09:50:29.534 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:29.535 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:50:29.535 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:29.535 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.535 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.535 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4183.0:106 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'68, mlcod=13'64)", 2024-09-11T09:50:29.535 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:11.712624+0000", 2024-09-11T09:50:29.535 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 17.750576661, 2024-09-11T09:50:29.535 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0010009629999999999, 2024-09-11T09:50:29.535 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:50:29.535 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:50:29.535 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:29.535 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:50:29.535 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.535 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:50:29.536 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.712624+0000", 2024-09-11T09:50:29.536 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:29.536 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.536 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.536 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:50:29.536 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.712624+0000", 2024-09-11T09:50:29.536 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:29.536 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.536 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.536 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:50:29.536 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.712624+0000", 2024-09-11T09:50:29.536 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.4799999999999998e-07 2024-09-11T09:50:29.536 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.537 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.537 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:50:29.537 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.712689+0000", 2024-09-11T09:50:29.537 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.4488999999999999e-05 2024-09-11T09:50:29.537 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.537 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.537 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:50:29.537 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.712707+0000", 2024-09-11T09:50:29.537 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.8272000000000001e-05 2024-09-11T09:50:29.537 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.537 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.537 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:50:29.537 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.712713+0000", 2024-09-11T09:50:29.538 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.2040000000000002e-06 2024-09-11T09:50:29.538 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.538 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.538 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:50:29.538 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.712783+0000", 2024-09-11T09:50:29.538 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.9534000000000002e-05 2024-09-11T09:50:29.538 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.538 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.538 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:50:29.538 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.712811+0000", 2024-09-11T09:50:29.538 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.8620999999999999e-05 2024-09-11T09:50:29.538 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.538 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.538 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:50:29.539 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.713586+0000", 2024-09-11T09:50:29.539 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00077462600000000005 2024-09-11T09:50:29.539 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.539 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.539 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:50:29.539 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.713624+0000", 2024-09-11T09:50:29.539 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.8668999999999999e-05 2024-09-11T09:50:29.539 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:29.539 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:50:29.539 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:29.539 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.539 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.539 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4183.0:118 1.0 e13/12 1:30306672:devicehealth::main.db.0000000000000000:head v 13'80, mlcod=13'78)", 2024-09-11T09:50:29.540 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:11.726126+0000", 2024-09-11T09:50:29.540 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 17.737074569000001, 2024-09-11T09:50:29.540 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00087776200000000003, 2024-09-11T09:50:29.540 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:50:29.540 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:50:29.540 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:29.540 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:50:29.540 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.540 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:50:29.540 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.726126+0000", 2024-09-11T09:50:29.540 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:29.540 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.541 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.541 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:50:29.541 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.726126+0000", 2024-09-11T09:50:29.541 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:29.541 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.541 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.541 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:50:29.541 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.726126+0000", 2024-09-11T09:50:29.541 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.2099999999999996e-07 2024-09-11T09:50:29.541 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.541 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.541 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:50:29.541 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.726167+0000", 2024-09-11T09:50:29.541 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.0621999999999998e-05 2024-09-11T09:50:29.542 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.542 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.542 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:50:29.542 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.726177+0000", 2024-09-11T09:50:29.542 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0299000000000001e-05 2024-09-11T09:50:29.542 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.542 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.542 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:50:29.542 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.726183+0000", 2024-09-11T09:50:29.542 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.2079999999999997e-06 2024-09-11T09:50:29.542 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.542 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.542 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:50:29.542 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.726256+0000", 2024-09-11T09:50:29.543 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.2632000000000005e-05 2024-09-11T09:50:29.543 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.543 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.543 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:50:29.543 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.726268+0000", 2024-09-11T09:50:29.543 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2461e-05 2024-09-11T09:50:29.543 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.543 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.543 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:50:29.543 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.726977+0000", 2024-09-11T09:50:29.543 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00070853500000000005 2024-09-11T09:50:29.543 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.543 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.544 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:50:29.544 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.727003+0000", 2024-09-11T09:50:29.544 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.6384e-05 2024-09-11T09:50:29.544 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:29.544 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:50:29.544 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:29.544 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.544 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.544 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4183.0:134 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'92, mlcod=13'86)", 2024-09-11T09:50:29.544 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:12.035698+0000", 2024-09-11T09:50:29.544 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 17.427502402000002, 2024-09-11T09:50:29.544 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00089550700000000003, 2024-09-11T09:50:29.544 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:50:29.545 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:50:29.545 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:29.545 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:50:29.545 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.545 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:50:29.545 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.035698+0000", 2024-09-11T09:50:29.545 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:29.545 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.545 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.545 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:50:29.545 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.035698+0000", 2024-09-11T09:50:29.545 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:29.545 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.546 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.546 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:50:29.546 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.035698+0000", 2024-09-11T09:50:29.546 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.0900000000000002e-07 2024-09-11T09:50:29.546 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.546 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.546 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:50:29.546 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.035758+0000", 2024-09-11T09:50:29.546 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.9481999999999997e-05 2024-09-11T09:50:29.546 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.546 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.546 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:50:29.546 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.035766+0000", 2024-09-11T09:50:29.547 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.3760000000000008e-06 2024-09-11T09:50:29.547 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.547 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.547 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:50:29.547 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.035772+0000", 2024-09-11T09:50:29.547 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.8930000000000002e-06 2024-09-11T09:50:29.547 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.547 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.547 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:50:29.547 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.035807+0000", 2024-09-11T09:50:29.547 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.4975999999999999e-05 2024-09-11T09:50:29.547 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.547 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.547 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:50:29.548 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.035817+0000", 2024-09-11T09:50:29.548 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0314e-05 2024-09-11T09:50:29.548 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.548 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.548 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:50:29.548 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.036542+0000", 2024-09-11T09:50:29.548 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00072434400000000003 2024-09-11T09:50:29.548 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.548 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.548 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:50:29.548 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.036593+0000", 2024-09-11T09:50:29.548 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.1613000000000002e-05 2024-09-11T09:50:29.548 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:29.549 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:50:29.549 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:29.549 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.549 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.549 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4183.0:136 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'94, mlcod=13'86)", 2024-09-11T09:50:29.549 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:12.036591+0000", 2024-09-11T09:50:29.552 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 17.426609192000001, 2024-09-11T09:50:29.552 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00090637900000000002, 2024-09-11T09:50:29.552 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:50:29.552 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:50:29.552 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:29.552 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:50:29.552 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.552 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:50:29.552 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.036591+0000", 2024-09-11T09:50:29.552 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:29.553 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.553 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.553 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:50:29.553 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.036591+0000", 2024-09-11T09:50:29.553 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:29.553 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.553 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.553 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:50:29.553 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.036592+0000", 2024-09-11T09:50:29.553 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.37e-07 2024-09-11T09:50:29.553 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.553 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.553 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:50:29.554 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.036600+0000", 2024-09-11T09:50:29.554 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.6910000000000003e-06 2024-09-11T09:50:29.554 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.554 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.554 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:50:29.554 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.036609+0000", 2024-09-11T09:50:29.554 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.0229999999999998e-06 2024-09-11T09:50:29.554 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.554 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.554 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:50:29.554 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.036617+0000", 2024-09-11T09:50:29.554 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.6979999999999998e-06 2024-09-11T09:50:29.554 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.554 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.555 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:50:29.555 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.036658+0000", 2024-09-11T09:50:29.555 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.0692000000000003e-05 2024-09-11T09:50:29.555 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.555 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.555 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:50:29.555 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.036670+0000", 2024-09-11T09:50:29.555 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2109e-05 2024-09-11T09:50:29.555 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.555 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.555 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:50:29.555 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.037475+0000", 2024-09-11T09:50:29.556 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00080487199999999999 2024-09-11T09:50:29.556 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.556 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.556 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:50:29.556 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.037497+0000", 2024-09-11T09:50:29.556 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.2557000000000001e-05 2024-09-11T09:50:29.556 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:29.556 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:50:29.556 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:29.556 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.556 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.556 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4183.0:158 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'112, mlcod=13'108)", 2024-09-11T09:50:29.556 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:12.455073+0000", 2024-09-11T09:50:29.557 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 17.008126939, 2024-09-11T09:50:29.557 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00090654500000000003, 2024-09-11T09:50:29.557 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:50:29.557 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:50:29.557 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:29.557 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:50:29.557 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.557 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:50:29.557 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.455073+0000", 2024-09-11T09:50:29.557 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:29.557 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.557 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.557 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:50:29.557 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.455073+0000", 2024-09-11T09:50:29.558 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:29.558 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.558 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.558 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:50:29.558 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.455074+0000", 2024-09-11T09:50:29.558 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.32e-07 2024-09-11T09:50:29.558 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.558 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.558 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:50:29.558 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.455126+0000", 2024-09-11T09:50:29.558 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.1898999999999999e-05 2024-09-11T09:50:29.558 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.558 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.559 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:50:29.559 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.455136+0000", 2024-09-11T09:50:29.559 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0651e-05 2024-09-11T09:50:29.559 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.559 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.559 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:50:29.559 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.455142+0000", 2024-09-11T09:50:29.559 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.0020000000000003e-06 2024-09-11T09:50:29.559 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.559 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.559 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:50:29.559 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.455192+0000", 2024-09-11T09:50:29.559 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.0164000000000003e-05 2024-09-11T09:50:29.559 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.560 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.560 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:50:29.560 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.455204+0000", 2024-09-11T09:50:29.560 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1178999999999999e-05 2024-09-11T09:50:29.560 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.560 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.560 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:50:29.560 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.455957+0000", 2024-09-11T09:50:29.560 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00075362300000000003 2024-09-11T09:50:29.560 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.560 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.560 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:50:29.560 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.455980+0000", 2024-09-11T09:50:29.561 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.2595e-05 2024-09-11T09:50:29.561 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:29.561 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:50:29.561 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:29.561 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:29.561 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:50:29.561 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:50:29.561 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-11T09:50:29.563 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-11T09:50:29.563 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:50:29.563+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:50:29.663 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:50:29.664+0000 7fdec5849640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31907) UID: 0 2024-09-11T09:50:29.713 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:50:29.714 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [], 2024-09-11T09:50:29.714 INFO:teuthology.orchestra.run.smithi017.stdout: "num_ops": 0 2024-09-11T09:50:29.714 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:50:29.723 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-11T09:50:29.764 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:50:29.765+0000 7f3c96ec9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31909) UID: 0 2024-09-11T09:50:29.830 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:50:29.831 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [], 2024-09-11T09:50:29.831 INFO:teuthology.orchestra.run.smithi017.stdout: "complaint_time": 30, 2024-09-11T09:50:29.831 INFO:teuthology.orchestra.run.smithi017.stdout: "num_blocked_ops": 0 2024-09-11T09:50:29.831 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:50:29.840 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-11T09:50:29.865 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:50:29.866+0000 7f3c96ec9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31909) UID: 0 2024-09-11T09:50:29.948 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:50:29.948 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 20, 2024-09-11T09:50:29.948 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 600, 2024-09-11T09:50:29.948 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [ 2024-09-11T09:50:29.948 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.948 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4230.0:5 2.2 e17/15 2:5cea7035:::rbd_info:head v 17'1, mlcod=0'0)", 2024-09-11T09:50:29.948 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:19.494757+0000", 2024-09-11T09:50:29.948 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 10.454736184, 2024-09-11T09:50:29.948 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00082098399999999997, 2024-09-11T09:50:29.949 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:50:29.949 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:50:29.949 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:29.949 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:50:29.949 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.949 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:50:29.949 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:19.494757+0000", 2024-09-11T09:50:29.949 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:29.949 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.949 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.949 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:50:29.949 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:19.494757+0000", 2024-09-11T09:50:29.949 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:29.950 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.950 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.950 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:50:29.950 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:19.494757+0000", 2024-09-11T09:50:29.950 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.97e-07 2024-09-11T09:50:29.950 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.950 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.950 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:50:29.950 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:19.494763+0000", 2024-09-11T09:50:29.950 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.3709999999999996e-06 2024-09-11T09:50:29.950 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.950 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.950 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:50:29.951 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:19.494778+0000", 2024-09-11T09:50:29.951 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.4960000000000001e-05 2024-09-11T09:50:29.951 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.951 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.951 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:50:29.951 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:19.494801+0000", 2024-09-11T09:50:29.951 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.2388999999999999e-05 2024-09-11T09:50:29.951 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.951 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.951 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:50:29.951 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:19.494859+0000", 2024-09-11T09:50:29.951 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.7961999999999998e-05 2024-09-11T09:50:29.951 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.952 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.952 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:50:29.952 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:19.494883+0000", 2024-09-11T09:50:29.952 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.4488999999999999e-05 2024-09-11T09:50:29.952 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.952 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.952 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:50:29.952 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:19.495534+0000", 2024-09-11T09:50:29.952 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00065075500000000004 2024-09-11T09:50:29.952 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.952 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.952 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:50:29.952 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:19.495578+0000", 2024-09-11T09:50:29.952 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.3860999999999999e-05 2024-09-11T09:50:29.953 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:29.953 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:50:29.953 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:29.953 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.953 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.953 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4230.0:7 2.2 e18/15 2:5cea7035:::rbd_info:head v 18'2, mlcod=0'0)", 2024-09-11T09:50:29.953 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:20.500782+0000", 2024-09-11T09:50:29.953 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 9.4487113039999997, 2024-09-11T09:50:29.953 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000790185, 2024-09-11T09:50:29.953 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:50:29.953 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:50:29.953 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:29.954 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:50:29.954 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.954 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:50:29.954 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:20.500782+0000", 2024-09-11T09:50:29.954 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:29.954 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.954 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.954 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:50:29.954 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:20.500782+0000", 2024-09-11T09:50:29.954 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:29.954 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.954 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.954 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:50:29.954 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:20.500782+0000", 2024-09-11T09:50:29.955 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.3299999999999998e-07 2024-09-11T09:50:29.955 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.955 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.955 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:50:29.955 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:20.500799+0000", 2024-09-11T09:50:29.955 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6830999999999999e-05 2024-09-11T09:50:29.955 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.955 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.955 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:50:29.955 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:20.500812+0000", 2024-09-11T09:50:29.955 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3013999999999999e-05 2024-09-11T09:50:29.955 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.955 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.955 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:50:29.956 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:20.500824+0000", 2024-09-11T09:50:29.956 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2593e-05 2024-09-11T09:50:29.956 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.956 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.956 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:50:29.956 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:20.500895+0000", 2024-09-11T09:50:29.956 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.1018000000000003e-05 2024-09-11T09:50:29.956 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.956 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.956 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:50:29.956 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:20.500912+0000", 2024-09-11T09:50:29.956 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6484e-05 2024-09-11T09:50:29.956 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.957 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.957 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:50:29.957 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:20.501531+0000", 2024-09-11T09:50:29.957 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00061903399999999997 2024-09-11T09:50:29.957 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:29.957 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:29.960 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:50:29.960 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:20.501572+0000", 2024-09-11T09:50:29.960 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.0877999999999997e-05 2024-09-11T09:50:29.960 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:29.960 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:50:29.960 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:29.960 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:29.960 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:50:29.960 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:50:29.961 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-11T09:50:29.966 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:50:29.966+0000 7f3c96ec9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31909) UID: 0 2024-09-11T09:50:30.066 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-09-11T09:50:30.067+0000 7f7b6c7d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31904) UID: 0 2024-09-11T09:50:30.069 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:50:30.069 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [], 2024-09-11T09:50:30.069 INFO:teuthology.orchestra.run.smithi017.stdout: "num_ops": 0 2024-09-11T09:50:30.069 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:50:30.078 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-11T09:50:30.169 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:50:30.169+0000 7f3c96ec9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31909) UID: 0 2024-09-11T09:50:30.185 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:50:30.185 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [], 2024-09-11T09:50:30.185 INFO:teuthology.orchestra.run.smithi017.stdout: "complaint_time": 30, 2024-09-11T09:50:30.185 INFO:teuthology.orchestra.run.smithi017.stdout: "num_blocked_ops": 0 2024-09-11T09:50:30.185 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:50:30.194 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-11T09:50:30.270 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:50:30.271+0000 7fdec5849640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31907) UID: 0 2024-09-11T09:50:30.301 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:50:30.301 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 20, 2024-09-11T09:50:30.301 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 600, 2024-09-11T09:50:30.302 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [ 2024-09-11T09:50:30.302 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:30.302 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4230.0:1 2.3 e16/15 2:c4c92e5a:::rbd_trash:head v 16'1, mlcod=0'0)", 2024-09-11T09:50:30.302 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:18.494378+0000", 2024-09-11T09:50:30.302 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 11.80863035, 2024-09-11T09:50:30.302 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00089752199999999999, 2024-09-11T09:50:30.302 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:50:30.302 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:50:30.302 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:30.302 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:50:30.302 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:30.302 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:50:30.303 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:18.494378+0000", 2024-09-11T09:50:30.303 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:30.303 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:30.303 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:30.303 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:50:30.303 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:18.494378+0000", 2024-09-11T09:50:30.303 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:30.303 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:30.303 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:30.303 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:50:30.303 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:18.494378+0000", 2024-09-11T09:50:30.303 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.9099999999999999e-07 2024-09-11T09:50:30.303 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:30.303 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:30.304 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:50:30.304 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:18.494385+0000", 2024-09-11T09:50:30.304 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.8129999999999998e-06 2024-09-11T09:50:30.304 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:30.304 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:30.304 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:50:30.304 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:18.494416+0000", 2024-09-11T09:50:30.304 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.0654000000000002e-05 2024-09-11T09:50:30.304 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:30.304 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:30.304 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:50:30.304 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:18.494430+0000", 2024-09-11T09:50:30.304 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3754000000000001e-05 2024-09-11T09:50:30.305 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:30.305 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:30.305 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:50:30.305 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:18.494524+0000", 2024-09-11T09:50:30.305 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.4896999999999996e-05 2024-09-11T09:50:30.305 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:30.305 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:30.305 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:50:30.305 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:18.494558+0000", 2024-09-11T09:50:30.305 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.3900999999999999e-05 2024-09-11T09:50:30.305 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:30.305 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:30.305 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:50:30.306 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:18.495231+0000", 2024-09-11T09:50:30.306 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00067233500000000003 2024-09-11T09:50:30.306 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:30.306 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:30.306 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:50:30.306 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:18.495275+0000", 2024-09-11T09:50:30.306 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.4777000000000002e-05 2024-09-11T09:50:30.306 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:30.306 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:50:30.306 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:30.306 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:30.306 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:50:30.306 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:50:30.310 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-11T09:50:30.371 INFO:tasks.ceph.osd.2.smithi017.stderr:2024-09-11T09:50:30.372+0000 7fd20b170640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31672) UID: 0 2024-09-11T09:50:30.425 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-11T09:50:30.425 INFO:teuthology.orchestra.run.smithi096.stdout: "ops": [], 2024-09-11T09:50:30.425 INFO:teuthology.orchestra.run.smithi096.stdout: "num_ops": 0 2024-09-11T09:50:30.425 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-11T09:50:30.435 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-11T09:50:30.472 INFO:tasks.ceph.osd.0.smithi017.stderr:2024-09-11T09:50:30.473+0000 7fb73c0f1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31665) UID: 0 2024-09-11T09:50:30.525 INFO:teuthology.orchestra.run.smithi017.stderr:nodeep-scrub is set 2024-09-11T09:50:30.541 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-11T09:50:30.541 INFO:teuthology.orchestra.run.smithi096.stdout: "ops": [], 2024-09-11T09:50:30.541 INFO:teuthology.orchestra.run.smithi096.stdout: "complaint_time": 30, 2024-09-11T09:50:30.541 INFO:teuthology.orchestra.run.smithi096.stdout: "num_blocked_ops": 0 2024-09-11T09:50:30.541 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-11T09:50:30.550 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-11T09:50:30.593 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:50:30.594+0000 7fdec5849640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31907) UID: 0 2024-09-11T09:50:30.655 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-11T09:50:30.655 INFO:teuthology.orchestra.run.smithi096.stdout: "size": 20, 2024-09-11T09:50:30.655 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 600, 2024-09-11T09:50:30.655 INFO:teuthology.orchestra.run.smithi096.stdout: "ops": [ 2024-09-11T09:50:30.655 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.655 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:74 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:50:30.656 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.539067+0000", 2024-09-11T09:50:30.656 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 19.115588367000001, 2024-09-11T09:50:30.656 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.006945957, 2024-09-11T09:50:30.656 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:50:30.656 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:50:30.656 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:30.656 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:50:30.656 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:50:30.656 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:50:30.656 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 74 2024-09-11T09:50:30.656 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.656 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:50:30.656 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.657 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:50:30.657 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.539067+0000", 2024-09-11T09:50:30.657 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:30.657 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.657 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.657 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:50:30.657 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.539067+0000", 2024-09-11T09:50:30.657 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:30.657 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.657 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.657 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:50:30.657 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.539068+0000", 2024-09-11T09:50:30.657 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.6000000000000003e-07 2024-09-11T09:50:30.658 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.658 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.658 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:50:30.658 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.539072+0000", 2024-09-11T09:50:30.658 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.0450000000000002e-06 2024-09-11T09:50:30.658 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.658 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.658 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:50:30.658 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.539078+0000", 2024-09-11T09:50:30.658 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.6889999999999998e-06 2024-09-11T09:50:30.658 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.658 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.658 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:50:30.658 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.539083+0000", 2024-09-11T09:50:30.659 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.7180000000000004e-06 2024-09-11T09:50:30.659 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.659 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.659 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:50:30.659 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.540230+0000", 2024-09-11T09:50:30.659 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0011466250000000001 2024-09-11T09:50:30.659 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.659 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.659 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:50:30.659 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.540279+0000", 2024-09-11T09:50:30.659 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.9667999999999999e-05 2024-09-11T09:50:30.659 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.659 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.660 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:50:30.660 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.540357+0000", 2024-09-11T09:50:30.660 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.7863999999999999e-05 2024-09-11T09:50:30.660 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.660 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.660 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:50:30.660 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.541086+0000", 2024-09-11T09:50:30.660 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00072921100000000001 2024-09-11T09:50:30.660 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.660 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.660 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:50:30.660 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.545949+0000", 2024-09-11T09:50:30.660 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0048626040000000004 2024-09-11T09:50:30.660 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.661 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.661 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:50:30.661 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.545989+0000", 2024-09-11T09:50:30.661 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.0259000000000002e-05 2024-09-11T09:50:30.661 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.661 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.661 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:50:30.661 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.546013+0000", 2024-09-11T09:50:30.661 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.3614000000000001e-05 2024-09-11T09:50:30.661 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:30.661 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:50:30.661 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:30.662 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.662 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.662 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:108 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:50:30.662 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.711510+0000", 2024-09-11T09:50:30.662 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 18.943145195, 2024-09-11T09:50:30.662 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0054615649999999998, 2024-09-11T09:50:30.662 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:50:30.662 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:50:30.662 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:30.662 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:50:30.662 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:50:30.662 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:50:30.662 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 108 2024-09-11T09:50:30.662 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.663 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:50:30.663 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.663 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:50:30.663 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711510+0000", 2024-09-11T09:50:30.663 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:30.663 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.663 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.663 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:50:30.663 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711510+0000", 2024-09-11T09:50:30.663 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:30.663 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.663 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.663 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:50:30.663 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711511+0000", 2024-09-11T09:50:30.664 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.1999999999999998e-07 2024-09-11T09:50:30.664 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.664 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.664 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:50:30.664 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711516+0000", 2024-09-11T09:50:30.664 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.4140000000000002e-06 2024-09-11T09:50:30.664 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.664 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.664 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:50:30.664 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711527+0000", 2024-09-11T09:50:30.664 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.0169e-05 2024-09-11T09:50:30.664 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.664 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.665 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:50:30.665 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711535+0000", 2024-09-11T09:50:30.665 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.3299999999999999e-06 2024-09-11T09:50:30.665 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.665 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.665 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:50:30.665 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.713763+0000", 2024-09-11T09:50:30.665 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0022281079999999999 2024-09-11T09:50:30.665 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.665 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.665 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:50:30.665 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.713812+0000", 2024-09-11T09:50:30.665 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.8544000000000001e-05 2024-09-11T09:50:30.666 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.666 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.666 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:50:30.666 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.713909+0000", 2024-09-11T09:50:30.666 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.7113000000000003e-05 2024-09-11T09:50:30.666 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.666 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.666 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:50:30.666 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.716873+0000", 2024-09-11T09:50:30.666 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0029642710000000001 2024-09-11T09:50:30.666 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.666 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.666 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:50:30.666 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.716918+0000", 2024-09-11T09:50:30.667 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.5447e-05 2024-09-11T09:50:30.667 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.667 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.667 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:50:30.667 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.716947+0000", 2024-09-11T09:50:30.667 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.8181e-05 2024-09-11T09:50:30.667 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.667 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.667 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:50:30.667 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.716972+0000", 2024-09-11T09:50:30.667 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.5068000000000001e-05 2024-09-11T09:50:30.667 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:30.667 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:50:30.668 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:30.668 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.668 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.668 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:109 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:50:30.668 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.711555+0000", 2024-09-11T09:50:30.668 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 18.943099946, 2024-09-11T09:50:30.668 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.005479902, 2024-09-11T09:50:30.668 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:50:30.668 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:50:30.668 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:30.668 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:50:30.668 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:50:30.668 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:50:30.669 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 109 2024-09-11T09:50:30.669 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.669 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:50:30.669 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.669 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:50:30.669 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711555+0000", 2024-09-11T09:50:30.669 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:30.669 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.669 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.669 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:50:30.669 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711555+0000", 2024-09-11T09:50:30.669 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:30.669 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.670 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.670 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:50:30.670 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711556+0000", 2024-09-11T09:50:30.670 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.3200000000000005e-07 2024-09-11T09:50:30.670 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.670 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.670 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:50:30.670 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711601+0000", 2024-09-11T09:50:30.670 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.5349000000000002e-05 2024-09-11T09:50:30.670 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.670 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.670 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:50:30.670 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711612+0000", 2024-09-11T09:50:30.670 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.0701e-05 2024-09-11T09:50:30.671 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.671 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.671 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:50:30.671 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711619+0000", 2024-09-11T09:50:30.671 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.7610000000000002e-06 2024-09-11T09:50:30.671 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.671 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.671 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:50:30.671 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.714371+0000", 2024-09-11T09:50:30.671 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0027526260000000002 2024-09-11T09:50:30.671 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.671 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.671 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:50:30.671 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.714417+0000", 2024-09-11T09:50:30.672 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.5893999999999998e-05 2024-09-11T09:50:30.672 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.672 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.672 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:50:30.672 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.714496+0000", 2024-09-11T09:50:30.672 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.8539000000000001e-05 2024-09-11T09:50:30.672 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.672 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.672 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:50:30.672 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.716910+0000", 2024-09-11T09:50:30.672 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0024137640000000001 2024-09-11T09:50:30.672 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.672 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.672 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:50:30.673 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.716989+0000", 2024-09-11T09:50:30.673 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.9198999999999998e-05 2024-09-11T09:50:30.673 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.673 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.673 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:50:30.673 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.717016+0000", 2024-09-11T09:50:30.673 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.6699e-05 2024-09-11T09:50:30.673 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.673 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.673 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:50:30.673 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.717035+0000", 2024-09-11T09:50:30.673 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.9738e-05 2024-09-11T09:50:30.673 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:30.673 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:50:30.673 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:30.674 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.674 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.674 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:110 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:50:30.674 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.711636+0000", 2024-09-11T09:50:30.674 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 18.943019486000001, 2024-09-11T09:50:30.674 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0067613070000000003, 2024-09-11T09:50:30.674 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:50:30.674 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:50:30.674 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:30.674 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:50:30.674 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:50:30.674 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:50:30.674 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 110 2024-09-11T09:50:30.674 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.675 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:50:30.675 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.675 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:50:30.675 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711636+0000", 2024-09-11T09:50:30.675 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:30.675 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.675 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.675 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:50:30.675 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711636+0000", 2024-09-11T09:50:30.675 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:30.675 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.675 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.675 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:50:30.675 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711637+0000", 2024-09-11T09:50:30.675 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.9999999999999997e-07 2024-09-11T09:50:30.675 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.676 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.676 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:50:30.676 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711641+0000", 2024-09-11T09:50:30.676 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.2570000000000001e-06 2024-09-11T09:50:30.676 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.676 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.676 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:50:30.676 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711651+0000", 2024-09-11T09:50:30.676 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.0285999999999999e-05 2024-09-11T09:50:30.676 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.676 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.676 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:50:30.676 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711658+0000", 2024-09-11T09:50:30.676 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.3540000000000002e-06 2024-09-11T09:50:30.677 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.677 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.677 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:50:30.677 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.716475+0000", 2024-09-11T09:50:30.677 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0048162819999999999 2024-09-11T09:50:30.677 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.677 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.677 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:50:30.677 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.716526+0000", 2024-09-11T09:50:30.677 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.0905000000000001e-05 2024-09-11T09:50:30.677 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.677 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.677 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:50:30.677 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.716601+0000", 2024-09-11T09:50:30.678 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.5283000000000004e-05 2024-09-11T09:50:30.678 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.678 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.678 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:50:30.678 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.717933+0000", 2024-09-11T09:50:30.678 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001331622 2024-09-11T09:50:30.678 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.678 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.678 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:50:30.678 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718332+0000", 2024-09-11T09:50:30.678 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00039909899999999998 2024-09-11T09:50:30.678 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.678 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.678 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:50:30.678 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718363+0000", 2024-09-11T09:50:30.678 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.1636e-05 2024-09-11T09:50:30.679 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.679 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.679 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:50:30.679 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718397+0000", 2024-09-11T09:50:30.679 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.3883000000000001e-05 2024-09-11T09:50:30.679 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:30.679 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:50:30.679 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:30.679 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.679 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.679 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:111 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:50:30.679 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.711668+0000", 2024-09-11T09:50:30.679 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 18.942987056, 2024-09-11T09:50:30.679 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0077145570000000004, 2024-09-11T09:50:30.679 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:50:30.680 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:50:30.680 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:30.680 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:50:30.680 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:50:30.680 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:50:30.680 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 111 2024-09-11T09:50:30.680 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.680 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:50:30.680 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.680 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:50:30.680 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711668+0000", 2024-09-11T09:50:30.680 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:30.680 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.680 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.681 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:50:30.681 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711668+0000", 2024-09-11T09:50:30.681 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:30.681 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.681 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.681 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:50:30.681 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711669+0000", 2024-09-11T09:50:30.681 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.99e-07 2024-09-11T09:50:30.681 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.681 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.681 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:50:30.681 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711673+0000", 2024-09-11T09:50:30.681 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.7440000000000001e-06 2024-09-11T09:50:30.681 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.682 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.682 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:50:30.682 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711686+0000", 2024-09-11T09:50:30.682 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.3317e-05 2024-09-11T09:50:30.682 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.682 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.682 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:50:30.682 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711697+0000", 2024-09-11T09:50:30.682 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1022000000000001e-05 2024-09-11T09:50:30.682 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.682 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.682 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:50:30.682 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.717065+0000", 2024-09-11T09:50:30.683 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.005367531 2024-09-11T09:50:30.683 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.683 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.683 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:50:30.683 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.717114+0000", 2024-09-11T09:50:30.683 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.9707999999999999e-05 2024-09-11T09:50:30.683 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.683 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.683 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:50:30.683 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.717190+0000", 2024-09-11T09:50:30.683 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.5394000000000004e-05 2024-09-11T09:50:30.683 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.683 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.683 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:50:30.684 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718793+0000", 2024-09-11T09:50:30.684 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001603653 2024-09-11T09:50:30.684 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.684 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.684 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:50:30.684 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719329+0000", 2024-09-11T09:50:30.684 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00053543799999999995 2024-09-11T09:50:30.684 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.684 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.684 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:50:30.684 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719359+0000", 2024-09-11T09:50:30.684 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.0420999999999999e-05 2024-09-11T09:50:30.684 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.684 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.684 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:50:30.685 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719383+0000", 2024-09-11T09:50:30.685 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.3730000000000001e-05 2024-09-11T09:50:30.685 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:30.685 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:50:30.685 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:30.685 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.685 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.685 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:112 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:50:30.685 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.711712+0000", 2024-09-11T09:50:30.685 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 18.942943496000002, 2024-09-11T09:50:30.685 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.008162084, 2024-09-11T09:50:30.685 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:50:30.685 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:50:30.685 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:30.686 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:50:30.686 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:50:30.686 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:50:30.686 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 112 2024-09-11T09:50:30.686 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.686 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:50:30.686 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.686 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:50:30.686 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711712+0000", 2024-09-11T09:50:30.686 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:30.686 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.686 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.686 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:50:30.686 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711712+0000", 2024-09-11T09:50:30.686 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:30.687 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.687 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.687 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:50:30.687 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711712+0000", 2024-09-11T09:50:30.687 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.6000000000000004e-07 2024-09-11T09:50:30.687 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.687 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.687 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:50:30.687 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711808+0000", 2024-09-11T09:50:30.687 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.5977999999999995e-05 2024-09-11T09:50:30.687 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.687 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.687 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:50:30.687 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711819+0000", 2024-09-11T09:50:30.688 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.0755e-05 2024-09-11T09:50:30.688 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.688 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.688 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:50:30.688 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711826+0000", 2024-09-11T09:50:30.688 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.7970000000000003e-06 2024-09-11T09:50:30.688 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.688 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.688 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:50:30.688 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.717435+0000", 2024-09-11T09:50:30.688 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0056090239999999998 2024-09-11T09:50:30.688 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.688 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.689 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:50:30.689 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.717496+0000", 2024-09-11T09:50:30.689 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.0543e-05 2024-09-11T09:50:30.689 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.689 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.689 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:50:30.689 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.717591+0000", 2024-09-11T09:50:30.689 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.5667000000000004e-05 2024-09-11T09:50:30.689 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.689 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.689 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:50:30.689 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718831+0000", 2024-09-11T09:50:30.689 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001240204 2024-09-11T09:50:30.690 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.690 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.690 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:50:30.690 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719819+0000", 2024-09-11T09:50:30.690 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00098751500000000001 2024-09-11T09:50:30.690 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.690 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.690 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:50:30.690 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719850+0000", 2024-09-11T09:50:30.690 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.1535999999999998e-05 2024-09-11T09:50:30.690 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.690 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.690 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:50:30.690 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719874+0000", 2024-09-11T09:50:30.690 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.3504999999999999e-05 2024-09-11T09:50:30.691 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:30.691 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:50:30.691 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:30.691 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.691 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.691 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:113 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:50:30.691 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.711842+0000", 2024-09-11T09:50:30.691 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 18.942812997000001, 2024-09-11T09:50:30.691 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0081160099999999999, 2024-09-11T09:50:30.691 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:50:30.691 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:50:30.691 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:30.691 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:50:30.692 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:50:30.692 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:50:30.692 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 113 2024-09-11T09:50:30.692 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.692 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:50:30.692 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.692 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:50:30.692 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711842+0000", 2024-09-11T09:50:30.692 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:30.692 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.692 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.692 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:50:30.692 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711842+0000", 2024-09-11T09:50:30.693 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:30.693 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.693 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.693 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:50:30.693 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711843+0000", 2024-09-11T09:50:30.693 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.4399999999999999e-07 2024-09-11T09:50:30.693 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.693 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.693 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:50:30.693 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711848+0000", 2024-09-11T09:50:30.693 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.4159999999999997e-06 2024-09-11T09:50:30.693 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.693 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.693 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:50:30.694 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711856+0000", 2024-09-11T09:50:30.694 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.7870000000000007e-06 2024-09-11T09:50:30.694 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.694 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.694 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:50:30.694 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711862+0000", 2024-09-11T09:50:30.694 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.3689999999999998e-06 2024-09-11T09:50:30.694 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.694 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.694 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:50:30.694 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.717965+0000", 2024-09-11T09:50:30.694 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0061028899999999997 2024-09-11T09:50:30.694 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.695 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.695 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:50:30.695 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718023+0000", 2024-09-11T09:50:30.695 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.8313000000000002e-05 2024-09-11T09:50:30.695 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.695 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.695 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:50:30.695 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718109+0000", 2024-09-11T09:50:30.695 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.6205000000000006e-05 2024-09-11T09:50:30.695 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.695 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.695 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:50:30.695 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719808+0000", 2024-09-11T09:50:30.695 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0016990060000000001 2024-09-11T09:50:30.696 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.696 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.696 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:50:30.696 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719883+0000", 2024-09-11T09:50:30.696 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.4624999999999998e-05 2024-09-11T09:50:30.696 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.696 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.696 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:50:30.696 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719910+0000", 2024-09-11T09:50:30.696 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.7512e-05 2024-09-11T09:50:30.696 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.696 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.696 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:50:30.697 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719958+0000", 2024-09-11T09:50:30.697 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.8143000000000003e-05 2024-09-11T09:50:30.697 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:30.697 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:50:30.697 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:30.697 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.697 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.697 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:50:30.697 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.711887+0000", 2024-09-11T09:50:30.697 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 18.942768383000001, 2024-09-11T09:50:30.697 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0086063589999999992, 2024-09-11T09:50:30.697 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:50:30.697 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:50:30.698 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:30.698 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:50:30.698 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:50:30.698 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:50:30.698 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 114 2024-09-11T09:50:30.698 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.698 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:50:30.698 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.698 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:50:30.698 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711887+0000", 2024-09-11T09:50:30.698 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:30.698 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.698 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.699 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:50:30.699 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711887+0000", 2024-09-11T09:50:30.699 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:30.699 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.699 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.699 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:50:30.699 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711887+0000", 2024-09-11T09:50:30.699 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.1500000000000005e-07 2024-09-11T09:50:30.699 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.699 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.699 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:50:30.699 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711897+0000", 2024-09-11T09:50:30.699 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.1900000000000001e-06 2024-09-11T09:50:30.699 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.700 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.700 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:50:30.700 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711902+0000", 2024-09-11T09:50:30.700 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.2349999999999999e-06 2024-09-11T09:50:30.700 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.700 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.700 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:50:30.700 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711906+0000", 2024-09-11T09:50:30.700 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.7129999999999999e-06 2024-09-11T09:50:30.700 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.700 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.700 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:50:30.700 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718429+0000", 2024-09-11T09:50:30.701 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0065231960000000002 2024-09-11T09:50:30.701 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.701 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.701 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:50:30.701 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718480+0000", 2024-09-11T09:50:30.701 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.1161e-05 2024-09-11T09:50:30.701 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.701 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.701 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:50:30.701 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718558+0000", 2024-09-11T09:50:30.701 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.7519000000000001e-05 2024-09-11T09:50:30.701 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.701 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.701 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:50:30.702 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720038+0000", 2024-09-11T09:50:30.702 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001480016 2024-09-11T09:50:30.702 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.702 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.702 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:50:30.702 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720417+0000", 2024-09-11T09:50:30.702 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00037943699999999999 2024-09-11T09:50:30.702 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.702 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.702 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:50:30.702 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720452+0000", 2024-09-11T09:50:30.702 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.4595000000000003e-05 2024-09-11T09:50:30.702 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.702 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.702 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:50:30.703 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720493+0000", 2024-09-11T09:50:30.703 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.1782000000000001e-05 2024-09-11T09:50:30.703 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:30.703 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:50:30.703 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:30.703 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.703 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.703 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:50:30.703 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.711912+0000", 2024-09-11T09:50:30.703 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 18.942743713999999, 2024-09-11T09:50:30.703 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0086864750000000008, 2024-09-11T09:50:30.703 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:50:30.703 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:50:30.704 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:30.704 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:50:30.704 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:50:30.704 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:50:30.704 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 115 2024-09-11T09:50:30.704 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.704 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:50:30.704 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.704 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:50:30.704 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711912+0000", 2024-09-11T09:50:30.704 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:30.704 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.704 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.704 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:50:30.705 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711912+0000", 2024-09-11T09:50:30.705 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:30.705 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.705 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.705 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:50:30.705 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711912+0000", 2024-09-11T09:50:30.705 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.3900000000000001e-07 2024-09-11T09:50:30.705 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.705 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.705 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:50:30.705 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711969+0000", 2024-09-11T09:50:30.705 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.7501999999999999e-05 2024-09-11T09:50:30.705 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.705 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.706 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:50:30.706 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711986+0000", 2024-09-11T09:50:30.706 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.6402999999999999e-05 2024-09-11T09:50:30.706 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.706 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.706 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:50:30.706 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711990+0000", 2024-09-11T09:50:30.706 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.929e-06 2024-09-11T09:50:30.706 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.706 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.706 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:50:30.706 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718861+0000", 2024-09-11T09:50:30.706 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0068710530000000002 2024-09-11T09:50:30.706 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.706 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.706 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:50:30.707 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718908+0000", 2024-09-11T09:50:30.707 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.7431e-05 2024-09-11T09:50:30.707 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.707 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.707 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:50:30.707 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718984+0000", 2024-09-11T09:50:30.707 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.5500999999999994e-05 2024-09-11T09:50:30.707 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.707 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.707 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:50:30.707 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720500+0000", 2024-09-11T09:50:30.707 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0015158820000000001 2024-09-11T09:50:30.707 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.707 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.707 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:50:30.707 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720536+0000", 2024-09-11T09:50:30.708 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.6118000000000002e-05 2024-09-11T09:50:30.708 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.708 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.708 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:50:30.708 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720570+0000", 2024-09-11T09:50:30.708 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.4387999999999997e-05 2024-09-11T09:50:30.708 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.708 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.708 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:50:30.708 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720598+0000", 2024-09-11T09:50:30.708 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.8028999999999999e-05 2024-09-11T09:50:30.708 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:30.708 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:50:30.708 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:30.708 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.709 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.709 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:50:30.709 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.711998+0000", 2024-09-11T09:50:30.709 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 18.942657534999999, 2024-09-11T09:50:30.709 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.010785813999999999, 2024-09-11T09:50:30.709 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:50:30.709 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:50:30.709 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:30.709 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:50:30.709 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:50:30.709 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:50:30.709 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 116 2024-09-11T09:50:30.709 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.709 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:50:30.709 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.710 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:50:30.710 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711998+0000", 2024-09-11T09:50:30.710 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:30.710 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.710 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.710 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:50:30.710 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711998+0000", 2024-09-11T09:50:30.710 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:30.710 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.710 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.710 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:50:30.710 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711998+0000", 2024-09-11T09:50:30.710 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.3300000000000003e-07 2024-09-11T09:50:30.710 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.711 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.711 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:50:30.711 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712001+0000", 2024-09-11T09:50:30.711 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.9349999999999999e-06 2024-09-11T09:50:30.711 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.711 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.711 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:50:30.711 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712006+0000", 2024-09-11T09:50:30.711 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.1630000000000004e-06 2024-09-11T09:50:30.711 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.711 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.711 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:50:30.711 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712010+0000", 2024-09-11T09:50:30.711 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.574e-06 2024-09-11T09:50:30.711 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.712 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.712 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:50:30.712 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719415+0000", 2024-09-11T09:50:30.712 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0074049099999999998 2024-09-11T09:50:30.712 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.712 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.712 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:50:30.712 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719465+0000", 2024-09-11T09:50:30.712 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.0129000000000001e-05 2024-09-11T09:50:30.712 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.712 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.712 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:50:30.712 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719545+0000", 2024-09-11T09:50:30.712 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.0253999999999994e-05 2024-09-11T09:50:30.713 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.713 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.713 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:50:30.713 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720636+0000", 2024-09-11T09:50:30.713 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0010909330000000001 2024-09-11T09:50:30.713 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.713 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.713 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:50:30.713 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.722707+0000", 2024-09-11T09:50:30.713 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0020713300000000001 2024-09-11T09:50:30.713 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.713 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.713 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:50:30.714 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.722743+0000", 2024-09-11T09:50:30.714 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.5351000000000001e-05 2024-09-11T09:50:30.714 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.714 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.714 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:50:30.714 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.722784+0000", 2024-09-11T09:50:30.714 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.0802e-05 2024-09-11T09:50:30.714 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:30.714 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:50:30.714 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:30.714 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.714 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.714 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:50:30.714 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.712014+0000", 2024-09-11T09:50:30.714 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 18.942641495, 2024-09-11T09:50:30.714 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.013420938, 2024-09-11T09:50:30.715 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:50:30.715 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:50:30.715 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:30.715 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:50:30.715 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:50:30.715 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:50:30.715 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 117 2024-09-11T09:50:30.715 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.715 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:50:30.715 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.715 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:50:30.715 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712014+0000", 2024-09-11T09:50:30.715 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:30.715 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.715 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.716 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:50:30.716 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712014+0000", 2024-09-11T09:50:30.716 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:30.716 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.716 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.716 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:50:30.716 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712014+0000", 2024-09-11T09:50:30.716 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.2000000000000001e-07 2024-09-11T09:50:30.716 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.716 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.716 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:50:30.716 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712016+0000", 2024-09-11T09:50:30.716 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.2309999999999999e-06 2024-09-11T09:50:30.716 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.716 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.716 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:50:30.717 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712020+0000", 2024-09-11T09:50:30.717 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.3830000000000001e-06 2024-09-11T09:50:30.717 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.717 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.717 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:50:30.717 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712023+0000", 2024-09-11T09:50:30.717 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.3340000000000002e-06 2024-09-11T09:50:30.717 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.717 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.717 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:50:30.717 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720083+0000", 2024-09-11T09:50:30.717 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0080601189999999993 2024-09-11T09:50:30.717 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.718 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.718 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:50:30.718 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720125+0000", 2024-09-11T09:50:30.718 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.2029e-05 2024-09-11T09:50:30.718 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.718 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.718 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:50:30.718 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720205+0000", 2024-09-11T09:50:30.718 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.0010000000000001e-05 2024-09-11T09:50:30.718 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.718 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.718 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:50:30.718 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720849+0000", 2024-09-11T09:50:30.718 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00064369899999999996 2024-09-11T09:50:30.718 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.719 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.719 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:50:30.719 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.725357+0000", 2024-09-11T09:50:30.719 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0045077379999999998 2024-09-11T09:50:30.719 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.719 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.719 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:50:30.719 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.725392+0000", 2024-09-11T09:50:30.719 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.5862999999999999e-05 2024-09-11T09:50:30.719 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.719 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.719 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:50:30.719 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.725435+0000", 2024-09-11T09:50:30.719 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.2311999999999997e-05 2024-09-11T09:50:30.719 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:30.720 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:50:30.720 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:30.720 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.720 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.720 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:119 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 131072~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:50:30.720 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.725673+0000", 2024-09-11T09:50:30.720 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 18.928982792999999, 2024-09-11T09:50:30.720 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0053927460000000003, 2024-09-11T09:50:30.720 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:50:30.720 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:50:30.720 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:30.720 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:50:30.720 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:50:30.720 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:50:30.720 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 119 2024-09-11T09:50:30.721 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.721 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:50:30.721 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.721 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:50:30.721 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.725673+0000", 2024-09-11T09:50:30.721 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:30.721 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.721 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.721 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:50:30.721 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.725673+0000", 2024-09-11T09:50:30.721 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:30.721 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.721 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.721 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:50:30.721 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.725673+0000", 2024-09-11T09:50:30.721 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.68e-07 2024-09-11T09:50:30.722 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.722 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.722 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:50:30.722 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.725720+0000", 2024-09-11T09:50:30.722 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.6526999999999997e-05 2024-09-11T09:50:30.722 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.722 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.722 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:50:30.722 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.725734+0000", 2024-09-11T09:50:30.722 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.4593000000000001e-05 2024-09-11T09:50:30.722 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.722 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.722 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:50:30.722 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.725741+0000", 2024-09-11T09:50:30.723 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.28e-06 2024-09-11T09:50:30.723 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.723 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.723 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:50:30.723 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.726290+0000", 2024-09-11T09:50:30.723 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00054892000000000005 2024-09-11T09:50:30.723 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.723 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.723 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:50:30.723 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.726338+0000", 2024-09-11T09:50:30.723 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.7982000000000002e-05 2024-09-11T09:50:30.723 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.723 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.723 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:50:30.723 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.726413+0000", 2024-09-11T09:50:30.724 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.5679000000000005e-05 2024-09-11T09:50:30.724 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.724 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.724 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:50:30.724 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.727555+0000", 2024-09-11T09:50:30.724 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0011420289999999999 2024-09-11T09:50:30.724 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.724 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.724 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:50:30.724 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.731004+0000", 2024-09-11T09:50:30.724 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0034484279999999999 2024-09-11T09:50:30.724 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.724 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.724 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:50:30.724 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.731040+0000", 2024-09-11T09:50:30.725 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.6315e-05 2024-09-11T09:50:30.725 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.725 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.725 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:50:30.725 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.731065+0000", 2024-09-11T09:50:30.725 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.5324999999999999e-05 2024-09-11T09:50:30.725 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:30.725 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:50:30.725 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:30.725 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.725 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.725 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:120 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 196608~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:50:30.725 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.725756+0000", 2024-09-11T09:50:30.725 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 18.928898883999999, 2024-09-11T09:50:30.726 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0053569100000000003, 2024-09-11T09:50:30.726 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:50:30.726 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:50:30.726 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:30.726 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:50:30.726 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:50:30.726 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:50:30.726 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 120 2024-09-11T09:50:30.726 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.726 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:50:30.726 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.726 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:50:30.726 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.725756+0000", 2024-09-11T09:50:30.726 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:30.726 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.727 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.727 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:50:30.727 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.725756+0000", 2024-09-11T09:50:30.727 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:30.727 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.727 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.727 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:50:30.727 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.725757+0000", 2024-09-11T09:50:30.727 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.3799999999999997e-07 2024-09-11T09:50:30.727 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.727 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.727 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:50:30.727 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.725802+0000", 2024-09-11T09:50:30.727 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.4790000000000003e-05 2024-09-11T09:50:30.727 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.728 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.728 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:50:30.728 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.725815+0000", 2024-09-11T09:50:30.728 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.2609000000000001e-05 2024-09-11T09:50:30.728 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.728 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.728 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:50:30.728 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.725822+0000", 2024-09-11T09:50:30.728 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.6450000000000005e-06 2024-09-11T09:50:30.728 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.728 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.728 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:50:30.728 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.726708+0000", 2024-09-11T09:50:30.728 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00088606500000000001 2024-09-11T09:50:30.729 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.729 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.729 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:50:30.729 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.726755+0000", 2024-09-11T09:50:30.729 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.6888000000000002e-05 2024-09-11T09:50:30.729 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.729 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.729 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:50:30.729 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.726828+0000", 2024-09-11T09:50:30.729 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.2510999999999997e-05 2024-09-11T09:50:30.729 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.729 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.729 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:50:30.729 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.727829+0000", 2024-09-11T09:50:30.729 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001001093 2024-09-11T09:50:30.730 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.730 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.730 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:50:30.730 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.731071+0000", 2024-09-11T09:50:30.730 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0032426009999999999 2024-09-11T09:50:30.730 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.730 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.730 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:50:30.730 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.731094+0000", 2024-09-11T09:50:30.730 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.2677e-05 2024-09-11T09:50:30.730 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.730 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.730 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:50:30.730 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.731113+0000", 2024-09-11T09:50:30.730 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.9392999999999999e-05 2024-09-11T09:50:30.731 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:30.731 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:50:30.731 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:30.731 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.731 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.731 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:138 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:50:30.731 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:12.034038+0000", 2024-09-11T09:50:30.731 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 18.620617553999999, 2024-09-11T09:50:30.731 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0054142749999999996, 2024-09-11T09:50:30.731 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:50:30.731 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:50:30.731 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:30.731 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:50:30.731 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:50:30.731 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:50:30.732 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 138 2024-09-11T09:50:30.732 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.732 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:50:30.732 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.732 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:50:30.732 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034038+0000", 2024-09-11T09:50:30.732 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:30.732 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.732 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.732 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:50:30.732 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034038+0000", 2024-09-11T09:50:30.732 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:30.732 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.732 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.733 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:50:30.733 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034038+0000", 2024-09-11T09:50:30.733 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.3499999999999996e-07 2024-09-11T09:50:30.733 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.733 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.733 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:50:30.733 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034043+0000", 2024-09-11T09:50:30.733 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.7310000000000001e-06 2024-09-11T09:50:30.733 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.733 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.733 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:50:30.733 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034050+0000", 2024-09-11T09:50:30.733 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.0419999999999997e-06 2024-09-11T09:50:30.733 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.733 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.734 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:50:30.734 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034055+0000", 2024-09-11T09:50:30.734 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.3299999999999998e-06 2024-09-11T09:50:30.734 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.734 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.734 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:50:30.734 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.037309+0000", 2024-09-11T09:50:30.734 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0032535440000000001 2024-09-11T09:50:30.734 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.734 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.734 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:50:30.734 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.037357+0000", 2024-09-11T09:50:30.734 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.8167999999999997e-05 2024-09-11T09:50:30.734 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.734 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.734 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:50:30.735 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.037430+0000", 2024-09-11T09:50:30.735 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.3275999999999994e-05 2024-09-11T09:50:30.735 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.735 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.735 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:50:30.735 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.039014+0000", 2024-09-11T09:50:30.735 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0015831350000000001 2024-09-11T09:50:30.735 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.735 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.735 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:50:30.735 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.039401+0000", 2024-09-11T09:50:30.735 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00038758500000000002 2024-09-11T09:50:30.735 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.736 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.736 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:50:30.736 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.039429+0000", 2024-09-11T09:50:30.736 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.7379e-05 2024-09-11T09:50:30.736 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.736 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.736 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:50:30.736 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.039452+0000", 2024-09-11T09:50:30.736 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.355e-05 2024-09-11T09:50:30.736 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:30.736 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:50:30.736 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:30.736 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.737 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.737 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:139 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:50:30.737 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:12.034064+0000", 2024-09-11T09:50:30.737 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 18.620591523000002, 2024-09-11T09:50:30.737 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0054736200000000002, 2024-09-11T09:50:30.737 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:50:30.737 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:50:30.737 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:30.737 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:50:30.737 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:50:30.737 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:50:30.737 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 139 2024-09-11T09:50:30.737 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.738 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:50:30.738 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.738 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:50:30.738 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034064+0000", 2024-09-11T09:50:30.738 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:30.738 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.738 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.738 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:50:30.738 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034064+0000", 2024-09-11T09:50:30.738 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:30.738 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.738 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.738 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:50:30.738 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034064+0000", 2024-09-11T09:50:30.739 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.89e-07 2024-09-11T09:50:30.739 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.739 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.739 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:50:30.739 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034069+0000", 2024-09-11T09:50:30.739 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.9799999999999998e-06 2024-09-11T09:50:30.739 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.739 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.739 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:50:30.739 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034079+0000", 2024-09-11T09:50:30.739 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.3549999999999994e-06 2024-09-11T09:50:30.739 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.739 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.739 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:50:30.740 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034084+0000", 2024-09-11T09:50:30.740 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.2209999999999996e-06 2024-09-11T09:50:30.740 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.740 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.740 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:50:30.740 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.038176+0000", 2024-09-11T09:50:30.740 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0040921610000000004 2024-09-11T09:50:30.740 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.740 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.740 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:50:30.740 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.038221+0000", 2024-09-11T09:50:30.740 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.5006000000000002e-05 2024-09-11T09:50:30.740 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.740 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.741 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:50:30.741 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.038293+0000", 2024-09-11T09:50:30.741 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.1952999999999994e-05 2024-09-11T09:50:30.741 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.741 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.741 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:50:30.741 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.039461+0000", 2024-09-11T09:50:30.741 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0011680340000000001 2024-09-11T09:50:30.741 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.741 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.741 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:50:30.741 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.039487+0000", 2024-09-11T09:50:30.741 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.5752e-05 2024-09-11T09:50:30.741 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.742 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.742 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:50:30.742 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.039515+0000", 2024-09-11T09:50:30.742 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.7763000000000002e-05 2024-09-11T09:50:30.742 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.742 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.742 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:50:30.742 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.039537+0000", 2024-09-11T09:50:30.742 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.2906000000000001e-05 2024-09-11T09:50:30.742 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:30.742 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:50:30.742 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:30.742 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.743 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.743 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:140 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:50:30.743 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:12.034093+0000", 2024-09-11T09:50:30.743 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 18.620562019000001, 2024-09-11T09:50:30.743 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0059112490000000004, 2024-09-11T09:50:30.743 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:50:30.743 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:50:30.743 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:30.743 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:50:30.743 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:50:30.743 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:50:30.743 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 140 2024-09-11T09:50:30.743 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.743 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:50:30.744 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.744 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:50:30.744 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034093+0000", 2024-09-11T09:50:30.744 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:30.744 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.744 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.744 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:50:30.744 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034093+0000", 2024-09-11T09:50:30.744 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:30.744 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.744 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.744 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:50:30.744 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034094+0000", 2024-09-11T09:50:30.744 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.4299999999999998e-07 2024-09-11T09:50:30.745 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.745 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.745 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:50:30.745 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034159+0000", 2024-09-11T09:50:30.745 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.567e-05 2024-09-11T09:50:30.745 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.745 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.745 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:50:30.745 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034167+0000", 2024-09-11T09:50:30.745 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.8490000000000006e-06 2024-09-11T09:50:30.745 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.745 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.746 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:50:30.746 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034173+0000", 2024-09-11T09:50:30.746 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.1770000000000003e-06 2024-09-11T09:50:30.746 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.746 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.746 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:50:30.746 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.038543+0000", 2024-09-11T09:50:30.746 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0043699150000000003 2024-09-11T09:50:30.746 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.746 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.746 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:50:30.746 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.038589+0000", 2024-09-11T09:50:30.746 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.5562999999999997e-05 2024-09-11T09:50:30.746 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.747 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.747 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:50:30.747 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.038662+0000", 2024-09-11T09:50:30.747 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.2748000000000002e-05 2024-09-11T09:50:30.747 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.747 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.747 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:50:30.747 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.039635+0000", 2024-09-11T09:50:30.747 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00097310799999999998 2024-09-11T09:50:30.747 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.747 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.747 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:50:30.747 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.039945+0000", 2024-09-11T09:50:30.747 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00031054100000000001 2024-09-11T09:50:30.748 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.748 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.748 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:50:30.748 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.039971+0000", 2024-09-11T09:50:30.748 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.5423999999999999e-05 2024-09-11T09:50:30.748 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.748 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.748 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:50:30.748 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.040005+0000", 2024-09-11T09:50:30.748 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.3810999999999999e-05 2024-09-11T09:50:30.748 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:30.748 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:50:30.748 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:30.748 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.748 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.749 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:50:30.749 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:12.034184+0000", 2024-09-11T09:50:30.749 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 18.620471321, 2024-09-11T09:50:30.749 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0061520380000000003, 2024-09-11T09:50:30.749 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:50:30.749 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:50:30.749 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:30.749 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:50:30.749 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:50:30.749 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:50:30.749 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 141 2024-09-11T09:50:30.749 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.749 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:50:30.749 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.750 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:50:30.750 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034184+0000", 2024-09-11T09:50:30.750 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:30.750 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.750 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.750 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:50:30.750 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034184+0000", 2024-09-11T09:50:30.750 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:30.750 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.750 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.750 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:50:30.750 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034185+0000", 2024-09-11T09:50:30.750 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.1099999999999995e-07 2024-09-11T09:50:30.750 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.751 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.751 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:50:30.751 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034190+0000", 2024-09-11T09:50:30.751 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.7309999999999998e-06 2024-09-11T09:50:30.751 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.751 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.751 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:50:30.751 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034197+0000", 2024-09-11T09:50:30.751 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.7440000000000001e-06 2024-09-11T09:50:30.751 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.751 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.751 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:50:30.751 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034204+0000", 2024-09-11T09:50:30.752 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.8970000000000002e-06 2024-09-11T09:50:30.752 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.752 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.752 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:50:30.752 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.039035+0000", 2024-09-11T09:50:30.752 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0048307039999999999 2024-09-11T09:50:30.752 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.752 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.752 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:50:30.752 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.039093+0000", 2024-09-11T09:50:30.752 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.8604000000000003e-05 2024-09-11T09:50:30.752 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.752 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.753 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:50:30.753 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.039184+0000", 2024-09-11T09:50:30.753 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.0196000000000002e-05 2024-09-11T09:50:30.753 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.753 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.753 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:50:30.753 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.039873+0000", 2024-09-11T09:50:30.753 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00068960600000000005 2024-09-11T09:50:30.753 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.753 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.753 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:50:30.753 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.040292+0000", 2024-09-11T09:50:30.753 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00041865199999999999 2024-09-11T09:50:30.753 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.754 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.754 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:50:30.754 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.040317+0000", 2024-09-11T09:50:30.754 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.5439000000000002e-05 2024-09-11T09:50:30.754 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.754 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.754 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:50:30.754 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.040336+0000", 2024-09-11T09:50:30.754 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.8754e-05 2024-09-11T09:50:30.754 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:30.754 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:50:30.754 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:30.754 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.754 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.755 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:157 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:50:30.755 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:12.452040+0000", 2024-09-11T09:50:30.755 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 18.202615468000001, 2024-09-11T09:50:30.755 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0068848269999999996, 2024-09-11T09:50:30.755 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:50:30.755 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:50:30.755 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:30.755 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:50:30.755 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:50:30.755 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:50:30.755 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 157 2024-09-11T09:50:30.755 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.755 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:50:30.755 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.755 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:50:30.756 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452040+0000", 2024-09-11T09:50:30.756 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:30.756 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.756 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.756 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:50:30.756 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452040+0000", 2024-09-11T09:50:30.756 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:30.756 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.756 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.756 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:50:30.756 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452040+0000", 2024-09-11T09:50:30.756 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.22e-07 2024-09-11T09:50:30.756 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.756 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.756 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:50:30.756 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452042+0000", 2024-09-11T09:50:30.757 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.7010000000000001e-06 2024-09-11T09:50:30.757 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.757 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.757 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:50:30.757 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452045+0000", 2024-09-11T09:50:30.757 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.1310000000000002e-06 2024-09-11T09:50:30.757 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.757 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.757 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:50:30.757 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452048+0000", 2024-09-11T09:50:30.757 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.65e-06 2024-09-11T09:50:30.757 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.757 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.757 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:50:30.757 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.454453+0000", 2024-09-11T09:50:30.758 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0024052959999999999 2024-09-11T09:50:30.758 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.758 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.758 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:50:30.758 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.454498+0000", 2024-09-11T09:50:30.758 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.4978999999999997e-05 2024-09-11T09:50:30.758 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.758 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.758 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:50:30.758 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.454567+0000", 2024-09-11T09:50:30.758 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.9110999999999996e-05 2024-09-11T09:50:30.758 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.758 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.758 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:50:30.758 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.455464+0000", 2024-09-11T09:50:30.759 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00089696199999999995 2024-09-11T09:50:30.759 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.759 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.759 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:50:30.759 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.458844+0000", 2024-09-11T09:50:30.759 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0033802469999999999 2024-09-11T09:50:30.759 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.759 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.759 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:50:30.759 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.458889+0000", 2024-09-11T09:50:30.759 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.4441999999999999e-05 2024-09-11T09:50:30.759 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.759 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.759 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:50:30.759 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.458925+0000", 2024-09-11T09:50:30.760 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.6086e-05 2024-09-11T09:50:30.760 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:30.760 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:50:30.760 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:30.760 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.760 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.760 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:158 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:50:30.760 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:12.452055+0000", 2024-09-11T09:50:30.760 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 18.202600074999999, 2024-09-11T09:50:30.760 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0069533829999999996, 2024-09-11T09:50:30.760 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:50:30.760 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:50:30.760 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:30.760 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:50:30.761 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:50:30.761 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:50:30.761 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 158 2024-09-11T09:50:30.761 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.761 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:50:30.761 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.761 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:50:30.761 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452055+0000", 2024-09-11T09:50:30.761 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:30.761 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.761 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.761 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:50:30.761 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452055+0000", 2024-09-11T09:50:30.761 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:30.761 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.762 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.762 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:50:30.762 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452056+0000", 2024-09-11T09:50:30.762 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.48e-07 2024-09-11T09:50:30.762 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.762 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.762 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:50:30.762 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452087+0000", 2024-09-11T09:50:30.762 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.1149000000000003e-05 2024-09-11T09:50:30.762 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.762 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.762 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:50:30.762 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452092+0000", 2024-09-11T09:50:30.762 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.8400000000000002e-06 2024-09-11T09:50:30.762 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.763 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.763 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:50:30.763 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452095+0000", 2024-09-11T09:50:30.763 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.0740000000000001e-06 2024-09-11T09:50:30.763 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.763 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.763 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:50:30.763 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.454796+0000", 2024-09-11T09:50:30.763 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0027009740000000001 2024-09-11T09:50:30.763 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.763 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.763 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:50:30.763 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.454869+0000", 2024-09-11T09:50:30.764 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.3678999999999997e-05 2024-09-11T09:50:30.764 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.764 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.764 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:50:30.764 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.454992+0000", 2024-09-11T09:50:30.764 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00012279000000000001 2024-09-11T09:50:30.764 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.764 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.764 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:50:30.764 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.456072+0000", 2024-09-11T09:50:30.764 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001080229 2024-09-11T09:50:30.764 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.764 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.764 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:50:30.764 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.458931+0000", 2024-09-11T09:50:30.765 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0028591789999999999 2024-09-11T09:50:30.765 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.765 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.765 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:50:30.765 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.458972+0000", 2024-09-11T09:50:30.765 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.0170999999999999e-05 2024-09-11T09:50:30.765 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.765 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.765 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:50:30.765 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.459009+0000", 2024-09-11T09:50:30.765 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.7049999999999999e-05 2024-09-11T09:50:30.765 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:30.765 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:50:30.765 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:30.765 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.765 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.766 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:159 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:50:30.766 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:12.452099+0000", 2024-09-11T09:50:30.766 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 18.202555907000001, 2024-09-11T09:50:30.766 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0069885950000000002, 2024-09-11T09:50:30.766 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:50:30.766 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:50:30.766 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:30.766 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:50:30.766 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:50:30.766 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:50:30.766 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 159 2024-09-11T09:50:30.766 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.766 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:50:30.766 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.766 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:50:30.766 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452099+0000", 2024-09-11T09:50:30.767 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:30.767 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.767 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.767 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:50:30.767 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452099+0000", 2024-09-11T09:50:30.767 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:30.767 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.767 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.767 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:50:30.767 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452100+0000", 2024-09-11T09:50:30.767 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.2800000000000003e-07 2024-09-11T09:50:30.767 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.767 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.767 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:50:30.767 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452102+0000", 2024-09-11T09:50:30.768 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.824e-06 2024-09-11T09:50:30.768 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.768 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.768 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:50:30.768 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452105+0000", 2024-09-11T09:50:30.768 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.0460000000000001e-06 2024-09-11T09:50:30.768 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.768 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.768 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:50:30.768 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452107+0000", 2024-09-11T09:50:30.768 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.543e-06 2024-09-11T09:50:30.768 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.768 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.768 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:50:30.768 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.455484+0000", 2024-09-11T09:50:30.768 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0033769759999999998 2024-09-11T09:50:30.769 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.769 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.769 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:50:30.769 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.455527+0000", 2024-09-11T09:50:30.769 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.2589999999999997e-05 2024-09-11T09:50:30.769 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.769 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.769 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:50:30.769 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.455594+0000", 2024-09-11T09:50:30.769 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.6775999999999999e-05 2024-09-11T09:50:30.769 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.769 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.769 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:50:30.769 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.456312+0000", 2024-09-11T09:50:30.769 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00071874700000000005 2024-09-11T09:50:30.770 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.770 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.770 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:50:30.770 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.459015+0000", 2024-09-11T09:50:30.770 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0027028540000000002 2024-09-11T09:50:30.770 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.770 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.770 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:50:30.770 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.459054+0000", 2024-09-11T09:50:30.770 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.8840000000000001e-05 2024-09-11T09:50:30.770 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:30.770 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:30.770 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:50:30.770 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.459088+0000", 2024-09-11T09:50:30.770 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.4071e-05 2024-09-11T09:50:30.770 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:30.771 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:50:30.771 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:30.771 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:30.771 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:50:30.771 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-11T09:50:30.772 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-11T09:50:30.773 INFO:tasks.ceph.osd.2.smithi017.stderr:2024-09-11T09:50:30.773+0000 7fd20b170640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31672) UID: 0 2024-09-11T09:50:30.832 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T09:50:30.877 INFO:tasks.ceph.osd.2.smithi017.stderr:2024-09-11T09:50:30.878+0000 7fd20b170640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31672) UID: 0 2024-09-11T09:50:30.915 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-11T09:50:30.915 INFO:teuthology.orchestra.run.smithi096.stdout: "ops": [], 2024-09-11T09:50:30.915 INFO:teuthology.orchestra.run.smithi096.stdout: "num_ops": 0 2024-09-11T09:50:30.916 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-11T09:50:30.924 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-11T09:50:30.962 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-11T09:50:30.963 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T09:50:30.973 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-09-11T09:50:30.974+0000 7f7b6c7d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31904) UID: 0 2024-09-11T09:50:30.979 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-11T09:50:30.980 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T09:50:30.996 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-11T09:50:30.996 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T09:50:31.013 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-11T09:50:31.013 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T09:50:31.029 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-11T09:50:31.029 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T09:50:31.030 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:50:31.030 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:50:31.038 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-11T09:50:31.046 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-11T09:50:31.046 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T09:50:31.079 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:50:31.080+0000 7f3c96ec9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31909) UID: 0 2024-09-11T09:50:31.175 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-09-11T09:50:31.176+0000 7f7b6c7d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31904) UID: 0 2024-09-11T09:50:31.181 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:50:31.181 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:50:31.190 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-11T09:50:31.276 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:50:31.276+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:50:31.333 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:50:31.333 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:50:31.342 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-11T09:50:31.385 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-09-11T09:50:31.386+0000 7f7b6c7d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31904) UID: 0 2024-09-11T09:50:31.444 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:50:31.444 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:50:31.453 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-11T09:50:31.477 INFO:tasks.ceph.osd.2.smithi017.stderr:2024-09-11T09:50:31.478+0000 7fd20b170640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31672) UID: 0 2024-09-11T09:50:31.578 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:50:31.579+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:50:31.596 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:50:31.596 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:50:31.605 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-11T09:50:31.679 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:50:31.680+0000 7f3c96ec9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31909) UID: 0 2024-09-11T09:50:31.719 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:50:31.719 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:50:31.728 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-11T09:50:31.780 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:50:31.781+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:50:31.836 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:50:31.836 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:50:31.845 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-11T09:50:31.882 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:50:31.882+0000 7f3c96ec9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31909) UID: 0 2024-09-11T09:50:31.982 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-09-11T09:50:31.983+0000 7f7b6c7d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31904) UID: 0 2024-09-11T09:50:31.993 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:50:31.993 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:50:32.003 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-11T09:50:32.083 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:50:32.084+0000 7fdec5849640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31907) UID: 0 2024-09-11T09:50:32.110 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:50:32.110 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:50:32.119 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-11T09:50:32.184 INFO:tasks.ceph.osd.2.smithi017.stderr:2024-09-11T09:50:32.185+0000 7fd20b170640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31672) UID: 0 2024-09-11T09:50:32.227 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:50:32.227 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:50:32.236 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-11T09:50:32.285 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:50:32.286+0000 7fdec5849640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31907) UID: 0 2024-09-11T09:50:32.383 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:50:32.383 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:50:32.386 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-09-11T09:50:32.387+0000 7f7b6c7d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31904) UID: 0 2024-09-11T09:50:32.392 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-11T09:50:32.487 INFO:tasks.ceph.osd.2.smithi017.stderr:2024-09-11T09:50:32.488+0000 7fd20b170640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31672) UID: 0 2024-09-11T09:50:32.500 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:50:32.500 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:50:32.509 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-11T09:50:32.533 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5] 2024-09-11T09:50:32.533 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-11T09:50:32.533 INFO:tasks.thrashosds.thrasher:Removing osd 0, in_osds are: [0, 1, 2, 3, 4, 5] 2024-09-11T09:50:32.534 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd out 0 2024-09-11T09:50:32.539 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-11T09:50:32.595 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:50:32.596+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:50:32.662 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:50:32.662 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:50:32.675 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-11T09:50:32.716 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:50:32.717+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:50:32.789 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:50:32.790+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:50:32.824 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:50:32.824 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:50:32.833 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-11T09:50:32.859 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-11T09:50:32.860+0000 7ff44db6e640 -1 mon.a@0(leader).osd e21 definitely_dead 0 2024-09-11T09:50:32.891 INFO:tasks.ceph.osd.0.smithi017.stderr:2024-09-11T09:50:32.892+0000 7fb73c0f1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31665) UID: 0 2024-09-11T09:50:32.949 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:50:32.949 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:50:32.958 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-11T09:50:33.002 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:50:33.003+0000 7fdec5849640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31907) UID: 0 2024-09-11T09:50:33.058 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T09:50:33.062 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:50:33.062 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:50:33.070 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-11T09:50:33.111 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:50:33.112+0000 7f3c96ec9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31909) UID: 0 2024-09-11T09:50:33.183 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-11T09:50:33.183 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T09:50:33.193 INFO:tasks.ceph.osd.2.smithi017.stderr:2024-09-11T09:50:33.194+0000 7fd20b170640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31672) UID: 0 2024-09-11T09:50:33.200 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-11T09:50:33.200 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T09:50:33.214 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:50:33.214 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:50:33.216 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-11T09:50:33.216 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T09:50:33.223 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-11T09:50:33.233 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-11T09:50:33.233 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T09:50:33.250 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-11T09:50:33.250 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T09:50:33.266 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-11T09:50:33.267 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T09:50:33.294 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:50:33.295+0000 7f3c96ec9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31909) UID: 0 2024-09-11T09:50:33.325 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-11T09:50:33.325 INFO:teuthology.orchestra.run.smithi096.stdout: "ops": [], 2024-09-11T09:50:33.325 INFO:teuthology.orchestra.run.smithi096.stdout: "num_ops": 0 2024-09-11T09:50:33.325 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-11T09:50:33.333 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-11T09:50:33.417 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-09-11T09:50:33.418+0000 7f7b6c7d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31904) UID: 0 2024-09-11T09:50:33.476 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-11T09:50:33.476 INFO:teuthology.orchestra.run.smithi096.stdout: "ops": [], 2024-09-11T09:50:33.476 INFO:teuthology.orchestra.run.smithi096.stdout: "complaint_time": 30, 2024-09-11T09:50:33.476 INFO:teuthology.orchestra.run.smithi096.stdout: "num_blocked_ops": 0 2024-09-11T09:50:33.476 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-11T09:50:33.485 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-11T09:50:33.496 INFO:tasks.ceph.osd.2.smithi017.stderr:2024-09-11T09:50:33.497+0000 7fd20b170640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31672) UID: 0 2024-09-11T09:50:33.532 INFO:teuthology.orchestra.run.smithi017.stderr:noscrub is unset 2024-09-11T09:50:33.532 INFO:teuthology.orchestra.run.smithi017.stderr:marked out osd.0. 2024-09-11T09:50:33.597 INFO:tasks.ceph.osd.0.smithi017.stderr:2024-09-11T09:50:33.598+0000 7fb73c0f1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31665) UID: 0 2024-09-11T09:50:33.627 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-11T09:50:33.627 INFO:teuthology.orchestra.run.smithi096.stdout: "size": 20, 2024-09-11T09:50:33.628 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 600, 2024-09-11T09:50:33.628 INFO:teuthology.orchestra.run.smithi096.stdout: "ops": [] 2024-09-11T09:50:33.628 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-11T09:50:33.636 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-11T09:50:33.698 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:50:33.699+0000 7f3c96ec9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31909) UID: 0 2024-09-11T09:50:33.738 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-11T09:50:33.738 INFO:teuthology.orchestra.run.smithi096.stdout: "ops": [], 2024-09-11T09:50:33.738 INFO:teuthology.orchestra.run.smithi096.stdout: "num_ops": 0 2024-09-11T09:50:33.738 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-11T09:50:33.747 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-11T09:50:33.831 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:50:33.832+0000 7f3c96ec9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31909) UID: 0 2024-09-11T09:50:33.889 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-11T09:50:33.890 INFO:teuthology.orchestra.run.smithi096.stdout: "ops": [], 2024-09-11T09:50:33.890 INFO:teuthology.orchestra.run.smithi096.stdout: "complaint_time": 30, 2024-09-11T09:50:33.890 INFO:teuthology.orchestra.run.smithi096.stdout: "num_blocked_ops": 0 2024-09-11T09:50:33.890 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-11T09:50:33.898 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-11T09:50:33.939 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:50:33.940+0000 7fdec5849640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31907) UID: 0 2024-09-11T09:50:34.000 INFO:tasks.ceph.osd.2.smithi017.stderr:2024-09-11T09:50:34.001+0000 7fd20b170640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31672) UID: 0 2024-09-11T09:50:34.042 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-11T09:50:34.042 INFO:teuthology.orchestra.run.smithi096.stdout: "size": 20, 2024-09-11T09:50:34.042 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 600, 2024-09-11T09:50:34.042 INFO:teuthology.orchestra.run.smithi096.stdout: "ops": [ 2024-09-11T09:50:34.042 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:34.043 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4230.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-11T09:50:34.043 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:18.490736+0000", 2024-09-11T09:50:34.043 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 15.552405926, 2024-09-11T09:50:34.043 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0059097569999999999, 2024-09-11T09:50:34.043 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:50:34.043 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:50:34.043 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:34.043 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:50:34.043 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4230", 2024-09-11T09:50:34.043 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.17:0/4101109576", 2024-09-11T09:50:34.043 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 1 2024-09-11T09:50:34.043 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:34.043 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:50:34.044 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:34.044 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:50:34.044 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.490736+0000", 2024-09-11T09:50:34.044 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:34.044 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:34.044 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:34.044 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:50:34.044 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.490736+0000", 2024-09-11T09:50:34.044 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:34.044 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:34.044 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:34.044 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:50:34.044 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.490742+0000", 2024-09-11T09:50:34.045 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.48e-06 2024-09-11T09:50:34.045 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:34.045 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:34.045 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:50:34.045 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.490749+0000", 2024-09-11T09:50:34.045 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.4569999999999999e-06 2024-09-11T09:50:34.045 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:34.045 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:34.045 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:50:34.045 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.490762+0000", 2024-09-11T09:50:34.045 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.2721000000000001e-05 2024-09-11T09:50:34.045 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:34.045 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:34.046 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:50:34.046 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.490774+0000", 2024-09-11T09:50:34.046 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1823e-05 2024-09-11T09:50:34.046 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:34.046 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:34.046 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:50:34.046 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.490846+0000", 2024-09-11T09:50:34.046 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.1878000000000005e-05 2024-09-11T09:50:34.046 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:34.046 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:34.046 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for peered", 2024-09-11T09:50:34.046 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.490857+0000", 2024-09-11T09:50:34.046 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1017e-05 2024-09-11T09:50:34.046 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:34.047 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:34.047 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:50:34.047 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.493856+0000", 2024-09-11T09:50:34.047 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00299885 2024-09-11T09:50:34.047 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:34.047 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:34.047 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:50:34.047 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.494020+0000", 2024-09-11T09:50:34.047 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00016436599999999999 2024-09-11T09:50:34.047 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:34.047 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:34.047 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 2", 2024-09-11T09:50:34.047 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.494186+0000", 2024-09-11T09:50:34.047 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00016562799999999999 2024-09-11T09:50:34.048 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:34.048 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:34.048 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:50:34.048 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.495407+0000", 2024-09-11T09:50:34.048 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0012217420000000001 2024-09-11T09:50:34.048 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:34.048 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:34.048 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:50:34.048 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.496579+0000", 2024-09-11T09:50:34.048 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001171844 2024-09-11T09:50:34.048 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:34.048 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:34.048 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:50:34.048 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.496620+0000", 2024-09-11T09:50:34.049 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.0479000000000003e-05 2024-09-11T09:50:34.049 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:34.049 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:34.049 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:50:34.049 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.496646+0000", 2024-09-11T09:50:34.049 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.6472e-05 2024-09-11T09:50:34.049 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:34.049 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:50:34.049 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:34.049 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:34.049 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:34.049 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_repop_reply(client.4230.0:1 2.3 e16/15 ondisk, result = 0)", 2024-09-11T09:50:34.049 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:18.495321+0000", 2024-09-11T09:50:34.050 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 15.547821260999999, 2024-09-11T09:50:34.052 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.4965999999999998e-05, 2024-09-11T09:50:34.052 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:50:34.052 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:50:34.052 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "started", 2024-09-11T09:50:34.052 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:50:34.052 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:34.052 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:50:34.052 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.495321+0000", 2024-09-11T09:50:34.052 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:34.053 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:34.053 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:34.053 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:50:34.053 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.495321+0000", 2024-09-11T09:50:34.053 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:34.053 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:34.053 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:34.053 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:50:34.053 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.495322+0000", 2024-09-11T09:50:34.053 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.7700000000000004e-07 2024-09-11T09:50:34.053 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:34.053 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:34.053 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:50:34.054 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.495325+0000", 2024-09-11T09:50:34.054 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.7780000000000001e-06 2024-09-11T09:50:34.054 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:34.054 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:34.054 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:50:34.054 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.495338+0000", 2024-09-11T09:50:34.054 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.2315e-05 2024-09-11T09:50:34.054 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:34.054 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:34.054 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:50:34.054 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.495345+0000", 2024-09-11T09:50:34.054 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.8889999999999996e-06 2024-09-11T09:50:34.054 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:34.055 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:34.055 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:50:34.055 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.495392+0000", 2024-09-11T09:50:34.055 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.7524000000000001e-05 2024-09-11T09:50:34.055 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:34.055 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:34.055 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:50:34.055 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.495402+0000", 2024-09-11T09:50:34.055 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.5840000000000001e-06 2024-09-11T09:50:34.055 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:34.055 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:34.055 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:50:34.055 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.495416+0000", 2024-09-11T09:50:34.055 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.4199000000000001e-05 2024-09-11T09:50:34.055 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:34.056 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:50:34.056 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:34.056 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:34.056 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:34.056 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4230.0:2 2.2 2:5cea7035:::rbd_info:head [call rbd.metadata_list in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e16)", 2024-09-11T09:50:34.056 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:18.499576+0000", 2024-09-11T09:50:34.056 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 15.543566754, 2024-09-11T09:50:34.056 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00018163000000000001, 2024-09-11T09:50:34.056 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:50:34.056 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:50:34.056 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "reached pg", 2024-09-11T09:50:34.056 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:50:34.056 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4230", 2024-09-11T09:50:34.056 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.17:0/4101109576", 2024-09-11T09:50:34.057 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 2 2024-09-11T09:50:34.057 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:34.057 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:50:34.057 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:34.057 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:50:34.057 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.499576+0000", 2024-09-11T09:50:34.057 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:34.057 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:34.057 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:34.057 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:50:34.057 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.499576+0000", 2024-09-11T09:50:34.057 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:34.057 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:34.057 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:34.058 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:50:34.058 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.499577+0000", 2024-09-11T09:50:34.058 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.124e-06 2024-09-11T09:50:34.058 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:34.058 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:34.058 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:50:34.058 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.499584+0000", 2024-09-11T09:50:34.058 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.1949999999999997e-06 2024-09-11T09:50:34.058 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:34.058 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:34.058 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:50:34.058 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.499597+0000", 2024-09-11T09:50:34.058 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.366e-05 2024-09-11T09:50:34.058 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:34.058 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:34.059 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:50:34.059 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.499604+0000", 2024-09-11T09:50:34.059 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.2709999999999998e-06 2024-09-11T09:50:34.059 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:34.059 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:34.059 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:50:34.059 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.499660+0000", 2024-09-11T09:50:34.059 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.6270999999999997e-05 2024-09-11T09:50:34.059 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:34.059 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:34.059 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:50:34.059 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.499757+0000", 2024-09-11T09:50:34.059 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.7108999999999995e-05 2024-09-11T09:50:34.060 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:34.060 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:50:34.060 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:34.060 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:34.060 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:34.060 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4230.0:3 2.2 2:5cea7035:::rbd_info:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e16)", 2024-09-11T09:50:34.060 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:18.499937+0000", 2024-09-11T09:50:34.060 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 15.543204964999999, 2024-09-11T09:50:34.060 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000143318, 2024-09-11T09:50:34.060 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:50:34.060 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:50:34.060 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "reached pg", 2024-09-11T09:50:34.060 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:50:34.061 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4230", 2024-09-11T09:50:34.061 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.17:0/4101109576", 2024-09-11T09:50:34.061 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 3 2024-09-11T09:50:34.061 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:34.061 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:50:34.061 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:34.061 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:50:34.061 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.499937+0000", 2024-09-11T09:50:34.061 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:34.061 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:34.061 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:34.061 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:50:34.061 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.499937+0000", 2024-09-11T09:50:34.061 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:34.062 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:34.062 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:34.062 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:50:34.062 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.499938+0000", 2024-09-11T09:50:34.062 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.195e-06 2024-09-11T09:50:34.062 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:34.062 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:34.062 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:50:34.062 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.499943+0000", 2024-09-11T09:50:34.062 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.9080000000000003e-06 2024-09-11T09:50:34.062 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:34.062 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:34.062 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:50:34.063 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.499956+0000", 2024-09-11T09:50:34.063 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.2728000000000001e-05 2024-09-11T09:50:34.063 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:34.063 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:34.063 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:50:34.063 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.499964+0000", 2024-09-11T09:50:34.063 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.8220000000000007e-06 2024-09-11T09:50:34.063 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:34.063 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:34.063 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:50:34.063 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.500017+0000", 2024-09-11T09:50:34.063 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.2809000000000002e-05 2024-09-11T09:50:34.063 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:34.063 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:34.064 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:50:34.064 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.500081+0000", 2024-09-11T09:50:34.064 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.3856000000000007e-05 2024-09-11T09:50:34.064 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:34.064 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:50:34.064 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:34.064 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:34.064 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:34.064 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4230.0:5 2.2 2:5cea7035:::rbd_info:head [create,write 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-11T09:50:34.064 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:19.492662+0000", 2024-09-11T09:50:34.064 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 14.550480305000001, 2024-09-11T09:50:34.064 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0043824759999999997, 2024-09-11T09:50:34.065 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:50:34.065 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:50:34.065 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:34.065 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:50:34.065 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4230", 2024-09-11T09:50:34.065 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.17:0/4101109576", 2024-09-11T09:50:34.065 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 5 2024-09-11T09:50:34.065 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:34.065 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:50:34.065 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:34.066 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:50:34.066 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.492662+0000", 2024-09-11T09:50:34.066 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:34.066 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:34.066 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:34.066 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:50:34.066 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.492662+0000", 2024-09-11T09:50:34.066 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:34.066 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:34.066 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:34.066 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:50:34.066 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.492664+0000", 2024-09-11T09:50:34.066 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.3599999999999999e-06 2024-09-11T09:50:34.067 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:34.067 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:34.067 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:50:34.067 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.492675+0000", 2024-09-11T09:50:34.067 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1182000000000001e-05 2024-09-11T09:50:34.067 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:34.067 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:34.067 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:50:34.067 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.492693+0000", 2024-09-11T09:50:34.067 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.7031999999999999e-05 2024-09-11T09:50:34.067 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:34.067 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:34.067 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:50:34.067 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.492702+0000", 2024-09-11T09:50:34.068 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.5540000000000003e-06 2024-09-11T09:50:34.068 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:34.068 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:34.068 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:50:34.068 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.492767+0000", 2024-09-11T09:50:34.068 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.5411000000000001e-05 2024-09-11T09:50:34.068 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:34.068 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:34.068 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op must wait for map", 2024-09-11T09:50:34.068 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.492773+0000", 2024-09-11T09:50:34.068 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.1039999999999998e-06 2024-09-11T09:50:34.068 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:34.068 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:34.069 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:50:34.069 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.494354+0000", 2024-09-11T09:50:34.069 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0015814049999999999 2024-09-11T09:50:34.069 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:34.069 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:34.069 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:50:34.069 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.494476+0000", 2024-09-11T09:50:34.069 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000122089 2024-09-11T09:50:34.069 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:34.069 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:34.069 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 1", 2024-09-11T09:50:34.069 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.494577+0000", 2024-09-11T09:50:34.069 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000100536 2024-09-11T09:50:34.069 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:34.070 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:34.070 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:50:34.070 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.495770+0000", 2024-09-11T09:50:34.070 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0011932469999999999 2024-09-11T09:50:34.070 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:34.070 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:34.070 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:50:34.070 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.496966+0000", 2024-09-11T09:50:34.070 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001196003 2024-09-11T09:50:34.070 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:34.070 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:34.070 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:50:34.070 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.497011+0000", 2024-09-11T09:50:34.070 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.5229999999999999e-05 2024-09-11T09:50:34.070 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:34.071 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:34.071 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:50:34.071 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.497044+0000", 2024-09-11T09:50:34.071 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.3322999999999999e-05 2024-09-11T09:50:34.071 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:34.071 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:50:34.071 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:34.071 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:34.071 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:34.071 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_repop_reply(client.4230.0:5 2.2 e17/15 ondisk, result = 0)", 2024-09-11T09:50:34.071 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:19.495669+0000", 2024-09-11T09:50:34.071 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 14.547473517, 2024-09-11T09:50:34.071 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000106431, 2024-09-11T09:50:34.072 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:50:34.072 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:50:34.072 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "started", 2024-09-11T09:50:34.072 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:50:34.072 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:34.072 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:50:34.072 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.495669+0000", 2024-09-11T09:50:34.072 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:34.072 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:34.072 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:34.072 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:50:34.072 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.495669+0000", 2024-09-11T09:50:34.072 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:34.072 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:34.073 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:34.073 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:50:34.073 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.495670+0000", 2024-09-11T09:50:34.073 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.218e-06 2024-09-11T09:50:34.073 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:34.073 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:34.073 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:50:34.073 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.495676+0000", 2024-09-11T09:50:34.073 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.5119999999999997e-06 2024-09-11T09:50:34.073 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:34.073 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:34.073 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:50:34.073 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.495690+0000", 2024-09-11T09:50:34.073 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.3312000000000001e-05 2024-09-11T09:50:34.073 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:34.074 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:34.074 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:50:34.074 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.495706+0000", 2024-09-11T09:50:34.074 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.6560000000000001e-05 2024-09-11T09:50:34.074 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:34.074 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:34.074 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:50:34.074 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.495759+0000", 2024-09-11T09:50:34.074 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.2828000000000003e-05 2024-09-11T09:50:34.074 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:34.074 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:34.074 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:50:34.074 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.495765+0000", 2024-09-11T09:50:34.074 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.7869999999999998e-06 2024-09-11T09:50:34.075 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:34.075 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:34.075 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:50:34.075 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.495775+0000", 2024-09-11T09:50:34.075 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.0214000000000001e-05 2024-09-11T09:50:34.075 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:34.075 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:50:34.075 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:34.075 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:34.075 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:34.075 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4230.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-09-11T09:50:34.075 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:20.498634+0000", 2024-09-11T09:50:34.075 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 13.544507905, 2024-09-11T09:50:34.076 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0050648539999999997, 2024-09-11T09:50:34.076 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:50:34.076 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:50:34.076 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:34.076 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:50:34.076 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4230", 2024-09-11T09:50:34.076 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.17:0/4101109576", 2024-09-11T09:50:34.076 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 7 2024-09-11T09:50:34.076 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:34.076 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:50:34.076 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:34.076 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:50:34.076 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.498634+0000", 2024-09-11T09:50:34.077 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:34.077 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:34.077 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:34.077 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:50:34.077 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.498634+0000", 2024-09-11T09:50:34.077 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:34.077 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:34.077 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:34.077 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:50:34.077 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.498637+0000", 2024-09-11T09:50:34.077 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.1919999999999999e-06 2024-09-11T09:50:34.077 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:34.077 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:34.078 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:50:34.078 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.498648+0000", 2024-09-11T09:50:34.078 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1116e-05 2024-09-11T09:50:34.078 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:34.078 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:34.078 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:50:34.078 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.498666+0000", 2024-09-11T09:50:34.078 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.8411999999999999e-05 2024-09-11T09:50:34.078 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:34.078 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:34.078 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:50:34.078 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.498675+0000", 2024-09-11T09:50:34.078 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.4059999999999994e-06 2024-09-11T09:50:34.078 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:34.079 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:34.079 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:50:34.079 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.498745+0000", 2024-09-11T09:50:34.079 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.9764000000000005e-05 2024-09-11T09:50:34.079 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:34.079 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:34.079 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op must wait for map", 2024-09-11T09:50:34.079 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.498754+0000", 2024-09-11T09:50:34.079 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.5969999999999992e-06 2024-09-11T09:50:34.079 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:34.079 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:34.079 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:50:34.080 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.500385+0000", 2024-09-11T09:50:34.080 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0016308360000000001 2024-09-11T09:50:34.080 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:34.080 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:34.080 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:50:34.080 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.500473+0000", 2024-09-11T09:50:34.080 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.7966000000000004e-05 2024-09-11T09:50:34.080 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:34.080 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:34.080 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 1", 2024-09-11T09:50:34.080 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.500591+0000", 2024-09-11T09:50:34.080 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000117895 2024-09-11T09:50:34.080 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:34.080 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:34.081 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:50:34.081 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.501685+0000", 2024-09-11T09:50:34.081 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001094886 2024-09-11T09:50:34.081 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:34.081 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:34.081 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:50:34.081 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.503627+0000", 2024-09-11T09:50:34.081 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001941199 2024-09-11T09:50:34.081 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:34.081 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:34.081 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:50:34.081 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.503667+0000", 2024-09-11T09:50:34.081 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.0645000000000001e-05 2024-09-11T09:50:34.082 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:34.082 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:34.082 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:50:34.082 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.503699+0000", 2024-09-11T09:50:34.082 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.1940000000000003e-05 2024-09-11T09:50:34.082 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:34.082 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:50:34.082 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:34.082 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:34.082 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:34.082 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_repop_reply(client.4230.0:7 2.2 e18/15 ondisk, result = 0)", 2024-09-11T09:50:34.082 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:20.501608+0000", 2024-09-11T09:50:34.082 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 13.541534383, 2024-09-11T09:50:34.082 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.2459000000000005e-05, 2024-09-11T09:50:34.082 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:50:34.083 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:50:34.083 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "started", 2024-09-11T09:50:34.083 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:50:34.083 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:34.083 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:50:34.083 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.501608+0000", 2024-09-11T09:50:34.083 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:34.083 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:34.083 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:34.083 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:50:34.083 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.501608+0000", 2024-09-11T09:50:34.083 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:34.083 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:34.083 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:34.083 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:50:34.084 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.501609+0000", 2024-09-11T09:50:34.084 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.0160000000000001e-06 2024-09-11T09:50:34.084 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:34.084 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:34.084 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:50:34.084 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.501613+0000", 2024-09-11T09:50:34.084 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.5959999999999997e-06 2024-09-11T09:50:34.084 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:34.084 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:34.084 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:50:34.084 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.501625+0000", 2024-09-11T09:50:34.084 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1888e-05 2024-09-11T09:50:34.084 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:34.085 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:34.085 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:50:34.085 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.501634+0000", 2024-09-11T09:50:34.085 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.6139999999999992e-06 2024-09-11T09:50:34.085 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:34.085 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:34.085 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:50:34.085 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.501676+0000", 2024-09-11T09:50:34.085 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.1953999999999998e-05 2024-09-11T09:50:34.085 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:34.085 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:34.085 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:50:34.085 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.501682+0000", 2024-09-11T09:50:34.086 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.8000000000000004e-06 2024-09-11T09:50:34.086 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:34.086 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:34.086 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:50:34.086 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.501690+0000", 2024-09-11T09:50:34.086 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.5909999999999996e-06 2024-09-11T09:50:34.086 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:34.086 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:50:34.086 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:34.086 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:34.086 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:50:34.086 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-11T09:50:34.087 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-11T09:50:34.100 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-09-11T09:50:34.101+0000 7f7b6c7d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31904) UID: 0 2024-09-11T09:50:34.201 INFO:tasks.ceph.osd.0.smithi017.stderr:2024-09-11T09:50:34.202+0000 7fb73c0f1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31665) UID: 0 2024-09-11T09:50:34.208 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:50:34.208 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [], 2024-09-11T09:50:34.208 INFO:teuthology.orchestra.run.smithi017.stdout: "num_ops": 0 2024-09-11T09:50:34.208 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:50:34.217 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-11T09:50:34.302 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:50:34.303+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:50:34.367 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:50:34.367 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [], 2024-09-11T09:50:34.367 INFO:teuthology.orchestra.run.smithi017.stdout: "complaint_time": 30, 2024-09-11T09:50:34.367 INFO:teuthology.orchestra.run.smithi017.stdout: "num_blocked_ops": 0 2024-09-11T09:50:34.367 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:50:34.376 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-11T09:50:34.417 INFO:tasks.ceph.osd.0.smithi017.stderr:2024-09-11T09:50:34.418+0000 7fb73c0f1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31665) UID: 0 2024-09-11T09:50:34.504 INFO:tasks.ceph.osd.0.smithi017.stderr:2024-09-11T09:50:34.505+0000 7fb73c0f1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31665) UID: 0 2024-09-11T09:50:34.525 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:50:34.525 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 20, 2024-09-11T09:50:34.525 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 600, 2024-09-11T09:50:34.525 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [ 2024-09-11T09:50:34.525 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.525 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4183.0:21 1.0 e13/12 1:30306672:devicehealth::main.db.0000000000000000:head v 13'11, mlcod=13'8)", 2024-09-11T09:50:34.526 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:11.491728+0000", 2024-09-11T09:50:34.526 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 23.033691096999998, 2024-09-11T09:50:34.526 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00093269800000000003, 2024-09-11T09:50:34.526 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:50:34.526 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:50:34.526 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:34.526 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:50:34.526 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.526 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:50:34.526 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.491728+0000", 2024-09-11T09:50:34.526 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:34.526 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:34.526 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.527 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:50:34.527 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.491728+0000", 2024-09-11T09:50:34.527 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:34.527 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:34.527 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.527 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:50:34.527 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.491729+0000", 2024-09-11T09:50:34.527 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.8599999999999997e-07 2024-09-11T09:50:34.527 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:34.527 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.527 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:50:34.527 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.491820+0000", 2024-09-11T09:50:34.527 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.0812999999999999e-05 2024-09-11T09:50:34.528 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:34.528 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.528 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:50:34.528 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.491833+0000", 2024-09-11T09:50:34.528 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.341e-05 2024-09-11T09:50:34.528 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:34.528 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.528 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:50:34.528 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.491841+0000", 2024-09-11T09:50:34.528 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.8730000000000007e-06 2024-09-11T09:50:34.528 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:34.528 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.528 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:50:34.529 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.491916+0000", 2024-09-11T09:50:34.529 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.4881000000000004e-05 2024-09-11T09:50:34.529 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:34.529 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.529 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:50:34.529 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.491931+0000", 2024-09-11T09:50:34.529 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.4825e-05 2024-09-11T09:50:34.529 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:34.529 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.529 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:50:34.529 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.492633+0000", 2024-09-11T09:50:34.529 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00070191600000000002 2024-09-11T09:50:34.529 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:34.530 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.530 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:50:34.530 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.492661+0000", 2024-09-11T09:50:34.530 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.8093999999999999e-05 2024-09-11T09:50:34.530 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:34.530 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:50:34.530 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:34.530 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:34.530 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.530 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4183.0:30 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'16, mlcod=13'14)", 2024-09-11T09:50:34.530 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:11.498866+0000", 2024-09-11T09:50:34.530 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 23.026553373999999, 2024-09-11T09:50:34.530 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00089294000000000005, 2024-09-11T09:50:34.530 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:50:34.531 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:50:34.531 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:34.531 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:50:34.531 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.531 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:50:34.531 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.498866+0000", 2024-09-11T09:50:34.531 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:34.531 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:34.531 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.531 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:50:34.531 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.498866+0000", 2024-09-11T09:50:34.532 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:34.532 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:34.532 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.532 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:50:34.532 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.498867+0000", 2024-09-11T09:50:34.532 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.7000000000000003e-07 2024-09-11T09:50:34.532 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:34.532 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.532 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:50:34.532 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.498873+0000", 2024-09-11T09:50:34.532 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.0140000000000004e-06 2024-09-11T09:50:34.533 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:34.533 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.533 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:50:34.533 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.498886+0000", 2024-09-11T09:50:34.533 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2745000000000001e-05 2024-09-11T09:50:34.533 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:34.533 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.533 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:50:34.533 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.498894+0000", 2024-09-11T09:50:34.533 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.2349999999999999e-06 2024-09-11T09:50:34.533 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:34.533 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.533 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:50:34.533 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.498941+0000", 2024-09-11T09:50:34.534 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.7679999999999998e-05 2024-09-11T09:50:34.534 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:34.534 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.534 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:50:34.534 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.498955+0000", 2024-09-11T09:50:34.534 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3446e-05 2024-09-11T09:50:34.534 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:34.534 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.534 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:50:34.534 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.499717+0000", 2024-09-11T09:50:34.534 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00076248199999999998 2024-09-11T09:50:34.534 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:34.535 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.535 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:50:34.535 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.499759+0000", 2024-09-11T09:50:34.535 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.1467999999999997e-05 2024-09-11T09:50:34.535 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:34.535 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:50:34.535 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:34.535 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:34.535 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.535 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4183.0:31 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'17, mlcod=13'14)", 2024-09-11T09:50:34.536 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:11.499319+0000", 2024-09-11T09:50:34.536 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 23.026100427999999, 2024-09-11T09:50:34.536 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0010073879999999999, 2024-09-11T09:50:34.536 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:50:34.536 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:50:34.536 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:34.536 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:50:34.536 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.536 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:50:34.536 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.499319+0000", 2024-09-11T09:50:34.536 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:34.537 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:34.537 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.537 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:50:34.537 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.499319+0000", 2024-09-11T09:50:34.537 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:34.537 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:34.537 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.537 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:50:34.537 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.499320+0000", 2024-09-11T09:50:34.537 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.7700000000000004e-07 2024-09-11T09:50:34.538 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:34.538 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.538 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:50:34.538 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.499326+0000", 2024-09-11T09:50:34.538 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.6420000000000001e-06 2024-09-11T09:50:34.538 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:34.538 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.539 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:50:34.539 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.499339+0000", 2024-09-11T09:50:34.539 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2537999999999999e-05 2024-09-11T09:50:34.539 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:34.539 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.539 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:50:34.539 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.499345+0000", 2024-09-11T09:50:34.539 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.8220000000000003e-06 2024-09-11T09:50:34.539 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:34.539 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.539 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:50:34.540 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.499403+0000", 2024-09-11T09:50:34.540 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.8814000000000003e-05 2024-09-11T09:50:34.540 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:34.540 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.540 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:50:34.540 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.499427+0000", 2024-09-11T09:50:34.540 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.3853999999999999e-05 2024-09-11T09:50:34.540 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:34.540 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.540 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:50:34.540 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.500294+0000", 2024-09-11T09:50:34.541 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00086622399999999997 2024-09-11T09:50:34.541 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:34.541 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.541 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:50:34.541 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.500326+0000", 2024-09-11T09:50:34.541 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.2716999999999999e-05 2024-09-11T09:50:34.541 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:34.541 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:50:34.541 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:34.541 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:34.541 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.541 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4183.0:32 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'18, mlcod=13'14)", 2024-09-11T09:50:34.541 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:11.499796+0000", 2024-09-11T09:50:34.542 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 23.025623222, 2024-09-11T09:50:34.542 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00092915100000000002, 2024-09-11T09:50:34.542 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:50:34.542 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:50:34.542 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:34.542 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:50:34.542 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.542 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:50:34.542 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.499796+0000", 2024-09-11T09:50:34.542 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:34.542 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:34.542 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.542 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:50:34.543 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.499796+0000", 2024-09-11T09:50:34.543 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:34.543 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:34.543 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.543 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:50:34.543 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.499797+0000", 2024-09-11T09:50:34.543 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.9400000000000004e-07 2024-09-11T09:50:34.543 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:34.543 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.543 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:50:34.543 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.499841+0000", 2024-09-11T09:50:34.543 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.3900000000000003e-05 2024-09-11T09:50:34.543 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:34.543 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.544 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:50:34.544 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.499852+0000", 2024-09-11T09:50:34.544 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1462e-05 2024-09-11T09:50:34.544 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:34.544 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.544 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:50:34.544 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.499858+0000", 2024-09-11T09:50:34.544 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.8320000000000002e-06 2024-09-11T09:50:34.544 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:34.544 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.545 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:50:34.545 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.499915+0000", 2024-09-11T09:50:34.545 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.7187e-05 2024-09-11T09:50:34.545 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:34.545 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.545 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:50:34.545 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.499930+0000", 2024-09-11T09:50:34.545 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.4928e-05 2024-09-11T09:50:34.545 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:34.545 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.545 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:50:34.546 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.500697+0000", 2024-09-11T09:50:34.546 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00076658399999999995 2024-09-11T09:50:34.546 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:34.546 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.546 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:50:34.546 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.500725+0000", 2024-09-11T09:50:34.546 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.8364000000000002e-05 2024-09-11T09:50:34.546 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:34.546 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:50:34.546 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:34.546 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:34.546 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.547 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4183.0:34 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'20, mlcod=13'15)", 2024-09-11T09:50:34.547 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:11.501016+0000", 2024-09-11T09:50:34.547 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 23.024403704000001, 2024-09-11T09:50:34.547 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00097815399999999996, 2024-09-11T09:50:34.547 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:50:34.547 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:50:34.547 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:34.547 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:50:34.547 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.547 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:50:34.548 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.501016+0000", 2024-09-11T09:50:34.548 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:34.548 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:34.548 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.548 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:50:34.548 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.501016+0000", 2024-09-11T09:50:34.548 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:34.548 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:34.548 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.548 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:50:34.548 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.501016+0000", 2024-09-11T09:50:34.549 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.3799999999999997e-07 2024-09-11T09:50:34.549 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:34.549 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.549 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:50:34.549 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.501023+0000", 2024-09-11T09:50:34.549 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.2180000000000002e-06 2024-09-11T09:50:34.549 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:34.549 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.549 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:50:34.549 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.501031+0000", 2024-09-11T09:50:34.550 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.5379999999999997e-06 2024-09-11T09:50:34.550 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:34.550 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.550 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:50:34.550 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.501035+0000", 2024-09-11T09:50:34.550 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.216e-06 2024-09-11T09:50:34.550 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:34.550 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.550 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:50:34.550 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.501105+0000", 2024-09-11T09:50:34.550 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.9757000000000003e-05 2024-09-11T09:50:34.550 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:34.550 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.550 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:50:34.551 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.501120+0000", 2024-09-11T09:50:34.551 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.5358000000000001e-05 2024-09-11T09:50:34.551 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:34.551 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.551 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:50:34.551 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.501955+0000", 2024-09-11T09:50:34.551 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00083505000000000005 2024-09-11T09:50:34.551 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:34.551 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.551 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:50:34.551 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.501994+0000", 2024-09-11T09:50:34.551 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.8479000000000002e-05 2024-09-11T09:50:34.551 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:34.551 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:50:34.552 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:34.552 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:34.552 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.552 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4183.0:38 1.0 e13/12 1:30306672:devicehealth::main.db.0000000000000000:head v 13'24, mlcod=13'22)", 2024-09-11T09:50:34.552 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:11.503891+0000", 2024-09-11T09:50:34.552 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 23.021528262, 2024-09-11T09:50:34.552 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000959351, 2024-09-11T09:50:34.552 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:50:34.552 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:50:34.552 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:34.552 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:50:34.552 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.552 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:50:34.553 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.503891+0000", 2024-09-11T09:50:34.553 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:34.553 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:34.553 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.553 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:50:34.553 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.503891+0000", 2024-09-11T09:50:34.553 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:34.553 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:34.553 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.553 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:50:34.553 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.503892+0000", 2024-09-11T09:50:34.553 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.5499999999999998e-07 2024-09-11T09:50:34.553 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:34.553 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.554 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:50:34.554 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.503967+0000", 2024-09-11T09:50:34.554 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.4993999999999994e-05 2024-09-11T09:50:34.554 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:34.554 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.554 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:50:34.554 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.503976+0000", 2024-09-11T09:50:34.554 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.4709999999999996e-06 2024-09-11T09:50:34.554 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:34.554 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.555 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:50:34.555 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.503981+0000", 2024-09-11T09:50:34.555 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.6789999999999996e-06 2024-09-11T09:50:34.555 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:34.555 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.555 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:50:34.555 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.504021+0000", 2024-09-11T09:50:34.555 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.0065999999999999e-05 2024-09-11T09:50:34.555 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:34.555 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.555 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:50:34.556 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.504036+0000", 2024-09-11T09:50:34.556 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.468e-05 2024-09-11T09:50:34.556 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:34.556 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.556 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:50:34.556 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.504813+0000", 2024-09-11T09:50:34.556 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00077718299999999998 2024-09-11T09:50:34.556 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:34.556 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.556 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:50:34.556 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.504850+0000", 2024-09-11T09:50:34.556 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.7722999999999998e-05 2024-09-11T09:50:34.556 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:34.557 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:50:34.557 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:34.557 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:34.557 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.557 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4183.0:54 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'32, mlcod=13'29)", 2024-09-11T09:50:34.557 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:11.515404+0000", 2024-09-11T09:50:34.557 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 23.010015657, 2024-09-11T09:50:34.557 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00089471799999999997, 2024-09-11T09:50:34.557 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:50:34.557 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:50:34.557 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:34.557 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:50:34.557 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.558 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:50:34.558 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.515404+0000", 2024-09-11T09:50:34.558 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:34.558 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:34.558 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.558 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:50:34.558 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.515404+0000", 2024-09-11T09:50:34.558 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:34.558 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:34.558 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.558 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:50:34.558 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.515404+0000", 2024-09-11T09:50:34.558 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.3799999999999996e-07 2024-09-11T09:50:34.558 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:34.559 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.559 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:50:34.559 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.515410+0000", 2024-09-11T09:50:34.559 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.1789999999999999e-06 2024-09-11T09:50:34.559 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:34.559 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.559 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:50:34.559 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.515422+0000", 2024-09-11T09:50:34.559 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1929999999999999e-05 2024-09-11T09:50:34.559 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:34.559 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.559 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:50:34.560 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.515427+0000", 2024-09-11T09:50:34.560 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.6060000000000002e-06 2024-09-11T09:50:34.560 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:34.560 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.560 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:50:34.560 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.515482+0000", 2024-09-11T09:50:34.560 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.4743000000000001e-05 2024-09-11T09:50:34.560 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:34.560 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.560 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:50:34.560 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.515498+0000", 2024-09-11T09:50:34.561 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.5778999999999999e-05 2024-09-11T09:50:34.561 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:34.561 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.561 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:50:34.561 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.516271+0000", 2024-09-11T09:50:34.561 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00077312599999999996 2024-09-11T09:50:34.561 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:34.561 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.561 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:50:34.561 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.516298+0000", 2024-09-11T09:50:34.561 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.7617e-05 2024-09-11T09:50:34.561 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:34.561 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:50:34.562 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:34.562 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:34.562 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.562 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4183.0:57 1.0 e13/12 1:30306672:devicehealth::main.db.0000000000000000:head v 13'35, mlcod=13'33)", 2024-09-11T09:50:34.562 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:11.519823+0000", 2024-09-11T09:50:34.562 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 23.005596600000001, 2024-09-11T09:50:34.562 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00090876100000000001, 2024-09-11T09:50:34.562 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:50:34.562 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:50:34.562 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:34.562 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:50:34.562 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.562 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:50:34.562 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.519823+0000", 2024-09-11T09:50:34.563 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:34.563 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:34.563 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.563 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:50:34.563 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.519823+0000", 2024-09-11T09:50:34.563 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:34.563 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:34.563 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.563 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:50:34.563 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.519823+0000", 2024-09-11T09:50:34.563 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.7000000000000004e-07 2024-09-11T09:50:34.564 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:34.564 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.564 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:50:34.564 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.519859+0000", 2024-09-11T09:50:34.564 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.5508e-05 2024-09-11T09:50:34.564 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:34.564 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.564 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:50:34.564 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.519871+0000", 2024-09-11T09:50:34.564 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.221e-05 2024-09-11T09:50:34.564 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:34.564 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.564 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:50:34.565 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.519877+0000", 2024-09-11T09:50:34.565 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.0410000000000003e-06 2024-09-11T09:50:34.565 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:34.565 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.565 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:50:34.565 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.519955+0000", 2024-09-11T09:50:34.565 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.8169000000000003e-05 2024-09-11T09:50:34.565 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:34.565 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.565 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:50:34.565 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.519972+0000", 2024-09-11T09:50:34.566 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6595999999999998e-05 2024-09-11T09:50:34.566 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:34.566 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.566 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:50:34.566 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.520691+0000", 2024-09-11T09:50:34.566 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000719405 2024-09-11T09:50:34.566 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:34.566 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.567 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:50:34.567 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.520731+0000", 2024-09-11T09:50:34.567 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.0061999999999997e-05 2024-09-11T09:50:34.567 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:34.567 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:50:34.567 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:34.567 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:34.567 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.567 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4183.0:58 1.0 e13/12 1:30306672:devicehealth::main.db.0000000000000000:head v 13'36, mlcod=13'33)", 2024-09-11T09:50:34.567 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:11.520285+0000", 2024-09-11T09:50:34.567 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 23.005134192, 2024-09-11T09:50:34.567 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00091014799999999995, 2024-09-11T09:50:34.568 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:50:34.568 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:50:34.568 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:34.568 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:50:34.568 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.568 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:50:34.568 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.520285+0000", 2024-09-11T09:50:34.568 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:34.568 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:34.568 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.568 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:50:34.568 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.520285+0000", 2024-09-11T09:50:34.568 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:34.569 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:34.569 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.569 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:50:34.569 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.520286+0000", 2024-09-11T09:50:34.569 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.4399999999999999e-07 2024-09-11T09:50:34.569 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:34.569 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.569 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:50:34.569 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.520348+0000", 2024-09-11T09:50:34.569 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.1760000000000005e-05 2024-09-11T09:50:34.569 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:34.569 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.569 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:50:34.570 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.520359+0000", 2024-09-11T09:50:34.570 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1362999999999999e-05 2024-09-11T09:50:34.570 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:34.570 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.570 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:50:34.570 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.520364+0000", 2024-09-11T09:50:34.570 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.3600000000000004e-06 2024-09-11T09:50:34.570 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:34.570 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.570 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:50:34.571 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.520413+0000", 2024-09-11T09:50:34.571 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.8226000000000002e-05 2024-09-11T09:50:34.571 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:34.571 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.571 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:50:34.571 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.520426+0000", 2024-09-11T09:50:34.571 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3560000000000001e-05 2024-09-11T09:50:34.571 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:34.571 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.571 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:50:34.571 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.521159+0000", 2024-09-11T09:50:34.571 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00073255499999999997 2024-09-11T09:50:34.571 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:34.571 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.572 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:50:34.572 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.521195+0000", 2024-09-11T09:50:34.572 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.6480000000000003e-05 2024-09-11T09:50:34.572 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:34.572 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:50:34.572 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:34.572 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:34.572 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.572 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4183.0:68 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'42, mlcod=13'39)", 2024-09-11T09:50:34.572 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:11.534957+0000", 2024-09-11T09:50:34.572 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 22.990462501, 2024-09-11T09:50:34.573 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.001065351, 2024-09-11T09:50:34.573 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:50:34.573 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:50:34.573 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:34.573 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:50:34.573 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.573 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:50:34.573 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.534957+0000", 2024-09-11T09:50:34.573 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:34.574 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:34.574 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.574 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:50:34.574 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.534957+0000", 2024-09-11T09:50:34.574 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:34.574 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:34.574 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.574 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:50:34.574 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.534957+0000", 2024-09-11T09:50:34.574 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.3799999999999998e-07 2024-09-11T09:50:34.574 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:34.575 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.575 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:50:34.575 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.534964+0000", 2024-09-11T09:50:34.575 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.9809999999999997e-06 2024-09-11T09:50:34.575 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:34.575 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.575 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:50:34.575 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.534976+0000", 2024-09-11T09:50:34.575 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2119999999999999e-05 2024-09-11T09:50:34.575 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:34.575 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.575 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:50:34.575 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.534980+0000", 2024-09-11T09:50:34.576 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.9890000000000003e-06 2024-09-11T09:50:34.576 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:34.576 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.576 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:50:34.576 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.535030+0000", 2024-09-11T09:50:34.576 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.9258000000000002e-05 2024-09-11T09:50:34.576 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:34.576 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.576 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:50:34.576 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.535044+0000", 2024-09-11T09:50:34.576 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3981e-05 2024-09-11T09:50:34.576 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:34.576 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.576 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:50:34.577 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.535995+0000", 2024-09-11T09:50:34.577 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000951804 2024-09-11T09:50:34.577 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:34.577 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.577 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:50:34.577 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.536022+0000", 2024-09-11T09:50:34.577 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.6780000000000001e-05 2024-09-11T09:50:34.577 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:34.577 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:50:34.578 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:34.578 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:34.578 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.578 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4183.0:69 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'43, mlcod=13'39)", 2024-09-11T09:50:34.578 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:11.535331+0000", 2024-09-11T09:50:34.578 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 22.990088178000001, 2024-09-11T09:50:34.578 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0011256090000000001, 2024-09-11T09:50:34.578 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:50:34.578 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:50:34.578 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:34.578 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:50:34.578 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.579 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:50:34.579 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.535331+0000", 2024-09-11T09:50:34.579 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:34.579 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:34.579 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.579 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:50:34.579 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.535331+0000", 2024-09-11T09:50:34.579 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:34.579 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:34.579 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.579 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:50:34.579 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.535332+0000", 2024-09-11T09:50:34.579 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.1600000000000001e-07 2024-09-11T09:50:34.580 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:34.580 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.580 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:50:34.580 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.535393+0000", 2024-09-11T09:50:34.580 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.1452999999999996e-05 2024-09-11T09:50:34.580 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:34.580 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.580 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:50:34.580 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.535402+0000", 2024-09-11T09:50:34.580 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.4300000000000006e-06 2024-09-11T09:50:34.580 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:34.580 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.581 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:50:34.581 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.535405+0000", 2024-09-11T09:50:34.581 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.512e-06 2024-09-11T09:50:34.581 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:34.581 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.581 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:50:34.581 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.535482+0000", 2024-09-11T09:50:34.581 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.7132999999999995e-05 2024-09-11T09:50:34.581 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:34.581 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.581 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:50:34.582 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.535506+0000", 2024-09-11T09:50:34.582 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.4070000000000002e-05 2024-09-11T09:50:34.582 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:34.582 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.582 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:50:34.582 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.536424+0000", 2024-09-11T09:50:34.582 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00091766700000000005 2024-09-11T09:50:34.582 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:34.582 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.582 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:50:34.582 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.536457+0000", 2024-09-11T09:50:34.582 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.2827999999999998e-05 2024-09-11T09:50:34.582 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:34.582 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:50:34.583 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:34.583 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:34.583 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.583 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4183.0:83 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'53, mlcod=13'50)", 2024-09-11T09:50:34.583 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:11.558836+0000", 2024-09-11T09:50:34.583 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 22.966582923000001, 2024-09-11T09:50:34.583 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00090313100000000005, 2024-09-11T09:50:34.583 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:50:34.583 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:50:34.583 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:34.583 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:50:34.583 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.583 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:50:34.584 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.558836+0000", 2024-09-11T09:50:34.584 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:34.584 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:34.584 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.584 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:50:34.584 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.558836+0000", 2024-09-11T09:50:34.584 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:34.584 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:34.584 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.584 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:50:34.584 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.558837+0000", 2024-09-11T09:50:34.584 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.4900000000000005e-07 2024-09-11T09:50:34.585 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:34.585 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.585 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:50:34.585 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.558857+0000", 2024-09-11T09:50:34.585 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.9502999999999999e-05 2024-09-11T09:50:34.585 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:34.585 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.585 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:50:34.585 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.558870+0000", 2024-09-11T09:50:34.585 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3020000000000001e-05 2024-09-11T09:50:34.585 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:34.585 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.585 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:50:34.585 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.558876+0000", 2024-09-11T09:50:34.586 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.7270000000000004e-06 2024-09-11T09:50:34.586 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:34.586 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.586 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:50:34.586 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.558934+0000", 2024-09-11T09:50:34.586 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.8100999999999998e-05 2024-09-11T09:50:34.586 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:34.586 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.586 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:50:34.586 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.558947+0000", 2024-09-11T09:50:34.586 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3269e-05 2024-09-11T09:50:34.586 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:34.586 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.587 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:50:34.587 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.559720+0000", 2024-09-11T09:50:34.587 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00077287600000000003 2024-09-11T09:50:34.587 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:34.587 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.587 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:50:34.587 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.559740+0000", 2024-09-11T09:50:34.587 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.9786e-05 2024-09-11T09:50:34.587 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:34.587 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:50:34.587 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:34.587 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:34.587 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.587 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4183.0:84 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'54, mlcod=13'50)", 2024-09-11T09:50:34.588 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:11.559267+0000", 2024-09-11T09:50:34.588 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 22.966152495999999, 2024-09-11T09:50:34.588 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00093990199999999995, 2024-09-11T09:50:34.588 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:50:34.588 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:50:34.588 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:34.588 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:50:34.588 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.588 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:50:34.588 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.559267+0000", 2024-09-11T09:50:34.588 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:34.588 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:34.588 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.589 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:50:34.589 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.559267+0000", 2024-09-11T09:50:34.589 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:34.589 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:34.589 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.589 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:50:34.589 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.559267+0000", 2024-09-11T09:50:34.589 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.2900000000000003e-07 2024-09-11T09:50:34.589 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:34.590 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.590 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:50:34.590 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.559322+0000", 2024-09-11T09:50:34.590 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.4429999999999999e-05 2024-09-11T09:50:34.590 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:34.590 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.590 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:50:34.590 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.559332+0000", 2024-09-11T09:50:34.590 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0473e-05 2024-09-11T09:50:34.590 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:34.590 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.590 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:50:34.590 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.559338+0000", 2024-09-11T09:50:34.590 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.0730000000000001e-06 2024-09-11T09:50:34.591 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:34.591 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.591 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:50:34.591 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.559373+0000", 2024-09-11T09:50:34.591 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.4626999999999999e-05 2024-09-11T09:50:34.591 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:34.591 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.591 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:50:34.591 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.559384+0000", 2024-09-11T09:50:34.592 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1307e-05 2024-09-11T09:50:34.592 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:34.592 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.592 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:50:34.592 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.560175+0000", 2024-09-11T09:50:34.592 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00079016399999999999 2024-09-11T09:50:34.592 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:34.592 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.592 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:50:34.592 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.560207+0000", 2024-09-11T09:50:34.592 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.2199000000000001e-05 2024-09-11T09:50:34.592 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:34.592 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:50:34.593 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:34.593 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:34.593 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.593 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4183.0:89 1.0 e13/12 1:30306672:devicehealth::main.db.0000000000000000:head v 13'59, mlcod=13'57)", 2024-09-11T09:50:34.593 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:11.563614+0000", 2024-09-11T09:50:34.593 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 22.961804875999999, 2024-09-11T09:50:34.593 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00091281399999999996, 2024-09-11T09:50:34.593 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:50:34.593 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:50:34.593 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:34.593 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:50:34.593 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.593 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:50:34.593 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.563614+0000", 2024-09-11T09:50:34.594 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:34.594 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:34.594 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.594 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:50:34.594 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.563614+0000", 2024-09-11T09:50:34.594 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:34.594 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:34.594 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.594 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:50:34.594 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.563615+0000", 2024-09-11T09:50:34.594 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.6899999999999998e-07 2024-09-11T09:50:34.594 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:34.594 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.595 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:50:34.595 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.563681+0000", 2024-09-11T09:50:34.595 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.5769e-05 2024-09-11T09:50:34.595 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:34.595 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.595 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:50:34.595 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.563690+0000", 2024-09-11T09:50:34.595 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.6800000000000005e-06 2024-09-11T09:50:34.595 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:34.595 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.595 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:50:34.595 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.563699+0000", 2024-09-11T09:50:34.595 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.3170000000000002e-06 2024-09-11T09:50:34.595 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:34.596 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.596 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:50:34.596 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.563734+0000", 2024-09-11T09:50:34.596 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.5023e-05 2024-09-11T09:50:34.596 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:34.596 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.596 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:50:34.596 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.563746+0000", 2024-09-11T09:50:34.596 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2661e-05 2024-09-11T09:50:34.596 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:34.597 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.597 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:50:34.597 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.564500+0000", 2024-09-11T09:50:34.597 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00075323100000000004 2024-09-11T09:50:34.597 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:34.597 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.597 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:50:34.598 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.564527+0000", 2024-09-11T09:50:34.598 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.7664000000000001e-05 2024-09-11T09:50:34.598 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:34.598 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:50:34.598 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:34.598 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:34.599 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.599 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4183.0:105 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'67, mlcod=13'64)", 2024-09-11T09:50:34.599 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:11.712146+0000", 2024-09-11T09:50:34.599 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 22.813273125999999, 2024-09-11T09:50:34.599 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0010916859999999999, 2024-09-11T09:50:34.599 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:50:34.599 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:50:34.600 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:34.600 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:50:34.600 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.600 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:50:34.600 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.712146+0000", 2024-09-11T09:50:34.601 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:34.601 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:34.601 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.601 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:50:34.601 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.712146+0000", 2024-09-11T09:50:34.601 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:34.601 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:34.602 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.602 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:50:34.602 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.712147+0000", 2024-09-11T09:50:34.602 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.8200000000000001e-07 2024-09-11T09:50:34.602 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:34.602 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.603 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:50:34.603 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.712155+0000", 2024-09-11T09:50:34.603 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.0309999999999998e-06 2024-09-11T09:50:34.603 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:34.603 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.603 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:50:34.604 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.712163+0000", 2024-09-11T09:50:34.604 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.5920000000000001e-06 2024-09-11T09:50:34.604 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:34.604 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.604 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:50:34.604 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.712169+0000", 2024-09-11T09:50:34.605 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.4299999999999997e-06 2024-09-11T09:50:34.605 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:34.605 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.605 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:50:34.605 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.712418+0000", 2024-09-11T09:50:34.605 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00024978499999999998 2024-09-11T09:50:34.606 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:34.606 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.606 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:50:34.606 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.712432+0000", 2024-09-11T09:50:34.606 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3190000000000001e-05 2024-09-11T09:50:34.606 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:34.607 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.607 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:50:34.607 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.713206+0000", 2024-09-11T09:50:34.607 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00077420199999999998 2024-09-11T09:50:34.607 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:34.608 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.608 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:50:34.608 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.713238+0000", 2024-09-11T09:50:34.608 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.2073999999999998e-05 2024-09-11T09:50:34.608 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:34.608 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:50:34.609 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:34.609 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:34.609 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.609 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4183.0:106 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'68, mlcod=13'64)", 2024-09-11T09:50:34.609 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:11.712624+0000", 2024-09-11T09:50:34.609 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 22.812795798, 2024-09-11T09:50:34.610 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0010009629999999999, 2024-09-11T09:50:34.610 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:50:34.610 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:50:34.610 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:34.610 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:50:34.610 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.611 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:50:34.611 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.712624+0000", 2024-09-11T09:50:34.611 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:34.611 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:34.611 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.611 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:50:34.612 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.712624+0000", 2024-09-11T09:50:34.612 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:34.612 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:34.612 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.612 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:50:34.612 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.712624+0000", 2024-09-11T09:50:34.613 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.4799999999999998e-07 2024-09-11T09:50:34.613 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:34.613 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.613 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:50:34.613 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.712689+0000", 2024-09-11T09:50:34.613 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.4488999999999999e-05 2024-09-11T09:50:34.614 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:34.614 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.614 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:50:34.614 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.712707+0000", 2024-09-11T09:50:34.614 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.8272000000000001e-05 2024-09-11T09:50:34.614 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:34.615 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.615 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:50:34.615 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.712713+0000", 2024-09-11T09:50:34.615 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.2040000000000002e-06 2024-09-11T09:50:34.615 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:34.616 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.616 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:50:34.616 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.712783+0000", 2024-09-11T09:50:34.616 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.9534000000000002e-05 2024-09-11T09:50:34.616 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:34.616 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.617 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:50:34.617 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.712811+0000", 2024-09-11T09:50:34.617 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.8620999999999999e-05 2024-09-11T09:50:34.617 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:34.617 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.617 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:50:34.618 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.713586+0000", 2024-09-11T09:50:34.618 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00077462600000000005 2024-09-11T09:50:34.618 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:34.618 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.618 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:50:34.618 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.713624+0000", 2024-09-11T09:50:34.619 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.8668999999999999e-05 2024-09-11T09:50:34.619 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:34.619 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:50:34.619 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:34.619 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:34.619 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.620 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4183.0:118 1.0 e13/12 1:30306672:devicehealth::main.db.0000000000000000:head v 13'80, mlcod=13'78)", 2024-09-11T09:50:34.620 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:11.726126+0000", 2024-09-11T09:50:34.620 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 22.799293706, 2024-09-11T09:50:34.620 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00087776200000000003, 2024-09-11T09:50:34.620 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:50:34.621 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:50:34.621 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:34.621 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:50:34.621 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.621 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:50:34.621 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.726126+0000", 2024-09-11T09:50:34.622 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:34.622 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:34.622 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.622 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:50:34.622 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.726126+0000", 2024-09-11T09:50:34.623 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:34.623 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:34.623 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.623 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:50:34.623 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.726126+0000", 2024-09-11T09:50:34.623 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.2099999999999996e-07 2024-09-11T09:50:34.624 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:34.624 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.624 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:50:34.624 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.726167+0000", 2024-09-11T09:50:34.624 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.0621999999999998e-05 2024-09-11T09:50:34.624 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:34.625 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.625 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:50:34.625 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.726177+0000", 2024-09-11T09:50:34.625 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0299000000000001e-05 2024-09-11T09:50:34.625 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:34.625 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.626 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:50:34.626 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.726183+0000", 2024-09-11T09:50:34.626 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.2079999999999997e-06 2024-09-11T09:50:34.626 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:34.626 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.627 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:50:34.627 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.726256+0000", 2024-09-11T09:50:34.627 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.2632000000000005e-05 2024-09-11T09:50:34.627 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:34.627 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.627 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:50:34.628 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.726268+0000", 2024-09-11T09:50:34.628 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2461e-05 2024-09-11T09:50:34.628 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:34.628 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.628 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:50:34.628 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.726977+0000", 2024-09-11T09:50:34.629 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00070853500000000005 2024-09-11T09:50:34.629 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:34.629 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.629 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:50:34.629 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.727003+0000", 2024-09-11T09:50:34.630 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.6384e-05 2024-09-11T09:50:34.630 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:34.630 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:50:34.630 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:34.630 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:34.630 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.631 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4183.0:134 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'92, mlcod=13'86)", 2024-09-11T09:50:34.631 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:12.035698+0000", 2024-09-11T09:50:34.631 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 22.489721539000001, 2024-09-11T09:50:34.631 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00089550700000000003, 2024-09-11T09:50:34.631 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:50:34.631 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:50:34.632 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:34.632 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:50:34.632 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.632 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:50:34.632 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.035698+0000", 2024-09-11T09:50:34.632 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:34.633 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:34.633 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.633 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:50:34.633 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.035698+0000", 2024-09-11T09:50:34.633 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:34.633 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:34.634 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.634 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:50:34.634 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.035698+0000", 2024-09-11T09:50:34.634 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.0900000000000002e-07 2024-09-11T09:50:34.634 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:34.635 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.635 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:50:34.635 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.035758+0000", 2024-09-11T09:50:34.635 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.9481999999999997e-05 2024-09-11T09:50:34.635 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:34.635 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.636 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:50:34.636 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.035766+0000", 2024-09-11T09:50:34.636 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.3760000000000008e-06 2024-09-11T09:50:34.636 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:34.636 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.636 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:50:34.637 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.035772+0000", 2024-09-11T09:50:34.637 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.8930000000000002e-06 2024-09-11T09:50:34.637 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:34.637 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.637 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:50:34.638 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.035807+0000", 2024-09-11T09:50:34.638 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.4975999999999999e-05 2024-09-11T09:50:34.638 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:34.638 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.638 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:50:34.638 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.035817+0000", 2024-09-11T09:50:34.638 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0314e-05 2024-09-11T09:50:34.638 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:34.638 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.638 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:50:34.639 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.036542+0000", 2024-09-11T09:50:34.639 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00072434400000000003 2024-09-11T09:50:34.639 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:34.639 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.639 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:50:34.639 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.036593+0000", 2024-09-11T09:50:34.639 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.1613000000000002e-05 2024-09-11T09:50:34.639 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:34.639 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:50:34.640 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:34.640 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:34.640 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.640 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4183.0:136 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'94, mlcod=13'86)", 2024-09-11T09:50:34.640 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:12.036591+0000", 2024-09-11T09:50:34.640 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 22.488828329, 2024-09-11T09:50:34.640 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00090637900000000002, 2024-09-11T09:50:34.640 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:50:34.640 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:50:34.640 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:34.641 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:50:34.641 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.641 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:50:34.641 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.036591+0000", 2024-09-11T09:50:34.641 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:34.641 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:34.641 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.641 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:50:34.641 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.036591+0000", 2024-09-11T09:50:34.642 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:34.642 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:34.642 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.642 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:50:34.642 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.036592+0000", 2024-09-11T09:50:34.642 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.37e-07 2024-09-11T09:50:34.642 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:34.642 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.642 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:50:34.643 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.036600+0000", 2024-09-11T09:50:34.643 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.6910000000000003e-06 2024-09-11T09:50:34.643 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:34.643 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.643 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:50:34.643 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.036609+0000", 2024-09-11T09:50:34.643 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.0229999999999998e-06 2024-09-11T09:50:34.643 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:34.643 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.644 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:50:34.644 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.036617+0000", 2024-09-11T09:50:34.644 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.6979999999999998e-06 2024-09-11T09:50:34.644 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:34.644 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.644 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:50:34.644 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.036658+0000", 2024-09-11T09:50:34.644 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.0692000000000003e-05 2024-09-11T09:50:34.644 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:34.645 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.645 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:50:34.645 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.036670+0000", 2024-09-11T09:50:34.645 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2109e-05 2024-09-11T09:50:34.645 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:34.645 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.645 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:50:34.645 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.037475+0000", 2024-09-11T09:50:34.646 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00080487199999999999 2024-09-11T09:50:34.646 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:34.646 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.646 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:50:34.646 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.037497+0000", 2024-09-11T09:50:34.646 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.2557000000000001e-05 2024-09-11T09:50:34.646 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:34.646 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:50:34.646 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:34.646 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:34.647 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.647 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4183.0:158 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'112, mlcod=13'108)", 2024-09-11T09:50:34.647 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:12.455073+0000", 2024-09-11T09:50:34.647 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 22.070346076, 2024-09-11T09:50:34.647 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00090654500000000003, 2024-09-11T09:50:34.647 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:50:34.647 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:50:34.647 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:34.647 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:50:34.648 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.648 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:50:34.648 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.455073+0000", 2024-09-11T09:50:34.648 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:34.648 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:34.648 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.648 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:50:34.648 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.455073+0000", 2024-09-11T09:50:34.648 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:34.648 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:34.649 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.649 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:50:34.649 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.455074+0000", 2024-09-11T09:50:34.649 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.32e-07 2024-09-11T09:50:34.649 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:34.649 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.649 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:50:34.650 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.455126+0000", 2024-09-11T09:50:34.650 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.1898999999999999e-05 2024-09-11T09:50:34.650 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:34.650 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.650 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:50:34.650 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.455136+0000", 2024-09-11T09:50:34.650 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0651e-05 2024-09-11T09:50:34.651 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:34.651 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.651 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:50:34.651 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.455142+0000", 2024-09-11T09:50:34.651 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.0020000000000003e-06 2024-09-11T09:50:34.651 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:34.652 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.652 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:50:34.652 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.455192+0000", 2024-09-11T09:50:34.652 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.0164000000000003e-05 2024-09-11T09:50:34.652 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:34.653 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.653 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:50:34.653 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.455204+0000", 2024-09-11T09:50:34.653 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1178999999999999e-05 2024-09-11T09:50:34.653 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:34.653 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.653 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:50:34.654 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.455957+0000", 2024-09-11T09:50:34.654 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00075362300000000003 2024-09-11T09:50:34.654 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:34.654 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:34.654 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:50:34.655 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.455980+0000", 2024-09-11T09:50:34.655 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.2595e-05 2024-09-11T09:50:34.655 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:34.655 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:50:34.655 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:34.655 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:34.656 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:50:34.656 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:50:34.658 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-11T09:50:34.659 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:50:34.658+0000 7fdec5849640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31907) UID: 0 2024-09-11T09:50:34.758 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-09-11T09:50:34.759+0000 7f7b6c7d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31904) UID: 0 2024-09-11T09:50:34.807 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:50:34.807 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [], 2024-09-11T09:50:34.807 INFO:teuthology.orchestra.run.smithi017.stdout: "num_ops": 0 2024-09-11T09:50:34.807 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:50:34.815 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-11T09:50:34.859 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:50:34.860+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:50:34.923 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:50:34.923 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [], 2024-09-11T09:50:34.923 INFO:teuthology.orchestra.run.smithi017.stdout: "complaint_time": 30, 2024-09-11T09:50:34.924 INFO:teuthology.orchestra.run.smithi017.stdout: "num_blocked_ops": 0 2024-09-11T09:50:34.924 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:50:34.933 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-11T09:50:34.974 INFO:tasks.ceph.osd.2.smithi017.stderr:2024-09-11T09:50:34.975+0000 7fd20b170640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31672) UID: 0 2024-09-11T09:50:35.061 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-09-11T09:50:35.062+0000 7f7b6c7d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31904) UID: 0 2024-09-11T09:50:35.081 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:50:35.081 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 20, 2024-09-11T09:50:35.081 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 600, 2024-09-11T09:50:35.081 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [ 2024-09-11T09:50:35.081 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:35.081 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4230.0:5 2.2 e17/15 2:5cea7035:::rbd_info:head v 17'1, mlcod=0'0)", 2024-09-11T09:50:35.081 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:19.494757+0000", 2024-09-11T09:50:35.081 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 15.587715699, 2024-09-11T09:50:35.081 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00082098399999999997, 2024-09-11T09:50:35.081 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:50:35.082 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:50:35.082 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:35.082 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:50:35.082 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:35.082 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:50:35.082 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:19.494757+0000", 2024-09-11T09:50:35.082 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:35.082 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:35.082 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:35.082 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:50:35.082 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:19.494757+0000", 2024-09-11T09:50:35.082 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:35.082 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:35.082 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:35.083 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:50:35.083 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:19.494757+0000", 2024-09-11T09:50:35.083 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.97e-07 2024-09-11T09:50:35.083 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:35.083 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:35.083 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:50:35.083 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:19.494763+0000", 2024-09-11T09:50:35.083 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.3709999999999996e-06 2024-09-11T09:50:35.083 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:35.083 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:35.084 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:50:35.084 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:19.494778+0000", 2024-09-11T09:50:35.084 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.4960000000000001e-05 2024-09-11T09:50:35.084 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:35.084 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:35.084 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:50:35.084 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:19.494801+0000", 2024-09-11T09:50:35.084 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.2388999999999999e-05 2024-09-11T09:50:35.084 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:35.084 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:35.085 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:50:35.085 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:19.494859+0000", 2024-09-11T09:50:35.085 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.7961999999999998e-05 2024-09-11T09:50:35.085 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:35.085 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:35.085 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:50:35.085 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:19.494883+0000", 2024-09-11T09:50:35.085 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.4488999999999999e-05 2024-09-11T09:50:35.085 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:35.085 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:35.085 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:50:35.085 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:19.495534+0000", 2024-09-11T09:50:35.085 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00065075500000000004 2024-09-11T09:50:35.086 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:35.086 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:35.086 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:50:35.086 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:19.495578+0000", 2024-09-11T09:50:35.086 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.3860999999999999e-05 2024-09-11T09:50:35.086 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:35.086 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:50:35.086 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:35.086 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:35.086 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:35.086 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4230.0:7 2.2 e18/15 2:5cea7035:::rbd_info:head v 18'2, mlcod=0'0)", 2024-09-11T09:50:35.086 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:20.500782+0000", 2024-09-11T09:50:35.087 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 14.581690819, 2024-09-11T09:50:35.087 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000790185, 2024-09-11T09:50:35.087 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:50:35.087 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:50:35.087 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:35.087 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:50:35.087 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:35.087 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:50:35.087 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:20.500782+0000", 2024-09-11T09:50:35.087 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:35.087 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:35.087 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:35.087 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:50:35.087 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:20.500782+0000", 2024-09-11T09:50:35.088 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:35.088 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:35.088 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:35.088 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:50:35.088 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:20.500782+0000", 2024-09-11T09:50:35.088 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.3299999999999998e-07 2024-09-11T09:50:35.088 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:35.088 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:35.088 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:50:35.088 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:20.500799+0000", 2024-09-11T09:50:35.088 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6830999999999999e-05 2024-09-11T09:50:35.088 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:35.088 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:35.089 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:50:35.089 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:20.500812+0000", 2024-09-11T09:50:35.089 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3013999999999999e-05 2024-09-11T09:50:35.089 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:35.089 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:35.089 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:50:35.089 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:20.500824+0000", 2024-09-11T09:50:35.089 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2593e-05 2024-09-11T09:50:35.089 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:35.089 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:35.089 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:50:35.089 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:20.500895+0000", 2024-09-11T09:50:35.089 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.1018000000000003e-05 2024-09-11T09:50:35.090 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:35.090 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:35.090 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:50:35.090 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:20.500912+0000", 2024-09-11T09:50:35.090 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6484e-05 2024-09-11T09:50:35.090 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:35.090 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:35.090 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:50:35.090 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:20.501531+0000", 2024-09-11T09:50:35.090 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00061903399999999997 2024-09-11T09:50:35.090 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:35.090 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:35.090 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:50:35.090 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:20.501572+0000", 2024-09-11T09:50:35.091 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.0877999999999997e-05 2024-09-11T09:50:35.091 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:35.091 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:50:35.091 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:35.091 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:35.091 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:50:35.091 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:50:35.092 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-11T09:50:35.162 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-09-11T09:50:35.163+0000 7f7b6c7d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31904) UID: 0 2024-09-11T09:50:35.200 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:50:35.200 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [], 2024-09-11T09:50:35.200 INFO:teuthology.orchestra.run.smithi017.stdout: "num_ops": 0 2024-09-11T09:50:35.200 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:50:35.209 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-11T09:50:35.263 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:50:35.264+0000 7fdec5849640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31907) UID: 0 2024-09-11T09:50:35.278 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T09:50:35.317 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:50:35.317 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [], 2024-09-11T09:50:35.317 INFO:teuthology.orchestra.run.smithi017.stdout: "complaint_time": 30, 2024-09-11T09:50:35.317 INFO:teuthology.orchestra.run.smithi017.stdout: "num_blocked_ops": 0 2024-09-11T09:50:35.317 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:50:35.326 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-11T09:50:35.364 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:50:35.364+0000 7f3c96ec9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31909) UID: 0 2024-09-11T09:50:35.405 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-11T09:50:35.405 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T09:50:35.422 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-11T09:50:35.422 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T09:50:35.438 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-11T09:50:35.439 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T09:50:35.456 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-11T09:50:35.457 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T09:50:35.465 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-09-11T09:50:35.466+0000 7f7b6c7d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31904) UID: 0 2024-09-11T09:50:35.473 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-11T09:50:35.474 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T09:50:35.478 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:50:35.478 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:50:35.488 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-11T09:50:35.492 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-11T09:50:35.492 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T09:50:35.549 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-11T09:50:35.566 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:50:35.567+0000 7fdec5849640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31907) UID: 0 2024-09-11T09:50:35.607 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:50:35.607 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:50:35.617 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-11T09:50:35.700 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:50:35.701+0000 7fdec5849640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31907) UID: 0 2024-09-11T09:50:35.759 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:50:35.759 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:50:35.768 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-11T09:50:35.769 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:50:35.768+0000 7fdec5849640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31907) UID: 0 2024-09-11T09:50:35.869 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-09-11T09:50:35.869+0000 7f7b6c7d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31904) UID: 0 2024-09-11T09:50:35.912 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:50:35.913 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:50:35.921 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-11T09:50:36.005 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-09-11T09:50:36.006+0000 7f7b6c7d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31904) UID: 0 2024-09-11T09:50:36.065 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:50:36.065 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:50:36.070 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-09-11T09:50:36.071+0000 7f7b6c7d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31904) UID: 0 2024-09-11T09:50:36.073 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-11T09:50:36.172 INFO:tasks.ceph.osd.0.smithi017.stderr:2024-09-11T09:50:36.172+0000 7fb73c0f1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31665) UID: 0 2024-09-11T09:50:36.216 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:50:36.216 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:50:36.224 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-11T09:50:36.272 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:50:36.273+0000 7f3c96ec9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31909) UID: 0 2024-09-11T09:50:36.328 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:50:36.328 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:50:36.336 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-11T09:50:36.373 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:50:36.373+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:50:36.474 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:50:36.475+0000 7f3c96ec9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31909) UID: 0 2024-09-11T09:50:36.480 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:50:36.480 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:50:36.489 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-11T09:50:36.545 INFO:teuthology.orchestra.run.smithi017.stderr:nodeep-scrub is unset 2024-09-11T09:50:36.602 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:50:36.603+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:50:36.633 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:50:36.634 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:50:36.643 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-11T09:50:36.675 INFO:tasks.ceph.osd.0.smithi017.stderr:2024-09-11T09:50:36.676+0000 7fb73c0f1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31665) UID: 0 2024-09-11T09:50:36.746 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:50:36.746 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:50:36.755 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-11T09:50:36.776 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-09-11T09:50:36.777+0000 7f7b6c7d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31904) UID: 0 2024-09-11T09:50:36.873 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:50:36.874 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:50:36.877 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:50:36.878+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:50:36.883 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-11T09:50:36.978 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:50:36.979+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:50:37.031 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:50:37.031 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:50:37.040 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-11T09:50:37.081 INFO:tasks.ceph.osd.2.smithi017.stderr:2024-09-11T09:50:37.082+0000 7fd20b170640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31672) UID: 0 2024-09-11T09:50:37.180 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-09-11T09:50:37.181+0000 7f7b6c7d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31904) UID: 0 2024-09-11T09:50:37.188 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:50:37.188 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:50:37.197 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-11T09:50:37.281 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:50:37.281+0000 7fdec5849640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31907) UID: 0 2024-09-11T09:50:37.305 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:50:37.305 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:50:37.314 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-11T09:50:37.381 INFO:tasks.ceph.osd.0.smithi017.stderr:2024-09-11T09:50:37.382+0000 7fb73c0f1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31665) UID: 0 2024-09-11T09:50:37.421 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:50:37.421 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:50:37.429 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-11T09:50:37.503 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T09:50:37.514 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:50:37.515+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:50:37.582 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:50:37.582 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:50:37.583 INFO:tasks.ceph.osd.2.smithi017.stderr:2024-09-11T09:50:37.584+0000 7fd20b170640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31672) UID: 0 2024-09-11T09:50:37.591 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-11T09:50:37.639 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-11T09:50:37.639 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T09:50:37.658 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-11T09:50:37.658 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T09:50:37.675 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-11T09:50:37.675 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T09:50:37.685 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:50:37.685+0000 7f3c96ec9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31909) UID: 0 2024-09-11T09:50:37.692 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-11T09:50:37.693 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T09:50:37.710 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-11T09:50:37.710 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T09:50:37.728 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-11T09:50:37.728 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T09:50:37.741 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:50:37.741 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [], 2024-09-11T09:50:37.742 INFO:teuthology.orchestra.run.smithi017.stdout: "num_ops": 0 2024-09-11T09:50:37.742 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:50:37.751 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-11T09:50:37.792 INFO:tasks.ceph.osd.0.smithi017.stderr:2024-09-11T09:50:37.793+0000 7fb73c0f1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31665) UID: 0 2024-09-11T09:50:37.886 INFO:tasks.ceph.osd.2.smithi017.stderr:2024-09-11T09:50:37.887+0000 7fd20b170640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31672) UID: 0 2024-09-11T09:50:37.898 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:50:37.898 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [], 2024-09-11T09:50:37.898 INFO:teuthology.orchestra.run.smithi017.stdout: "complaint_time": 30, 2024-09-11T09:50:37.899 INFO:teuthology.orchestra.run.smithi017.stdout: "num_blocked_ops": 0 2024-09-11T09:50:37.899 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:50:37.908 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-11T09:50:37.992 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:50:37.993+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:50:38.056 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:50:38.056 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 20, 2024-09-11T09:50:38.056 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 600, 2024-09-11T09:50:38.056 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [ 2024-09-11T09:50:38.056 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:38.056 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4230.0:1 2.3 e16/15 2:c4c92e5a:::rbd_trash:head v 16'1, mlcod=0'0)", 2024-09-11T09:50:38.057 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:18.494378+0000", 2024-09-11T09:50:38.057 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 19.563149605, 2024-09-11T09:50:38.057 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00089752199999999999, 2024-09-11T09:50:38.057 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:50:38.057 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:50:38.057 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:38.057 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:50:38.057 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:38.057 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:50:38.057 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:18.494378+0000", 2024-09-11T09:50:38.057 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:38.057 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:38.057 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:38.057 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:50:38.058 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:18.494378+0000", 2024-09-11T09:50:38.058 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:38.058 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:38.058 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:38.058 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:50:38.058 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:18.494378+0000", 2024-09-11T09:50:38.058 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.9099999999999999e-07 2024-09-11T09:50:38.058 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:38.058 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:38.058 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:50:38.058 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:18.494385+0000", 2024-09-11T09:50:38.058 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.8129999999999998e-06 2024-09-11T09:50:38.058 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:38.059 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:38.059 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:50:38.059 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:18.494416+0000", 2024-09-11T09:50:38.059 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.0654000000000002e-05 2024-09-11T09:50:38.059 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:38.059 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:38.059 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:50:38.059 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:18.494430+0000", 2024-09-11T09:50:38.059 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3754000000000001e-05 2024-09-11T09:50:38.059 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:38.059 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:38.059 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:50:38.059 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:18.494524+0000", 2024-09-11T09:50:38.059 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.4896999999999996e-05 2024-09-11T09:50:38.060 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:38.060 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:38.060 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:50:38.060 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:18.494558+0000", 2024-09-11T09:50:38.060 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.3900999999999999e-05 2024-09-11T09:50:38.060 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:38.060 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:38.060 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:50:38.060 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:18.495231+0000", 2024-09-11T09:50:38.060 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00067233500000000003 2024-09-11T09:50:38.060 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:38.060 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:38.060 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:50:38.060 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:18.495275+0000", 2024-09-11T09:50:38.061 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.4777000000000002e-05 2024-09-11T09:50:38.061 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:38.061 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:50:38.061 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:38.061 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:38.061 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:50:38.061 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:50:38.065 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-11T09:50:38.109 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-09-11T09:50:38.110+0000 7f7b6c7d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31904) UID: 0 2024-09-11T09:50:38.180 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-11T09:50:38.180 INFO:teuthology.orchestra.run.smithi096.stdout: "ops": [], 2024-09-11T09:50:38.180 INFO:teuthology.orchestra.run.smithi096.stdout: "num_ops": 0 2024-09-11T09:50:38.180 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-11T09:50:38.188 INFO:tasks.ceph.osd.2.smithi017.stderr:2024-09-11T09:50:38.189+0000 7fd20b170640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31672) UID: 0 2024-09-11T09:50:38.190 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-11T09:50:38.289 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:50:38.290+0000 7f3c96ec9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31909) UID: 0 2024-09-11T09:50:38.334 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-11T09:50:38.335 INFO:teuthology.orchestra.run.smithi096.stdout: "ops": [], 2024-09-11T09:50:38.335 INFO:teuthology.orchestra.run.smithi096.stdout: "complaint_time": 30, 2024-09-11T09:50:38.335 INFO:teuthology.orchestra.run.smithi096.stdout: "num_blocked_ops": 0 2024-09-11T09:50:38.335 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-11T09:50:38.343 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-11T09:50:38.390 INFO:tasks.ceph.osd.0.smithi017.stderr:2024-09-11T09:50:38.391+0000 7fb73c0f1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31665) UID: 0 2024-09-11T09:50:38.488 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-11T09:50:38.489 INFO:teuthology.orchestra.run.smithi096.stdout: "size": 20, 2024-09-11T09:50:38.489 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 600, 2024-09-11T09:50:38.489 INFO:teuthology.orchestra.run.smithi096.stdout: "ops": [ 2024-09-11T09:50:38.489 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.489 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:74 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:50:38.489 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.539067+0000", 2024-09-11T09:50:38.489 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 26.949004731999999, 2024-09-11T09:50:38.489 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.006945957, 2024-09-11T09:50:38.489 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:50:38.489 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:50:38.489 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:38.490 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:50:38.490 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:50:38.490 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:50:38.490 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 74 2024-09-11T09:50:38.490 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.490 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:50:38.490 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.490 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:50:38.490 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.539067+0000", 2024-09-11T09:50:38.490 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:38.490 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.490 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.490 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:50:38.491 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.539067+0000", 2024-09-11T09:50:38.491 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:38.491 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.491 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.491 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:50:38.491 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.539068+0000", 2024-09-11T09:50:38.491 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.6000000000000003e-07 2024-09-11T09:50:38.491 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.491 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.491 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:50:38.491 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.539072+0000", 2024-09-11T09:50:38.491 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.0450000000000002e-06 2024-09-11T09:50:38.492 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.492 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.492 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:50:38.492 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.539078+0000", 2024-09-11T09:50:38.492 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.6889999999999998e-06 2024-09-11T09:50:38.492 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.492 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.492 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:50:38.492 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.539083+0000", 2024-09-11T09:50:38.492 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.7180000000000004e-06 2024-09-11T09:50:38.492 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.492 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.492 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:50:38.493 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.540230+0000", 2024-09-11T09:50:38.493 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0011466250000000001 2024-09-11T09:50:38.493 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.493 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.493 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:50:38.493 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.540279+0000", 2024-09-11T09:50:38.493 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.9667999999999999e-05 2024-09-11T09:50:38.493 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.493 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.493 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:50:38.493 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.540357+0000", 2024-09-11T09:50:38.493 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.7863999999999999e-05 2024-09-11T09:50:38.493 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.494 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.494 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:50:38.494 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.541086+0000", 2024-09-11T09:50:38.494 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00072921100000000001 2024-09-11T09:50:38.494 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.494 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.494 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:50:38.494 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.545949+0000", 2024-09-11T09:50:38.494 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0048626040000000004 2024-09-11T09:50:38.494 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.494 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.494 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:50:38.494 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.545989+0000", 2024-09-11T09:50:38.495 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.0259000000000002e-05 2024-09-11T09:50:38.495 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.495 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.495 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:50:38.495 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.546013+0000", 2024-09-11T09:50:38.495 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.3614000000000001e-05 2024-09-11T09:50:38.495 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:38.495 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:50:38.495 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:38.495 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.495 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.495 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:108 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:50:38.495 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.711510+0000", 2024-09-11T09:50:38.496 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 26.776561560000001, 2024-09-11T09:50:38.496 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0054615649999999998, 2024-09-11T09:50:38.496 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:50:38.496 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:50:38.496 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:38.496 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:50:38.496 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:50:38.496 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:50:38.496 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 108 2024-09-11T09:50:38.496 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.496 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:50:38.496 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.496 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:50:38.497 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711510+0000", 2024-09-11T09:50:38.497 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:38.497 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.497 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.497 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:50:38.497 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711510+0000", 2024-09-11T09:50:38.497 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:38.497 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.497 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.497 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:50:38.497 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711511+0000", 2024-09-11T09:50:38.497 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.1999999999999998e-07 2024-09-11T09:50:38.497 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.498 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.498 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:50:38.498 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711516+0000", 2024-09-11T09:50:38.498 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.4140000000000002e-06 2024-09-11T09:50:38.498 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.498 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.498 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:50:38.498 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711527+0000", 2024-09-11T09:50:38.498 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.0169e-05 2024-09-11T09:50:38.498 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.498 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.498 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:50:38.499 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711535+0000", 2024-09-11T09:50:38.499 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.3299999999999999e-06 2024-09-11T09:50:38.499 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.499 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.499 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:50:38.499 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.713763+0000", 2024-09-11T09:50:38.499 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0022281079999999999 2024-09-11T09:50:38.499 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.499 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.499 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:50:38.499 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.713812+0000", 2024-09-11T09:50:38.499 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.8544000000000001e-05 2024-09-11T09:50:38.499 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.500 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.500 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:50:38.500 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.713909+0000", 2024-09-11T09:50:38.500 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.7113000000000003e-05 2024-09-11T09:50:38.500 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.500 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.500 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:50:38.500 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.716873+0000", 2024-09-11T09:50:38.500 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0029642710000000001 2024-09-11T09:50:38.500 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.500 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.500 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:50:38.500 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.716918+0000", 2024-09-11T09:50:38.501 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.5447e-05 2024-09-11T09:50:38.501 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.501 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.501 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:50:38.501 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.716947+0000", 2024-09-11T09:50:38.501 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.8181e-05 2024-09-11T09:50:38.501 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.501 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.501 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:50:38.501 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.716972+0000", 2024-09-11T09:50:38.501 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.5068000000000001e-05 2024-09-11T09:50:38.501 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:38.501 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:50:38.502 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:38.502 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.502 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.502 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:109 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:50:38.502 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.711555+0000", 2024-09-11T09:50:38.502 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 26.776516311000002, 2024-09-11T09:50:38.502 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.005479902, 2024-09-11T09:50:38.502 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:50:38.502 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:50:38.502 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:38.502 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:50:38.502 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:50:38.502 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:50:38.503 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 109 2024-09-11T09:50:38.503 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.503 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:50:38.503 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.503 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:50:38.503 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711555+0000", 2024-09-11T09:50:38.503 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:38.503 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.503 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.503 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:50:38.503 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711555+0000", 2024-09-11T09:50:38.503 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:38.504 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.504 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.504 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:50:38.504 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711556+0000", 2024-09-11T09:50:38.504 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.3200000000000005e-07 2024-09-11T09:50:38.504 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.504 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.504 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:50:38.504 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711601+0000", 2024-09-11T09:50:38.504 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.5349000000000002e-05 2024-09-11T09:50:38.504 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.504 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.505 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:50:38.505 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711612+0000", 2024-09-11T09:50:38.505 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.0701e-05 2024-09-11T09:50:38.505 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.505 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.505 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:50:38.505 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711619+0000", 2024-09-11T09:50:38.505 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.7610000000000002e-06 2024-09-11T09:50:38.505 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.505 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.505 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:50:38.505 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.714371+0000", 2024-09-11T09:50:38.505 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0027526260000000002 2024-09-11T09:50:38.506 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.506 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.506 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:50:38.506 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.714417+0000", 2024-09-11T09:50:38.506 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.5893999999999998e-05 2024-09-11T09:50:38.506 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.506 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.506 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:50:38.506 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.714496+0000", 2024-09-11T09:50:38.506 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.8539000000000001e-05 2024-09-11T09:50:38.506 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.506 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.506 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:50:38.507 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.716910+0000", 2024-09-11T09:50:38.507 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0024137640000000001 2024-09-11T09:50:38.507 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.507 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.507 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:50:38.507 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.716989+0000", 2024-09-11T09:50:38.507 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.9198999999999998e-05 2024-09-11T09:50:38.507 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.507 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.507 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:50:38.507 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.717016+0000", 2024-09-11T09:50:38.507 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.6699e-05 2024-09-11T09:50:38.507 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.508 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.508 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:50:38.508 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.717035+0000", 2024-09-11T09:50:38.508 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.9738e-05 2024-09-11T09:50:38.508 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:38.508 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:50:38.508 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:38.508 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.508 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.508 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:110 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:50:38.508 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.711636+0000", 2024-09-11T09:50:38.508 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 26.776435850999999, 2024-09-11T09:50:38.508 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0067613070000000003, 2024-09-11T09:50:38.509 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:50:38.509 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:50:38.509 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:38.509 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:50:38.509 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:50:38.509 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:50:38.509 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 110 2024-09-11T09:50:38.509 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.509 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:50:38.509 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.509 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:50:38.509 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711636+0000", 2024-09-11T09:50:38.509 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:38.510 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.510 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.510 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:50:38.510 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711636+0000", 2024-09-11T09:50:38.510 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:38.510 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.510 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.510 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:50:38.510 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711637+0000", 2024-09-11T09:50:38.510 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.9999999999999997e-07 2024-09-11T09:50:38.510 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.510 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.510 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:50:38.511 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711641+0000", 2024-09-11T09:50:38.511 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.2570000000000001e-06 2024-09-11T09:50:38.511 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.511 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.511 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:50:38.511 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711651+0000", 2024-09-11T09:50:38.511 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.0285999999999999e-05 2024-09-11T09:50:38.511 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.511 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.511 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:50:38.511 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711658+0000", 2024-09-11T09:50:38.511 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.3540000000000002e-06 2024-09-11T09:50:38.511 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.512 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.512 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:50:38.512 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.716475+0000", 2024-09-11T09:50:38.512 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0048162819999999999 2024-09-11T09:50:38.512 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.512 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.512 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:50:38.512 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.716526+0000", 2024-09-11T09:50:38.512 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.0905000000000001e-05 2024-09-11T09:50:38.512 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.512 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.512 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:50:38.512 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.716601+0000", 2024-09-11T09:50:38.513 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.5283000000000004e-05 2024-09-11T09:50:38.513 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.513 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.513 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:50:38.513 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.717933+0000", 2024-09-11T09:50:38.513 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001331622 2024-09-11T09:50:38.513 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.513 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.513 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:50:38.513 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718332+0000", 2024-09-11T09:50:38.513 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00039909899999999998 2024-09-11T09:50:38.513 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.514 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.514 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:50:38.514 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718363+0000", 2024-09-11T09:50:38.514 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.1636e-05 2024-09-11T09:50:38.514 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.514 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.514 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:50:38.514 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718397+0000", 2024-09-11T09:50:38.514 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.3883000000000001e-05 2024-09-11T09:50:38.514 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:38.514 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:50:38.514 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:38.514 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.514 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.515 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:111 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:50:38.515 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.711668+0000", 2024-09-11T09:50:38.515 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 26.776403421000001, 2024-09-11T09:50:38.515 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0077145570000000004, 2024-09-11T09:50:38.515 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:50:38.515 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:50:38.515 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:38.515 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:50:38.515 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:50:38.515 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:50:38.515 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 111 2024-09-11T09:50:38.515 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.515 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:50:38.516 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.516 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:50:38.516 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711668+0000", 2024-09-11T09:50:38.516 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:38.516 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.516 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.516 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:50:38.516 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711668+0000", 2024-09-11T09:50:38.516 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:38.516 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.516 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.517 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:50:38.517 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711669+0000", 2024-09-11T09:50:38.517 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.99e-07 2024-09-11T09:50:38.517 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.517 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.517 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:50:38.517 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711673+0000", 2024-09-11T09:50:38.517 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.7440000000000001e-06 2024-09-11T09:50:38.517 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.517 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.517 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:50:38.517 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711686+0000", 2024-09-11T09:50:38.518 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.3317e-05 2024-09-11T09:50:38.518 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.518 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.518 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:50:38.518 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711697+0000", 2024-09-11T09:50:38.518 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1022000000000001e-05 2024-09-11T09:50:38.518 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.518 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.518 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:50:38.518 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.717065+0000", 2024-09-11T09:50:38.518 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.005367531 2024-09-11T09:50:38.518 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.518 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.518 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:50:38.519 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.717114+0000", 2024-09-11T09:50:38.519 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.9707999999999999e-05 2024-09-11T09:50:38.519 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.519 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.519 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:50:38.519 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.717190+0000", 2024-09-11T09:50:38.519 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.5394000000000004e-05 2024-09-11T09:50:38.519 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.519 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.519 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:50:38.519 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718793+0000", 2024-09-11T09:50:38.519 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001603653 2024-09-11T09:50:38.520 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.520 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.520 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:50:38.520 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719329+0000", 2024-09-11T09:50:38.520 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00053543799999999995 2024-09-11T09:50:38.520 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.520 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.520 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:50:38.521 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719359+0000", 2024-09-11T09:50:38.521 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.0420999999999999e-05 2024-09-11T09:50:38.521 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.521 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.521 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:50:38.521 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719383+0000", 2024-09-11T09:50:38.521 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.3730000000000001e-05 2024-09-11T09:50:38.521 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:38.521 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:50:38.521 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:38.521 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.522 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.522 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:112 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:50:38.522 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.711712+0000", 2024-09-11T09:50:38.522 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 26.776359861, 2024-09-11T09:50:38.522 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.008162084, 2024-09-11T09:50:38.522 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:50:38.522 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:50:38.522 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:38.522 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:50:38.522 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:50:38.522 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:50:38.523 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 112 2024-09-11T09:50:38.523 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.523 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:50:38.523 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.523 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:50:38.523 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711712+0000", 2024-09-11T09:50:38.523 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:38.523 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.523 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.523 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:50:38.523 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711712+0000", 2024-09-11T09:50:38.523 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:38.523 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.523 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.524 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:50:38.524 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711712+0000", 2024-09-11T09:50:38.524 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.6000000000000004e-07 2024-09-11T09:50:38.524 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.524 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.524 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:50:38.524 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711808+0000", 2024-09-11T09:50:38.524 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.5977999999999995e-05 2024-09-11T09:50:38.524 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.524 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.524 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:50:38.524 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711819+0000", 2024-09-11T09:50:38.524 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.0755e-05 2024-09-11T09:50:38.525 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.525 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.525 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:50:38.525 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711826+0000", 2024-09-11T09:50:38.525 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.7970000000000003e-06 2024-09-11T09:50:38.525 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.525 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.525 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:50:38.525 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.717435+0000", 2024-09-11T09:50:38.525 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0056090239999999998 2024-09-11T09:50:38.525 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.525 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.526 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:50:38.526 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.717496+0000", 2024-09-11T09:50:38.526 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.0543e-05 2024-09-11T09:50:38.526 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.526 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.526 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:50:38.526 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.717591+0000", 2024-09-11T09:50:38.526 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.5667000000000004e-05 2024-09-11T09:50:38.526 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.526 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.526 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:50:38.526 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718831+0000", 2024-09-11T09:50:38.526 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001240204 2024-09-11T09:50:38.527 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.527 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.527 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:50:38.527 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719819+0000", 2024-09-11T09:50:38.527 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00098751500000000001 2024-09-11T09:50:38.527 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.527 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.527 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:50:38.527 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719850+0000", 2024-09-11T09:50:38.527 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.1535999999999998e-05 2024-09-11T09:50:38.527 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.527 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.527 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:50:38.528 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719874+0000", 2024-09-11T09:50:38.528 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.3504999999999999e-05 2024-09-11T09:50:38.528 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:38.528 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:50:38.528 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:38.528 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.528 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.528 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:113 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:50:38.528 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.711842+0000", 2024-09-11T09:50:38.528 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 26.776229361999999, 2024-09-11T09:50:38.528 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0081160099999999999, 2024-09-11T09:50:38.528 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:50:38.528 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:50:38.529 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:38.529 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:50:38.529 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:50:38.529 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:50:38.529 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 113 2024-09-11T09:50:38.529 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.529 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:50:38.529 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.529 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:50:38.529 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711842+0000", 2024-09-11T09:50:38.529 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:38.529 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.529 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.530 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:50:38.530 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711842+0000", 2024-09-11T09:50:38.530 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:38.530 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.530 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.530 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:50:38.530 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711843+0000", 2024-09-11T09:50:38.530 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.4399999999999999e-07 2024-09-11T09:50:38.530 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.530 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.530 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:50:38.530 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711848+0000", 2024-09-11T09:50:38.530 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.4159999999999997e-06 2024-09-11T09:50:38.531 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.531 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.531 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:50:38.531 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711856+0000", 2024-09-11T09:50:38.531 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.7870000000000007e-06 2024-09-11T09:50:38.531 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.531 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.531 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:50:38.531 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711862+0000", 2024-09-11T09:50:38.531 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.3689999999999998e-06 2024-09-11T09:50:38.531 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.531 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.531 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:50:38.532 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.717965+0000", 2024-09-11T09:50:38.532 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0061028899999999997 2024-09-11T09:50:38.532 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.532 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.532 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:50:38.532 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718023+0000", 2024-09-11T09:50:38.532 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.8313000000000002e-05 2024-09-11T09:50:38.532 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.532 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.532 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:50:38.532 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718109+0000", 2024-09-11T09:50:38.532 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.6205000000000006e-05 2024-09-11T09:50:38.532 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.533 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.533 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:50:38.533 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719808+0000", 2024-09-11T09:50:38.533 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0016990060000000001 2024-09-11T09:50:38.533 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.533 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.533 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:50:38.533 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719883+0000", 2024-09-11T09:50:38.533 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.4624999999999998e-05 2024-09-11T09:50:38.533 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.533 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.533 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:50:38.533 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719910+0000", 2024-09-11T09:50:38.534 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.7512e-05 2024-09-11T09:50:38.534 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.534 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.534 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:50:38.534 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719958+0000", 2024-09-11T09:50:38.534 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.8143000000000003e-05 2024-09-11T09:50:38.534 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:38.534 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:50:38.534 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:38.534 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.534 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.534 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:50:38.534 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.711887+0000", 2024-09-11T09:50:38.535 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 26.776184747999999, 2024-09-11T09:50:38.535 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0086063589999999992, 2024-09-11T09:50:38.535 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:50:38.535 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:50:38.535 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:38.535 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:50:38.535 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:50:38.535 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:50:38.535 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 114 2024-09-11T09:50:38.535 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.535 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:50:38.535 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.535 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:50:38.536 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711887+0000", 2024-09-11T09:50:38.536 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:38.536 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.536 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.536 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:50:38.536 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711887+0000", 2024-09-11T09:50:38.536 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:38.536 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.536 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.536 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:50:38.536 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711887+0000", 2024-09-11T09:50:38.536 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.1500000000000005e-07 2024-09-11T09:50:38.536 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.537 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.537 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:50:38.537 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711897+0000", 2024-09-11T09:50:38.537 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.1900000000000001e-06 2024-09-11T09:50:38.537 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.537 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.537 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:50:38.537 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711902+0000", 2024-09-11T09:50:38.537 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.2349999999999999e-06 2024-09-11T09:50:38.537 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.537 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.537 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:50:38.537 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711906+0000", 2024-09-11T09:50:38.538 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.7129999999999999e-06 2024-09-11T09:50:38.538 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.538 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.538 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:50:38.538 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718429+0000", 2024-09-11T09:50:38.538 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0065231960000000002 2024-09-11T09:50:38.538 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.538 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.538 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:50:38.538 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718480+0000", 2024-09-11T09:50:38.538 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.1161e-05 2024-09-11T09:50:38.538 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.538 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.538 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:50:38.539 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718558+0000", 2024-09-11T09:50:38.539 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.7519000000000001e-05 2024-09-11T09:50:38.539 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.539 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.539 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:50:38.539 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720038+0000", 2024-09-11T09:50:38.539 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001480016 2024-09-11T09:50:38.539 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.539 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.539 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:50:38.539 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720417+0000", 2024-09-11T09:50:38.539 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00037943699999999999 2024-09-11T09:50:38.539 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.539 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.540 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:50:38.540 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720452+0000", 2024-09-11T09:50:38.540 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.4595000000000003e-05 2024-09-11T09:50:38.540 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.540 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.540 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:50:38.540 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720493+0000", 2024-09-11T09:50:38.540 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.1782000000000001e-05 2024-09-11T09:50:38.540 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:38.540 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:50:38.540 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:38.540 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.540 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.541 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:50:38.541 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.711912+0000", 2024-09-11T09:50:38.541 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 26.776160079, 2024-09-11T09:50:38.541 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0086864750000000008, 2024-09-11T09:50:38.541 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:50:38.541 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:50:38.541 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:38.541 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:50:38.541 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:50:38.541 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:50:38.541 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 115 2024-09-11T09:50:38.541 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.541 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:50:38.542 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.542 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:50:38.542 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711912+0000", 2024-09-11T09:50:38.542 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:38.542 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.542 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.542 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:50:38.542 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711912+0000", 2024-09-11T09:50:38.542 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:38.542 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.542 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.542 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:50:38.542 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711912+0000", 2024-09-11T09:50:38.543 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.3900000000000001e-07 2024-09-11T09:50:38.543 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.543 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.543 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:50:38.543 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711969+0000", 2024-09-11T09:50:38.543 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.7501999999999999e-05 2024-09-11T09:50:38.543 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.543 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.543 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:50:38.543 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711986+0000", 2024-09-11T09:50:38.543 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.6402999999999999e-05 2024-09-11T09:50:38.543 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.543 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.544 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:50:38.544 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711990+0000", 2024-09-11T09:50:38.544 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.929e-06 2024-09-11T09:50:38.544 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.544 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.544 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:50:38.544 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718861+0000", 2024-09-11T09:50:38.544 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0068710530000000002 2024-09-11T09:50:38.544 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.544 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.544 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:50:38.544 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718908+0000", 2024-09-11T09:50:38.545 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.7431e-05 2024-09-11T09:50:38.545 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.545 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.545 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:50:38.545 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718984+0000", 2024-09-11T09:50:38.545 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.5500999999999994e-05 2024-09-11T09:50:38.545 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.545 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.545 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:50:38.545 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720500+0000", 2024-09-11T09:50:38.545 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0015158820000000001 2024-09-11T09:50:38.545 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.546 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.546 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:50:38.546 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720536+0000", 2024-09-11T09:50:38.546 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.6118000000000002e-05 2024-09-11T09:50:38.546 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.546 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.546 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:50:38.546 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720570+0000", 2024-09-11T09:50:38.546 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.4387999999999997e-05 2024-09-11T09:50:38.546 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.547 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.547 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:50:38.547 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720598+0000", 2024-09-11T09:50:38.547 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.8028999999999999e-05 2024-09-11T09:50:38.547 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:38.547 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:50:38.547 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:38.547 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.547 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.547 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:50:38.547 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.711998+0000", 2024-09-11T09:50:38.547 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 26.7760739, 2024-09-11T09:50:38.547 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.010785813999999999, 2024-09-11T09:50:38.548 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:50:38.548 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:50:38.548 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:38.548 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:50:38.548 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:50:38.548 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:50:38.548 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 116 2024-09-11T09:50:38.548 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.548 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:50:38.548 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.548 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:50:38.548 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711998+0000", 2024-09-11T09:50:38.548 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:38.549 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.549 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.549 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:50:38.549 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711998+0000", 2024-09-11T09:50:38.549 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:38.549 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.549 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.549 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:50:38.549 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711998+0000", 2024-09-11T09:50:38.549 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.3300000000000003e-07 2024-09-11T09:50:38.549 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.549 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.549 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:50:38.550 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712001+0000", 2024-09-11T09:50:38.550 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.9349999999999999e-06 2024-09-11T09:50:38.550 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.550 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.550 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:50:38.550 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712006+0000", 2024-09-11T09:50:38.550 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.1630000000000004e-06 2024-09-11T09:50:38.550 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.550 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.550 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:50:38.550 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712010+0000", 2024-09-11T09:50:38.550 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.574e-06 2024-09-11T09:50:38.550 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.551 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.551 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:50:38.551 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719415+0000", 2024-09-11T09:50:38.551 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0074049099999999998 2024-09-11T09:50:38.551 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.551 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.551 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:50:38.551 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719465+0000", 2024-09-11T09:50:38.551 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.0129000000000001e-05 2024-09-11T09:50:38.551 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.551 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.551 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:50:38.551 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719545+0000", 2024-09-11T09:50:38.552 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.0253999999999994e-05 2024-09-11T09:50:38.552 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.552 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.552 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:50:38.552 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720636+0000", 2024-09-11T09:50:38.552 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0010909330000000001 2024-09-11T09:50:38.552 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.552 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.552 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:50:38.552 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.722707+0000", 2024-09-11T09:50:38.552 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0020713300000000001 2024-09-11T09:50:38.552 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.553 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.553 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:50:38.553 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.722743+0000", 2024-09-11T09:50:38.553 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.5351000000000001e-05 2024-09-11T09:50:38.553 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.553 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.553 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:50:38.553 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.722784+0000", 2024-09-11T09:50:38.553 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.0802e-05 2024-09-11T09:50:38.553 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:38.553 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:50:38.553 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:38.553 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.554 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.554 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:50:38.554 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.712014+0000", 2024-09-11T09:50:38.554 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 26.776057860000002, 2024-09-11T09:50:38.554 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.013420938, 2024-09-11T09:50:38.554 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:50:38.554 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:50:38.554 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:38.554 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:50:38.554 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:50:38.554 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:50:38.554 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 117 2024-09-11T09:50:38.554 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.555 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:50:38.555 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.555 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:50:38.555 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712014+0000", 2024-09-11T09:50:38.555 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:38.555 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.555 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.555 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:50:38.555 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712014+0000", 2024-09-11T09:50:38.555 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:38.555 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.555 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.555 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:50:38.555 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712014+0000", 2024-09-11T09:50:38.556 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.2000000000000001e-07 2024-09-11T09:50:38.556 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.556 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.556 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:50:38.556 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712016+0000", 2024-09-11T09:50:38.556 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.2309999999999999e-06 2024-09-11T09:50:38.556 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.556 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.556 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:50:38.556 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712020+0000", 2024-09-11T09:50:38.556 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.3830000000000001e-06 2024-09-11T09:50:38.556 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.556 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.557 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:50:38.557 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712023+0000", 2024-09-11T09:50:38.557 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.3340000000000002e-06 2024-09-11T09:50:38.557 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.557 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.557 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:50:38.557 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720083+0000", 2024-09-11T09:50:38.557 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0080601189999999993 2024-09-11T09:50:38.557 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.557 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.557 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:50:38.557 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720125+0000", 2024-09-11T09:50:38.557 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.2029e-05 2024-09-11T09:50:38.558 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.558 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.558 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:50:38.558 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720205+0000", 2024-09-11T09:50:38.558 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.0010000000000001e-05 2024-09-11T09:50:38.558 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.558 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.558 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:50:38.558 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720849+0000", 2024-09-11T09:50:38.558 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00064369899999999996 2024-09-11T09:50:38.558 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.558 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.558 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:50:38.558 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.725357+0000", 2024-09-11T09:50:38.559 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0045077379999999998 2024-09-11T09:50:38.559 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.559 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.559 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:50:38.559 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.725392+0000", 2024-09-11T09:50:38.559 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.5862999999999999e-05 2024-09-11T09:50:38.559 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.559 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.559 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:50:38.560 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.725435+0000", 2024-09-11T09:50:38.560 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.2311999999999997e-05 2024-09-11T09:50:38.560 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:38.560 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:50:38.560 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:38.560 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.560 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.560 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:119 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 131072~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:50:38.560 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.725673+0000", 2024-09-11T09:50:38.561 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 26.762399158000001, 2024-09-11T09:50:38.561 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0053927460000000003, 2024-09-11T09:50:38.561 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:50:38.561 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:50:38.561 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:38.561 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:50:38.561 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:50:38.561 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:50:38.561 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 119 2024-09-11T09:50:38.561 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.561 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:50:38.561 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.561 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:50:38.562 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.725673+0000", 2024-09-11T09:50:38.562 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:38.562 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.562 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.562 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:50:38.562 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.725673+0000", 2024-09-11T09:50:38.562 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:38.562 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.562 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.562 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:50:38.562 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.725673+0000", 2024-09-11T09:50:38.562 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.68e-07 2024-09-11T09:50:38.563 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.563 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.563 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:50:38.563 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.725720+0000", 2024-09-11T09:50:38.563 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.6526999999999997e-05 2024-09-11T09:50:38.563 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.563 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.563 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:50:38.563 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.725734+0000", 2024-09-11T09:50:38.563 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.4593000000000001e-05 2024-09-11T09:50:38.563 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.563 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.563 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:50:38.563 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.725741+0000", 2024-09-11T09:50:38.563 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.28e-06 2024-09-11T09:50:38.564 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.564 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.564 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:50:38.564 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.726290+0000", 2024-09-11T09:50:38.564 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00054892000000000005 2024-09-11T09:50:38.564 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.564 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.564 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:50:38.564 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.726338+0000", 2024-09-11T09:50:38.564 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.7982000000000002e-05 2024-09-11T09:50:38.564 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.564 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.565 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:50:38.565 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.726413+0000", 2024-09-11T09:50:38.565 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.5679000000000005e-05 2024-09-11T09:50:38.565 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.565 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.565 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:50:38.565 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.727555+0000", 2024-09-11T09:50:38.565 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0011420289999999999 2024-09-11T09:50:38.565 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.565 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.565 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:50:38.565 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.731004+0000", 2024-09-11T09:50:38.565 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0034484279999999999 2024-09-11T09:50:38.566 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.566 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.566 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:50:38.566 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.731040+0000", 2024-09-11T09:50:38.566 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.6315e-05 2024-09-11T09:50:38.566 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.566 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.566 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:50:38.566 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.731065+0000", 2024-09-11T09:50:38.566 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.5324999999999999e-05 2024-09-11T09:50:38.566 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:38.566 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:50:38.566 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:38.566 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.567 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.567 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:120 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 196608~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:50:38.567 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.725756+0000", 2024-09-11T09:50:38.567 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 26.762315249, 2024-09-11T09:50:38.567 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0053569100000000003, 2024-09-11T09:50:38.567 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:50:38.567 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:50:38.567 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:38.567 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:50:38.567 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:50:38.567 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:50:38.567 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 120 2024-09-11T09:50:38.568 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.568 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:50:38.568 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.568 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:50:38.568 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.725756+0000", 2024-09-11T09:50:38.568 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:38.568 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.568 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.568 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:50:38.568 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.725756+0000", 2024-09-11T09:50:38.568 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:38.568 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.568 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.569 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:50:38.569 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.725757+0000", 2024-09-11T09:50:38.569 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.3799999999999997e-07 2024-09-11T09:50:38.569 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.569 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.569 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:50:38.569 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.725802+0000", 2024-09-11T09:50:38.569 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.4790000000000003e-05 2024-09-11T09:50:38.569 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.569 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.569 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:50:38.569 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.725815+0000", 2024-09-11T09:50:38.569 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.2609000000000001e-05 2024-09-11T09:50:38.570 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.570 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.570 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:50:38.570 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.725822+0000", 2024-09-11T09:50:38.570 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.6450000000000005e-06 2024-09-11T09:50:38.570 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.570 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.570 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:50:38.570 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.726708+0000", 2024-09-11T09:50:38.570 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00088606500000000001 2024-09-11T09:50:38.570 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.570 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.570 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:50:38.571 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.726755+0000", 2024-09-11T09:50:38.571 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.6888000000000002e-05 2024-09-11T09:50:38.571 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.571 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.571 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:50:38.571 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.726828+0000", 2024-09-11T09:50:38.571 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.2510999999999997e-05 2024-09-11T09:50:38.571 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.571 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.571 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:50:38.571 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.727829+0000", 2024-09-11T09:50:38.571 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001001093 2024-09-11T09:50:38.571 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.572 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.572 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:50:38.572 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.731071+0000", 2024-09-11T09:50:38.572 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0032426009999999999 2024-09-11T09:50:38.572 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.572 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.572 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:50:38.572 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.731094+0000", 2024-09-11T09:50:38.572 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.2677e-05 2024-09-11T09:50:38.572 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.572 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.572 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:50:38.573 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.731113+0000", 2024-09-11T09:50:38.573 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.9392999999999999e-05 2024-09-11T09:50:38.573 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:38.573 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:50:38.573 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:38.573 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.573 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.573 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:138 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:50:38.573 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:12.034038+0000", 2024-09-11T09:50:38.573 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 26.454033919, 2024-09-11T09:50:38.573 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0054142749999999996, 2024-09-11T09:50:38.573 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:50:38.573 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:50:38.574 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:38.574 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:50:38.574 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:50:38.574 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:50:38.574 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 138 2024-09-11T09:50:38.574 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.574 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:50:38.574 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.574 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:50:38.574 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034038+0000", 2024-09-11T09:50:38.574 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:38.574 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.574 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.575 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:50:38.575 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034038+0000", 2024-09-11T09:50:38.575 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:38.575 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.575 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.575 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:50:38.575 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034038+0000", 2024-09-11T09:50:38.575 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.3499999999999996e-07 2024-09-11T09:50:38.575 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.575 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.575 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:50:38.575 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034043+0000", 2024-09-11T09:50:38.575 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.7310000000000001e-06 2024-09-11T09:50:38.575 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.576 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.576 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:50:38.576 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034050+0000", 2024-09-11T09:50:38.576 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.0419999999999997e-06 2024-09-11T09:50:38.576 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.576 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.576 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:50:38.576 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034055+0000", 2024-09-11T09:50:38.576 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.3299999999999998e-06 2024-09-11T09:50:38.576 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.576 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.576 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:50:38.576 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.037309+0000", 2024-09-11T09:50:38.577 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0032535440000000001 2024-09-11T09:50:38.577 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.577 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.577 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:50:38.577 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.037357+0000", 2024-09-11T09:50:38.577 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.8167999999999997e-05 2024-09-11T09:50:38.577 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.577 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.577 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:50:38.577 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.037430+0000", 2024-09-11T09:50:38.577 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.3275999999999994e-05 2024-09-11T09:50:38.577 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.578 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.578 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:50:38.578 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.039014+0000", 2024-09-11T09:50:38.578 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0015831350000000001 2024-09-11T09:50:38.578 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.578 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.578 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:50:38.578 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.039401+0000", 2024-09-11T09:50:38.578 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00038758500000000002 2024-09-11T09:50:38.578 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.578 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.578 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:50:38.578 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.039429+0000", 2024-09-11T09:50:38.578 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.7379e-05 2024-09-11T09:50:38.579 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.579 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.579 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:50:38.579 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.039452+0000", 2024-09-11T09:50:38.579 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.355e-05 2024-09-11T09:50:38.579 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:38.579 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:50:38.579 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:38.579 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.579 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.579 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:139 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:50:38.579 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:12.034064+0000", 2024-09-11T09:50:38.580 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 26.454007888, 2024-09-11T09:50:38.580 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0054736200000000002, 2024-09-11T09:50:38.580 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:50:38.580 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:50:38.580 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:38.580 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:50:38.580 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:50:38.580 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:50:38.581 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 139 2024-09-11T09:50:38.581 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.581 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:50:38.581 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.581 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:50:38.581 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034064+0000", 2024-09-11T09:50:38.581 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:38.581 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.581 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.581 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:50:38.581 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034064+0000", 2024-09-11T09:50:38.581 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:38.581 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.582 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.582 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:50:38.582 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034064+0000", 2024-09-11T09:50:38.582 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.89e-07 2024-09-11T09:50:38.582 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.582 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.582 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:50:38.582 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034069+0000", 2024-09-11T09:50:38.582 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.9799999999999998e-06 2024-09-11T09:50:38.582 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.583 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.583 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:50:38.583 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034079+0000", 2024-09-11T09:50:38.583 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.3549999999999994e-06 2024-09-11T09:50:38.583 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.583 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.583 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:50:38.583 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034084+0000", 2024-09-11T09:50:38.583 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.2209999999999996e-06 2024-09-11T09:50:38.583 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.583 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.583 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:50:38.584 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.038176+0000", 2024-09-11T09:50:38.584 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0040921610000000004 2024-09-11T09:50:38.584 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.584 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.584 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:50:38.584 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.038221+0000", 2024-09-11T09:50:38.584 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.5006000000000002e-05 2024-09-11T09:50:38.584 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.584 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.584 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:50:38.584 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.038293+0000", 2024-09-11T09:50:38.584 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.1952999999999994e-05 2024-09-11T09:50:38.584 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.585 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.585 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:50:38.585 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.039461+0000", 2024-09-11T09:50:38.585 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0011680340000000001 2024-09-11T09:50:38.585 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.585 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.585 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:50:38.585 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.039487+0000", 2024-09-11T09:50:38.585 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.5752e-05 2024-09-11T09:50:38.585 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.585 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.585 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:50:38.585 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.039515+0000", 2024-09-11T09:50:38.586 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.7763000000000002e-05 2024-09-11T09:50:38.586 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.586 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.586 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:50:38.586 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.039537+0000", 2024-09-11T09:50:38.586 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.2906000000000001e-05 2024-09-11T09:50:38.586 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:38.586 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:50:38.586 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:38.586 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.586 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.586 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:140 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:50:38.586 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:12.034093+0000", 2024-09-11T09:50:38.586 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 26.453978383999999, 2024-09-11T09:50:38.587 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0059112490000000004, 2024-09-11T09:50:38.587 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:50:38.587 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:50:38.587 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:38.587 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:50:38.587 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:50:38.587 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:50:38.587 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 140 2024-09-11T09:50:38.587 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.587 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:50:38.587 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.587 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:50:38.588 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034093+0000", 2024-09-11T09:50:38.588 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:38.588 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.588 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.588 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:50:38.588 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034093+0000", 2024-09-11T09:50:38.588 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:38.588 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.588 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.588 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:50:38.588 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034094+0000", 2024-09-11T09:50:38.588 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.4299999999999998e-07 2024-09-11T09:50:38.588 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.589 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.589 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:50:38.589 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034159+0000", 2024-09-11T09:50:38.589 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.567e-05 2024-09-11T09:50:38.589 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.589 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.589 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:50:38.589 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034167+0000", 2024-09-11T09:50:38.589 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.8490000000000006e-06 2024-09-11T09:50:38.589 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.589 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.589 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:50:38.589 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034173+0000", 2024-09-11T09:50:38.590 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.1770000000000003e-06 2024-09-11T09:50:38.590 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.590 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.590 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:50:38.590 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.038543+0000", 2024-09-11T09:50:38.590 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0043699150000000003 2024-09-11T09:50:38.590 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.590 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.590 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:50:38.591 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.038589+0000", 2024-09-11T09:50:38.591 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.5562999999999997e-05 2024-09-11T09:50:38.591 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.591 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.591 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:50:38.591 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.038662+0000", 2024-09-11T09:50:38.591 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.2748000000000002e-05 2024-09-11T09:50:38.591 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.591 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.591 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:50:38.591 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.039635+0000", 2024-09-11T09:50:38.591 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00097310799999999998 2024-09-11T09:50:38.591 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.592 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.592 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:50:38.592 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.039945+0000", 2024-09-11T09:50:38.592 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00031054100000000001 2024-09-11T09:50:38.592 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.592 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.592 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:50:38.592 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.039971+0000", 2024-09-11T09:50:38.592 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.5423999999999999e-05 2024-09-11T09:50:38.592 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.592 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.593 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:50:38.593 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.040005+0000", 2024-09-11T09:50:38.593 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.3810999999999999e-05 2024-09-11T09:50:38.593 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:38.593 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:50:38.593 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:38.593 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.593 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.593 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:50:38.593 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:12.034184+0000", 2024-09-11T09:50:38.593 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 26.453887686000002, 2024-09-11T09:50:38.593 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0061520380000000003, 2024-09-11T09:50:38.594 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:50:38.594 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:50:38.594 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:38.594 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:50:38.594 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:50:38.594 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:50:38.594 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 141 2024-09-11T09:50:38.594 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.594 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:50:38.595 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.595 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:50:38.595 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034184+0000", 2024-09-11T09:50:38.595 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:38.595 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.595 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.595 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:50:38.595 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034184+0000", 2024-09-11T09:50:38.595 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:38.595 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.595 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.596 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:50:38.596 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034185+0000", 2024-09-11T09:50:38.596 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.1099999999999995e-07 2024-09-11T09:50:38.596 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.596 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.596 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:50:38.596 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034190+0000", 2024-09-11T09:50:38.596 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.7309999999999998e-06 2024-09-11T09:50:38.596 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.596 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.596 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:50:38.596 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034197+0000", 2024-09-11T09:50:38.597 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.7440000000000001e-06 2024-09-11T09:50:38.597 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.597 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.597 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:50:38.597 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034204+0000", 2024-09-11T09:50:38.597 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.8970000000000002e-06 2024-09-11T09:50:38.597 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.597 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.597 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:50:38.597 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.039035+0000", 2024-09-11T09:50:38.597 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0048307039999999999 2024-09-11T09:50:38.597 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.598 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.598 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:50:38.598 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.039093+0000", 2024-09-11T09:50:38.598 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.8604000000000003e-05 2024-09-11T09:50:38.598 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.598 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.598 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:50:38.598 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.039184+0000", 2024-09-11T09:50:38.598 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.0196000000000002e-05 2024-09-11T09:50:38.598 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.598 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.598 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:50:38.598 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.039873+0000", 2024-09-11T09:50:38.598 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00068960600000000005 2024-09-11T09:50:38.599 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.599 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.599 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:50:38.599 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.040292+0000", 2024-09-11T09:50:38.599 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00041865199999999999 2024-09-11T09:50:38.599 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.599 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.599 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:50:38.599 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.040317+0000", 2024-09-11T09:50:38.599 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.5439000000000002e-05 2024-09-11T09:50:38.599 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.599 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.599 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:50:38.600 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.040336+0000", 2024-09-11T09:50:38.600 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.8754e-05 2024-09-11T09:50:38.600 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:38.600 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:50:38.600 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:38.600 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.600 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.600 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:157 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:50:38.600 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:12.452040+0000", 2024-09-11T09:50:38.600 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 26.036031832999999, 2024-09-11T09:50:38.600 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0068848269999999996, 2024-09-11T09:50:38.600 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:50:38.600 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:50:38.601 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:38.601 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:50:38.601 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:50:38.601 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:50:38.601 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 157 2024-09-11T09:50:38.601 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.601 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:50:38.601 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.601 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:50:38.601 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452040+0000", 2024-09-11T09:50:38.601 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:38.601 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.601 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.602 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:50:38.602 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452040+0000", 2024-09-11T09:50:38.602 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:38.602 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.602 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.602 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:50:38.602 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452040+0000", 2024-09-11T09:50:38.602 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.22e-07 2024-09-11T09:50:38.602 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.602 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.602 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:50:38.602 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452042+0000", 2024-09-11T09:50:38.602 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.7010000000000001e-06 2024-09-11T09:50:38.603 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.603 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.603 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:50:38.603 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452045+0000", 2024-09-11T09:50:38.603 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.1310000000000002e-06 2024-09-11T09:50:38.603 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.603 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.603 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:50:38.604 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452048+0000", 2024-09-11T09:50:38.604 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.65e-06 2024-09-11T09:50:38.604 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.604 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.604 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:50:38.604 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.454453+0000", 2024-09-11T09:50:38.604 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0024052959999999999 2024-09-11T09:50:38.604 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.604 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.604 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:50:38.604 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.454498+0000", 2024-09-11T09:50:38.604 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.4978999999999997e-05 2024-09-11T09:50:38.604 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.605 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.605 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:50:38.605 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.454567+0000", 2024-09-11T09:50:38.605 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.9110999999999996e-05 2024-09-11T09:50:38.605 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.605 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.605 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:50:38.605 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.455464+0000", 2024-09-11T09:50:38.605 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00089696199999999995 2024-09-11T09:50:38.605 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.605 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.605 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:50:38.606 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.458844+0000", 2024-09-11T09:50:38.606 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0033802469999999999 2024-09-11T09:50:38.606 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.606 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.606 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:50:38.606 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.458889+0000", 2024-09-11T09:50:38.606 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.4441999999999999e-05 2024-09-11T09:50:38.606 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.606 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.606 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:50:38.606 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.458925+0000", 2024-09-11T09:50:38.606 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.6086e-05 2024-09-11T09:50:38.606 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:38.607 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:50:38.607 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:38.607 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.607 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.607 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:158 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:50:38.607 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:12.452055+0000", 2024-09-11T09:50:38.607 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 26.036016440000001, 2024-09-11T09:50:38.607 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0069533829999999996, 2024-09-11T09:50:38.607 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:50:38.607 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:50:38.607 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:38.607 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:50:38.608 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:50:38.608 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:50:38.608 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 158 2024-09-11T09:50:38.608 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.608 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:50:38.608 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.608 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:50:38.608 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452055+0000", 2024-09-11T09:50:38.608 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:38.608 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.609 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.609 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:50:38.609 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452055+0000", 2024-09-11T09:50:38.609 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:38.609 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.609 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.609 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:50:38.609 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452056+0000", 2024-09-11T09:50:38.609 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.48e-07 2024-09-11T09:50:38.609 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.609 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.610 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:50:38.610 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452087+0000", 2024-09-11T09:50:38.610 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.1149000000000003e-05 2024-09-11T09:50:38.610 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.610 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.610 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:50:38.610 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452092+0000", 2024-09-11T09:50:38.610 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.8400000000000002e-06 2024-09-11T09:50:38.610 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.610 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.610 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:50:38.610 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452095+0000", 2024-09-11T09:50:38.610 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.0740000000000001e-06 2024-09-11T09:50:38.611 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.611 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.611 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:50:38.611 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.454796+0000", 2024-09-11T09:50:38.611 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0027009740000000001 2024-09-11T09:50:38.611 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.611 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.611 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:50:38.611 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.454869+0000", 2024-09-11T09:50:38.611 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.3678999999999997e-05 2024-09-11T09:50:38.611 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.611 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.611 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:50:38.612 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.454992+0000", 2024-09-11T09:50:38.612 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00012279000000000001 2024-09-11T09:50:38.612 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.612 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.612 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:50:38.612 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.456072+0000", 2024-09-11T09:50:38.612 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001080229 2024-09-11T09:50:38.612 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.612 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.612 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:50:38.612 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.458931+0000", 2024-09-11T09:50:38.612 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0028591789999999999 2024-09-11T09:50:38.612 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.613 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.613 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:50:38.613 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.458972+0000", 2024-09-11T09:50:38.613 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.0170999999999999e-05 2024-09-11T09:50:38.613 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.613 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.613 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:50:38.613 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.459009+0000", 2024-09-11T09:50:38.613 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.7049999999999999e-05 2024-09-11T09:50:38.613 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:38.614 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:50:38.614 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:38.614 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.614 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.614 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:159 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:50:38.614 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:12.452099+0000", 2024-09-11T09:50:38.614 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 26.035972271999999, 2024-09-11T09:50:38.614 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0069885950000000002, 2024-09-11T09:50:38.614 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:50:38.614 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:50:38.614 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:38.614 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:50:38.615 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:50:38.615 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:50:38.615 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 159 2024-09-11T09:50:38.615 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.615 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:50:38.615 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.615 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:50:38.615 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452099+0000", 2024-09-11T09:50:38.615 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:38.615 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.615 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.615 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:50:38.615 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452099+0000", 2024-09-11T09:50:38.615 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:38.616 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.616 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.616 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:50:38.616 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452100+0000", 2024-09-11T09:50:38.616 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.2800000000000003e-07 2024-09-11T09:50:38.616 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.616 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.616 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:50:38.616 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452102+0000", 2024-09-11T09:50:38.616 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.824e-06 2024-09-11T09:50:38.616 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.616 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.617 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:50:38.617 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452105+0000", 2024-09-11T09:50:38.617 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.0460000000000001e-06 2024-09-11T09:50:38.617 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.617 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.617 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:50:38.617 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452107+0000", 2024-09-11T09:50:38.617 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.543e-06 2024-09-11T09:50:38.617 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.617 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.617 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:50:38.618 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.455484+0000", 2024-09-11T09:50:38.618 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0033769759999999998 2024-09-11T09:50:38.618 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.618 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.618 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:50:38.618 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.455527+0000", 2024-09-11T09:50:38.618 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.2589999999999997e-05 2024-09-11T09:50:38.618 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.618 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.618 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:50:38.618 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.455594+0000", 2024-09-11T09:50:38.618 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.6775999999999999e-05 2024-09-11T09:50:38.618 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.618 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.618 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:50:38.619 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.456312+0000", 2024-09-11T09:50:38.619 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00071874700000000005 2024-09-11T09:50:38.619 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.619 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.619 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:50:38.619 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.459015+0000", 2024-09-11T09:50:38.619 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0027028540000000002 2024-09-11T09:50:38.619 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.619 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.619 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:50:38.619 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.459054+0000", 2024-09-11T09:50:38.619 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.8840000000000001e-05 2024-09-11T09:50:38.619 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:38.620 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:38.620 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:50:38.620 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.459088+0000", 2024-09-11T09:50:38.620 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.4071e-05 2024-09-11T09:50:38.620 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:38.620 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:50:38.620 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:38.620 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:38.620 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:50:38.620 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-11T09:50:38.621 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 5] out_osds: [0] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5] 2024-09-11T09:50:38.621 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-09-11T09:50:38.621 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-11T09:50:38.623 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-11T09:50:38.624 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-09-11T09:50:38.622+0000 7f7b6c7d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31904) UID: 0 2024-09-11T09:50:38.722 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-09-11T09:50:38.723+0000 7f7b6c7d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31904) UID: 0 2024-09-11T09:50:38.766 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-11T09:50:38.767 INFO:teuthology.orchestra.run.smithi096.stdout: "ops": [], 2024-09-11T09:50:38.767 INFO:teuthology.orchestra.run.smithi096.stdout: "num_ops": 0 2024-09-11T09:50:38.767 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-11T09:50:38.776 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-11T09:50:38.860 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-09-11T09:50:38.861+0000 7f7b6c7d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31904) UID: 0 2024-09-11T09:50:38.920 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-11T09:50:38.920 INFO:teuthology.orchestra.run.smithi096.stdout: "ops": [], 2024-09-11T09:50:38.920 INFO:teuthology.orchestra.run.smithi096.stdout: "complaint_time": 30, 2024-09-11T09:50:38.920 INFO:teuthology.orchestra.run.smithi096.stdout: "num_blocked_ops": 0 2024-09-11T09:50:38.920 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-11T09:50:38.924 INFO:tasks.ceph.osd.0.smithi017.stderr:2024-09-11T09:50:38.925+0000 7fb73c0f1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31665) UID: 0 2024-09-11T09:50:38.928 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-11T09:50:39.025 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:50:39.026+0000 7fdec5849640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31907) UID: 0 2024-09-11T09:50:39.070 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-11T09:50:39.070 INFO:teuthology.orchestra.run.smithi096.stdout: "size": 20, 2024-09-11T09:50:39.070 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 600, 2024-09-11T09:50:39.070 INFO:teuthology.orchestra.run.smithi096.stdout: "ops": [] 2024-09-11T09:50:39.070 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-11T09:50:39.079 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-11T09:50:39.127 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:50:39.127+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:50:39.226 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-11T09:50:39.226 INFO:teuthology.orchestra.run.smithi096.stdout: "ops": [], 2024-09-11T09:50:39.227 INFO:teuthology.orchestra.run.smithi096.stdout: "num_ops": 0 2024-09-11T09:50:39.227 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-11T09:50:39.228 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-09-11T09:50:39.229+0000 7f7b6c7d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31904) UID: 0 2024-09-11T09:50:39.235 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-11T09:50:39.329 INFO:tasks.ceph.osd.2.smithi017.stderr:2024-09-11T09:50:39.330+0000 7fd20b170640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31672) UID: 0 2024-09-11T09:50:39.379 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-11T09:50:39.379 INFO:teuthology.orchestra.run.smithi096.stdout: "ops": [], 2024-09-11T09:50:39.379 INFO:teuthology.orchestra.run.smithi096.stdout: "complaint_time": 30, 2024-09-11T09:50:39.379 INFO:teuthology.orchestra.run.smithi096.stdout: "num_blocked_ops": 0 2024-09-11T09:50:39.380 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-11T09:50:39.387 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-11T09:50:39.430 INFO:tasks.ceph.osd.2.smithi017.stderr:2024-09-11T09:50:39.431+0000 7fd20b170640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31672) UID: 0 2024-09-11T09:50:39.491 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-11T09:50:39.491 INFO:teuthology.orchestra.run.smithi096.stdout: "size": 20, 2024-09-11T09:50:39.491 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 600, 2024-09-11T09:50:39.491 INFO:teuthology.orchestra.run.smithi096.stdout: "ops": [ 2024-09-11T09:50:39.491 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:39.492 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4230.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-11T09:50:39.492 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:18.490736+0000", 2024-09-11T09:50:39.492 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 21.001084007999999, 2024-09-11T09:50:39.492 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0059097569999999999, 2024-09-11T09:50:39.492 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:50:39.492 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:50:39.492 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:39.492 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:50:39.492 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4230", 2024-09-11T09:50:39.492 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.17:0/4101109576", 2024-09-11T09:50:39.492 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 1 2024-09-11T09:50:39.492 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:39.493 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:50:39.493 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:39.493 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:50:39.493 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.490736+0000", 2024-09-11T09:50:39.493 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:39.493 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:39.493 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:39.493 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:50:39.493 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.490736+0000", 2024-09-11T09:50:39.493 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:39.493 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:39.493 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:39.494 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:50:39.494 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.490742+0000", 2024-09-11T09:50:39.494 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.48e-06 2024-09-11T09:50:39.494 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:39.494 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:39.494 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:50:39.494 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.490749+0000", 2024-09-11T09:50:39.494 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.4569999999999999e-06 2024-09-11T09:50:39.494 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:39.494 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:39.494 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:50:39.494 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.490762+0000", 2024-09-11T09:50:39.494 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.2721000000000001e-05 2024-09-11T09:50:39.495 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:39.495 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:39.495 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:50:39.495 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.490774+0000", 2024-09-11T09:50:39.495 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1823e-05 2024-09-11T09:50:39.495 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:39.495 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:39.495 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:50:39.495 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.490846+0000", 2024-09-11T09:50:39.496 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.1878000000000005e-05 2024-09-11T09:50:39.496 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:39.496 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:39.496 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for peered", 2024-09-11T09:50:39.496 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.490857+0000", 2024-09-11T09:50:39.496 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1017e-05 2024-09-11T09:50:39.496 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:39.496 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:39.496 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:50:39.497 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.493856+0000", 2024-09-11T09:50:39.497 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00299885 2024-09-11T09:50:39.497 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:39.497 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:39.497 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:50:39.497 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.494020+0000", 2024-09-11T09:50:39.497 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00016436599999999999 2024-09-11T09:50:39.497 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:39.497 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:39.497 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 2", 2024-09-11T09:50:39.497 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.494186+0000", 2024-09-11T09:50:39.497 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00016562799999999999 2024-09-11T09:50:39.497 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:39.498 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:39.498 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:50:39.498 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.495407+0000", 2024-09-11T09:50:39.498 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0012217420000000001 2024-09-11T09:50:39.498 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:39.498 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:39.498 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:50:39.498 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.496579+0000", 2024-09-11T09:50:39.498 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001171844 2024-09-11T09:50:39.498 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:39.499 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:39.499 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:50:39.499 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.496620+0000", 2024-09-11T09:50:39.499 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.0479000000000003e-05 2024-09-11T09:50:39.499 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:39.499 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:39.500 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:50:39.500 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.496646+0000", 2024-09-11T09:50:39.500 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.6472e-05 2024-09-11T09:50:39.500 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:39.500 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:50:39.500 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:39.501 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:39.501 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:39.501 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_repop_reply(client.4230.0:1 2.3 e16/15 ondisk, result = 0)", 2024-09-11T09:50:39.501 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:18.495321+0000", 2024-09-11T09:50:39.501 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 20.996499343, 2024-09-11T09:50:39.501 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.4965999999999998e-05, 2024-09-11T09:50:39.502 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:50:39.502 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:50:39.502 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "started", 2024-09-11T09:50:39.502 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:50:39.502 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:39.502 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:50:39.503 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.495321+0000", 2024-09-11T09:50:39.503 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:39.503 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:39.503 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:39.503 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:50:39.503 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.495321+0000", 2024-09-11T09:50:39.504 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:39.504 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:39.504 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:39.504 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:50:39.504 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.495322+0000", 2024-09-11T09:50:39.504 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.7700000000000004e-07 2024-09-11T09:50:39.505 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:39.505 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:39.505 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:50:39.505 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.495325+0000", 2024-09-11T09:50:39.505 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.7780000000000001e-06 2024-09-11T09:50:39.506 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:39.506 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:39.506 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:50:39.506 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.495338+0000", 2024-09-11T09:50:39.506 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.2315e-05 2024-09-11T09:50:39.506 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:39.507 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:39.507 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:50:39.507 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.495345+0000", 2024-09-11T09:50:39.507 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.8889999999999996e-06 2024-09-11T09:50:39.507 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:39.507 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:39.508 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:50:39.508 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.495392+0000", 2024-09-11T09:50:39.508 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.7524000000000001e-05 2024-09-11T09:50:39.508 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:39.508 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:39.508 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:50:39.509 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.495402+0000", 2024-09-11T09:50:39.509 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.5840000000000001e-06 2024-09-11T09:50:39.509 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:39.509 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:39.509 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:50:39.509 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.495416+0000", 2024-09-11T09:50:39.510 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.4199000000000001e-05 2024-09-11T09:50:39.510 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:39.510 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:50:39.510 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:39.510 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:39.511 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:39.511 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4230.0:2 2.2 2:5cea7035:::rbd_info:head [call rbd.metadata_list in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e16)", 2024-09-11T09:50:39.511 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:18.499576+0000", 2024-09-11T09:50:39.511 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 20.992244836000001, 2024-09-11T09:50:39.511 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00018163000000000001, 2024-09-11T09:50:39.511 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:50:39.512 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:50:39.512 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "reached pg", 2024-09-11T09:50:39.512 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:50:39.512 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4230", 2024-09-11T09:50:39.512 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.17:0/4101109576", 2024-09-11T09:50:39.512 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 2 2024-09-11T09:50:39.513 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:39.513 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:50:39.513 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:39.513 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:50:39.513 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.499576+0000", 2024-09-11T09:50:39.513 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:39.514 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:39.514 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:39.514 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:50:39.514 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.499576+0000", 2024-09-11T09:50:39.514 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:39.514 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:39.515 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:39.515 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:50:39.515 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.499577+0000", 2024-09-11T09:50:39.515 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.124e-06 2024-09-11T09:50:39.515 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:39.515 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:39.516 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:50:39.516 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.499584+0000", 2024-09-11T09:50:39.516 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.1949999999999997e-06 2024-09-11T09:50:39.516 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:39.516 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:39.517 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:50:39.517 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.499597+0000", 2024-09-11T09:50:39.517 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.366e-05 2024-09-11T09:50:39.517 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:39.517 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:39.517 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:50:39.518 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.499604+0000", 2024-09-11T09:50:39.518 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.2709999999999998e-06 2024-09-11T09:50:39.518 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:39.518 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:39.518 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:50:39.519 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.499660+0000", 2024-09-11T09:50:39.519 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.6270999999999997e-05 2024-09-11T09:50:39.519 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:39.519 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:39.519 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:50:39.520 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.499757+0000", 2024-09-11T09:50:39.520 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.7108999999999995e-05 2024-09-11T09:50:39.520 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:39.520 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:50:39.520 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:39.521 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:39.521 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:39.521 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4230.0:3 2.2 2:5cea7035:::rbd_info:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e16)", 2024-09-11T09:50:39.521 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:18.499937+0000", 2024-09-11T09:50:39.521 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 20.991883047000002, 2024-09-11T09:50:39.522 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000143318, 2024-09-11T09:50:39.522 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:50:39.522 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:50:39.522 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "reached pg", 2024-09-11T09:50:39.522 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:50:39.522 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4230", 2024-09-11T09:50:39.523 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.17:0/4101109576", 2024-09-11T09:50:39.523 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 3 2024-09-11T09:50:39.523 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:39.523 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:50:39.523 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:39.524 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:50:39.524 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.499937+0000", 2024-09-11T09:50:39.524 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:39.524 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:39.524 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:39.524 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:50:39.525 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.499937+0000", 2024-09-11T09:50:39.525 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:39.525 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:39.525 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:39.525 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:50:39.526 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.499938+0000", 2024-09-11T09:50:39.526 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.195e-06 2024-09-11T09:50:39.526 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:39.526 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:39.526 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:50:39.526 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.499943+0000", 2024-09-11T09:50:39.527 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.9080000000000003e-06 2024-09-11T09:50:39.527 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:39.527 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:39.527 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:50:39.527 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.499956+0000", 2024-09-11T09:50:39.528 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.2728000000000001e-05 2024-09-11T09:50:39.528 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:39.528 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:39.528 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:50:39.528 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.499964+0000", 2024-09-11T09:50:39.529 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.8220000000000007e-06 2024-09-11T09:50:39.529 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:39.529 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:39.529 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:50:39.529 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.500017+0000", 2024-09-11T09:50:39.529 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.2809000000000002e-05 2024-09-11T09:50:39.530 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:39.530 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:39.530 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:50:39.530 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.500081+0000", 2024-09-11T09:50:39.530 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.3856000000000007e-05 2024-09-11T09:50:39.530 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:39.530 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:50:39.530 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:39.530 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:39.530 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:39.531 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4230.0:5 2.2 2:5cea7035:::rbd_info:head [create,write 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-11T09:50:39.531 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:19.492662+0000", 2024-09-11T09:50:39.531 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 19.999158387000001, 2024-09-11T09:50:39.531 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0043824759999999997, 2024-09-11T09:50:39.531 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:50:39.531 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:50:39.531 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:39.531 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:50:39.531 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4230", 2024-09-11T09:50:39.532 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.17:0/4101109576", 2024-09-11T09:50:39.532 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 5 2024-09-11T09:50:39.532 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:39.532 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:50:39.532 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:39.532 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:50:39.532 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.492662+0000", 2024-09-11T09:50:39.532 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:39.532 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:39.533 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:39.533 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:50:39.533 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.492662+0000", 2024-09-11T09:50:39.533 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:39.533 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:39.533 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:39.533 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:50:39.533 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.492664+0000", 2024-09-11T09:50:39.534 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.3599999999999999e-06 2024-09-11T09:50:39.534 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:39.534 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:39.534 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:50:39.534 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.492675+0000", 2024-09-11T09:50:39.534 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1182000000000001e-05 2024-09-11T09:50:39.534 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:39.534 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:39.534 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:50:39.535 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.492693+0000", 2024-09-11T09:50:39.535 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.7031999999999999e-05 2024-09-11T09:50:39.535 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:39.535 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:39.535 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:50:39.535 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.492702+0000", 2024-09-11T09:50:39.535 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.5540000000000003e-06 2024-09-11T09:50:39.535 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:39.535 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:39.536 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:50:39.536 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.492767+0000", 2024-09-11T09:50:39.536 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.5411000000000001e-05 2024-09-11T09:50:39.536 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:39.536 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:39.536 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op must wait for map", 2024-09-11T09:50:39.536 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.492773+0000", 2024-09-11T09:50:39.536 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.1039999999999998e-06 2024-09-11T09:50:39.537 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:39.537 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:39.537 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:50:39.537 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.494354+0000", 2024-09-11T09:50:39.537 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0015814049999999999 2024-09-11T09:50:39.537 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:39.537 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:39.537 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:50:39.537 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.494476+0000", 2024-09-11T09:50:39.538 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000122089 2024-09-11T09:50:39.538 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:39.538 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:39.538 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 1", 2024-09-11T09:50:39.538 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.494577+0000", 2024-09-11T09:50:39.538 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000100536 2024-09-11T09:50:39.538 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:39.538 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:39.538 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:50:39.539 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.495770+0000", 2024-09-11T09:50:39.539 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0011932469999999999 2024-09-11T09:50:39.539 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:39.539 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:39.539 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:50:39.539 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.496966+0000", 2024-09-11T09:50:39.539 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001196003 2024-09-11T09:50:39.539 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:39.539 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:39.540 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:50:39.540 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.497011+0000", 2024-09-11T09:50:39.540 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.5229999999999999e-05 2024-09-11T09:50:39.540 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:39.540 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:39.540 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:50:39.540 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.497044+0000", 2024-09-11T09:50:39.540 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.3322999999999999e-05 2024-09-11T09:50:39.540 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:39.541 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:50:39.541 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:39.541 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:39.541 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:39.541 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_repop_reply(client.4230.0:5 2.2 e17/15 ondisk, result = 0)", 2024-09-11T09:50:39.541 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:19.495669+0000", 2024-09-11T09:50:39.542 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 19.996151599000001, 2024-09-11T09:50:39.542 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000106431, 2024-09-11T09:50:39.542 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:50:39.542 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:50:39.542 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "started", 2024-09-11T09:50:39.542 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:50:39.543 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:39.543 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:50:39.543 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.495669+0000", 2024-09-11T09:50:39.543 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:39.543 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:39.544 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:39.544 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:50:39.544 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.495669+0000", 2024-09-11T09:50:39.544 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:39.544 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:39.544 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:39.544 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:50:39.545 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.495670+0000", 2024-09-11T09:50:39.545 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.218e-06 2024-09-11T09:50:39.545 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:39.545 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:39.545 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:50:39.546 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.495676+0000", 2024-09-11T09:50:39.546 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.5119999999999997e-06 2024-09-11T09:50:39.546 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:39.546 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:39.546 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:50:39.547 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.495690+0000", 2024-09-11T09:50:39.547 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.3312000000000001e-05 2024-09-11T09:50:39.547 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:39.547 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:39.547 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:50:39.548 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.495706+0000", 2024-09-11T09:50:39.548 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.6560000000000001e-05 2024-09-11T09:50:39.548 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:39.548 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:39.548 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:50:39.548 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.495759+0000", 2024-09-11T09:50:39.549 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.2828000000000003e-05 2024-09-11T09:50:39.549 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:39.549 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:39.549 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:50:39.549 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.495765+0000", 2024-09-11T09:50:39.549 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.7869999999999998e-06 2024-09-11T09:50:39.550 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:39.550 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:39.550 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:50:39.550 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.495775+0000", 2024-09-11T09:50:39.550 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.0214000000000001e-05 2024-09-11T09:50:39.551 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:39.551 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:50:39.551 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:39.551 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:39.551 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:39.551 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4230.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-09-11T09:50:39.552 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:20.498634+0000", 2024-09-11T09:50:39.552 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 18.993185987, 2024-09-11T09:50:39.552 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0050648539999999997, 2024-09-11T09:50:39.552 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:50:39.552 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:50:39.553 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:39.553 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:50:39.553 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4230", 2024-09-11T09:50:39.553 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.17:0/4101109576", 2024-09-11T09:50:39.553 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 7 2024-09-11T09:50:39.554 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:39.554 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:50:39.554 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:39.554 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:50:39.554 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.498634+0000", 2024-09-11T09:50:39.554 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:39.555 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:39.555 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:39.555 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:50:39.555 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.498634+0000", 2024-09-11T09:50:39.555 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:39.556 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:39.556 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:39.556 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:50:39.556 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.498637+0000", 2024-09-11T09:50:39.556 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.1919999999999999e-06 2024-09-11T09:50:39.557 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:39.557 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:39.557 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:50:39.557 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.498648+0000", 2024-09-11T09:50:39.558 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1116e-05 2024-09-11T09:50:39.558 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:39.558 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:39.558 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:50:39.558 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.498666+0000", 2024-09-11T09:50:39.559 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.8411999999999999e-05 2024-09-11T09:50:39.559 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:39.559 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:39.559 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:50:39.559 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.498675+0000", 2024-09-11T09:50:39.559 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.4059999999999994e-06 2024-09-11T09:50:39.559 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:39.560 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:39.560 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:50:39.560 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.498745+0000", 2024-09-11T09:50:39.560 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.9764000000000005e-05 2024-09-11T09:50:39.560 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:39.560 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:39.560 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op must wait for map", 2024-09-11T09:50:39.561 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.498754+0000", 2024-09-11T09:50:39.561 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.5969999999999992e-06 2024-09-11T09:50:39.561 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:39.561 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:39.561 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:50:39.561 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.500385+0000", 2024-09-11T09:50:39.561 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0016308360000000001 2024-09-11T09:50:39.561 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:39.562 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:39.562 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:50:39.562 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.500473+0000", 2024-09-11T09:50:39.562 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.7966000000000004e-05 2024-09-11T09:50:39.562 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:39.562 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:39.562 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 1", 2024-09-11T09:50:39.562 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.500591+0000", 2024-09-11T09:50:39.563 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000117895 2024-09-11T09:50:39.563 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:39.563 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:39.563 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:50:39.563 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.501685+0000", 2024-09-11T09:50:39.563 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001094886 2024-09-11T09:50:39.563 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:39.564 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:39.564 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:50:39.564 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.503627+0000", 2024-09-11T09:50:39.564 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001941199 2024-09-11T09:50:39.564 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:39.564 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:39.564 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:50:39.564 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.503667+0000", 2024-09-11T09:50:39.564 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.0645000000000001e-05 2024-09-11T09:50:39.565 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:39.565 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:39.565 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:50:39.565 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.503699+0000", 2024-09-11T09:50:39.565 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.1940000000000003e-05 2024-09-11T09:50:39.565 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:39.565 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:50:39.565 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:39.565 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:39.565 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:39.565 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_repop_reply(client.4230.0:7 2.2 e18/15 ondisk, result = 0)", 2024-09-11T09:50:39.565 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:20.501608+0000", 2024-09-11T09:50:39.565 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 18.990212464999999, 2024-09-11T09:50:39.566 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.2459000000000005e-05, 2024-09-11T09:50:39.566 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:50:39.566 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:50:39.566 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "started", 2024-09-11T09:50:39.566 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:50:39.566 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:39.566 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:50:39.566 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.501608+0000", 2024-09-11T09:50:39.566 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:39.567 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:39.567 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:39.567 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:50:39.567 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.501608+0000", 2024-09-11T09:50:39.567 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:39.567 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:39.567 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:39.567 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:50:39.567 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.501609+0000", 2024-09-11T09:50:39.567 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.0160000000000001e-06 2024-09-11T09:50:39.567 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:39.567 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:39.567 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:50:39.568 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.501613+0000", 2024-09-11T09:50:39.568 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.5959999999999997e-06 2024-09-11T09:50:39.568 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:39.568 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:39.568 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:50:39.568 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.501625+0000", 2024-09-11T09:50:39.568 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1888e-05 2024-09-11T09:50:39.568 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:39.568 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:39.568 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:50:39.568 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.501634+0000", 2024-09-11T09:50:39.568 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.6139999999999992e-06 2024-09-11T09:50:39.569 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:39.569 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:39.569 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:50:39.569 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.501676+0000", 2024-09-11T09:50:39.569 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.1953999999999998e-05 2024-09-11T09:50:39.569 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:39.569 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:39.569 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:50:39.569 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.501682+0000", 2024-09-11T09:50:39.569 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.8000000000000004e-06 2024-09-11T09:50:39.569 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:39.569 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:39.569 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:50:39.570 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.501690+0000", 2024-09-11T09:50:39.570 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.5909999999999996e-06 2024-09-11T09:50:39.570 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:39.570 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:50:39.570 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:39.570 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:39.570 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:50:39.570 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-11T09:50:39.571 INFO:teuthology.orchestra.run.smithi017.stderr:noscrub is set 2024-09-11T09:50:39.572 INFO:teuthology.orchestra.run.smithi017.stderr:reweighted osd.1 to 1 (10000) 2024-09-11T09:50:39.572 INFO:tasks.ceph.osd.0.smithi017.stderr:2024-09-11T09:50:39.572+0000 7fb73c0f1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31665) UID: 0 2024-09-11T09:50:39.573 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-11T09:50:39.580 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-09-11T09:50:39.672 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:50:39.673+0000 7f3c96ec9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31909) UID: 0 2024-09-11T09:50:39.688 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:50:39.689 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [], 2024-09-11T09:50:39.689 INFO:teuthology.orchestra.run.smithi017.stdout: "num_ops": 0 2024-09-11T09:50:39.689 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:50:39.699 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-11T09:50:39.740 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T09:50:39.773 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:50:39.774+0000 7f3c96ec9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31909) UID: 0 2024-09-11T09:50:39.809 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:50:39.809 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [], 2024-09-11T09:50:39.809 INFO:teuthology.orchestra.run.smithi017.stdout: "complaint_time": 30, 2024-09-11T09:50:39.809 INFO:teuthology.orchestra.run.smithi017.stdout: "num_blocked_ops": 0 2024-09-11T09:50:39.810 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:50:39.819 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-11T09:50:39.873 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:50:39.875+0000 7fdec5849640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31907) UID: 0 2024-09-11T09:50:39.882 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-11T09:50:39.883 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T09:50:39.902 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-11T09:50:39.902 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T09:50:39.919 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-11T09:50:39.919 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T09:50:39.929 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:50:39.929 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:50:39.938 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-11T09:50:39.954 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-11T09:50:39.954 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T09:50:39.973 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-11T09:50:39.973 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T09:50:39.974 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:50:39.975+0000 7fdec5849640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31907) UID: 0 2024-09-11T09:50:39.990 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-11T09:50:39.990 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T09:50:40.047 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:50:40.047 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:50:40.056 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-11T09:50:40.075 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:50:40.076+0000 7fdec5849640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31907) UID: 0 2024-09-11T09:50:40.163 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:50:40.163 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:50:40.172 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-11T09:50:40.176 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:50:40.177+0000 7fdec5849640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31907) UID: 0 2024-09-11T09:50:40.277 INFO:tasks.ceph.osd.2.smithi017.stderr:2024-09-11T09:50:40.278+0000 7fd20b170640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31672) UID: 0 2024-09-11T09:50:40.279 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:50:40.279 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:50:40.287 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-11T09:50:40.378 INFO:tasks.ceph.osd.0.smithi017.stderr:2024-09-11T09:50:40.379+0000 7fb73c0f1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31665) UID: 0 2024-09-11T09:50:40.436 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:50:40.436 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:50:40.445 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-11T09:50:40.486 INFO:tasks.ceph.osd.0.smithi017.stderr:2024-09-11T09:50:40.487+0000 7fb73c0f1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31665) UID: 0 2024-09-11T09:50:40.563 INFO:teuthology.orchestra.run.smithi017.stderr:reweighted osd.2 to 1 (10000) 2024-09-11T09:50:40.577 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-09-11T09:50:40.579 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:50:40.580+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:50:40.595 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:50:40.595 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:50:40.605 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-11T09:50:40.689 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:50:40.690+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:50:40.781 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-09-11T09:50:40.781+0000 7f7b6c7d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31904) UID: 0 2024-09-11T09:50:40.786 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:50:40.786 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:50:40.795 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-11T09:50:40.881 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:50:40.882+0000 7fdec5849640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31907) UID: 0 2024-09-11T09:50:40.909 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:50:40.910 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:50:40.919 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-11T09:50:41.003 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-09-11T09:50:41.004+0000 7f7b6c7d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31904) UID: 0 2024-09-11T09:50:41.063 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:50:41.063 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:50:41.072 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-11T09:50:41.083 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:50:41.084+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:50:41.184 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:50:41.185+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:50:41.216 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:50:41.217 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:50:41.224 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-11T09:50:41.285 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:50:41.285+0000 7fdec5849640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31907) UID: 0 2024-09-11T09:50:41.329 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:50:41.329 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:50:41.337 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-11T09:50:41.386 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:50:41.387+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:50:41.480 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:50:41.480 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:50:41.486 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:50:41.487+0000 7f3c96ec9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31909) UID: 0 2024-09-11T09:50:41.489 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-11T09:50:41.570 INFO:teuthology.orchestra.run.smithi017.stderr:reweighted osd.3 to 1 (10000) 2024-09-11T09:50:41.577 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-11T09:50:41.586 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 4 1 2024-09-11T09:50:41.628 INFO:tasks.ceph.osd.0.smithi017.stderr:2024-09-11T09:50:41.629+0000 7fb73c0f1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31665) UID: 0 2024-09-11T09:50:41.632 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:50:41.632 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:50:41.640 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-11T09:50:41.689 INFO:tasks.ceph.osd.0.smithi017.stderr:2024-09-11T09:50:41.690+0000 7fb73c0f1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31665) UID: 0 2024-09-11T09:50:41.743 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:50:41.744 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:50:41.752 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-11T09:50:41.790 INFO:tasks.ceph.osd.0.smithi017.stderr:2024-09-11T09:50:41.791+0000 7fb73c0f1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31665) UID: 0 2024-09-11T09:50:41.854 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:50:41.855 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:50:41.863 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-11T09:50:41.891 INFO:tasks.ceph.osd.0.smithi017.stderr:2024-09-11T09:50:41.892+0000 7fb73c0f1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31665) UID: 0 2024-09-11T09:50:41.966 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:50:41.966 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:50:41.974 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-11T09:50:42.002 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T09:50:42.018 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:50:42.019+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:50:42.083 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:50:42.084 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:50:42.093 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-11T09:50:42.094 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:50:42.094+0000 7f3c96ec9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31909) UID: 0 2024-09-11T09:50:42.143 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-11T09:50:42.143 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T09:50:42.161 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-11T09:50:42.161 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T09:50:42.178 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-11T09:50:42.178 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T09:50:42.193 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:50:42.194+0000 7fdec5849640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31907) UID: 0 2024-09-11T09:50:42.195 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-11T09:50:42.195 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T09:50:42.203 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:50:42.203 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [], 2024-09-11T09:50:42.203 INFO:teuthology.orchestra.run.smithi017.stdout: "complaint_time": 30, 2024-09-11T09:50:42.203 INFO:teuthology.orchestra.run.smithi017.stdout: "num_blocked_ops": 0 2024-09-11T09:50:42.203 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:50:42.212 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-11T09:50:42.213 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-11T09:50:42.213 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T09:50:42.230 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-11T09:50:42.230 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T09:50:42.294 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:50:42.295+0000 7fdec5849640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31907) UID: 0 2024-09-11T09:50:42.323 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:50:42.323 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 20, 2024-09-11T09:50:42.323 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 600, 2024-09-11T09:50:42.323 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [ 2024-09-11T09:50:42.323 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.323 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4183.0:21 1.0 e13/12 1:30306672:devicehealth::main.db.0000000000000000:head v 13'11, mlcod=13'8)", 2024-09-11T09:50:42.323 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:11.491728+0000", 2024-09-11T09:50:42.323 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 30.831017623000001, 2024-09-11T09:50:42.323 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00093269800000000003, 2024-09-11T09:50:42.323 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:50:42.323 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:50:42.324 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:42.324 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:50:42.324 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.324 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:50:42.324 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.491728+0000", 2024-09-11T09:50:42.324 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:42.324 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.324 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.324 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:50:42.324 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.491728+0000", 2024-09-11T09:50:42.324 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:42.324 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.324 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.325 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:50:42.325 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.491729+0000", 2024-09-11T09:50:42.325 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.8599999999999997e-07 2024-09-11T09:50:42.325 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.325 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.325 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:50:42.325 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.491820+0000", 2024-09-11T09:50:42.325 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.0812999999999999e-05 2024-09-11T09:50:42.325 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.325 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.325 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:50:42.325 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.491833+0000", 2024-09-11T09:50:42.325 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.341e-05 2024-09-11T09:50:42.326 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.326 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.326 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:50:42.326 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.491841+0000", 2024-09-11T09:50:42.326 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.8730000000000007e-06 2024-09-11T09:50:42.326 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.326 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.326 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:50:42.326 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.491916+0000", 2024-09-11T09:50:42.326 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.4881000000000004e-05 2024-09-11T09:50:42.326 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.326 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.326 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:50:42.327 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.491931+0000", 2024-09-11T09:50:42.327 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.4825e-05 2024-09-11T09:50:42.327 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.327 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.327 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:50:42.327 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.492633+0000", 2024-09-11T09:50:42.327 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00070191600000000002 2024-09-11T09:50:42.327 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.327 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.327 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:50:42.327 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.492661+0000", 2024-09-11T09:50:42.327 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.8093999999999999e-05 2024-09-11T09:50:42.327 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:42.327 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:50:42.328 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:42.328 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.328 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.328 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4183.0:30 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'16, mlcod=13'14)", 2024-09-11T09:50:42.328 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:11.498866+0000", 2024-09-11T09:50:42.328 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 30.823879900000001, 2024-09-11T09:50:42.328 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00089294000000000005, 2024-09-11T09:50:42.328 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:50:42.328 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:50:42.328 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:42.328 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:50:42.328 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.328 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:50:42.329 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.498866+0000", 2024-09-11T09:50:42.329 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:42.329 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.329 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.329 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:50:42.329 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.498866+0000", 2024-09-11T09:50:42.329 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:42.329 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.329 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.329 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:50:42.329 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.498867+0000", 2024-09-11T09:50:42.329 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.7000000000000003e-07 2024-09-11T09:50:42.329 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.330 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.330 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:50:42.330 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.498873+0000", 2024-09-11T09:50:42.330 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.0140000000000004e-06 2024-09-11T09:50:42.330 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.330 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.330 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:50:42.330 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.498886+0000", 2024-09-11T09:50:42.330 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2745000000000001e-05 2024-09-11T09:50:42.331 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.331 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.331 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:50:42.331 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.498894+0000", 2024-09-11T09:50:42.331 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.2349999999999999e-06 2024-09-11T09:50:42.331 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.331 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.331 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:50:42.331 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.498941+0000", 2024-09-11T09:50:42.331 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.7679999999999998e-05 2024-09-11T09:50:42.331 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.331 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.331 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:50:42.331 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.498955+0000", 2024-09-11T09:50:42.332 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3446e-05 2024-09-11T09:50:42.332 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.332 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.332 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:50:42.332 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.499717+0000", 2024-09-11T09:50:42.332 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00076248199999999998 2024-09-11T09:50:42.332 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.332 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.332 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:50:42.332 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.499759+0000", 2024-09-11T09:50:42.332 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.1467999999999997e-05 2024-09-11T09:50:42.332 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:42.332 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:50:42.333 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:42.333 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.333 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.333 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4183.0:31 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'17, mlcod=13'14)", 2024-09-11T09:50:42.333 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:11.499319+0000", 2024-09-11T09:50:42.333 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 30.823426953999999, 2024-09-11T09:50:42.333 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0010073879999999999, 2024-09-11T09:50:42.333 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:50:42.333 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:50:42.333 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:42.333 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:50:42.333 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.333 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:50:42.334 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.499319+0000", 2024-09-11T09:50:42.334 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:42.334 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.334 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.334 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:50:42.334 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.499319+0000", 2024-09-11T09:50:42.334 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:42.334 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.334 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.334 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:50:42.334 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.499320+0000", 2024-09-11T09:50:42.334 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.7700000000000004e-07 2024-09-11T09:50:42.334 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.334 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.335 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:50:42.335 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.499326+0000", 2024-09-11T09:50:42.335 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.6420000000000001e-06 2024-09-11T09:50:42.335 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.335 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.335 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:50:42.335 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.499339+0000", 2024-09-11T09:50:42.335 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2537999999999999e-05 2024-09-11T09:50:42.335 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.335 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.335 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:50:42.335 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.499345+0000", 2024-09-11T09:50:42.335 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.8220000000000003e-06 2024-09-11T09:50:42.336 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.336 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.336 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:50:42.336 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.499403+0000", 2024-09-11T09:50:42.336 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.8814000000000003e-05 2024-09-11T09:50:42.336 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.336 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.336 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:50:42.336 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.499427+0000", 2024-09-11T09:50:42.336 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.3853999999999999e-05 2024-09-11T09:50:42.336 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.336 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.336 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:50:42.337 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.500294+0000", 2024-09-11T09:50:42.337 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00086622399999999997 2024-09-11T09:50:42.337 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.337 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.337 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:50:42.337 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.500326+0000", 2024-09-11T09:50:42.337 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.2716999999999999e-05 2024-09-11T09:50:42.337 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:42.337 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:50:42.337 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:42.337 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.337 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.337 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4183.0:32 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'18, mlcod=13'14)", 2024-09-11T09:50:42.337 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:11.499796+0000", 2024-09-11T09:50:42.338 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 30.822949747999999, 2024-09-11T09:50:42.338 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00092915100000000002, 2024-09-11T09:50:42.338 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:50:42.338 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:50:42.338 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:42.338 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:50:42.338 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.338 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:50:42.338 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.499796+0000", 2024-09-11T09:50:42.338 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:42.338 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.338 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.338 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:50:42.339 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.499796+0000", 2024-09-11T09:50:42.339 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:42.339 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.339 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.339 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:50:42.339 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.499797+0000", 2024-09-11T09:50:42.339 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.9400000000000004e-07 2024-09-11T09:50:42.339 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.339 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.339 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:50:42.339 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.499841+0000", 2024-09-11T09:50:42.339 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.3900000000000003e-05 2024-09-11T09:50:42.339 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.340 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.340 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:50:42.340 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.499852+0000", 2024-09-11T09:50:42.340 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1462e-05 2024-09-11T09:50:42.340 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.340 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.340 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:50:42.340 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.499858+0000", 2024-09-11T09:50:42.340 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.8320000000000002e-06 2024-09-11T09:50:42.340 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.340 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.340 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:50:42.340 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.499915+0000", 2024-09-11T09:50:42.341 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.7187e-05 2024-09-11T09:50:42.341 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.341 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.341 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:50:42.341 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.499930+0000", 2024-09-11T09:50:42.341 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.4928e-05 2024-09-11T09:50:42.341 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.341 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.341 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:50:42.341 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.500697+0000", 2024-09-11T09:50:42.341 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00076658399999999995 2024-09-11T09:50:42.341 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.341 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.341 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:50:42.342 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.500725+0000", 2024-09-11T09:50:42.342 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.8364000000000002e-05 2024-09-11T09:50:42.342 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:42.342 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:50:42.342 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:42.342 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.342 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.342 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4183.0:34 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'20, mlcod=13'15)", 2024-09-11T09:50:42.342 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:11.501016+0000", 2024-09-11T09:50:42.342 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 30.82173023, 2024-09-11T09:50:42.342 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00097815399999999996, 2024-09-11T09:50:42.342 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:50:42.342 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:50:42.342 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:42.343 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:50:42.343 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.343 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:50:42.343 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.501016+0000", 2024-09-11T09:50:42.343 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:42.343 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.343 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.343 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:50:42.343 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.501016+0000", 2024-09-11T09:50:42.343 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:42.343 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.343 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.343 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:50:42.344 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.501016+0000", 2024-09-11T09:50:42.344 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.3799999999999997e-07 2024-09-11T09:50:42.344 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.344 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.344 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:50:42.344 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.501023+0000", 2024-09-11T09:50:42.344 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.2180000000000002e-06 2024-09-11T09:50:42.344 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.344 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.344 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:50:42.344 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.501031+0000", 2024-09-11T09:50:42.344 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.5379999999999997e-06 2024-09-11T09:50:42.344 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.344 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.345 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:50:42.345 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.501035+0000", 2024-09-11T09:50:42.345 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.216e-06 2024-09-11T09:50:42.345 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.345 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.345 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:50:42.345 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.501105+0000", 2024-09-11T09:50:42.345 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.9757000000000003e-05 2024-09-11T09:50:42.345 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.345 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.345 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:50:42.345 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.501120+0000", 2024-09-11T09:50:42.345 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.5358000000000001e-05 2024-09-11T09:50:42.346 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.346 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.346 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:50:42.346 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.501955+0000", 2024-09-11T09:50:42.346 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00083505000000000005 2024-09-11T09:50:42.346 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.346 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.346 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:50:42.346 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.501994+0000", 2024-09-11T09:50:42.346 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.8479000000000002e-05 2024-09-11T09:50:42.346 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:42.346 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:50:42.346 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:42.346 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.347 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.347 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4183.0:38 1.0 e13/12 1:30306672:devicehealth::main.db.0000000000000000:head v 13'24, mlcod=13'22)", 2024-09-11T09:50:42.347 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:11.503891+0000", 2024-09-11T09:50:42.347 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 30.818854787999999, 2024-09-11T09:50:42.347 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000959351, 2024-09-11T09:50:42.347 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:50:42.347 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:50:42.347 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:42.347 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:50:42.347 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.347 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:50:42.347 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.503891+0000", 2024-09-11T09:50:42.347 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:42.347 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.348 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.348 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:50:42.348 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.503891+0000", 2024-09-11T09:50:42.348 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:42.348 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.348 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.348 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:50:42.348 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.503892+0000", 2024-09-11T09:50:42.348 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.5499999999999998e-07 2024-09-11T09:50:42.348 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.348 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.348 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:50:42.348 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.503967+0000", 2024-09-11T09:50:42.349 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.4993999999999994e-05 2024-09-11T09:50:42.349 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.349 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.349 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:50:42.349 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.503976+0000", 2024-09-11T09:50:42.349 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.4709999999999996e-06 2024-09-11T09:50:42.349 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.349 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.349 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:50:42.349 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.503981+0000", 2024-09-11T09:50:42.349 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.6789999999999996e-06 2024-09-11T09:50:42.349 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.349 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.349 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:50:42.350 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.504021+0000", 2024-09-11T09:50:42.350 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.0065999999999999e-05 2024-09-11T09:50:42.350 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.350 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.350 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:50:42.350 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.504036+0000", 2024-09-11T09:50:42.350 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.468e-05 2024-09-11T09:50:42.350 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.350 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.350 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:50:42.350 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.504813+0000", 2024-09-11T09:50:42.350 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00077718299999999998 2024-09-11T09:50:42.350 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.351 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.351 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:50:42.351 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.504850+0000", 2024-09-11T09:50:42.351 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.7722999999999998e-05 2024-09-11T09:50:42.351 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:42.351 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:50:42.351 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:42.351 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.351 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.351 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4183.0:54 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'32, mlcod=13'29)", 2024-09-11T09:50:42.351 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:11.515404+0000", 2024-09-11T09:50:42.351 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 30.807342182999999, 2024-09-11T09:50:42.351 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00089471799999999997, 2024-09-11T09:50:42.351 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:50:42.352 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:50:42.352 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:42.352 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:50:42.352 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.352 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:50:42.352 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.515404+0000", 2024-09-11T09:50:42.352 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:42.352 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.352 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.352 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:50:42.352 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.515404+0000", 2024-09-11T09:50:42.352 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:42.352 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.353 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.353 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:50:42.353 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.515404+0000", 2024-09-11T09:50:42.353 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.3799999999999996e-07 2024-09-11T09:50:42.353 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.353 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.353 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:50:42.353 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.515410+0000", 2024-09-11T09:50:42.353 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.1789999999999999e-06 2024-09-11T09:50:42.353 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.353 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.353 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:50:42.353 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.515422+0000", 2024-09-11T09:50:42.353 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1929999999999999e-05 2024-09-11T09:50:42.354 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.354 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.354 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:50:42.354 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.515427+0000", 2024-09-11T09:50:42.354 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.6060000000000002e-06 2024-09-11T09:50:42.354 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.354 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.354 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:50:42.354 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.515482+0000", 2024-09-11T09:50:42.354 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.4743000000000001e-05 2024-09-11T09:50:42.354 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.354 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.354 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:50:42.354 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.515498+0000", 2024-09-11T09:50:42.355 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.5778999999999999e-05 2024-09-11T09:50:42.355 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.355 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.355 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:50:42.355 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.516271+0000", 2024-09-11T09:50:42.355 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00077312599999999996 2024-09-11T09:50:42.355 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.355 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.355 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:50:42.355 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.516298+0000", 2024-09-11T09:50:42.355 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.7617e-05 2024-09-11T09:50:42.355 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:42.355 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:50:42.356 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:42.356 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.356 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.356 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4183.0:57 1.0 e13/12 1:30306672:devicehealth::main.db.0000000000000000:head v 13'35, mlcod=13'33)", 2024-09-11T09:50:42.356 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:11.519823+0000", 2024-09-11T09:50:42.356 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 30.802923126, 2024-09-11T09:50:42.356 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00090876100000000001, 2024-09-11T09:50:42.356 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:50:42.356 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:50:42.356 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:42.356 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:50:42.356 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.356 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:50:42.356 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.519823+0000", 2024-09-11T09:50:42.357 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:42.357 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.357 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.357 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:50:42.357 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.519823+0000", 2024-09-11T09:50:42.357 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:42.357 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.357 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.357 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:50:42.357 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.519823+0000", 2024-09-11T09:50:42.357 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.7000000000000004e-07 2024-09-11T09:50:42.357 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.357 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.358 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:50:42.358 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.519859+0000", 2024-09-11T09:50:42.358 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.5508e-05 2024-09-11T09:50:42.358 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.358 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.358 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:50:42.358 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.519871+0000", 2024-09-11T09:50:42.358 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.221e-05 2024-09-11T09:50:42.358 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.358 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.358 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:50:42.358 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.519877+0000", 2024-09-11T09:50:42.358 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.0410000000000003e-06 2024-09-11T09:50:42.358 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.359 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.359 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:50:42.359 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.519955+0000", 2024-09-11T09:50:42.359 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.8169000000000003e-05 2024-09-11T09:50:42.359 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.359 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.359 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:50:42.359 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.519972+0000", 2024-09-11T09:50:42.359 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6595999999999998e-05 2024-09-11T09:50:42.359 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.359 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.359 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:50:42.359 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.520691+0000", 2024-09-11T09:50:42.360 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000719405 2024-09-11T09:50:42.360 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.360 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.360 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:50:42.360 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.520731+0000", 2024-09-11T09:50:42.360 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.0061999999999997e-05 2024-09-11T09:50:42.360 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:42.360 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:50:42.360 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:42.360 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.360 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.360 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4183.0:58 1.0 e13/12 1:30306672:devicehealth::main.db.0000000000000000:head v 13'36, mlcod=13'33)", 2024-09-11T09:50:42.360 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:11.520285+0000", 2024-09-11T09:50:42.361 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 30.802460717999999, 2024-09-11T09:50:42.361 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00091014799999999995, 2024-09-11T09:50:42.361 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:50:42.361 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:50:42.361 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:42.361 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:50:42.361 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.361 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:50:42.361 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.520285+0000", 2024-09-11T09:50:42.361 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:42.361 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.361 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.361 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:50:42.361 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.520285+0000", 2024-09-11T09:50:42.362 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:42.362 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.362 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.362 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:50:42.362 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.520286+0000", 2024-09-11T09:50:42.362 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.4399999999999999e-07 2024-09-11T09:50:42.362 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.362 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.363 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:50:42.363 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.520348+0000", 2024-09-11T09:50:42.363 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.1760000000000005e-05 2024-09-11T09:50:42.363 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.363 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.363 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:50:42.363 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.520359+0000", 2024-09-11T09:50:42.363 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1362999999999999e-05 2024-09-11T09:50:42.363 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.364 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.364 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:50:42.364 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.520364+0000", 2024-09-11T09:50:42.364 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.3600000000000004e-06 2024-09-11T09:50:42.364 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.364 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.364 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:50:42.364 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.520413+0000", 2024-09-11T09:50:42.364 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.8226000000000002e-05 2024-09-11T09:50:42.364 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.364 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.364 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:50:42.364 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.520426+0000", 2024-09-11T09:50:42.365 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3560000000000001e-05 2024-09-11T09:50:42.365 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.365 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.365 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:50:42.365 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.521159+0000", 2024-09-11T09:50:42.365 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00073255499999999997 2024-09-11T09:50:42.365 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.365 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.365 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:50:42.365 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.521195+0000", 2024-09-11T09:50:42.365 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.6480000000000003e-05 2024-09-11T09:50:42.365 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:42.365 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:50:42.366 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:42.366 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.366 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.366 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4183.0:68 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'42, mlcod=13'39)", 2024-09-11T09:50:42.366 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:11.534957+0000", 2024-09-11T09:50:42.366 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 30.787789026999999, 2024-09-11T09:50:42.366 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.001065351, 2024-09-11T09:50:42.366 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:50:42.366 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:50:42.366 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:42.366 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:50:42.366 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.366 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:50:42.366 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.534957+0000", 2024-09-11T09:50:42.367 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:42.367 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.367 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.367 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:50:42.367 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.534957+0000", 2024-09-11T09:50:42.367 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:42.367 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.367 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.367 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:50:42.367 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.534957+0000", 2024-09-11T09:50:42.367 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.3799999999999998e-07 2024-09-11T09:50:42.367 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.367 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.368 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:50:42.368 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.534964+0000", 2024-09-11T09:50:42.368 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.9809999999999997e-06 2024-09-11T09:50:42.368 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.368 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.368 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:50:42.368 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.534976+0000", 2024-09-11T09:50:42.368 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2119999999999999e-05 2024-09-11T09:50:42.368 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.368 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.368 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:50:42.368 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.534980+0000", 2024-09-11T09:50:42.368 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.9890000000000003e-06 2024-09-11T09:50:42.369 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.369 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.369 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:50:42.369 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.535030+0000", 2024-09-11T09:50:42.369 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.9258000000000002e-05 2024-09-11T09:50:42.369 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.369 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.369 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:50:42.369 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.535044+0000", 2024-09-11T09:50:42.369 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3981e-05 2024-09-11T09:50:42.369 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.369 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.369 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:50:42.370 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.535995+0000", 2024-09-11T09:50:42.370 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000951804 2024-09-11T09:50:42.370 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.370 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.370 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:50:42.370 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.536022+0000", 2024-09-11T09:50:42.370 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.6780000000000001e-05 2024-09-11T09:50:42.370 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:42.370 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:50:42.370 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:42.370 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.370 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.370 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4183.0:69 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'43, mlcod=13'39)", 2024-09-11T09:50:42.370 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:11.535331+0000", 2024-09-11T09:50:42.371 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 30.787414704, 2024-09-11T09:50:42.371 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0011256090000000001, 2024-09-11T09:50:42.371 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:50:42.371 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:50:42.371 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:42.371 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:50:42.371 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.371 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:50:42.371 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.535331+0000", 2024-09-11T09:50:42.371 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:42.371 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.371 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.371 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:50:42.371 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.535331+0000", 2024-09-11T09:50:42.372 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:42.372 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.372 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.372 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:50:42.372 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.535332+0000", 2024-09-11T09:50:42.372 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.1600000000000001e-07 2024-09-11T09:50:42.372 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.372 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.372 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:50:42.372 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.535393+0000", 2024-09-11T09:50:42.372 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.1452999999999996e-05 2024-09-11T09:50:42.372 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.372 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.373 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:50:42.373 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.535402+0000", 2024-09-11T09:50:42.373 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.4300000000000006e-06 2024-09-11T09:50:42.373 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.373 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.373 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:50:42.373 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.535405+0000", 2024-09-11T09:50:42.373 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.512e-06 2024-09-11T09:50:42.373 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.373 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.373 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:50:42.373 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.535482+0000", 2024-09-11T09:50:42.373 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.7132999999999995e-05 2024-09-11T09:50:42.374 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.374 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.374 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:50:42.374 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.535506+0000", 2024-09-11T09:50:42.374 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.4070000000000002e-05 2024-09-11T09:50:42.374 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.374 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.374 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:50:42.374 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.536424+0000", 2024-09-11T09:50:42.374 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00091766700000000005 2024-09-11T09:50:42.375 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.375 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.375 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:50:42.375 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.536457+0000", 2024-09-11T09:50:42.375 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.2827999999999998e-05 2024-09-11T09:50:42.375 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:42.375 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:50:42.375 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:42.375 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.375 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.375 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4183.0:83 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'53, mlcod=13'50)", 2024-09-11T09:50:42.375 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:11.558836+0000", 2024-09-11T09:50:42.375 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 30.763909449, 2024-09-11T09:50:42.375 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00090313100000000005, 2024-09-11T09:50:42.376 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:50:42.376 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:50:42.376 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:42.376 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:50:42.376 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.376 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:50:42.376 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.558836+0000", 2024-09-11T09:50:42.376 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:42.376 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.376 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.376 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:50:42.376 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.558836+0000", 2024-09-11T09:50:42.377 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:42.377 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.377 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.377 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:50:42.377 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.558837+0000", 2024-09-11T09:50:42.377 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.4900000000000005e-07 2024-09-11T09:50:42.377 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.377 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.377 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:50:42.377 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.558857+0000", 2024-09-11T09:50:42.377 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.9502999999999999e-05 2024-09-11T09:50:42.377 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.377 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.377 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:50:42.378 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.558870+0000", 2024-09-11T09:50:42.378 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3020000000000001e-05 2024-09-11T09:50:42.378 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.378 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.378 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:50:42.378 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.558876+0000", 2024-09-11T09:50:42.378 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.7270000000000004e-06 2024-09-11T09:50:42.378 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.378 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.378 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:50:42.378 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.558934+0000", 2024-09-11T09:50:42.378 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.8100999999999998e-05 2024-09-11T09:50:42.378 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.378 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.379 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:50:42.379 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.558947+0000", 2024-09-11T09:50:42.379 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3269e-05 2024-09-11T09:50:42.379 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.379 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.379 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:50:42.379 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.559720+0000", 2024-09-11T09:50:42.379 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00077287600000000003 2024-09-11T09:50:42.379 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.379 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.379 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:50:42.379 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.559740+0000", 2024-09-11T09:50:42.379 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.9786e-05 2024-09-11T09:50:42.380 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:42.380 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:50:42.380 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:42.380 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.380 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.380 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4183.0:84 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'54, mlcod=13'50)", 2024-09-11T09:50:42.380 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:11.559267+0000", 2024-09-11T09:50:42.380 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 30.763479021999999, 2024-09-11T09:50:42.380 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00093990199999999995, 2024-09-11T09:50:42.380 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:50:42.380 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:50:42.380 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:42.380 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:50:42.380 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.381 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:50:42.381 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.559267+0000", 2024-09-11T09:50:42.381 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:42.381 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.381 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.381 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:50:42.381 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.559267+0000", 2024-09-11T09:50:42.381 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:42.381 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.381 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.381 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:50:42.381 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.559267+0000", 2024-09-11T09:50:42.381 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.2900000000000003e-07 2024-09-11T09:50:42.381 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.382 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.382 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:50:42.382 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.559322+0000", 2024-09-11T09:50:42.382 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.4429999999999999e-05 2024-09-11T09:50:42.382 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.382 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.382 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:50:42.382 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.559332+0000", 2024-09-11T09:50:42.382 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0473e-05 2024-09-11T09:50:42.383 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.383 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.383 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:50:42.383 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.559338+0000", 2024-09-11T09:50:42.383 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.0730000000000001e-06 2024-09-11T09:50:42.383 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.383 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.383 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:50:42.383 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.559373+0000", 2024-09-11T09:50:42.383 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.4626999999999999e-05 2024-09-11T09:50:42.383 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.383 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.383 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:50:42.383 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.559384+0000", 2024-09-11T09:50:42.384 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1307e-05 2024-09-11T09:50:42.384 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.384 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.384 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:50:42.384 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.560175+0000", 2024-09-11T09:50:42.384 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00079016399999999999 2024-09-11T09:50:42.384 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.384 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.384 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:50:42.384 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.560207+0000", 2024-09-11T09:50:42.384 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.2199000000000001e-05 2024-09-11T09:50:42.384 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:42.384 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:50:42.385 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:42.385 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.385 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.385 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4183.0:89 1.0 e13/12 1:30306672:devicehealth::main.db.0000000000000000:head v 13'59, mlcod=13'57)", 2024-09-11T09:50:42.385 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:11.563614+0000", 2024-09-11T09:50:42.385 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 30.759131402000001, 2024-09-11T09:50:42.385 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00091281399999999996, 2024-09-11T09:50:42.385 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:50:42.385 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:50:42.385 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:42.385 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:50:42.385 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.385 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:50:42.385 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.563614+0000", 2024-09-11T09:50:42.386 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:42.386 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.386 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.386 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:50:42.386 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.563614+0000", 2024-09-11T09:50:42.386 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:42.386 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.386 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.386 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:50:42.386 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.563615+0000", 2024-09-11T09:50:42.386 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.6899999999999998e-07 2024-09-11T09:50:42.386 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.386 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.387 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:50:42.387 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.563681+0000", 2024-09-11T09:50:42.387 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.5769e-05 2024-09-11T09:50:42.387 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.387 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.387 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:50:42.387 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.563690+0000", 2024-09-11T09:50:42.387 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.6800000000000005e-06 2024-09-11T09:50:42.387 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.387 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.387 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:50:42.387 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.563699+0000", 2024-09-11T09:50:42.387 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.3170000000000002e-06 2024-09-11T09:50:42.387 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.388 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.388 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:50:42.388 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.563734+0000", 2024-09-11T09:50:42.388 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.5023e-05 2024-09-11T09:50:42.388 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.388 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.388 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:50:42.388 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.563746+0000", 2024-09-11T09:50:42.388 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2661e-05 2024-09-11T09:50:42.388 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.388 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.388 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:50:42.388 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.564500+0000", 2024-09-11T09:50:42.389 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00075323100000000004 2024-09-11T09:50:42.389 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.389 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.389 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:50:42.389 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.564527+0000", 2024-09-11T09:50:42.389 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.7664000000000001e-05 2024-09-11T09:50:42.389 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:42.389 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:50:42.389 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:42.389 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.389 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.389 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4183.0:105 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'67, mlcod=13'64)", 2024-09-11T09:50:42.389 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:11.712146+0000", 2024-09-11T09:50:42.389 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 30.610599652000001, 2024-09-11T09:50:42.390 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0010916859999999999, 2024-09-11T09:50:42.390 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:50:42.390 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:50:42.390 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:42.390 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:50:42.390 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.390 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:50:42.390 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.712146+0000", 2024-09-11T09:50:42.390 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:42.390 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.390 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.390 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:50:42.390 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.712146+0000", 2024-09-11T09:50:42.390 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:42.391 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.391 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.391 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:50:42.391 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.712147+0000", 2024-09-11T09:50:42.391 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.8200000000000001e-07 2024-09-11T09:50:42.391 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.391 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.391 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:50:42.391 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.712155+0000", 2024-09-11T09:50:42.391 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.0309999999999998e-06 2024-09-11T09:50:42.391 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.391 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.391 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:50:42.392 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.712163+0000", 2024-09-11T09:50:42.392 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.5920000000000001e-06 2024-09-11T09:50:42.392 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.392 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.392 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:50:42.392 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.712169+0000", 2024-09-11T09:50:42.392 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.4299999999999997e-06 2024-09-11T09:50:42.392 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.392 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.392 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:50:42.392 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.712418+0000", 2024-09-11T09:50:42.392 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00024978499999999998 2024-09-11T09:50:42.392 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.393 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.393 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:50:42.393 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.712432+0000", 2024-09-11T09:50:42.393 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3190000000000001e-05 2024-09-11T09:50:42.393 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.393 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.393 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:50:42.393 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.713206+0000", 2024-09-11T09:50:42.393 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00077420199999999998 2024-09-11T09:50:42.393 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.393 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.393 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:50:42.393 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.713238+0000", 2024-09-11T09:50:42.393 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.2073999999999998e-05 2024-09-11T09:50:42.394 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:42.394 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:50:42.394 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:42.394 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.394 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.394 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4183.0:106 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'68, mlcod=13'64)", 2024-09-11T09:50:42.394 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:11.712624+0000", 2024-09-11T09:50:42.394 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 30.610122323999999, 2024-09-11T09:50:42.394 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0010009629999999999, 2024-09-11T09:50:42.394 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:50:42.394 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:50:42.394 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:42.394 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:50:42.394 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.395 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:50:42.395 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.712624+0000", 2024-09-11T09:50:42.395 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:42.395 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.395 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.395 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:50:42.395 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.712624+0000", 2024-09-11T09:50:42.395 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:42.395 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.395 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.395 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:50:42.395 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.712624+0000", 2024-09-11T09:50:42.395 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.4799999999999998e-07 2024-09-11T09:50:42.396 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.396 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.396 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:50:42.396 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.712689+0000", 2024-09-11T09:50:42.396 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.4488999999999999e-05 2024-09-11T09:50:42.396 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.396 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.396 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:50:42.396 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.712707+0000", 2024-09-11T09:50:42.396 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.8272000000000001e-05 2024-09-11T09:50:42.396 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.396 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.396 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:50:42.397 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.712713+0000", 2024-09-11T09:50:42.397 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.2040000000000002e-06 2024-09-11T09:50:42.397 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.397 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.397 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:50:42.397 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.712783+0000", 2024-09-11T09:50:42.397 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.9534000000000002e-05 2024-09-11T09:50:42.397 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.397 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.398 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:50:42.398 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.712811+0000", 2024-09-11T09:50:42.398 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.8620999999999999e-05 2024-09-11T09:50:42.398 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.398 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.398 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:50:42.398 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.713586+0000", 2024-09-11T09:50:42.398 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00077462600000000005 2024-09-11T09:50:42.398 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.398 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.398 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:50:42.398 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.713624+0000", 2024-09-11T09:50:42.398 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.8668999999999999e-05 2024-09-11T09:50:42.398 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:42.399 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:50:42.399 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:42.399 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.399 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.399 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4183.0:118 1.0 e13/12 1:30306672:devicehealth::main.db.0000000000000000:head v 13'80, mlcod=13'78)", 2024-09-11T09:50:42.399 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:11.726126+0000", 2024-09-11T09:50:42.399 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 30.596620231999999, 2024-09-11T09:50:42.399 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00087776200000000003, 2024-09-11T09:50:42.399 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:50:42.399 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:50:42.399 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:42.399 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:50:42.399 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.400 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:50:42.400 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.726126+0000", 2024-09-11T09:50:42.400 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:42.400 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.400 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.400 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:50:42.400 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.726126+0000", 2024-09-11T09:50:42.400 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:42.400 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.400 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.400 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:50:42.400 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.726126+0000", 2024-09-11T09:50:42.400 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.2099999999999996e-07 2024-09-11T09:50:42.401 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.401 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.401 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:50:42.401 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.726167+0000", 2024-09-11T09:50:42.401 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.0621999999999998e-05 2024-09-11T09:50:42.401 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.401 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.401 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:50:42.401 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.726177+0000", 2024-09-11T09:50:42.401 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0299000000000001e-05 2024-09-11T09:50:42.401 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.401 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.401 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:50:42.401 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.726183+0000", 2024-09-11T09:50:42.402 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.2079999999999997e-06 2024-09-11T09:50:42.402 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.402 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.402 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:50:42.402 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.726256+0000", 2024-09-11T09:50:42.402 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.2632000000000005e-05 2024-09-11T09:50:42.402 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.402 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.402 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:50:42.402 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.726268+0000", 2024-09-11T09:50:42.402 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2461e-05 2024-09-11T09:50:42.402 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.402 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.402 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:50:42.403 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.726977+0000", 2024-09-11T09:50:42.403 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00070853500000000005 2024-09-11T09:50:42.403 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.403 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.403 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:50:42.403 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.727003+0000", 2024-09-11T09:50:42.403 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.6384e-05 2024-09-11T09:50:42.403 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:42.403 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:50:42.403 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:42.403 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.403 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.403 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4183.0:134 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'92, mlcod=13'86)", 2024-09-11T09:50:42.404 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:12.035698+0000", 2024-09-11T09:50:42.404 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 30.287048065, 2024-09-11T09:50:42.404 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00089550700000000003, 2024-09-11T09:50:42.404 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:50:42.404 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:50:42.404 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:42.404 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:50:42.404 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.404 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:50:42.404 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.035698+0000", 2024-09-11T09:50:42.404 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:42.404 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.404 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.404 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:50:42.405 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.035698+0000", 2024-09-11T09:50:42.405 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:42.405 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.405 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.405 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:50:42.405 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.035698+0000", 2024-09-11T09:50:42.405 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.0900000000000002e-07 2024-09-11T09:50:42.405 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.405 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.405 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:50:42.405 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.035758+0000", 2024-09-11T09:50:42.405 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.9481999999999997e-05 2024-09-11T09:50:42.405 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.406 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.406 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:50:42.406 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.035766+0000", 2024-09-11T09:50:42.406 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.3760000000000008e-06 2024-09-11T09:50:42.406 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.406 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.406 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:50:42.406 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.035772+0000", 2024-09-11T09:50:42.406 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.8930000000000002e-06 2024-09-11T09:50:42.406 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.406 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.406 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:50:42.406 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.035807+0000", 2024-09-11T09:50:42.406 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.4975999999999999e-05 2024-09-11T09:50:42.407 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.407 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.407 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:50:42.407 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.035817+0000", 2024-09-11T09:50:42.407 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0314e-05 2024-09-11T09:50:42.407 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.407 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.407 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:50:42.407 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.036542+0000", 2024-09-11T09:50:42.407 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00072434400000000003 2024-09-11T09:50:42.407 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.407 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.407 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:50:42.408 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.036593+0000", 2024-09-11T09:50:42.408 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.1613000000000002e-05 2024-09-11T09:50:42.408 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:42.408 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:50:42.408 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:42.408 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.408 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.408 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4183.0:136 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'94, mlcod=13'86)", 2024-09-11T09:50:42.408 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:12.036591+0000", 2024-09-11T09:50:42.408 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 30.286154854999999, 2024-09-11T09:50:42.408 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00090637900000000002, 2024-09-11T09:50:42.408 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:50:42.408 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:50:42.408 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:42.409 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:50:42.409 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.409 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:50:42.409 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.036591+0000", 2024-09-11T09:50:42.409 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:42.409 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.409 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.409 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:50:42.409 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.036591+0000", 2024-09-11T09:50:42.409 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:42.409 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.409 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.409 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:50:42.409 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.036592+0000", 2024-09-11T09:50:42.410 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.37e-07 2024-09-11T09:50:42.410 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.410 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.410 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:50:42.410 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.036600+0000", 2024-09-11T09:50:42.410 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.6910000000000003e-06 2024-09-11T09:50:42.410 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.410 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.410 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:50:42.410 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.036609+0000", 2024-09-11T09:50:42.410 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.0229999999999998e-06 2024-09-11T09:50:42.410 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.410 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.411 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:50:42.411 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.036617+0000", 2024-09-11T09:50:42.411 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.6979999999999998e-06 2024-09-11T09:50:42.411 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.411 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.411 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:50:42.411 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.036658+0000", 2024-09-11T09:50:42.411 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.0692000000000003e-05 2024-09-11T09:50:42.411 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.411 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.411 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:50:42.411 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.036670+0000", 2024-09-11T09:50:42.411 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2109e-05 2024-09-11T09:50:42.411 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.412 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.412 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:50:42.412 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.037475+0000", 2024-09-11T09:50:42.412 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00080487199999999999 2024-09-11T09:50:42.412 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.412 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.412 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:50:42.412 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.037497+0000", 2024-09-11T09:50:42.412 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.2557000000000001e-05 2024-09-11T09:50:42.412 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:42.412 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:50:42.412 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:42.412 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.413 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.413 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4183.0:158 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'112, mlcod=13'108)", 2024-09-11T09:50:42.413 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:12.455073+0000", 2024-09-11T09:50:42.413 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 29.867672601999999, 2024-09-11T09:50:42.413 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00090654500000000003, 2024-09-11T09:50:42.413 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:50:42.413 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:50:42.413 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:42.413 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:50:42.413 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.413 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:50:42.413 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.455073+0000", 2024-09-11T09:50:42.413 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:42.414 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.414 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.414 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:50:42.414 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.455073+0000", 2024-09-11T09:50:42.414 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:42.414 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.414 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.414 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:50:42.414 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.455074+0000", 2024-09-11T09:50:42.414 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.32e-07 2024-09-11T09:50:42.414 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.414 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.414 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:50:42.414 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.455126+0000", 2024-09-11T09:50:42.415 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.1898999999999999e-05 2024-09-11T09:50:42.415 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.415 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.415 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:50:42.415 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.455136+0000", 2024-09-11T09:50:42.415 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0651e-05 2024-09-11T09:50:42.415 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.415 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.415 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:50:42.415 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.455142+0000", 2024-09-11T09:50:42.415 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.0020000000000003e-06 2024-09-11T09:50:42.415 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.415 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.416 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:50:42.416 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.455192+0000", 2024-09-11T09:50:42.416 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.0164000000000003e-05 2024-09-11T09:50:42.416 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.416 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.416 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:50:42.416 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.455204+0000", 2024-09-11T09:50:42.416 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1178999999999999e-05 2024-09-11T09:50:42.416 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.416 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.416 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:50:42.416 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.455957+0000", 2024-09-11T09:50:42.417 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00075362300000000003 2024-09-11T09:50:42.417 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.417 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.417 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:50:42.417 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.455980+0000", 2024-09-11T09:50:42.417 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.2595e-05 2024-09-11T09:50:42.417 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:42.417 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:50:42.417 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:42.417 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:42.417 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:50:42.417 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:50:42.419 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-11T09:50:42.419 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:50:42.419+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:50:42.519 INFO:tasks.ceph.osd.2.smithi017.stderr:2024-09-11T09:50:42.520+0000 7fd20b170640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31672) UID: 0 2024-09-11T09:50:42.567 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:50:42.568 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [], 2024-09-11T09:50:42.568 INFO:teuthology.orchestra.run.smithi017.stdout: "num_ops": 0 2024-09-11T09:50:42.568 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:50:42.571 INFO:teuthology.orchestra.run.smithi017.stderr:nodeep-scrub is set 2024-09-11T09:50:42.571 INFO:teuthology.orchestra.run.smithi017.stderr:reweighted osd.4 to 1 (10000) 2024-09-11T09:50:42.580 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-11T09:50:42.587 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 5 1 2024-09-11T09:50:42.629 INFO:tasks.ceph.osd.2.smithi017.stderr:2024-09-11T09:50:42.630+0000 7fd20b170640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31672) UID: 0 2024-09-11T09:50:42.722 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-09-11T09:50:42.723+0000 7f7b6c7d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31904) UID: 0 2024-09-11T09:50:42.736 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:50:42.736 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [], 2024-09-11T09:50:42.736 INFO:teuthology.orchestra.run.smithi017.stdout: "complaint_time": 30, 2024-09-11T09:50:42.736 INFO:teuthology.orchestra.run.smithi017.stdout: "num_blocked_ops": 0 2024-09-11T09:50:42.736 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:50:42.746 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-11T09:50:42.822 INFO:tasks.ceph.osd.2.smithi017.stderr:2024-09-11T09:50:42.823+0000 7fd20b170640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31672) UID: 0 2024-09-11T09:50:42.856 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:50:42.856 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 20, 2024-09-11T09:50:42.857 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 600, 2024-09-11T09:50:42.857 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [ 2024-09-11T09:50:42.857 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.857 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4230.0:5 2.2 e17/15 2:5cea7035:::rbd_info:head v 17'1, mlcod=0'0)", 2024-09-11T09:50:42.857 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:19.494757+0000", 2024-09-11T09:50:42.857 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 23.360308455999999, 2024-09-11T09:50:42.857 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00082098399999999997, 2024-09-11T09:50:42.857 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:50:42.857 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:50:42.857 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:42.857 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:50:42.857 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.858 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:50:42.858 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:19.494757+0000", 2024-09-11T09:50:42.858 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:42.858 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.858 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.858 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:50:42.858 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:19.494757+0000", 2024-09-11T09:50:42.858 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:42.858 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.858 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.858 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:50:42.858 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:19.494757+0000", 2024-09-11T09:50:42.858 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.97e-07 2024-09-11T09:50:42.858 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.859 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.859 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:50:42.859 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:19.494763+0000", 2024-09-11T09:50:42.859 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.3709999999999996e-06 2024-09-11T09:50:42.859 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.859 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.859 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:50:42.859 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:19.494778+0000", 2024-09-11T09:50:42.859 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.4960000000000001e-05 2024-09-11T09:50:42.859 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.859 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.859 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:50:42.859 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:19.494801+0000", 2024-09-11T09:50:42.860 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.2388999999999999e-05 2024-09-11T09:50:42.860 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.860 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.860 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:50:42.860 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:19.494859+0000", 2024-09-11T09:50:42.860 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.7961999999999998e-05 2024-09-11T09:50:42.860 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.860 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.860 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:50:42.860 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:19.494883+0000", 2024-09-11T09:50:42.860 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.4488999999999999e-05 2024-09-11T09:50:42.860 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.860 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.861 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:50:42.861 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:19.495534+0000", 2024-09-11T09:50:42.861 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00065075500000000004 2024-09-11T09:50:42.861 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.861 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.861 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:50:42.861 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:19.495578+0000", 2024-09-11T09:50:42.861 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.3860999999999999e-05 2024-09-11T09:50:42.861 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:42.861 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:50:42.861 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:42.862 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.862 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.862 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4230.0:7 2.2 e18/15 2:5cea7035:::rbd_info:head v 18'2, mlcod=0'0)", 2024-09-11T09:50:42.862 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:20.500782+0000", 2024-09-11T09:50:42.862 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 22.354283576, 2024-09-11T09:50:42.862 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000790185, 2024-09-11T09:50:42.862 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:50:42.863 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:50:42.863 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:42.863 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:50:42.863 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.863 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:50:42.863 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:20.500782+0000", 2024-09-11T09:50:42.863 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:42.863 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.863 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.863 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:50:42.863 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:20.500782+0000", 2024-09-11T09:50:42.863 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:42.863 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.864 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.864 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:50:42.864 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:20.500782+0000", 2024-09-11T09:50:42.864 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.3299999999999998e-07 2024-09-11T09:50:42.864 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.864 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.864 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:50:42.865 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:20.500799+0000", 2024-09-11T09:50:42.865 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6830999999999999e-05 2024-09-11T09:50:42.865 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.865 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.865 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:50:42.865 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:20.500812+0000", 2024-09-11T09:50:42.865 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3013999999999999e-05 2024-09-11T09:50:42.865 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.865 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.865 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:50:42.865 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:20.500824+0000", 2024-09-11T09:50:42.865 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2593e-05 2024-09-11T09:50:42.865 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.866 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.866 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:50:42.866 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:20.500895+0000", 2024-09-11T09:50:42.866 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.1018000000000003e-05 2024-09-11T09:50:42.866 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.866 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.866 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:50:42.866 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:20.500912+0000", 2024-09-11T09:50:42.866 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6484e-05 2024-09-11T09:50:42.866 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.866 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.866 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:50:42.866 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:20.501531+0000", 2024-09-11T09:50:42.866 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00061903399999999997 2024-09-11T09:50:42.867 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:42.867 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:42.867 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:50:42.867 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:20.501572+0000", 2024-09-11T09:50:42.867 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.0877999999999997e-05 2024-09-11T09:50:42.867 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:42.867 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:50:42.867 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:42.867 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:42.867 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:50:42.867 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:50:42.868 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-11T09:50:42.924 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-09-11T09:50:42.925+0000 7f7b6c7d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31904) UID: 0 2024-09-11T09:50:43.015 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:50:43.015 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [], 2024-09-11T09:50:43.015 INFO:teuthology.orchestra.run.smithi017.stdout: "num_ops": 0 2024-09-11T09:50:43.015 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:50:43.024 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-11T09:50:43.026 INFO:tasks.ceph.osd.2.smithi017.stderr:2024-09-11T09:50:43.026+0000 7fd20b170640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31672) UID: 0 2024-09-11T09:50:43.127 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:50:43.128+0000 7fdec5849640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31907) UID: 0 2024-09-11T09:50:43.131 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:50:43.131 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [], 2024-09-11T09:50:43.132 INFO:teuthology.orchestra.run.smithi017.stdout: "complaint_time": 30, 2024-09-11T09:50:43.132 INFO:teuthology.orchestra.run.smithi017.stdout: "num_blocked_ops": 0 2024-09-11T09:50:43.132 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:50:43.141 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-11T09:50:43.228 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:50:43.229+0000 7f3c96ec9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31909) UID: 0 2024-09-11T09:50:43.248 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:50:43.248 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 20, 2024-09-11T09:50:43.248 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 600, 2024-09-11T09:50:43.249 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [ 2024-09-11T09:50:43.249 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:43.249 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4230.0:1 2.3 e16/15 2:c4c92e5a:::rbd_trash:head v 16'1, mlcod=0'0)", 2024-09-11T09:50:43.249 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:18.494378+0000", 2024-09-11T09:50:43.249 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 24.755552609999999, 2024-09-11T09:50:43.249 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00089752199999999999, 2024-09-11T09:50:43.249 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:50:43.249 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:50:43.249 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:43.249 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:50:43.249 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:43.249 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:50:43.249 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:18.494378+0000", 2024-09-11T09:50:43.250 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:43.250 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:43.250 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:43.250 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:50:43.250 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:18.494378+0000", 2024-09-11T09:50:43.250 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:43.250 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:43.250 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:43.250 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:50:43.250 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:18.494378+0000", 2024-09-11T09:50:43.250 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.9099999999999999e-07 2024-09-11T09:50:43.250 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:43.250 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:43.251 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:50:43.251 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:18.494385+0000", 2024-09-11T09:50:43.251 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.8129999999999998e-06 2024-09-11T09:50:43.251 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:43.251 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:43.251 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:50:43.251 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:18.494416+0000", 2024-09-11T09:50:43.251 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.0654000000000002e-05 2024-09-11T09:50:43.251 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:43.251 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:43.251 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:50:43.251 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:18.494430+0000", 2024-09-11T09:50:43.251 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3754000000000001e-05 2024-09-11T09:50:43.251 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:43.252 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:43.252 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:50:43.252 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:18.494524+0000", 2024-09-11T09:50:43.252 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.4896999999999996e-05 2024-09-11T09:50:43.252 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:43.252 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:43.252 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:50:43.252 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:18.494558+0000", 2024-09-11T09:50:43.252 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.3900999999999999e-05 2024-09-11T09:50:43.252 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:43.252 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:43.252 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:50:43.252 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:18.495231+0000", 2024-09-11T09:50:43.253 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00067233500000000003 2024-09-11T09:50:43.253 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:43.253 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:43.253 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:50:43.253 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:18.495275+0000", 2024-09-11T09:50:43.253 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.4777000000000002e-05 2024-09-11T09:50:43.253 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:43.253 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:50:43.253 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:43.253 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:43.253 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:50:43.253 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:50:43.257 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-11T09:50:43.328 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:50:43.329+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:50:43.372 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-11T09:50:43.372 INFO:teuthology.orchestra.run.smithi096.stdout: "ops": [], 2024-09-11T09:50:43.372 INFO:teuthology.orchestra.run.smithi096.stdout: "num_ops": 0 2024-09-11T09:50:43.372 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-11T09:50:43.381 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-11T09:50:43.429 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:50:43.430+0000 7fdec5849640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31907) UID: 0 2024-09-11T09:50:43.484 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-11T09:50:43.485 INFO:teuthology.orchestra.run.smithi096.stdout: "ops": [], 2024-09-11T09:50:43.485 INFO:teuthology.orchestra.run.smithi096.stdout: "complaint_time": 30, 2024-09-11T09:50:43.485 INFO:teuthology.orchestra.run.smithi096.stdout: "num_blocked_ops": 0 2024-09-11T09:50:43.485 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-11T09:50:43.493 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-11T09:50:43.534 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-09-11T09:50:43.535+0000 7f7b6c7d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31904) UID: 0 2024-09-11T09:50:43.575 INFO:teuthology.orchestra.run.smithi017.stderr:reweighted osd.5 to 1 (10000) 2024-09-11T09:50:43.590 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-09-11T09:50:43.591 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T09:50:43.631 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:50:43.632+0000 7fdec5849640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31907) UID: 0 2024-09-11T09:50:43.638 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-11T09:50:43.639 INFO:teuthology.orchestra.run.smithi096.stdout: "size": 20, 2024-09-11T09:50:43.639 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 600, 2024-09-11T09:50:43.639 INFO:teuthology.orchestra.run.smithi096.stdout: "ops": [ 2024-09-11T09:50:43.639 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.639 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:74 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:50:43.639 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.539067+0000", 2024-09-11T09:50:43.639 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 32.099354232000003, 2024-09-11T09:50:43.639 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.006945957, 2024-09-11T09:50:43.639 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:50:43.639 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:50:43.639 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:43.639 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:50:43.639 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:50:43.640 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:50:43.640 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 74 2024-09-11T09:50:43.640 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.640 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:50:43.640 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.640 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:50:43.640 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.539067+0000", 2024-09-11T09:50:43.640 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:43.640 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.640 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.640 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:50:43.640 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.539067+0000", 2024-09-11T09:50:43.640 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:43.641 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.641 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.641 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:50:43.641 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.539068+0000", 2024-09-11T09:50:43.641 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.6000000000000003e-07 2024-09-11T09:50:43.641 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.641 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.641 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:50:43.641 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.539072+0000", 2024-09-11T09:50:43.641 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.0450000000000002e-06 2024-09-11T09:50:43.641 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.641 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.641 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:50:43.641 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.539078+0000", 2024-09-11T09:50:43.642 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.6889999999999998e-06 2024-09-11T09:50:43.642 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.642 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.642 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:50:43.642 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.539083+0000", 2024-09-11T09:50:43.642 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.7180000000000004e-06 2024-09-11T09:50:43.642 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.642 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.642 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:50:43.642 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.540230+0000", 2024-09-11T09:50:43.642 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0011466250000000001 2024-09-11T09:50:43.642 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.642 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.642 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:50:43.642 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.540279+0000", 2024-09-11T09:50:43.643 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.9667999999999999e-05 2024-09-11T09:50:43.643 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.643 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.643 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:50:43.643 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.540357+0000", 2024-09-11T09:50:43.643 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.7863999999999999e-05 2024-09-11T09:50:43.643 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.643 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.643 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:50:43.643 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.541086+0000", 2024-09-11T09:50:43.643 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00072921100000000001 2024-09-11T09:50:43.643 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.643 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.643 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:50:43.644 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.545949+0000", 2024-09-11T09:50:43.644 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0048626040000000004 2024-09-11T09:50:43.644 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.644 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.644 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:50:43.644 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.545989+0000", 2024-09-11T09:50:43.644 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.0259000000000002e-05 2024-09-11T09:50:43.644 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.644 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.644 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:50:43.644 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.546013+0000", 2024-09-11T09:50:43.644 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.3614000000000001e-05 2024-09-11T09:50:43.644 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:43.644 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:50:43.645 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:43.645 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.645 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.645 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:108 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:50:43.645 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.711510+0000", 2024-09-11T09:50:43.645 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 31.926911059999998, 2024-09-11T09:50:43.645 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0054615649999999998, 2024-09-11T09:50:43.645 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:50:43.645 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:50:43.645 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:43.645 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:50:43.645 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:50:43.645 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:50:43.645 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 108 2024-09-11T09:50:43.646 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.646 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:50:43.646 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.646 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:50:43.646 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711510+0000", 2024-09-11T09:50:43.646 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:43.646 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.646 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.646 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:50:43.646 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711510+0000", 2024-09-11T09:50:43.646 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:43.646 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.646 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.647 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:50:43.647 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711511+0000", 2024-09-11T09:50:43.647 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.1999999999999998e-07 2024-09-11T09:50:43.647 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.647 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.647 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:50:43.647 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711516+0000", 2024-09-11T09:50:43.647 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.4140000000000002e-06 2024-09-11T09:50:43.647 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.647 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.647 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:50:43.647 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711527+0000", 2024-09-11T09:50:43.647 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.0169e-05 2024-09-11T09:50:43.647 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.648 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.648 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:50:43.648 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711535+0000", 2024-09-11T09:50:43.648 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.3299999999999999e-06 2024-09-11T09:50:43.648 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.648 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.648 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:50:43.648 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.713763+0000", 2024-09-11T09:50:43.648 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0022281079999999999 2024-09-11T09:50:43.648 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.648 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.648 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:50:43.648 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.713812+0000", 2024-09-11T09:50:43.648 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.8544000000000001e-05 2024-09-11T09:50:43.649 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.649 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.649 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:50:43.649 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.713909+0000", 2024-09-11T09:50:43.649 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.7113000000000003e-05 2024-09-11T09:50:43.649 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.649 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.649 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:50:43.649 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.716873+0000", 2024-09-11T09:50:43.649 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0029642710000000001 2024-09-11T09:50:43.649 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.649 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.649 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:50:43.650 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.716918+0000", 2024-09-11T09:50:43.650 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.5447e-05 2024-09-11T09:50:43.650 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.650 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.650 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:50:43.650 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.716947+0000", 2024-09-11T09:50:43.650 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.8181e-05 2024-09-11T09:50:43.650 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.650 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.650 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:50:43.650 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.716972+0000", 2024-09-11T09:50:43.650 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.5068000000000001e-05 2024-09-11T09:50:43.650 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:43.650 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:50:43.650 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:43.651 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.651 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.651 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:109 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:50:43.651 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.711555+0000", 2024-09-11T09:50:43.651 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 31.926865810999999, 2024-09-11T09:50:43.651 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.005479902, 2024-09-11T09:50:43.651 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:50:43.651 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:50:43.651 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:43.651 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:50:43.651 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:50:43.651 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:50:43.651 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 109 2024-09-11T09:50:43.651 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.652 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:50:43.652 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.652 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:50:43.652 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711555+0000", 2024-09-11T09:50:43.652 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:43.652 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.652 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.652 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:50:43.652 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711555+0000", 2024-09-11T09:50:43.652 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:43.652 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.652 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.652 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:50:43.653 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711556+0000", 2024-09-11T09:50:43.653 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.3200000000000005e-07 2024-09-11T09:50:43.653 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.653 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.653 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:50:43.653 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711601+0000", 2024-09-11T09:50:43.653 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.5349000000000002e-05 2024-09-11T09:50:43.653 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.653 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.653 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:50:43.653 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711612+0000", 2024-09-11T09:50:43.653 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.0701e-05 2024-09-11T09:50:43.653 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.653 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.654 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:50:43.654 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711619+0000", 2024-09-11T09:50:43.654 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.7610000000000002e-06 2024-09-11T09:50:43.654 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.654 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.654 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:50:43.654 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.714371+0000", 2024-09-11T09:50:43.654 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0027526260000000002 2024-09-11T09:50:43.654 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.654 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.654 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:50:43.654 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.714417+0000", 2024-09-11T09:50:43.654 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.5893999999999998e-05 2024-09-11T09:50:43.654 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.655 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.655 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:50:43.655 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.714496+0000", 2024-09-11T09:50:43.655 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.8539000000000001e-05 2024-09-11T09:50:43.655 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.655 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.655 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:50:43.655 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.716910+0000", 2024-09-11T09:50:43.655 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0024137640000000001 2024-09-11T09:50:43.655 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.655 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.655 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:50:43.655 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.716989+0000", 2024-09-11T09:50:43.655 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.9198999999999998e-05 2024-09-11T09:50:43.656 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.656 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.656 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:50:43.656 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.717016+0000", 2024-09-11T09:50:43.656 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.6699e-05 2024-09-11T09:50:43.656 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.656 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.656 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:50:43.656 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.717035+0000", 2024-09-11T09:50:43.656 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.9738e-05 2024-09-11T09:50:43.656 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:43.656 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:50:43.656 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:43.656 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.657 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.657 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:110 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:50:43.657 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.711636+0000", 2024-09-11T09:50:43.657 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 31.926785350999999, 2024-09-11T09:50:43.657 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0067613070000000003, 2024-09-11T09:50:43.657 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:50:43.657 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:50:43.657 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:43.657 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:50:43.657 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:50:43.657 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:50:43.657 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 110 2024-09-11T09:50:43.657 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.658 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:50:43.658 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.658 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:50:43.658 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711636+0000", 2024-09-11T09:50:43.658 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:43.658 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.658 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.658 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:50:43.658 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711636+0000", 2024-09-11T09:50:43.658 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:43.658 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.658 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.658 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:50:43.658 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711637+0000", 2024-09-11T09:50:43.658 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.9999999999999997e-07 2024-09-11T09:50:43.659 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.659 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.659 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:50:43.659 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711641+0000", 2024-09-11T09:50:43.659 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.2570000000000001e-06 2024-09-11T09:50:43.659 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.659 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.659 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:50:43.659 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711651+0000", 2024-09-11T09:50:43.659 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.0285999999999999e-05 2024-09-11T09:50:43.659 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.659 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.659 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:50:43.659 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711658+0000", 2024-09-11T09:50:43.660 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.3540000000000002e-06 2024-09-11T09:50:43.660 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.660 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.660 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:50:43.660 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.716475+0000", 2024-09-11T09:50:43.660 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0048162819999999999 2024-09-11T09:50:43.660 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.660 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.660 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:50:43.660 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.716526+0000", 2024-09-11T09:50:43.660 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.0905000000000001e-05 2024-09-11T09:50:43.660 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.660 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.661 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:50:43.661 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.716601+0000", 2024-09-11T09:50:43.661 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.5283000000000004e-05 2024-09-11T09:50:43.661 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.661 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.661 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:50:43.661 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.717933+0000", 2024-09-11T09:50:43.661 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001331622 2024-09-11T09:50:43.661 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.661 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.661 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:50:43.661 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718332+0000", 2024-09-11T09:50:43.661 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00039909899999999998 2024-09-11T09:50:43.661 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.662 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.662 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:50:43.662 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718363+0000", 2024-09-11T09:50:43.662 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.1636e-05 2024-09-11T09:50:43.662 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.662 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.662 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:50:43.662 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718397+0000", 2024-09-11T09:50:43.662 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.3883000000000001e-05 2024-09-11T09:50:43.662 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:43.662 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:50:43.662 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:43.662 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.662 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.662 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:111 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:50:43.663 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.711668+0000", 2024-09-11T09:50:43.663 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 31.926752920999999, 2024-09-11T09:50:43.663 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0077145570000000004, 2024-09-11T09:50:43.663 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:50:43.663 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:50:43.663 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:43.663 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:50:43.663 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:50:43.663 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:50:43.663 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 111 2024-09-11T09:50:43.663 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.663 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:50:43.663 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.663 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:50:43.664 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711668+0000", 2024-09-11T09:50:43.664 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:43.664 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.664 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.664 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:50:43.664 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711668+0000", 2024-09-11T09:50:43.664 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:43.664 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.664 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.664 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:50:43.664 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711669+0000", 2024-09-11T09:50:43.664 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.99e-07 2024-09-11T09:50:43.664 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.664 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.665 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:50:43.665 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711673+0000", 2024-09-11T09:50:43.665 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.7440000000000001e-06 2024-09-11T09:50:43.665 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.665 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.665 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:50:43.665 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711686+0000", 2024-09-11T09:50:43.665 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.3317e-05 2024-09-11T09:50:43.665 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.665 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.665 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:50:43.665 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711697+0000", 2024-09-11T09:50:43.665 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1022000000000001e-05 2024-09-11T09:50:43.665 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.666 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.666 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:50:43.666 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.717065+0000", 2024-09-11T09:50:43.666 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.005367531 2024-09-11T09:50:43.666 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.666 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.666 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:50:43.666 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.717114+0000", 2024-09-11T09:50:43.666 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.9707999999999999e-05 2024-09-11T09:50:43.666 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.666 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.666 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:50:43.666 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.717190+0000", 2024-09-11T09:50:43.666 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.5394000000000004e-05 2024-09-11T09:50:43.667 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.667 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.667 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:50:43.667 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718793+0000", 2024-09-11T09:50:43.667 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001603653 2024-09-11T09:50:43.667 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.667 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.667 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:50:43.667 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719329+0000", 2024-09-11T09:50:43.667 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00053543799999999995 2024-09-11T09:50:43.667 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.667 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.667 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:50:43.667 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719359+0000", 2024-09-11T09:50:43.668 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.0420999999999999e-05 2024-09-11T09:50:43.668 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.668 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.668 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:50:43.668 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719383+0000", 2024-09-11T09:50:43.668 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.3730000000000001e-05 2024-09-11T09:50:43.668 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:43.668 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:50:43.668 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:43.668 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.668 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.668 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:112 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:50:43.668 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.711712+0000", 2024-09-11T09:50:43.669 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 31.926709361, 2024-09-11T09:50:43.669 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.008162084, 2024-09-11T09:50:43.669 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:50:43.672 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:50:43.672 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:43.672 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:50:43.672 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:50:43.672 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:50:43.672 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 112 2024-09-11T09:50:43.672 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.672 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:50:43.673 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.673 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:50:43.673 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711712+0000", 2024-09-11T09:50:43.673 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:43.673 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.673 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.673 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:50:43.673 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711712+0000", 2024-09-11T09:50:43.673 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:43.673 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.673 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.673 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:50:43.673 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711712+0000", 2024-09-11T09:50:43.674 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.6000000000000004e-07 2024-09-11T09:50:43.674 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.674 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.674 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:50:43.674 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711808+0000", 2024-09-11T09:50:43.674 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.5977999999999995e-05 2024-09-11T09:50:43.674 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.674 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.674 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:50:43.674 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711819+0000", 2024-09-11T09:50:43.674 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.0755e-05 2024-09-11T09:50:43.674 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.674 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.674 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:50:43.675 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711826+0000", 2024-09-11T09:50:43.675 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.7970000000000003e-06 2024-09-11T09:50:43.675 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.675 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.675 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:50:43.675 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.717435+0000", 2024-09-11T09:50:43.675 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0056090239999999998 2024-09-11T09:50:43.675 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.675 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.675 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:50:43.675 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.717496+0000", 2024-09-11T09:50:43.675 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.0543e-05 2024-09-11T09:50:43.675 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.675 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.676 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:50:43.676 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.717591+0000", 2024-09-11T09:50:43.676 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.5667000000000004e-05 2024-09-11T09:50:43.676 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.676 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.676 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:50:43.676 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718831+0000", 2024-09-11T09:50:43.676 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001240204 2024-09-11T09:50:43.676 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.676 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.676 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:50:43.676 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719819+0000", 2024-09-11T09:50:43.676 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00098751500000000001 2024-09-11T09:50:43.677 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.677 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.677 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:50:43.677 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719850+0000", 2024-09-11T09:50:43.677 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.1535999999999998e-05 2024-09-11T09:50:43.677 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.677 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.677 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:50:43.677 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719874+0000", 2024-09-11T09:50:43.677 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.3504999999999999e-05 2024-09-11T09:50:43.677 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:43.677 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:50:43.677 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:43.677 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.678 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.678 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:113 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:50:43.678 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.711842+0000", 2024-09-11T09:50:43.678 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 31.926578862, 2024-09-11T09:50:43.678 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0081160099999999999, 2024-09-11T09:50:43.678 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:50:43.678 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:50:43.678 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:43.678 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:50:43.678 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:50:43.678 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:50:43.678 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 113 2024-09-11T09:50:43.678 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.678 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:50:43.679 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.679 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:50:43.679 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711842+0000", 2024-09-11T09:50:43.679 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:43.679 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.679 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.679 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:50:43.679 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711842+0000", 2024-09-11T09:50:43.679 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:43.679 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.679 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.679 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:50:43.679 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711843+0000", 2024-09-11T09:50:43.679 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.4399999999999999e-07 2024-09-11T09:50:43.679 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.680 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.680 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:50:43.680 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711848+0000", 2024-09-11T09:50:43.680 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.4159999999999997e-06 2024-09-11T09:50:43.680 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.680 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.680 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:50:43.680 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711856+0000", 2024-09-11T09:50:43.680 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.7870000000000007e-06 2024-09-11T09:50:43.680 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.680 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.680 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:50:43.680 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711862+0000", 2024-09-11T09:50:43.681 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.3689999999999998e-06 2024-09-11T09:50:43.681 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.681 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.681 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:50:43.681 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.717965+0000", 2024-09-11T09:50:43.681 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0061028899999999997 2024-09-11T09:50:43.681 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.681 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.681 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:50:43.681 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718023+0000", 2024-09-11T09:50:43.681 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.8313000000000002e-05 2024-09-11T09:50:43.681 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.681 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.681 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:50:43.682 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718109+0000", 2024-09-11T09:50:43.682 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.6205000000000006e-05 2024-09-11T09:50:43.682 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.682 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.682 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:50:43.682 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719808+0000", 2024-09-11T09:50:43.682 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0016990060000000001 2024-09-11T09:50:43.682 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.682 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.682 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:50:43.682 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719883+0000", 2024-09-11T09:50:43.682 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.4624999999999998e-05 2024-09-11T09:50:43.682 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.682 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.682 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:50:43.683 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719910+0000", 2024-09-11T09:50:43.683 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.7512e-05 2024-09-11T09:50:43.683 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.683 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.683 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:50:43.683 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719958+0000", 2024-09-11T09:50:43.683 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.8143000000000003e-05 2024-09-11T09:50:43.683 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:43.683 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:50:43.683 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:43.683 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.683 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.683 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:50:43.684 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.711887+0000", 2024-09-11T09:50:43.684 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 31.926534247999999, 2024-09-11T09:50:43.684 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0086063589999999992, 2024-09-11T09:50:43.684 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:50:43.684 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:50:43.684 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:43.684 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:50:43.684 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:50:43.684 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:50:43.684 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 114 2024-09-11T09:50:43.684 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.684 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:50:43.684 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.685 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:50:43.685 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711887+0000", 2024-09-11T09:50:43.685 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:43.685 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.685 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.685 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:50:43.685 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711887+0000", 2024-09-11T09:50:43.685 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:43.685 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.685 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.685 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:50:43.685 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711887+0000", 2024-09-11T09:50:43.685 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.1500000000000005e-07 2024-09-11T09:50:43.685 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.686 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.686 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:50:43.686 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711897+0000", 2024-09-11T09:50:43.686 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.1900000000000001e-06 2024-09-11T09:50:43.686 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.686 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.686 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:50:43.686 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711902+0000", 2024-09-11T09:50:43.686 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.2349999999999999e-06 2024-09-11T09:50:43.686 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.686 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.686 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:50:43.686 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711906+0000", 2024-09-11T09:50:43.686 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.7129999999999999e-06 2024-09-11T09:50:43.687 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.687 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.687 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:50:43.687 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718429+0000", 2024-09-11T09:50:43.687 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0065231960000000002 2024-09-11T09:50:43.687 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.687 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.687 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:50:43.687 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718480+0000", 2024-09-11T09:50:43.687 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.1161e-05 2024-09-11T09:50:43.687 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.687 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.687 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:50:43.687 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718558+0000", 2024-09-11T09:50:43.688 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.7519000000000001e-05 2024-09-11T09:50:43.688 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.688 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.688 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:50:43.688 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720038+0000", 2024-09-11T09:50:43.688 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001480016 2024-09-11T09:50:43.688 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.688 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.688 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:50:43.688 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720417+0000", 2024-09-11T09:50:43.688 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00037943699999999999 2024-09-11T09:50:43.688 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.688 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.689 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:50:43.689 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720452+0000", 2024-09-11T09:50:43.689 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.4595000000000003e-05 2024-09-11T09:50:43.689 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.689 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.689 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:50:43.689 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720493+0000", 2024-09-11T09:50:43.689 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.1782000000000001e-05 2024-09-11T09:50:43.689 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:43.689 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:50:43.689 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:43.689 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.689 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.690 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:50:43.690 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.711912+0000", 2024-09-11T09:50:43.690 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 31.926509579000001, 2024-09-11T09:50:43.690 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0086864750000000008, 2024-09-11T09:50:43.690 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:50:43.690 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:50:43.690 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:43.690 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:50:43.690 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:50:43.690 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:50:43.690 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 115 2024-09-11T09:50:43.690 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.690 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:50:43.690 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.691 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:50:43.691 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711912+0000", 2024-09-11T09:50:43.691 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:43.691 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.691 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.691 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:50:43.691 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711912+0000", 2024-09-11T09:50:43.691 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:43.691 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.691 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.691 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:50:43.691 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711912+0000", 2024-09-11T09:50:43.691 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.3900000000000001e-07 2024-09-11T09:50:43.691 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.692 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.692 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:50:43.692 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711969+0000", 2024-09-11T09:50:43.692 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.7501999999999999e-05 2024-09-11T09:50:43.692 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.692 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.692 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:50:43.692 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711986+0000", 2024-09-11T09:50:43.692 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.6402999999999999e-05 2024-09-11T09:50:43.692 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.692 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.692 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:50:43.692 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711990+0000", 2024-09-11T09:50:43.693 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.929e-06 2024-09-11T09:50:43.693 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.693 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.693 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:50:43.693 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718861+0000", 2024-09-11T09:50:43.693 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0068710530000000002 2024-09-11T09:50:43.693 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.693 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.693 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:50:43.693 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718908+0000", 2024-09-11T09:50:43.693 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.7431e-05 2024-09-11T09:50:43.693 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.693 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.693 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:50:43.694 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718984+0000", 2024-09-11T09:50:43.694 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.5500999999999994e-05 2024-09-11T09:50:43.694 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.694 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.694 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:50:43.694 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720500+0000", 2024-09-11T09:50:43.694 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0015158820000000001 2024-09-11T09:50:43.694 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.694 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.694 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:50:43.694 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720536+0000", 2024-09-11T09:50:43.694 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.6118000000000002e-05 2024-09-11T09:50:43.694 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.694 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.694 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:50:43.695 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720570+0000", 2024-09-11T09:50:43.695 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.4387999999999997e-05 2024-09-11T09:50:43.695 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.695 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.695 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:50:43.695 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720598+0000", 2024-09-11T09:50:43.695 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.8028999999999999e-05 2024-09-11T09:50:43.695 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:43.695 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:50:43.695 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:43.695 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.695 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.695 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:50:43.695 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.711998+0000", 2024-09-11T09:50:43.696 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 31.926423400000001, 2024-09-11T09:50:43.696 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.010785813999999999, 2024-09-11T09:50:43.696 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:50:43.696 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:50:43.696 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:43.696 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:50:43.696 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:50:43.696 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:50:43.696 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 116 2024-09-11T09:50:43.696 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.696 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:50:43.696 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.696 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:50:43.696 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711998+0000", 2024-09-11T09:50:43.697 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:43.697 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.697 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.697 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:50:43.697 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711998+0000", 2024-09-11T09:50:43.697 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:43.697 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.697 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.697 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:50:43.697 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711998+0000", 2024-09-11T09:50:43.697 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.3300000000000003e-07 2024-09-11T09:50:43.697 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.697 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.698 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:50:43.698 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712001+0000", 2024-09-11T09:50:43.698 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.9349999999999999e-06 2024-09-11T09:50:43.698 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.698 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.698 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:50:43.698 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712006+0000", 2024-09-11T09:50:43.698 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.1630000000000004e-06 2024-09-11T09:50:43.698 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.698 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.698 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:50:43.698 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712010+0000", 2024-09-11T09:50:43.698 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.574e-06 2024-09-11T09:50:43.698 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.699 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.699 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:50:43.699 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719415+0000", 2024-09-11T09:50:43.699 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0074049099999999998 2024-09-11T09:50:43.699 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.699 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.699 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:50:43.699 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719465+0000", 2024-09-11T09:50:43.699 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.0129000000000001e-05 2024-09-11T09:50:43.699 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.699 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.699 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:50:43.699 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719545+0000", 2024-09-11T09:50:43.699 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.0253999999999994e-05 2024-09-11T09:50:43.700 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.700 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.700 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:50:43.700 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720636+0000", 2024-09-11T09:50:43.700 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0010909330000000001 2024-09-11T09:50:43.700 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.700 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.700 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:50:43.700 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.722707+0000", 2024-09-11T09:50:43.700 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0020713300000000001 2024-09-11T09:50:43.700 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.700 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.700 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:50:43.700 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.722743+0000", 2024-09-11T09:50:43.701 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.5351000000000001e-05 2024-09-11T09:50:43.701 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.701 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.701 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:50:43.701 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.722784+0000", 2024-09-11T09:50:43.701 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.0802e-05 2024-09-11T09:50:43.701 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:43.701 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:50:43.701 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:43.701 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.701 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.701 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:50:43.701 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.712014+0000", 2024-09-11T09:50:43.702 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 31.926407359999999, 2024-09-11T09:50:43.702 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.013420938, 2024-09-11T09:50:43.702 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:50:43.702 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:50:43.702 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:43.702 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:50:43.703 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:50:43.703 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:50:43.703 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 117 2024-09-11T09:50:43.703 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.703 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:50:43.703 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.703 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:50:43.703 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712014+0000", 2024-09-11T09:50:43.703 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:43.703 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.703 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.703 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:50:43.703 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712014+0000", 2024-09-11T09:50:43.703 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:43.704 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.704 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.704 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:50:43.704 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712014+0000", 2024-09-11T09:50:43.704 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.2000000000000001e-07 2024-09-11T09:50:43.704 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.704 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.704 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:50:43.704 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712016+0000", 2024-09-11T09:50:43.704 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.2309999999999999e-06 2024-09-11T09:50:43.704 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.704 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.704 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:50:43.705 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712020+0000", 2024-09-11T09:50:43.705 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.3830000000000001e-06 2024-09-11T09:50:43.705 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.705 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.705 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:50:43.705 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712023+0000", 2024-09-11T09:50:43.705 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.3340000000000002e-06 2024-09-11T09:50:43.705 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.705 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.705 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:50:43.705 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720083+0000", 2024-09-11T09:50:43.705 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0080601189999999993 2024-09-11T09:50:43.705 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.705 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.706 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:50:43.706 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720125+0000", 2024-09-11T09:50:43.706 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.2029e-05 2024-09-11T09:50:43.706 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.706 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.706 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:50:43.706 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720205+0000", 2024-09-11T09:50:43.706 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.0010000000000001e-05 2024-09-11T09:50:43.706 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.706 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.706 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:50:43.706 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720849+0000", 2024-09-11T09:50:43.706 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00064369899999999996 2024-09-11T09:50:43.706 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.707 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.707 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:50:43.707 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.725357+0000", 2024-09-11T09:50:43.707 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0045077379999999998 2024-09-11T09:50:43.707 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.707 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.707 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:50:43.707 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.725392+0000", 2024-09-11T09:50:43.707 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.5862999999999999e-05 2024-09-11T09:50:43.707 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.707 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.707 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:50:43.707 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.725435+0000", 2024-09-11T09:50:43.707 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.2311999999999997e-05 2024-09-11T09:50:43.708 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:43.708 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:50:43.708 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:43.708 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.708 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.708 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:119 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 131072~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:50:43.708 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.725673+0000", 2024-09-11T09:50:43.708 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 31.912748658000002, 2024-09-11T09:50:43.708 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0053927460000000003, 2024-09-11T09:50:43.708 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:50:43.708 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:50:43.708 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:43.708 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:50:43.708 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:50:43.709 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:50:43.709 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 119 2024-09-11T09:50:43.709 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.709 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:50:43.709 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.709 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:50:43.709 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.725673+0000", 2024-09-11T09:50:43.709 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:43.709 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.709 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.709 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:50:43.709 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.725673+0000", 2024-09-11T09:50:43.709 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:43.709 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.710 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.710 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:50:43.710 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.725673+0000", 2024-09-11T09:50:43.710 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.68e-07 2024-09-11T09:50:43.710 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.710 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.710 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:50:43.710 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.725720+0000", 2024-09-11T09:50:43.710 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.6526999999999997e-05 2024-09-11T09:50:43.710 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.710 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.710 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:50:43.710 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.725734+0000", 2024-09-11T09:50:43.710 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.4593000000000001e-05 2024-09-11T09:50:43.711 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.711 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.711 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:50:43.711 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.725741+0000", 2024-09-11T09:50:43.711 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.28e-06 2024-09-11T09:50:43.711 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.711 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.711 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:50:43.711 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.726290+0000", 2024-09-11T09:50:43.711 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00054892000000000005 2024-09-11T09:50:43.711 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.711 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.711 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:50:43.711 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.726338+0000", 2024-09-11T09:50:43.712 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.7982000000000002e-05 2024-09-11T09:50:43.712 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.712 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.712 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:50:43.712 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.726413+0000", 2024-09-11T09:50:43.713 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.5679000000000005e-05 2024-09-11T09:50:43.713 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.713 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.713 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:50:43.713 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.727555+0000", 2024-09-11T09:50:43.713 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0011420289999999999 2024-09-11T09:50:43.713 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.713 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.713 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:50:43.714 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.731004+0000", 2024-09-11T09:50:43.714 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0034484279999999999 2024-09-11T09:50:43.714 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.714 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.714 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:50:43.714 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.731040+0000", 2024-09-11T09:50:43.714 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.6315e-05 2024-09-11T09:50:43.714 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.714 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.714 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:50:43.714 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.731065+0000", 2024-09-11T09:50:43.715 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.5324999999999999e-05 2024-09-11T09:50:43.715 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:43.715 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:50:43.715 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:43.715 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.715 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.715 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:120 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 196608~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:50:43.715 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.725756+0000", 2024-09-11T09:50:43.715 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 31.912664749000001, 2024-09-11T09:50:43.715 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0053569100000000003, 2024-09-11T09:50:43.716 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:50:43.716 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:50:43.716 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:43.716 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:50:43.716 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:50:43.716 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:50:43.716 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 120 2024-09-11T09:50:43.716 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.716 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:50:43.716 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.716 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:50:43.717 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.725756+0000", 2024-09-11T09:50:43.717 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:43.717 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.717 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.717 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:50:43.717 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.725756+0000", 2024-09-11T09:50:43.717 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:43.717 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.717 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.717 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:50:43.717 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.725757+0000", 2024-09-11T09:50:43.718 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.3799999999999997e-07 2024-09-11T09:50:43.718 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.718 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.718 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:50:43.718 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.725802+0000", 2024-09-11T09:50:43.718 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.4790000000000003e-05 2024-09-11T09:50:43.718 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.718 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.718 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:50:43.718 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.725815+0000", 2024-09-11T09:50:43.719 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.2609000000000001e-05 2024-09-11T09:50:43.719 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.719 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.719 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:50:43.719 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.725822+0000", 2024-09-11T09:50:43.719 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.6450000000000005e-06 2024-09-11T09:50:43.719 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.719 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.719 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:50:43.719 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.726708+0000", 2024-09-11T09:50:43.719 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00088606500000000001 2024-09-11T09:50:43.720 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.720 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.720 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:50:43.720 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.726755+0000", 2024-09-11T09:50:43.720 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.6888000000000002e-05 2024-09-11T09:50:43.720 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.720 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.720 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:50:43.720 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.726828+0000", 2024-09-11T09:50:43.720 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.2510999999999997e-05 2024-09-11T09:50:43.721 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.721 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.721 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:50:43.721 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.727829+0000", 2024-09-11T09:50:43.721 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001001093 2024-09-11T09:50:43.721 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.721 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.721 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:50:43.721 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.731071+0000", 2024-09-11T09:50:43.721 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0032426009999999999 2024-09-11T09:50:43.721 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.722 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.722 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:50:43.722 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.731094+0000", 2024-09-11T09:50:43.722 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.2677e-05 2024-09-11T09:50:43.722 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.722 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.722 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:50:43.722 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.731113+0000", 2024-09-11T09:50:43.722 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.9392999999999999e-05 2024-09-11T09:50:43.722 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:43.722 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:50:43.722 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:43.723 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.723 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.723 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:138 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:50:43.723 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:12.034038+0000", 2024-09-11T09:50:43.723 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 31.604383419000001, 2024-09-11T09:50:43.723 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0054142749999999996, 2024-09-11T09:50:43.723 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:50:43.723 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:50:43.723 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:43.723 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:50:43.723 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:50:43.724 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:50:43.724 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 138 2024-09-11T09:50:43.724 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.724 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:50:43.724 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.724 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:50:43.724 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034038+0000", 2024-09-11T09:50:43.724 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:43.724 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.724 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.725 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:50:43.725 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034038+0000", 2024-09-11T09:50:43.725 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:43.725 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.725 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.725 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:50:43.725 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034038+0000", 2024-09-11T09:50:43.725 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.3499999999999996e-07 2024-09-11T09:50:43.725 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.725 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.725 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:50:43.725 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034043+0000", 2024-09-11T09:50:43.726 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.7310000000000001e-06 2024-09-11T09:50:43.726 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.726 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.726 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:50:43.726 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034050+0000", 2024-09-11T09:50:43.726 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.0419999999999997e-06 2024-09-11T09:50:43.726 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.726 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.726 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:50:43.726 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034055+0000", 2024-09-11T09:50:43.726 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.3299999999999998e-06 2024-09-11T09:50:43.727 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.727 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.727 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:50:43.727 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.037309+0000", 2024-09-11T09:50:43.727 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0032535440000000001 2024-09-11T09:50:43.727 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.727 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.727 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:50:43.727 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.037357+0000", 2024-09-11T09:50:43.727 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.8167999999999997e-05 2024-09-11T09:50:43.727 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.727 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.727 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:50:43.728 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.037430+0000", 2024-09-11T09:50:43.728 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.3275999999999994e-05 2024-09-11T09:50:43.728 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.728 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.728 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:50:43.728 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.039014+0000", 2024-09-11T09:50:43.728 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0015831350000000001 2024-09-11T09:50:43.728 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.728 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.728 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:50:43.728 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.039401+0000", 2024-09-11T09:50:43.728 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00038758500000000002 2024-09-11T09:50:43.729 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.729 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.729 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:50:43.729 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.039429+0000", 2024-09-11T09:50:43.729 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.7379e-05 2024-09-11T09:50:43.729 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.729 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.729 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:50:43.729 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.039452+0000", 2024-09-11T09:50:43.729 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.355e-05 2024-09-11T09:50:43.730 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:43.730 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:50:43.730 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:43.730 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.730 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.730 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:139 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:50:43.730 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:12.034064+0000", 2024-09-11T09:50:43.730 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 31.604357388, 2024-09-11T09:50:43.730 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0054736200000000002, 2024-09-11T09:50:43.730 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:50:43.730 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:50:43.730 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:43.730 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:50:43.731 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:50:43.731 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:50:43.731 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 139 2024-09-11T09:50:43.731 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.731 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:50:43.731 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.731 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:50:43.731 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034064+0000", 2024-09-11T09:50:43.731 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:43.731 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.731 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.732 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:50:43.732 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034064+0000", 2024-09-11T09:50:43.732 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:43.732 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.732 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.732 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:50:43.732 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034064+0000", 2024-09-11T09:50:43.732 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.89e-07 2024-09-11T09:50:43.732 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.732 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.732 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:50:43.733 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034069+0000", 2024-09-11T09:50:43.733 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.9799999999999998e-06 2024-09-11T09:50:43.733 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.733 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.733 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:50:43.733 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034079+0000", 2024-09-11T09:50:43.733 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.3549999999999994e-06 2024-09-11T09:50:43.733 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.733 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.733 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:50:43.733 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034084+0000", 2024-09-11T09:50:43.734 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.2209999999999996e-06 2024-09-11T09:50:43.734 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.734 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.734 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:50:43.734 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.038176+0000", 2024-09-11T09:50:43.734 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0040921610000000004 2024-09-11T09:50:43.734 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.734 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.734 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:50:43.734 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.038221+0000", 2024-09-11T09:50:43.734 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.5006000000000002e-05 2024-09-11T09:50:43.734 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.735 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.735 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:50:43.735 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.038293+0000", 2024-09-11T09:50:43.735 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.1952999999999994e-05 2024-09-11T09:50:43.735 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.735 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.735 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:50:43.735 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.039461+0000", 2024-09-11T09:50:43.735 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0011680340000000001 2024-09-11T09:50:43.735 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.735 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.735 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:50:43.736 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.039487+0000", 2024-09-11T09:50:43.736 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.5752e-05 2024-09-11T09:50:43.736 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.736 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.736 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:50:43.736 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.039515+0000", 2024-09-11T09:50:43.736 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.7763000000000002e-05 2024-09-11T09:50:43.736 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.736 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.736 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:50:43.736 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.039537+0000", 2024-09-11T09:50:43.737 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.2906000000000001e-05 2024-09-11T09:50:43.737 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:43.737 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:50:43.737 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:43.737 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.737 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.737 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:140 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:50:43.737 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:12.034093+0000", 2024-09-11T09:50:43.737 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 31.604327884, 2024-09-11T09:50:43.738 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0059112490000000004, 2024-09-11T09:50:43.738 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:50:43.738 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:50:43.738 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:43.738 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:50:43.738 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:50:43.738 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:50:43.738 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 140 2024-09-11T09:50:43.738 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.738 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:50:43.738 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.738 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:50:43.739 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034093+0000", 2024-09-11T09:50:43.739 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:43.739 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.739 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.739 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:50:43.739 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034093+0000", 2024-09-11T09:50:43.739 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:43.739 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.739 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.739 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:50:43.739 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034094+0000", 2024-09-11T09:50:43.739 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.4299999999999998e-07 2024-09-11T09:50:43.740 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.740 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.740 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:50:43.740 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034159+0000", 2024-09-11T09:50:43.740 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.567e-05 2024-09-11T09:50:43.740 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.740 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.740 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:50:43.740 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034167+0000", 2024-09-11T09:50:43.740 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.8490000000000006e-06 2024-09-11T09:50:43.740 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.740 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.741 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:50:43.741 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034173+0000", 2024-09-11T09:50:43.741 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.1770000000000003e-06 2024-09-11T09:50:43.741 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.741 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.741 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:50:43.741 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.038543+0000", 2024-09-11T09:50:43.741 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0043699150000000003 2024-09-11T09:50:43.741 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.741 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.741 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:50:43.741 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.038589+0000", 2024-09-11T09:50:43.742 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.5562999999999997e-05 2024-09-11T09:50:43.742 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.742 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.742 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:50:43.742 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.038662+0000", 2024-09-11T09:50:43.742 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.2748000000000002e-05 2024-09-11T09:50:43.742 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.742 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.742 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:50:43.742 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.039635+0000", 2024-09-11T09:50:43.742 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00097310799999999998 2024-09-11T09:50:43.743 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.743 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.743 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:50:43.743 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.039945+0000", 2024-09-11T09:50:43.743 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00031054100000000001 2024-09-11T09:50:43.743 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.743 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.743 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:50:43.743 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.039971+0000", 2024-09-11T09:50:43.743 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.5423999999999999e-05 2024-09-11T09:50:43.743 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.743 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.743 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:50:43.744 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.040005+0000", 2024-09-11T09:50:43.744 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.3810999999999999e-05 2024-09-11T09:50:43.744 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:43.744 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:50:43.744 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:43.744 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.744 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.744 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:50:43.744 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:12.034184+0000", 2024-09-11T09:50:43.744 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 31.604237185999999, 2024-09-11T09:50:43.744 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0061520380000000003, 2024-09-11T09:50:43.745 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:50:43.745 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:50:43.745 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:43.745 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:50:43.745 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:50:43.745 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:50:43.745 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 141 2024-09-11T09:50:43.745 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.745 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:50:43.745 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.745 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:50:43.745 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034184+0000", 2024-09-11T09:50:43.746 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:43.746 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.746 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.746 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:50:43.746 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034184+0000", 2024-09-11T09:50:43.746 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:43.746 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.746 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.746 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:50:43.746 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034185+0000", 2024-09-11T09:50:43.746 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.1099999999999995e-07 2024-09-11T09:50:43.747 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.747 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.747 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:50:43.747 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034190+0000", 2024-09-11T09:50:43.747 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.7309999999999998e-06 2024-09-11T09:50:43.747 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.747 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.747 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:50:43.747 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034197+0000", 2024-09-11T09:50:43.747 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.7440000000000001e-06 2024-09-11T09:50:43.747 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.748 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.748 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:50:43.748 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034204+0000", 2024-09-11T09:50:43.748 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.8970000000000002e-06 2024-09-11T09:50:43.748 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.748 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.748 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:50:43.748 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.039035+0000", 2024-09-11T09:50:43.748 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0048307039999999999 2024-09-11T09:50:43.748 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.748 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.749 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:50:43.749 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.039093+0000", 2024-09-11T09:50:43.749 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.8604000000000003e-05 2024-09-11T09:50:43.749 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.749 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.749 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:50:43.749 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.039184+0000", 2024-09-11T09:50:43.749 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.0196000000000002e-05 2024-09-11T09:50:43.749 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.749 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.749 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:50:43.750 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.039873+0000", 2024-09-11T09:50:43.750 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00068960600000000005 2024-09-11T09:50:43.750 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.750 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.750 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:50:43.750 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.040292+0000", 2024-09-11T09:50:43.750 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00041865199999999999 2024-09-11T09:50:43.750 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.750 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.750 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:50:43.750 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.040317+0000", 2024-09-11T09:50:43.750 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.5439000000000002e-05 2024-09-11T09:50:43.751 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.751 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.751 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:50:43.751 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.040336+0000", 2024-09-11T09:50:43.751 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.8754e-05 2024-09-11T09:50:43.751 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:43.751 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:50:43.751 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:43.751 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.751 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.751 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:157 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:50:43.752 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:12.452040+0000", 2024-09-11T09:50:43.752 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 31.186381333, 2024-09-11T09:50:43.752 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0068848269999999996, 2024-09-11T09:50:43.752 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:50:43.752 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:50:43.752 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:43.752 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:50:43.752 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:50:43.752 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:50:43.752 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 157 2024-09-11T09:50:43.752 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.753 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:50:43.753 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.753 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:50:43.753 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452040+0000", 2024-09-11T09:50:43.753 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:43.753 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.753 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.753 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:50:43.753 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452040+0000", 2024-09-11T09:50:43.753 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:43.754 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.754 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.754 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:50:43.754 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452040+0000", 2024-09-11T09:50:43.754 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.22e-07 2024-09-11T09:50:43.754 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.754 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.754 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:50:43.754 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452042+0000", 2024-09-11T09:50:43.754 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.7010000000000001e-06 2024-09-11T09:50:43.754 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.755 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.755 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:50:43.755 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452045+0000", 2024-09-11T09:50:43.755 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.1310000000000002e-06 2024-09-11T09:50:43.755 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.755 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.755 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:50:43.755 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452048+0000", 2024-09-11T09:50:43.755 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.65e-06 2024-09-11T09:50:43.755 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.755 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.756 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:50:43.756 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.454453+0000", 2024-09-11T09:50:43.756 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0024052959999999999 2024-09-11T09:50:43.756 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.756 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.756 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:50:43.756 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.454498+0000", 2024-09-11T09:50:43.756 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.4978999999999997e-05 2024-09-11T09:50:43.756 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.756 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.756 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:50:43.757 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.454567+0000", 2024-09-11T09:50:43.757 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.9110999999999996e-05 2024-09-11T09:50:43.757 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.757 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.757 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:50:43.757 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.455464+0000", 2024-09-11T09:50:43.757 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00089696199999999995 2024-09-11T09:50:43.757 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.757 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.757 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:50:43.757 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.458844+0000", 2024-09-11T09:50:43.758 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0033802469999999999 2024-09-11T09:50:43.758 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.758 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.758 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:50:43.758 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.458889+0000", 2024-09-11T09:50:43.758 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.4441999999999999e-05 2024-09-11T09:50:43.758 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.758 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.758 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:50:43.758 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.458925+0000", 2024-09-11T09:50:43.758 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.6086e-05 2024-09-11T09:50:43.759 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:43.759 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:50:43.759 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:43.759 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.759 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.759 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:158 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:50:43.759 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:12.452055+0000", 2024-09-11T09:50:43.759 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 31.186365940000002, 2024-09-11T09:50:43.759 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0069533829999999996, 2024-09-11T09:50:43.759 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:50:43.759 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:50:43.760 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:43.760 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:50:43.760 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:50:43.760 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:50:43.760 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 158 2024-09-11T09:50:43.760 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.760 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:50:43.760 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.760 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:50:43.760 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452055+0000", 2024-09-11T09:50:43.760 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:43.760 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.761 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.761 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:50:43.761 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452055+0000", 2024-09-11T09:50:43.761 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:43.761 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.761 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.761 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:50:43.761 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452056+0000", 2024-09-11T09:50:43.761 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.48e-07 2024-09-11T09:50:43.761 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.761 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.762 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:50:43.762 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452087+0000", 2024-09-11T09:50:43.762 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.1149000000000003e-05 2024-09-11T09:50:43.762 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.762 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.762 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:50:43.762 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452092+0000", 2024-09-11T09:50:43.762 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.8400000000000002e-06 2024-09-11T09:50:43.762 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.762 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.762 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:50:43.762 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452095+0000", 2024-09-11T09:50:43.763 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.0740000000000001e-06 2024-09-11T09:50:43.763 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.763 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.763 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:50:43.763 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.454796+0000", 2024-09-11T09:50:43.763 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0027009740000000001 2024-09-11T09:50:43.763 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.763 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.763 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:50:43.763 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.454869+0000", 2024-09-11T09:50:43.763 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.3678999999999997e-05 2024-09-11T09:50:43.764 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.764 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.764 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:50:43.764 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.454992+0000", 2024-09-11T09:50:43.764 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00012279000000000001 2024-09-11T09:50:43.764 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.764 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.764 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:50:43.764 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.456072+0000", 2024-09-11T09:50:43.764 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001080229 2024-09-11T09:50:43.764 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.765 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.765 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:50:43.765 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.458931+0000", 2024-09-11T09:50:43.765 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0028591789999999999 2024-09-11T09:50:43.765 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.765 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.765 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:50:43.765 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.458972+0000", 2024-09-11T09:50:43.765 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.0170999999999999e-05 2024-09-11T09:50:43.765 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.765 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.765 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:50:43.766 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.459009+0000", 2024-09-11T09:50:43.766 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.7049999999999999e-05 2024-09-11T09:50:43.766 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:43.766 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:50:43.766 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:43.766 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.766 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.766 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:159 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:50:43.766 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:12.452099+0000", 2024-09-11T09:50:43.766 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 31.186321771999999, 2024-09-11T09:50:43.766 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0069885950000000002, 2024-09-11T09:50:43.766 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:50:43.767 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:50:43.767 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:43.767 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:50:43.767 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:50:43.767 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:50:43.767 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 159 2024-09-11T09:50:43.767 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.767 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:50:43.767 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.767 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:50:43.767 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452099+0000", 2024-09-11T09:50:43.767 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:43.768 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.768 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.768 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:50:43.768 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452099+0000", 2024-09-11T09:50:43.768 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:43.768 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.768 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.768 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:50:43.768 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452100+0000", 2024-09-11T09:50:43.768 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.2800000000000003e-07 2024-09-11T09:50:43.768 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.768 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.769 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:50:43.769 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452102+0000", 2024-09-11T09:50:43.769 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.824e-06 2024-09-11T09:50:43.769 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.769 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.769 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:50:43.769 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452105+0000", 2024-09-11T09:50:43.769 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.0460000000000001e-06 2024-09-11T09:50:43.769 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.769 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.769 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:50:43.770 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452107+0000", 2024-09-11T09:50:43.770 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.543e-06 2024-09-11T09:50:43.770 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.770 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.770 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:50:43.770 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.455484+0000", 2024-09-11T09:50:43.770 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0033769759999999998 2024-09-11T09:50:43.770 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.770 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.770 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:50:43.770 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.455527+0000", 2024-09-11T09:50:43.771 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.2589999999999997e-05 2024-09-11T09:50:43.771 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.771 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.771 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:50:43.771 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.455594+0000", 2024-09-11T09:50:43.771 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.6775999999999999e-05 2024-09-11T09:50:43.771 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.771 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.771 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:50:43.771 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.456312+0000", 2024-09-11T09:50:43.771 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00071874700000000005 2024-09-11T09:50:43.771 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.772 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.772 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:50:43.772 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.459015+0000", 2024-09-11T09:50:43.772 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0027028540000000002 2024-09-11T09:50:43.772 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.772 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.772 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:50:43.772 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.459054+0000", 2024-09-11T09:50:43.772 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.8840000000000001e-05 2024-09-11T09:50:43.772 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:43.772 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:43.772 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:50:43.773 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.459088+0000", 2024-09-11T09:50:43.773 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.4071e-05 2024-09-11T09:50:43.773 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:43.773 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:50:43.773 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:43.773 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:43.773 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:50:43.773 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-11T09:50:43.775 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-11T09:50:43.775 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:50:43.775+0000 7f3c96ec9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31909) UID: 0 2024-09-11T09:50:43.877 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:50:43.878+0000 7fdec5849640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31907) UID: 0 2024-09-11T09:50:43.920 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-11T09:50:43.920 INFO:teuthology.orchestra.run.smithi096.stdout: "ops": [], 2024-09-11T09:50:43.920 INFO:teuthology.orchestra.run.smithi096.stdout: "num_ops": 0 2024-09-11T09:50:43.920 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-11T09:50:43.922 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-11T09:50:43.923 INFO:teuthology.orchestra.run.smithi017.stderr:dumped all 2024-09-11T09:50:43.929 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-11T09:50:43.934 INFO:teuthology.orchestra.run.smithi017.stdout:{"pg_ready":true,"pg_map":{"version":42,"stamp":"2024-09-11T09:50:43.574666+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":161,"num_write_kb":2418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120,"ondisk_log_size":120,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":5,"num_per_pool_osds":5,"num_per_pool_omap_osds":5,"kb":471859200,"kb_used":144944,"kb_used_data":1476,"kb_used_omap":7,"kb_used_meta":143352,"kb_avail":471714256,"statfs":{"total":483183820800,"available":483035398144,"internally_reserved":0,"allocated":1511424,"data_stored":1663558,"data_compressed":66628,"data_compressed_allocated":90112,"data_compressed_original":1163264,"omap_allocated":7971,"internal_metadata":146792669},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.011069"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":39,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-11T09:50:40.645878+0000","last_change":"2024-09-11T09:50:20.497924+0000","last_active":"2024-09-11T09:50:40.645878+0000","last_peered":"2024-09-11T09:50:40.645878+0000","last_clean":"2024-09-11T09:50:40.645878+0000","last_became_active":"2024-09-11T09:50:18.544390+0000","last_became_peered":"2024-09-11T09:50:18.544390+0000","last_unstale":"2024-09-11T09:50:40.645878+0000","last_undegraded":"2024-09-11T09:50:40.645878+0000","last_fullsized":"2024-09-11T09:50:40.645878+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_clean_scrub_stamp":"2024-09-11T09:50:17.477890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T17:32:48.479885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00046094899999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":37,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T09:50:39.562954+0000","last_change":"2024-09-11T09:50:20.497446+0000","last_active":"2024-09-11T09:50:39.562954+0000","last_peered":"2024-09-11T09:50:39.562954+0000","last_clean":"2024-09-11T09:50:39.562954+0000","last_became_active":"2024-09-11T09:50:18.497481+0000","last_became_peered":"2024-09-11T09:50:18.497481+0000","last_unstale":"2024-09-11T09:50:39.562954+0000","last_undegraded":"2024-09-11T09:50:39.562954+0000","last_fullsized":"2024-09-11T09:50:39.562954+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_clean_scrub_stamp":"2024-09-11T09:50:17.477890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T14:12:09.857529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00050697600000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":44,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-11T09:50:40.645866+0000","last_change":"2024-09-11T09:50:34.946175+0000","last_active":"2024-09-11T09:50:40.645866+0000","last_peered":"2024-09-11T09:50:40.645866+0000","last_clean":"2024-09-11T09:50:40.645866+0000","last_became_active":"2024-09-11T09:50:34.945865+0000","last_became_peered":"2024-09-11T09:50:34.945865+0000","last_unstale":"2024-09-11T09:50:40.645866+0000","last_undegraded":"2024-09-11T09:50:40.645866+0000","last_fullsized":"2024-09-11T09:50:40.645866+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_clean_scrub_stamp":"2024-09-11T09:50:17.477890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T17:29:31.422615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045546699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":42,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T09:50:39.562963+0000","last_change":"2024-09-11T09:50:34.946105+0000","last_active":"2024-09-11T09:50:39.562963+0000","last_peered":"2024-09-11T09:50:39.562963+0000","last_clean":"2024-09-11T09:50:39.562963+0000","last_became_active":"2024-09-11T09:50:34.945846+0000","last_became_peered":"2024-09-11T09:50:34.945846+0000","last_unstale":"2024-09-11T09:50:39.562963+0000","last_undegraded":"2024-09-11T09:50:39.562963+0000","last_fullsized":"2024-09-11T09:50:39.562963+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_clean_scrub_stamp":"2024-09-11T09:50:17.477890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:49:48.552858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00050337500000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"18'2","reported_seq":39,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T09:50:40.239035+0000","last_change":"2024-09-11T09:50:20.501218+0000","last_active":"2024-09-11T09:50:40.239035+0000","last_peered":"2024-09-11T09:50:40.239035+0000","last_clean":"2024-09-11T09:50:40.239035+0000","last_became_active":"2024-09-11T09:50:18.493165+0000","last_became_peered":"2024-09-11T09:50:18.493165+0000","last_unstale":"2024-09-11T09:50:40.239035+0000","last_undegraded":"2024-09-11T09:50:40.239035+0000","last_fullsized":"2024-09-11T09:50:40.239035+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_clean_scrub_stamp":"2024-09-11T09:50:17.477890+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T18:04:31.250555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00094463900000000005,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":35,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T09:50:37.770084+0000","last_change":"2024-09-11T09:50:21.030566+0000","last_active":"2024-09-11T09:50:37.770084+0000","last_peered":"2024-09-11T09:50:37.770084+0000","last_clean":"2024-09-11T09:50:37.770084+0000","last_became_active":"2024-09-11T09:50:18.493109+0000","last_became_peered":"2024-09-11T09:50:18.493109+0000","last_unstale":"2024-09-11T09:50:37.770084+0000","last_undegraded":"2024-09-11T09:50:37.770084+0000","last_fullsized":"2024-09-11T09:50:37.770084+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_clean_scrub_stamp":"2024-09-11T09:50:17.477890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:37:16.328368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041008499999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":39,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-11T09:50:40.645965+0000","last_change":"2024-09-11T09:50:20.498284+0000","last_active":"2024-09-11T09:50:40.645965+0000","last_peered":"2024-09-11T09:50:40.645965+0000","last_clean":"2024-09-11T09:50:40.645965+0000","last_became_active":"2024-09-11T09:50:18.493210+0000","last_became_peered":"2024-09-11T09:50:18.493210+0000","last_unstale":"2024-09-11T09:50:40.645965+0000","last_undegraded":"2024-09-11T09:50:40.645965+0000","last_fullsized":"2024-09-11T09:50:40.645965+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_clean_scrub_stamp":"2024-09-11T09:50:17.477890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T18:16:13.887681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079749499999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"16'1","reported_seq":38,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T09:50:40.239047+0000","last_change":"2024-09-11T09:50:20.500687+0000","last_active":"2024-09-11T09:50:40.239047+0000","last_peered":"2024-09-11T09:50:40.239047+0000","last_clean":"2024-09-11T09:50:40.239047+0000","last_became_active":"2024-09-11T09:50:18.493650+0000","last_became_peered":"2024-09-11T09:50:18.493650+0000","last_unstale":"2024-09-11T09:50:40.239047+0000","last_undegraded":"2024-09-11T09:50:40.239047+0000","last_fullsized":"2024-09-11T09:50:40.239047+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_clean_scrub_stamp":"2024-09-11T09:50:17.477890+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T14:26:32.838143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041261100000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"13'117","reported_seq":203,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-11T09:50:40.645927+0000","last_change":"2024-09-11T09:50:37.558081+0000","last_active":"2024-09-11T09:50:40.645927+0000","last_peered":"2024-09-11T09:50:40.645927+0000","last_clean":"2024-09-11T09:50:40.645927+0000","last_became_active":"2024-09-11T09:50:37.557755+0000","last_became_peered":"2024-09-11T09:50:37.557755+0000","last_unstale":"2024-09-11T09:50:40.645927+0000","last_undegraded":"2024-09-11T09:50:40.645927+0000","last_fullsized":"2024-09-11T09:50:40.645927+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:50:10.457262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:50:10.457262+0000","last_clean_scrub_stamp":"2024-09-11T09:50:10.457262+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:53:34.686406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":180224,"data_stored":1771104,"data_compressed":99124,"data_compressed_allocated":135168,"data_compressed_original":1736704,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":12,"seq":51539607562,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27380,"kb_used_data":352,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344460,"statfs":{"total":96636764160,"available":96608727040,"internally_reserved":0,"allocated":360448,"data_stored":693334,"data_compressed":32496,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":0,"seq":51539607561,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":11,"seq":47244640266,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":280,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":286720,"data_stored":98407,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":11,"seq":47244640266,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27292,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344548,"statfs":{"total":96636764160,"available":96608817152,"internally_reserved":0,"allocated":270336,"data_stored":93810,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640265,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31456,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340384,"statfs":{"total":96636764160,"available":96604553216,"internally_reserved":0,"allocated":270336,"data_stored":93810,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640265,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31508,"kb_used_data":316,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340332,"statfs":{"total":96636764160,"available":96604499968,"internally_reserved":0,"allocated":323584,"data_stored":684197,"data_compressed":34132,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32496,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":34132,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32496,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T09:50:43.936 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T09:50:44.013 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:50:44.014+0000 7f3c96ec9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31909) UID: 0 2024-09-11T09:50:44.073 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-11T09:50:44.073 INFO:teuthology.orchestra.run.smithi096.stdout: "ops": [], 2024-09-11T09:50:44.073 INFO:teuthology.orchestra.run.smithi096.stdout: "complaint_time": 30, 2024-09-11T09:50:44.073 INFO:teuthology.orchestra.run.smithi096.stdout: "num_blocked_ops": 0 2024-09-11T09:50:44.073 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-11T09:50:44.083 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-11T09:50:44.084 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:50:44.084+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:50:44.184 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-09-11T09:50:44.185+0000 7f7b6c7d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31904) UID: 0 2024-09-11T09:50:44.222 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-11T09:50:44.223 INFO:teuthology.orchestra.run.smithi017.stderr:dumped all 2024-09-11T09:50:44.226 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-11T09:50:44.227 INFO:teuthology.orchestra.run.smithi096.stdout: "size": 20, 2024-09-11T09:50:44.227 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 600, 2024-09-11T09:50:44.227 INFO:teuthology.orchestra.run.smithi096.stdout: "ops": [] 2024-09-11T09:50:44.227 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-11T09:50:44.235 INFO:teuthology.orchestra.run.smithi017.stdout:{"pg_ready":true,"pg_map":{"version":42,"stamp":"2024-09-11T09:50:43.574666+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":161,"num_write_kb":2418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120,"ondisk_log_size":120,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":5,"num_per_pool_osds":5,"num_per_pool_omap_osds":5,"kb":471859200,"kb_used":144944,"kb_used_data":1476,"kb_used_omap":7,"kb_used_meta":143352,"kb_avail":471714256,"statfs":{"total":483183820800,"available":483035398144,"internally_reserved":0,"allocated":1511424,"data_stored":1663558,"data_compressed":66628,"data_compressed_allocated":90112,"data_compressed_original":1163264,"omap_allocated":7971,"internal_metadata":146792669},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.011069"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":39,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-11T09:50:40.645878+0000","last_change":"2024-09-11T09:50:20.497924+0000","last_active":"2024-09-11T09:50:40.645878+0000","last_peered":"2024-09-11T09:50:40.645878+0000","last_clean":"2024-09-11T09:50:40.645878+0000","last_became_active":"2024-09-11T09:50:18.544390+0000","last_became_peered":"2024-09-11T09:50:18.544390+0000","last_unstale":"2024-09-11T09:50:40.645878+0000","last_undegraded":"2024-09-11T09:50:40.645878+0000","last_fullsized":"2024-09-11T09:50:40.645878+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_clean_scrub_stamp":"2024-09-11T09:50:17.477890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T17:32:48.479885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00046094899999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":37,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T09:50:39.562954+0000","last_change":"2024-09-11T09:50:20.497446+0000","last_active":"2024-09-11T09:50:39.562954+0000","last_peered":"2024-09-11T09:50:39.562954+0000","last_clean":"2024-09-11T09:50:39.562954+0000","last_became_active":"2024-09-11T09:50:18.497481+0000","last_became_peered":"2024-09-11T09:50:18.497481+0000","last_unstale":"2024-09-11T09:50:39.562954+0000","last_undegraded":"2024-09-11T09:50:39.562954+0000","last_fullsized":"2024-09-11T09:50:39.562954+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_clean_scrub_stamp":"2024-09-11T09:50:17.477890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T14:12:09.857529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00050697600000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":44,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-11T09:50:40.645866+0000","last_change":"2024-09-11T09:50:34.946175+0000","last_active":"2024-09-11T09:50:40.645866+0000","last_peered":"2024-09-11T09:50:40.645866+0000","last_clean":"2024-09-11T09:50:40.645866+0000","last_became_active":"2024-09-11T09:50:34.945865+0000","last_became_peered":"2024-09-11T09:50:34.945865+0000","last_unstale":"2024-09-11T09:50:40.645866+0000","last_undegraded":"2024-09-11T09:50:40.645866+0000","last_fullsized":"2024-09-11T09:50:40.645866+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_clean_scrub_stamp":"2024-09-11T09:50:17.477890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T17:29:31.422615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045546699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":42,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T09:50:39.562963+0000","last_change":"2024-09-11T09:50:34.946105+0000","last_active":"2024-09-11T09:50:39.562963+0000","last_peered":"2024-09-11T09:50:39.562963+0000","last_clean":"2024-09-11T09:50:39.562963+0000","last_became_active":"2024-09-11T09:50:34.945846+0000","last_became_peered":"2024-09-11T09:50:34.945846+0000","last_unstale":"2024-09-11T09:50:39.562963+0000","last_undegraded":"2024-09-11T09:50:39.562963+0000","last_fullsized":"2024-09-11T09:50:39.562963+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_clean_scrub_stamp":"2024-09-11T09:50:17.477890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:49:48.552858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00050337500000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"18'2","reported_seq":39,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T09:50:40.239035+0000","last_change":"2024-09-11T09:50:20.501218+0000","last_active":"2024-09-11T09:50:40.239035+0000","last_peered":"2024-09-11T09:50:40.239035+0000","last_clean":"2024-09-11T09:50:40.239035+0000","last_became_active":"2024-09-11T09:50:18.493165+0000","last_became_peered":"2024-09-11T09:50:18.493165+0000","last_unstale":"2024-09-11T09:50:40.239035+0000","last_undegraded":"2024-09-11T09:50:40.239035+0000","last_fullsized":"2024-09-11T09:50:40.239035+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_clean_scrub_stamp":"2024-09-11T09:50:17.477890+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T18:04:31.250555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00094463900000000005,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":35,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-09-11T09:50:37.770084+0000","last_change":"2024-09-11T09:50:21.030566+0000","last_active":"2024-09-11T09:50:37.770084+0000","last_peered":"2024-09-11T09:50:37.770084+0000","last_clean":"2024-09-11T09:50:37.770084+0000","last_became_active":"2024-09-11T09:50:18.493109+0000","last_became_peered":"2024-09-11T09:50:18.493109+0000","last_unstale":"2024-09-11T09:50:37.770084+0000","last_undegraded":"2024-09-11T09:50:37.770084+0000","last_fullsized":"2024-09-11T09:50:37.770084+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_clean_scrub_stamp":"2024-09-11T09:50:17.477890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:37:16.328368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041008499999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":39,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-11T09:50:40.645965+0000","last_change":"2024-09-11T09:50:20.498284+0000","last_active":"2024-09-11T09:50:40.645965+0000","last_peered":"2024-09-11T09:50:40.645965+0000","last_clean":"2024-09-11T09:50:40.645965+0000","last_became_active":"2024-09-11T09:50:18.493210+0000","last_became_peered":"2024-09-11T09:50:18.493210+0000","last_unstale":"2024-09-11T09:50:40.645965+0000","last_undegraded":"2024-09-11T09:50:40.645965+0000","last_fullsized":"2024-09-11T09:50:40.645965+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_clean_scrub_stamp":"2024-09-11T09:50:17.477890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T18:16:13.887681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079749499999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"16'1","reported_seq":38,"reported_epoch":27,"state":"active+clean","last_fresh":"2024-09-11T09:50:40.239047+0000","last_change":"2024-09-11T09:50:20.500687+0000","last_active":"2024-09-11T09:50:40.239047+0000","last_peered":"2024-09-11T09:50:40.239047+0000","last_clean":"2024-09-11T09:50:40.239047+0000","last_became_active":"2024-09-11T09:50:18.493650+0000","last_became_peered":"2024-09-11T09:50:18.493650+0000","last_unstale":"2024-09-11T09:50:40.239047+0000","last_undegraded":"2024-09-11T09:50:40.239047+0000","last_fullsized":"2024-09-11T09:50:40.239047+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_clean_scrub_stamp":"2024-09-11T09:50:17.477890+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T14:26:32.838143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041261100000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"13'117","reported_seq":203,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-11T09:50:40.645927+0000","last_change":"2024-09-11T09:50:37.558081+0000","last_active":"2024-09-11T09:50:40.645927+0000","last_peered":"2024-09-11T09:50:40.645927+0000","last_clean":"2024-09-11T09:50:40.645927+0000","last_became_active":"2024-09-11T09:50:37.557755+0000","last_became_peered":"2024-09-11T09:50:37.557755+0000","last_unstale":"2024-09-11T09:50:40.645927+0000","last_undegraded":"2024-09-11T09:50:40.645927+0000","last_fullsized":"2024-09-11T09:50:40.645927+0000","mapping_epoch":25,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:50:10.457262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:50:10.457262+0000","last_clean_scrub_stamp":"2024-09-11T09:50:10.457262+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:53:34.686406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":180224,"data_stored":1771104,"data_compressed":99124,"data_compressed_allocated":135168,"data_compressed_original":1736704,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":12,"seq":51539607562,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27380,"kb_used_data":352,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344460,"statfs":{"total":96636764160,"available":96608727040,"internally_reserved":0,"allocated":360448,"data_stored":693334,"data_compressed":32496,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":0,"seq":51539607561,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":11,"seq":47244640266,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27308,"kb_used_data":280,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344532,"statfs":{"total":96636764160,"available":96608800768,"internally_reserved":0,"allocated":286720,"data_stored":98407,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":11,"seq":47244640266,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27292,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344548,"statfs":{"total":96636764160,"available":96608817152,"internally_reserved":0,"allocated":270336,"data_stored":93810,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640265,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31456,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340384,"statfs":{"total":96636764160,"available":96604553216,"internally_reserved":0,"allocated":270336,"data_stored":93810,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640265,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31508,"kb_used_data":316,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340332,"statfs":{"total":96636764160,"available":96604499968,"internally_reserved":0,"allocated":323584,"data_stored":684197,"data_compressed":34132,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32496,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":34132,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32496,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T09:50:44.236 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-09-11T09:50:44.236 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-11T09:50:44.236 INFO:tasks.thrashosds.thrasher:Adding osd 0 2024-09-11T09:50:44.236 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd in 0 2024-09-11T09:50:44.237 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-11T09:50:44.241 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T09:50:44.320 INFO:tasks.ceph.osd.2.smithi017.stderr:2024-09-11T09:50:44.321+0000 7fd20b170640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31672) UID: 0 2024-09-11T09:50:44.380 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-11T09:50:44.380 INFO:teuthology.orchestra.run.smithi096.stdout: "ops": [], 2024-09-11T09:50:44.380 INFO:teuthology.orchestra.run.smithi096.stdout: "num_ops": 0 2024-09-11T09:50:44.380 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-11T09:50:44.386 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:50:44.387+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:50:44.389 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-11T09:50:44.401 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-11T09:50:44.401 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T09:50:44.418 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-11T09:50:44.418 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T09:50:44.435 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-11T09:50:44.435 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T09:50:44.451 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-11T09:50:44.452 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T09:50:44.469 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-11T09:50:44.469 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T09:50:44.486 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-11T09:50:44.487 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T09:50:44.487 INFO:tasks.ceph.osd.0.smithi017.stderr:2024-09-11T09:50:44.487+0000 7fb73c0f1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31665) UID: 0 2024-09-11T09:50:44.493 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:50:44.493 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:50:44.502 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-11T09:50:44.581 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-11T09:50:44.581+0000 7ff44db6e640 -1 mon.a@0(leader).osd e31 definitely_dead 0 2024-09-11T09:50:44.587 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-11T09:50:44.588 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:50:44.588+0000 7fdec5849640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31907) UID: 0 2024-09-11T09:50:44.605 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:50:44.605 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:50:44.614 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-11T09:50:44.689 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:50:44.689+0000 7f3c96ec9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31909) UID: 0 2024-09-11T09:50:44.769 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:50:44.769 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:50:44.781 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-11T09:50:44.826 INFO:tasks.ceph.osd.0.smithi017.stderr:2024-09-11T09:50:44.827+0000 7fb73c0f1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31665) UID: 0 2024-09-11T09:50:44.889 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:50:44.889 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:50:44.890 INFO:tasks.ceph.osd.0.smithi017.stderr:2024-09-11T09:50:44.891+0000 7fb73c0f1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31665) UID: 0 2024-09-11T09:50:44.902 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-11T09:50:44.991 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:50:44.992+0000 7f3c96ec9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31909) UID: 0 2024-09-11T09:50:45.050 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:50:45.050 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:50:45.059 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-11T09:50:45.104 INFO:tasks.ceph.osd.0.smithi017.stderr:2024-09-11T09:50:45.104+0000 7fb73c0f1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31665) UID: 0 2024-09-11T09:50:45.166 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:50:45.167 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:50:45.175 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-11T09:50:45.216 INFO:tasks.ceph.osd.0.smithi017.stderr:2024-09-11T09:50:45.217+0000 7fb73c0f1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31665) UID: 0 2024-09-11T09:50:45.294 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:50:45.295+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:50:45.323 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:50:45.323 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:50:45.331 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-11T09:50:45.416 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:50:45.417+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:50:45.480 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:50:45.480 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:50:45.488 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-11T09:50:45.496 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:50:45.497+0000 7fdec5849640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31907) UID: 0 2024-09-11T09:50:45.584 INFO:teuthology.orchestra.run.smithi017.stderr:marked in osd.0. 2024-09-11T09:50:45.584 INFO:teuthology.orchestra.run.smithi017.stderr:noscrub is unset 2024-09-11T09:50:45.597 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:50:45.598+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:50:45.601 INFO:tasks.thrashosds.thrasher:Added osd 0 2024-09-11T09:50:45.640 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:50:45.640 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:50:45.649 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-11T09:50:45.734 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:50:45.735+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:50:45.797 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:50:45.797 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:50:45.798 INFO:tasks.ceph.osd.0.smithi017.stderr:2024-09-11T09:50:45.799+0000 7fb73c0f1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31665) UID: 0 2024-09-11T09:50:45.806 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-11T09:50:45.900 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:50:45.901+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:50:45.953 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:50:45.953 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:50:45.962 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-11T09:50:46.006 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:50:46.007+0000 7fdec5849640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31907) UID: 0 2024-09-11T09:50:46.077 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:50:46.077 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:50:46.086 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-11T09:50:46.129 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-09-11T09:50:46.130+0000 7f7b6c7d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31904) UID: 0 2024-09-11T09:50:46.202 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:50:46.203+0000 7f3c96ec9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31909) UID: 0 2024-09-11T09:50:46.231 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:50:46.231 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:50:46.240 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-11T09:50:46.303 INFO:tasks.ceph.osd.2.smithi017.stderr:2024-09-11T09:50:46.304+0000 7fd20b170640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31672) UID: 0 2024-09-11T09:50:46.383 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:50:46.383 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:50:46.392 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-11T09:50:46.435 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-09-11T09:50:46.436+0000 7f7b6c7d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31904) UID: 0 2024-09-11T09:50:46.495 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:50:46.496 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:50:46.498 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T09:50:46.504 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-11T09:50:46.542 INFO:tasks.ceph.osd.2.smithi017.stderr:2024-09-11T09:50:46.543+0000 7fd20b170640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31672) UID: 0 2024-09-11T09:50:46.605 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-09-11T09:50:46.605+0000 7f7b6c7d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31904) UID: 0 2024-09-11T09:50:46.631 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-11T09:50:46.631 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T09:50:46.648 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-11T09:50:46.649 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T09:50:46.649 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:50:46.649 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:50:46.658 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-11T09:50:46.665 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-11T09:50:46.665 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T09:50:46.682 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-11T09:50:46.682 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T09:50:46.698 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-11T09:50:46.699 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T09:50:46.715 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-11T09:50:46.715 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T09:50:46.742 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:50:46.743+0000 7fdec5849640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31907) UID: 0 2024-09-11T09:50:46.801 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-11T09:50:46.801 INFO:teuthology.orchestra.run.smithi096.stdout: "size": 20, 2024-09-11T09:50:46.801 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 600, 2024-09-11T09:50:46.801 INFO:teuthology.orchestra.run.smithi096.stdout: "ops": [] 2024-09-11T09:50:46.801 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-11T09:50:46.806 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:50:46.807+0000 7fdec5849640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31907) UID: 0 2024-09-11T09:50:46.809 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-11T09:50:46.908 INFO:tasks.ceph.osd.0.smithi017.stderr:2024-09-11T09:50:46.909+0000 7fb73c0f1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31665) UID: 0 2024-09-11T09:50:46.952 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-11T09:50:46.952 INFO:teuthology.orchestra.run.smithi096.stdout: "ops": [], 2024-09-11T09:50:46.952 INFO:teuthology.orchestra.run.smithi096.stdout: "num_ops": 0 2024-09-11T09:50:46.953 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-11T09:50:46.961 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-11T09:50:47.009 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:50:47.010+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:50:47.064 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-11T09:50:47.064 INFO:teuthology.orchestra.run.smithi096.stdout: "ops": [], 2024-09-11T09:50:47.065 INFO:teuthology.orchestra.run.smithi096.stdout: "complaint_time": 30, 2024-09-11T09:50:47.065 INFO:teuthology.orchestra.run.smithi096.stdout: "num_blocked_ops": 0 2024-09-11T09:50:47.065 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-11T09:50:47.073 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-11T09:50:47.116 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:50:47.117+0000 7f3c96ec9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31909) UID: 0 2024-09-11T09:50:47.177 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-11T09:50:47.177 INFO:teuthology.orchestra.run.smithi096.stdout: "size": 20, 2024-09-11T09:50:47.177 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 600, 2024-09-11T09:50:47.177 INFO:teuthology.orchestra.run.smithi096.stdout: "ops": [ 2024-09-11T09:50:47.177 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:47.178 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4230.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-11T09:50:47.178 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:18.490736+0000", 2024-09-11T09:50:47.178 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 28.687189658000001, 2024-09-11T09:50:47.178 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0059097569999999999, 2024-09-11T09:50:47.178 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:50:47.178 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:50:47.178 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:47.178 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:50:47.178 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4230", 2024-09-11T09:50:47.178 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.17:0/4101109576", 2024-09-11T09:50:47.179 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 1 2024-09-11T09:50:47.179 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:47.179 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:50:47.179 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:47.179 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:50:47.179 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.490736+0000", 2024-09-11T09:50:47.179 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:47.179 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:47.179 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:47.180 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:50:47.180 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.490736+0000", 2024-09-11T09:50:47.180 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:47.180 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:47.180 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:47.180 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:50:47.180 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.490742+0000", 2024-09-11T09:50:47.180 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.48e-06 2024-09-11T09:50:47.181 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:47.181 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:47.181 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:50:47.181 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.490749+0000", 2024-09-11T09:50:47.181 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.4569999999999999e-06 2024-09-11T09:50:47.181 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:47.181 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:47.181 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:50:47.182 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.490762+0000", 2024-09-11T09:50:47.182 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.2721000000000001e-05 2024-09-11T09:50:47.182 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:47.182 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:47.182 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:50:47.182 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.490774+0000", 2024-09-11T09:50:47.182 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1823e-05 2024-09-11T09:50:47.182 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:47.183 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:47.183 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:50:47.183 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.490846+0000", 2024-09-11T09:50:47.183 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.1878000000000005e-05 2024-09-11T09:50:47.183 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:47.183 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:47.183 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for peered", 2024-09-11T09:50:47.183 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.490857+0000", 2024-09-11T09:50:47.184 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1017e-05 2024-09-11T09:50:47.184 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:47.184 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:47.184 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:50:47.184 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.493856+0000", 2024-09-11T09:50:47.184 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00299885 2024-09-11T09:50:47.184 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:47.185 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:47.185 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:50:47.185 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.494020+0000", 2024-09-11T09:50:47.185 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00016436599999999999 2024-09-11T09:50:47.185 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:47.185 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:47.185 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 2", 2024-09-11T09:50:47.186 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.494186+0000", 2024-09-11T09:50:47.186 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00016562799999999999 2024-09-11T09:50:47.186 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:47.186 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:47.186 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:50:47.186 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.495407+0000", 2024-09-11T09:50:47.186 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0012217420000000001 2024-09-11T09:50:47.186 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:47.187 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:47.187 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:50:47.187 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.496579+0000", 2024-09-11T09:50:47.187 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001171844 2024-09-11T09:50:47.187 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:47.187 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:47.187 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:50:47.187 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.496620+0000", 2024-09-11T09:50:47.188 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.0479000000000003e-05 2024-09-11T09:50:47.188 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:47.188 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:47.188 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:50:47.188 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.496646+0000", 2024-09-11T09:50:47.188 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.6472e-05 2024-09-11T09:50:47.188 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:47.188 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:50:47.189 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:47.189 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:47.189 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:47.189 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_repop_reply(client.4230.0:1 2.3 e16/15 ondisk, result = 0)", 2024-09-11T09:50:47.189 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:18.495321+0000", 2024-09-11T09:50:47.189 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 28.682604992999998, 2024-09-11T09:50:47.189 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.4965999999999998e-05, 2024-09-11T09:50:47.189 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:50:47.190 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:50:47.190 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "started", 2024-09-11T09:50:47.190 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:50:47.190 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:47.190 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:50:47.190 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.495321+0000", 2024-09-11T09:50:47.190 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:47.190 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:47.190 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:47.191 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:50:47.191 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.495321+0000", 2024-09-11T09:50:47.191 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:47.191 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:47.191 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:47.191 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:50:47.191 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.495322+0000", 2024-09-11T09:50:47.192 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.7700000000000004e-07 2024-09-11T09:50:47.192 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:47.192 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:47.192 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:50:47.192 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.495325+0000", 2024-09-11T09:50:47.192 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.7780000000000001e-06 2024-09-11T09:50:47.192 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:47.192 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:47.193 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:50:47.193 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.495338+0000", 2024-09-11T09:50:47.193 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.2315e-05 2024-09-11T09:50:47.193 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:47.193 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:47.193 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:50:47.193 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.495345+0000", 2024-09-11T09:50:47.193 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.8889999999999996e-06 2024-09-11T09:50:47.194 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:47.194 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:47.194 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:50:47.194 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.495392+0000", 2024-09-11T09:50:47.194 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.7524000000000001e-05 2024-09-11T09:50:47.194 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:47.194 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:47.194 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:50:47.194 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.495402+0000", 2024-09-11T09:50:47.194 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.5840000000000001e-06 2024-09-11T09:50:47.194 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:47.194 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:47.195 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:50:47.195 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.495416+0000", 2024-09-11T09:50:47.195 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.4199000000000001e-05 2024-09-11T09:50:47.195 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:47.195 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:50:47.195 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:47.195 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:47.195 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:47.195 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4230.0:2 2.2 2:5cea7035:::rbd_info:head [call rbd.metadata_list in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e16)", 2024-09-11T09:50:47.195 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:18.499576+0000", 2024-09-11T09:50:47.196 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 28.678350485999999, 2024-09-11T09:50:47.196 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00018163000000000001, 2024-09-11T09:50:47.196 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:50:47.196 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:50:47.196 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "reached pg", 2024-09-11T09:50:47.196 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:50:47.196 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4230", 2024-09-11T09:50:47.196 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.17:0/4101109576", 2024-09-11T09:50:47.196 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 2 2024-09-11T09:50:47.196 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:47.196 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:50:47.196 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:47.196 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:50:47.197 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.499576+0000", 2024-09-11T09:50:47.197 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:47.197 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:47.197 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:47.197 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:50:47.197 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.499576+0000", 2024-09-11T09:50:47.197 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:47.197 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:47.197 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:47.197 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:50:47.197 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.499577+0000", 2024-09-11T09:50:47.197 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.124e-06 2024-09-11T09:50:47.198 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:47.198 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:47.198 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:50:47.198 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.499584+0000", 2024-09-11T09:50:47.198 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.1949999999999997e-06 2024-09-11T09:50:47.198 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:47.198 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:47.198 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:50:47.198 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.499597+0000", 2024-09-11T09:50:47.198 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.366e-05 2024-09-11T09:50:47.198 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:47.198 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:47.198 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:50:47.199 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.499604+0000", 2024-09-11T09:50:47.199 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.2709999999999998e-06 2024-09-11T09:50:47.199 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:47.199 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:47.199 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:50:47.199 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.499660+0000", 2024-09-11T09:50:47.199 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.6270999999999997e-05 2024-09-11T09:50:47.199 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:47.200 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:47.200 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:50:47.200 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.499757+0000", 2024-09-11T09:50:47.200 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.7108999999999995e-05 2024-09-11T09:50:47.200 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:47.200 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:50:47.200 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:47.200 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:47.200 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:47.201 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4230.0:3 2.2 2:5cea7035:::rbd_info:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e16)", 2024-09-11T09:50:47.201 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:18.499937+0000", 2024-09-11T09:50:47.201 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 28.677988697, 2024-09-11T09:50:47.201 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000143318, 2024-09-11T09:50:47.201 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:50:47.201 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:50:47.201 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "reached pg", 2024-09-11T09:50:47.201 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:50:47.201 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4230", 2024-09-11T09:50:47.201 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.17:0/4101109576", 2024-09-11T09:50:47.201 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 3 2024-09-11T09:50:47.201 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:47.201 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:50:47.202 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:47.202 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:50:47.202 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.499937+0000", 2024-09-11T09:50:47.202 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:47.202 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:47.202 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:47.202 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:50:47.202 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.499937+0000", 2024-09-11T09:50:47.202 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:47.202 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:47.202 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:47.202 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:50:47.203 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.499938+0000", 2024-09-11T09:50:47.203 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.195e-06 2024-09-11T09:50:47.203 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:47.203 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:47.203 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:50:47.203 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.499943+0000", 2024-09-11T09:50:47.203 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.9080000000000003e-06 2024-09-11T09:50:47.203 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:47.203 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:47.203 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:50:47.203 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.499956+0000", 2024-09-11T09:50:47.203 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.2728000000000001e-05 2024-09-11T09:50:47.204 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:47.204 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:47.204 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:50:47.204 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.499964+0000", 2024-09-11T09:50:47.204 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.8220000000000007e-06 2024-09-11T09:50:47.204 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:47.204 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:47.204 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:50:47.204 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.500017+0000", 2024-09-11T09:50:47.204 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.2809000000000002e-05 2024-09-11T09:50:47.204 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:47.205 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:47.205 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:50:47.205 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.500081+0000", 2024-09-11T09:50:47.205 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.3856000000000007e-05 2024-09-11T09:50:47.205 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:47.205 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:50:47.205 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:47.205 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:47.205 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:47.205 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4230.0:5 2.2 2:5cea7035:::rbd_info:head [create,write 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-11T09:50:47.205 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:19.492662+0000", 2024-09-11T09:50:47.205 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 27.685264037, 2024-09-11T09:50:47.205 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0043824759999999997, 2024-09-11T09:50:47.206 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:50:47.206 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:50:47.206 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:47.206 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:50:47.206 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4230", 2024-09-11T09:50:47.206 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.17:0/4101109576", 2024-09-11T09:50:47.206 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 5 2024-09-11T09:50:47.206 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:47.206 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:50:47.206 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:47.206 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:50:47.206 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.492662+0000", 2024-09-11T09:50:47.207 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:47.207 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:47.207 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:47.207 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:50:47.207 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.492662+0000", 2024-09-11T09:50:47.207 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:47.207 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:47.207 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:47.207 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:50:47.207 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.492664+0000", 2024-09-11T09:50:47.207 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.3599999999999999e-06 2024-09-11T09:50:47.207 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:47.207 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:47.208 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:50:47.208 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.492675+0000", 2024-09-11T09:50:47.208 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1182000000000001e-05 2024-09-11T09:50:47.208 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:47.208 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:47.208 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:50:47.208 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.492693+0000", 2024-09-11T09:50:47.208 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.7031999999999999e-05 2024-09-11T09:50:47.208 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:47.208 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:47.208 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:50:47.208 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.492702+0000", 2024-09-11T09:50:47.209 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.5540000000000003e-06 2024-09-11T09:50:47.209 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:47.209 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:47.209 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:50:47.209 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.492767+0000", 2024-09-11T09:50:47.209 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.5411000000000001e-05 2024-09-11T09:50:47.209 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:47.209 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:47.209 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op must wait for map", 2024-09-11T09:50:47.209 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.492773+0000", 2024-09-11T09:50:47.209 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.1039999999999998e-06 2024-09-11T09:50:47.209 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:47.210 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:47.210 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:50:47.210 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.494354+0000", 2024-09-11T09:50:47.210 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0015814049999999999 2024-09-11T09:50:47.210 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:47.210 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:47.210 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:50:47.210 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.494476+0000", 2024-09-11T09:50:47.210 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000122089 2024-09-11T09:50:47.210 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:47.210 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:47.210 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 1", 2024-09-11T09:50:47.210 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.494577+0000", 2024-09-11T09:50:47.211 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000100536 2024-09-11T09:50:47.211 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:47.211 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:47.211 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:50:47.211 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.495770+0000", 2024-09-11T09:50:47.211 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0011932469999999999 2024-09-11T09:50:47.211 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:47.211 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:47.211 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:50:47.211 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.496966+0000", 2024-09-11T09:50:47.211 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001196003 2024-09-11T09:50:47.211 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:47.211 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:47.212 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:50:47.212 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.497011+0000", 2024-09-11T09:50:47.212 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.5229999999999999e-05 2024-09-11T09:50:47.212 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:47.212 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:47.212 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:50:47.212 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.497044+0000", 2024-09-11T09:50:47.212 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.3322999999999999e-05 2024-09-11T09:50:47.212 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:47.212 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:50:47.212 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:47.213 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:47.213 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:47.213 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_repop_reply(client.4230.0:5 2.2 e17/15 ondisk, result = 0)", 2024-09-11T09:50:47.213 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:19.495669+0000", 2024-09-11T09:50:47.213 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 27.682257248999999, 2024-09-11T09:50:47.213 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000106431, 2024-09-11T09:50:47.213 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:50:47.213 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:50:47.213 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "started", 2024-09-11T09:50:47.213 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:50:47.213 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:47.213 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:50:47.213 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.495669+0000", 2024-09-11T09:50:47.213 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:47.214 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:47.214 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:47.214 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:50:47.214 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.495669+0000", 2024-09-11T09:50:47.214 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:47.214 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:47.214 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:47.214 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:50:47.214 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.495670+0000", 2024-09-11T09:50:47.214 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.218e-06 2024-09-11T09:50:47.215 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:47.215 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:47.215 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:50:47.215 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.495676+0000", 2024-09-11T09:50:47.215 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.5119999999999997e-06 2024-09-11T09:50:47.215 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:47.215 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:47.215 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:50:47.215 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.495690+0000", 2024-09-11T09:50:47.215 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.3312000000000001e-05 2024-09-11T09:50:47.216 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:47.216 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:47.216 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:50:47.216 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.495706+0000", 2024-09-11T09:50:47.216 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.6560000000000001e-05 2024-09-11T09:50:47.216 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:47.216 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:47.216 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:50:47.216 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.495759+0000", 2024-09-11T09:50:47.216 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.2828000000000003e-05 2024-09-11T09:50:47.217 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:47.217 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:47.217 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:50:47.217 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.495765+0000", 2024-09-11T09:50:47.217 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.7869999999999998e-06 2024-09-11T09:50:47.217 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:47.217 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:47.217 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:50:47.217 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.495775+0000", 2024-09-11T09:50:47.217 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.0214000000000001e-05 2024-09-11T09:50:47.217 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:47.217 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:50:47.217 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:47.218 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:47.218 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:47.218 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4230.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-09-11T09:50:47.218 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:20.498634+0000", 2024-09-11T09:50:47.218 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 26.679291636999999, 2024-09-11T09:50:47.218 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0050648539999999997, 2024-09-11T09:50:47.218 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:50:47.218 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:50:47.218 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:47.218 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:50:47.218 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4230", 2024-09-11T09:50:47.219 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.17:0/4101109576", 2024-09-11T09:50:47.219 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 7 2024-09-11T09:50:47.219 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:47.219 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:50:47.219 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:47.219 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:50:47.219 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.498634+0000", 2024-09-11T09:50:47.219 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:47.219 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:47.219 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:47.219 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:50:47.220 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.498634+0000", 2024-09-11T09:50:47.220 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:47.220 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:47.220 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:47.220 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:50:47.220 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.498637+0000", 2024-09-11T09:50:47.220 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.1919999999999999e-06 2024-09-11T09:50:47.220 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:47.220 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:47.220 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:50:47.220 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.498648+0000", 2024-09-11T09:50:47.221 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1116e-05 2024-09-11T09:50:47.221 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:47.221 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:47.221 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:50:47.221 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.498666+0000", 2024-09-11T09:50:47.221 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.8411999999999999e-05 2024-09-11T09:50:47.221 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:47.221 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:47.221 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:50:47.221 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.498675+0000", 2024-09-11T09:50:47.221 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.4059999999999994e-06 2024-09-11T09:50:47.221 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:47.221 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:47.221 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:50:47.222 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.498745+0000", 2024-09-11T09:50:47.222 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.9764000000000005e-05 2024-09-11T09:50:47.222 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:47.222 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:47.222 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op must wait for map", 2024-09-11T09:50:47.222 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.498754+0000", 2024-09-11T09:50:47.222 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.5969999999999992e-06 2024-09-11T09:50:47.222 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:47.222 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:47.222 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:50:47.222 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.500385+0000", 2024-09-11T09:50:47.222 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0016308360000000001 2024-09-11T09:50:47.222 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:47.223 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:47.223 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:50:47.223 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.500473+0000", 2024-09-11T09:50:47.223 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.7966000000000004e-05 2024-09-11T09:50:47.223 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:47.223 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:47.223 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 1", 2024-09-11T09:50:47.223 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.500591+0000", 2024-09-11T09:50:47.223 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000117895 2024-09-11T09:50:47.223 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:47.223 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:47.223 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:50:47.223 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.501685+0000", 2024-09-11T09:50:47.224 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001094886 2024-09-11T09:50:47.224 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:47.224 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:47.224 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:50:47.224 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.503627+0000", 2024-09-11T09:50:47.224 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001941199 2024-09-11T09:50:47.224 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:47.224 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:47.224 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:50:47.224 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.503667+0000", 2024-09-11T09:50:47.224 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.0645000000000001e-05 2024-09-11T09:50:47.225 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:47.225 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:47.225 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:50:47.225 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.503699+0000", 2024-09-11T09:50:47.225 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.1940000000000003e-05 2024-09-11T09:50:47.225 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:47.225 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:50:47.226 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:47.226 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:47.226 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:47.226 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_repop_reply(client.4230.0:7 2.2 e18/15 ondisk, result = 0)", 2024-09-11T09:50:47.226 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:20.501608+0000", 2024-09-11T09:50:47.226 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 26.676318115000001, 2024-09-11T09:50:47.226 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.2459000000000005e-05, 2024-09-11T09:50:47.226 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:50:47.226 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:50:47.226 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "started", 2024-09-11T09:50:47.227 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:50:47.227 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:47.227 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:50:47.227 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.501608+0000", 2024-09-11T09:50:47.227 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:47.227 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:47.227 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:47.227 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:50:47.227 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.501608+0000", 2024-09-11T09:50:47.227 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:47.228 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:47.228 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:47.228 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:50:47.228 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.501609+0000", 2024-09-11T09:50:47.228 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.0160000000000001e-06 2024-09-11T09:50:47.228 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:47.228 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:47.228 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:50:47.228 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.501613+0000", 2024-09-11T09:50:47.228 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.5959999999999997e-06 2024-09-11T09:50:47.228 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:47.228 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:47.228 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:50:47.229 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.501625+0000", 2024-09-11T09:50:47.229 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1888e-05 2024-09-11T09:50:47.229 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:47.229 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:47.229 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:50:47.229 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.501634+0000", 2024-09-11T09:50:47.229 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.6139999999999992e-06 2024-09-11T09:50:47.229 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:47.229 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:47.229 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:50:47.229 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.501676+0000", 2024-09-11T09:50:47.229 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.1953999999999998e-05 2024-09-11T09:50:47.229 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:47.230 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:47.230 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:50:47.230 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.501682+0000", 2024-09-11T09:50:47.230 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.8000000000000004e-06 2024-09-11T09:50:47.230 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:47.230 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:47.230 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:50:47.230 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.501690+0000", 2024-09-11T09:50:47.230 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.5909999999999996e-06 2024-09-11T09:50:47.230 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:47.230 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:50:47.230 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:47.230 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:47.231 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:50:47.231 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-11T09:50:47.232 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:50:47.233+0000 7fdec5849640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31907) UID: 0 2024-09-11T09:50:47.233 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-11T09:50:47.332 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:50:47.334+0000 7fdec5849640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31907) UID: 0 2024-09-11T09:50:47.349 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:50:47.349 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [], 2024-09-11T09:50:47.349 INFO:teuthology.orchestra.run.smithi017.stdout: "num_ops": 0 2024-09-11T09:50:47.349 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:50:47.361 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-11T09:50:47.433 INFO:tasks.ceph.osd.2.smithi017.stderr:2024-09-11T09:50:47.434+0000 7fd20b170640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31672) UID: 0 2024-09-11T09:50:47.469 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:50:47.469 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [], 2024-09-11T09:50:47.469 INFO:teuthology.orchestra.run.smithi017.stdout: "complaint_time": 30, 2024-09-11T09:50:47.469 INFO:teuthology.orchestra.run.smithi017.stdout: "num_blocked_ops": 0 2024-09-11T09:50:47.469 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:50:47.477 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-11T09:50:47.534 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:50:47.535+0000 7fdec5849640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31907) UID: 0 2024-09-11T09:50:47.601 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-11T09:50:47.630 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:50:47.631 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 20, 2024-09-11T09:50:47.631 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 600, 2024-09-11T09:50:47.631 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [ 2024-09-11T09:50:47.631 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.631 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4183.0:21 1.0 e13/12 1:30306672:devicehealth::main.db.0000000000000000:head v 13'11, mlcod=13'8)", 2024-09-11T09:50:47.631 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:11.491728+0000", 2024-09-11T09:50:47.631 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 36.137974843999999, 2024-09-11T09:50:47.631 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00093269800000000003, 2024-09-11T09:50:47.631 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:50:47.631 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:50:47.631 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:47.631 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:50:47.631 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.632 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:50:47.632 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.491728+0000", 2024-09-11T09:50:47.632 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:47.632 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:47.632 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.632 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:50:47.632 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.491728+0000", 2024-09-11T09:50:47.632 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:47.632 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:47.632 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.632 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:50:47.633 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.491729+0000", 2024-09-11T09:50:47.633 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.8599999999999997e-07 2024-09-11T09:50:47.633 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:47.633 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.633 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:50:47.633 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.491820+0000", 2024-09-11T09:50:47.633 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.0812999999999999e-05 2024-09-11T09:50:47.633 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:47.633 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.633 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:50:47.633 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.491833+0000", 2024-09-11T09:50:47.633 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.341e-05 2024-09-11T09:50:47.633 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:47.634 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.634 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:50:47.634 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.491841+0000", 2024-09-11T09:50:47.634 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.8730000000000007e-06 2024-09-11T09:50:47.634 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:47.634 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.634 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:50:47.634 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.491916+0000", 2024-09-11T09:50:47.634 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.4881000000000004e-05 2024-09-11T09:50:47.634 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:47.634 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.634 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:50:47.634 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.491931+0000", 2024-09-11T09:50:47.634 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.4825e-05 2024-09-11T09:50:47.634 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:47.635 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.635 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:50:47.635 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.492633+0000", 2024-09-11T09:50:47.635 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00070191600000000002 2024-09-11T09:50:47.635 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:47.635 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.635 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:50:47.635 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.492661+0000", 2024-09-11T09:50:47.635 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.8093999999999999e-05 2024-09-11T09:50:47.635 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:47.635 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:50:47.635 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:47.635 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:47.635 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.636 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4183.0:30 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'16, mlcod=13'14)", 2024-09-11T09:50:47.636 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:11.498866+0000", 2024-09-11T09:50:47.636 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 36.130837120999999, 2024-09-11T09:50:47.636 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00089294000000000005, 2024-09-11T09:50:47.636 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:50:47.636 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:50:47.636 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:47.636 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:50:47.636 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.636 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:50:47.637 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.498866+0000", 2024-09-11T09:50:47.637 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:47.637 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:47.637 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.637 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:50:47.637 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.498866+0000", 2024-09-11T09:50:47.638 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:47.638 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:47.638 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.638 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:50:47.638 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.498867+0000", 2024-09-11T09:50:47.638 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.7000000000000003e-07 2024-09-11T09:50:47.638 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:47.638 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.638 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:50:47.638 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.498873+0000", 2024-09-11T09:50:47.638 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.0140000000000004e-06 2024-09-11T09:50:47.638 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:47.638 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.639 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:50:47.639 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.498886+0000", 2024-09-11T09:50:47.639 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2745000000000001e-05 2024-09-11T09:50:47.639 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:47.639 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.639 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:50:47.639 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.498894+0000", 2024-09-11T09:50:47.639 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.2349999999999999e-06 2024-09-11T09:50:47.639 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:47.639 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.639 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:50:47.639 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.498941+0000", 2024-09-11T09:50:47.639 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.7679999999999998e-05 2024-09-11T09:50:47.640 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:47.640 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.640 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:50:47.640 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.498955+0000", 2024-09-11T09:50:47.640 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3446e-05 2024-09-11T09:50:47.640 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:47.640 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.640 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:50:47.640 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.499717+0000", 2024-09-11T09:50:47.640 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00076248199999999998 2024-09-11T09:50:47.640 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:47.640 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.640 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:50:47.641 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.499759+0000", 2024-09-11T09:50:47.641 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.1467999999999997e-05 2024-09-11T09:50:47.641 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:47.641 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:50:47.641 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:47.641 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:47.641 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.641 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4183.0:31 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'17, mlcod=13'14)", 2024-09-11T09:50:47.641 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:11.499319+0000", 2024-09-11T09:50:47.642 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 36.130384175000003, 2024-09-11T09:50:47.642 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0010073879999999999, 2024-09-11T09:50:47.642 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:50:47.642 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:50:47.642 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:47.642 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:50:47.642 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.642 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:50:47.642 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.499319+0000", 2024-09-11T09:50:47.642 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:47.642 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:47.642 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.643 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:50:47.643 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.499319+0000", 2024-09-11T09:50:47.643 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:47.643 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:47.643 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.643 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:50:47.643 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.499320+0000", 2024-09-11T09:50:47.643 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.7700000000000004e-07 2024-09-11T09:50:47.643 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:47.643 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.643 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:50:47.643 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.499326+0000", 2024-09-11T09:50:47.644 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.6420000000000001e-06 2024-09-11T09:50:47.644 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:47.644 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.644 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:50:47.644 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.499339+0000", 2024-09-11T09:50:47.644 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2537999999999999e-05 2024-09-11T09:50:47.644 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:47.644 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.644 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:50:47.644 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.499345+0000", 2024-09-11T09:50:47.644 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.8220000000000003e-06 2024-09-11T09:50:47.644 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:47.644 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.644 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:50:47.645 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.499403+0000", 2024-09-11T09:50:47.645 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.8814000000000003e-05 2024-09-11T09:50:47.645 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:47.645 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.645 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:50:47.645 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.499427+0000", 2024-09-11T09:50:47.645 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.3853999999999999e-05 2024-09-11T09:50:47.645 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:47.645 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.645 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:50:47.645 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.500294+0000", 2024-09-11T09:50:47.645 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00086622399999999997 2024-09-11T09:50:47.645 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:47.646 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.646 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:50:47.646 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.500326+0000", 2024-09-11T09:50:47.646 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.2716999999999999e-05 2024-09-11T09:50:47.646 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:47.646 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:50:47.646 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:47.646 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:47.646 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.646 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4183.0:32 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'18, mlcod=13'14)", 2024-09-11T09:50:47.646 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:11.499796+0000", 2024-09-11T09:50:47.646 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 36.129906968999997, 2024-09-11T09:50:47.646 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00092915100000000002, 2024-09-11T09:50:47.647 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:50:47.647 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:50:47.647 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:47.647 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:50:47.647 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.647 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:50:47.647 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.499796+0000", 2024-09-11T09:50:47.647 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:47.647 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:47.647 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.647 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:50:47.647 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.499796+0000", 2024-09-11T09:50:47.648 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:47.648 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:47.648 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.648 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:50:47.648 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.499797+0000", 2024-09-11T09:50:47.648 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.9400000000000004e-07 2024-09-11T09:50:47.648 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:47.648 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.648 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:50:47.648 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.499841+0000", 2024-09-11T09:50:47.648 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.3900000000000003e-05 2024-09-11T09:50:47.648 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:47.648 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.649 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:50:47.649 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.499852+0000", 2024-09-11T09:50:47.649 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1462e-05 2024-09-11T09:50:47.649 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:47.649 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.649 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:50:47.649 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.499858+0000", 2024-09-11T09:50:47.649 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.8320000000000002e-06 2024-09-11T09:50:47.649 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:47.649 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.649 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:50:47.649 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.499915+0000", 2024-09-11T09:50:47.650 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.7187e-05 2024-09-11T09:50:47.650 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:47.650 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.650 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:50:47.650 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.499930+0000", 2024-09-11T09:50:47.650 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.4928e-05 2024-09-11T09:50:47.650 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:47.650 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.650 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:50:47.650 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.500697+0000", 2024-09-11T09:50:47.650 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00076658399999999995 2024-09-11T09:50:47.650 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:47.650 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.651 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:50:47.651 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.500725+0000", 2024-09-11T09:50:47.651 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.8364000000000002e-05 2024-09-11T09:50:47.651 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:47.651 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:50:47.651 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:47.651 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:47.651 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.651 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4183.0:34 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'20, mlcod=13'15)", 2024-09-11T09:50:47.651 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:11.501016+0000", 2024-09-11T09:50:47.651 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 36.128687450999998, 2024-09-11T09:50:47.651 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00097815399999999996, 2024-09-11T09:50:47.651 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:50:47.652 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:50:47.652 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:47.652 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:50:47.652 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.652 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:50:47.652 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.501016+0000", 2024-09-11T09:50:47.652 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:47.652 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:47.652 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.652 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:50:47.652 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.501016+0000", 2024-09-11T09:50:47.652 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:47.652 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:47.653 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.653 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:50:47.653 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.501016+0000", 2024-09-11T09:50:47.653 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.3799999999999997e-07 2024-09-11T09:50:47.653 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:47.653 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.653 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:50:47.653 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.501023+0000", 2024-09-11T09:50:47.653 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.2180000000000002e-06 2024-09-11T09:50:47.653 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:47.654 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.654 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:50:47.654 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.501031+0000", 2024-09-11T09:50:47.654 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.5379999999999997e-06 2024-09-11T09:50:47.654 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:47.654 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.654 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:50:47.654 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.501035+0000", 2024-09-11T09:50:47.654 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.216e-06 2024-09-11T09:50:47.654 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:47.654 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.655 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:50:47.655 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.501105+0000", 2024-09-11T09:50:47.655 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.9757000000000003e-05 2024-09-11T09:50:47.655 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:47.655 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.655 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:50:47.655 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.501120+0000", 2024-09-11T09:50:47.655 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.5358000000000001e-05 2024-09-11T09:50:47.655 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:47.655 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.655 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:50:47.655 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.501955+0000", 2024-09-11T09:50:47.655 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00083505000000000005 2024-09-11T09:50:47.656 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:47.656 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.656 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:50:47.656 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.501994+0000", 2024-09-11T09:50:47.656 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.8479000000000002e-05 2024-09-11T09:50:47.656 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:47.656 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:50:47.656 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:47.656 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:47.656 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.656 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4183.0:38 1.0 e13/12 1:30306672:devicehealth::main.db.0000000000000000:head v 13'24, mlcod=13'22)", 2024-09-11T09:50:47.656 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:11.503891+0000", 2024-09-11T09:50:47.656 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 36.125812009000001, 2024-09-11T09:50:47.657 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000959351, 2024-09-11T09:50:47.657 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:50:47.657 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:50:47.657 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:47.657 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:50:47.657 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.657 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:50:47.657 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.503891+0000", 2024-09-11T09:50:47.657 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:47.657 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:47.657 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.657 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:50:47.657 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.503891+0000", 2024-09-11T09:50:47.658 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:47.658 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:47.658 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.658 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:50:47.658 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.503892+0000", 2024-09-11T09:50:47.658 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.5499999999999998e-07 2024-09-11T09:50:47.658 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:47.658 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.658 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:50:47.658 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.503967+0000", 2024-09-11T09:50:47.658 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.4993999999999994e-05 2024-09-11T09:50:47.658 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:47.658 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.659 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:50:47.659 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.503976+0000", 2024-09-11T09:50:47.659 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.4709999999999996e-06 2024-09-11T09:50:47.659 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:47.659 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.659 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:50:47.659 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.503981+0000", 2024-09-11T09:50:47.659 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.6789999999999996e-06 2024-09-11T09:50:47.659 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:47.659 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.659 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:50:47.659 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.504021+0000", 2024-09-11T09:50:47.659 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.0065999999999999e-05 2024-09-11T09:50:47.660 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:47.660 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.660 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:50:47.660 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.504036+0000", 2024-09-11T09:50:47.660 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.468e-05 2024-09-11T09:50:47.660 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:47.660 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.660 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:50:47.660 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.504813+0000", 2024-09-11T09:50:47.660 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00077718299999999998 2024-09-11T09:50:47.661 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:47.661 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.661 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:50:47.661 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.504850+0000", 2024-09-11T09:50:47.661 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.7722999999999998e-05 2024-09-11T09:50:47.661 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:47.661 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:50:47.661 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:47.661 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:47.661 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.661 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4183.0:54 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'32, mlcod=13'29)", 2024-09-11T09:50:47.661 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:11.515404+0000", 2024-09-11T09:50:47.661 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 36.114299404, 2024-09-11T09:50:47.662 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00089471799999999997, 2024-09-11T09:50:47.662 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:50:47.662 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:50:47.662 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:47.662 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:50:47.662 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.662 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:50:47.662 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.515404+0000", 2024-09-11T09:50:47.662 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:47.662 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:47.662 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.662 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:50:47.663 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.515404+0000", 2024-09-11T09:50:47.663 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:47.663 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:47.663 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.663 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:50:47.663 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.515404+0000", 2024-09-11T09:50:47.663 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.3799999999999996e-07 2024-09-11T09:50:47.663 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:47.663 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.663 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:50:47.663 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.515410+0000", 2024-09-11T09:50:47.663 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.1789999999999999e-06 2024-09-11T09:50:47.663 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:47.663 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.664 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:50:47.664 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.515422+0000", 2024-09-11T09:50:47.664 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1929999999999999e-05 2024-09-11T09:50:47.664 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:47.664 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.664 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:50:47.664 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.515427+0000", 2024-09-11T09:50:47.664 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.6060000000000002e-06 2024-09-11T09:50:47.664 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:47.664 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.664 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:50:47.664 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.515482+0000", 2024-09-11T09:50:47.664 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.4743000000000001e-05 2024-09-11T09:50:47.665 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:47.665 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.665 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:50:47.665 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.515498+0000", 2024-09-11T09:50:47.665 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.5778999999999999e-05 2024-09-11T09:50:47.665 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:47.665 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.665 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:50:47.665 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.516271+0000", 2024-09-11T09:50:47.665 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00077312599999999996 2024-09-11T09:50:47.666 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:47.666 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.666 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:50:47.666 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.516298+0000", 2024-09-11T09:50:47.666 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.7617e-05 2024-09-11T09:50:47.666 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:47.666 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:50:47.666 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:47.666 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:47.666 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.666 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4183.0:57 1.0 e13/12 1:30306672:devicehealth::main.db.0000000000000000:head v 13'35, mlcod=13'33)", 2024-09-11T09:50:47.666 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:11.519823+0000", 2024-09-11T09:50:47.666 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 36.109880347000001, 2024-09-11T09:50:47.667 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00090876100000000001, 2024-09-11T09:50:47.667 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:50:47.667 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:50:47.667 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:47.667 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:50:47.667 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.668 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:50:47.668 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.519823+0000", 2024-09-11T09:50:47.668 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:47.668 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:47.668 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.668 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:50:47.668 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.519823+0000", 2024-09-11T09:50:47.668 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:47.669 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:47.669 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.669 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:50:47.669 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.519823+0000", 2024-09-11T09:50:47.669 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.7000000000000004e-07 2024-09-11T09:50:47.669 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:47.669 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.669 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:50:47.670 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.519859+0000", 2024-09-11T09:50:47.670 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.5508e-05 2024-09-11T09:50:47.670 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:47.670 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.670 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:50:47.670 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.519871+0000", 2024-09-11T09:50:47.670 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.221e-05 2024-09-11T09:50:47.670 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:47.671 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.671 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:50:47.671 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.519877+0000", 2024-09-11T09:50:47.671 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.0410000000000003e-06 2024-09-11T09:50:47.671 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:47.671 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.671 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:50:47.672 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.519955+0000", 2024-09-11T09:50:47.672 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.8169000000000003e-05 2024-09-11T09:50:47.672 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:47.672 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.672 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:50:47.673 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.519972+0000", 2024-09-11T09:50:47.673 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6595999999999998e-05 2024-09-11T09:50:47.673 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:47.673 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.673 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:50:47.673 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.520691+0000", 2024-09-11T09:50:47.673 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000719405 2024-09-11T09:50:47.673 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:47.674 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.674 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:50:47.674 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.520731+0000", 2024-09-11T09:50:47.674 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.0061999999999997e-05 2024-09-11T09:50:47.674 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:47.674 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:50:47.674 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:47.674 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:47.675 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.675 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4183.0:58 1.0 e13/12 1:30306672:devicehealth::main.db.0000000000000000:head v 13'36, mlcod=13'33)", 2024-09-11T09:50:47.675 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:11.520285+0000", 2024-09-11T09:50:47.675 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 36.109417938999997, 2024-09-11T09:50:47.675 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00091014799999999995, 2024-09-11T09:50:47.675 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:50:47.676 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:50:47.676 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:47.676 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:50:47.676 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.676 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:50:47.676 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.520285+0000", 2024-09-11T09:50:47.677 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:47.677 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:47.677 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.677 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:50:47.677 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.520285+0000", 2024-09-11T09:50:47.677 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:47.677 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:47.677 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.678 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:50:47.678 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.520286+0000", 2024-09-11T09:50:47.678 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.4399999999999999e-07 2024-09-11T09:50:47.678 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:47.678 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.679 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:50:47.679 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.520348+0000", 2024-09-11T09:50:47.679 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.1760000000000005e-05 2024-09-11T09:50:47.679 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:47.679 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.679 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:50:47.679 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.520359+0000", 2024-09-11T09:50:47.679 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1362999999999999e-05 2024-09-11T09:50:47.679 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:47.679 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.680 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:50:47.680 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.520364+0000", 2024-09-11T09:50:47.680 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.3600000000000004e-06 2024-09-11T09:50:47.680 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:47.680 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.680 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:50:47.680 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.520413+0000", 2024-09-11T09:50:47.680 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.8226000000000002e-05 2024-09-11T09:50:47.680 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:47.680 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.681 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:50:47.681 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.520426+0000", 2024-09-11T09:50:47.681 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3560000000000001e-05 2024-09-11T09:50:47.681 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:47.681 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.681 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:50:47.682 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.521159+0000", 2024-09-11T09:50:47.682 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00073255499999999997 2024-09-11T09:50:47.682 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:47.682 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.682 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:50:47.682 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.521195+0000", 2024-09-11T09:50:47.682 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.6480000000000003e-05 2024-09-11T09:50:47.683 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:47.683 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:50:47.683 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:47.683 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:47.683 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.683 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4183.0:68 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'42, mlcod=13'39)", 2024-09-11T09:50:47.683 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:11.534957+0000", 2024-09-11T09:50:47.684 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 36.094746248, 2024-09-11T09:50:47.684 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.001065351, 2024-09-11T09:50:47.684 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:50:47.684 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:50:47.684 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:47.684 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:50:47.684 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.684 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:50:47.684 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.534957+0000", 2024-09-11T09:50:47.684 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:47.684 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:47.685 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.685 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:50:47.685 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.534957+0000", 2024-09-11T09:50:47.685 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:47.685 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:47.685 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.685 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:50:47.685 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.534957+0000", 2024-09-11T09:50:47.685 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.3799999999999998e-07 2024-09-11T09:50:47.685 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:47.685 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.685 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:50:47.685 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.534964+0000", 2024-09-11T09:50:47.686 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.9809999999999997e-06 2024-09-11T09:50:47.686 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:47.686 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.686 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:50:47.686 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.534976+0000", 2024-09-11T09:50:47.686 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2119999999999999e-05 2024-09-11T09:50:47.686 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:47.686 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.686 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:50:47.686 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.534980+0000", 2024-09-11T09:50:47.686 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.9890000000000003e-06 2024-09-11T09:50:47.686 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:47.686 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.687 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:50:47.687 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.535030+0000", 2024-09-11T09:50:47.687 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.9258000000000002e-05 2024-09-11T09:50:47.687 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:47.687 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.687 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:50:47.687 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.535044+0000", 2024-09-11T09:50:47.687 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3981e-05 2024-09-11T09:50:47.687 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:47.687 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.688 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:50:47.688 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.535995+0000", 2024-09-11T09:50:47.688 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000951804 2024-09-11T09:50:47.688 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:47.688 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.688 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:50:47.688 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.536022+0000", 2024-09-11T09:50:47.688 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.6780000000000001e-05 2024-09-11T09:50:47.688 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:47.688 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:50:47.689 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:47.689 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:47.689 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.689 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4183.0:69 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'43, mlcod=13'39)", 2024-09-11T09:50:47.689 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:11.535331+0000", 2024-09-11T09:50:47.689 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 36.094371924999997, 2024-09-11T09:50:47.689 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0011256090000000001, 2024-09-11T09:50:47.689 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:50:47.689 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:50:47.689 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:47.689 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:50:47.689 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.690 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:50:47.690 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.535331+0000", 2024-09-11T09:50:47.690 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:47.690 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:47.690 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.690 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:50:47.690 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.535331+0000", 2024-09-11T09:50:47.690 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:47.690 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:47.690 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.690 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:50:47.690 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.535332+0000", 2024-09-11T09:50:47.691 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.1600000000000001e-07 2024-09-11T09:50:47.691 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:47.691 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.691 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:50:47.691 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.535393+0000", 2024-09-11T09:50:47.691 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.1452999999999996e-05 2024-09-11T09:50:47.691 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:47.691 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.691 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:50:47.691 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.535402+0000", 2024-09-11T09:50:47.691 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.4300000000000006e-06 2024-09-11T09:50:47.691 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:47.692 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.692 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:50:47.692 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.535405+0000", 2024-09-11T09:50:47.692 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.512e-06 2024-09-11T09:50:47.692 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:47.692 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.692 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:50:47.693 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.535482+0000", 2024-09-11T09:50:47.693 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.7132999999999995e-05 2024-09-11T09:50:47.693 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:47.693 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.693 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:50:47.693 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.535506+0000", 2024-09-11T09:50:47.694 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.4070000000000002e-05 2024-09-11T09:50:47.694 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:47.694 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.694 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:50:47.694 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.536424+0000", 2024-09-11T09:50:47.694 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00091766700000000005 2024-09-11T09:50:47.694 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:47.694 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.694 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:50:47.695 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.536457+0000", 2024-09-11T09:50:47.695 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.2827999999999998e-05 2024-09-11T09:50:47.695 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:47.695 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:50:47.695 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:47.695 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:47.695 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.695 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4183.0:83 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'53, mlcod=13'50)", 2024-09-11T09:50:47.695 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:11.558836+0000", 2024-09-11T09:50:47.695 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 36.070866670000001, 2024-09-11T09:50:47.695 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00090313100000000005, 2024-09-11T09:50:47.695 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:50:47.695 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:50:47.696 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:47.696 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:50:47.696 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.696 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:50:47.696 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.558836+0000", 2024-09-11T09:50:47.696 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:47.696 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:47.696 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.696 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:50:47.696 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.558836+0000", 2024-09-11T09:50:47.696 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:47.696 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:47.696 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.697 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:50:47.697 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.558837+0000", 2024-09-11T09:50:47.697 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.4900000000000005e-07 2024-09-11T09:50:47.697 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:47.697 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.697 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:50:47.697 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.558857+0000", 2024-09-11T09:50:47.697 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.9502999999999999e-05 2024-09-11T09:50:47.697 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:47.697 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.697 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:50:47.698 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.558870+0000", 2024-09-11T09:50:47.698 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3020000000000001e-05 2024-09-11T09:50:47.698 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:47.698 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.698 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:50:47.698 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.558876+0000", 2024-09-11T09:50:47.698 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.7270000000000004e-06 2024-09-11T09:50:47.698 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:47.698 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.698 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:50:47.698 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.558934+0000", 2024-09-11T09:50:47.698 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.8100999999999998e-05 2024-09-11T09:50:47.698 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:47.699 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.699 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:50:47.699 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.558947+0000", 2024-09-11T09:50:47.699 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3269e-05 2024-09-11T09:50:47.699 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:47.700 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.700 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:50:47.700 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.559720+0000", 2024-09-11T09:50:47.700 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00077287600000000003 2024-09-11T09:50:47.700 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:47.700 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.700 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:50:47.700 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.559740+0000", 2024-09-11T09:50:47.700 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.9786e-05 2024-09-11T09:50:47.700 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:47.700 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:50:47.700 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:47.700 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:47.700 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.701 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4183.0:84 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'54, mlcod=13'50)", 2024-09-11T09:50:47.701 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:11.559267+0000", 2024-09-11T09:50:47.701 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 36.070436243000003, 2024-09-11T09:50:47.701 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00093990199999999995, 2024-09-11T09:50:47.701 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:50:47.701 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:50:47.701 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:47.701 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:50:47.701 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.701 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:50:47.701 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.559267+0000", 2024-09-11T09:50:47.701 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:47.701 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:47.702 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.702 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:50:47.702 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.559267+0000", 2024-09-11T09:50:47.702 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:47.702 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:47.702 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.702 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:50:47.702 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.559267+0000", 2024-09-11T09:50:47.702 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.2900000000000003e-07 2024-09-11T09:50:47.702 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:47.702 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.702 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:50:47.702 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.559322+0000", 2024-09-11T09:50:47.703 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.4429999999999999e-05 2024-09-11T09:50:47.703 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:47.703 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.703 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:50:47.703 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.559332+0000", 2024-09-11T09:50:47.703 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0473e-05 2024-09-11T09:50:47.703 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:47.703 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.703 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:50:47.703 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.559338+0000", 2024-09-11T09:50:47.703 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.0730000000000001e-06 2024-09-11T09:50:47.703 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:47.703 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.704 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:50:47.704 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.559373+0000", 2024-09-11T09:50:47.704 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.4626999999999999e-05 2024-09-11T09:50:47.704 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:47.704 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.704 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:50:47.704 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.559384+0000", 2024-09-11T09:50:47.704 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1307e-05 2024-09-11T09:50:47.704 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:47.704 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.704 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:50:47.704 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.560175+0000", 2024-09-11T09:50:47.705 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00079016399999999999 2024-09-11T09:50:47.705 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:47.705 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.705 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:50:47.705 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.560207+0000", 2024-09-11T09:50:47.705 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.2199000000000001e-05 2024-09-11T09:50:47.705 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:47.705 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:50:47.705 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:47.705 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:47.705 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.705 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4183.0:89 1.0 e13/12 1:30306672:devicehealth::main.db.0000000000000000:head v 13'59, mlcod=13'57)", 2024-09-11T09:50:47.705 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:11.563614+0000", 2024-09-11T09:50:47.706 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 36.066088622999999, 2024-09-11T09:50:47.706 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00091281399999999996, 2024-09-11T09:50:47.706 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:50:47.706 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:50:47.706 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:47.706 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:50:47.706 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.706 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:50:47.706 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.563614+0000", 2024-09-11T09:50:47.706 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:47.707 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:47.707 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.707 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:50:47.707 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.563614+0000", 2024-09-11T09:50:47.707 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:47.707 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:47.707 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.707 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:50:47.707 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.563615+0000", 2024-09-11T09:50:47.707 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.6899999999999998e-07 2024-09-11T09:50:47.707 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:47.708 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.708 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:50:47.708 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.563681+0000", 2024-09-11T09:50:47.708 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.5769e-05 2024-09-11T09:50:47.708 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:47.708 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.708 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:50:47.708 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.563690+0000", 2024-09-11T09:50:47.708 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.6800000000000005e-06 2024-09-11T09:50:47.708 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:47.708 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.708 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:50:47.709 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.563699+0000", 2024-09-11T09:50:47.709 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.3170000000000002e-06 2024-09-11T09:50:47.709 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:47.709 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.709 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:50:47.709 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.563734+0000", 2024-09-11T09:50:47.709 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.5023e-05 2024-09-11T09:50:47.709 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:47.709 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.709 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:50:47.709 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.563746+0000", 2024-09-11T09:50:47.709 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2661e-05 2024-09-11T09:50:47.710 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:47.710 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.710 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:50:47.710 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.564500+0000", 2024-09-11T09:50:47.710 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00075323100000000004 2024-09-11T09:50:47.710 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:47.710 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.710 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:50:47.710 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.564527+0000", 2024-09-11T09:50:47.710 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.7664000000000001e-05 2024-09-11T09:50:47.710 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:47.710 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:50:47.711 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:47.711 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:47.711 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.711 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4183.0:105 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'67, mlcod=13'64)", 2024-09-11T09:50:47.711 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:11.712146+0000", 2024-09-11T09:50:47.711 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 35.917556873000002, 2024-09-11T09:50:47.711 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0010916859999999999, 2024-09-11T09:50:47.711 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:50:47.711 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:50:47.711 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:47.711 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:50:47.711 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.712 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:50:47.712 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.712146+0000", 2024-09-11T09:50:47.712 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:47.712 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:47.712 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.712 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:50:47.712 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.712146+0000", 2024-09-11T09:50:47.712 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:47.712 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:47.712 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.712 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:50:47.713 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.712147+0000", 2024-09-11T09:50:47.713 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.8200000000000001e-07 2024-09-11T09:50:47.713 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:47.713 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.713 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:50:47.713 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.712155+0000", 2024-09-11T09:50:47.713 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.0309999999999998e-06 2024-09-11T09:50:47.713 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:47.713 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.713 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:50:47.713 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.712163+0000", 2024-09-11T09:50:47.713 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.5920000000000001e-06 2024-09-11T09:50:47.713 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:47.714 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.714 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:50:47.714 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.712169+0000", 2024-09-11T09:50:47.714 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.4299999999999997e-06 2024-09-11T09:50:47.714 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:47.714 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.714 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:50:47.714 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.712418+0000", 2024-09-11T09:50:47.714 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00024978499999999998 2024-09-11T09:50:47.714 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:47.714 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.714 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:50:47.714 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.712432+0000", 2024-09-11T09:50:47.714 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3190000000000001e-05 2024-09-11T09:50:47.715 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:47.715 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.715 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:50:47.715 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.713206+0000", 2024-09-11T09:50:47.715 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00077420199999999998 2024-09-11T09:50:47.715 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:47.715 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.715 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:50:47.715 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.713238+0000", 2024-09-11T09:50:47.715 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.2073999999999998e-05 2024-09-11T09:50:47.715 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:47.715 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:50:47.715 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:47.716 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:47.716 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.716 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4183.0:106 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'68, mlcod=13'64)", 2024-09-11T09:50:47.716 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:11.712624+0000", 2024-09-11T09:50:47.716 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 35.917079545, 2024-09-11T09:50:47.716 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0010009629999999999, 2024-09-11T09:50:47.716 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:50:47.716 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:50:47.716 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:47.716 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:50:47.716 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.716 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:50:47.716 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.712624+0000", 2024-09-11T09:50:47.717 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:47.717 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:47.717 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.717 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:50:47.717 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.712624+0000", 2024-09-11T09:50:47.717 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:47.717 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:47.717 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.717 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:50:47.717 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.712624+0000", 2024-09-11T09:50:47.717 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.4799999999999998e-07 2024-09-11T09:50:47.717 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:47.717 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.718 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:50:47.718 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.712689+0000", 2024-09-11T09:50:47.718 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.4488999999999999e-05 2024-09-11T09:50:47.718 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:47.718 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.718 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:50:47.718 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.712707+0000", 2024-09-11T09:50:47.718 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.8272000000000001e-05 2024-09-11T09:50:47.718 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:47.718 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.718 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:50:47.718 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.712713+0000", 2024-09-11T09:50:47.719 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.2040000000000002e-06 2024-09-11T09:50:47.719 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:47.719 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.719 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:50:47.719 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.712783+0000", 2024-09-11T09:50:47.719 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.9534000000000002e-05 2024-09-11T09:50:47.719 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:47.719 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.719 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:50:47.719 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.712811+0000", 2024-09-11T09:50:47.719 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.8620999999999999e-05 2024-09-11T09:50:47.719 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:47.719 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.720 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:50:47.720 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.713586+0000", 2024-09-11T09:50:47.720 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00077462600000000005 2024-09-11T09:50:47.720 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:47.720 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.720 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:50:47.720 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.713624+0000", 2024-09-11T09:50:47.720 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.8668999999999999e-05 2024-09-11T09:50:47.720 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:47.720 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:50:47.720 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:47.720 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:47.720 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.721 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4183.0:118 1.0 e13/12 1:30306672:devicehealth::main.db.0000000000000000:head v 13'80, mlcod=13'78)", 2024-09-11T09:50:47.721 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:11.726126+0000", 2024-09-11T09:50:47.721 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 35.903577452999997, 2024-09-11T09:50:47.721 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00087776200000000003, 2024-09-11T09:50:47.721 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:50:47.721 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:50:47.721 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:47.721 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:50:47.721 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.721 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:50:47.721 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.726126+0000", 2024-09-11T09:50:47.721 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:47.721 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:47.722 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.722 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:50:47.722 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.726126+0000", 2024-09-11T09:50:47.722 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:47.722 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:47.722 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.722 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:50:47.722 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.726126+0000", 2024-09-11T09:50:47.722 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.2099999999999996e-07 2024-09-11T09:50:47.722 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:47.722 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.722 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:50:47.722 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.726167+0000", 2024-09-11T09:50:47.722 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.0621999999999998e-05 2024-09-11T09:50:47.723 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:47.723 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.723 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:50:47.723 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.726177+0000", 2024-09-11T09:50:47.723 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0299000000000001e-05 2024-09-11T09:50:47.723 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:47.723 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.723 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:50:47.723 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.726183+0000", 2024-09-11T09:50:47.723 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.2079999999999997e-06 2024-09-11T09:50:47.723 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:47.724 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.724 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:50:47.724 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.726256+0000", 2024-09-11T09:50:47.724 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.2632000000000005e-05 2024-09-11T09:50:47.724 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:47.724 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.724 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:50:47.724 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.726268+0000", 2024-09-11T09:50:47.724 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2461e-05 2024-09-11T09:50:47.724 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:47.724 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.724 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:50:47.724 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.726977+0000", 2024-09-11T09:50:47.725 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00070853500000000005 2024-09-11T09:50:47.725 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:47.725 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.725 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:50:47.725 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.727003+0000", 2024-09-11T09:50:47.725 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.6384e-05 2024-09-11T09:50:47.725 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:47.725 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:50:47.725 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:47.725 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:47.725 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.725 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4183.0:134 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'92, mlcod=13'86)", 2024-09-11T09:50:47.725 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:12.035698+0000", 2024-09-11T09:50:47.725 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 35.594005285999998, 2024-09-11T09:50:47.726 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00089550700000000003, 2024-09-11T09:50:47.726 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:50:47.726 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:50:47.726 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:47.726 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:50:47.726 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.726 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:50:47.726 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.035698+0000", 2024-09-11T09:50:47.726 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:47.726 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:47.726 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.726 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:50:47.726 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.035698+0000", 2024-09-11T09:50:47.726 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:47.727 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:47.727 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.727 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:50:47.727 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.035698+0000", 2024-09-11T09:50:47.727 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.0900000000000002e-07 2024-09-11T09:50:47.727 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:47.727 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.727 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:50:47.727 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.035758+0000", 2024-09-11T09:50:47.727 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.9481999999999997e-05 2024-09-11T09:50:47.727 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:47.727 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.727 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:50:47.727 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.035766+0000", 2024-09-11T09:50:47.727 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.3760000000000008e-06 2024-09-11T09:50:47.728 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:47.728 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.728 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:50:47.728 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.035772+0000", 2024-09-11T09:50:47.728 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.8930000000000002e-06 2024-09-11T09:50:47.728 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:47.728 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.728 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:50:47.728 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.035807+0000", 2024-09-11T09:50:47.728 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.4975999999999999e-05 2024-09-11T09:50:47.728 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:47.728 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.728 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:50:47.729 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.035817+0000", 2024-09-11T09:50:47.729 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0314e-05 2024-09-11T09:50:47.729 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:47.729 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.729 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:50:47.729 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.036542+0000", 2024-09-11T09:50:47.729 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00072434400000000003 2024-09-11T09:50:47.729 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:47.729 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.730 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:50:47.730 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.036593+0000", 2024-09-11T09:50:47.730 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.1613000000000002e-05 2024-09-11T09:50:47.730 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:47.730 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:50:47.730 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:47.730 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:47.730 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.730 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4183.0:136 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'94, mlcod=13'86)", 2024-09-11T09:50:47.730 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:12.036591+0000", 2024-09-11T09:50:47.731 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 35.593112075999997, 2024-09-11T09:50:47.731 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00090637900000000002, 2024-09-11T09:50:47.731 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:50:47.731 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:50:47.731 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:47.731 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:50:47.731 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.731 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:50:47.731 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.036591+0000", 2024-09-11T09:50:47.731 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:47.731 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:47.731 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.731 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:50:47.732 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.036591+0000", 2024-09-11T09:50:47.732 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:47.732 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:47.732 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.732 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:50:47.732 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.036592+0000", 2024-09-11T09:50:47.732 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.37e-07 2024-09-11T09:50:47.732 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:47.732 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.732 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:50:47.732 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.036600+0000", 2024-09-11T09:50:47.732 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.6910000000000003e-06 2024-09-11T09:50:47.732 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:47.733 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.733 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:50:47.733 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.036609+0000", 2024-09-11T09:50:47.733 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.0229999999999998e-06 2024-09-11T09:50:47.733 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:47.733 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.733 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:50:47.733 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.036617+0000", 2024-09-11T09:50:47.733 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.6979999999999998e-06 2024-09-11T09:50:47.733 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:47.733 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.733 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:50:47.733 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.036658+0000", 2024-09-11T09:50:47.734 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.0692000000000003e-05 2024-09-11T09:50:47.734 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:47.734 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.734 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:50:47.734 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.036670+0000", 2024-09-11T09:50:47.734 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2109e-05 2024-09-11T09:50:47.734 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:47.734 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.734 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:50:47.734 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.037475+0000", 2024-09-11T09:50:47.734 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00080487199999999999 2024-09-11T09:50:47.734 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:47.735 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.735 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:50:47.735 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.037497+0000", 2024-09-11T09:50:47.735 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.2557000000000001e-05 2024-09-11T09:50:47.735 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:47.735 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:50:47.735 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:47.735 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:47.735 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.735 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4183.0:158 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'112, mlcod=13'108)", 2024-09-11T09:50:47.735 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:12.455073+0000", 2024-09-11T09:50:47.735 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 35.174629822999997, 2024-09-11T09:50:47.735 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00090654500000000003, 2024-09-11T09:50:47.736 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:50:47.736 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:50:47.736 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:47.736 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:50:47.736 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.736 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:50:47.736 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.455073+0000", 2024-09-11T09:50:47.736 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:47.736 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:47.736 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.736 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:50:47.736 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.455073+0000", 2024-09-11T09:50:47.737 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:47.737 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:47.737 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.737 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:50:47.737 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.455074+0000", 2024-09-11T09:50:47.737 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.32e-07 2024-09-11T09:50:47.737 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:47.737 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.737 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:50:47.737 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.455126+0000", 2024-09-11T09:50:47.737 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.1898999999999999e-05 2024-09-11T09:50:47.737 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:47.737 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.738 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:50:47.738 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.455136+0000", 2024-09-11T09:50:47.738 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0651e-05 2024-09-11T09:50:47.738 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:47.738 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.738 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:50:47.738 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.455142+0000", 2024-09-11T09:50:47.738 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.0020000000000003e-06 2024-09-11T09:50:47.738 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:47.738 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.738 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:50:47.738 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.455192+0000", 2024-09-11T09:50:47.739 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.0164000000000003e-05 2024-09-11T09:50:47.739 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:47.739 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.739 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:50:47.739 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.455204+0000", 2024-09-11T09:50:47.739 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1178999999999999e-05 2024-09-11T09:50:47.739 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:47.739 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.739 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:50:47.739 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.455957+0000", 2024-09-11T09:50:47.739 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00075362300000000003 2024-09-11T09:50:47.739 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:47.739 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:47.740 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:50:47.740 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.455980+0000", 2024-09-11T09:50:47.740 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.2595e-05 2024-09-11T09:50:47.740 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:47.740 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:50:47.740 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:47.740 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:47.740 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:50:47.740 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:50:47.742 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-11T09:50:47.742 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:50:47.742+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:50:47.843 INFO:tasks.ceph.osd.2.smithi017.stderr:2024-09-11T09:50:47.844+0000 7fd20b170640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31672) UID: 0 2024-09-11T09:50:47.890 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:50:47.891 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [], 2024-09-11T09:50:47.891 INFO:teuthology.orchestra.run.smithi017.stdout: "num_ops": 0 2024-09-11T09:50:47.891 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:50:47.900 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-11T09:50:47.944 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:50:47.945+0000 7f3c96ec9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31909) UID: 0 2024-09-11T09:50:48.044 INFO:tasks.ceph.osd.0.smithi017.stderr:2024-09-11T09:50:48.045+0000 7fb73c0f1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31665) UID: 0 2024-09-11T09:50:48.048 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:50:48.048 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [], 2024-09-11T09:50:48.048 INFO:teuthology.orchestra.run.smithi017.stdout: "complaint_time": 30, 2024-09-11T09:50:48.048 INFO:teuthology.orchestra.run.smithi017.stdout: "num_blocked_ops": 0 2024-09-11T09:50:48.048 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:50:48.057 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-11T09:50:48.146 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-09-11T09:50:48.147+0000 7f7b6c7d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31904) UID: 0 2024-09-11T09:50:48.204 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:50:48.204 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 20, 2024-09-11T09:50:48.204 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 600, 2024-09-11T09:50:48.204 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [ 2024-09-11T09:50:48.204 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:48.204 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4230.0:5 2.2 e17/15 2:5cea7035:::rbd_info:head v 17'1, mlcod=0'0)", 2024-09-11T09:50:48.205 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:19.494757+0000", 2024-09-11T09:50:48.205 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 28.710904187000001, 2024-09-11T09:50:48.205 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00082098399999999997, 2024-09-11T09:50:48.205 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:50:48.205 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:50:48.205 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:48.205 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:50:48.205 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:48.205 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:50:48.205 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:19.494757+0000", 2024-09-11T09:50:48.205 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:48.205 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:48.205 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:48.205 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:50:48.206 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:19.494757+0000", 2024-09-11T09:50:48.206 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:48.206 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:48.206 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:48.206 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:50:48.206 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:19.494757+0000", 2024-09-11T09:50:48.206 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.97e-07 2024-09-11T09:50:48.206 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:48.206 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:48.206 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:50:48.206 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:19.494763+0000", 2024-09-11T09:50:48.206 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.3709999999999996e-06 2024-09-11T09:50:48.206 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:48.206 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:48.207 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:50:48.207 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:19.494778+0000", 2024-09-11T09:50:48.207 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.4960000000000001e-05 2024-09-11T09:50:48.207 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:48.207 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:48.207 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:50:48.207 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:19.494801+0000", 2024-09-11T09:50:48.207 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.2388999999999999e-05 2024-09-11T09:50:48.207 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:48.207 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:48.207 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:50:48.207 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:19.494859+0000", 2024-09-11T09:50:48.207 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.7961999999999998e-05 2024-09-11T09:50:48.208 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:48.208 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:48.208 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:50:48.208 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:19.494883+0000", 2024-09-11T09:50:48.208 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.4488999999999999e-05 2024-09-11T09:50:48.208 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:48.208 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:48.208 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:50:48.208 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:19.495534+0000", 2024-09-11T09:50:48.208 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00065075500000000004 2024-09-11T09:50:48.208 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:48.208 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:48.208 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:50:48.208 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:19.495578+0000", 2024-09-11T09:50:48.209 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.3860999999999999e-05 2024-09-11T09:50:48.209 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:48.209 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:50:48.209 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:48.209 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:48.209 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:48.209 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4230.0:7 2.2 e18/15 2:5cea7035:::rbd_info:head v 18'2, mlcod=0'0)", 2024-09-11T09:50:48.209 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:20.500782+0000", 2024-09-11T09:50:48.209 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 27.704879306999999, 2024-09-11T09:50:48.209 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000790185, 2024-09-11T09:50:48.209 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:50:48.209 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:50:48.209 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:48.209 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:50:48.210 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:48.210 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:50:48.210 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:20.500782+0000", 2024-09-11T09:50:48.210 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:48.210 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:48.210 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:48.210 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:50:48.210 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:20.500782+0000", 2024-09-11T09:50:48.210 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:48.210 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:48.210 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:48.210 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:50:48.210 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:20.500782+0000", 2024-09-11T09:50:48.210 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.3299999999999998e-07 2024-09-11T09:50:48.211 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:48.211 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:48.211 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:50:48.211 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:20.500799+0000", 2024-09-11T09:50:48.211 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6830999999999999e-05 2024-09-11T09:50:48.211 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:48.211 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:48.211 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:50:48.211 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:20.500812+0000", 2024-09-11T09:50:48.211 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3013999999999999e-05 2024-09-11T09:50:48.211 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:48.211 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:48.211 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:50:48.211 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:20.500824+0000", 2024-09-11T09:50:48.212 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2593e-05 2024-09-11T09:50:48.212 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:48.212 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:48.212 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:50:48.212 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:20.500895+0000", 2024-09-11T09:50:48.212 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.1018000000000003e-05 2024-09-11T09:50:48.212 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:48.212 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:48.212 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:50:48.212 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:20.500912+0000", 2024-09-11T09:50:48.212 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6484e-05 2024-09-11T09:50:48.212 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:48.212 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:48.213 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:50:48.213 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:20.501531+0000", 2024-09-11T09:50:48.213 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00061903399999999997 2024-09-11T09:50:48.213 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:48.213 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:48.213 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:50:48.213 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:20.501572+0000", 2024-09-11T09:50:48.213 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.0877999999999997e-05 2024-09-11T09:50:48.213 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:48.213 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:50:48.213 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:48.213 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:48.213 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:50:48.214 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:50:48.215 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-11T09:50:48.260 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:50:48.261+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:50:48.322 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:50:48.322 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [], 2024-09-11T09:50:48.322 INFO:teuthology.orchestra.run.smithi017.stdout: "num_ops": 0 2024-09-11T09:50:48.322 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:50:48.331 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-11T09:50:48.347 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:50:48.348+0000 7fdec5849640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31907) UID: 0 2024-09-11T09:50:48.448 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-09-11T09:50:48.449+0000 7f7b6c7d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31904) UID: 0 2024-09-11T09:50:48.478 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:50:48.478 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [], 2024-09-11T09:50:48.478 INFO:teuthology.orchestra.run.smithi017.stdout: "complaint_time": 30, 2024-09-11T09:50:48.478 INFO:teuthology.orchestra.run.smithi017.stdout: "num_blocked_ops": 0 2024-09-11T09:50:48.478 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:50:48.487 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-11T09:50:48.549 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:50:48.550+0000 7f3c96ec9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31909) UID: 0 2024-09-11T09:50:48.594 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:50:48.594 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 20, 2024-09-11T09:50:48.594 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 600, 2024-09-11T09:50:48.595 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [ 2024-09-11T09:50:48.595 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:48.595 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4230.0:1 2.3 e16/15 2:c4c92e5a:::rbd_trash:head v 16'1, mlcod=0'0)", 2024-09-11T09:50:48.595 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:18.494378+0000", 2024-09-11T09:50:48.595 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 30.101242389999999, 2024-09-11T09:50:48.595 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00089752199999999999, 2024-09-11T09:50:48.595 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:50:48.595 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:50:48.595 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:48.595 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:50:48.595 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:48.595 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:50:48.595 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:18.494378+0000", 2024-09-11T09:50:48.595 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:48.596 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:48.596 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:48.596 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:50:48.596 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:18.494378+0000", 2024-09-11T09:50:48.596 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:48.596 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:48.596 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:48.596 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:50:48.596 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:18.494378+0000", 2024-09-11T09:50:48.596 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.9099999999999999e-07 2024-09-11T09:50:48.596 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:48.596 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:48.596 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:50:48.597 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:18.494385+0000", 2024-09-11T09:50:48.597 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.8129999999999998e-06 2024-09-11T09:50:48.597 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:48.597 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:48.597 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:50:48.597 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:18.494416+0000", 2024-09-11T09:50:48.597 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.0654000000000002e-05 2024-09-11T09:50:48.597 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:48.597 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:48.597 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:50:48.597 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:18.494430+0000", 2024-09-11T09:50:48.597 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3754000000000001e-05 2024-09-11T09:50:48.597 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:48.597 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:48.598 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:50:48.598 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:18.494524+0000", 2024-09-11T09:50:48.598 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.4896999999999996e-05 2024-09-11T09:50:48.598 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:48.598 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:48.598 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:50:48.598 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:18.494558+0000", 2024-09-11T09:50:48.598 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.3900999999999999e-05 2024-09-11T09:50:48.598 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:48.598 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:48.598 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:50:48.598 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:18.495231+0000", 2024-09-11T09:50:48.598 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00067233500000000003 2024-09-11T09:50:48.599 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:48.599 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:48.599 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:50:48.599 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:18.495275+0000", 2024-09-11T09:50:48.599 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.4777000000000002e-05 2024-09-11T09:50:48.599 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:48.599 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:50:48.599 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:48.599 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:48.599 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:50:48.599 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:50:48.600 INFO:teuthology.orchestra.run.smithi017.stderr:nodeep-scrub is unset 2024-09-11T09:50:48.605 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-11T09:50:48.650 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:50:48.651+0000 7f3c96ec9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31909) UID: 0 2024-09-11T09:50:48.720 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-11T09:50:48.720 INFO:teuthology.orchestra.run.smithi096.stdout: "ops": [], 2024-09-11T09:50:48.720 INFO:teuthology.orchestra.run.smithi096.stdout: "num_ops": 0 2024-09-11T09:50:48.720 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-11T09:50:48.727 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T09:50:48.729 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-11T09:50:48.772 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:50:48.773+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:50:48.852 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-09-11T09:50:48.852+0000 7f7b6c7d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31904) UID: 0 2024-09-11T09:50:48.853 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-11T09:50:48.853 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T09:50:48.870 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-11T09:50:48.871 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T09:50:48.874 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-11T09:50:48.874 INFO:teuthology.orchestra.run.smithi096.stdout: "ops": [], 2024-09-11T09:50:48.875 INFO:teuthology.orchestra.run.smithi096.stdout: "complaint_time": 30, 2024-09-11T09:50:48.875 INFO:teuthology.orchestra.run.smithi096.stdout: "num_blocked_ops": 0 2024-09-11T09:50:48.875 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-11T09:50:48.884 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-11T09:50:48.886 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-11T09:50:48.886 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T09:50:48.903 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-11T09:50:48.903 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T09:50:48.919 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-11T09:50:48.919 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T09:50:48.936 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-11T09:50:48.936 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T09:50:48.968 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:50:48.969+0000 7fdec5849640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31907) UID: 0 2024-09-11T09:50:49.028 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:50:49.028 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:50:49.036 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-11T09:50:49.077 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:50:49.078+0000 7fdec5849640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31907) UID: 0 2024-09-11T09:50:49.156 INFO:tasks.ceph.osd.0.smithi017.stderr:2024-09-11T09:50:49.156+0000 7fb73c0f1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31665) UID: 0 2024-09-11T09:50:49.179 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:50:49.179 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:50:49.187 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-11T09:50:49.256 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:50:49.257+0000 7fdec5849640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31907) UID: 0 2024-09-11T09:50:49.290 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:50:49.290 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:50:49.299 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-11T09:50:49.357 INFO:tasks.ceph.osd.0.smithi017.stderr:2024-09-11T09:50:49.358+0000 7fb73c0f1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31665) UID: 0 2024-09-11T09:50:49.442 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:50:49.442 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:50:49.451 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-11T09:50:49.458 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:50:49.459+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:50:49.553 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:50:49.553 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:50:49.559 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:50:49.559+0000 7f3c96ec9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31909) UID: 0 2024-09-11T09:50:49.562 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-11T09:50:49.660 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:50:49.660+0000 7fdec5849640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31907) UID: 0 2024-09-11T09:50:49.706 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:50:49.706 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:50:49.715 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-11T09:50:49.760 INFO:tasks.ceph.osd.2.smithi017.stderr:2024-09-11T09:50:49.761+0000 7fd20b170640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31672) UID: 0 2024-09-11T09:50:49.858 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:50:49.858 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:50:49.861 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:50:49.862+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:50:49.867 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-11T09:50:49.961 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:50:49.962+0000 7fdec5849640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31907) UID: 0 2024-09-11T09:50:49.988 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:50:49.988 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:50:49.997 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-11T09:50:50.062 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:50:50.063+0000 7fdec5849640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31907) UID: 0 2024-09-11T09:50:50.105 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:50:50.105 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:50:50.114 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-11T09:50:50.163 INFO:tasks.ceph.osd.0.smithi017.stderr:2024-09-11T09:50:50.164+0000 7fb73c0f1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31665) UID: 0 2024-09-11T09:50:50.222 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:50:50.222 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:50:50.232 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-11T09:50:50.273 INFO:tasks.ceph.osd.0.smithi017.stderr:2024-09-11T09:50:50.274+0000 7fb73c0f1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31665) UID: 0 2024-09-11T09:50:50.365 INFO:tasks.ceph.osd.2.smithi017.stderr:2024-09-11T09:50:50.366+0000 7fd20b170640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31672) UID: 0 2024-09-11T09:50:50.379 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:50:50.379 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:50:50.388 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-11T09:50:50.472 INFO:tasks.ceph.osd.0.smithi017.stderr:2024-09-11T09:50:50.473+0000 7fb73c0f1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31665) UID: 0 2024-09-11T09:50:50.536 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:50:50.536 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:50:50.545 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-11T09:50:50.586 INFO:tasks.ceph.osd.0.smithi017.stderr:2024-09-11T09:50:50.587+0000 7fb73c0f1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31665) UID: 0 2024-09-11T09:50:50.602 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 5, 0] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5] 2024-09-11T09:50:50.602 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-11T09:50:50.602 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-09-11T09:50:50.602 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump -f json-pretty 2024-09-11T09:50:50.611 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-11T09:50:50.673 INFO:tasks.ceph.osd.2.smithi017.stderr:2024-09-11T09:50:50.674+0000 7fd20b170640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31672) UID: 0 2024-09-11T09:50:50.698 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:50:50.736 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:50:50.738 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-11T09:50:50.768 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-09-11T09:50:50.769+0000 7f7b6c7d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31904) UID: 0 2024-09-11T09:50:50.869 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:50:50.870+0000 7fdec5849640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31907) UID: 0 2024-09-11T09:50:50.887 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:50:50.887 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:50:50.896 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-11T09:50:50.929 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-11T09:50:50.929 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:50:50.929 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 36, 2024-09-11T09:50:50.929 INFO:teuthology.orchestra.run.smithi017.stdout: "fsid": "a2f1daec-d3b0-4bf5-82c8-72ecbb93d4e4", 2024-09-11T09:50:50.929 INFO:teuthology.orchestra.run.smithi017.stdout: "created": "2024-09-11T09:50:00.165804+0000", 2024-09-11T09:50:50.930 INFO:teuthology.orchestra.run.smithi017.stdout: "modified": "2024-09-11T09:50:49.597656+0000", 2024-09-11T09:50:50.930 INFO:teuthology.orchestra.run.smithi017.stdout: "last_up_change": "2024-09-11T09:50:10.457262+0000", 2024-09-11T09:50:50.930 INFO:teuthology.orchestra.run.smithi017.stdout: "last_in_change": "2024-09-11T09:50:45.581301+0000", 2024-09-11T09:50:50.930 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-11T09:50:50.930 INFO:teuthology.orchestra.run.smithi017.stdout: "flags_num": 5799936, 2024-09-11T09:50:50.930 INFO:teuthology.orchestra.run.smithi017.stdout: "flags_set": [ 2024-09-11T09:50:50.930 INFO:teuthology.orchestra.run.smithi017.stdout: "pglog_hardlimit", 2024-09-11T09:50:50.930 INFO:teuthology.orchestra.run.smithi017.stdout: "purged_snapdirs", 2024-09-11T09:50:50.930 INFO:teuthology.orchestra.run.smithi017.stdout: "recovery_deletes", 2024-09-11T09:50:50.930 INFO:teuthology.orchestra.run.smithi017.stdout: "sortbitwise" 2024-09-11T09:50:50.930 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-11T09:50:50.930 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_version": 5, 2024-09-11T09:50:50.930 INFO:teuthology.orchestra.run.smithi017.stdout: "full_ratio": 0.94999998807907104, 2024-09-11T09:50:50.930 INFO:teuthology.orchestra.run.smithi017.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-11T09:50:50.931 INFO:teuthology.orchestra.run.smithi017.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-11T09:50:50.931 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_snapshot": "", 2024-09-11T09:50:50.931 INFO:teuthology.orchestra.run.smithi017.stdout: "pool_max": 2, 2024-09-11T09:50:50.931 INFO:teuthology.orchestra.run.smithi017.stdout: "max_osd": 6, 2024-09-11T09:50:50.931 INFO:teuthology.orchestra.run.smithi017.stdout: "require_min_compat_client": "luminous", 2024-09-11T09:50:50.931 INFO:teuthology.orchestra.run.smithi017.stdout: "min_compat_client": "jewel", 2024-09-11T09:50:50.931 INFO:teuthology.orchestra.run.smithi017.stdout: "require_osd_release": "squid", 2024-09-11T09:50:50.931 INFO:teuthology.orchestra.run.smithi017.stdout: "allow_crimson": false, 2024-09-11T09:50:50.931 INFO:teuthology.orchestra.run.smithi017.stdout: "pools": [ 2024-09-11T09:50:50.931 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:50.931 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 1, 2024-09-11T09:50:50.931 INFO:teuthology.orchestra.run.smithi017.stdout: "pool_name": ".mgr", 2024-09-11T09:50:50.932 INFO:teuthology.orchestra.run.smithi017.stdout: "create_time": "2024-09-11T09:50:10.457184+0000", 2024-09-11T09:50:50.932 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 1, 2024-09-11T09:50:50.932 INFO:teuthology.orchestra.run.smithi017.stdout: "flags_names": "hashpspool", 2024-09-11T09:50:50.932 INFO:teuthology.orchestra.run.smithi017.stdout: "type": 1, 2024-09-11T09:50:50.932 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 2, 2024-09-11T09:50:50.932 INFO:teuthology.orchestra.run.smithi017.stdout: "min_size": 1, 2024-09-11T09:50:50.932 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_rule": 0, 2024-09-11T09:50:50.932 INFO:teuthology.orchestra.run.smithi017.stdout: "peering_crush_bucket_count": 0, 2024-09-11T09:50:50.932 INFO:teuthology.orchestra.run.smithi017.stdout: "peering_crush_bucket_target": 0, 2024-09-11T09:50:50.932 INFO:teuthology.orchestra.run.smithi017.stdout: "peering_crush_bucket_barrier": 0, 2024-09-11T09:50:50.932 INFO:teuthology.orchestra.run.smithi017.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-11T09:50:50.932 INFO:teuthology.orchestra.run.smithi017.stdout: "is_stretch_pool": false, 2024-09-11T09:50:50.932 INFO:teuthology.orchestra.run.smithi017.stdout: "object_hash": 2, 2024-09-11T09:50:50.932 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_autoscale_mode": "off", 2024-09-11T09:50:50.933 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_num": 1, 2024-09-11T09:50:50.933 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_placement_num": 1, 2024-09-11T09:50:50.933 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_placement_num_target": 1, 2024-09-11T09:50:50.933 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_num_target": 1, 2024-09-11T09:50:50.933 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_num_pending": 1, 2024-09-11T09:50:50.933 INFO:teuthology.orchestra.run.smithi017.stdout: "last_pg_merge_meta": { 2024-09-11T09:50:50.933 INFO:teuthology.orchestra.run.smithi017.stdout: "source_pgid": "0.0", 2024-09-11T09:50:50.933 INFO:teuthology.orchestra.run.smithi017.stdout: "ready_epoch": 0, 2024-09-11T09:50:50.933 INFO:teuthology.orchestra.run.smithi017.stdout: "last_epoch_started": 0, 2024-09-11T09:50:50.933 INFO:teuthology.orchestra.run.smithi017.stdout: "last_epoch_clean": 0, 2024-09-11T09:50:50.933 INFO:teuthology.orchestra.run.smithi017.stdout: "source_version": "0'0", 2024-09-11T09:50:50.933 INFO:teuthology.orchestra.run.smithi017.stdout: "target_version": "0'0" 2024-09-11T09:50:50.934 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:50.934 INFO:teuthology.orchestra.run.smithi017.stdout: "last_change": "14", 2024-09-11T09:50:50.934 INFO:teuthology.orchestra.run.smithi017.stdout: "last_force_op_resend": "0", 2024-09-11T09:50:50.934 INFO:teuthology.orchestra.run.smithi017.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-11T09:50:50.934 INFO:teuthology.orchestra.run.smithi017.stdout: "last_force_op_resend_preluminous": "0", 2024-09-11T09:50:50.934 INFO:teuthology.orchestra.run.smithi017.stdout: "auid": 0, 2024-09-11T09:50:50.934 INFO:teuthology.orchestra.run.smithi017.stdout: "snap_mode": "selfmanaged", 2024-09-11T09:50:50.934 INFO:teuthology.orchestra.run.smithi017.stdout: "snap_seq": 0, 2024-09-11T09:50:50.934 INFO:teuthology.orchestra.run.smithi017.stdout: "snap_epoch": 0, 2024-09-11T09:50:50.934 INFO:teuthology.orchestra.run.smithi017.stdout: "pool_snaps": [], 2024-09-11T09:50:50.934 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_snaps": "[]", 2024-09-11T09:50:50.934 INFO:teuthology.orchestra.run.smithi017.stdout: "quota_max_bytes": 0, 2024-09-11T09:50:50.934 INFO:teuthology.orchestra.run.smithi017.stdout: "quota_max_objects": 0, 2024-09-11T09:50:50.935 INFO:teuthology.orchestra.run.smithi017.stdout: "tiers": [], 2024-09-11T09:50:50.935 INFO:teuthology.orchestra.run.smithi017.stdout: "tier_of": -1, 2024-09-11T09:50:50.935 INFO:teuthology.orchestra.run.smithi017.stdout: "read_tier": -1, 2024-09-11T09:50:50.935 INFO:teuthology.orchestra.run.smithi017.stdout: "write_tier": -1, 2024-09-11T09:50:50.935 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_mode": "none", 2024-09-11T09:50:50.935 INFO:teuthology.orchestra.run.smithi017.stdout: "target_max_bytes": 0, 2024-09-11T09:50:50.935 INFO:teuthology.orchestra.run.smithi017.stdout: "target_max_objects": 0, 2024-09-11T09:50:50.935 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-11T09:50:50.935 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-11T09:50:50.935 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-11T09:50:50.935 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_min_flush_age": 0, 2024-09-11T09:50:50.935 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_min_evict_age": 0, 2024-09-11T09:50:50.935 INFO:teuthology.orchestra.run.smithi017.stdout: "erasure_code_profile": "", 2024-09-11T09:50:50.935 INFO:teuthology.orchestra.run.smithi017.stdout: "hit_set_params": { 2024-09-11T09:50:50.936 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "none" 2024-09-11T09:50:50.936 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:50.936 INFO:teuthology.orchestra.run.smithi017.stdout: "hit_set_period": 0, 2024-09-11T09:50:50.936 INFO:teuthology.orchestra.run.smithi017.stdout: "hit_set_count": 0, 2024-09-11T09:50:50.936 INFO:teuthology.orchestra.run.smithi017.stdout: "use_gmt_hitset": true, 2024-09-11T09:50:50.936 INFO:teuthology.orchestra.run.smithi017.stdout: "min_read_recency_for_promote": 0, 2024-09-11T09:50:50.936 INFO:teuthology.orchestra.run.smithi017.stdout: "min_write_recency_for_promote": 0, 2024-09-11T09:50:50.936 INFO:teuthology.orchestra.run.smithi017.stdout: "hit_set_grade_decay_rate": 0, 2024-09-11T09:50:50.936 INFO:teuthology.orchestra.run.smithi017.stdout: "hit_set_search_last_n": 0, 2024-09-11T09:50:50.936 INFO:teuthology.orchestra.run.smithi017.stdout: "grade_table": [], 2024-09-11T09:50:50.936 INFO:teuthology.orchestra.run.smithi017.stdout: "stripe_width": 0, 2024-09-11T09:50:50.936 INFO:teuthology.orchestra.run.smithi017.stdout: "expected_num_objects": 0, 2024-09-11T09:50:50.936 INFO:teuthology.orchestra.run.smithi017.stdout: "fast_read": false, 2024-09-11T09:50:50.936 INFO:teuthology.orchestra.run.smithi017.stdout: "options": { 2024-09-11T09:50:50.937 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_num_max": 32, 2024-09-11T09:50:50.937 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_num_min": 1 2024-09-11T09:50:50.937 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:50.937 INFO:teuthology.orchestra.run.smithi017.stdout: "application_metadata": { 2024-09-11T09:50:50.937 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr": {} 2024-09-11T09:50:50.937 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:50.937 INFO:teuthology.orchestra.run.smithi017.stdout: "read_balance": { 2024-09-11T09:50:50.937 INFO:teuthology.orchestra.run.smithi017.stdout: "score_type": "Fair distribution", 2024-09-11T09:50:50.937 INFO:teuthology.orchestra.run.smithi017.stdout: "score_acting": 6.059999942779541, 2024-09-11T09:50:50.937 INFO:teuthology.orchestra.run.smithi017.stdout: "score_stable": 6.059999942779541, 2024-09-11T09:50:50.937 INFO:teuthology.orchestra.run.smithi017.stdout: "optimal_score": 0.33000001311302185, 2024-09-11T09:50:50.937 INFO:teuthology.orchestra.run.smithi017.stdout: "raw_score_acting": 2, 2024-09-11T09:50:50.937 INFO:teuthology.orchestra.run.smithi017.stdout: "raw_score_stable": 2, 2024-09-11T09:50:50.938 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity_weighted": 1, 2024-09-11T09:50:50.938 INFO:teuthology.orchestra.run.smithi017.stdout: "average_primary_affinity": 1, 2024-09-11T09:50:50.938 INFO:teuthology.orchestra.run.smithi017.stdout: "average_primary_affinity_weighted": 1 2024-09-11T09:50:50.938 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:50.938 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:50.938 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:50.938 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 2, 2024-09-11T09:50:50.938 INFO:teuthology.orchestra.run.smithi017.stdout: "pool_name": "rbd", 2024-09-11T09:50:50.938 INFO:teuthology.orchestra.run.smithi017.stdout: "create_time": "2024-09-11T09:50:16.993202+0000", 2024-09-11T09:50:50.938 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 8193, 2024-09-11T09:50:50.938 INFO:teuthology.orchestra.run.smithi017.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-11T09:50:50.938 INFO:teuthology.orchestra.run.smithi017.stdout: "type": 1, 2024-09-11T09:50:50.938 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 2, 2024-09-11T09:50:50.938 INFO:teuthology.orchestra.run.smithi017.stdout: "min_size": 1, 2024-09-11T09:50:50.939 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_rule": 0, 2024-09-11T09:50:50.939 INFO:teuthology.orchestra.run.smithi017.stdout: "peering_crush_bucket_count": 0, 2024-09-11T09:50:50.939 INFO:teuthology.orchestra.run.smithi017.stdout: "peering_crush_bucket_target": 0, 2024-09-11T09:50:50.939 INFO:teuthology.orchestra.run.smithi017.stdout: "peering_crush_bucket_barrier": 0, 2024-09-11T09:50:50.939 INFO:teuthology.orchestra.run.smithi017.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-11T09:50:50.939 INFO:teuthology.orchestra.run.smithi017.stdout: "is_stretch_pool": false, 2024-09-11T09:50:50.939 INFO:teuthology.orchestra.run.smithi017.stdout: "object_hash": 2, 2024-09-11T09:50:50.939 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_autoscale_mode": "off", 2024-09-11T09:50:50.939 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_num": 8, 2024-09-11T09:50:50.939 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_placement_num": 8, 2024-09-11T09:50:50.939 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_placement_num_target": 8, 2024-09-11T09:50:50.939 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_num_target": 8, 2024-09-11T09:50:50.939 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_num_pending": 8, 2024-09-11T09:50:50.939 INFO:teuthology.orchestra.run.smithi017.stdout: "last_pg_merge_meta": { 2024-09-11T09:50:50.940 INFO:teuthology.orchestra.run.smithi017.stdout: "source_pgid": "0.0", 2024-09-11T09:50:50.940 INFO:teuthology.orchestra.run.smithi017.stdout: "ready_epoch": 0, 2024-09-11T09:50:50.940 INFO:teuthology.orchestra.run.smithi017.stdout: "last_epoch_started": 0, 2024-09-11T09:50:50.940 INFO:teuthology.orchestra.run.smithi017.stdout: "last_epoch_clean": 0, 2024-09-11T09:50:50.940 INFO:teuthology.orchestra.run.smithi017.stdout: "source_version": "0'0", 2024-09-11T09:50:50.940 INFO:teuthology.orchestra.run.smithi017.stdout: "target_version": "0'0" 2024-09-11T09:50:50.940 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:50.940 INFO:teuthology.orchestra.run.smithi017.stdout: "last_change": "18", 2024-09-11T09:50:50.940 INFO:teuthology.orchestra.run.smithi017.stdout: "last_force_op_resend": "0", 2024-09-11T09:50:50.940 INFO:teuthology.orchestra.run.smithi017.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-11T09:50:50.940 INFO:teuthology.orchestra.run.smithi017.stdout: "last_force_op_resend_preluminous": "0", 2024-09-11T09:50:50.940 INFO:teuthology.orchestra.run.smithi017.stdout: "auid": 0, 2024-09-11T09:50:50.940 INFO:teuthology.orchestra.run.smithi017.stdout: "snap_mode": "selfmanaged", 2024-09-11T09:50:50.941 INFO:teuthology.orchestra.run.smithi017.stdout: "snap_seq": 2, 2024-09-11T09:50:50.941 INFO:teuthology.orchestra.run.smithi017.stdout: "snap_epoch": 18, 2024-09-11T09:50:50.941 INFO:teuthology.orchestra.run.smithi017.stdout: "pool_snaps": [], 2024-09-11T09:50:50.941 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_snaps": "[]", 2024-09-11T09:50:50.941 INFO:teuthology.orchestra.run.smithi017.stdout: "quota_max_bytes": 0, 2024-09-11T09:50:50.941 INFO:teuthology.orchestra.run.smithi017.stdout: "quota_max_objects": 0, 2024-09-11T09:50:50.941 INFO:teuthology.orchestra.run.smithi017.stdout: "tiers": [], 2024-09-11T09:50:50.941 INFO:teuthology.orchestra.run.smithi017.stdout: "tier_of": -1, 2024-09-11T09:50:50.941 INFO:teuthology.orchestra.run.smithi017.stdout: "read_tier": -1, 2024-09-11T09:50:50.941 INFO:teuthology.orchestra.run.smithi017.stdout: "write_tier": -1, 2024-09-11T09:50:50.941 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_mode": "none", 2024-09-11T09:50:50.941 INFO:teuthology.orchestra.run.smithi017.stdout: "target_max_bytes": 0, 2024-09-11T09:50:50.941 INFO:teuthology.orchestra.run.smithi017.stdout: "target_max_objects": 0, 2024-09-11T09:50:50.942 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-11T09:50:50.942 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-11T09:50:50.942 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-11T09:50:50.942 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_min_flush_age": 0, 2024-09-11T09:50:50.942 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_min_evict_age": 0, 2024-09-11T09:50:50.942 INFO:teuthology.orchestra.run.smithi017.stdout: "erasure_code_profile": "", 2024-09-11T09:50:50.942 INFO:teuthology.orchestra.run.smithi017.stdout: "hit_set_params": { 2024-09-11T09:50:50.942 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "none" 2024-09-11T09:50:50.942 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:50.942 INFO:teuthology.orchestra.run.smithi017.stdout: "hit_set_period": 0, 2024-09-11T09:50:50.942 INFO:teuthology.orchestra.run.smithi017.stdout: "hit_set_count": 0, 2024-09-11T09:50:50.942 INFO:teuthology.orchestra.run.smithi017.stdout: "use_gmt_hitset": true, 2024-09-11T09:50:50.942 INFO:teuthology.orchestra.run.smithi017.stdout: "min_read_recency_for_promote": 0, 2024-09-11T09:50:50.943 INFO:teuthology.orchestra.run.smithi017.stdout: "min_write_recency_for_promote": 0, 2024-09-11T09:50:50.943 INFO:teuthology.orchestra.run.smithi017.stdout: "hit_set_grade_decay_rate": 0, 2024-09-11T09:50:50.943 INFO:teuthology.orchestra.run.smithi017.stdout: "hit_set_search_last_n": 0, 2024-09-11T09:50:50.943 INFO:teuthology.orchestra.run.smithi017.stdout: "grade_table": [], 2024-09-11T09:50:50.943 INFO:teuthology.orchestra.run.smithi017.stdout: "stripe_width": 0, 2024-09-11T09:50:50.943 INFO:teuthology.orchestra.run.smithi017.stdout: "expected_num_objects": 0, 2024-09-11T09:50:50.943 INFO:teuthology.orchestra.run.smithi017.stdout: "fast_read": false, 2024-09-11T09:50:50.943 INFO:teuthology.orchestra.run.smithi017.stdout: "options": {}, 2024-09-11T09:50:50.943 INFO:teuthology.orchestra.run.smithi017.stdout: "application_metadata": { 2024-09-11T09:50:50.943 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd": {} 2024-09-11T09:50:50.943 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:50.943 INFO:teuthology.orchestra.run.smithi017.stdout: "read_balance": { 2024-09-11T09:50:50.943 INFO:teuthology.orchestra.run.smithi017.stdout: "score_type": "Fair distribution", 2024-09-11T09:50:50.944 INFO:teuthology.orchestra.run.smithi017.stdout: "score_acting": 2.25, 2024-09-11T09:50:50.944 INFO:teuthology.orchestra.run.smithi017.stdout: "score_stable": 2.25, 2024-09-11T09:50:50.944 INFO:teuthology.orchestra.run.smithi017.stdout: "optimal_score": 1, 2024-09-11T09:50:50.944 INFO:teuthology.orchestra.run.smithi017.stdout: "raw_score_acting": 2.25, 2024-09-11T09:50:50.944 INFO:teuthology.orchestra.run.smithi017.stdout: "raw_score_stable": 2.25, 2024-09-11T09:50:50.944 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity_weighted": 1, 2024-09-11T09:50:50.944 INFO:teuthology.orchestra.run.smithi017.stdout: "average_primary_affinity": 1, 2024-09-11T09:50:50.944 INFO:teuthology.orchestra.run.smithi017.stdout: "average_primary_affinity_weighted": 1 2024-09-11T09:50:50.944 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:50.944 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:50.944 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-11T09:50:50.944 INFO:teuthology.orchestra.run.smithi017.stdout: "osds": [ 2024-09-11T09:50:50.944 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:50.945 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 0, 2024-09-11T09:50:50.945 INFO:teuthology.orchestra.run.smithi017.stdout: "uuid": "4829b7a1-d607-4a56-b6d2-55036cb8acf8", 2024-09-11T09:50:50.945 INFO:teuthology.orchestra.run.smithi017.stdout: "up": 1, 2024-09-11T09:50:50.945 INFO:teuthology.orchestra.run.smithi017.stdout: "in": 1, 2024-09-11T09:50:50.945 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 1, 2024-09-11T09:50:50.945 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity": 1, 2024-09-11T09:50:50.945 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_begin": 0, 2024-09-11T09:50:50.945 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_end": 0, 2024-09-11T09:50:50.945 INFO:teuthology.orchestra.run.smithi017.stdout: "up_from": 12, 2024-09-11T09:50:50.945 INFO:teuthology.orchestra.run.smithi017.stdout: "up_thru": 0, 2024-09-11T09:50:50.945 INFO:teuthology.orchestra.run.smithi017.stdout: "down_at": 0, 2024-09-11T09:50:50.945 INFO:teuthology.orchestra.run.smithi017.stdout: "lost_at": 0, 2024-09-11T09:50:50.946 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-11T09:50:50.946 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-11T09:50:50.946 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:50.946 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-11T09:50:50.946 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6808", 2024-09-11T09:50:50.946 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 4154449929 2024-09-11T09:50:50.946 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:50.946 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:50.946 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-11T09:50:50.946 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6809", 2024-09-11T09:50:50.946 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 4154449929 2024-09-11T09:50:50.946 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:50.946 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:50:50.946 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:50.947 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addrs": { 2024-09-11T09:50:50.947 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-11T09:50:50.947 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:50.947 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-11T09:50:50.947 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6810", 2024-09-11T09:50:50.947 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 4154449929 2024-09-11T09:50:50.947 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:50.947 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:50.947 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-11T09:50:50.947 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6811", 2024-09-11T09:50:50.947 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 4154449929 2024-09-11T09:50:50.947 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:50.947 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:50:50.947 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:50.948 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addrs": { 2024-09-11T09:50:50.948 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-11T09:50:50.948 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:50.948 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-11T09:50:50.948 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6814", 2024-09-11T09:50:50.948 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 4154449929 2024-09-11T09:50:50.948 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:50.948 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:50.948 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-11T09:50:50.948 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6815", 2024-09-11T09:50:50.948 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 4154449929 2024-09-11T09:50:50.948 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:50.948 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:50:50.948 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:50.949 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addrs": { 2024-09-11T09:50:50.949 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-11T09:50:50.949 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:50.949 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-11T09:50:50.949 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6812", 2024-09-11T09:50:50.949 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 4154449929 2024-09-11T09:50:50.949 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:50.949 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:50.949 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-11T09:50:50.949 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6813", 2024-09-11T09:50:50.949 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 4154449929 2024-09-11T09:50:50.949 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:50.950 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:50:50.950 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:50.950 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6809/4154449929", 2024-09-11T09:50:50.950 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addr": "172.21.15.17:6811/4154449929", 2024-09-11T09:50:50.950 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addr": "172.21.15.17:6815/4154449929", 2024-09-11T09:50:50.950 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addr": "172.21.15.17:6813/4154449929", 2024-09-11T09:50:50.950 INFO:teuthology.orchestra.run.smithi017.stdout: "state": [ 2024-09-11T09:50:50.950 INFO:teuthology.orchestra.run.smithi017.stdout: "exists", 2024-09-11T09:50:50.950 INFO:teuthology.orchestra.run.smithi017.stdout: "up" 2024-09-11T09:50:50.950 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:50:50.950 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:50.950 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:50.950 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 1, 2024-09-11T09:50:50.950 INFO:teuthology.orchestra.run.smithi017.stdout: "uuid": "ab1d5df0-3c0e-44cd-88a0-96b07964148b", 2024-09-11T09:50:50.951 INFO:teuthology.orchestra.run.smithi017.stdout: "up": 1, 2024-09-11T09:50:50.951 INFO:teuthology.orchestra.run.smithi017.stdout: "in": 1, 2024-09-11T09:50:50.951 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 1, 2024-09-11T09:50:50.951 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity": 1, 2024-09-11T09:50:50.951 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_begin": 0, 2024-09-11T09:50:50.951 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_end": 0, 2024-09-11T09:50:50.951 INFO:teuthology.orchestra.run.smithi017.stdout: "up_from": 11, 2024-09-11T09:50:50.951 INFO:teuthology.orchestra.run.smithi017.stdout: "up_thru": 32, 2024-09-11T09:50:50.951 INFO:teuthology.orchestra.run.smithi017.stdout: "down_at": 0, 2024-09-11T09:50:50.951 INFO:teuthology.orchestra.run.smithi017.stdout: "lost_at": 0, 2024-09-11T09:50:50.951 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-11T09:50:50.952 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-11T09:50:50.952 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:50.952 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-11T09:50:50.952 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6800", 2024-09-11T09:50:50.952 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 4139511644 2024-09-11T09:50:50.952 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:50.952 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:50.952 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-11T09:50:50.952 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6801", 2024-09-11T09:50:50.952 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 4139511644 2024-09-11T09:50:50.952 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:50.952 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:50:50.952 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:50.953 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addrs": { 2024-09-11T09:50:50.953 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-11T09:50:50.953 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:50.953 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-11T09:50:50.953 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6802", 2024-09-11T09:50:50.953 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 4139511644 2024-09-11T09:50:50.953 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:50.953 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:50.953 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-11T09:50:50.953 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6803", 2024-09-11T09:50:50.953 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 4139511644 2024-09-11T09:50:50.953 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:50.954 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:50:50.954 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:50.954 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addrs": { 2024-09-11T09:50:50.954 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-11T09:50:50.954 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:50.954 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-11T09:50:50.954 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6806", 2024-09-11T09:50:50.954 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 4139511644 2024-09-11T09:50:50.954 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:50.954 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:50.954 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-11T09:50:50.954 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6807", 2024-09-11T09:50:50.954 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 4139511644 2024-09-11T09:50:50.954 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:50.954 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:50:50.955 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:50.955 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addrs": { 2024-09-11T09:50:50.955 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-11T09:50:50.955 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:50.955 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-11T09:50:50.955 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6804", 2024-09-11T09:50:50.955 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 4139511644 2024-09-11T09:50:50.955 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:50.955 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:50.955 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-11T09:50:50.955 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6805", 2024-09-11T09:50:50.955 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 4139511644 2024-09-11T09:50:50.955 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:50.955 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:50:50.956 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:50.956 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6801/4139511644", 2024-09-11T09:50:50.956 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addr": "172.21.15.17:6803/4139511644", 2024-09-11T09:50:50.956 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addr": "172.21.15.17:6807/4139511644", 2024-09-11T09:50:50.956 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addr": "172.21.15.17:6805/4139511644", 2024-09-11T09:50:50.956 INFO:teuthology.orchestra.run.smithi017.stdout: "state": [ 2024-09-11T09:50:50.956 INFO:teuthology.orchestra.run.smithi017.stdout: "exists", 2024-09-11T09:50:50.956 INFO:teuthology.orchestra.run.smithi017.stdout: "up" 2024-09-11T09:50:50.956 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:50:50.956 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:50.956 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:50.956 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 2, 2024-09-11T09:50:50.956 INFO:teuthology.orchestra.run.smithi017.stdout: "uuid": "4c8e50db-b49c-4d44-a913-47b5b30f8699", 2024-09-11T09:50:50.956 INFO:teuthology.orchestra.run.smithi017.stdout: "up": 1, 2024-09-11T09:50:50.957 INFO:teuthology.orchestra.run.smithi017.stdout: "in": 1, 2024-09-11T09:50:50.957 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 1, 2024-09-11T09:50:50.957 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity": 1, 2024-09-11T09:50:50.957 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_begin": 0, 2024-09-11T09:50:50.957 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_end": 0, 2024-09-11T09:50:50.957 INFO:teuthology.orchestra.run.smithi017.stdout: "up_from": 11, 2024-09-11T09:50:50.957 INFO:teuthology.orchestra.run.smithi017.stdout: "up_thru": 15, 2024-09-11T09:50:50.957 INFO:teuthology.orchestra.run.smithi017.stdout: "down_at": 0, 2024-09-11T09:50:50.957 INFO:teuthology.orchestra.run.smithi017.stdout: "lost_at": 0, 2024-09-11T09:50:50.957 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-11T09:50:50.957 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-11T09:50:50.957 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:50.957 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-11T09:50:50.958 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6816", 2024-09-11T09:50:50.958 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1151282506 2024-09-11T09:50:50.958 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:50.958 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:50.958 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-11T09:50:50.958 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6817", 2024-09-11T09:50:50.958 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1151282506 2024-09-11T09:50:50.958 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:50.958 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:50:50.958 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:50.958 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addrs": { 2024-09-11T09:50:50.958 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-11T09:50:50.958 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:50.959 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-11T09:50:50.959 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6818", 2024-09-11T09:50:50.959 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1151282506 2024-09-11T09:50:50.959 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:50.959 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:50.959 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-11T09:50:50.959 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6819", 2024-09-11T09:50:50.959 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1151282506 2024-09-11T09:50:50.959 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:50.959 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:50:50.959 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:50.959 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addrs": { 2024-09-11T09:50:50.959 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-11T09:50:50.960 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:50.960 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-11T09:50:50.960 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6822", 2024-09-11T09:50:50.960 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1151282506 2024-09-11T09:50:50.960 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:50.960 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:50.960 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-11T09:50:50.960 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6823", 2024-09-11T09:50:50.960 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1151282506 2024-09-11T09:50:50.960 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:50.960 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:50:50.960 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:50.961 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addrs": { 2024-09-11T09:50:50.961 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-11T09:50:50.961 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:50.961 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-11T09:50:50.961 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6820", 2024-09-11T09:50:50.961 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1151282506 2024-09-11T09:50:50.961 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:50.961 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:50.961 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-11T09:50:50.961 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6821", 2024-09-11T09:50:50.961 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1151282506 2024-09-11T09:50:50.961 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:50.962 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:50:50.962 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:50.962 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6817/1151282506", 2024-09-11T09:50:50.962 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addr": "172.21.15.17:6819/1151282506", 2024-09-11T09:50:50.962 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addr": "172.21.15.17:6823/1151282506", 2024-09-11T09:50:50.962 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addr": "172.21.15.17:6821/1151282506", 2024-09-11T09:50:50.962 INFO:teuthology.orchestra.run.smithi017.stdout: "state": [ 2024-09-11T09:50:50.962 INFO:teuthology.orchestra.run.smithi017.stdout: "exists", 2024-09-11T09:50:50.962 INFO:teuthology.orchestra.run.smithi017.stdout: "up" 2024-09-11T09:50:50.962 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:50:50.962 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:50.962 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:50.962 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 3, 2024-09-11T09:50:50.962 INFO:teuthology.orchestra.run.smithi017.stdout: "uuid": "37abc55b-a1eb-49d5-b1ff-b507ee2cb5a6", 2024-09-11T09:50:50.963 INFO:teuthology.orchestra.run.smithi017.stdout: "up": 1, 2024-09-11T09:50:50.963 INFO:teuthology.orchestra.run.smithi017.stdout: "in": 1, 2024-09-11T09:50:50.963 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 1, 2024-09-11T09:50:50.963 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity": 1, 2024-09-11T09:50:50.963 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_begin": 0, 2024-09-11T09:50:50.963 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_end": 0, 2024-09-11T09:50:50.963 INFO:teuthology.orchestra.run.smithi017.stdout: "up_from": 12, 2024-09-11T09:50:50.963 INFO:teuthology.orchestra.run.smithi017.stdout: "up_thru": 35, 2024-09-11T09:50:50.963 INFO:teuthology.orchestra.run.smithi017.stdout: "down_at": 0, 2024-09-11T09:50:50.963 INFO:teuthology.orchestra.run.smithi017.stdout: "lost_at": 0, 2024-09-11T09:50:50.963 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-11T09:50:50.963 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-11T09:50:50.963 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:50.963 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-11T09:50:50.964 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.96:6816", 2024-09-11T09:50:50.964 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 332854807 2024-09-11T09:50:50.964 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:50.964 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:50.964 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-11T09:50:50.964 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.96:6817", 2024-09-11T09:50:50.964 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 332854807 2024-09-11T09:50:50.964 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:50.964 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:50:50.964 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:50.964 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addrs": { 2024-09-11T09:50:50.964 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-11T09:50:50.964 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:50.964 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-11T09:50:50.965 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.96:6818", 2024-09-11T09:50:50.965 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 332854807 2024-09-11T09:50:50.965 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:50.965 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:50.965 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-11T09:50:50.965 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.96:6819", 2024-09-11T09:50:50.965 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 332854807 2024-09-11T09:50:50.965 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:50.965 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:50:50.965 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:50.965 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addrs": { 2024-09-11T09:50:50.965 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-11T09:50:50.965 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:50.966 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-11T09:50:50.966 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.96:6822", 2024-09-11T09:50:50.966 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 332854807 2024-09-11T09:50:50.966 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:50.966 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:50.966 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-11T09:50:50.966 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.96:6823", 2024-09-11T09:50:50.966 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 332854807 2024-09-11T09:50:50.966 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:50.966 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:50:50.966 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:50.966 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addrs": { 2024-09-11T09:50:50.966 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-11T09:50:50.966 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:50.967 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-11T09:50:50.967 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.96:6820", 2024-09-11T09:50:50.967 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 332854807 2024-09-11T09:50:50.967 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:50.967 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:50.967 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-11T09:50:50.967 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.96:6821", 2024-09-11T09:50:50.967 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 332854807 2024-09-11T09:50:50.967 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:50.967 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:50:50.967 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:50.967 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.96:6817/332854807", 2024-09-11T09:50:50.967 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addr": "172.21.15.96:6819/332854807", 2024-09-11T09:50:50.967 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addr": "172.21.15.96:6823/332854807", 2024-09-11T09:50:50.968 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addr": "172.21.15.96:6821/332854807", 2024-09-11T09:50:50.968 INFO:teuthology.orchestra.run.smithi017.stdout: "state": [ 2024-09-11T09:50:50.968 INFO:teuthology.orchestra.run.smithi017.stdout: "exists", 2024-09-11T09:50:50.968 INFO:teuthology.orchestra.run.smithi017.stdout: "up" 2024-09-11T09:50:50.968 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:50:50.968 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:50.968 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:50.968 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 4, 2024-09-11T09:50:50.968 INFO:teuthology.orchestra.run.smithi017.stdout: "uuid": "067d9586-4f5a-4d17-96c5-86a7b8bcd114", 2024-09-11T09:50:50.968 INFO:teuthology.orchestra.run.smithi017.stdout: "up": 1, 2024-09-11T09:50:50.968 INFO:teuthology.orchestra.run.smithi017.stdout: "in": 1, 2024-09-11T09:50:50.968 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 1, 2024-09-11T09:50:50.968 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity": 1, 2024-09-11T09:50:50.968 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_begin": 0, 2024-09-11T09:50:50.969 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_end": 0, 2024-09-11T09:50:50.969 INFO:teuthology.orchestra.run.smithi017.stdout: "up_from": 11, 2024-09-11T09:50:50.969 INFO:teuthology.orchestra.run.smithi017.stdout: "up_thru": 0, 2024-09-11T09:50:50.969 INFO:teuthology.orchestra.run.smithi017.stdout: "down_at": 0, 2024-09-11T09:50:50.969 INFO:teuthology.orchestra.run.smithi017.stdout: "lost_at": 0, 2024-09-11T09:50:50.969 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-11T09:50:50.969 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-11T09:50:50.969 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:50.969 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-11T09:50:50.969 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.96:6800", 2024-09-11T09:50:50.969 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2785176937 2024-09-11T09:50:50.969 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:50.969 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:50.970 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-11T09:50:50.970 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.96:6801", 2024-09-11T09:50:50.970 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2785176937 2024-09-11T09:50:50.970 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:50.970 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:50:50.970 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:50.970 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addrs": { 2024-09-11T09:50:50.970 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-11T09:50:50.970 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:50.970 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-11T09:50:50.970 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.96:6802", 2024-09-11T09:50:50.970 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2785176937 2024-09-11T09:50:50.970 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:50.970 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:50.971 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-11T09:50:50.971 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.96:6803", 2024-09-11T09:50:50.971 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2785176937 2024-09-11T09:50:50.971 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:50.971 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:50:50.971 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:50.971 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addrs": { 2024-09-11T09:50:50.971 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-11T09:50:50.971 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:50.971 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-11T09:50:50.971 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.96:6806", 2024-09-11T09:50:50.971 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2785176937 2024-09-11T09:50:50.971 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:50.971 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:50.972 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-11T09:50:50.972 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.96:6807", 2024-09-11T09:50:50.972 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2785176937 2024-09-11T09:50:50.972 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:50.972 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:50:50.972 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:50.972 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addrs": { 2024-09-11T09:50:50.972 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-11T09:50:50.972 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:50.972 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-11T09:50:50.972 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.96:6804", 2024-09-11T09:50:50.972 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2785176937 2024-09-11T09:50:50.972 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:50.973 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:50.973 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-11T09:50:50.973 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.96:6805", 2024-09-11T09:50:50.973 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 2785176937 2024-09-11T09:50:50.973 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:50.973 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:50:50.973 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:50.973 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.96:6801/2785176937", 2024-09-11T09:50:50.973 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addr": "172.21.15.96:6803/2785176937", 2024-09-11T09:50:50.973 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addr": "172.21.15.96:6807/2785176937", 2024-09-11T09:50:50.973 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addr": "172.21.15.96:6805/2785176937", 2024-09-11T09:50:50.973 INFO:teuthology.orchestra.run.smithi017.stdout: "state": [ 2024-09-11T09:50:50.974 INFO:teuthology.orchestra.run.smithi017.stdout: "exists", 2024-09-11T09:50:50.974 INFO:teuthology.orchestra.run.smithi017.stdout: "up" 2024-09-11T09:50:50.974 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:50:50.974 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:50.974 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:50.974 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 5, 2024-09-11T09:50:50.974 INFO:teuthology.orchestra.run.smithi017.stdout: "uuid": "8ff7a724-0a3a-4aa2-baf1-fd52f88d476c", 2024-09-11T09:50:50.974 INFO:teuthology.orchestra.run.smithi017.stdout: "up": 1, 2024-09-11T09:50:50.974 INFO:teuthology.orchestra.run.smithi017.stdout: "in": 1, 2024-09-11T09:50:50.974 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 1, 2024-09-11T09:50:50.974 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity": 1, 2024-09-11T09:50:50.974 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_begin": 0, 2024-09-11T09:50:50.974 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_end": 0, 2024-09-11T09:50:50.974 INFO:teuthology.orchestra.run.smithi017.stdout: "up_from": 11, 2024-09-11T09:50:50.974 INFO:teuthology.orchestra.run.smithi017.stdout: "up_thru": 15, 2024-09-11T09:50:50.975 INFO:teuthology.orchestra.run.smithi017.stdout: "down_at": 0, 2024-09-11T09:50:50.975 INFO:teuthology.orchestra.run.smithi017.stdout: "lost_at": 0, 2024-09-11T09:50:50.975 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-11T09:50:50.975 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-11T09:50:50.975 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:50.975 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-11T09:50:50.975 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.96:6808", 2024-09-11T09:50:50.975 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 509935370 2024-09-11T09:50:50.975 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:50.975 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:50.975 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-11T09:50:50.975 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.96:6809", 2024-09-11T09:50:50.975 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 509935370 2024-09-11T09:50:50.975 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:50.976 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:50:50.976 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:50.976 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addrs": { 2024-09-11T09:50:50.976 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-11T09:50:50.976 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:50.976 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-11T09:50:50.976 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.96:6810", 2024-09-11T09:50:50.976 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 509935370 2024-09-11T09:50:50.976 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:50.976 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:50.976 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-11T09:50:50.976 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.96:6811", 2024-09-11T09:50:50.976 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 509935370 2024-09-11T09:50:50.976 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:50.977 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:50:50.977 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:50.977 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addrs": { 2024-09-11T09:50:50.977 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-11T09:50:50.977 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:50.977 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-11T09:50:50.977 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.96:6814", 2024-09-11T09:50:50.977 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 509935370 2024-09-11T09:50:50.977 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:50.977 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:50.977 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-11T09:50:50.977 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.96:6815", 2024-09-11T09:50:50.977 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 509935370 2024-09-11T09:50:50.978 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:50.978 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:50:50.978 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:50.978 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addrs": { 2024-09-11T09:50:50.978 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-11T09:50:50.978 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:50.978 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-11T09:50:50.978 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.96:6812", 2024-09-11T09:50:50.978 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 509935370 2024-09-11T09:50:50.978 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:50.978 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:50.978 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-11T09:50:50.978 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.96:6813", 2024-09-11T09:50:50.978 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 509935370 2024-09-11T09:50:50.979 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:50.979 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:50:50.979 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:50.979 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.96:6809/509935370", 2024-09-11T09:50:50.979 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addr": "172.21.15.96:6811/509935370", 2024-09-11T09:50:50.979 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addr": "172.21.15.96:6815/509935370", 2024-09-11T09:50:50.979 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addr": "172.21.15.96:6813/509935370", 2024-09-11T09:50:50.979 INFO:teuthology.orchestra.run.smithi017.stdout: "state": [ 2024-09-11T09:50:50.979 INFO:teuthology.orchestra.run.smithi017.stdout: "exists", 2024-09-11T09:50:50.979 INFO:teuthology.orchestra.run.smithi017.stdout: "up" 2024-09-11T09:50:50.979 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:50:50.979 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:50.979 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-11T09:50:50.979 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_xinfo": [ 2024-09-11T09:50:50.979 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:50.980 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 0, 2024-09-11T09:50:50.980 INFO:teuthology.orchestra.run.smithi017.stdout: "down_stamp": "0.000000", 2024-09-11T09:50:50.980 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_probability": 0, 2024-09-11T09:50:50.980 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_interval": 0, 2024-09-11T09:50:50.980 INFO:teuthology.orchestra.run.smithi017.stdout: "features": 4540701547738038271, 2024-09-11T09:50:50.980 INFO:teuthology.orchestra.run.smithi017.stdout: "old_weight": 0, 2024-09-11T09:50:50.980 INFO:teuthology.orchestra.run.smithi017.stdout: "last_purged_snaps_scrub": "2024-09-11T09:50:08.513270+0000", 2024-09-11T09:50:50.980 INFO:teuthology.orchestra.run.smithi017.stdout: "dead_epoch": 0 2024-09-11T09:50:50.980 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:50.980 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:50.980 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 1, 2024-09-11T09:50:50.980 INFO:teuthology.orchestra.run.smithi017.stdout: "down_stamp": "0.000000", 2024-09-11T09:50:50.980 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_probability": 0, 2024-09-11T09:50:50.981 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_interval": 0, 2024-09-11T09:50:50.981 INFO:teuthology.orchestra.run.smithi017.stdout: "features": 4540701547738038271, 2024-09-11T09:50:50.981 INFO:teuthology.orchestra.run.smithi017.stdout: "old_weight": 0, 2024-09-11T09:50:50.981 INFO:teuthology.orchestra.run.smithi017.stdout: "last_purged_snaps_scrub": "2024-09-11T09:50:07.522863+0000", 2024-09-11T09:50:50.981 INFO:teuthology.orchestra.run.smithi017.stdout: "dead_epoch": 0 2024-09-11T09:50:50.981 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:50.981 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:50.981 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 2, 2024-09-11T09:50:50.981 INFO:teuthology.orchestra.run.smithi017.stdout: "down_stamp": "0.000000", 2024-09-11T09:50:50.981 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_probability": 0, 2024-09-11T09:50:50.981 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_interval": 0, 2024-09-11T09:50:50.981 INFO:teuthology.orchestra.run.smithi017.stdout: "features": 4540701547738038271, 2024-09-11T09:50:50.982 INFO:teuthology.orchestra.run.smithi017.stdout: "old_weight": 0, 2024-09-11T09:50:50.982 INFO:teuthology.orchestra.run.smithi017.stdout: "last_purged_snaps_scrub": "2024-09-11T09:50:08.134364+0000", 2024-09-11T09:50:50.982 INFO:teuthology.orchestra.run.smithi017.stdout: "dead_epoch": 0 2024-09-11T09:50:50.982 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:50.982 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:50.982 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 3, 2024-09-11T09:50:50.982 INFO:teuthology.orchestra.run.smithi017.stdout: "down_stamp": "0.000000", 2024-09-11T09:50:50.982 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_probability": 0, 2024-09-11T09:50:50.982 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_interval": 0, 2024-09-11T09:50:50.982 INFO:teuthology.orchestra.run.smithi017.stdout: "features": 4540701547738038271, 2024-09-11T09:50:50.982 INFO:teuthology.orchestra.run.smithi017.stdout: "old_weight": 0, 2024-09-11T09:50:50.982 INFO:teuthology.orchestra.run.smithi017.stdout: "last_purged_snaps_scrub": "2024-09-11T09:50:08.622366+0000", 2024-09-11T09:50:50.982 INFO:teuthology.orchestra.run.smithi017.stdout: "dead_epoch": 0 2024-09-11T09:50:50.982 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:50.982 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:50.983 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 4, 2024-09-11T09:50:50.983 INFO:teuthology.orchestra.run.smithi017.stdout: "down_stamp": "0.000000", 2024-09-11T09:50:50.983 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_probability": 0, 2024-09-11T09:50:50.983 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_interval": 0, 2024-09-11T09:50:50.983 INFO:teuthology.orchestra.run.smithi017.stdout: "features": 4540701547738038271, 2024-09-11T09:50:50.983 INFO:teuthology.orchestra.run.smithi017.stdout: "old_weight": 0, 2024-09-11T09:50:50.983 INFO:teuthology.orchestra.run.smithi017.stdout: "last_purged_snaps_scrub": "2024-09-11T09:50:08.156142+0000", 2024-09-11T09:50:50.983 INFO:teuthology.orchestra.run.smithi017.stdout: "dead_epoch": 0 2024-09-11T09:50:50.983 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:50.983 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:50.983 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 5, 2024-09-11T09:50:50.983 INFO:teuthology.orchestra.run.smithi017.stdout: "down_stamp": "0.000000", 2024-09-11T09:50:50.983 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_probability": 0, 2024-09-11T09:50:50.983 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_interval": 0, 2024-09-11T09:50:50.983 INFO:teuthology.orchestra.run.smithi017.stdout: "features": 4540701547738038271, 2024-09-11T09:50:50.984 INFO:teuthology.orchestra.run.smithi017.stdout: "old_weight": 0, 2024-09-11T09:50:50.984 INFO:teuthology.orchestra.run.smithi017.stdout: "last_purged_snaps_scrub": "2024-09-11T09:50:08.108116+0000", 2024-09-11T09:50:50.984 INFO:teuthology.orchestra.run.smithi017.stdout: "dead_epoch": 0 2024-09-11T09:50:50.984 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:50.984 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-11T09:50:50.984 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_upmap": [], 2024-09-11T09:50:50.984 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_upmap_items": [], 2024-09-11T09:50:50.984 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_upmap_primaries": [], 2024-09-11T09:50:50.984 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_temp": [], 2024-09-11T09:50:50.984 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_temp": [], 2024-09-11T09:50:50.984 INFO:teuthology.orchestra.run.smithi017.stdout: "blocklist": {}, 2024-09-11T09:50:50.984 INFO:teuthology.orchestra.run.smithi017.stdout: "range_blocklist": {}, 2024-09-11T09:50:50.984 INFO:teuthology.orchestra.run.smithi017.stdout: "erasure_code_profiles": { 2024-09-11T09:50:50.985 INFO:teuthology.orchestra.run.smithi017.stdout: "default": { 2024-09-11T09:50:50.985 INFO:teuthology.orchestra.run.smithi017.stdout: "crush-failure-domain": "osd", 2024-09-11T09:50:50.985 INFO:teuthology.orchestra.run.smithi017.stdout: "k": "2", 2024-09-11T09:50:50.985 INFO:teuthology.orchestra.run.smithi017.stdout: "m": "1", 2024-09-11T09:50:50.985 INFO:teuthology.orchestra.run.smithi017.stdout: "plugin": "jerasure", 2024-09-11T09:50:50.985 INFO:teuthology.orchestra.run.smithi017.stdout: "technique": "reed_sol_van" 2024-09-11T09:50:50.985 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:50.985 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:50.985 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_snaps_queue": [], 2024-09-11T09:50:50.985 INFO:teuthology.orchestra.run.smithi017.stdout: "new_removed_snaps": [], 2024-09-11T09:50:50.985 INFO:teuthology.orchestra.run.smithi017.stdout: "new_purged_snaps": [], 2024-09-11T09:50:50.985 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_node_flags": {}, 2024-09-11T09:50:50.985 INFO:teuthology.orchestra.run.smithi017.stdout: "device_class_flags": {}, 2024-09-11T09:50:50.985 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": { 2024-09-11T09:50:50.986 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode_enabled": false, 2024-09-11T09:50:50.986 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_bucket_count": 0, 2024-09-11T09:50:50.986 INFO:teuthology.orchestra.run.smithi017.stdout: "degraded_stretch_mode": 0, 2024-09-11T09:50:50.986 INFO:teuthology.orchestra.run.smithi017.stdout: "recovering_stretch_mode": 0, 2024-09-11T09:50:50.986 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode_bucket": 0 2024-09-11T09:50:50.986 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:50.986 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:50:50.986 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T09:50:50.988 INFO:tasks.thrashosds.thrasher:j is {'epoch': 36, 'fsid': 'a2f1daec-d3b0-4bf5-82c8-72ecbb93d4e4', 'created': '2024-09-11T09:50:00.165804+0000', 'modified': '2024-09-11T09:50:49.597656+0000', 'last_up_change': '2024-09-11T09:50:10.457262+0000', 'last_in_change': '2024-09-11T09:50:45.581301+0000', 'flags': 'sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5799936, 'flags_set': ['pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 5, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 2, 'max_osd': 6, 'require_min_compat_client': 'luminous', 'min_compat_client': 'jewel', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-11T09:50:10.457184+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '14', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 6.059999942779541, 'score_stable': 6.059999942779541, 'optimal_score': 0.33000001311302185, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2024-09-11T09:50:16.993202+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 18, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}], 'osds': [{'osd': 0, 'uuid': '4829b7a1-d607-4a56-b6d2-55036cb8acf8', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 12, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6808', 'nonce': 4154449929}, {'type': 'v1', 'addr': '172.21.15.17:6809', 'nonce': 4154449929}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6810', 'nonce': 4154449929}, {'type': 'v1', 'addr': '172.21.15.17:6811', 'nonce': 4154449929}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6814', 'nonce': 4154449929}, {'type': 'v1', 'addr': '172.21.15.17:6815', 'nonce': 4154449929}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6812', 'nonce': 4154449929}, {'type': 'v1', 'addr': '172.21.15.17:6813', 'nonce': 4154449929}]}, 'public_addr': '172.21.15.17:6809/4154449929', 'cluster_addr': '172.21.15.17:6811/4154449929', 'heartbeat_back_addr': '172.21.15.17:6815/4154449929', 'heartbeat_front_addr': '172.21.15.17:6813/4154449929', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'ab1d5df0-3c0e-44cd-88a0-96b07964148b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 32, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6800', 'nonce': 4139511644}, {'type': 'v1', 'addr': '172.21.15.17:6801', 'nonce': 4139511644}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6802', 'nonce': 4139511644}, {'type': 'v1', 'addr': '172.21.15.17:6803', 'nonce': 4139511644}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6806', 'nonce': 4139511644}, {'type': 'v1', 'addr': '172.21.15.17:6807', 'nonce': 4139511644}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6804', 'nonce': 4139511644}, {'type': 'v1', 'addr': '172.21.15.17:6805', 'nonce': 4139511644}]}, 'public_addr': '172.21.15.17:6801/4139511644', 'cluster_addr': '172.21.15.17:6803/4139511644', 'heartbeat_back_addr': '172.21.15.17:6807/4139511644', 'heartbeat_front_addr': '172.21.15.17:6805/4139511644', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '4c8e50db-b49c-4d44-a913-47b5b30f8699', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 15, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6816', 'nonce': 1151282506}, {'type': 'v1', 'addr': '172.21.15.17:6817', 'nonce': 1151282506}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6818', 'nonce': 1151282506}, {'type': 'v1', 'addr': '172.21.15.17:6819', 'nonce': 1151282506}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6822', 'nonce': 1151282506}, {'type': 'v1', 'addr': '172.21.15.17:6823', 'nonce': 1151282506}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6820', 'nonce': 1151282506}, {'type': 'v1', 'addr': '172.21.15.17:6821', 'nonce': 1151282506}]}, 'public_addr': '172.21.15.17:6817/1151282506', 'cluster_addr': '172.21.15.17:6819/1151282506', 'heartbeat_back_addr': '172.21.15.17:6823/1151282506', 'heartbeat_front_addr': '172.21.15.17:6821/1151282506', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '37abc55b-a1eb-49d5-b1ff-b507ee2cb5a6', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 12, 'up_thru': 35, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.96:6816', 'nonce': 332854807}, {'type': 'v1', 'addr': '172.21.15.96:6817', 'nonce': 332854807}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.96:6818', 'nonce': 332854807}, {'type': 'v1', 'addr': '172.21.15.96:6819', 'nonce': 332854807}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.96:6822', 'nonce': 332854807}, {'type': 'v1', 'addr': '172.21.15.96:6823', 'nonce': 332854807}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.96:6820', 'nonce': 332854807}, {'type': 'v1', 'addr': '172.21.15.96:6821', 'nonce': 332854807}]}, 'public_addr': '172.21.15.96:6817/332854807', 'cluster_addr': '172.21.15.96:6819/332854807', 'heartbeat_back_addr': '172.21.15.96:6823/332854807', 'heartbeat_front_addr': '172.21.15.96:6821/332854807', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '067d9586-4f5a-4d17-96c5-86a7b8bcd114', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 0, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.96:6800', 'nonce': 2785176937}, {'type': 'v1', 'addr': '172.21.15.96:6801', 'nonce': 2785176937}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.96:6802', 'nonce': 2785176937}, {'type': 'v1', 'addr': '172.21.15.96:6803', 'nonce': 2785176937}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.96:6806', 'nonce': 2785176937}, {'type': 'v1', 'addr': '172.21.15.96:6807', 'nonce': 2785176937}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.96:6804', 'nonce': 2785176937}, {'type': 'v1', 'addr': '172.21.15.96:6805', 'nonce': 2785176937}]}, 'public_addr': '172.21.15.96:6801/2785176937', 'cluster_addr': '172.21.15.96:6803/2785176937', 'heartbeat_back_addr': '172.21.15.96:6807/2785176937', 'heartbeat_front_addr': '172.21.15.96:6805/2785176937', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '8ff7a724-0a3a-4aa2-baf1-fd52f88d476c', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 15, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.96:6808', 'nonce': 509935370}, {'type': 'v1', 'addr': '172.21.15.96:6809', 'nonce': 509935370}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.96:6810', 'nonce': 509935370}, {'type': 'v1', 'addr': '172.21.15.96:6811', 'nonce': 509935370}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.96:6814', 'nonce': 509935370}, {'type': 'v1', 'addr': '172.21.15.96:6815', 'nonce': 509935370}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.96:6812', 'nonce': 509935370}, {'type': 'v1', 'addr': '172.21.15.96:6813', 'nonce': 509935370}]}, 'public_addr': '172.21.15.96:6809/509935370', 'cluster_addr': '172.21.15.96:6811/509935370', 'heartbeat_back_addr': '172.21.15.96:6815/509935370', 'heartbeat_front_addr': '172.21.15.96:6813/509935370', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-11T09:50:08.513270+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-11T09:50:07.522863+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-11T09:50:08.134364+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-11T09:50:08.622366+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-11T09:50:08.156142+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-11T09:50:08.108116+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-11T09:50:50.989 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T09:50:50.989 INFO:tasks.ceph.osd.0.smithi017.stderr:2024-09-11T09:50:50.988+0000 7fb73c0f1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31665) UID: 0 2024-09-11T09:50:51.004 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:50:51.005 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:50:51.014 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-11T09:50:51.089 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:50:51.090+0000 7f3c96ec9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31909) UID: 0 2024-09-11T09:50:51.135 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-11T09:50:51.135 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T09:50:51.152 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-11T09:50:51.152 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T09:50:51.169 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-11T09:50:51.169 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T09:50:51.171 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:50:51.171 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 20, 2024-09-11T09:50:51.172 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 600, 2024-09-11T09:50:51.172 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [ 2024-09-11T09:50:51.172 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:51.172 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4230.0:1 2.3 e16/15 2:c4c92e5a:::rbd_trash:head v 16'1, mlcod=0'0)", 2024-09-11T09:50:51.172 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:18.494378+0000", 2024-09-11T09:50:51.172 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 32.678809747000003, 2024-09-11T09:50:51.172 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00089752199999999999, 2024-09-11T09:50:51.172 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:50:51.172 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:50:51.172 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:51.172 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:50:51.172 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:51.172 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:50:51.173 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:18.494378+0000", 2024-09-11T09:50:51.173 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:51.173 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:51.173 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:51.173 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:50:51.173 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:18.494378+0000", 2024-09-11T09:50:51.173 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:51.173 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:51.173 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:51.173 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:50:51.173 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:18.494378+0000", 2024-09-11T09:50:51.173 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.9099999999999999e-07 2024-09-11T09:50:51.173 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:51.174 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:51.174 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:50:51.174 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:18.494385+0000", 2024-09-11T09:50:51.174 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.8129999999999998e-06 2024-09-11T09:50:51.174 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:51.174 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:51.174 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:50:51.174 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:18.494416+0000", 2024-09-11T09:50:51.174 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.0654000000000002e-05 2024-09-11T09:50:51.174 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:51.174 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:51.174 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:50:51.174 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:18.494430+0000", 2024-09-11T09:50:51.174 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3754000000000001e-05 2024-09-11T09:50:51.175 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:51.175 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:51.175 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:50:51.175 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:18.494524+0000", 2024-09-11T09:50:51.175 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.4896999999999996e-05 2024-09-11T09:50:51.175 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:51.175 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:51.175 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:50:51.175 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:18.494558+0000", 2024-09-11T09:50:51.175 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.3900999999999999e-05 2024-09-11T09:50:51.175 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:51.175 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:51.175 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:50:51.175 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:18.495231+0000", 2024-09-11T09:50:51.175 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00067233500000000003 2024-09-11T09:50:51.176 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:51.176 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:51.176 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:50:51.176 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:18.495275+0000", 2024-09-11T09:50:51.176 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.4777000000000002e-05 2024-09-11T09:50:51.176 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:51.176 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:50:51.176 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:51.176 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:51.176 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:50:51.176 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:50:51.181 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-11T09:50:51.186 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-11T09:50:51.186 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T09:50:51.203 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-11T09:50:51.203 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T09:50:51.221 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-11T09:50:51.221 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T09:50:51.225 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-09-11T09:50:51.226+0000 7f7b6c7d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31904) UID: 0 2024-09-11T09:50:51.290 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:50:51.291+0000 7fdec5849640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31907) UID: 0 2024-09-11T09:50:51.296 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-11T09:50:51.297 INFO:teuthology.orchestra.run.smithi096.stdout: "ops": [], 2024-09-11T09:50:51.297 INFO:teuthology.orchestra.run.smithi096.stdout: "num_ops": 0 2024-09-11T09:50:51.297 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-11T09:50:51.306 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-11T09:50:51.306 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-11T09:50:51.307 INFO:teuthology.orchestra.run.smithi017.stderr:dumped all 2024-09-11T09:50:51.319 INFO:teuthology.orchestra.run.smithi017.stdout:{"pg_ready":true,"pg_map":{"version":51,"stamp":"2024-09-11T09:50:50.421631+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":161,"num_write_kb":2418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120,"ondisk_log_size":120,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":5,"kb":566231040,"kb_used":176752,"kb_used_data":2076,"kb_used_omap":9,"kb_used_meta":174518,"kb_avail":566054288,"statfs":{"total":579820584960,"available":579639590912,"internally_reserved":0,"allocated":2125824,"data_stored":1885665,"data_compressed":66628,"data_compressed_allocated":90112,"data_compressed_original":1163264,"omap_allocated":9566,"internal_metadata":178707106},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000807"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":47,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-11T09:50:45.587946+0000","last_change":"2024-09-11T09:50:20.497924+0000","last_active":"2024-09-11T09:50:45.587946+0000","last_peered":"2024-09-11T09:50:45.587946+0000","last_clean":"2024-09-11T09:50:45.587946+0000","last_became_active":"2024-09-11T09:50:18.544390+0000","last_became_peered":"2024-09-11T09:50:18.544390+0000","last_unstale":"2024-09-11T09:50:45.587946+0000","last_undegraded":"2024-09-11T09:50:45.587946+0000","last_fullsized":"2024-09-11T09:50:45.587946+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_clean_scrub_stamp":"2024-09-11T09:50:17.477890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T17:32:48.479885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00046094899999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":45,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-11T09:50:43.575924+0000","last_change":"2024-09-11T09:50:20.497446+0000","last_active":"2024-09-11T09:50:43.575924+0000","last_peered":"2024-09-11T09:50:43.575924+0000","last_clean":"2024-09-11T09:50:43.575924+0000","last_became_active":"2024-09-11T09:50:18.497481+0000","last_became_peered":"2024-09-11T09:50:18.497481+0000","last_unstale":"2024-09-11T09:50:43.575924+0000","last_undegraded":"2024-09-11T09:50:43.575924+0000","last_fullsized":"2024-09-11T09:50:43.575924+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_clean_scrub_stamp":"2024-09-11T09:50:17.477890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T14:12:09.857529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00050697600000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":51,"reported_epoch":31,"state":"peering","last_fresh":"2024-09-11T09:50:43.576045+0000","last_change":"2024-09-11T09:50:45.588289+0000","last_active":"2024-09-11T09:50:45.588289+0000","last_peered":"2024-09-11T09:50:43.576045+0000","last_clean":"2024-09-11T09:50:43.576045+0000","last_became_active":"2024-09-11T09:50:34.945865+0000","last_became_peered":"2024-09-11T09:50:34.945865+0000","last_unstale":"2024-09-11T09:50:43.576045+0000","last_undegraded":"2024-09-11T09:50:43.576045+0000","last_fullsized":"2024-09-11T09:50:43.576045+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_clean_scrub_stamp":"2024-09-11T09:50:17.477890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045546699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,4],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":50,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-11T09:50:43.575912+0000","last_change":"2024-09-11T09:50:34.946105+0000","last_active":"2024-09-11T09:50:43.575912+0000","last_peered":"2024-09-11T09:50:43.575912+0000","last_clean":"2024-09-11T09:50:43.575912+0000","last_became_active":"2024-09-11T09:50:34.945846+0000","last_became_peered":"2024-09-11T09:50:34.945846+0000","last_unstale":"2024-09-11T09:50:43.575912+0000","last_undegraded":"2024-09-11T09:50:43.575912+0000","last_fullsized":"2024-09-11T09:50:43.575912+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_clean_scrub_stamp":"2024-09-11T09:50:17.477890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:49:48.552858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00050337500000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"18'2","reported_seq":47,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-11T09:50:44.145878+0000","last_change":"2024-09-11T09:50:20.501218+0000","last_active":"2024-09-11T09:50:44.145878+0000","last_peered":"2024-09-11T09:50:44.145878+0000","last_clean":"2024-09-11T09:50:44.145878+0000","last_became_active":"2024-09-11T09:50:18.493165+0000","last_became_peered":"2024-09-11T09:50:18.493165+0000","last_unstale":"2024-09-11T09:50:44.145878+0000","last_undegraded":"2024-09-11T09:50:44.145878+0000","last_fullsized":"2024-09-11T09:50:44.145878+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_clean_scrub_stamp":"2024-09-11T09:50:17.477890+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T18:04:31.250555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00094463900000000005,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":44,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-11T09:50:44.145773+0000","last_change":"2024-09-11T09:50:21.030566+0000","last_active":"2024-09-11T09:50:44.145773+0000","last_peered":"2024-09-11T09:50:44.145773+0000","last_clean":"2024-09-11T09:50:44.145773+0000","last_became_active":"2024-09-11T09:50:18.493109+0000","last_became_peered":"2024-09-11T09:50:18.493109+0000","last_unstale":"2024-09-11T09:50:44.145773+0000","last_undegraded":"2024-09-11T09:50:44.145773+0000","last_fullsized":"2024-09-11T09:50:44.145773+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_clean_scrub_stamp":"2024-09-11T09:50:17.477890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:37:16.328368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041008499999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":47,"reported_epoch":32,"state":"active+clean","last_fresh":"2024-09-11T09:50:45.587892+0000","last_change":"2024-09-11T09:50:20.498284+0000","last_active":"2024-09-11T09:50:45.587892+0000","last_peered":"2024-09-11T09:50:45.587892+0000","last_clean":"2024-09-11T09:50:45.587892+0000","last_became_active":"2024-09-11T09:50:18.493210+0000","last_became_peered":"2024-09-11T09:50:18.493210+0000","last_unstale":"2024-09-11T09:50:45.587892+0000","last_undegraded":"2024-09-11T09:50:45.587892+0000","last_fullsized":"2024-09-11T09:50:45.587892+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_clean_scrub_stamp":"2024-09-11T09:50:17.477890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T18:16:13.887681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079749499999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"16'1","reported_seq":46,"reported_epoch":31,"state":"active+clean","last_fresh":"2024-09-11T09:50:44.145888+0000","last_change":"2024-09-11T09:50:20.500687+0000","last_active":"2024-09-11T09:50:44.145888+0000","last_peered":"2024-09-11T09:50:44.145888+0000","last_clean":"2024-09-11T09:50:44.145888+0000","last_became_active":"2024-09-11T09:50:18.493650+0000","last_became_peered":"2024-09-11T09:50:18.493650+0000","last_unstale":"2024-09-11T09:50:44.145888+0000","last_undegraded":"2024-09-11T09:50:44.145888+0000","last_fullsized":"2024-09-11T09:50:44.145888+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_clean_scrub_stamp":"2024-09-11T09:50:17.477890+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T14:26:32.838143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041261100000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"13'117","reported_seq":210,"reported_epoch":31,"state":"peering","last_fresh":"2024-09-11T09:50:43.576061+0000","last_change":"2024-09-11T09:50:45.588277+0000","last_active":"2024-09-11T09:50:45.588277+0000","last_peered":"2024-09-11T09:50:43.576061+0000","last_clean":"2024-09-11T09:50:43.576061+0000","last_became_active":"2024-09-11T09:50:37.557755+0000","last_became_peered":"2024-09-11T09:50:37.557755+0000","last_unstale":"2024-09-11T09:50:43.576061+0000","last_undegraded":"2024-09-11T09:50:43.576061+0000","last_fullsized":"2024-09-11T09:50:43.576061+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":26,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:50:10.457262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:50:10.457262+0000","last_clean_scrub_stamp":"2024-09-11T09:50:10.457262+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,1],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":180224,"data_stored":1771104,"data_compressed":99124,"data_compressed_allocated":135168,"data_compressed_original":1736704,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":12,"seq":51539607563,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27432,"kb_used_data":400,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344408,"statfs":{"total":96636764160,"available":96608673792,"internally_reserved":0,"allocated":409600,"data_stored":711759,"data_compressed":32496,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":12,"seq":51539607562,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31512,"kb_used_data":324,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340328,"statfs":{"total":96636764160,"available":96604495872,"internally_reserved":0,"allocated":331776,"data_stored":116700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":11,"seq":47244640267,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27364,"kb_used_data":328,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344476,"statfs":{"total":96636764160,"available":96608743424,"internally_reserved":0,"allocated":335872,"data_stored":116719,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":11,"seq":47244640267,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27356,"kb_used_data":324,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344484,"statfs":{"total":96636764160,"available":96608751616,"internally_reserved":0,"allocated":331776,"data_stored":116700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640266,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31520,"kb_used_data":324,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340320,"statfs":{"total":96636764160,"available":96604487680,"internally_reserved":0,"allocated":331776,"data_stored":116700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640266,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31568,"kb_used_data":376,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340272,"statfs":{"total":96636764160,"available":96604438528,"internally_reserved":0,"allocated":385024,"data_stored":707087,"data_compressed":34132,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32496,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":34132,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32496,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T09:50:51.320 INFO:tasks.thrashosds.thrasher:Setting 2.7 to [4, 5, 3, 0] 2024-09-11T09:50:51.320 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '2.7', '4', '5', '3', '0'] 2024-09-11T09:50:51.320 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pg-upmap-items 2.7 4 5 3 0 2024-09-11T09:50:51.391 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:50:51.392+0000 7fdec5849640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31907) UID: 0 2024-09-11T09:50:51.452 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-11T09:50:51.452 INFO:teuthology.orchestra.run.smithi096.stdout: "ops": [], 2024-09-11T09:50:51.452 INFO:teuthology.orchestra.run.smithi096.stdout: "complaint_time": 30, 2024-09-11T09:50:51.452 INFO:teuthology.orchestra.run.smithi096.stdout: "num_blocked_ops": 0 2024-09-11T09:50:51.452 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-11T09:50:51.460 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-11T09:50:51.492 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:50:51.493+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:50:51.593 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:50:51.594+0000 7fdec5849640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31907) UID: 0 2024-09-11T09:50:51.603 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-11T09:50:51.604 INFO:teuthology.orchestra.run.smithi096.stdout: "size": 20, 2024-09-11T09:50:51.604 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 600, 2024-09-11T09:50:51.604 INFO:teuthology.orchestra.run.smithi096.stdout: "ops": [ 2024-09-11T09:50:51.604 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.604 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:74 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:50:51.604 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.539067+0000", 2024-09-11T09:50:51.604 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 40.064471017999999, 2024-09-11T09:50:51.604 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.006945957, 2024-09-11T09:50:51.604 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:50:51.604 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:50:51.604 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:51.605 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:50:51.605 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:50:51.605 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:50:51.605 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 74 2024-09-11T09:50:51.605 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.605 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:50:51.605 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.605 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:50:51.605 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.539067+0000", 2024-09-11T09:50:51.605 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:51.605 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.605 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.606 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:50:51.606 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.539067+0000", 2024-09-11T09:50:51.606 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:51.606 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.606 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.606 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:50:51.606 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.539068+0000", 2024-09-11T09:50:51.606 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.6000000000000003e-07 2024-09-11T09:50:51.606 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.606 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.606 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:50:51.606 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.539072+0000", 2024-09-11T09:50:51.606 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.0450000000000002e-06 2024-09-11T09:50:51.606 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.607 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.607 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:50:51.607 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.539078+0000", 2024-09-11T09:50:51.607 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.6889999999999998e-06 2024-09-11T09:50:51.607 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.607 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.607 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:50:51.607 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.539083+0000", 2024-09-11T09:50:51.607 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.7180000000000004e-06 2024-09-11T09:50:51.607 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.607 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.607 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:50:51.607 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.540230+0000", 2024-09-11T09:50:51.608 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0011466250000000001 2024-09-11T09:50:51.608 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.608 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.608 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:50:51.608 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.540279+0000", 2024-09-11T09:50:51.608 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.9667999999999999e-05 2024-09-11T09:50:51.608 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.608 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.608 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:50:51.608 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.540357+0000", 2024-09-11T09:50:51.608 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.7863999999999999e-05 2024-09-11T09:50:51.608 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.609 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.609 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:50:51.609 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.541086+0000", 2024-09-11T09:50:51.609 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00072921100000000001 2024-09-11T09:50:51.609 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.609 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.609 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:50:51.609 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.545949+0000", 2024-09-11T09:50:51.609 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0048626040000000004 2024-09-11T09:50:51.609 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.609 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.609 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:50:51.609 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.545989+0000", 2024-09-11T09:50:51.610 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.0259000000000002e-05 2024-09-11T09:50:51.610 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.610 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.610 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:50:51.610 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.546013+0000", 2024-09-11T09:50:51.610 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.3614000000000001e-05 2024-09-11T09:50:51.610 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:51.610 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:50:51.610 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:51.610 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.610 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.610 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:108 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:50:51.610 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.711510+0000", 2024-09-11T09:50:51.611 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 39.892027845999998, 2024-09-11T09:50:51.611 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0054615649999999998, 2024-09-11T09:50:51.611 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:50:51.611 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:50:51.611 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:51.611 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:50:51.611 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:50:51.611 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:50:51.611 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 108 2024-09-11T09:50:51.611 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.611 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:50:51.611 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.611 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:50:51.612 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711510+0000", 2024-09-11T09:50:51.612 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:51.612 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.612 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.612 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:50:51.612 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711510+0000", 2024-09-11T09:50:51.612 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:51.612 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.612 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.612 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:50:51.612 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711511+0000", 2024-09-11T09:50:51.612 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.1999999999999998e-07 2024-09-11T09:50:51.612 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.613 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.613 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:50:51.613 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711516+0000", 2024-09-11T09:50:51.613 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.4140000000000002e-06 2024-09-11T09:50:51.613 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.613 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.613 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:50:51.613 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711527+0000", 2024-09-11T09:50:51.613 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.0169e-05 2024-09-11T09:50:51.613 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.613 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.613 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:50:51.614 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711535+0000", 2024-09-11T09:50:51.614 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.3299999999999999e-06 2024-09-11T09:50:51.614 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.614 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.614 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:50:51.614 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.713763+0000", 2024-09-11T09:50:51.614 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0022281079999999999 2024-09-11T09:50:51.614 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.614 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.614 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:50:51.614 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.713812+0000", 2024-09-11T09:50:51.614 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.8544000000000001e-05 2024-09-11T09:50:51.615 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.615 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.615 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:50:51.615 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.713909+0000", 2024-09-11T09:50:51.615 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.7113000000000003e-05 2024-09-11T09:50:51.615 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.615 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.615 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:50:51.615 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.716873+0000", 2024-09-11T09:50:51.615 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0029642710000000001 2024-09-11T09:50:51.615 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.615 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.615 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:50:51.615 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.716918+0000", 2024-09-11T09:50:51.616 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.5447e-05 2024-09-11T09:50:51.616 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.616 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.616 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:50:51.616 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.716947+0000", 2024-09-11T09:50:51.616 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.8181e-05 2024-09-11T09:50:51.616 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.616 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.616 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:50:51.616 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.716972+0000", 2024-09-11T09:50:51.616 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.5068000000000001e-05 2024-09-11T09:50:51.616 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:51.617 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:50:51.617 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:51.617 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.617 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.617 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:109 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:50:51.617 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.711555+0000", 2024-09-11T09:50:51.617 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 39.891982597000002, 2024-09-11T09:50:51.617 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.005479902, 2024-09-11T09:50:51.617 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:50:51.617 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:50:51.617 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:51.618 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:50:51.618 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:50:51.618 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:50:51.618 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 109 2024-09-11T09:50:51.618 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.618 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:50:51.618 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.618 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:50:51.618 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711555+0000", 2024-09-11T09:50:51.618 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:51.618 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.618 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.618 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:50:51.619 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711555+0000", 2024-09-11T09:50:51.619 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:51.619 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.619 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.619 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:50:51.619 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711556+0000", 2024-09-11T09:50:51.619 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.3200000000000005e-07 2024-09-11T09:50:51.619 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.619 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.619 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:50:51.619 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711601+0000", 2024-09-11T09:50:51.620 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.5349000000000002e-05 2024-09-11T09:50:51.620 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.620 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.620 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:50:51.620 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711612+0000", 2024-09-11T09:50:51.620 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.0701e-05 2024-09-11T09:50:51.620 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.620 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.620 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:50:51.620 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711619+0000", 2024-09-11T09:50:51.620 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.7610000000000002e-06 2024-09-11T09:50:51.620 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.620 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.621 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:50:51.621 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.714371+0000", 2024-09-11T09:50:51.621 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0027526260000000002 2024-09-11T09:50:51.621 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.621 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.621 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:50:51.621 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.714417+0000", 2024-09-11T09:50:51.621 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.5893999999999998e-05 2024-09-11T09:50:51.621 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.621 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.621 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:50:51.621 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.714496+0000", 2024-09-11T09:50:51.622 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.8539000000000001e-05 2024-09-11T09:50:51.622 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.622 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.622 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:50:51.622 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.716910+0000", 2024-09-11T09:50:51.622 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0024137640000000001 2024-09-11T09:50:51.622 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.622 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.622 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:50:51.622 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.716989+0000", 2024-09-11T09:50:51.622 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.9198999999999998e-05 2024-09-11T09:50:51.622 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.622 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.623 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:50:51.623 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.717016+0000", 2024-09-11T09:50:51.623 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.6699e-05 2024-09-11T09:50:51.623 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.623 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.623 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:50:51.623 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.717035+0000", 2024-09-11T09:50:51.623 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.9738e-05 2024-09-11T09:50:51.623 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:51.623 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:50:51.623 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:51.623 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.623 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.623 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:110 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:50:51.624 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.711636+0000", 2024-09-11T09:50:51.624 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 39.891902137000002, 2024-09-11T09:50:51.624 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0067613070000000003, 2024-09-11T09:50:51.624 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:50:51.624 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:50:51.624 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:51.624 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:50:51.624 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:50:51.624 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:50:51.624 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 110 2024-09-11T09:50:51.624 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.624 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:50:51.624 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.624 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:50:51.625 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711636+0000", 2024-09-11T09:50:51.625 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:51.625 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.625 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.625 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:50:51.625 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711636+0000", 2024-09-11T09:50:51.625 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:51.625 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.625 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.625 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:50:51.625 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711637+0000", 2024-09-11T09:50:51.625 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.9999999999999997e-07 2024-09-11T09:50:51.625 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.625 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.626 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:50:51.626 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711641+0000", 2024-09-11T09:50:51.626 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.2570000000000001e-06 2024-09-11T09:50:51.626 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.626 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.626 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:50:51.626 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711651+0000", 2024-09-11T09:50:51.626 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.0285999999999999e-05 2024-09-11T09:50:51.626 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.626 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.626 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:50:51.626 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711658+0000", 2024-09-11T09:50:51.626 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.3540000000000002e-06 2024-09-11T09:50:51.627 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.627 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.627 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:50:51.627 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.716475+0000", 2024-09-11T09:50:51.627 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0048162819999999999 2024-09-11T09:50:51.627 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.627 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.627 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:50:51.627 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.716526+0000", 2024-09-11T09:50:51.627 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.0905000000000001e-05 2024-09-11T09:50:51.627 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.627 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.628 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:50:51.628 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.716601+0000", 2024-09-11T09:50:51.628 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.5283000000000004e-05 2024-09-11T09:50:51.628 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.628 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.628 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:50:51.628 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.717933+0000", 2024-09-11T09:50:51.628 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001331622 2024-09-11T09:50:51.628 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.628 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.628 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:50:51.628 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718332+0000", 2024-09-11T09:50:51.628 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00039909899999999998 2024-09-11T09:50:51.629 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.629 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.629 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:50:51.629 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718363+0000", 2024-09-11T09:50:51.629 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.1636e-05 2024-09-11T09:50:51.629 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.629 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.629 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:50:51.629 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718397+0000", 2024-09-11T09:50:51.629 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.3883000000000001e-05 2024-09-11T09:50:51.629 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:51.629 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:50:51.629 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:51.630 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.630 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.630 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:111 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:50:51.630 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.711668+0000", 2024-09-11T09:50:51.630 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 39.891869706999998, 2024-09-11T09:50:51.630 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0077145570000000004, 2024-09-11T09:50:51.630 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:50:51.630 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:50:51.630 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:51.630 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:50:51.630 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:50:51.630 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:50:51.630 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 111 2024-09-11T09:50:51.631 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.631 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:50:51.631 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.631 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:50:51.631 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711668+0000", 2024-09-11T09:50:51.631 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:51.631 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.631 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.631 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:50:51.631 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711668+0000", 2024-09-11T09:50:51.631 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:51.631 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.632 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.632 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:50:51.632 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711669+0000", 2024-09-11T09:50:51.632 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.99e-07 2024-09-11T09:50:51.632 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.632 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.632 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:50:51.632 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711673+0000", 2024-09-11T09:50:51.632 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.7440000000000001e-06 2024-09-11T09:50:51.632 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.632 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.632 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:50:51.633 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711686+0000", 2024-09-11T09:50:51.633 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.3317e-05 2024-09-11T09:50:51.633 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.633 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.633 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:50:51.633 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711697+0000", 2024-09-11T09:50:51.633 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1022000000000001e-05 2024-09-11T09:50:51.633 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.633 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.633 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:50:51.633 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.717065+0000", 2024-09-11T09:50:51.633 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.005367531 2024-09-11T09:50:51.633 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.634 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.634 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:50:51.634 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.717114+0000", 2024-09-11T09:50:51.634 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.9707999999999999e-05 2024-09-11T09:50:51.634 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.634 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.634 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:50:51.634 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.717190+0000", 2024-09-11T09:50:51.634 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.5394000000000004e-05 2024-09-11T09:50:51.634 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.634 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.634 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:50:51.634 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718793+0000", 2024-09-11T09:50:51.634 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001603653 2024-09-11T09:50:51.635 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.635 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.635 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:50:51.635 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719329+0000", 2024-09-11T09:50:51.635 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00053543799999999995 2024-09-11T09:50:51.635 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.635 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.635 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:50:51.635 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719359+0000", 2024-09-11T09:50:51.635 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.0420999999999999e-05 2024-09-11T09:50:51.635 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.635 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.635 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:50:51.636 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719383+0000", 2024-09-11T09:50:51.636 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.3730000000000001e-05 2024-09-11T09:50:51.636 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:51.636 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:50:51.636 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:51.636 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.636 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.636 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:112 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:50:51.636 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.711712+0000", 2024-09-11T09:50:51.636 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 39.891826147000003, 2024-09-11T09:50:51.636 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.008162084, 2024-09-11T09:50:51.636 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:50:51.636 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:50:51.636 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:51.637 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:50:51.637 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:50:51.637 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:50:51.637 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 112 2024-09-11T09:50:51.637 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.637 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:50:51.637 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.637 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:50:51.637 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711712+0000", 2024-09-11T09:50:51.637 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:51.637 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.637 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.637 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:50:51.637 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711712+0000", 2024-09-11T09:50:51.638 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:51.638 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.638 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.638 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:50:51.638 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711712+0000", 2024-09-11T09:50:51.638 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.6000000000000004e-07 2024-09-11T09:50:51.638 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.638 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.638 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:50:51.638 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711808+0000", 2024-09-11T09:50:51.638 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.5977999999999995e-05 2024-09-11T09:50:51.638 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.639 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.639 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:50:51.639 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711819+0000", 2024-09-11T09:50:51.639 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.0755e-05 2024-09-11T09:50:51.639 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.639 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.639 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:50:51.639 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711826+0000", 2024-09-11T09:50:51.639 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.7970000000000003e-06 2024-09-11T09:50:51.639 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.639 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.639 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:50:51.640 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.717435+0000", 2024-09-11T09:50:51.640 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0056090239999999998 2024-09-11T09:50:51.640 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.640 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.640 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:50:51.640 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.717496+0000", 2024-09-11T09:50:51.640 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.0543e-05 2024-09-11T09:50:51.640 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.640 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.640 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:50:51.640 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.717591+0000", 2024-09-11T09:50:51.640 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.5667000000000004e-05 2024-09-11T09:50:51.640 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.640 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.641 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:50:51.641 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718831+0000", 2024-09-11T09:50:51.641 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001240204 2024-09-11T09:50:51.641 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.641 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.641 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:50:51.641 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719819+0000", 2024-09-11T09:50:51.641 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00098751500000000001 2024-09-11T09:50:51.641 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.641 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.641 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:50:51.641 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719850+0000", 2024-09-11T09:50:51.641 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.1535999999999998e-05 2024-09-11T09:50:51.641 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.642 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.642 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:50:51.642 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719874+0000", 2024-09-11T09:50:51.642 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.3504999999999999e-05 2024-09-11T09:50:51.642 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:51.642 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:50:51.642 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:51.642 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.642 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.642 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:113 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:50:51.642 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.711842+0000", 2024-09-11T09:50:51.642 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 39.891695648000002, 2024-09-11T09:50:51.642 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0081160099999999999, 2024-09-11T09:50:51.643 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:50:51.643 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:50:51.643 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:51.643 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:50:51.643 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:50:51.643 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:50:51.643 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 113 2024-09-11T09:50:51.643 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.643 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:50:51.643 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.643 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:50:51.643 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711842+0000", 2024-09-11T09:50:51.643 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:51.644 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.644 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.644 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:50:51.644 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711842+0000", 2024-09-11T09:50:51.644 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:51.644 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.644 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.644 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:50:51.644 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711843+0000", 2024-09-11T09:50:51.644 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.4399999999999999e-07 2024-09-11T09:50:51.644 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.644 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.645 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:50:51.645 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711848+0000", 2024-09-11T09:50:51.645 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.4159999999999997e-06 2024-09-11T09:50:51.645 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.645 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.645 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:50:51.645 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711856+0000", 2024-09-11T09:50:51.645 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.7870000000000007e-06 2024-09-11T09:50:51.645 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.646 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.646 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:50:51.646 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711862+0000", 2024-09-11T09:50:51.646 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.3689999999999998e-06 2024-09-11T09:50:51.646 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.646 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.646 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:50:51.646 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.717965+0000", 2024-09-11T09:50:51.646 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0061028899999999997 2024-09-11T09:50:51.646 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.646 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.646 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:50:51.646 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718023+0000", 2024-09-11T09:50:51.647 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.8313000000000002e-05 2024-09-11T09:50:51.647 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.647 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.647 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:50:51.647 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718109+0000", 2024-09-11T09:50:51.647 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.6205000000000006e-05 2024-09-11T09:50:51.647 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.647 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.648 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:50:51.648 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719808+0000", 2024-09-11T09:50:51.648 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0016990060000000001 2024-09-11T09:50:51.648 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.648 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.648 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:50:51.648 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719883+0000", 2024-09-11T09:50:51.648 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.4624999999999998e-05 2024-09-11T09:50:51.648 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.648 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.648 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:50:51.648 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719910+0000", 2024-09-11T09:50:51.648 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.7512e-05 2024-09-11T09:50:51.649 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.649 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.649 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:50:51.649 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719958+0000", 2024-09-11T09:50:51.649 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.8143000000000003e-05 2024-09-11T09:50:51.649 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:51.649 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:50:51.649 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:51.649 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.649 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.649 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:50:51.649 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.711887+0000", 2024-09-11T09:50:51.650 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 39.891651033999999, 2024-09-11T09:50:51.650 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0086063589999999992, 2024-09-11T09:50:51.650 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:50:51.650 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:50:51.650 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:51.650 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:50:51.650 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:50:51.650 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:50:51.650 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 114 2024-09-11T09:50:51.650 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.650 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:50:51.650 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.650 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:50:51.651 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711887+0000", 2024-09-11T09:50:51.651 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:51.651 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.651 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.651 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:50:51.651 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711887+0000", 2024-09-11T09:50:51.651 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:51.651 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.651 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.652 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:50:51.652 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711887+0000", 2024-09-11T09:50:51.652 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.1500000000000005e-07 2024-09-11T09:50:51.652 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.652 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.652 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:50:51.652 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711897+0000", 2024-09-11T09:50:51.652 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.1900000000000001e-06 2024-09-11T09:50:51.652 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.652 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.652 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:50:51.652 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711902+0000", 2024-09-11T09:50:51.652 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.2349999999999999e-06 2024-09-11T09:50:51.652 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.653 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.653 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:50:51.653 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711906+0000", 2024-09-11T09:50:51.653 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.7129999999999999e-06 2024-09-11T09:50:51.653 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.653 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.653 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:50:51.653 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718429+0000", 2024-09-11T09:50:51.653 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0065231960000000002 2024-09-11T09:50:51.654 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.654 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.654 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:50:51.654 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718480+0000", 2024-09-11T09:50:51.654 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.1161e-05 2024-09-11T09:50:51.654 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.654 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.654 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:50:51.654 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718558+0000", 2024-09-11T09:50:51.654 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.7519000000000001e-05 2024-09-11T09:50:51.654 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.654 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.655 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:50:51.655 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720038+0000", 2024-09-11T09:50:51.655 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001480016 2024-09-11T09:50:51.655 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.655 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.655 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:50:51.655 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720417+0000", 2024-09-11T09:50:51.655 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00037943699999999999 2024-09-11T09:50:51.655 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.655 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.656 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:50:51.656 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720452+0000", 2024-09-11T09:50:51.656 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.4595000000000003e-05 2024-09-11T09:50:51.656 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.656 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.656 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:50:51.656 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720493+0000", 2024-09-11T09:50:51.656 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.1782000000000001e-05 2024-09-11T09:50:51.656 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:51.656 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:50:51.656 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:51.656 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.657 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.657 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:50:51.657 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.711912+0000", 2024-09-11T09:50:51.657 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 39.891626365, 2024-09-11T09:50:51.657 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0086864750000000008, 2024-09-11T09:50:51.657 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:50:51.657 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:50:51.657 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:51.657 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:50:51.657 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:50:51.658 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:50:51.658 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 115 2024-09-11T09:50:51.658 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.658 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:50:51.658 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.658 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:50:51.658 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711912+0000", 2024-09-11T09:50:51.658 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:51.658 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.658 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.658 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:50:51.658 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711912+0000", 2024-09-11T09:50:51.658 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:51.658 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.659 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.659 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:50:51.659 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711912+0000", 2024-09-11T09:50:51.659 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.3900000000000001e-07 2024-09-11T09:50:51.659 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.659 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.659 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:50:51.659 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711969+0000", 2024-09-11T09:50:51.659 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.7501999999999999e-05 2024-09-11T09:50:51.659 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.659 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.659 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:50:51.659 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711986+0000", 2024-09-11T09:50:51.660 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.6402999999999999e-05 2024-09-11T09:50:51.660 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.660 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.660 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:50:51.660 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711990+0000", 2024-09-11T09:50:51.660 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.929e-06 2024-09-11T09:50:51.660 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.660 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.660 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:50:51.660 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718861+0000", 2024-09-11T09:50:51.660 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0068710530000000002 2024-09-11T09:50:51.660 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.661 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.661 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:50:51.661 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718908+0000", 2024-09-11T09:50:51.661 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.7431e-05 2024-09-11T09:50:51.661 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.661 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.661 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:50:51.661 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718984+0000", 2024-09-11T09:50:51.661 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.5500999999999994e-05 2024-09-11T09:50:51.661 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.661 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.661 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:50:51.661 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720500+0000", 2024-09-11T09:50:51.661 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0015158820000000001 2024-09-11T09:50:51.662 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.662 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.662 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:50:51.662 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720536+0000", 2024-09-11T09:50:51.662 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.6118000000000002e-05 2024-09-11T09:50:51.662 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.662 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.662 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:50:51.662 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720570+0000", 2024-09-11T09:50:51.662 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.4387999999999997e-05 2024-09-11T09:50:51.662 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.662 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.662 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:50:51.663 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720598+0000", 2024-09-11T09:50:51.663 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.8028999999999999e-05 2024-09-11T09:50:51.663 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:51.663 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:50:51.663 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:51.663 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.663 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.663 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:50:51.663 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.711998+0000", 2024-09-11T09:50:51.663 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 39.891540186, 2024-09-11T09:50:51.663 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.010785813999999999, 2024-09-11T09:50:51.663 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:50:51.663 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:50:51.663 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:51.664 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:50:51.664 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:50:51.664 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:50:51.664 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 116 2024-09-11T09:50:51.664 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.664 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:50:51.664 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.664 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:50:51.664 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711998+0000", 2024-09-11T09:50:51.664 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:51.664 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.664 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.664 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:50:51.665 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711998+0000", 2024-09-11T09:50:51.665 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:51.665 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.665 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.665 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:50:51.665 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711998+0000", 2024-09-11T09:50:51.665 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.3300000000000003e-07 2024-09-11T09:50:51.665 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.665 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.665 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:50:51.665 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712001+0000", 2024-09-11T09:50:51.665 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.9349999999999999e-06 2024-09-11T09:50:51.665 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.666 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.666 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:50:51.666 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712006+0000", 2024-09-11T09:50:51.666 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.1630000000000004e-06 2024-09-11T09:50:51.666 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.666 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.666 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:50:51.666 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712010+0000", 2024-09-11T09:50:51.666 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.574e-06 2024-09-11T09:50:51.666 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.666 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.666 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:50:51.666 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719415+0000", 2024-09-11T09:50:51.666 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0074049099999999998 2024-09-11T09:50:51.666 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.667 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.667 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:50:51.667 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719465+0000", 2024-09-11T09:50:51.667 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.0129000000000001e-05 2024-09-11T09:50:51.667 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.667 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.667 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:50:51.667 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719545+0000", 2024-09-11T09:50:51.667 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.0253999999999994e-05 2024-09-11T09:50:51.667 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.667 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.667 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:50:51.668 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720636+0000", 2024-09-11T09:50:51.668 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0010909330000000001 2024-09-11T09:50:51.668 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.668 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.668 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:50:51.668 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.722707+0000", 2024-09-11T09:50:51.668 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0020713300000000001 2024-09-11T09:50:51.668 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.668 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.668 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:50:51.668 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.722743+0000", 2024-09-11T09:50:51.668 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.5351000000000001e-05 2024-09-11T09:50:51.668 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.669 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.669 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:50:51.669 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.722784+0000", 2024-09-11T09:50:51.669 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.0802e-05 2024-09-11T09:50:51.669 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:51.669 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:50:51.669 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:51.669 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.669 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.669 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:50:51.669 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.712014+0000", 2024-09-11T09:50:51.669 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 39.891524146000002, 2024-09-11T09:50:51.669 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.013420938, 2024-09-11T09:50:51.670 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:50:51.670 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:50:51.670 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:51.670 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:50:51.670 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:50:51.670 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:50:51.670 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 117 2024-09-11T09:50:51.670 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.670 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:50:51.670 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.670 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:50:51.670 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712014+0000", 2024-09-11T09:50:51.671 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:51.671 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.671 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.671 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:50:51.671 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712014+0000", 2024-09-11T09:50:51.671 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:51.671 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.671 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.671 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:50:51.671 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712014+0000", 2024-09-11T09:50:51.671 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.2000000000000001e-07 2024-09-11T09:50:51.671 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.672 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.672 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:50:51.672 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712016+0000", 2024-09-11T09:50:51.672 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.2309999999999999e-06 2024-09-11T09:50:51.672 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.672 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.672 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:50:51.672 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712020+0000", 2024-09-11T09:50:51.672 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.3830000000000001e-06 2024-09-11T09:50:51.672 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.672 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.672 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:50:51.672 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712023+0000", 2024-09-11T09:50:51.672 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.3340000000000002e-06 2024-09-11T09:50:51.673 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.673 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.673 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:50:51.673 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720083+0000", 2024-09-11T09:50:51.673 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0080601189999999993 2024-09-11T09:50:51.673 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.673 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.673 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:50:51.673 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720125+0000", 2024-09-11T09:50:51.673 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.2029e-05 2024-09-11T09:50:51.673 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.673 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.673 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:50:51.674 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720205+0000", 2024-09-11T09:50:51.674 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.0010000000000001e-05 2024-09-11T09:50:51.674 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.674 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.674 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:50:51.674 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720849+0000", 2024-09-11T09:50:51.674 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00064369899999999996 2024-09-11T09:50:51.674 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.674 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.674 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:50:51.674 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.725357+0000", 2024-09-11T09:50:51.674 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0045077379999999998 2024-09-11T09:50:51.675 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.675 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.675 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:50:51.675 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.725392+0000", 2024-09-11T09:50:51.675 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.5862999999999999e-05 2024-09-11T09:50:51.675 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.675 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.675 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:50:51.675 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.725435+0000", 2024-09-11T09:50:51.675 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.2311999999999997e-05 2024-09-11T09:50:51.675 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:51.675 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:50:51.675 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:51.675 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.676 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.676 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:119 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 131072~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:50:51.676 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.725673+0000", 2024-09-11T09:50:51.676 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 39.877865444000001, 2024-09-11T09:50:51.676 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0053927460000000003, 2024-09-11T09:50:51.676 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:50:51.676 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:50:51.676 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:51.676 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:50:51.676 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:50:51.676 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:50:51.677 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 119 2024-09-11T09:50:51.677 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.677 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:50:51.677 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.677 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:50:51.677 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.725673+0000", 2024-09-11T09:50:51.677 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:51.677 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.677 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.677 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:50:51.677 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.725673+0000", 2024-09-11T09:50:51.677 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:51.677 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.678 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.678 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:50:51.678 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.725673+0000", 2024-09-11T09:50:51.678 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.68e-07 2024-09-11T09:50:51.678 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.678 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.678 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:50:51.678 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.725720+0000", 2024-09-11T09:50:51.678 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.6526999999999997e-05 2024-09-11T09:50:51.678 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.678 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.679 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:50:51.679 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.725734+0000", 2024-09-11T09:50:51.679 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.4593000000000001e-05 2024-09-11T09:50:51.679 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.679 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.679 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:50:51.679 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.725741+0000", 2024-09-11T09:50:51.679 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.28e-06 2024-09-11T09:50:51.679 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.679 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.679 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:50:51.679 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.726290+0000", 2024-09-11T09:50:51.680 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00054892000000000005 2024-09-11T09:50:51.680 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.680 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.680 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:50:51.680 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.726338+0000", 2024-09-11T09:50:51.680 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.7982000000000002e-05 2024-09-11T09:50:51.680 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.680 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.680 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:50:51.680 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.726413+0000", 2024-09-11T09:50:51.680 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.5679000000000005e-05 2024-09-11T09:50:51.680 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.681 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.681 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:50:51.681 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.727555+0000", 2024-09-11T09:50:51.681 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0011420289999999999 2024-09-11T09:50:51.681 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.681 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.681 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:50:51.681 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.731004+0000", 2024-09-11T09:50:51.681 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0034484279999999999 2024-09-11T09:50:51.681 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.681 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.681 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:50:51.681 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.731040+0000", 2024-09-11T09:50:51.682 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.6315e-05 2024-09-11T09:50:51.682 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.682 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.682 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:50:51.682 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.731065+0000", 2024-09-11T09:50:51.682 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.5324999999999999e-05 2024-09-11T09:50:51.682 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:51.682 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:50:51.682 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:51.682 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.682 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.682 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:120 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 196608~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:50:51.682 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.725756+0000", 2024-09-11T09:50:51.683 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 39.877781534999997, 2024-09-11T09:50:51.683 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0053569100000000003, 2024-09-11T09:50:51.683 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:50:51.683 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:50:51.683 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:51.683 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:50:51.683 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:50:51.683 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:50:51.683 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 120 2024-09-11T09:50:51.683 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.683 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:50:51.683 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.683 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:50:51.683 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.725756+0000", 2024-09-11T09:50:51.684 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:51.684 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.684 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.684 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:50:51.684 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.725756+0000", 2024-09-11T09:50:51.684 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:51.684 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.684 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.684 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:50:51.684 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.725757+0000", 2024-09-11T09:50:51.684 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.3799999999999997e-07 2024-09-11T09:50:51.684 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.685 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.685 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:50:51.685 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.725802+0000", 2024-09-11T09:50:51.685 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.4790000000000003e-05 2024-09-11T09:50:51.685 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.685 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.685 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:50:51.685 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.725815+0000", 2024-09-11T09:50:51.685 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.2609000000000001e-05 2024-09-11T09:50:51.685 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.685 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.685 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:50:51.686 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.725822+0000", 2024-09-11T09:50:51.686 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.6450000000000005e-06 2024-09-11T09:50:51.686 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.686 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.686 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:50:51.686 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.726708+0000", 2024-09-11T09:50:51.686 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00088606500000000001 2024-09-11T09:50:51.686 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.686 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.686 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:50:51.686 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.726755+0000", 2024-09-11T09:50:51.686 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.6888000000000002e-05 2024-09-11T09:50:51.686 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.687 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.687 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:50:51.687 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.726828+0000", 2024-09-11T09:50:51.687 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.2510999999999997e-05 2024-09-11T09:50:51.687 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.687 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.687 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:50:51.687 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.727829+0000", 2024-09-11T09:50:51.687 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001001093 2024-09-11T09:50:51.687 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.687 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.687 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:50:51.688 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.731071+0000", 2024-09-11T09:50:51.688 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0032426009999999999 2024-09-11T09:50:51.688 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.688 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.688 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:50:51.688 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.731094+0000", 2024-09-11T09:50:51.688 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.2677e-05 2024-09-11T09:50:51.688 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.688 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.688 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:50:51.688 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.731113+0000", 2024-09-11T09:50:51.688 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.9392999999999999e-05 2024-09-11T09:50:51.689 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:51.689 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:50:51.689 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:51.689 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.689 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.689 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:138 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:50:51.689 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:12.034038+0000", 2024-09-11T09:50:51.689 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 39.569500204999997, 2024-09-11T09:50:51.689 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0054142749999999996, 2024-09-11T09:50:51.689 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:50:51.689 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:50:51.689 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:51.689 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:50:51.689 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:50:51.690 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:50:51.690 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 138 2024-09-11T09:50:51.690 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.690 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:50:51.690 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.690 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:50:51.690 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034038+0000", 2024-09-11T09:50:51.690 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:51.690 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.690 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.690 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:50:51.690 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034038+0000", 2024-09-11T09:50:51.690 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:51.691 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.691 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.691 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:50:51.691 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034038+0000", 2024-09-11T09:50:51.691 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.3499999999999996e-07 2024-09-11T09:50:51.691 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.691 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.691 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:50:51.691 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034043+0000", 2024-09-11T09:50:51.691 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.7310000000000001e-06 2024-09-11T09:50:51.691 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.691 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.691 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:50:51.691 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034050+0000", 2024-09-11T09:50:51.691 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.0419999999999997e-06 2024-09-11T09:50:51.692 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.692 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.692 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:50:51.692 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034055+0000", 2024-09-11T09:50:51.692 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.3299999999999998e-06 2024-09-11T09:50:51.692 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.692 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.692 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:50:51.692 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.037309+0000", 2024-09-11T09:50:51.692 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0032535440000000001 2024-09-11T09:50:51.692 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.692 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.692 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:50:51.693 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.037357+0000", 2024-09-11T09:50:51.693 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.8167999999999997e-05 2024-09-11T09:50:51.693 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.693 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.693 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:50:51.693 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.037430+0000", 2024-09-11T09:50:51.693 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.3275999999999994e-05 2024-09-11T09:50:51.693 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.693 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.693 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:50:51.693 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.039014+0000", 2024-09-11T09:50:51.693 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0015831350000000001 2024-09-11T09:50:51.693 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.694 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.694 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:50:51.694 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.039401+0000", 2024-09-11T09:50:51.694 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00038758500000000002 2024-09-11T09:50:51.694 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.694 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.694 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:50:51.694 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.039429+0000", 2024-09-11T09:50:51.694 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.7379e-05 2024-09-11T09:50:51.694 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.694 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.694 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:50:51.694 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.039452+0000", 2024-09-11T09:50:51.695 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.355e-05 2024-09-11T09:50:51.695 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:51.695 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:50:51.695 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:51.695 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.695 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.695 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:139 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:50:51.695 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:12.034064+0000", 2024-09-11T09:50:51.695 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 39.569474174, 2024-09-11T09:50:51.695 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0054736200000000002, 2024-09-11T09:50:51.695 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:50:51.695 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:50:51.695 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:51.696 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:50:51.696 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:50:51.696 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:50:51.696 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 139 2024-09-11T09:50:51.696 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.696 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:50:51.696 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.696 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:50:51.696 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034064+0000", 2024-09-11T09:50:51.696 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:51.696 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.696 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.697 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:50:51.697 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034064+0000", 2024-09-11T09:50:51.697 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:51.697 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.697 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.697 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:50:51.697 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034064+0000", 2024-09-11T09:50:51.697 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.89e-07 2024-09-11T09:50:51.697 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.697 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.697 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:50:51.697 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034069+0000", 2024-09-11T09:50:51.697 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.9799999999999998e-06 2024-09-11T09:50:51.698 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.698 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.698 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:50:51.698 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034079+0000", 2024-09-11T09:50:51.698 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.3549999999999994e-06 2024-09-11T09:50:51.698 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.698 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.698 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:50:51.698 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034084+0000", 2024-09-11T09:50:51.698 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.2209999999999996e-06 2024-09-11T09:50:51.698 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.698 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.698 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:50:51.699 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.038176+0000", 2024-09-11T09:50:51.699 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0040921610000000004 2024-09-11T09:50:51.699 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.699 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.699 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:50:51.699 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.038221+0000", 2024-09-11T09:50:51.699 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.5006000000000002e-05 2024-09-11T09:50:51.699 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.699 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.699 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:50:51.699 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.038293+0000", 2024-09-11T09:50:51.699 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.1952999999999994e-05 2024-09-11T09:50:51.699 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.700 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.700 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:50:51.700 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.039461+0000", 2024-09-11T09:50:51.700 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0011680340000000001 2024-09-11T09:50:51.700 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.700 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.700 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:50:51.700 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.039487+0000", 2024-09-11T09:50:51.700 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.5752e-05 2024-09-11T09:50:51.700 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.700 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.700 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:50:51.700 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.039515+0000", 2024-09-11T09:50:51.700 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.7763000000000002e-05 2024-09-11T09:50:51.701 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.701 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.701 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:50:51.701 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.039537+0000", 2024-09-11T09:50:51.701 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.2906000000000001e-05 2024-09-11T09:50:51.701 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:51.701 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:50:51.701 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:51.701 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.701 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.701 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:140 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:50:51.702 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:12.034093+0000", 2024-09-11T09:50:51.702 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 39.569444670000003, 2024-09-11T09:50:51.702 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0059112490000000004, 2024-09-11T09:50:51.702 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:50:51.702 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:50:51.702 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:51.702 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:50:51.702 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:50:51.702 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:50:51.702 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 140 2024-09-11T09:50:51.702 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.702 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:50:51.702 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.702 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:50:51.703 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034093+0000", 2024-09-11T09:50:51.703 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:51.703 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.703 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.703 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:50:51.703 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034093+0000", 2024-09-11T09:50:51.703 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:51.703 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.703 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.703 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:50:51.703 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034094+0000", 2024-09-11T09:50:51.703 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.4299999999999998e-07 2024-09-11T09:50:51.703 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.703 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.703 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:50:51.704 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034159+0000", 2024-09-11T09:50:51.704 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.567e-05 2024-09-11T09:50:51.704 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.704 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.704 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:50:51.704 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034167+0000", 2024-09-11T09:50:51.704 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.8490000000000006e-06 2024-09-11T09:50:51.704 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.704 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.704 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:50:51.704 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034173+0000", 2024-09-11T09:50:51.704 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.1770000000000003e-06 2024-09-11T09:50:51.704 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.704 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.705 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:50:51.705 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.038543+0000", 2024-09-11T09:50:51.705 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0043699150000000003 2024-09-11T09:50:51.705 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.705 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.705 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:50:51.705 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.038589+0000", 2024-09-11T09:50:51.705 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.5562999999999997e-05 2024-09-11T09:50:51.705 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.705 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.705 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:50:51.705 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.038662+0000", 2024-09-11T09:50:51.706 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.2748000000000002e-05 2024-09-11T09:50:51.706 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.706 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.706 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:50:51.706 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.039635+0000", 2024-09-11T09:50:51.706 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00097310799999999998 2024-09-11T09:50:51.706 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.706 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.706 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:50:51.706 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.039945+0000", 2024-09-11T09:50:51.706 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00031054100000000001 2024-09-11T09:50:51.706 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.706 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.706 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:50:51.707 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.039971+0000", 2024-09-11T09:50:51.707 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.5423999999999999e-05 2024-09-11T09:50:51.707 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.707 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.707 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:50:51.707 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.040005+0000", 2024-09-11T09:50:51.707 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.3810999999999999e-05 2024-09-11T09:50:51.707 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:51.707 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:50:51.707 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:51.707 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.707 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.707 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:50:51.708 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:12.034184+0000", 2024-09-11T09:50:51.708 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 39.569353972000002, 2024-09-11T09:50:51.708 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0061520380000000003, 2024-09-11T09:50:51.708 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:50:51.708 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:50:51.708 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:51.708 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:50:51.708 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:50:51.708 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:50:51.708 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 141 2024-09-11T09:50:51.708 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.708 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:50:51.708 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.708 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:50:51.709 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034184+0000", 2024-09-11T09:50:51.709 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:51.709 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.709 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.709 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:50:51.709 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034184+0000", 2024-09-11T09:50:51.709 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:51.709 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.709 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.709 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:50:51.709 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034185+0000", 2024-09-11T09:50:51.709 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.1099999999999995e-07 2024-09-11T09:50:51.710 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.710 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.710 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:50:51.710 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034190+0000", 2024-09-11T09:50:51.710 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.7309999999999998e-06 2024-09-11T09:50:51.710 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.710 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.710 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:50:51.710 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034197+0000", 2024-09-11T09:50:51.710 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.7440000000000001e-06 2024-09-11T09:50:51.710 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.710 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.710 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:50:51.711 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034204+0000", 2024-09-11T09:50:51.711 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.8970000000000002e-06 2024-09-11T09:50:51.711 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.711 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.711 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:50:51.711 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.039035+0000", 2024-09-11T09:50:51.711 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0048307039999999999 2024-09-11T09:50:51.711 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.711 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.711 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:50:51.711 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.039093+0000", 2024-09-11T09:50:51.711 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.8604000000000003e-05 2024-09-11T09:50:51.711 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.711 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.712 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:50:51.712 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.039184+0000", 2024-09-11T09:50:51.712 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.0196000000000002e-05 2024-09-11T09:50:51.712 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.712 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.712 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:50:51.712 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.039873+0000", 2024-09-11T09:50:51.712 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00068960600000000005 2024-09-11T09:50:51.712 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.712 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.712 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:50:51.713 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.040292+0000", 2024-09-11T09:50:51.713 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00041865199999999999 2024-09-11T09:50:51.713 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.713 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.713 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:50:51.713 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.040317+0000", 2024-09-11T09:50:51.713 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.5439000000000002e-05 2024-09-11T09:50:51.713 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.713 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.713 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:50:51.713 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.040336+0000", 2024-09-11T09:50:51.713 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.8754e-05 2024-09-11T09:50:51.713 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:51.714 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:50:51.714 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:51.714 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.714 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.714 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:157 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:50:51.714 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:12.452040+0000", 2024-09-11T09:50:51.714 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 39.151498119000003, 2024-09-11T09:50:51.714 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0068848269999999996, 2024-09-11T09:50:51.714 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:50:51.714 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:50:51.714 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:51.714 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:50:51.714 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:50:51.715 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:50:51.715 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 157 2024-09-11T09:50:51.715 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.715 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:50:51.715 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.715 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:50:51.715 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452040+0000", 2024-09-11T09:50:51.715 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:51.715 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.715 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.715 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:50:51.715 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452040+0000", 2024-09-11T09:50:51.715 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:51.716 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.716 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.716 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:50:51.716 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452040+0000", 2024-09-11T09:50:51.716 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.22e-07 2024-09-11T09:50:51.716 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.716 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.716 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:50:51.716 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452042+0000", 2024-09-11T09:50:51.716 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.7010000000000001e-06 2024-09-11T09:50:51.716 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.716 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.717 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:50:51.717 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452045+0000", 2024-09-11T09:50:51.717 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.1310000000000002e-06 2024-09-11T09:50:51.717 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.717 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.717 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:50:51.717 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452048+0000", 2024-09-11T09:50:51.717 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.65e-06 2024-09-11T09:50:51.717 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.717 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.717 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:50:51.717 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.454453+0000", 2024-09-11T09:50:51.717 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0024052959999999999 2024-09-11T09:50:51.718 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.718 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.718 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:50:51.718 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.454498+0000", 2024-09-11T09:50:51.718 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.4978999999999997e-05 2024-09-11T09:50:51.718 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.718 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.718 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:50:51.718 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.454567+0000", 2024-09-11T09:50:51.718 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.9110999999999996e-05 2024-09-11T09:50:51.718 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.718 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.719 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:50:51.719 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.455464+0000", 2024-09-11T09:50:51.719 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00089696199999999995 2024-09-11T09:50:51.719 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.719 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.719 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:50:51.719 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.458844+0000", 2024-09-11T09:50:51.719 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0033802469999999999 2024-09-11T09:50:51.719 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.719 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.719 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:50:51.719 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.458889+0000", 2024-09-11T09:50:51.719 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.4441999999999999e-05 2024-09-11T09:50:51.720 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.720 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.720 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:50:51.720 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.458925+0000", 2024-09-11T09:50:51.720 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.6086e-05 2024-09-11T09:50:51.720 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:51.720 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:50:51.720 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:51.720 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.720 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.720 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:158 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:50:51.720 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:12.452055+0000", 2024-09-11T09:50:51.720 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 39.151482725999998, 2024-09-11T09:50:51.721 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0069533829999999996, 2024-09-11T09:50:51.721 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:50:51.721 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:50:51.721 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:51.721 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:50:51.721 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:50:51.721 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:50:51.721 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 158 2024-09-11T09:50:51.721 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.721 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:50:51.721 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.721 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:50:51.721 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452055+0000", 2024-09-11T09:50:51.722 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:51.722 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.723 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.723 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:50:51.723 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452055+0000", 2024-09-11T09:50:51.723 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:51.723 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.723 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.723 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:50:51.723 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452056+0000", 2024-09-11T09:50:51.723 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.48e-07 2024-09-11T09:50:51.723 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.723 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.723 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:50:51.724 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452087+0000", 2024-09-11T09:50:51.724 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.1149000000000003e-05 2024-09-11T09:50:51.724 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.724 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.724 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:50:51.724 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452092+0000", 2024-09-11T09:50:51.724 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.8400000000000002e-06 2024-09-11T09:50:51.724 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.724 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.724 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:50:51.724 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452095+0000", 2024-09-11T09:50:51.724 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.0740000000000001e-06 2024-09-11T09:50:51.724 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.725 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.725 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:50:51.725 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.454796+0000", 2024-09-11T09:50:51.725 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0027009740000000001 2024-09-11T09:50:51.725 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.725 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.725 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:50:51.725 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.454869+0000", 2024-09-11T09:50:51.725 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.3678999999999997e-05 2024-09-11T09:50:51.725 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.725 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.725 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:50:51.725 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.454992+0000", 2024-09-11T09:50:51.725 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00012279000000000001 2024-09-11T09:50:51.726 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.726 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.726 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:50:51.726 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.456072+0000", 2024-09-11T09:50:51.726 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001080229 2024-09-11T09:50:51.726 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.726 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.726 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:50:51.726 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.458931+0000", 2024-09-11T09:50:51.726 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0028591789999999999 2024-09-11T09:50:51.726 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.726 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.726 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:50:51.727 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.458972+0000", 2024-09-11T09:50:51.727 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.0170999999999999e-05 2024-09-11T09:50:51.727 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.727 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.727 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:50:51.727 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.459009+0000", 2024-09-11T09:50:51.727 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.7049999999999999e-05 2024-09-11T09:50:51.727 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:51.727 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:50:51.727 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:51.727 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.727 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.727 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:159 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:50:51.728 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:12.452099+0000", 2024-09-11T09:50:51.728 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 39.151438558000002, 2024-09-11T09:50:51.728 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0069885950000000002, 2024-09-11T09:50:51.728 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:50:51.728 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:50:51.728 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:51.728 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:50:51.728 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:50:51.728 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:50:51.728 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 159 2024-09-11T09:50:51.728 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.728 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:50:51.729 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.729 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:50:51.729 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452099+0000", 2024-09-11T09:50:51.729 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:51.729 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.729 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.729 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:50:51.729 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452099+0000", 2024-09-11T09:50:51.729 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:51.729 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.729 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.729 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:50:51.729 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452100+0000", 2024-09-11T09:50:51.730 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.2800000000000003e-07 2024-09-11T09:50:51.730 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.730 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.730 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:50:51.730 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452102+0000", 2024-09-11T09:50:51.730 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.824e-06 2024-09-11T09:50:51.730 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.730 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.730 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:50:51.730 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452105+0000", 2024-09-11T09:50:51.730 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.0460000000000001e-06 2024-09-11T09:50:51.730 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.730 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.731 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:50:51.731 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452107+0000", 2024-09-11T09:50:51.731 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.543e-06 2024-09-11T09:50:51.731 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.731 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.731 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:50:51.731 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.455484+0000", 2024-09-11T09:50:51.731 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0033769759999999998 2024-09-11T09:50:51.731 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.731 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.731 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:50:51.731 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.455527+0000", 2024-09-11T09:50:51.731 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.2589999999999997e-05 2024-09-11T09:50:51.731 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.732 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.732 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:50:51.732 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.455594+0000", 2024-09-11T09:50:51.732 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.6775999999999999e-05 2024-09-11T09:50:51.732 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.732 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.732 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:50:51.732 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.456312+0000", 2024-09-11T09:50:51.732 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00071874700000000005 2024-09-11T09:50:51.732 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.732 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.732 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:50:51.732 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.459015+0000", 2024-09-11T09:50:51.732 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0027028540000000002 2024-09-11T09:50:51.733 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.733 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.733 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:50:51.733 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.459054+0000", 2024-09-11T09:50:51.733 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.8840000000000001e-05 2024-09-11T09:50:51.733 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:51.733 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:51.733 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:50:51.733 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.459088+0000", 2024-09-11T09:50:51.733 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.4071e-05 2024-09-11T09:50:51.733 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:51.733 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:50:51.733 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:51.733 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:51.734 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:50:51.734 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-11T09:50:51.736 INFO:teuthology.orchestra.run.smithi017.stderr:noscrub is set 2024-09-11T09:50:51.736 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-11T09:50:51.736 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-09-11T09:50:51.736+0000 7f7b6c7d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31904) UID: 0 2024-09-11T09:50:51.836 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:50:51.836+0000 7fdec5849640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31907) UID: 0 2024-09-11T09:50:51.879 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-11T09:50:51.879 INFO:teuthology.orchestra.run.smithi096.stdout: "ops": [], 2024-09-11T09:50:51.879 INFO:teuthology.orchestra.run.smithi096.stdout: "num_ops": 0 2024-09-11T09:50:51.879 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-11T09:50:51.888 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-11T09:50:51.972 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:50:51.973+0000 7f3c96ec9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31909) UID: 0 2024-09-11T09:50:52.031 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-11T09:50:52.031 INFO:teuthology.orchestra.run.smithi096.stdout: "ops": [], 2024-09-11T09:50:52.031 INFO:teuthology.orchestra.run.smithi096.stdout: "complaint_time": 30, 2024-09-11T09:50:52.031 INFO:teuthology.orchestra.run.smithi096.stdout: "num_blocked_ops": 0 2024-09-11T09:50:52.031 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-11T09:50:52.037 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:50:52.038+0000 7f3c96ec9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31909) UID: 0 2024-09-11T09:50:52.039 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-11T09:50:52.138 INFO:tasks.ceph.osd.2.smithi017.stderr:2024-09-11T09:50:52.139+0000 7fd20b170640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31672) UID: 0 2024-09-11T09:50:52.182 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-11T09:50:52.182 INFO:teuthology.orchestra.run.smithi096.stdout: "size": 20, 2024-09-11T09:50:52.182 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 600, 2024-09-11T09:50:52.183 INFO:teuthology.orchestra.run.smithi096.stdout: "ops": [] 2024-09-11T09:50:52.183 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-11T09:50:52.191 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-11T09:50:52.239 INFO:tasks.ceph.osd.2.smithi017.stderr:2024-09-11T09:50:52.240+0000 7fd20b170640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31672) UID: 0 2024-09-11T09:50:52.294 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-11T09:50:52.294 INFO:teuthology.orchestra.run.smithi096.stdout: "ops": [], 2024-09-11T09:50:52.294 INFO:teuthology.orchestra.run.smithi096.stdout: "num_ops": 0 2024-09-11T09:50:52.294 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-11T09:50:52.303 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-11T09:50:52.339 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:50:52.340+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:50:52.405 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-11T09:50:52.405 INFO:teuthology.orchestra.run.smithi096.stdout: "ops": [], 2024-09-11T09:50:52.405 INFO:teuthology.orchestra.run.smithi096.stdout: "complaint_time": 30, 2024-09-11T09:50:52.405 INFO:teuthology.orchestra.run.smithi096.stdout: "num_blocked_ops": 0 2024-09-11T09:50:52.405 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-11T09:50:52.413 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-11T09:50:52.440 INFO:tasks.ceph.osd.0.smithi017.stderr:2024-09-11T09:50:52.441+0000 7fb73c0f1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31665) UID: 0 2024-09-11T09:50:52.515 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-11T09:50:52.515 INFO:teuthology.orchestra.run.smithi096.stdout: "size": 20, 2024-09-11T09:50:52.515 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 600, 2024-09-11T09:50:52.515 INFO:teuthology.orchestra.run.smithi096.stdout: "ops": [ 2024-09-11T09:50:52.515 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:52.516 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4230.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-11T09:50:52.516 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:18.490736+0000", 2024-09-11T09:50:52.516 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 34.025367967999998, 2024-09-11T09:50:52.516 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0059097569999999999, 2024-09-11T09:50:52.516 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:50:52.516 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:50:52.516 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:52.516 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:50:52.516 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4230", 2024-09-11T09:50:52.516 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.17:0/4101109576", 2024-09-11T09:50:52.516 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 1 2024-09-11T09:50:52.516 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:52.516 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:50:52.517 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:52.517 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:50:52.517 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.490736+0000", 2024-09-11T09:50:52.517 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:52.517 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:52.517 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:52.517 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:50:52.517 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.490736+0000", 2024-09-11T09:50:52.517 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:52.517 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:52.517 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:52.517 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:50:52.518 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.490742+0000", 2024-09-11T09:50:52.518 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.48e-06 2024-09-11T09:50:52.518 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:52.518 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:52.518 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:50:52.518 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.490749+0000", 2024-09-11T09:50:52.518 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.4569999999999999e-06 2024-09-11T09:50:52.518 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:52.518 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:52.518 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:50:52.518 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.490762+0000", 2024-09-11T09:50:52.518 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.2721000000000001e-05 2024-09-11T09:50:52.518 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:52.519 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:52.519 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:50:52.519 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.490774+0000", 2024-09-11T09:50:52.519 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1823e-05 2024-09-11T09:50:52.519 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:52.519 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:52.519 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:50:52.519 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.490846+0000", 2024-09-11T09:50:52.519 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.1878000000000005e-05 2024-09-11T09:50:52.519 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:52.519 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:52.519 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for peered", 2024-09-11T09:50:52.519 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.490857+0000", 2024-09-11T09:50:52.520 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1017e-05 2024-09-11T09:50:52.520 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:52.520 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:52.520 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:50:52.520 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.493856+0000", 2024-09-11T09:50:52.520 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00299885 2024-09-11T09:50:52.520 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:52.520 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:52.520 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:50:52.520 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.494020+0000", 2024-09-11T09:50:52.520 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00016436599999999999 2024-09-11T09:50:52.520 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:52.520 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:52.521 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 2", 2024-09-11T09:50:52.521 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.494186+0000", 2024-09-11T09:50:52.521 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00016562799999999999 2024-09-11T09:50:52.521 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:52.521 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:52.521 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:50:52.521 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.495407+0000", 2024-09-11T09:50:52.521 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0012217420000000001 2024-09-11T09:50:52.521 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:52.521 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:52.521 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:50:52.521 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.496579+0000", 2024-09-11T09:50:52.522 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001171844 2024-09-11T09:50:52.522 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:52.522 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:52.522 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:50:52.522 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.496620+0000", 2024-09-11T09:50:52.522 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.0479000000000003e-05 2024-09-11T09:50:52.522 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:52.522 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:52.522 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:50:52.522 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.496646+0000", 2024-09-11T09:50:52.522 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.6472e-05 2024-09-11T09:50:52.522 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:52.523 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:50:52.523 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:52.523 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:52.523 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:52.523 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_repop_reply(client.4230.0:1 2.3 e16/15 ondisk, result = 0)", 2024-09-11T09:50:52.523 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:18.495321+0000", 2024-09-11T09:50:52.523 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 34.020783303000002, 2024-09-11T09:50:52.523 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.4965999999999998e-05, 2024-09-11T09:50:52.523 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:50:52.523 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:50:52.523 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "started", 2024-09-11T09:50:52.523 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:50:52.523 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:52.524 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:50:52.524 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.495321+0000", 2024-09-11T09:50:52.524 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:52.524 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:52.524 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:52.524 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:50:52.524 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.495321+0000", 2024-09-11T09:50:52.524 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:52.524 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:52.524 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:52.524 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:50:52.524 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.495322+0000", 2024-09-11T09:50:52.525 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.7700000000000004e-07 2024-09-11T09:50:52.525 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:52.525 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:52.525 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:50:52.525 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.495325+0000", 2024-09-11T09:50:52.525 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.7780000000000001e-06 2024-09-11T09:50:52.525 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:52.525 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:52.525 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:50:52.525 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.495338+0000", 2024-09-11T09:50:52.525 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.2315e-05 2024-09-11T09:50:52.525 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:52.525 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:52.526 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:50:52.526 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.495345+0000", 2024-09-11T09:50:52.526 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.8889999999999996e-06 2024-09-11T09:50:52.526 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:52.526 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:52.526 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:50:52.526 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.495392+0000", 2024-09-11T09:50:52.526 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.7524000000000001e-05 2024-09-11T09:50:52.526 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:52.526 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:52.526 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:50:52.526 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.495402+0000", 2024-09-11T09:50:52.526 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.5840000000000001e-06 2024-09-11T09:50:52.526 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:52.527 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:52.527 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:50:52.527 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.495416+0000", 2024-09-11T09:50:52.527 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.4199000000000001e-05 2024-09-11T09:50:52.527 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:52.527 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:50:52.527 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:52.527 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:52.527 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:52.527 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4230.0:2 2.2 2:5cea7035:::rbd_info:head [call rbd.metadata_list in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e16)", 2024-09-11T09:50:52.527 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:18.499576+0000", 2024-09-11T09:50:52.527 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 34.016528796000003, 2024-09-11T09:50:52.527 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00018163000000000001, 2024-09-11T09:50:52.528 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:50:52.528 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:50:52.528 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "reached pg", 2024-09-11T09:50:52.528 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:50:52.528 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4230", 2024-09-11T09:50:52.528 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.17:0/4101109576", 2024-09-11T09:50:52.528 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 2 2024-09-11T09:50:52.528 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:52.528 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:50:52.528 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:52.528 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:50:52.528 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.499576+0000", 2024-09-11T09:50:52.528 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:52.529 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:52.529 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:52.529 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:50:52.529 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.499576+0000", 2024-09-11T09:50:52.529 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:52.529 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:52.529 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:52.529 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:50:52.529 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.499577+0000", 2024-09-11T09:50:52.529 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.124e-06 2024-09-11T09:50:52.530 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:52.530 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:52.530 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:50:52.530 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.499584+0000", 2024-09-11T09:50:52.530 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.1949999999999997e-06 2024-09-11T09:50:52.530 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:52.530 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:52.530 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:50:52.530 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.499597+0000", 2024-09-11T09:50:52.530 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.366e-05 2024-09-11T09:50:52.530 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:52.530 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:52.530 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:50:52.531 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.499604+0000", 2024-09-11T09:50:52.531 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.2709999999999998e-06 2024-09-11T09:50:52.531 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:52.531 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:52.531 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:50:52.531 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.499660+0000", 2024-09-11T09:50:52.531 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.6270999999999997e-05 2024-09-11T09:50:52.531 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:52.531 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:52.531 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:50:52.531 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.499757+0000", 2024-09-11T09:50:52.531 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.7108999999999995e-05 2024-09-11T09:50:52.532 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:52.532 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:50:52.532 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:52.532 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:52.532 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:52.532 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4230.0:3 2.2 2:5cea7035:::rbd_info:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e16)", 2024-09-11T09:50:52.532 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:18.499937+0000", 2024-09-11T09:50:52.532 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 34.016167007, 2024-09-11T09:50:52.532 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000143318, 2024-09-11T09:50:52.532 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:50:52.532 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:50:52.532 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "reached pg", 2024-09-11T09:50:52.532 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:50:52.533 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4230", 2024-09-11T09:50:52.533 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.17:0/4101109576", 2024-09-11T09:50:52.533 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 3 2024-09-11T09:50:52.533 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:52.533 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:50:52.533 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:52.533 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:50:52.533 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.499937+0000", 2024-09-11T09:50:52.533 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:52.533 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:52.533 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:52.533 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:50:52.533 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.499937+0000", 2024-09-11T09:50:52.534 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:52.534 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:52.534 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:52.534 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:50:52.534 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.499938+0000", 2024-09-11T09:50:52.534 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.195e-06 2024-09-11T09:50:52.534 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:52.534 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:52.534 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:50:52.534 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.499943+0000", 2024-09-11T09:50:52.534 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.9080000000000003e-06 2024-09-11T09:50:52.534 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:52.534 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:52.535 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:50:52.535 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.499956+0000", 2024-09-11T09:50:52.535 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.2728000000000001e-05 2024-09-11T09:50:52.535 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:52.535 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:52.535 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:50:52.535 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.499964+0000", 2024-09-11T09:50:52.535 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.8220000000000007e-06 2024-09-11T09:50:52.536 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:52.536 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:52.536 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:50:52.536 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.500017+0000", 2024-09-11T09:50:52.536 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.2809000000000002e-05 2024-09-11T09:50:52.536 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:52.536 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:52.536 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:50:52.536 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.500081+0000", 2024-09-11T09:50:52.536 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.3856000000000007e-05 2024-09-11T09:50:52.536 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:52.536 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:50:52.537 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:52.537 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:52.537 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:52.537 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4230.0:5 2.2 2:5cea7035:::rbd_info:head [create,write 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-11T09:50:52.537 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:19.492662+0000", 2024-09-11T09:50:52.537 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 33.023442347, 2024-09-11T09:50:52.537 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0043824759999999997, 2024-09-11T09:50:52.537 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:50:52.537 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:50:52.537 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:52.537 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:50:52.537 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4230", 2024-09-11T09:50:52.537 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.17:0/4101109576", 2024-09-11T09:50:52.537 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 5 2024-09-11T09:50:52.538 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:52.538 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:50:52.538 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:52.538 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:50:52.538 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.492662+0000", 2024-09-11T09:50:52.538 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:52.538 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:52.538 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:52.538 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:50:52.538 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.492662+0000", 2024-09-11T09:50:52.538 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:52.538 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:52.538 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:52.539 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:50:52.539 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.492664+0000", 2024-09-11T09:50:52.539 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.3599999999999999e-06 2024-09-11T09:50:52.539 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:52.539 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:52.539 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:50:52.539 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.492675+0000", 2024-09-11T09:50:52.539 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1182000000000001e-05 2024-09-11T09:50:52.539 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:52.539 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:52.539 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:50:52.539 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.492693+0000", 2024-09-11T09:50:52.539 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.7031999999999999e-05 2024-09-11T09:50:52.540 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:52.540 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:52.540 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:50:52.540 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.492702+0000", 2024-09-11T09:50:52.540 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.5540000000000003e-06 2024-09-11T09:50:52.540 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:52.540 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:52.540 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:50:52.540 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.492767+0000", 2024-09-11T09:50:52.540 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.5411000000000001e-05 2024-09-11T09:50:52.540 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:52.540 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:52.541 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op must wait for map", 2024-09-11T09:50:52.541 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.492773+0000", 2024-09-11T09:50:52.541 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.1039999999999998e-06 2024-09-11T09:50:52.541 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:52.541 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:52.541 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:50:52.541 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.494354+0000", 2024-09-11T09:50:52.541 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0015814049999999999 2024-09-11T09:50:52.541 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:52.541 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:52.541 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:50:52.541 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.494476+0000", 2024-09-11T09:50:52.541 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000122089 2024-09-11T09:50:52.541 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:52.542 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:52.542 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 1", 2024-09-11T09:50:52.542 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.494577+0000", 2024-09-11T09:50:52.542 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000100536 2024-09-11T09:50:52.542 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:52.542 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:52.542 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:50:52.542 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.495770+0000", 2024-09-11T09:50:52.542 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0011932469999999999 2024-09-11T09:50:52.542 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:52.542 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:52.542 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:50:52.542 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.496966+0000", 2024-09-11T09:50:52.543 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001196003 2024-09-11T09:50:52.543 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:52.543 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:52.543 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:50:52.543 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.497011+0000", 2024-09-11T09:50:52.543 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.5229999999999999e-05 2024-09-11T09:50:52.543 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:52.543 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:52.543 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:50:52.543 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.497044+0000", 2024-09-11T09:50:52.543 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.3322999999999999e-05 2024-09-11T09:50:52.543 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:52.543 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:50:52.544 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:52.544 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:52.544 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:52.544 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_repop_reply(client.4230.0:5 2.2 e17/15 ondisk, result = 0)", 2024-09-11T09:50:52.544 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:19.495669+0000", 2024-09-11T09:50:52.544 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 33.020435558999999, 2024-09-11T09:50:52.544 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000106431, 2024-09-11T09:50:52.544 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:50:52.544 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:50:52.544 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "started", 2024-09-11T09:50:52.544 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:50:52.544 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:52.544 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:50:52.545 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.495669+0000", 2024-09-11T09:50:52.545 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:52.545 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:52.545 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:52.545 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:50:52.545 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.495669+0000", 2024-09-11T09:50:52.545 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:52.545 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:52.545 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:52.545 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:50:52.545 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.495670+0000", 2024-09-11T09:50:52.545 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.218e-06 2024-09-11T09:50:52.545 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:52.545 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:52.546 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:50:52.546 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.495676+0000", 2024-09-11T09:50:52.546 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.5119999999999997e-06 2024-09-11T09:50:52.546 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:52.546 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:52.546 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:50:52.546 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.495690+0000", 2024-09-11T09:50:52.546 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.3312000000000001e-05 2024-09-11T09:50:52.546 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:52.546 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:52.546 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:50:52.546 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.495706+0000", 2024-09-11T09:50:52.546 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.6560000000000001e-05 2024-09-11T09:50:52.547 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:52.547 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:52.547 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:50:52.547 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.495759+0000", 2024-09-11T09:50:52.547 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.2828000000000003e-05 2024-09-11T09:50:52.547 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:52.547 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:52.547 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:50:52.547 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.495765+0000", 2024-09-11T09:50:52.547 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.7869999999999998e-06 2024-09-11T09:50:52.547 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:52.547 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:52.548 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:50:52.548 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.495775+0000", 2024-09-11T09:50:52.548 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.0214000000000001e-05 2024-09-11T09:50:52.548 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:52.548 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:50:52.548 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:52.548 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:52.548 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:52.548 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4230.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-09-11T09:50:52.548 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:20.498634+0000", 2024-09-11T09:50:52.548 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 32.017469947000002, 2024-09-11T09:50:52.548 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0050648539999999997, 2024-09-11T09:50:52.549 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:50:52.549 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:50:52.549 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:52.549 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:50:52.549 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4230", 2024-09-11T09:50:52.549 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.17:0/4101109576", 2024-09-11T09:50:52.549 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 7 2024-09-11T09:50:52.549 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:52.549 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:50:52.549 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:52.549 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:50:52.549 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.498634+0000", 2024-09-11T09:50:52.549 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:52.550 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:52.550 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:52.550 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:50:52.550 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.498634+0000", 2024-09-11T09:50:52.550 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:52.550 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:52.550 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:52.550 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:50:52.550 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.498637+0000", 2024-09-11T09:50:52.550 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.1919999999999999e-06 2024-09-11T09:50:52.550 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:52.550 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:52.550 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:50:52.550 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.498648+0000", 2024-09-11T09:50:52.551 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1116e-05 2024-09-11T09:50:52.551 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:52.551 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:52.551 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:50:52.551 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.498666+0000", 2024-09-11T09:50:52.551 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.8411999999999999e-05 2024-09-11T09:50:52.551 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:52.551 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:52.551 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:50:52.551 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.498675+0000", 2024-09-11T09:50:52.551 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.4059999999999994e-06 2024-09-11T09:50:52.551 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:52.551 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:52.552 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:50:52.552 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.498745+0000", 2024-09-11T09:50:52.552 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.9764000000000005e-05 2024-09-11T09:50:52.552 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:52.552 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:52.552 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op must wait for map", 2024-09-11T09:50:52.552 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.498754+0000", 2024-09-11T09:50:52.552 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.5969999999999992e-06 2024-09-11T09:50:52.552 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:52.552 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:52.552 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:50:52.552 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.500385+0000", 2024-09-11T09:50:52.552 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0016308360000000001 2024-09-11T09:50:52.553 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:52.553 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:52.553 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:50:52.553 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.500473+0000", 2024-09-11T09:50:52.553 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.7966000000000004e-05 2024-09-11T09:50:52.553 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:52.553 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:52.553 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 1", 2024-09-11T09:50:52.553 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.500591+0000", 2024-09-11T09:50:52.553 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000117895 2024-09-11T09:50:52.553 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:52.553 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:52.553 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:50:52.554 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.501685+0000", 2024-09-11T09:50:52.554 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001094886 2024-09-11T09:50:52.554 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:52.554 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:52.554 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:50:52.554 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.503627+0000", 2024-09-11T09:50:52.554 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001941199 2024-09-11T09:50:52.554 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:52.554 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:52.554 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:50:52.554 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.503667+0000", 2024-09-11T09:50:52.554 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.0645000000000001e-05 2024-09-11T09:50:52.554 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:52.554 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:52.555 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:50:52.555 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.503699+0000", 2024-09-11T09:50:52.555 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.1940000000000003e-05 2024-09-11T09:50:52.555 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:52.555 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:50:52.555 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:52.555 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:52.555 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:52.555 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_repop_reply(client.4230.0:7 2.2 e18/15 ondisk, result = 0)", 2024-09-11T09:50:52.555 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:20.501608+0000", 2024-09-11T09:50:52.555 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 32.014496424999997, 2024-09-11T09:50:52.555 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.2459000000000005e-05, 2024-09-11T09:50:52.555 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:50:52.556 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:50:52.556 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "started", 2024-09-11T09:50:52.556 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:50:52.556 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:52.556 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:50:52.556 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.501608+0000", 2024-09-11T09:50:52.556 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:52.556 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:52.556 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:52.556 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:50:52.556 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.501608+0000", 2024-09-11T09:50:52.556 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:52.556 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:52.557 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:52.557 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:50:52.557 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.501609+0000", 2024-09-11T09:50:52.557 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.0160000000000001e-06 2024-09-11T09:50:52.557 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:52.557 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:52.557 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:50:52.557 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.501613+0000", 2024-09-11T09:50:52.557 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.5959999999999997e-06 2024-09-11T09:50:52.557 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:52.557 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:52.557 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:50:52.557 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.501625+0000", 2024-09-11T09:50:52.557 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1888e-05 2024-09-11T09:50:52.558 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:52.558 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:52.558 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:50:52.558 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.501634+0000", 2024-09-11T09:50:52.558 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.6139999999999992e-06 2024-09-11T09:50:52.558 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:52.558 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:52.558 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:50:52.558 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.501676+0000", 2024-09-11T09:50:52.558 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.1953999999999998e-05 2024-09-11T09:50:52.558 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:52.558 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:52.558 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:50:52.559 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.501682+0000", 2024-09-11T09:50:52.559 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.8000000000000004e-06 2024-09-11T09:50:52.559 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:52.559 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:52.559 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:50:52.559 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.501690+0000", 2024-09-11T09:50:52.559 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.5909999999999996e-06 2024-09-11T09:50:52.559 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:52.559 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:50:52.559 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:52.559 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:52.559 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:50:52.559 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-11T09:50:52.561 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-11T09:50:52.562 INFO:tasks.ceph.osd.2.smithi017.stderr:2024-09-11T09:50:52.562+0000 7fd20b170640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31672) UID: 0 2024-09-11T09:50:52.612 INFO:teuthology.orchestra.run.smithi017.stderr:set 2.7 pg_upmap_items mapping to [4->5,3->0] 2024-09-11T09:50:52.662 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:50:52.663+0000 7f3c96ec9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31909) UID: 0 2024-09-11T09:50:52.686 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:50:52.686 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [], 2024-09-11T09:50:52.686 INFO:teuthology.orchestra.run.smithi017.stdout: "num_ops": 0 2024-09-11T09:50:52.686 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:50:52.695 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-11T09:50:52.762 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:50:52.764+0000 7fdec5849640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31907) UID: 0 2024-09-11T09:50:52.803 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:50:52.803 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [], 2024-09-11T09:50:52.803 INFO:teuthology.orchestra.run.smithi017.stdout: "complaint_time": 30, 2024-09-11T09:50:52.803 INFO:teuthology.orchestra.run.smithi017.stdout: "num_blocked_ops": 0 2024-09-11T09:50:52.803 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:50:52.812 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-11T09:50:52.863 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-09-11T09:50:52.864+0000 7f7b6c7d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31904) UID: 0 2024-09-11T09:50:52.921 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:50:52.921 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 20, 2024-09-11T09:50:52.921 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 600, 2024-09-11T09:50:52.921 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [ 2024-09-11T09:50:52.922 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:52.922 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4183.0:21 1.0 e13/12 1:30306672:devicehealth::main.db.0000000000000000:head v 13'11, mlcod=13'8)", 2024-09-11T09:50:52.922 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:11.491728+0000", 2024-09-11T09:50:52.922 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 41.429697355999998, 2024-09-11T09:50:52.922 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00093269800000000003, 2024-09-11T09:50:52.922 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:50:52.922 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:50:52.922 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:52.922 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:50:52.922 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:52.922 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:50:52.922 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.491728+0000", 2024-09-11T09:50:52.922 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:52.923 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:52.923 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:52.923 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:50:52.923 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.491728+0000", 2024-09-11T09:50:52.923 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:52.923 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:52.923 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:52.923 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:50:52.923 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.491729+0000", 2024-09-11T09:50:52.923 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.8599999999999997e-07 2024-09-11T09:50:52.923 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:52.923 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:52.923 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:50:52.923 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.491820+0000", 2024-09-11T09:50:52.924 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.0812999999999999e-05 2024-09-11T09:50:52.924 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:52.924 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:52.924 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:50:52.924 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.491833+0000", 2024-09-11T09:50:52.924 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.341e-05 2024-09-11T09:50:52.924 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:52.924 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:52.924 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:50:52.924 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.491841+0000", 2024-09-11T09:50:52.924 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.8730000000000007e-06 2024-09-11T09:50:52.924 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:52.924 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:52.925 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:50:52.925 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.491916+0000", 2024-09-11T09:50:52.925 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.4881000000000004e-05 2024-09-11T09:50:52.925 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:52.925 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:52.925 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:50:52.925 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.491931+0000", 2024-09-11T09:50:52.925 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.4825e-05 2024-09-11T09:50:52.925 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:52.925 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:52.925 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:50:52.925 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.492633+0000", 2024-09-11T09:50:52.925 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00070191600000000002 2024-09-11T09:50:52.925 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:52.926 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:52.926 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:50:52.926 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.492661+0000", 2024-09-11T09:50:52.926 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.8093999999999999e-05 2024-09-11T09:50:52.926 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:52.926 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:50:52.926 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:52.926 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:52.926 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:52.926 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4183.0:31 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'17, mlcod=13'14)", 2024-09-11T09:50:52.926 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:11.499319+0000", 2024-09-11T09:50:52.926 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 41.422106687000003, 2024-09-11T09:50:52.926 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0010073879999999999, 2024-09-11T09:50:52.927 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:50:52.927 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:50:52.927 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:52.927 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:50:52.927 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:52.927 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:50:52.927 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.499319+0000", 2024-09-11T09:50:52.927 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:52.927 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:52.927 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:52.927 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:50:52.927 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.499319+0000", 2024-09-11T09:50:52.927 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:52.928 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:52.928 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:52.928 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:50:52.928 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.499320+0000", 2024-09-11T09:50:52.928 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.7700000000000004e-07 2024-09-11T09:50:52.928 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:52.928 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:52.928 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:50:52.928 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.499326+0000", 2024-09-11T09:50:52.928 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.6420000000000001e-06 2024-09-11T09:50:52.928 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:52.928 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:52.928 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:50:52.929 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.499339+0000", 2024-09-11T09:50:52.929 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2537999999999999e-05 2024-09-11T09:50:52.929 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:52.929 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:52.929 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:50:52.929 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.499345+0000", 2024-09-11T09:50:52.929 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.8220000000000003e-06 2024-09-11T09:50:52.929 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:52.929 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:52.929 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:50:52.929 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.499403+0000", 2024-09-11T09:50:52.929 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.8814000000000003e-05 2024-09-11T09:50:52.929 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:52.929 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:52.930 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:50:52.930 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.499427+0000", 2024-09-11T09:50:52.930 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.3853999999999999e-05 2024-09-11T09:50:52.930 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:52.930 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:52.930 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:50:52.930 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.500294+0000", 2024-09-11T09:50:52.930 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00086622399999999997 2024-09-11T09:50:52.930 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:52.930 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:52.930 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:50:52.930 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.500326+0000", 2024-09-11T09:50:52.930 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.2716999999999999e-05 2024-09-11T09:50:52.930 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:52.931 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:50:52.931 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:52.931 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:52.931 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:52.931 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4183.0:32 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'18, mlcod=13'14)", 2024-09-11T09:50:52.931 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:11.499796+0000", 2024-09-11T09:50:52.931 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 41.421629480999997, 2024-09-11T09:50:52.931 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00092915100000000002, 2024-09-11T09:50:52.931 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:50:52.931 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:50:52.931 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:52.931 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:50:52.931 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:52.931 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:50:52.932 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.499796+0000", 2024-09-11T09:50:52.932 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:52.932 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:52.932 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:52.932 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:50:52.932 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.499796+0000", 2024-09-11T09:50:52.932 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:52.932 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:52.932 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:52.932 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:50:52.932 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.499797+0000", 2024-09-11T09:50:52.932 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.9400000000000004e-07 2024-09-11T09:50:52.932 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:52.933 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:52.933 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:50:52.933 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.499841+0000", 2024-09-11T09:50:52.933 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.3900000000000003e-05 2024-09-11T09:50:52.933 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:52.933 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:52.933 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:50:52.933 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.499852+0000", 2024-09-11T09:50:52.933 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1462e-05 2024-09-11T09:50:52.933 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:52.933 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:52.933 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:50:52.933 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.499858+0000", 2024-09-11T09:50:52.933 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.8320000000000002e-06 2024-09-11T09:50:52.934 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:52.934 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:52.934 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:50:52.934 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.499915+0000", 2024-09-11T09:50:52.934 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.7187e-05 2024-09-11T09:50:52.934 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:52.934 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:52.934 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:50:52.934 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.499930+0000", 2024-09-11T09:50:52.934 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.4928e-05 2024-09-11T09:50:52.934 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:52.934 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:52.934 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:50:52.934 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.500697+0000", 2024-09-11T09:50:52.935 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00076658399999999995 2024-09-11T09:50:52.935 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:52.935 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:52.935 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:50:52.935 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.500725+0000", 2024-09-11T09:50:52.935 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.8364000000000002e-05 2024-09-11T09:50:52.935 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:52.935 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:50:52.935 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:52.935 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:52.935 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:52.935 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4183.0:34 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'20, mlcod=13'15)", 2024-09-11T09:50:52.935 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:11.501016+0000", 2024-09-11T09:50:52.935 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 41.420409962999997, 2024-09-11T09:50:52.936 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00097815399999999996, 2024-09-11T09:50:52.936 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:50:52.936 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:50:52.936 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:52.936 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:50:52.936 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:52.936 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:50:52.936 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.501016+0000", 2024-09-11T09:50:52.936 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:52.936 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:52.936 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:52.937 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:50:52.937 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.501016+0000", 2024-09-11T09:50:52.937 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:52.937 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:52.937 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:52.937 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:50:52.937 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.501016+0000", 2024-09-11T09:50:52.937 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.3799999999999997e-07 2024-09-11T09:50:52.937 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:52.937 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:52.937 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:50:52.937 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.501023+0000", 2024-09-11T09:50:52.937 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.2180000000000002e-06 2024-09-11T09:50:52.937 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:52.938 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:52.938 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:50:52.938 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.501031+0000", 2024-09-11T09:50:52.938 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.5379999999999997e-06 2024-09-11T09:50:52.938 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:52.938 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:52.938 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:50:52.938 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.501035+0000", 2024-09-11T09:50:52.938 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.216e-06 2024-09-11T09:50:52.938 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:52.938 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:52.938 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:50:52.938 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.501105+0000", 2024-09-11T09:50:52.938 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.9757000000000003e-05 2024-09-11T09:50:52.939 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:52.939 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:52.939 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:50:52.939 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.501120+0000", 2024-09-11T09:50:52.939 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.5358000000000001e-05 2024-09-11T09:50:52.939 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:52.939 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:52.939 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:50:52.939 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.501955+0000", 2024-09-11T09:50:52.939 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00083505000000000005 2024-09-11T09:50:52.939 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:52.939 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:52.939 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:50:52.940 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.501994+0000", 2024-09-11T09:50:52.940 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.8479000000000002e-05 2024-09-11T09:50:52.940 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:52.940 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:50:52.940 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:52.940 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:52.940 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:52.940 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4183.0:38 1.0 e13/12 1:30306672:devicehealth::main.db.0000000000000000:head v 13'24, mlcod=13'22)", 2024-09-11T09:50:52.940 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:11.503891+0000", 2024-09-11T09:50:52.940 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 41.417534521, 2024-09-11T09:50:52.940 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000959351, 2024-09-11T09:50:52.940 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:50:52.940 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:50:52.941 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:52.941 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:50:52.941 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:52.941 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:50:52.941 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.503891+0000", 2024-09-11T09:50:52.941 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:52.941 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:52.941 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:52.941 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:50:52.941 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.503891+0000", 2024-09-11T09:50:52.941 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:52.941 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:52.941 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:52.942 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:50:52.942 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.503892+0000", 2024-09-11T09:50:52.942 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.5499999999999998e-07 2024-09-11T09:50:52.942 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:52.942 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:52.942 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:50:52.942 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.503967+0000", 2024-09-11T09:50:52.942 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.4993999999999994e-05 2024-09-11T09:50:52.942 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:52.942 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:52.942 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:50:52.942 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.503976+0000", 2024-09-11T09:50:52.942 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.4709999999999996e-06 2024-09-11T09:50:52.942 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:52.943 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:52.943 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:50:52.943 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.503981+0000", 2024-09-11T09:50:52.943 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.6789999999999996e-06 2024-09-11T09:50:52.943 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:52.943 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:52.943 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:50:52.943 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.504021+0000", 2024-09-11T09:50:52.943 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.0065999999999999e-05 2024-09-11T09:50:52.943 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:52.943 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:52.943 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:50:52.943 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.504036+0000", 2024-09-11T09:50:52.943 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.468e-05 2024-09-11T09:50:52.944 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:52.944 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:52.944 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:50:52.944 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.504813+0000", 2024-09-11T09:50:52.944 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00077718299999999998 2024-09-11T09:50:52.944 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:52.944 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:52.944 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:50:52.944 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.504850+0000", 2024-09-11T09:50:52.944 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.7722999999999998e-05 2024-09-11T09:50:52.944 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:52.945 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:50:52.945 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:52.945 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:52.945 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:52.945 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4183.0:54 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'32, mlcod=13'29)", 2024-09-11T09:50:52.945 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:11.515404+0000", 2024-09-11T09:50:52.945 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 41.406021916, 2024-09-11T09:50:52.945 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00089471799999999997, 2024-09-11T09:50:52.945 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:50:52.945 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:50:52.945 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:52.945 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:50:52.945 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:52.946 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:50:52.946 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.515404+0000", 2024-09-11T09:50:52.946 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:52.946 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:52.946 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:52.946 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:50:52.946 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.515404+0000", 2024-09-11T09:50:52.946 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:52.946 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:52.946 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:52.946 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:50:52.946 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.515404+0000", 2024-09-11T09:50:52.946 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.3799999999999996e-07 2024-09-11T09:50:52.946 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:52.947 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:52.947 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:50:52.947 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.515410+0000", 2024-09-11T09:50:52.947 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.1789999999999999e-06 2024-09-11T09:50:52.947 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:52.947 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:52.947 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:50:52.947 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.515422+0000", 2024-09-11T09:50:52.947 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1929999999999999e-05 2024-09-11T09:50:52.947 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:52.947 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:52.947 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:50:52.947 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.515427+0000", 2024-09-11T09:50:52.947 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.6060000000000002e-06 2024-09-11T09:50:52.948 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:52.948 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:52.948 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:50:52.948 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.515482+0000", 2024-09-11T09:50:52.948 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.4743000000000001e-05 2024-09-11T09:50:52.948 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:52.948 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:52.948 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:50:52.948 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.515498+0000", 2024-09-11T09:50:52.948 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.5778999999999999e-05 2024-09-11T09:50:52.949 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:52.949 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:52.949 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:50:52.949 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.516271+0000", 2024-09-11T09:50:52.949 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00077312599999999996 2024-09-11T09:50:52.949 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:52.949 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:52.949 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:50:52.949 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.516298+0000", 2024-09-11T09:50:52.949 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.7617e-05 2024-09-11T09:50:52.949 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:52.950 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:50:52.950 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:52.950 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:52.950 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:52.950 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4183.0:57 1.0 e13/12 1:30306672:devicehealth::main.db.0000000000000000:head v 13'35, mlcod=13'33)", 2024-09-11T09:50:52.950 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:11.519823+0000", 2024-09-11T09:50:52.950 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 41.401602859, 2024-09-11T09:50:52.950 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00090876100000000001, 2024-09-11T09:50:52.950 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:50:52.950 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:50:52.950 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:52.950 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:50:52.951 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:52.951 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:50:52.951 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.519823+0000", 2024-09-11T09:50:52.951 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:52.951 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:52.951 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:52.951 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:50:52.951 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.519823+0000", 2024-09-11T09:50:52.951 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:52.951 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:52.951 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:52.951 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:50:52.951 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.519823+0000", 2024-09-11T09:50:52.952 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.7000000000000004e-07 2024-09-11T09:50:52.952 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:52.952 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:52.952 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:50:52.952 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.519859+0000", 2024-09-11T09:50:52.952 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.5508e-05 2024-09-11T09:50:52.952 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:52.952 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:52.952 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:50:52.952 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.519871+0000", 2024-09-11T09:50:52.952 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.221e-05 2024-09-11T09:50:52.952 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:52.952 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:52.952 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:50:52.953 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.519877+0000", 2024-09-11T09:50:52.953 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.0410000000000003e-06 2024-09-11T09:50:52.953 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:52.953 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:52.953 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:50:52.953 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.519955+0000", 2024-09-11T09:50:52.953 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.8169000000000003e-05 2024-09-11T09:50:52.953 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:52.953 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:52.953 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:50:52.953 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.519972+0000", 2024-09-11T09:50:52.953 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6595999999999998e-05 2024-09-11T09:50:52.953 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:52.954 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:52.954 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:50:52.954 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.520691+0000", 2024-09-11T09:50:52.954 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000719405 2024-09-11T09:50:52.954 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:52.954 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:52.954 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:50:52.954 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.520731+0000", 2024-09-11T09:50:52.954 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.0061999999999997e-05 2024-09-11T09:50:52.954 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:52.954 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:50:52.954 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:52.954 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:52.954 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:52.955 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4183.0:58 1.0 e13/12 1:30306672:devicehealth::main.db.0000000000000000:head v 13'36, mlcod=13'33)", 2024-09-11T09:50:52.955 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:11.520285+0000", 2024-09-11T09:50:52.955 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 41.401140451000003, 2024-09-11T09:50:52.955 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00091014799999999995, 2024-09-11T09:50:52.955 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:50:52.955 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:50:52.955 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:52.955 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:50:52.955 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:52.955 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:50:52.955 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.520285+0000", 2024-09-11T09:50:52.955 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:52.955 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:52.955 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:52.956 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:50:52.956 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.520285+0000", 2024-09-11T09:50:52.956 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:52.956 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:52.956 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:52.956 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:50:52.956 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.520286+0000", 2024-09-11T09:50:52.956 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.4399999999999999e-07 2024-09-11T09:50:52.956 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:52.956 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:52.956 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:50:52.956 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.520348+0000", 2024-09-11T09:50:52.956 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.1760000000000005e-05 2024-09-11T09:50:52.957 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:52.957 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:52.957 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:50:52.957 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.520359+0000", 2024-09-11T09:50:52.957 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1362999999999999e-05 2024-09-11T09:50:52.957 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:52.957 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:52.957 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:50:52.957 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.520364+0000", 2024-09-11T09:50:52.957 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.3600000000000004e-06 2024-09-11T09:50:52.957 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:52.957 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:52.957 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:50:52.958 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.520413+0000", 2024-09-11T09:50:52.958 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.8226000000000002e-05 2024-09-11T09:50:52.958 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:52.958 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:52.958 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:50:52.958 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.520426+0000", 2024-09-11T09:50:52.958 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3560000000000001e-05 2024-09-11T09:50:52.958 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:52.958 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:52.958 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:50:52.958 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.521159+0000", 2024-09-11T09:50:52.958 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00073255499999999997 2024-09-11T09:50:52.958 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:52.959 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:52.959 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:50:52.959 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.521195+0000", 2024-09-11T09:50:52.959 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.6480000000000003e-05 2024-09-11T09:50:52.959 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:52.959 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:50:52.959 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:52.959 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:52.959 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:52.959 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4183.0:68 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'42, mlcod=13'39)", 2024-09-11T09:50:52.959 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:11.534957+0000", 2024-09-11T09:50:52.959 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 41.38646876, 2024-09-11T09:50:52.959 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.001065351, 2024-09-11T09:50:52.960 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:50:52.960 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:50:52.960 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:52.960 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:50:52.960 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:52.960 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:50:52.960 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.534957+0000", 2024-09-11T09:50:52.960 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:52.960 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:52.960 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:52.960 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:50:52.960 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.534957+0000", 2024-09-11T09:50:52.960 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:52.961 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:52.961 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:52.961 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:50:52.961 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.534957+0000", 2024-09-11T09:50:52.961 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.3799999999999998e-07 2024-09-11T09:50:52.961 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:52.961 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:52.961 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:50:52.961 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.534964+0000", 2024-09-11T09:50:52.961 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.9809999999999997e-06 2024-09-11T09:50:52.961 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:52.961 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:52.961 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:50:52.962 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.534976+0000", 2024-09-11T09:50:52.962 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2119999999999999e-05 2024-09-11T09:50:52.962 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:52.962 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:52.962 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:50:52.962 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.534980+0000", 2024-09-11T09:50:52.962 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.9890000000000003e-06 2024-09-11T09:50:52.962 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:52.962 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:52.962 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:50:52.962 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.535030+0000", 2024-09-11T09:50:52.962 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.9258000000000002e-05 2024-09-11T09:50:52.962 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:52.962 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:52.963 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:50:52.963 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.535044+0000", 2024-09-11T09:50:52.963 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3981e-05 2024-09-11T09:50:52.963 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:52.963 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:52.963 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:50:52.963 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.535995+0000", 2024-09-11T09:50:52.963 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000951804 2024-09-11T09:50:52.963 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:52.963 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:52.963 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:50:52.963 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.536022+0000", 2024-09-11T09:50:52.963 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.6780000000000001e-05 2024-09-11T09:50:52.964 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:52.964 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:50:52.964 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:52.964 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:52.964 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:52.964 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4183.0:69 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'43, mlcod=13'39)", 2024-09-11T09:50:52.964 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:11.535331+0000", 2024-09-11T09:50:52.964 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 41.386094436999997, 2024-09-11T09:50:52.964 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0011256090000000001, 2024-09-11T09:50:52.964 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:50:52.964 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:50:52.964 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:52.964 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:50:52.964 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:52.965 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:50:52.965 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.535331+0000", 2024-09-11T09:50:52.965 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:52.965 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:52.965 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:52.965 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:50:52.965 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.535331+0000", 2024-09-11T09:50:52.965 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:52.965 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:52.965 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:52.965 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:50:52.965 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.535332+0000", 2024-09-11T09:50:52.966 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.1600000000000001e-07 2024-09-11T09:50:52.966 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:52.966 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:52.966 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:50:52.966 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.535393+0000", 2024-09-11T09:50:52.966 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.1452999999999996e-05 2024-09-11T09:50:52.966 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:52.966 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:52.966 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:50:52.966 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.535402+0000", 2024-09-11T09:50:52.966 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.4300000000000006e-06 2024-09-11T09:50:52.966 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:52.966 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:52.967 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:50:52.967 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.535405+0000", 2024-09-11T09:50:52.967 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.512e-06 2024-09-11T09:50:52.967 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:52.967 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:52.967 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:50:52.967 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.535482+0000", 2024-09-11T09:50:52.967 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.7132999999999995e-05 2024-09-11T09:50:52.967 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:52.967 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:52.967 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:50:52.967 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.535506+0000", 2024-09-11T09:50:52.967 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.4070000000000002e-05 2024-09-11T09:50:52.967 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:52.968 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:52.968 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:50:52.968 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.536424+0000", 2024-09-11T09:50:52.968 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00091766700000000005 2024-09-11T09:50:52.968 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:52.968 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:52.968 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:50:52.968 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.536457+0000", 2024-09-11T09:50:52.968 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.2827999999999998e-05 2024-09-11T09:50:52.968 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:52.968 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:50:52.968 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:52.968 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:52.969 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:52.969 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4183.0:83 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'53, mlcod=13'50)", 2024-09-11T09:50:52.969 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:11.558836+0000", 2024-09-11T09:50:52.969 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 41.362589182000001, 2024-09-11T09:50:52.969 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00090313100000000005, 2024-09-11T09:50:52.969 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:50:52.969 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:50:52.969 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:52.969 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:50:52.969 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:52.969 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:50:52.969 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.558836+0000", 2024-09-11T09:50:52.969 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:52.970 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:52.970 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:52.970 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:50:52.970 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.558836+0000", 2024-09-11T09:50:52.970 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:52.970 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:52.970 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:52.970 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:50:52.970 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.558837+0000", 2024-09-11T09:50:52.970 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.4900000000000005e-07 2024-09-11T09:50:52.970 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:52.970 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:52.970 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:50:52.970 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.558857+0000", 2024-09-11T09:50:52.971 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.9502999999999999e-05 2024-09-11T09:50:52.971 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:52.971 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:52.971 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:50:52.971 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.558870+0000", 2024-09-11T09:50:52.971 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3020000000000001e-05 2024-09-11T09:50:52.971 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:52.971 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:52.971 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:50:52.971 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.558876+0000", 2024-09-11T09:50:52.971 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.7270000000000004e-06 2024-09-11T09:50:52.971 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:52.971 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:52.972 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:50:52.972 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.558934+0000", 2024-09-11T09:50:52.972 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.8100999999999998e-05 2024-09-11T09:50:52.972 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:52.972 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:52.972 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:50:52.972 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.558947+0000", 2024-09-11T09:50:52.972 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3269e-05 2024-09-11T09:50:52.972 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:52.972 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:52.972 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:50:52.972 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.559720+0000", 2024-09-11T09:50:52.972 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00077287600000000003 2024-09-11T09:50:52.972 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:52.973 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:52.973 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:50:52.973 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.559740+0000", 2024-09-11T09:50:52.973 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.9786e-05 2024-09-11T09:50:52.973 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:52.973 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:50:52.973 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:52.973 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:52.973 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:52.973 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4183.0:84 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'54, mlcod=13'50)", 2024-09-11T09:50:52.973 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:11.559267+0000", 2024-09-11T09:50:52.973 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 41.362158755000003, 2024-09-11T09:50:52.973 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00093990199999999995, 2024-09-11T09:50:52.974 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:50:52.974 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:50:52.974 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:52.974 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:50:52.974 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:52.974 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:50:52.974 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.559267+0000", 2024-09-11T09:50:52.974 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:52.974 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:52.974 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:52.974 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:50:52.974 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.559267+0000", 2024-09-11T09:50:52.974 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:52.974 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:52.975 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:52.975 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:50:52.975 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.559267+0000", 2024-09-11T09:50:52.975 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.2900000000000003e-07 2024-09-11T09:50:52.975 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:52.975 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:52.975 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:50:52.975 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.559322+0000", 2024-09-11T09:50:52.975 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.4429999999999999e-05 2024-09-11T09:50:52.975 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:52.975 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:52.975 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:50:52.975 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.559332+0000", 2024-09-11T09:50:52.975 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0473e-05 2024-09-11T09:50:52.976 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:52.976 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:52.976 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:50:52.976 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.559338+0000", 2024-09-11T09:50:52.976 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.0730000000000001e-06 2024-09-11T09:50:52.976 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:52.976 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:52.976 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:50:52.976 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.559373+0000", 2024-09-11T09:50:52.976 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.4626999999999999e-05 2024-09-11T09:50:52.976 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:52.976 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:52.976 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:50:52.977 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.559384+0000", 2024-09-11T09:50:52.977 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1307e-05 2024-09-11T09:50:52.977 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:52.977 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:52.977 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:50:52.977 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.560175+0000", 2024-09-11T09:50:52.977 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00079016399999999999 2024-09-11T09:50:52.977 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:52.977 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:52.977 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:50:52.977 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.560207+0000", 2024-09-11T09:50:52.977 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.2199000000000001e-05 2024-09-11T09:50:52.977 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:52.978 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:50:52.978 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:52.978 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:52.978 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:52.978 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4183.0:89 1.0 e13/12 1:30306672:devicehealth::main.db.0000000000000000:head v 13'59, mlcod=13'57)", 2024-09-11T09:50:52.978 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:11.563614+0000", 2024-09-11T09:50:52.978 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 41.357811134999999, 2024-09-11T09:50:52.978 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00091281399999999996, 2024-09-11T09:50:52.978 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:50:52.978 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:50:52.978 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:52.978 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:50:52.978 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:52.979 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:50:52.979 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.563614+0000", 2024-09-11T09:50:52.979 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:52.979 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:52.979 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:52.979 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:50:52.979 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.563614+0000", 2024-09-11T09:50:52.979 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:52.979 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:52.979 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:52.979 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:50:52.979 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.563615+0000", 2024-09-11T09:50:52.979 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.6899999999999998e-07 2024-09-11T09:50:52.979 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:52.980 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:52.980 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:50:52.980 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.563681+0000", 2024-09-11T09:50:52.980 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.5769e-05 2024-09-11T09:50:52.980 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:52.980 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:52.980 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:50:52.980 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.563690+0000", 2024-09-11T09:50:52.980 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.6800000000000005e-06 2024-09-11T09:50:52.980 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:52.980 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:52.980 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:50:52.980 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.563699+0000", 2024-09-11T09:50:52.981 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.3170000000000002e-06 2024-09-11T09:50:52.981 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:52.981 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:52.981 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:50:52.981 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.563734+0000", 2024-09-11T09:50:52.981 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.5023e-05 2024-09-11T09:50:52.981 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:52.981 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:52.981 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:50:52.981 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.563746+0000", 2024-09-11T09:50:52.981 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2661e-05 2024-09-11T09:50:52.981 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:52.981 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:52.981 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:50:52.982 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.564500+0000", 2024-09-11T09:50:52.982 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00075323100000000004 2024-09-11T09:50:52.982 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:52.982 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:52.982 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:50:52.982 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.564527+0000", 2024-09-11T09:50:52.982 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.7664000000000001e-05 2024-09-11T09:50:52.982 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:52.982 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:50:52.982 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:52.982 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:52.982 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:52.982 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4183.0:105 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'67, mlcod=13'64)", 2024-09-11T09:50:52.983 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:11.712146+0000", 2024-09-11T09:50:52.983 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 41.209279385000002, 2024-09-11T09:50:52.983 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0010916859999999999, 2024-09-11T09:50:52.983 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:50:52.983 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:50:52.983 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:52.983 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:50:52.983 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:52.983 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:50:52.983 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.712146+0000", 2024-09-11T09:50:52.983 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:52.983 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:52.983 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:52.984 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:50:52.984 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.712146+0000", 2024-09-11T09:50:52.984 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:52.984 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:52.984 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:52.984 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:50:52.984 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.712147+0000", 2024-09-11T09:50:52.984 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.8200000000000001e-07 2024-09-11T09:50:52.984 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:52.984 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:52.984 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:50:52.984 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.712155+0000", 2024-09-11T09:50:52.984 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.0309999999999998e-06 2024-09-11T09:50:52.984 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:52.985 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:52.985 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:50:52.985 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.712163+0000", 2024-09-11T09:50:52.985 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.5920000000000001e-06 2024-09-11T09:50:52.985 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:52.985 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:52.985 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:50:52.985 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.712169+0000", 2024-09-11T09:50:52.985 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.4299999999999997e-06 2024-09-11T09:50:52.985 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:52.985 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:52.985 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:50:52.985 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.712418+0000", 2024-09-11T09:50:52.986 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00024978499999999998 2024-09-11T09:50:52.986 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:52.986 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:52.986 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:50:52.986 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.712432+0000", 2024-09-11T09:50:52.986 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3190000000000001e-05 2024-09-11T09:50:52.986 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:52.986 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:52.986 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:50:52.986 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.713206+0000", 2024-09-11T09:50:52.986 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00077420199999999998 2024-09-11T09:50:52.986 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:52.986 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:52.986 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:50:52.987 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.713238+0000", 2024-09-11T09:50:52.987 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.2073999999999998e-05 2024-09-11T09:50:52.987 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:52.987 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:50:52.987 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:52.987 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:52.987 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:52.987 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4183.0:106 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'68, mlcod=13'64)", 2024-09-11T09:50:52.987 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:11.712624+0000", 2024-09-11T09:50:52.987 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 41.208802057, 2024-09-11T09:50:52.987 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0010009629999999999, 2024-09-11T09:50:52.987 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:50:52.987 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:50:52.987 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:52.988 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:50:52.988 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:52.988 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:50:52.988 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.712624+0000", 2024-09-11T09:50:52.988 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:52.988 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:52.988 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:52.988 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:50:52.988 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.712624+0000", 2024-09-11T09:50:52.988 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:52.988 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:52.988 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:52.988 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:50:52.988 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.712624+0000", 2024-09-11T09:50:52.989 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.4799999999999998e-07 2024-09-11T09:50:52.989 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:52.989 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:52.989 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:50:52.989 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.712689+0000", 2024-09-11T09:50:52.989 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.4488999999999999e-05 2024-09-11T09:50:52.989 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:52.989 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:52.989 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:50:52.989 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.712707+0000", 2024-09-11T09:50:52.989 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.8272000000000001e-05 2024-09-11T09:50:52.989 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:52.989 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:52.989 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:50:52.990 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.712713+0000", 2024-09-11T09:50:52.990 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.2040000000000002e-06 2024-09-11T09:50:52.990 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:52.990 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:52.990 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:50:52.990 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.712783+0000", 2024-09-11T09:50:52.990 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.9534000000000002e-05 2024-09-11T09:50:52.990 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:52.990 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:52.990 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:50:52.990 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.712811+0000", 2024-09-11T09:50:52.990 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.8620999999999999e-05 2024-09-11T09:50:52.990 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:52.990 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:52.991 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:50:52.991 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.713586+0000", 2024-09-11T09:50:52.991 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00077462600000000005 2024-09-11T09:50:52.991 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:52.991 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:52.991 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:50:52.991 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.713624+0000", 2024-09-11T09:50:52.991 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.8668999999999999e-05 2024-09-11T09:50:52.991 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:52.991 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:50:52.991 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:52.991 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:52.991 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:52.991 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4183.0:134 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'92, mlcod=13'86)", 2024-09-11T09:50:52.992 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:12.035698+0000", 2024-09-11T09:50:52.992 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 40.885727797999998, 2024-09-11T09:50:52.992 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00089550700000000003, 2024-09-11T09:50:52.992 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:50:52.992 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:50:52.992 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:52.992 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:50:52.992 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:52.992 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:50:52.992 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.035698+0000", 2024-09-11T09:50:52.992 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:52.992 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:52.992 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:52.992 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:50:52.993 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.035698+0000", 2024-09-11T09:50:52.993 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:52.993 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:52.993 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:52.993 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:50:52.993 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.035698+0000", 2024-09-11T09:50:52.993 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.0900000000000002e-07 2024-09-11T09:50:52.993 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:52.993 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:52.993 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:50:52.993 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.035758+0000", 2024-09-11T09:50:52.993 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.9481999999999997e-05 2024-09-11T09:50:52.993 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:52.994 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:52.994 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:50:52.994 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.035766+0000", 2024-09-11T09:50:52.994 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.3760000000000008e-06 2024-09-11T09:50:52.994 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:52.994 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:52.994 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:50:52.994 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.035772+0000", 2024-09-11T09:50:52.994 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.8930000000000002e-06 2024-09-11T09:50:52.994 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:52.994 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:52.994 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:50:52.994 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.035807+0000", 2024-09-11T09:50:52.995 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.4975999999999999e-05 2024-09-11T09:50:52.995 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:52.995 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:52.995 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:50:52.995 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.035817+0000", 2024-09-11T09:50:52.995 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0314e-05 2024-09-11T09:50:52.995 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:52.995 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:52.995 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:50:52.995 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.036542+0000", 2024-09-11T09:50:52.995 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00072434400000000003 2024-09-11T09:50:52.995 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:52.995 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:52.995 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:50:52.996 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.036593+0000", 2024-09-11T09:50:52.996 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.1613000000000002e-05 2024-09-11T09:50:52.996 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:52.996 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:50:52.996 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:52.996 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:52.996 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:52.996 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4183.0:136 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'94, mlcod=13'86)", 2024-09-11T09:50:52.996 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:12.036591+0000", 2024-09-11T09:50:52.996 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 40.884834587999997, 2024-09-11T09:50:52.996 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00090637900000000002, 2024-09-11T09:50:52.996 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:50:52.996 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:50:52.997 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:52.997 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:50:52.997 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:52.997 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:50:52.997 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.036591+0000", 2024-09-11T09:50:52.997 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:52.997 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:52.997 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:52.997 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:50:52.997 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.036591+0000", 2024-09-11T09:50:52.997 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:52.997 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:52.997 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:52.998 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:50:52.998 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.036592+0000", 2024-09-11T09:50:52.998 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.37e-07 2024-09-11T09:50:52.998 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:52.998 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:52.998 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:50:52.998 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.036600+0000", 2024-09-11T09:50:52.998 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.6910000000000003e-06 2024-09-11T09:50:52.998 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:52.998 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:52.998 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:50:52.998 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.036609+0000", 2024-09-11T09:50:52.998 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.0229999999999998e-06 2024-09-11T09:50:52.998 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:52.999 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:52.999 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:50:52.999 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.036617+0000", 2024-09-11T09:50:52.999 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.6979999999999998e-06 2024-09-11T09:50:52.999 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:52.999 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:52.999 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:50:52.999 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.036658+0000", 2024-09-11T09:50:52.999 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.0692000000000003e-05 2024-09-11T09:50:52.999 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:52.999 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:52.999 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:50:52.999 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.036670+0000", 2024-09-11T09:50:53.000 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2109e-05 2024-09-11T09:50:53.000 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:53.000 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:53.000 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:50:53.000 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.037475+0000", 2024-09-11T09:50:53.000 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00080487199999999999 2024-09-11T09:50:53.000 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:53.000 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:53.000 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:50:53.000 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.037497+0000", 2024-09-11T09:50:53.000 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.2557000000000001e-05 2024-09-11T09:50:53.000 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:53.000 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:50:53.000 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:53.001 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:53.001 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:53.001 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4183.0:158 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'112, mlcod=13'108)", 2024-09-11T09:50:53.001 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:12.455073+0000", 2024-09-11T09:50:53.001 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 40.466352335000003, 2024-09-11T09:50:53.001 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00090654500000000003, 2024-09-11T09:50:53.001 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:50:53.001 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:50:53.001 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:53.001 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:50:53.001 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:53.001 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:50:53.001 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.455073+0000", 2024-09-11T09:50:53.002 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:53.002 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:53.002 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:53.002 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:50:53.002 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.455073+0000", 2024-09-11T09:50:53.002 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:53.002 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:53.002 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:53.002 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:50:53.002 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.455074+0000", 2024-09-11T09:50:53.002 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.32e-07 2024-09-11T09:50:53.002 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:53.002 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:53.003 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:50:53.003 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.455126+0000", 2024-09-11T09:50:53.003 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.1898999999999999e-05 2024-09-11T09:50:53.003 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:53.003 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:53.003 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:50:53.003 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.455136+0000", 2024-09-11T09:50:53.003 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0651e-05 2024-09-11T09:50:53.003 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:53.003 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:53.003 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:50:53.003 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.455142+0000", 2024-09-11T09:50:53.003 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.0020000000000003e-06 2024-09-11T09:50:53.004 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:53.004 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:53.004 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:50:53.004 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.455192+0000", 2024-09-11T09:50:53.004 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.0164000000000003e-05 2024-09-11T09:50:53.004 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:53.004 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:53.004 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:50:53.004 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.455204+0000", 2024-09-11T09:50:53.004 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1178999999999999e-05 2024-09-11T09:50:53.004 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:53.004 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:53.004 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:50:53.005 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.455957+0000", 2024-09-11T09:50:53.005 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00075362300000000003 2024-09-11T09:50:53.005 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:53.005 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:53.005 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:50:53.005 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.455980+0000", 2024-09-11T09:50:53.005 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.2595e-05 2024-09-11T09:50:53.005 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:53.005 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:50:53.005 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:53.005 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:53.005 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:53.005 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "MOSDPGPush(1.0 34/33 [PushOp(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head, version: 13'116, data_included: [0~262688], data_size: 262688, omap_header_size: 0, omap_entries_size: 0, attrset_size: 9, recovery_info: ObjectRecoveryInfo(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head@13'116, size: 262688, copy_subset: [(0, 262688)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 262688, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-11T09:50:53.006 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:47.599030+0000", 2024-09-11T09:50:53.006 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 5.3223957090000003, 2024-09-11T09:50:53.006 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0010572209999999999, 2024-09-11T09:50:53.006 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:50:53.006 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:50:53.006 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "started", 2024-09-11T09:50:53.006 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:50:53.006 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:53.006 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:50:53.006 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:47.599030+0000", 2024-09-11T09:50:53.006 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:53.006 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:53.006 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:53.007 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:50:53.007 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:47.599030+0000", 2024-09-11T09:50:53.007 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:53.007 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:53.007 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:53.007 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:50:53.007 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:47.599031+0000", 2024-09-11T09:50:53.007 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.92e-07 2024-09-11T09:50:53.007 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:53.007 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:53.007 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:50:53.007 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:47.599328+0000", 2024-09-11T09:50:53.007 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00029708399999999998 2024-09-11T09:50:53.007 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:53.008 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:53.008 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:50:53.008 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:47.599353+0000", 2024-09-11T09:50:53.008 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.5721e-05 2024-09-11T09:50:53.008 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:53.008 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:53.008 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:50:53.008 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:47.599375+0000", 2024-09-11T09:50:53.008 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.1990000000000001e-05 2024-09-11T09:50:53.008 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:53.008 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:53.008 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:50:53.009 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:47.599466+0000", 2024-09-11T09:50:53.009 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.0175999999999999e-05 2024-09-11T09:50:53.009 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:53.009 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:53.009 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:50:53.009 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:47.599480+0000", 2024-09-11T09:50:53.009 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.4873e-05 2024-09-11T09:50:53.009 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:53.009 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:53.009 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:50:53.009 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:47.600087+0000", 2024-09-11T09:50:53.009 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00060668500000000004 2024-09-11T09:50:53.009 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:53.009 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:50:53.010 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:53.010 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:53.010 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:53.010 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "MOSDPGPush(1.0 34/33 [PushOp(1:30306672:devicehealth::main.db.0000000000000000:head, version: 13'117, data_included: [0~327680], data_size: 327680, omap_header_size: 0, omap_entries_size: 0, attrset_size: 10, recovery_info: ObjectRecoveryInfo(1:30306672:devicehealth::main.db.0000000000000000:head@13'117, size: 327680, copy_subset: [(0, 327680)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 327680, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-11T09:50:53.010 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:47.601120+0000", 2024-09-11T09:50:53.010 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 5.3203052209999999, 2024-09-11T09:50:53.010 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00099708000000000001, 2024-09-11T09:50:53.010 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:50:53.010 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:50:53.010 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "started", 2024-09-11T09:50:53.010 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:50:53.010 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:53.010 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:50:53.010 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:47.601120+0000", 2024-09-11T09:50:53.011 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:53.011 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:53.011 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:53.011 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:50:53.011 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:47.601120+0000", 2024-09-11T09:50:53.011 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:53.011 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:53.011 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:53.011 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:50:53.011 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:47.601121+0000", 2024-09-11T09:50:53.011 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.0200000000000001e-07 2024-09-11T09:50:53.011 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:53.011 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:53.011 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:50:53.012 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:47.601430+0000", 2024-09-11T09:50:53.012 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00030917899999999998 2024-09-11T09:50:53.012 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:53.012 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:53.012 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:50:53.012 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:47.601454+0000", 2024-09-11T09:50:53.012 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.4000999999999999e-05 2024-09-11T09:50:53.012 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:53.012 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:53.012 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:50:53.012 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:47.601475+0000", 2024-09-11T09:50:53.012 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.0305999999999998e-05 2024-09-11T09:50:53.012 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:53.012 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:53.013 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:50:53.013 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:47.601561+0000", 2024-09-11T09:50:53.013 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.6414000000000003e-05 2024-09-11T09:50:53.013 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:53.013 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:53.013 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:50:53.013 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:47.601573+0000", 2024-09-11T09:50:53.013 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1894e-05 2024-09-11T09:50:53.013 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:53.013 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:53.013 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:50:53.013 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:47.602117+0000", 2024-09-11T09:50:53.013 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00054458399999999998 2024-09-11T09:50:53.013 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:53.014 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:50:53.014 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:53.014 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:53.014 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:50:53.014 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:50:53.015 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-11T09:50:53.016 INFO:tasks.ceph.osd.0.smithi017.stderr:2024-09-11T09:50:53.016+0000 7fb73c0f1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31665) UID: 0 2024-09-11T09:50:53.116 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:50:53.117+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:50:53.163 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:50:53.163 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [], 2024-09-11T09:50:53.163 INFO:teuthology.orchestra.run.smithi017.stdout: "num_ops": 0 2024-09-11T09:50:53.163 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:50:53.172 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-11T09:50:53.217 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:50:53.218+0000 7fdec5849640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31907) UID: 0 2024-09-11T09:50:53.233 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T09:50:53.318 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:50:53.319+0000 7fdec5849640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31907) UID: 0 2024-09-11T09:50:53.320 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:50:53.320 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [], 2024-09-11T09:50:53.320 INFO:teuthology.orchestra.run.smithi017.stdout: "complaint_time": 30, 2024-09-11T09:50:53.320 INFO:teuthology.orchestra.run.smithi017.stdout: "num_blocked_ops": 0 2024-09-11T09:50:53.320 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:50:53.329 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-11T09:50:53.381 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-11T09:50:53.382 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T09:50:53.399 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-11T09:50:53.399 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T09:50:53.416 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-11T09:50:53.416 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T09:50:53.418 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:50:53.419+0000 7f3c96ec9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31909) UID: 0 2024-09-11T09:50:53.433 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-11T09:50:53.433 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T09:50:53.441 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:50:53.441 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:50:53.450 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-11T09:50:53.451 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-11T09:50:53.451 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T09:50:53.469 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-11T09:50:53.469 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T09:50:53.522 INFO:tasks.ceph.osd.0.smithi017.stderr:2024-09-11T09:50:53.523+0000 7fb73c0f1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31665) UID: 0 2024-09-11T09:50:53.559 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:50:53.559 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:50:53.567 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-11T09:50:53.652 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:50:53.653+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:50:53.716 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:50:53.716 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:50:53.721 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-09-11T09:50:53.723+0000 7f7b6c7d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31904) UID: 0 2024-09-11T09:50:53.725 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-11T09:50:53.737 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-11T09:50:53.823 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-09-11T09:50:53.823+0000 7f7b6c7d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31904) UID: 0 2024-09-11T09:50:53.875 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:50:53.875 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:50:53.885 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-11T09:50:53.923 INFO:tasks.ceph.osd.2.smithi017.stderr:2024-09-11T09:50:53.924+0000 7fd20b170640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31672) UID: 0 2024-09-11T09:50:53.999 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:50:53.999 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:50:54.008 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-11T09:50:54.025 INFO:tasks.ceph.osd.0.smithi017.stderr:2024-09-11T09:50:54.026+0000 7fb73c0f1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31665) UID: 0 2024-09-11T09:50:54.111 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:50:54.111 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:50:54.120 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-11T09:50:54.165 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:50:54.166+0000 7f3c96ec9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31909) UID: 0 2024-09-11T09:50:54.222 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:50:54.223 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:50:54.226 INFO:tasks.ceph.osd.2.smithi017.stderr:2024-09-11T09:50:54.227+0000 7fd20b170640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31672) UID: 0 2024-09-11T09:50:54.232 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-11T09:50:54.327 INFO:tasks.ceph.osd.0.smithi017.stderr:2024-09-11T09:50:54.328+0000 7fb73c0f1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31665) UID: 0 2024-09-11T09:50:54.376 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:50:54.376 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:50:54.384 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-11T09:50:54.428 INFO:tasks.ceph.osd.0.smithi017.stderr:2024-09-11T09:50:54.429+0000 7fb73c0f1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31665) UID: 0 2024-09-11T09:50:54.486 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:50:54.486 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:50:54.495 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-11T09:50:54.538 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:50:54.539+0000 7fdec5849640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31907) UID: 0 2024-09-11T09:50:54.598 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:50:54.598 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:50:54.607 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-11T09:50:54.625 INFO:teuthology.orchestra.run.smithi017.stderr:nodeep-scrub is set 2024-09-11T09:50:54.629 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:50:54.630+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:50:54.730 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:50:54.731+0000 7f3c96ec9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31909) UID: 0 2024-09-11T09:50:54.750 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:50:54.750 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:50:54.759 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-11T09:50:54.831 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:50:54.832+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:50:54.902 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:50:54.902 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:50:54.911 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-11T09:50:54.932 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:50:54.933+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:50:55.014 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:50:55.014 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:50:55.023 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-11T09:50:55.033 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-09-11T09:50:55.034+0000 7f7b6c7d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31904) UID: 0 2024-09-11T09:50:55.134 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:50:55.135+0000 7fdec5849640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31907) UID: 0 2024-09-11T09:50:55.144 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:50:55.144 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:50:55.153 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-11T09:50:55.235 INFO:tasks.ceph.osd.2.smithi017.stderr:2024-09-11T09:50:55.236+0000 7fd20b170640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31672) UID: 0 2024-09-11T09:50:55.262 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:50:55.262 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:50:55.271 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-11T09:50:55.355 INFO:tasks.ceph.osd.0.smithi017.stderr:2024-09-11T09:50:55.356+0000 7fb73c0f1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31665) UID: 0 2024-09-11T09:50:55.419 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:50:55.419 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:50:55.427 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-11T09:50:55.437 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:50:55.438+0000 7fdec5849640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31907) UID: 0 2024-09-11T09:50:55.481 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T09:50:55.538 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-09-11T09:50:55.539+0000 7f7b6c7d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31904) UID: 0 2024-09-11T09:50:55.575 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:50:55.576 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:50:55.585 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-11T09:50:55.636 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-11T09:50:55.636 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T09:50:55.639 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:50:55.640+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:50:55.653 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-11T09:50:55.653 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T09:50:55.670 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-11T09:50:55.671 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T09:50:55.687 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-11T09:50:55.687 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T09:50:55.696 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:50:55.696 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [], 2024-09-11T09:50:55.696 INFO:teuthology.orchestra.run.smithi017.stdout: "complaint_time": 30, 2024-09-11T09:50:55.696 INFO:teuthology.orchestra.run.smithi017.stdout: "num_blocked_ops": 0 2024-09-11T09:50:55.696 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:50:55.704 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-11T09:50:55.704 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T09:50:55.705 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-11T09:50:55.721 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-11T09:50:55.721 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T09:50:55.740 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-09-11T09:50:55.741+0000 7f7b6c7d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31904) UID: 0 2024-09-11T09:50:55.828 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:50:55.828 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 20, 2024-09-11T09:50:55.828 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 600, 2024-09-11T09:50:55.828 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [ 2024-09-11T09:50:55.829 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:55.829 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4230.0:5 2.2 e17/15 2:5cea7035:::rbd_info:head v 17'1, mlcod=0'0)", 2024-09-11T09:50:55.829 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:19.494757+0000", 2024-09-11T09:50:55.829 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 36.335054045, 2024-09-11T09:50:55.829 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00082098399999999997, 2024-09-11T09:50:55.829 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:50:55.829 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:50:55.829 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:55.829 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:50:55.829 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:55.829 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:50:55.829 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:19.494757+0000", 2024-09-11T09:50:55.829 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:55.830 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:55.830 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:55.830 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:50:55.830 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:19.494757+0000", 2024-09-11T09:50:55.830 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:55.830 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:55.830 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:55.830 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:50:55.830 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:19.494757+0000", 2024-09-11T09:50:55.830 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.97e-07 2024-09-11T09:50:55.830 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:55.830 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:55.831 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:50:55.831 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:19.494763+0000", 2024-09-11T09:50:55.831 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.3709999999999996e-06 2024-09-11T09:50:55.831 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:55.831 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:55.831 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:50:55.831 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:19.494778+0000", 2024-09-11T09:50:55.831 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.4960000000000001e-05 2024-09-11T09:50:55.831 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:55.831 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:55.831 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:50:55.831 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:19.494801+0000", 2024-09-11T09:50:55.831 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.2388999999999999e-05 2024-09-11T09:50:55.831 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:55.832 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:55.832 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:50:55.832 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:19.494859+0000", 2024-09-11T09:50:55.832 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.7961999999999998e-05 2024-09-11T09:50:55.832 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:55.832 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:55.832 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:50:55.832 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:19.494883+0000", 2024-09-11T09:50:55.832 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.4488999999999999e-05 2024-09-11T09:50:55.832 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:55.832 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:55.832 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:50:55.832 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:19.495534+0000", 2024-09-11T09:50:55.832 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00065075500000000004 2024-09-11T09:50:55.833 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:55.833 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:55.833 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:50:55.833 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:19.495578+0000", 2024-09-11T09:50:55.833 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.3860999999999999e-05 2024-09-11T09:50:55.833 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:55.833 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:50:55.833 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:55.833 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:55.833 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:55.833 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4230.0:7 2.2 e18/15 2:5cea7035:::rbd_info:head v 18'2, mlcod=0'0)", 2024-09-11T09:50:55.833 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:20.500782+0000", 2024-09-11T09:50:55.834 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 35.329029165000001, 2024-09-11T09:50:55.834 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000790185, 2024-09-11T09:50:55.834 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:50:55.834 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:50:55.834 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:55.834 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:50:55.834 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:55.834 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:50:55.834 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:20.500782+0000", 2024-09-11T09:50:55.834 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:55.834 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:55.834 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:55.834 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:50:55.835 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:20.500782+0000", 2024-09-11T09:50:55.835 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:55.835 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:55.835 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:55.835 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:50:55.835 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:20.500782+0000", 2024-09-11T09:50:55.835 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.3299999999999998e-07 2024-09-11T09:50:55.835 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:55.835 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:55.835 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:50:55.835 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:20.500799+0000", 2024-09-11T09:50:55.835 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6830999999999999e-05 2024-09-11T09:50:55.836 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:55.836 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:55.836 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:50:55.836 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:20.500812+0000", 2024-09-11T09:50:55.836 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3013999999999999e-05 2024-09-11T09:50:55.836 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:55.836 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:55.836 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:50:55.836 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:20.500824+0000", 2024-09-11T09:50:55.836 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2593e-05 2024-09-11T09:50:55.836 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:55.836 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:55.837 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:50:55.837 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:20.500895+0000", 2024-09-11T09:50:55.837 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.1018000000000003e-05 2024-09-11T09:50:55.837 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:55.837 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:55.837 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:50:55.837 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:20.500912+0000", 2024-09-11T09:50:55.837 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6484e-05 2024-09-11T09:50:55.837 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:55.837 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:55.837 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:50:55.837 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:20.501531+0000", 2024-09-11T09:50:55.837 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00061903399999999997 2024-09-11T09:50:55.838 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:55.838 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:55.838 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:50:55.838 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:20.501572+0000", 2024-09-11T09:50:55.838 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.0877999999999997e-05 2024-09-11T09:50:55.838 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:55.838 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:50:55.838 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:55.838 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:55.838 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:50:55.838 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:50:55.840 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-11T09:50:55.841 INFO:tasks.ceph.osd.2.smithi017.stderr:2024-09-11T09:50:55.842+0000 7fd20b170640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31672) UID: 0 2024-09-11T09:50:55.942 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:50:55.943+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:50:55.947 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:50:55.947 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [], 2024-09-11T09:50:55.947 INFO:teuthology.orchestra.run.smithi017.stdout: "num_ops": 0 2024-09-11T09:50:55.947 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:50:55.956 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-11T09:50:56.043 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:50:56.044+0000 7f3c96ec9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31909) UID: 0 2024-09-11T09:50:56.103 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:50:56.103 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [], 2024-09-11T09:50:56.103 INFO:teuthology.orchestra.run.smithi017.stdout: "complaint_time": 30, 2024-09-11T09:50:56.103 INFO:teuthology.orchestra.run.smithi017.stdout: "num_blocked_ops": 0 2024-09-11T09:50:56.103 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:50:56.112 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-11T09:50:56.144 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:50:56.145+0000 7f3c96ec9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31909) UID: 0 2024-09-11T09:50:56.219 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:50:56.219 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 20, 2024-09-11T09:50:56.220 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 600, 2024-09-11T09:50:56.220 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [ 2024-09-11T09:50:56.220 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:56.220 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4230.0:1 2.3 e16/15 2:c4c92e5a:::rbd_trash:head v 16'1, mlcod=0'0)", 2024-09-11T09:50:56.220 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:18.494378+0000", 2024-09-11T09:50:56.220 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 37.726686641000001, 2024-09-11T09:50:56.220 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00089752199999999999, 2024-09-11T09:50:56.220 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:50:56.220 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:50:56.220 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:56.220 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:50:56.220 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:56.221 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:50:56.221 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:18.494378+0000", 2024-09-11T09:50:56.221 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:56.221 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:56.221 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:56.221 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:50:56.221 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:18.494378+0000", 2024-09-11T09:50:56.221 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:50:56.221 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:56.221 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:56.221 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:50:56.221 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:18.494378+0000", 2024-09-11T09:50:56.221 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.9099999999999999e-07 2024-09-11T09:50:56.222 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:56.222 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:56.222 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:50:56.222 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:18.494385+0000", 2024-09-11T09:50:56.222 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.8129999999999998e-06 2024-09-11T09:50:56.222 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:56.222 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:56.222 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:50:56.222 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:18.494416+0000", 2024-09-11T09:50:56.222 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.0654000000000002e-05 2024-09-11T09:50:56.222 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:56.222 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:56.222 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:50:56.223 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:18.494430+0000", 2024-09-11T09:50:56.223 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3754000000000001e-05 2024-09-11T09:50:56.223 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:56.223 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:56.223 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:50:56.223 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:18.494524+0000", 2024-09-11T09:50:56.223 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.4896999999999996e-05 2024-09-11T09:50:56.223 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:56.223 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:56.223 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:50:56.223 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:18.494558+0000", 2024-09-11T09:50:56.223 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.3900999999999999e-05 2024-09-11T09:50:56.223 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:56.223 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:56.224 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:50:56.224 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:18.495231+0000", 2024-09-11T09:50:56.224 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00067233500000000003 2024-09-11T09:50:56.224 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:50:56.224 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:50:56.224 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:50:56.224 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:18.495275+0000", 2024-09-11T09:50:56.224 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.4777000000000002e-05 2024-09-11T09:50:56.224 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:56.224 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:50:56.224 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:56.224 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:50:56.225 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:50:56.225 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:50:56.228 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-11T09:50:56.273 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-09-11T09:50:56.274+0000 7f7b6c7d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31904) UID: 0 2024-09-11T09:50:56.343 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-11T09:50:56.343 INFO:teuthology.orchestra.run.smithi096.stdout: "ops": [], 2024-09-11T09:50:56.343 INFO:teuthology.orchestra.run.smithi096.stdout: "num_ops": 0 2024-09-11T09:50:56.344 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-11T09:50:56.345 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:50:56.345+0000 7f3c96ec9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31909) UID: 0 2024-09-11T09:50:56.352 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-11T09:50:56.445 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:50:56.446+0000 7f3c96ec9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31909) UID: 0 2024-09-11T09:50:56.497 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-11T09:50:56.497 INFO:teuthology.orchestra.run.smithi096.stdout: "ops": [], 2024-09-11T09:50:56.497 INFO:teuthology.orchestra.run.smithi096.stdout: "complaint_time": 30, 2024-09-11T09:50:56.497 INFO:teuthology.orchestra.run.smithi096.stdout: "num_blocked_ops": 0 2024-09-11T09:50:56.497 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-11T09:50:56.506 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-11T09:50:56.547 INFO:tasks.ceph.osd.2.smithi017.stderr:2024-09-11T09:50:56.547+0000 7fd20b170640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31672) UID: 0 2024-09-11T09:50:56.642 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-11T09:50:56.647 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:50:56.648+0000 7fdec5849640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31907) UID: 0 2024-09-11T09:50:56.651 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-11T09:50:56.652 INFO:teuthology.orchestra.run.smithi096.stdout: "size": 20, 2024-09-11T09:50:56.652 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 600, 2024-09-11T09:50:56.652 INFO:teuthology.orchestra.run.smithi096.stdout: "ops": [ 2024-09-11T09:50:56.652 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.652 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:74 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:50:56.652 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.539067+0000", 2024-09-11T09:50:56.652 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 45.112297126000001, 2024-09-11T09:50:56.652 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.006945957, 2024-09-11T09:50:56.652 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:50:56.652 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:50:56.652 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:56.653 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:50:56.653 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:50:56.653 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:50:56.653 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 74 2024-09-11T09:50:56.653 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.653 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:50:56.653 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.653 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:50:56.653 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.539067+0000", 2024-09-11T09:50:56.653 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:56.653 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.653 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.654 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:50:56.654 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.539067+0000", 2024-09-11T09:50:56.654 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:56.654 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.654 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.654 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:50:56.654 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.539068+0000", 2024-09-11T09:50:56.654 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.6000000000000003e-07 2024-09-11T09:50:56.654 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.654 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.654 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:50:56.654 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.539072+0000", 2024-09-11T09:50:56.654 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.0450000000000002e-06 2024-09-11T09:50:56.654 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.655 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.655 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:50:56.655 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.539078+0000", 2024-09-11T09:50:56.655 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.6889999999999998e-06 2024-09-11T09:50:56.655 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.655 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.655 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:50:56.655 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.539083+0000", 2024-09-11T09:50:56.655 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.7180000000000004e-06 2024-09-11T09:50:56.655 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.655 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.655 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:50:56.655 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.540230+0000", 2024-09-11T09:50:56.655 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0011466250000000001 2024-09-11T09:50:56.655 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.656 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.656 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:50:56.656 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.540279+0000", 2024-09-11T09:50:56.656 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.9667999999999999e-05 2024-09-11T09:50:56.656 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.656 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.656 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:50:56.656 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.540357+0000", 2024-09-11T09:50:56.656 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.7863999999999999e-05 2024-09-11T09:50:56.656 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.656 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.656 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:50:56.656 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.541086+0000", 2024-09-11T09:50:56.656 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00072921100000000001 2024-09-11T09:50:56.657 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.657 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.657 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:50:56.657 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.545949+0000", 2024-09-11T09:50:56.657 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0048626040000000004 2024-09-11T09:50:56.657 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.657 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.657 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:50:56.657 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.545989+0000", 2024-09-11T09:50:56.657 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.0259000000000002e-05 2024-09-11T09:50:56.657 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.657 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.657 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:50:56.657 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.546013+0000", 2024-09-11T09:50:56.658 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.3614000000000001e-05 2024-09-11T09:50:56.658 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:56.658 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:50:56.658 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:56.658 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.658 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.658 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:108 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:50:56.658 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.711510+0000", 2024-09-11T09:50:56.658 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 44.939853954, 2024-09-11T09:50:56.658 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0054615649999999998, 2024-09-11T09:50:56.658 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:50:56.658 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:50:56.658 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:56.658 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:50:56.659 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:50:56.659 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:50:56.659 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 108 2024-09-11T09:50:56.659 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.659 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:50:56.659 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.659 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:50:56.659 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711510+0000", 2024-09-11T09:50:56.659 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:56.659 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.659 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.659 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:50:56.659 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711510+0000", 2024-09-11T09:50:56.660 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:56.660 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.660 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.660 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:50:56.660 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711511+0000", 2024-09-11T09:50:56.660 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.1999999999999998e-07 2024-09-11T09:50:56.660 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.660 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.660 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:50:56.660 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711516+0000", 2024-09-11T09:50:56.660 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.4140000000000002e-06 2024-09-11T09:50:56.660 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.660 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.661 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:50:56.661 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711527+0000", 2024-09-11T09:50:56.661 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.0169e-05 2024-09-11T09:50:56.661 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.661 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.661 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:50:56.661 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711535+0000", 2024-09-11T09:50:56.661 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.3299999999999999e-06 2024-09-11T09:50:56.661 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.661 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.661 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:50:56.661 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.713763+0000", 2024-09-11T09:50:56.661 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0022281079999999999 2024-09-11T09:50:56.662 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.662 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.662 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:50:56.662 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.713812+0000", 2024-09-11T09:50:56.662 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.8544000000000001e-05 2024-09-11T09:50:56.662 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.662 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.662 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:50:56.662 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.713909+0000", 2024-09-11T09:50:56.662 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.7113000000000003e-05 2024-09-11T09:50:56.662 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.662 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.662 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:50:56.663 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.716873+0000", 2024-09-11T09:50:56.663 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0029642710000000001 2024-09-11T09:50:56.663 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.663 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.663 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:50:56.663 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.716918+0000", 2024-09-11T09:50:56.663 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.5447e-05 2024-09-11T09:50:56.663 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.663 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.663 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:50:56.663 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.716947+0000", 2024-09-11T09:50:56.663 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.8181e-05 2024-09-11T09:50:56.663 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.664 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.664 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:50:56.664 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.716972+0000", 2024-09-11T09:50:56.664 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.5068000000000001e-05 2024-09-11T09:50:56.664 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:56.664 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:50:56.664 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:56.664 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.664 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.664 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:109 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:50:56.664 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.711555+0000", 2024-09-11T09:50:56.664 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 44.939808704999997, 2024-09-11T09:50:56.664 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.005479902, 2024-09-11T09:50:56.665 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:50:56.665 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:50:56.665 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:56.665 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:50:56.665 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:50:56.665 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:50:56.665 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 109 2024-09-11T09:50:56.665 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.665 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:50:56.665 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.665 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:50:56.665 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711555+0000", 2024-09-11T09:50:56.665 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:56.666 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.666 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.666 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:50:56.666 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711555+0000", 2024-09-11T09:50:56.666 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:56.666 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.666 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.666 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:50:56.666 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711556+0000", 2024-09-11T09:50:56.666 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.3200000000000005e-07 2024-09-11T09:50:56.666 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.666 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.666 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:50:56.667 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711601+0000", 2024-09-11T09:50:56.667 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.5349000000000002e-05 2024-09-11T09:50:56.667 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.667 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.667 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:50:56.667 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711612+0000", 2024-09-11T09:50:56.667 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.0701e-05 2024-09-11T09:50:56.667 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.667 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.667 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:50:56.667 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711619+0000", 2024-09-11T09:50:56.667 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.7610000000000002e-06 2024-09-11T09:50:56.667 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.667 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.668 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:50:56.668 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.714371+0000", 2024-09-11T09:50:56.668 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0027526260000000002 2024-09-11T09:50:56.668 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.668 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.668 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:50:56.668 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.714417+0000", 2024-09-11T09:50:56.668 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.5893999999999998e-05 2024-09-11T09:50:56.668 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.668 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.668 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:50:56.668 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.714496+0000", 2024-09-11T09:50:56.668 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.8539000000000001e-05 2024-09-11T09:50:56.669 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.669 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.669 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:50:56.669 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.716910+0000", 2024-09-11T09:50:56.669 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0024137640000000001 2024-09-11T09:50:56.669 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.669 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.669 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:50:56.669 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.716989+0000", 2024-09-11T09:50:56.669 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.9198999999999998e-05 2024-09-11T09:50:56.669 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.669 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.669 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:50:56.669 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.717016+0000", 2024-09-11T09:50:56.670 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.6699e-05 2024-09-11T09:50:56.670 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.670 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.670 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:50:56.670 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.717035+0000", 2024-09-11T09:50:56.670 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.9738e-05 2024-09-11T09:50:56.670 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:56.670 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:50:56.670 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:56.670 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.670 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.670 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:110 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:50:56.670 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.711636+0000", 2024-09-11T09:50:56.670 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 44.939728244999998, 2024-09-11T09:50:56.671 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0067613070000000003, 2024-09-11T09:50:56.671 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:50:56.671 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:50:56.671 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:56.671 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:50:56.671 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:50:56.671 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:50:56.671 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 110 2024-09-11T09:50:56.671 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.671 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:50:56.671 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.671 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:50:56.671 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711636+0000", 2024-09-11T09:50:56.672 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:56.672 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.672 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.672 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:50:56.672 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711636+0000", 2024-09-11T09:50:56.672 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:56.672 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.672 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.672 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:50:56.672 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711637+0000", 2024-09-11T09:50:56.672 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.9999999999999997e-07 2024-09-11T09:50:56.672 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.673 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.673 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:50:56.673 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711641+0000", 2024-09-11T09:50:56.673 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.2570000000000001e-06 2024-09-11T09:50:56.673 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.673 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.673 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:50:56.673 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711651+0000", 2024-09-11T09:50:56.673 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.0285999999999999e-05 2024-09-11T09:50:56.673 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.673 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.673 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:50:56.674 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711658+0000", 2024-09-11T09:50:56.674 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.3540000000000002e-06 2024-09-11T09:50:56.674 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.674 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.674 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:50:56.674 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.716475+0000", 2024-09-11T09:50:56.674 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0048162819999999999 2024-09-11T09:50:56.674 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.674 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.674 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:50:56.674 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.716526+0000", 2024-09-11T09:50:56.674 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.0905000000000001e-05 2024-09-11T09:50:56.674 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.674 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.674 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:50:56.675 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.716601+0000", 2024-09-11T09:50:56.675 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.5283000000000004e-05 2024-09-11T09:50:56.675 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.675 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.675 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:50:56.675 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.717933+0000", 2024-09-11T09:50:56.675 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001331622 2024-09-11T09:50:56.675 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.675 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.675 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:50:56.675 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718332+0000", 2024-09-11T09:50:56.675 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00039909899999999998 2024-09-11T09:50:56.675 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.675 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.675 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:50:56.676 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718363+0000", 2024-09-11T09:50:56.676 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.1636e-05 2024-09-11T09:50:56.676 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.676 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.676 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:50:56.676 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718397+0000", 2024-09-11T09:50:56.676 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.3883000000000001e-05 2024-09-11T09:50:56.676 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:56.676 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:50:56.676 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:56.676 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.676 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.676 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:111 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:50:56.676 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.711668+0000", 2024-09-11T09:50:56.676 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 44.939695815, 2024-09-11T09:50:56.677 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0077145570000000004, 2024-09-11T09:50:56.677 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:50:56.677 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:50:56.677 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:56.677 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:50:56.677 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:50:56.677 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:50:56.677 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 111 2024-09-11T09:50:56.677 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.677 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:50:56.677 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.677 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:50:56.677 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711668+0000", 2024-09-11T09:50:56.677 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:56.677 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.678 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.678 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:50:56.678 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711668+0000", 2024-09-11T09:50:56.678 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:56.678 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.678 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.678 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:50:56.678 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711669+0000", 2024-09-11T09:50:56.678 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.99e-07 2024-09-11T09:50:56.678 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.678 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.678 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:50:56.678 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711673+0000", 2024-09-11T09:50:56.678 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.7440000000000001e-06 2024-09-11T09:50:56.678 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.679 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.679 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:50:56.679 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711686+0000", 2024-09-11T09:50:56.679 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.3317e-05 2024-09-11T09:50:56.679 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.679 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.679 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:50:56.679 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711697+0000", 2024-09-11T09:50:56.679 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1022000000000001e-05 2024-09-11T09:50:56.679 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.679 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.679 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:50:56.679 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.717065+0000", 2024-09-11T09:50:56.679 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.005367531 2024-09-11T09:50:56.679 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.680 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.680 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:50:56.680 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.717114+0000", 2024-09-11T09:50:56.680 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.9707999999999999e-05 2024-09-11T09:50:56.680 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.680 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.680 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:50:56.680 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.717190+0000", 2024-09-11T09:50:56.680 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.5394000000000004e-05 2024-09-11T09:50:56.680 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.680 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.680 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:50:56.680 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718793+0000", 2024-09-11T09:50:56.680 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001603653 2024-09-11T09:50:56.681 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.681 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.681 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:50:56.681 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719329+0000", 2024-09-11T09:50:56.681 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00053543799999999995 2024-09-11T09:50:56.681 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.681 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.681 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:50:56.681 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719359+0000", 2024-09-11T09:50:56.681 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.0420999999999999e-05 2024-09-11T09:50:56.681 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.681 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.681 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:50:56.681 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719383+0000", 2024-09-11T09:50:56.682 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.3730000000000001e-05 2024-09-11T09:50:56.682 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:56.682 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:50:56.682 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:56.682 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.682 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.682 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:112 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:50:56.682 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.711712+0000", 2024-09-11T09:50:56.682 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 44.939652254999999, 2024-09-11T09:50:56.682 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.008162084, 2024-09-11T09:50:56.682 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:50:56.682 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:50:56.683 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:56.683 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:50:56.683 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:50:56.683 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:50:56.683 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 112 2024-09-11T09:50:56.683 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.683 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:50:56.683 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.683 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:50:56.683 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711712+0000", 2024-09-11T09:50:56.683 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:56.683 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.683 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.683 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:50:56.684 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711712+0000", 2024-09-11T09:50:56.684 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:56.684 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.684 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.684 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:50:56.684 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711712+0000", 2024-09-11T09:50:56.684 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.6000000000000004e-07 2024-09-11T09:50:56.684 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.684 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.684 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:50:56.684 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711808+0000", 2024-09-11T09:50:56.684 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.5977999999999995e-05 2024-09-11T09:50:56.685 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.685 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.685 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:50:56.685 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711819+0000", 2024-09-11T09:50:56.685 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.0755e-05 2024-09-11T09:50:56.685 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.685 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.685 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:50:56.685 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711826+0000", 2024-09-11T09:50:56.685 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.7970000000000003e-06 2024-09-11T09:50:56.685 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.685 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.685 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:50:56.686 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.717435+0000", 2024-09-11T09:50:56.686 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0056090239999999998 2024-09-11T09:50:56.686 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.686 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.686 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:50:56.686 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.717496+0000", 2024-09-11T09:50:56.686 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.0543e-05 2024-09-11T09:50:56.686 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.686 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.686 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:50:56.686 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.717591+0000", 2024-09-11T09:50:56.686 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.5667000000000004e-05 2024-09-11T09:50:56.686 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.687 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.687 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:50:56.687 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718831+0000", 2024-09-11T09:50:56.687 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001240204 2024-09-11T09:50:56.687 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.687 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.687 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:50:56.687 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719819+0000", 2024-09-11T09:50:56.687 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00098751500000000001 2024-09-11T09:50:56.687 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.687 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.688 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:50:56.688 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719850+0000", 2024-09-11T09:50:56.688 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.1535999999999998e-05 2024-09-11T09:50:56.688 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.688 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.688 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:50:56.688 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719874+0000", 2024-09-11T09:50:56.688 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.3504999999999999e-05 2024-09-11T09:50:56.688 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:56.688 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:50:56.688 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:56.688 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.689 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.689 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:113 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:50:56.689 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.711842+0000", 2024-09-11T09:50:56.689 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 44.939521755999998, 2024-09-11T09:50:56.689 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0081160099999999999, 2024-09-11T09:50:56.689 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:50:56.689 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:50:56.689 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:56.689 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:50:56.689 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:50:56.689 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:50:56.689 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 113 2024-09-11T09:50:56.690 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.690 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:50:56.690 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.690 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:50:56.690 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711842+0000", 2024-09-11T09:50:56.690 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:56.690 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.690 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.690 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:50:56.690 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711842+0000", 2024-09-11T09:50:56.690 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:56.690 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.690 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.690 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:50:56.691 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711843+0000", 2024-09-11T09:50:56.691 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.4399999999999999e-07 2024-09-11T09:50:56.691 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.691 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.691 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:50:56.691 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711848+0000", 2024-09-11T09:50:56.691 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.4159999999999997e-06 2024-09-11T09:50:56.691 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.691 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.691 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:50:56.691 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711856+0000", 2024-09-11T09:50:56.691 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.7870000000000007e-06 2024-09-11T09:50:56.691 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.692 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.692 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:50:56.692 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711862+0000", 2024-09-11T09:50:56.692 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.3689999999999998e-06 2024-09-11T09:50:56.692 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.692 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.692 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:50:56.692 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.717965+0000", 2024-09-11T09:50:56.692 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0061028899999999997 2024-09-11T09:50:56.692 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.692 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.692 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:50:56.692 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718023+0000", 2024-09-11T09:50:56.692 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.8313000000000002e-05 2024-09-11T09:50:56.693 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.693 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.693 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:50:56.693 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718109+0000", 2024-09-11T09:50:56.693 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.6205000000000006e-05 2024-09-11T09:50:56.693 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.693 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.693 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:50:56.693 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719808+0000", 2024-09-11T09:50:56.693 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0016990060000000001 2024-09-11T09:50:56.693 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.693 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.693 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:50:56.694 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719883+0000", 2024-09-11T09:50:56.694 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.4624999999999998e-05 2024-09-11T09:50:56.694 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.694 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.694 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:50:56.694 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719910+0000", 2024-09-11T09:50:56.694 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.7512e-05 2024-09-11T09:50:56.694 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.694 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.694 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:50:56.694 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719958+0000", 2024-09-11T09:50:56.694 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.8143000000000003e-05 2024-09-11T09:50:56.694 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:56.694 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:50:56.694 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:56.695 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.695 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.695 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:50:56.695 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.711887+0000", 2024-09-11T09:50:56.695 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 44.939477142000001, 2024-09-11T09:50:56.695 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0086063589999999992, 2024-09-11T09:50:56.695 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:50:56.695 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:50:56.695 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:56.695 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:50:56.695 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:50:56.695 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:50:56.695 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 114 2024-09-11T09:50:56.695 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.696 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:50:56.696 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.696 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:50:56.696 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711887+0000", 2024-09-11T09:50:56.696 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:56.696 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.696 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.696 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:50:56.696 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711887+0000", 2024-09-11T09:50:56.696 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:56.696 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.696 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.697 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:50:56.697 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711887+0000", 2024-09-11T09:50:56.697 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.1500000000000005e-07 2024-09-11T09:50:56.697 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.697 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.697 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:50:56.697 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711897+0000", 2024-09-11T09:50:56.697 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.1900000000000001e-06 2024-09-11T09:50:56.697 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.697 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.697 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:50:56.697 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711902+0000", 2024-09-11T09:50:56.697 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.2349999999999999e-06 2024-09-11T09:50:56.697 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.697 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.698 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:50:56.698 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711906+0000", 2024-09-11T09:50:56.698 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.7129999999999999e-06 2024-09-11T09:50:56.698 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.698 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.698 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:50:56.698 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718429+0000", 2024-09-11T09:50:56.698 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0065231960000000002 2024-09-11T09:50:56.698 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.698 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.698 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:50:56.698 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718480+0000", 2024-09-11T09:50:56.698 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.1161e-05 2024-09-11T09:50:56.698 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.698 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.699 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:50:56.699 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718558+0000", 2024-09-11T09:50:56.699 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.7519000000000001e-05 2024-09-11T09:50:56.699 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.699 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.699 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:50:56.699 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720038+0000", 2024-09-11T09:50:56.699 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001480016 2024-09-11T09:50:56.699 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.699 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.699 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:50:56.699 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720417+0000", 2024-09-11T09:50:56.699 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00037943699999999999 2024-09-11T09:50:56.699 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.699 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.700 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:50:56.700 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720452+0000", 2024-09-11T09:50:56.700 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.4595000000000003e-05 2024-09-11T09:50:56.700 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.700 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.700 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:50:56.700 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720493+0000", 2024-09-11T09:50:56.700 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.1782000000000001e-05 2024-09-11T09:50:56.700 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:56.700 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:50:56.700 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:56.700 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.700 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.700 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:50:56.701 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.711912+0000", 2024-09-11T09:50:56.701 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 44.939452473000003, 2024-09-11T09:50:56.701 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0086864750000000008, 2024-09-11T09:50:56.701 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:50:56.701 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:50:56.701 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:56.701 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:50:56.701 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:50:56.701 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:50:56.701 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 115 2024-09-11T09:50:56.701 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.701 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:50:56.701 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.702 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:50:56.702 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711912+0000", 2024-09-11T09:50:56.702 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:56.702 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.702 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.702 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:50:56.702 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711912+0000", 2024-09-11T09:50:56.702 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:56.702 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.702 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.702 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:50:56.702 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711912+0000", 2024-09-11T09:50:56.702 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.3900000000000001e-07 2024-09-11T09:50:56.702 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.702 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.703 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:50:56.703 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711969+0000", 2024-09-11T09:50:56.703 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.7501999999999999e-05 2024-09-11T09:50:56.703 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.703 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.703 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:50:56.703 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711986+0000", 2024-09-11T09:50:56.703 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.6402999999999999e-05 2024-09-11T09:50:56.703 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.703 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.703 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:50:56.703 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711990+0000", 2024-09-11T09:50:56.703 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.929e-06 2024-09-11T09:50:56.703 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.703 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.704 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:50:56.704 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718861+0000", 2024-09-11T09:50:56.704 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0068710530000000002 2024-09-11T09:50:56.704 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.704 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.704 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:50:56.704 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718908+0000", 2024-09-11T09:50:56.704 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.7431e-05 2024-09-11T09:50:56.704 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.704 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.704 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:50:56.704 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718984+0000", 2024-09-11T09:50:56.704 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.5500999999999994e-05 2024-09-11T09:50:56.705 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.705 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.705 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:50:56.705 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720500+0000", 2024-09-11T09:50:56.705 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0015158820000000001 2024-09-11T09:50:56.705 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.705 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.705 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:50:56.705 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720536+0000", 2024-09-11T09:50:56.705 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.6118000000000002e-05 2024-09-11T09:50:56.705 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.705 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.705 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:50:56.705 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720570+0000", 2024-09-11T09:50:56.705 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.4387999999999997e-05 2024-09-11T09:50:56.706 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.706 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.706 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:50:56.706 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720598+0000", 2024-09-11T09:50:56.706 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.8028999999999999e-05 2024-09-11T09:50:56.706 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:56.706 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:50:56.706 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:56.706 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.706 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.706 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:50:56.706 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.711998+0000", 2024-09-11T09:50:56.706 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 44.939366294000003, 2024-09-11T09:50:56.706 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.010785813999999999, 2024-09-11T09:50:56.706 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:50:56.707 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:50:56.707 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:56.707 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:50:56.707 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:50:56.707 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:50:56.707 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 116 2024-09-11T09:50:56.707 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.707 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:50:56.707 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.707 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:50:56.707 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711998+0000", 2024-09-11T09:50:56.707 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:56.707 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.707 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.708 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:50:56.708 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711998+0000", 2024-09-11T09:50:56.708 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:56.708 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.708 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.708 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:50:56.708 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711998+0000", 2024-09-11T09:50:56.708 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.3300000000000003e-07 2024-09-11T09:50:56.708 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.708 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.708 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:50:56.708 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712001+0000", 2024-09-11T09:50:56.708 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.9349999999999999e-06 2024-09-11T09:50:56.709 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.709 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.709 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:50:56.709 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712006+0000", 2024-09-11T09:50:56.709 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.1630000000000004e-06 2024-09-11T09:50:56.709 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.709 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.709 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:50:56.709 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712010+0000", 2024-09-11T09:50:56.709 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.574e-06 2024-09-11T09:50:56.709 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.709 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.709 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:50:56.709 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719415+0000", 2024-09-11T09:50:56.710 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0074049099999999998 2024-09-11T09:50:56.710 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.710 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.710 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:50:56.710 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719465+0000", 2024-09-11T09:50:56.710 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.0129000000000001e-05 2024-09-11T09:50:56.710 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.710 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.710 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:50:56.710 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719545+0000", 2024-09-11T09:50:56.710 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.0253999999999994e-05 2024-09-11T09:50:56.710 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.710 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.711 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:50:56.711 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720636+0000", 2024-09-11T09:50:56.711 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0010909330000000001 2024-09-11T09:50:56.711 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.711 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.711 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:50:56.711 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.722707+0000", 2024-09-11T09:50:56.711 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0020713300000000001 2024-09-11T09:50:56.711 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.711 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.711 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:50:56.711 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.722743+0000", 2024-09-11T09:50:56.711 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.5351000000000001e-05 2024-09-11T09:50:56.711 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.712 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.712 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:50:56.712 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.722784+0000", 2024-09-11T09:50:56.712 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.0802e-05 2024-09-11T09:50:56.712 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:56.712 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:50:56.712 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:56.712 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.712 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.712 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:50:56.712 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.712014+0000", 2024-09-11T09:50:56.712 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 44.939350253999997, 2024-09-11T09:50:56.712 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.013420938, 2024-09-11T09:50:56.713 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:50:56.713 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:50:56.713 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:56.713 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:50:56.713 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:50:56.713 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:50:56.713 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 117 2024-09-11T09:50:56.713 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.713 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:50:56.713 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.713 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:50:56.713 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712014+0000", 2024-09-11T09:50:56.714 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:56.714 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.714 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.714 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:50:56.714 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712014+0000", 2024-09-11T09:50:56.714 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:56.714 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.714 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.714 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:50:56.714 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712014+0000", 2024-09-11T09:50:56.714 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.2000000000000001e-07 2024-09-11T09:50:56.714 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.714 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.714 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:50:56.715 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712016+0000", 2024-09-11T09:50:56.715 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.2309999999999999e-06 2024-09-11T09:50:56.715 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.715 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.715 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:50:56.715 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712020+0000", 2024-09-11T09:50:56.715 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.3830000000000001e-06 2024-09-11T09:50:56.715 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.715 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.715 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:50:56.715 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712023+0000", 2024-09-11T09:50:56.715 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.3340000000000002e-06 2024-09-11T09:50:56.715 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.716 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.716 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:50:56.716 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720083+0000", 2024-09-11T09:50:56.716 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0080601189999999993 2024-09-11T09:50:56.716 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.716 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.716 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:50:56.716 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720125+0000", 2024-09-11T09:50:56.716 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.2029e-05 2024-09-11T09:50:56.716 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.716 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.716 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:50:56.716 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720205+0000", 2024-09-11T09:50:56.717 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.0010000000000001e-05 2024-09-11T09:50:56.717 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.717 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.717 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:50:56.717 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720849+0000", 2024-09-11T09:50:56.717 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00064369899999999996 2024-09-11T09:50:56.717 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.717 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.717 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:50:56.717 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.725357+0000", 2024-09-11T09:50:56.717 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0045077379999999998 2024-09-11T09:50:56.717 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.717 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.717 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:50:56.718 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.725392+0000", 2024-09-11T09:50:56.718 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.5862999999999999e-05 2024-09-11T09:50:56.718 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.718 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.718 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:50:56.718 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.725435+0000", 2024-09-11T09:50:56.718 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.2311999999999997e-05 2024-09-11T09:50:56.718 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:56.718 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:50:56.718 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:56.718 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.718 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.718 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:119 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 131072~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:50:56.719 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.725673+0000", 2024-09-11T09:50:56.719 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 44.925691552000004, 2024-09-11T09:50:56.719 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0053927460000000003, 2024-09-11T09:50:56.719 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:50:56.719 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:50:56.719 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:56.719 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:50:56.719 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:50:56.719 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:50:56.719 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 119 2024-09-11T09:50:56.719 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.719 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:50:56.719 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.720 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:50:56.720 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.725673+0000", 2024-09-11T09:50:56.720 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:56.720 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.720 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.720 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:50:56.720 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.725673+0000", 2024-09-11T09:50:56.720 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:56.720 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.720 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.720 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:50:56.720 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.725673+0000", 2024-09-11T09:50:56.720 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.68e-07 2024-09-11T09:50:56.721 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.721 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.721 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:50:56.721 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.725720+0000", 2024-09-11T09:50:56.721 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.6526999999999997e-05 2024-09-11T09:50:56.721 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.721 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.721 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:50:56.721 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.725734+0000", 2024-09-11T09:50:56.721 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.4593000000000001e-05 2024-09-11T09:50:56.722 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.722 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.722 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:50:56.722 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.725741+0000", 2024-09-11T09:50:56.722 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.28e-06 2024-09-11T09:50:56.722 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.722 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.722 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:50:56.722 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.726290+0000", 2024-09-11T09:50:56.722 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00054892000000000005 2024-09-11T09:50:56.722 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.722 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.722 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:50:56.723 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.726338+0000", 2024-09-11T09:50:56.723 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.7982000000000002e-05 2024-09-11T09:50:56.723 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.723 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.723 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:50:56.723 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.726413+0000", 2024-09-11T09:50:56.723 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.5679000000000005e-05 2024-09-11T09:50:56.723 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.723 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.723 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:50:56.723 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.727555+0000", 2024-09-11T09:50:56.723 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0011420289999999999 2024-09-11T09:50:56.724 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.724 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.724 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:50:56.724 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.731004+0000", 2024-09-11T09:50:56.724 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0034484279999999999 2024-09-11T09:50:56.724 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.724 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.724 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:50:56.724 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.731040+0000", 2024-09-11T09:50:56.724 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.6315e-05 2024-09-11T09:50:56.724 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.724 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.725 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:50:56.725 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.731065+0000", 2024-09-11T09:50:56.725 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.5324999999999999e-05 2024-09-11T09:50:56.725 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:56.725 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:50:56.725 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:56.725 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.725 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.725 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:120 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 196608~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:50:56.725 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.725756+0000", 2024-09-11T09:50:56.725 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 44.925607642999999, 2024-09-11T09:50:56.725 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0053569100000000003, 2024-09-11T09:50:56.725 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:50:56.726 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:50:56.726 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:56.726 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:50:56.726 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:50:56.726 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:50:56.726 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 120 2024-09-11T09:50:56.726 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.726 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:50:56.726 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.726 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:50:56.726 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.725756+0000", 2024-09-11T09:50:56.726 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:56.726 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.726 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.727 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:50:56.727 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.725756+0000", 2024-09-11T09:50:56.727 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:56.727 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.727 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.727 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:50:56.727 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.725757+0000", 2024-09-11T09:50:56.727 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.3799999999999997e-07 2024-09-11T09:50:56.727 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.727 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.727 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:50:56.727 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.725802+0000", 2024-09-11T09:50:56.727 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.4790000000000003e-05 2024-09-11T09:50:56.727 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.727 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.728 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:50:56.728 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.725815+0000", 2024-09-11T09:50:56.728 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.2609000000000001e-05 2024-09-11T09:50:56.728 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.728 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.728 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:50:56.728 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.725822+0000", 2024-09-11T09:50:56.728 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.6450000000000005e-06 2024-09-11T09:50:56.728 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.728 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.728 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:50:56.728 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.726708+0000", 2024-09-11T09:50:56.728 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00088606500000000001 2024-09-11T09:50:56.728 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.729 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.729 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:50:56.729 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.726755+0000", 2024-09-11T09:50:56.729 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.6888000000000002e-05 2024-09-11T09:50:56.729 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.729 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.729 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:50:56.729 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.726828+0000", 2024-09-11T09:50:56.729 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.2510999999999997e-05 2024-09-11T09:50:56.729 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.729 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.729 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:50:56.729 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.727829+0000", 2024-09-11T09:50:56.730 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001001093 2024-09-11T09:50:56.730 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.730 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.730 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:50:56.730 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.731071+0000", 2024-09-11T09:50:56.730 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0032426009999999999 2024-09-11T09:50:56.730 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.730 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.730 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:50:56.730 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.731094+0000", 2024-09-11T09:50:56.730 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.2677e-05 2024-09-11T09:50:56.730 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.730 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.731 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:50:56.731 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.731113+0000", 2024-09-11T09:50:56.731 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.9392999999999999e-05 2024-09-11T09:50:56.731 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:56.731 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:50:56.731 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:56.731 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.731 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.731 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:138 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:50:56.731 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:12.034038+0000", 2024-09-11T09:50:56.731 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 44.617326313, 2024-09-11T09:50:56.731 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0054142749999999996, 2024-09-11T09:50:56.731 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:50:56.732 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:50:56.732 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:56.732 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:50:56.732 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:50:56.732 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:50:56.732 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 138 2024-09-11T09:50:56.732 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.732 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:50:56.732 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.732 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:50:56.732 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034038+0000", 2024-09-11T09:50:56.732 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:56.733 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.733 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.733 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:50:56.733 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034038+0000", 2024-09-11T09:50:56.733 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:56.733 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.733 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.733 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:50:56.733 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034038+0000", 2024-09-11T09:50:56.733 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.3499999999999996e-07 2024-09-11T09:50:56.733 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.733 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.733 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:50:56.733 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034043+0000", 2024-09-11T09:50:56.734 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.7310000000000001e-06 2024-09-11T09:50:56.734 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.734 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.734 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:50:56.734 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034050+0000", 2024-09-11T09:50:56.734 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.0419999999999997e-06 2024-09-11T09:50:56.734 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.734 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.734 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:50:56.734 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034055+0000", 2024-09-11T09:50:56.734 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.3299999999999998e-06 2024-09-11T09:50:56.734 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.734 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.735 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:50:56.735 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.037309+0000", 2024-09-11T09:50:56.735 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0032535440000000001 2024-09-11T09:50:56.735 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.735 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.735 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:50:56.735 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.037357+0000", 2024-09-11T09:50:56.735 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.8167999999999997e-05 2024-09-11T09:50:56.735 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.735 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.735 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:50:56.735 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.037430+0000", 2024-09-11T09:50:56.735 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.3275999999999994e-05 2024-09-11T09:50:56.735 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.736 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.736 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:50:56.736 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.039014+0000", 2024-09-11T09:50:56.736 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0015831350000000001 2024-09-11T09:50:56.736 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.736 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.736 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:50:56.736 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.039401+0000", 2024-09-11T09:50:56.736 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00038758500000000002 2024-09-11T09:50:56.736 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.736 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.736 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:50:56.736 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.039429+0000", 2024-09-11T09:50:56.737 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.7379e-05 2024-09-11T09:50:56.737 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.737 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.737 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:50:56.737 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.039452+0000", 2024-09-11T09:50:56.737 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.355e-05 2024-09-11T09:50:56.737 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:56.737 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:50:56.737 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:56.737 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.737 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.737 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:139 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:50:56.737 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:12.034064+0000", 2024-09-11T09:50:56.737 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 44.617300282000002, 2024-09-11T09:50:56.738 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0054736200000000002, 2024-09-11T09:50:56.738 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:50:56.738 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:50:56.738 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:56.738 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:50:56.738 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:50:56.738 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:50:56.738 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 139 2024-09-11T09:50:56.738 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.738 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:50:56.738 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.738 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:50:56.738 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034064+0000", 2024-09-11T09:50:56.738 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:56.739 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.739 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.739 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:50:56.739 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034064+0000", 2024-09-11T09:50:56.739 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:56.739 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.739 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.739 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:50:56.739 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034064+0000", 2024-09-11T09:50:56.739 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.89e-07 2024-09-11T09:50:56.739 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.739 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.739 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:50:56.740 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034069+0000", 2024-09-11T09:50:56.740 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.9799999999999998e-06 2024-09-11T09:50:56.740 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.740 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.740 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:50:56.740 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034079+0000", 2024-09-11T09:50:56.740 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.3549999999999994e-06 2024-09-11T09:50:56.740 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.740 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.740 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:50:56.740 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034084+0000", 2024-09-11T09:50:56.740 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.2209999999999996e-06 2024-09-11T09:50:56.740 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.741 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.741 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:50:56.741 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.038176+0000", 2024-09-11T09:50:56.741 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0040921610000000004 2024-09-11T09:50:56.741 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.741 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.741 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:50:56.741 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.038221+0000", 2024-09-11T09:50:56.741 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.5006000000000002e-05 2024-09-11T09:50:56.741 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.741 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.741 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:50:56.741 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.038293+0000", 2024-09-11T09:50:56.742 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.1952999999999994e-05 2024-09-11T09:50:56.742 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.742 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.742 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:50:56.742 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.039461+0000", 2024-09-11T09:50:56.742 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0011680340000000001 2024-09-11T09:50:56.742 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.742 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.742 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:50:56.742 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.039487+0000", 2024-09-11T09:50:56.742 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.5752e-05 2024-09-11T09:50:56.742 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.742 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.743 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:50:56.743 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.039515+0000", 2024-09-11T09:50:56.743 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.7763000000000002e-05 2024-09-11T09:50:56.743 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.743 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.743 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:50:56.743 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.039537+0000", 2024-09-11T09:50:56.743 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.2906000000000001e-05 2024-09-11T09:50:56.743 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:56.743 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:50:56.743 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:56.743 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.743 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.743 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:140 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:50:56.744 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:12.034093+0000", 2024-09-11T09:50:56.744 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 44.617270777999998, 2024-09-11T09:50:56.744 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0059112490000000004, 2024-09-11T09:50:56.744 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:50:56.744 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:50:56.744 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:56.744 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:50:56.744 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:50:56.744 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:50:56.744 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 140 2024-09-11T09:50:56.744 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.744 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:50:56.744 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.745 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:50:56.745 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034093+0000", 2024-09-11T09:50:56.745 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:56.745 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.745 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.745 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:50:56.745 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034093+0000", 2024-09-11T09:50:56.745 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:56.745 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.745 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.745 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:50:56.745 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034094+0000", 2024-09-11T09:50:56.745 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.4299999999999998e-07 2024-09-11T09:50:56.745 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.746 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.746 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:50:56.746 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034159+0000", 2024-09-11T09:50:56.746 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.567e-05 2024-09-11T09:50:56.746 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.746 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.746 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:50:56.746 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034167+0000", 2024-09-11T09:50:56.746 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.8490000000000006e-06 2024-09-11T09:50:56.746 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.746 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.746 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:50:56.746 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034173+0000", 2024-09-11T09:50:56.747 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.1770000000000003e-06 2024-09-11T09:50:56.747 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.747 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.747 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:50:56.747 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.038543+0000", 2024-09-11T09:50:56.747 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0043699150000000003 2024-09-11T09:50:56.747 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.747 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.747 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:50:56.747 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.038589+0000", 2024-09-11T09:50:56.747 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.5562999999999997e-05 2024-09-11T09:50:56.747 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.747 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.747 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:50:56.748 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.038662+0000", 2024-09-11T09:50:56.748 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.2748000000000002e-05 2024-09-11T09:50:56.748 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.748 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.748 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:50:56.748 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.039635+0000", 2024-09-11T09:50:56.748 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00097310799999999998 2024-09-11T09:50:56.748 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.748 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.748 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:50:56.748 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.039945+0000", 2024-09-11T09:50:56.748 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00031054100000000001 2024-09-11T09:50:56.748 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.749 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.749 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:50:56.749 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.039971+0000", 2024-09-11T09:50:56.749 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.5423999999999999e-05 2024-09-11T09:50:56.749 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.749 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.749 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:50:56.749 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.040005+0000", 2024-09-11T09:50:56.749 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.3810999999999999e-05 2024-09-11T09:50:56.749 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:56.749 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:50:56.749 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:56.749 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.749 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.749 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:50:56.750 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:12.034184+0000", 2024-09-11T09:50:56.750 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 44.617180079999997, 2024-09-11T09:50:56.750 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0061520380000000003, 2024-09-11T09:50:56.750 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:50:56.750 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:50:56.750 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:56.750 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:50:56.750 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:50:56.750 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:50:56.750 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 141 2024-09-11T09:50:56.750 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.750 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:50:56.750 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.750 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:50:56.751 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034184+0000", 2024-09-11T09:50:56.751 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:56.751 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.751 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.751 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:50:56.751 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034184+0000", 2024-09-11T09:50:56.751 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:56.751 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.751 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.751 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:50:56.751 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034185+0000", 2024-09-11T09:50:56.751 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.1099999999999995e-07 2024-09-11T09:50:56.751 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.751 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.752 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:50:56.752 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034190+0000", 2024-09-11T09:50:56.752 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.7309999999999998e-06 2024-09-11T09:50:56.752 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.752 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.752 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:50:56.752 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034197+0000", 2024-09-11T09:50:56.752 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.7440000000000001e-06 2024-09-11T09:50:56.752 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.752 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.752 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:50:56.753 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034204+0000", 2024-09-11T09:50:56.753 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.8970000000000002e-06 2024-09-11T09:50:56.753 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.753 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.753 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:50:56.753 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.039035+0000", 2024-09-11T09:50:56.753 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0048307039999999999 2024-09-11T09:50:56.753 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.753 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.753 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:50:56.753 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.039093+0000", 2024-09-11T09:50:56.753 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.8604000000000003e-05 2024-09-11T09:50:56.754 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.754 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.754 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:50:56.754 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.039184+0000", 2024-09-11T09:50:56.754 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.0196000000000002e-05 2024-09-11T09:50:56.754 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.754 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.754 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:50:56.754 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.039873+0000", 2024-09-11T09:50:56.754 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00068960600000000005 2024-09-11T09:50:56.754 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.754 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.754 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:50:56.755 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.040292+0000", 2024-09-11T09:50:56.755 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00041865199999999999 2024-09-11T09:50:56.755 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.755 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.755 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:50:56.755 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.040317+0000", 2024-09-11T09:50:56.755 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.5439000000000002e-05 2024-09-11T09:50:56.755 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.755 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.755 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:50:56.755 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.040336+0000", 2024-09-11T09:50:56.755 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.8754e-05 2024-09-11T09:50:56.755 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:56.756 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:50:56.756 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:56.756 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.756 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.756 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:157 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:50:56.756 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:12.452040+0000", 2024-09-11T09:50:56.756 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 44.199324226999998, 2024-09-11T09:50:56.756 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0068848269999999996, 2024-09-11T09:50:56.756 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:50:56.756 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:50:56.756 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:56.756 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:50:56.757 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:50:56.757 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:50:56.757 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 157 2024-09-11T09:50:56.757 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.757 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:50:56.757 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.757 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:50:56.757 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452040+0000", 2024-09-11T09:50:56.757 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:56.757 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.757 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.757 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:50:56.758 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452040+0000", 2024-09-11T09:50:56.758 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:56.758 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.758 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.758 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:50:56.758 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452040+0000", 2024-09-11T09:50:56.758 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.22e-07 2024-09-11T09:50:56.758 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.758 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.758 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:50:56.758 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452042+0000", 2024-09-11T09:50:56.758 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.7010000000000001e-06 2024-09-11T09:50:56.758 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.759 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.759 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:50:56.759 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452045+0000", 2024-09-11T09:50:56.759 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.1310000000000002e-06 2024-09-11T09:50:56.759 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.759 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.759 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:50:56.759 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452048+0000", 2024-09-11T09:50:56.759 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.65e-06 2024-09-11T09:50:56.759 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.759 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.759 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:50:56.759 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.454453+0000", 2024-09-11T09:50:56.760 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0024052959999999999 2024-09-11T09:50:56.760 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.760 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.760 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:50:56.760 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.454498+0000", 2024-09-11T09:50:56.760 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.4978999999999997e-05 2024-09-11T09:50:56.760 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.760 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.760 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:50:56.760 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.454567+0000", 2024-09-11T09:50:56.760 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.9110999999999996e-05 2024-09-11T09:50:56.760 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.760 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.761 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:50:56.761 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.455464+0000", 2024-09-11T09:50:56.761 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00089696199999999995 2024-09-11T09:50:56.761 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.761 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.761 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:50:56.761 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.458844+0000", 2024-09-11T09:50:56.761 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0033802469999999999 2024-09-11T09:50:56.761 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.761 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.761 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:50:56.761 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.458889+0000", 2024-09-11T09:50:56.762 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.4441999999999999e-05 2024-09-11T09:50:56.762 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.762 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.762 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:50:56.762 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.458925+0000", 2024-09-11T09:50:56.762 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.6086e-05 2024-09-11T09:50:56.762 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:56.762 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:50:56.762 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:56.762 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.762 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.762 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:158 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:50:56.762 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:12.452055+0000", 2024-09-11T09:50:56.763 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 44.199308834, 2024-09-11T09:50:56.763 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0069533829999999996, 2024-09-11T09:50:56.763 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:50:56.763 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:50:56.763 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:56.763 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:50:56.763 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:50:56.763 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:50:56.763 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 158 2024-09-11T09:50:56.763 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.763 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:50:56.763 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.763 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:50:56.763 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452055+0000", 2024-09-11T09:50:56.764 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:56.764 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.764 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.764 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:50:56.764 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452055+0000", 2024-09-11T09:50:56.764 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:56.764 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.764 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.764 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:50:56.764 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452056+0000", 2024-09-11T09:50:56.764 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.48e-07 2024-09-11T09:50:56.765 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.765 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.765 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:50:56.765 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452087+0000", 2024-09-11T09:50:56.765 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.1149000000000003e-05 2024-09-11T09:50:56.765 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.765 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.765 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:50:56.765 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452092+0000", 2024-09-11T09:50:56.765 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.8400000000000002e-06 2024-09-11T09:50:56.765 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.765 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.766 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:50:56.766 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452095+0000", 2024-09-11T09:50:56.766 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.0740000000000001e-06 2024-09-11T09:50:56.766 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.766 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.766 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:50:56.766 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.454796+0000", 2024-09-11T09:50:56.766 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0027009740000000001 2024-09-11T09:50:56.766 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.766 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.766 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:50:56.766 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.454869+0000", 2024-09-11T09:50:56.767 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.3678999999999997e-05 2024-09-11T09:50:56.767 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.767 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.767 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:50:56.767 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.454992+0000", 2024-09-11T09:50:56.767 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00012279000000000001 2024-09-11T09:50:56.767 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.767 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.767 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:50:56.767 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.456072+0000", 2024-09-11T09:50:56.767 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001080229 2024-09-11T09:50:56.767 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.767 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.767 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:50:56.768 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.458931+0000", 2024-09-11T09:50:56.768 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0028591789999999999 2024-09-11T09:50:56.768 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.768 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.768 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:50:56.768 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.458972+0000", 2024-09-11T09:50:56.768 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.0170999999999999e-05 2024-09-11T09:50:56.768 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.768 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.768 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:50:56.768 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.459009+0000", 2024-09-11T09:50:56.768 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.7049999999999999e-05 2024-09-11T09:50:56.769 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:56.769 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:50:56.769 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:56.769 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.769 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.769 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:159 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:50:56.769 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:12.452099+0000", 2024-09-11T09:50:56.769 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 44.199264665999998, 2024-09-11T09:50:56.769 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0069885950000000002, 2024-09-11T09:50:56.769 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:50:56.769 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:50:56.769 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:56.770 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:50:56.770 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:50:56.770 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:50:56.770 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 159 2024-09-11T09:50:56.770 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.770 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:50:56.770 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.770 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:50:56.770 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452099+0000", 2024-09-11T09:50:56.770 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:56.770 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.770 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.770 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:50:56.771 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452099+0000", 2024-09-11T09:50:56.771 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:56.771 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.771 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.771 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:50:56.771 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452100+0000", 2024-09-11T09:50:56.771 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.2800000000000003e-07 2024-09-11T09:50:56.771 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.771 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.771 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:50:56.771 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452102+0000", 2024-09-11T09:50:56.771 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.824e-06 2024-09-11T09:50:56.772 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.772 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.772 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:50:56.772 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452105+0000", 2024-09-11T09:50:56.772 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.0460000000000001e-06 2024-09-11T09:50:56.772 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.772 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.772 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:50:56.772 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452107+0000", 2024-09-11T09:50:56.772 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.543e-06 2024-09-11T09:50:56.772 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.773 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.773 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:50:56.773 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.455484+0000", 2024-09-11T09:50:56.773 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0033769759999999998 2024-09-11T09:50:56.773 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.773 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.773 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:50:56.773 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.455527+0000", 2024-09-11T09:50:56.773 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.2589999999999997e-05 2024-09-11T09:50:56.773 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.773 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.773 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:50:56.774 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.455594+0000", 2024-09-11T09:50:56.774 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.6775999999999999e-05 2024-09-11T09:50:56.774 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.774 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.774 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:50:56.774 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.456312+0000", 2024-09-11T09:50:56.774 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00071874700000000005 2024-09-11T09:50:56.774 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.774 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.774 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:50:56.774 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.459015+0000", 2024-09-11T09:50:56.774 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0027028540000000002 2024-09-11T09:50:56.774 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.774 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.775 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:50:56.775 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.459054+0000", 2024-09-11T09:50:56.775 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.8840000000000001e-05 2024-09-11T09:50:56.775 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:56.775 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:56.775 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:50:56.775 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.459088+0000", 2024-09-11T09:50:56.775 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.4071e-05 2024-09-11T09:50:56.775 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:56.775 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:50:56.775 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:56.775 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:56.775 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:50:56.775 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-11T09:50:56.777 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-11T09:50:56.779 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-09-11T09:50:56.777+0000 7f7b6c7d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31904) UID: 0 2024-09-11T09:50:56.877 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:50:56.878+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:50:56.921 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-11T09:50:56.921 INFO:teuthology.orchestra.run.smithi096.stdout: "ops": [], 2024-09-11T09:50:56.921 INFO:teuthology.orchestra.run.smithi096.stdout: "num_ops": 0 2024-09-11T09:50:56.921 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-11T09:50:56.929 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-11T09:50:56.978 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:50:56.979+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:50:57.032 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-11T09:50:57.032 INFO:teuthology.orchestra.run.smithi096.stdout: "ops": [], 2024-09-11T09:50:57.032 INFO:teuthology.orchestra.run.smithi096.stdout: "complaint_time": 30, 2024-09-11T09:50:57.032 INFO:teuthology.orchestra.run.smithi096.stdout: "num_blocked_ops": 0 2024-09-11T09:50:57.032 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-11T09:50:57.041 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-11T09:50:57.084 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:50:57.085+0000 7fdec5849640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31907) UID: 0 2024-09-11T09:50:57.144 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-11T09:50:57.144 INFO:teuthology.orchestra.run.smithi096.stdout: "size": 20, 2024-09-11T09:50:57.144 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 600, 2024-09-11T09:50:57.144 INFO:teuthology.orchestra.run.smithi096.stdout: "ops": [] 2024-09-11T09:50:57.145 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-11T09:50:57.153 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-11T09:50:57.180 INFO:tasks.ceph.osd.0.smithi017.stderr:2024-09-11T09:50:57.180+0000 7fb73c0f1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31665) UID: 0 2024-09-11T09:50:57.280 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:50:57.280+0000 7fdec5849640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31907) UID: 0 2024-09-11T09:50:57.297 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-11T09:50:57.297 INFO:teuthology.orchestra.run.smithi096.stdout: "ops": [], 2024-09-11T09:50:57.297 INFO:teuthology.orchestra.run.smithi096.stdout: "num_ops": 0 2024-09-11T09:50:57.298 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-11T09:50:57.306 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-11T09:50:57.391 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-09-11T09:50:57.391+0000 7f7b6c7d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31904) UID: 0 2024-09-11T09:50:57.449 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-11T09:50:57.449 INFO:teuthology.orchestra.run.smithi096.stdout: "ops": [], 2024-09-11T09:50:57.449 INFO:teuthology.orchestra.run.smithi096.stdout: "complaint_time": 30, 2024-09-11T09:50:57.449 INFO:teuthology.orchestra.run.smithi096.stdout: "num_blocked_ops": 0 2024-09-11T09:50:57.449 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-11T09:50:57.458 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-11T09:50:57.499 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:50:57.500+0000 7fdec5849640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31907) UID: 0 2024-09-11T09:50:57.583 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-09-11T09:50:57.584+0000 7f7b6c7d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31904) UID: 0 2024-09-11T09:50:57.601 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-11T09:50:57.601 INFO:teuthology.orchestra.run.smithi096.stdout: "size": 20, 2024-09-11T09:50:57.601 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 600, 2024-09-11T09:50:57.601 INFO:teuthology.orchestra.run.smithi096.stdout: "ops": [ 2024-09-11T09:50:57.601 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:57.601 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4230.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-11T09:50:57.602 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:18.490736+0000", 2024-09-11T09:50:57.602 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 39.111596392000003, 2024-09-11T09:50:57.602 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0059097569999999999, 2024-09-11T09:50:57.602 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:50:57.602 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:50:57.602 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:57.602 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:50:57.602 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4230", 2024-09-11T09:50:57.602 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.17:0/4101109576", 2024-09-11T09:50:57.602 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 1 2024-09-11T09:50:57.602 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:57.602 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:50:57.603 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:57.603 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:50:57.603 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.490736+0000", 2024-09-11T09:50:57.603 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:57.603 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:57.603 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:57.603 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:50:57.603 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.490736+0000", 2024-09-11T09:50:57.603 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:57.603 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:57.603 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:57.603 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:50:57.603 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.490742+0000", 2024-09-11T09:50:57.604 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.48e-06 2024-09-11T09:50:57.604 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:57.604 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:57.604 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:50:57.604 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.490749+0000", 2024-09-11T09:50:57.604 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.4569999999999999e-06 2024-09-11T09:50:57.604 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:57.604 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:57.604 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:50:57.604 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.490762+0000", 2024-09-11T09:50:57.604 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.2721000000000001e-05 2024-09-11T09:50:57.604 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:57.604 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:57.605 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:50:57.605 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.490774+0000", 2024-09-11T09:50:57.605 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1823e-05 2024-09-11T09:50:57.605 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:57.605 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:57.605 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:50:57.605 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.490846+0000", 2024-09-11T09:50:57.605 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.1878000000000005e-05 2024-09-11T09:50:57.605 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:57.605 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:57.605 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for peered", 2024-09-11T09:50:57.605 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.490857+0000", 2024-09-11T09:50:57.605 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1017e-05 2024-09-11T09:50:57.605 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:57.606 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:57.606 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:50:57.606 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.493856+0000", 2024-09-11T09:50:57.606 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00299885 2024-09-11T09:50:57.606 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:57.606 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:57.606 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:50:57.606 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.494020+0000", 2024-09-11T09:50:57.606 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00016436599999999999 2024-09-11T09:50:57.606 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:57.606 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:57.606 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 2", 2024-09-11T09:50:57.606 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.494186+0000", 2024-09-11T09:50:57.606 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00016562799999999999 2024-09-11T09:50:57.606 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:57.607 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:57.607 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:50:57.607 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.495407+0000", 2024-09-11T09:50:57.607 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0012217420000000001 2024-09-11T09:50:57.607 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:57.607 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:57.607 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:50:57.607 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.496579+0000", 2024-09-11T09:50:57.607 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001171844 2024-09-11T09:50:57.607 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:57.607 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:57.607 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:50:57.607 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.496620+0000", 2024-09-11T09:50:57.607 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.0479000000000003e-05 2024-09-11T09:50:57.608 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:57.608 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:57.608 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:50:57.608 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.496646+0000", 2024-09-11T09:50:57.608 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.6472e-05 2024-09-11T09:50:57.608 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:57.608 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:50:57.608 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:57.608 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:57.608 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:57.608 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_repop_reply(client.4230.0:1 2.3 e16/15 ondisk, result = 0)", 2024-09-11T09:50:57.608 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:18.495321+0000", 2024-09-11T09:50:57.608 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 39.107011727, 2024-09-11T09:50:57.608 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.4965999999999998e-05, 2024-09-11T09:50:57.609 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:50:57.609 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:50:57.609 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "started", 2024-09-11T09:50:57.609 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:50:57.609 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:57.609 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:50:57.609 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.495321+0000", 2024-09-11T09:50:57.609 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:57.609 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:57.609 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:57.609 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:50:57.609 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.495321+0000", 2024-09-11T09:50:57.610 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:57.610 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:57.610 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:57.610 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:50:57.610 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.495322+0000", 2024-09-11T09:50:57.610 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.7700000000000004e-07 2024-09-11T09:50:57.610 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:57.610 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:57.610 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:50:57.610 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.495325+0000", 2024-09-11T09:50:57.610 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.7780000000000001e-06 2024-09-11T09:50:57.610 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:57.610 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:57.610 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:50:57.611 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.495338+0000", 2024-09-11T09:50:57.611 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.2315e-05 2024-09-11T09:50:57.611 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:57.611 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:57.611 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:50:57.611 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.495345+0000", 2024-09-11T09:50:57.611 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.8889999999999996e-06 2024-09-11T09:50:57.611 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:57.611 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:57.611 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:50:57.611 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.495392+0000", 2024-09-11T09:50:57.611 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.7524000000000001e-05 2024-09-11T09:50:57.611 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:57.611 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:57.612 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:50:57.612 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.495402+0000", 2024-09-11T09:50:57.612 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.5840000000000001e-06 2024-09-11T09:50:57.612 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:57.612 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:57.612 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:50:57.612 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.495416+0000", 2024-09-11T09:50:57.612 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.4199000000000001e-05 2024-09-11T09:50:57.612 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:57.612 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:50:57.612 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:57.612 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:57.613 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:57.613 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4230.0:2 2.2 2:5cea7035:::rbd_info:head [call rbd.metadata_list in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e16)", 2024-09-11T09:50:57.613 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:18.499576+0000", 2024-09-11T09:50:57.613 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 39.102757220000001, 2024-09-11T09:50:57.613 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00018163000000000001, 2024-09-11T09:50:57.613 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:50:57.613 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:50:57.613 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "reached pg", 2024-09-11T09:50:57.613 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:50:57.613 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4230", 2024-09-11T09:50:57.613 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.17:0/4101109576", 2024-09-11T09:50:57.613 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 2 2024-09-11T09:50:57.613 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:57.613 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:50:57.614 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:57.614 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:50:57.614 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.499576+0000", 2024-09-11T09:50:57.614 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:57.614 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:57.614 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:57.614 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:50:57.614 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.499576+0000", 2024-09-11T09:50:57.614 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:57.614 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:57.614 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:57.614 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:50:57.614 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.499577+0000", 2024-09-11T09:50:57.614 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.124e-06 2024-09-11T09:50:57.614 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:57.615 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:57.615 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:50:57.615 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.499584+0000", 2024-09-11T09:50:57.615 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.1949999999999997e-06 2024-09-11T09:50:57.615 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:57.615 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:57.615 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:50:57.615 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.499597+0000", 2024-09-11T09:50:57.615 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.366e-05 2024-09-11T09:50:57.615 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:57.615 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:57.615 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:50:57.615 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.499604+0000", 2024-09-11T09:50:57.615 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.2709999999999998e-06 2024-09-11T09:50:57.615 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:57.616 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:57.616 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:50:57.616 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.499660+0000", 2024-09-11T09:50:57.616 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.6270999999999997e-05 2024-09-11T09:50:57.616 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:57.616 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:57.616 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:50:57.616 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.499757+0000", 2024-09-11T09:50:57.616 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.7108999999999995e-05 2024-09-11T09:50:57.616 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:57.616 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:50:57.616 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:57.617 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:57.617 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:57.617 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4230.0:3 2.2 2:5cea7035:::rbd_info:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e16)", 2024-09-11T09:50:57.617 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:18.499937+0000", 2024-09-11T09:50:57.617 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 39.102395430999998, 2024-09-11T09:50:57.617 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000143318, 2024-09-11T09:50:57.617 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:50:57.617 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:50:57.617 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "reached pg", 2024-09-11T09:50:57.617 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:50:57.617 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4230", 2024-09-11T09:50:57.617 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.17:0/4101109576", 2024-09-11T09:50:57.617 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 3 2024-09-11T09:50:57.617 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:57.618 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:50:57.618 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:57.618 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:50:57.618 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.499937+0000", 2024-09-11T09:50:57.618 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:57.618 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:57.618 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:57.618 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:50:57.618 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.499937+0000", 2024-09-11T09:50:57.618 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:57.618 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:57.618 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:57.618 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:50:57.618 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.499938+0000", 2024-09-11T09:50:57.619 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.195e-06 2024-09-11T09:50:57.619 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:57.619 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:57.619 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:50:57.619 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.499943+0000", 2024-09-11T09:50:57.619 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.9080000000000003e-06 2024-09-11T09:50:57.619 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:57.619 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:57.619 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:50:57.619 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.499956+0000", 2024-09-11T09:50:57.619 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.2728000000000001e-05 2024-09-11T09:50:57.619 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:57.619 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:57.619 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:50:57.620 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.499964+0000", 2024-09-11T09:50:57.620 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.8220000000000007e-06 2024-09-11T09:50:57.620 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:57.620 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:57.620 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:50:57.620 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.500017+0000", 2024-09-11T09:50:57.620 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.2809000000000002e-05 2024-09-11T09:50:57.620 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:57.620 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:57.620 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:50:57.620 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.500081+0000", 2024-09-11T09:50:57.620 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.3856000000000007e-05 2024-09-11T09:50:57.620 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:57.620 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:50:57.621 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:57.621 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:57.621 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:57.621 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4230.0:5 2.2 2:5cea7035:::rbd_info:head [create,write 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-11T09:50:57.621 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:19.492662+0000", 2024-09-11T09:50:57.621 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 38.109670770999998, 2024-09-11T09:50:57.621 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0043824759999999997, 2024-09-11T09:50:57.621 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:50:57.621 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:50:57.621 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:57.621 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:50:57.621 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4230", 2024-09-11T09:50:57.621 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.17:0/4101109576", 2024-09-11T09:50:57.622 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 5 2024-09-11T09:50:57.622 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:57.622 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:50:57.622 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:57.622 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:50:57.622 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.492662+0000", 2024-09-11T09:50:57.622 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:57.622 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:57.622 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:57.622 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:50:57.622 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.492662+0000", 2024-09-11T09:50:57.622 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:57.622 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:57.623 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:57.623 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:50:57.623 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.492664+0000", 2024-09-11T09:50:57.623 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.3599999999999999e-06 2024-09-11T09:50:57.623 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:57.623 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:57.623 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:50:57.623 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.492675+0000", 2024-09-11T09:50:57.623 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1182000000000001e-05 2024-09-11T09:50:57.623 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:57.623 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:57.623 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:50:57.623 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.492693+0000", 2024-09-11T09:50:57.623 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.7031999999999999e-05 2024-09-11T09:50:57.624 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:57.624 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:57.624 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:50:57.624 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.492702+0000", 2024-09-11T09:50:57.624 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.5540000000000003e-06 2024-09-11T09:50:57.624 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:57.624 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:57.624 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:50:57.624 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.492767+0000", 2024-09-11T09:50:57.624 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.5411000000000001e-05 2024-09-11T09:50:57.624 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:57.624 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:57.624 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op must wait for map", 2024-09-11T09:50:57.624 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.492773+0000", 2024-09-11T09:50:57.625 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.1039999999999998e-06 2024-09-11T09:50:57.625 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:57.625 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:57.625 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:50:57.625 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.494354+0000", 2024-09-11T09:50:57.625 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0015814049999999999 2024-09-11T09:50:57.625 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:57.625 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:57.625 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:50:57.625 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.494476+0000", 2024-09-11T09:50:57.625 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000122089 2024-09-11T09:50:57.625 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:57.625 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:57.625 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 1", 2024-09-11T09:50:57.626 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.494577+0000", 2024-09-11T09:50:57.626 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000100536 2024-09-11T09:50:57.626 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:57.626 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:57.626 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:50:57.626 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.495770+0000", 2024-09-11T09:50:57.626 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0011932469999999999 2024-09-11T09:50:57.626 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:57.626 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:57.626 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:50:57.626 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.496966+0000", 2024-09-11T09:50:57.626 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001196003 2024-09-11T09:50:57.626 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:57.626 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:57.627 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:50:57.627 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.497011+0000", 2024-09-11T09:50:57.627 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.5229999999999999e-05 2024-09-11T09:50:57.627 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:57.627 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:57.627 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:50:57.627 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.497044+0000", 2024-09-11T09:50:57.627 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.3322999999999999e-05 2024-09-11T09:50:57.627 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:57.627 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:50:57.627 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:57.627 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:57.627 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:57.627 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_repop_reply(client.4230.0:5 2.2 e17/15 ondisk, result = 0)", 2024-09-11T09:50:57.628 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:19.495669+0000", 2024-09-11T09:50:57.628 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 38.106663982999997, 2024-09-11T09:50:57.628 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000106431, 2024-09-11T09:50:57.628 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:50:57.628 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:50:57.628 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "started", 2024-09-11T09:50:57.628 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:50:57.628 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:57.628 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:50:57.628 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.495669+0000", 2024-09-11T09:50:57.628 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:57.628 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:57.628 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:57.628 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:50:57.629 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.495669+0000", 2024-09-11T09:50:57.629 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:57.629 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:57.629 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:57.629 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:50:57.629 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.495670+0000", 2024-09-11T09:50:57.629 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.218e-06 2024-09-11T09:50:57.629 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:57.629 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:57.629 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:50:57.629 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.495676+0000", 2024-09-11T09:50:57.629 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.5119999999999997e-06 2024-09-11T09:50:57.629 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:57.629 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:57.629 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:50:57.630 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.495690+0000", 2024-09-11T09:50:57.630 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.3312000000000001e-05 2024-09-11T09:50:57.630 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:57.630 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:57.630 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:50:57.630 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.495706+0000", 2024-09-11T09:50:57.630 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.6560000000000001e-05 2024-09-11T09:50:57.630 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:57.630 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:57.630 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:50:57.630 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.495759+0000", 2024-09-11T09:50:57.630 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.2828000000000003e-05 2024-09-11T09:50:57.630 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:57.631 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:57.631 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:50:57.631 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.495765+0000", 2024-09-11T09:50:57.631 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.7869999999999998e-06 2024-09-11T09:50:57.631 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:57.631 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:57.631 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:50:57.631 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.495775+0000", 2024-09-11T09:50:57.631 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.0214000000000001e-05 2024-09-11T09:50:57.631 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:57.631 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:50:57.631 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:57.631 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:57.632 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:57.632 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4230.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-09-11T09:50:57.632 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:20.498634+0000", 2024-09-11T09:50:57.632 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 37.103698371, 2024-09-11T09:50:57.632 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0050648539999999997, 2024-09-11T09:50:57.632 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:50:57.632 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:50:57.632 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:50:57.632 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:50:57.632 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4230", 2024-09-11T09:50:57.632 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.17:0/4101109576", 2024-09-11T09:50:57.632 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 7 2024-09-11T09:50:57.632 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:57.633 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:50:57.633 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:57.633 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:50:57.633 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.498634+0000", 2024-09-11T09:50:57.633 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:57.633 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:57.633 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:57.633 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:50:57.633 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.498634+0000", 2024-09-11T09:50:57.633 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:57.633 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:57.633 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:57.633 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:50:57.633 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.498637+0000", 2024-09-11T09:50:57.634 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.1919999999999999e-06 2024-09-11T09:50:57.634 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:57.634 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:57.634 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:50:57.634 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.498648+0000", 2024-09-11T09:50:57.634 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1116e-05 2024-09-11T09:50:57.634 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:57.634 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:57.634 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:50:57.634 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.498666+0000", 2024-09-11T09:50:57.634 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.8411999999999999e-05 2024-09-11T09:50:57.634 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:57.634 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:57.634 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:50:57.634 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.498675+0000", 2024-09-11T09:50:57.635 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.4059999999999994e-06 2024-09-11T09:50:57.635 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:57.635 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:57.635 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:50:57.635 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.498745+0000", 2024-09-11T09:50:57.635 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.9764000000000005e-05 2024-09-11T09:50:57.635 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:57.635 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:57.635 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op must wait for map", 2024-09-11T09:50:57.635 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.498754+0000", 2024-09-11T09:50:57.635 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.5969999999999992e-06 2024-09-11T09:50:57.635 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:57.635 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:57.636 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:50:57.636 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.500385+0000", 2024-09-11T09:50:57.636 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0016308360000000001 2024-09-11T09:50:57.636 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:57.636 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:57.636 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:50:57.636 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.500473+0000", 2024-09-11T09:50:57.636 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.7966000000000004e-05 2024-09-11T09:50:57.636 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:57.636 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:57.637 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 1", 2024-09-11T09:50:57.637 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.500591+0000", 2024-09-11T09:50:57.637 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000117895 2024-09-11T09:50:57.637 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:57.637 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:57.637 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:50:57.637 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.501685+0000", 2024-09-11T09:50:57.637 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001094886 2024-09-11T09:50:57.637 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:57.637 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:57.637 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:50:57.637 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.503627+0000", 2024-09-11T09:50:57.637 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001941199 2024-09-11T09:50:57.638 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:57.638 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:57.638 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:50:57.638 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.503667+0000", 2024-09-11T09:50:57.638 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.0645000000000001e-05 2024-09-11T09:50:57.638 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:57.638 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:57.638 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:50:57.638 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.503699+0000", 2024-09-11T09:50:57.638 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.1940000000000003e-05 2024-09-11T09:50:57.638 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:57.638 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:50:57.638 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:57.639 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:57.639 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:57.639 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_repop_reply(client.4230.0:7 2.2 e18/15 ondisk, result = 0)", 2024-09-11T09:50:57.639 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:20.501608+0000", 2024-09-11T09:50:57.639 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 37.100724849000002, 2024-09-11T09:50:57.639 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.2459000000000005e-05, 2024-09-11T09:50:57.639 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:50:57.639 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:50:57.639 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "started", 2024-09-11T09:50:57.639 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:50:57.639 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:57.639 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:50:57.639 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.501608+0000", 2024-09-11T09:50:57.640 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:57.640 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:57.640 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:57.640 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:50:57.640 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.501608+0000", 2024-09-11T09:50:57.640 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:50:57.640 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:57.640 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:57.640 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:50:57.640 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.501609+0000", 2024-09-11T09:50:57.640 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.0160000000000001e-06 2024-09-11T09:50:57.640 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:57.640 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:57.640 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:50:57.641 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.501613+0000", 2024-09-11T09:50:57.641 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.5959999999999997e-06 2024-09-11T09:50:57.641 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:57.641 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:57.641 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:50:57.641 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.501625+0000", 2024-09-11T09:50:57.641 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1888e-05 2024-09-11T09:50:57.641 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:57.641 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:57.641 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:50:57.641 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.501634+0000", 2024-09-11T09:50:57.641 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.6139999999999992e-06 2024-09-11T09:50:57.642 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:57.642 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:57.642 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:50:57.642 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.501676+0000", 2024-09-11T09:50:57.642 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.1953999999999998e-05 2024-09-11T09:50:57.642 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:57.642 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:57.642 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:50:57.642 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.501682+0000", 2024-09-11T09:50:57.642 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.8000000000000004e-06 2024-09-11T09:50:57.642 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:50:57.642 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:50:57.642 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:50:57.643 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.501690+0000", 2024-09-11T09:50:57.643 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.5909999999999996e-06 2024-09-11T09:50:57.643 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:57.643 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:50:57.643 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:57.643 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:50:57.643 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:50:57.643 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-11T09:50:57.643 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 5, 0] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5] 2024-09-11T09:50:57.643 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-11T09:50:57.643 INFO:tasks.thrashosds.thrasher:inject_pause on osd.0 2024-09-11T09:50:57.644 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-11T09:50:57.644 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-11T09:50:57.644 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok config set filestore_inject_stall 3 2024-09-11T09:50:57.645 INFO:teuthology.orchestra.run.smithi017.stderr:noscrub is unset 2024-09-11T09:50:57.646 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-11T09:50:57.689 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:50:57.690+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:50:57.733 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T09:50:57.757 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:50:57.757 INFO:teuthology.orchestra.run.smithi017.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-11T09:50:57.757 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:50:57.784 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-09-11T09:50:57.786+0000 7f7b6c7d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31904) UID: 0 2024-09-11T09:50:57.798 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:50:57.798 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [], 2024-09-11T09:50:57.798 INFO:teuthology.orchestra.run.smithi017.stdout: "num_ops": 0 2024-09-11T09:50:57.798 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:50:57.808 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-11T09:50:57.857 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-11T09:50:57.858 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T09:50:57.877 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-11T09:50:57.877 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T09:50:57.885 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:50:57.886+0000 7f3c96ec9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31909) UID: 0 2024-09-11T09:50:57.892 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-11T09:50:57.892 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T09:50:57.909 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-11T09:50:57.909 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T09:50:57.926 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-11T09:50:57.926 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T09:50:57.943 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-11T09:50:57.943 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T09:50:57.958 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:50:57.959 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:50:57.968 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-11T09:50:58.009 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:50:58.010+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:50:58.086 INFO:tasks.ceph.osd.2.smithi017.stderr:2024-09-11T09:50:58.087+0000 7fd20b170640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31672) UID: 0 2024-09-11T09:50:58.115 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:50:58.115 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:50:58.124 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-11T09:50:58.187 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:50:58.188+0000 7fdec5849640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31907) UID: 0 2024-09-11T09:50:58.271 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:50:58.271 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:50:58.280 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-11T09:50:58.287 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:50:58.289+0000 7fdec5849640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31907) UID: 0 2024-09-11T09:50:58.387 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:50:58.387 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:50:58.388 INFO:tasks.ceph.osd.2.smithi017.stderr:2024-09-11T09:50:58.389+0000 7fd20b170640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31672) UID: 0 2024-09-11T09:50:58.396 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-11T09:50:58.489 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:50:58.490+0000 7f3c96ec9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31909) UID: 0 2024-09-11T09:50:58.544 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:50:58.544 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:50:58.553 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-11T09:50:58.597 INFO:tasks.ceph.osd.0.smithi017.stderr:2024-09-11T09:50:58.598+0000 7fb73c0f1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31665) UID: 0 2024-09-11T09:50:58.661 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:50:58.661 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:50:58.669 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-11T09:50:58.691 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:50:58.691+0000 7fdec5849640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31907) UID: 0 2024-09-11T09:50:58.792 INFO:tasks.ceph.osd.0.smithi017.stderr:2024-09-11T09:50:58.793+0000 7fb73c0f1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31665) UID: 0 2024-09-11T09:50:58.817 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:50:58.818 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:50:58.826 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-11T09:50:58.893 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:50:58.894+0000 7fdec5849640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31907) UID: 0 2024-09-11T09:50:58.974 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:50:58.974 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:50:58.982 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-11T09:50:58.994 INFO:tasks.ceph.osd.0.smithi017.stderr:2024-09-11T09:50:58.995+0000 7fb73c0f1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31665) UID: 0 2024-09-11T09:50:59.095 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:50:59.094+0000 7fdec5849640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31907) UID: 0 2024-09-11T09:50:59.097 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:50:59.098 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:50:59.107 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-11T09:50:59.196 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:50:59.196+0000 7fdec5849640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31907) UID: 0 2024-09-11T09:50:59.252 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:50:59.252 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:50:59.261 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-11T09:50:59.297 INFO:tasks.ceph.osd.0.smithi017.stderr:2024-09-11T09:50:59.298+0000 7fb73c0f1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31665) UID: 0 2024-09-11T09:50:59.398 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:50:59.397+0000 7fdec5849640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31907) UID: 0 2024-09-11T09:50:59.404 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:50:59.404 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:50:59.413 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-11T09:50:59.498 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:50:59.498+0000 7f3c96ec9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31909) UID: 0 2024-09-11T09:50:59.558 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:50:59.558 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:50:59.567 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-11T09:50:59.599 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:50:59.600+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:50:59.648 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-11T09:50:59.700 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:50:59.700+0000 7fdec5849640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31907) UID: 0 2024-09-11T09:50:59.709 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:50:59.710 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:50:59.718 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-11T09:50:59.802 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:50:59.802+0000 7fdec5849640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31907) UID: 0 2024-09-11T09:50:59.860 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:50:59.860 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:50:59.870 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-11T09:50:59.911 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:50:59.912+0000 7f3c96ec9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31909) UID: 0 2024-09-11T09:50:59.955 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T09:51:00.002 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:51:00.003+0000 7fdec5849640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31907) UID: 0 2024-09-11T09:51:00.015 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:51:00.015 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:51:00.023 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-11T09:51:00.077 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-11T09:51:00.078 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T09:51:00.095 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-11T09:51:00.095 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T09:51:00.103 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:51:00.104+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:51:00.110 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-11T09:51:00.111 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T09:51:00.127 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-11T09:51:00.127 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T09:51:00.144 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-11T09:51:00.144 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T09:51:00.160 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-11T09:51:00.161 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T09:51:00.167 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-11T09:51:00.167 INFO:teuthology.orchestra.run.smithi096.stdout: "ops": [], 2024-09-11T09:51:00.167 INFO:teuthology.orchestra.run.smithi096.stdout: "complaint_time": 30, 2024-09-11T09:51:00.167 INFO:teuthology.orchestra.run.smithi096.stdout: "num_blocked_ops": 0 2024-09-11T09:51:00.167 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-11T09:51:00.176 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-11T09:51:00.220 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:51:00.221+0000 7f3c96ec9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31909) UID: 0 2024-09-11T09:51:00.280 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-11T09:51:00.280 INFO:teuthology.orchestra.run.smithi096.stdout: "size": 20, 2024-09-11T09:51:00.280 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 600, 2024-09-11T09:51:00.280 INFO:teuthology.orchestra.run.smithi096.stdout: "ops": [ 2024-09-11T09:51:00.280 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:00.280 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4230.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-11T09:51:00.280 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:18.490736+0000", 2024-09-11T09:51:00.280 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 41.790170547000002, 2024-09-11T09:51:00.280 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0059097569999999999, 2024-09-11T09:51:00.281 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:00.281 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:00.281 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:00.281 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:00.281 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4230", 2024-09-11T09:51:00.281 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.17:0/4101109576", 2024-09-11T09:51:00.281 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 1 2024-09-11T09:51:00.281 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:00.281 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:00.281 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:00.281 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:00.281 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.490736+0000", 2024-09-11T09:51:00.281 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:00.282 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:00.282 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:00.282 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:00.282 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.490736+0000", 2024-09-11T09:51:00.282 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:00.282 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:00.282 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:00.282 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:00.282 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.490742+0000", 2024-09-11T09:51:00.282 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.48e-06 2024-09-11T09:51:00.282 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:00.282 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:00.282 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:00.282 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.490749+0000", 2024-09-11T09:51:00.283 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.4569999999999999e-06 2024-09-11T09:51:00.283 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:00.283 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:00.283 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:00.283 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.490762+0000", 2024-09-11T09:51:00.283 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.2721000000000001e-05 2024-09-11T09:51:00.283 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:00.283 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:00.283 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:00.283 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.490774+0000", 2024-09-11T09:51:00.283 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1823e-05 2024-09-11T09:51:00.284 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:00.284 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:00.284 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:00.284 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.490846+0000", 2024-09-11T09:51:00.284 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.1878000000000005e-05 2024-09-11T09:51:00.284 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:00.284 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:00.284 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for peered", 2024-09-11T09:51:00.284 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.490857+0000", 2024-09-11T09:51:00.284 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1017e-05 2024-09-11T09:51:00.284 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:00.284 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:00.284 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:00.285 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.493856+0000", 2024-09-11T09:51:00.285 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00299885 2024-09-11T09:51:00.285 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:00.285 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:00.285 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:00.285 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.494020+0000", 2024-09-11T09:51:00.285 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00016436599999999999 2024-09-11T09:51:00.285 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:00.285 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:00.285 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 2", 2024-09-11T09:51:00.285 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.494186+0000", 2024-09-11T09:51:00.285 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00016562799999999999 2024-09-11T09:51:00.285 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:00.286 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:00.286 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:00.286 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.495407+0000", 2024-09-11T09:51:00.286 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0012217420000000001 2024-09-11T09:51:00.286 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:00.286 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:00.286 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:00.286 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.496579+0000", 2024-09-11T09:51:00.286 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001171844 2024-09-11T09:51:00.286 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:00.286 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:00.286 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:00.286 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.496620+0000", 2024-09-11T09:51:00.287 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.0479000000000003e-05 2024-09-11T09:51:00.287 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:00.287 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:00.287 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:00.287 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.496646+0000", 2024-09-11T09:51:00.287 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.6472e-05 2024-09-11T09:51:00.287 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:00.287 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:00.287 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:00.287 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:00.288 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:00.288 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_repop_reply(client.4230.0:1 2.3 e16/15 ondisk, result = 0)", 2024-09-11T09:51:00.288 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:18.495321+0000", 2024-09-11T09:51:00.288 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 41.785585881999999, 2024-09-11T09:51:00.288 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.4965999999999998e-05, 2024-09-11T09:51:00.288 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:00.288 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:00.288 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "started", 2024-09-11T09:51:00.288 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:00.288 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:00.289 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:00.289 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.495321+0000", 2024-09-11T09:51:00.289 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:00.289 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:00.289 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:00.289 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:00.289 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.495321+0000", 2024-09-11T09:51:00.289 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:00.289 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:00.289 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:00.289 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:00.289 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.495322+0000", 2024-09-11T09:51:00.289 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.7700000000000004e-07 2024-09-11T09:51:00.290 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:00.290 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:00.290 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:00.290 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.495325+0000", 2024-09-11T09:51:00.290 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.7780000000000001e-06 2024-09-11T09:51:00.290 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:00.290 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:00.290 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:00.290 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.495338+0000", 2024-09-11T09:51:00.290 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.2315e-05 2024-09-11T09:51:00.290 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:00.290 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:00.290 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:00.290 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.495345+0000", 2024-09-11T09:51:00.291 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.8889999999999996e-06 2024-09-11T09:51:00.291 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:00.291 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:00.291 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:00.291 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.495392+0000", 2024-09-11T09:51:00.291 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.7524000000000001e-05 2024-09-11T09:51:00.291 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:00.291 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:00.291 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:00.291 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.495402+0000", 2024-09-11T09:51:00.291 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.5840000000000001e-06 2024-09-11T09:51:00.291 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:00.291 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:00.292 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:00.292 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.495416+0000", 2024-09-11T09:51:00.292 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.4199000000000001e-05 2024-09-11T09:51:00.292 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:00.292 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:00.292 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:00.292 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:00.292 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:00.292 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4230.0:2 2.2 2:5cea7035:::rbd_info:head [call rbd.metadata_list in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e16)", 2024-09-11T09:51:00.292 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:18.499576+0000", 2024-09-11T09:51:00.292 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 41.781331375000001, 2024-09-11T09:51:00.292 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00018163000000000001, 2024-09-11T09:51:00.292 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:00.293 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:00.293 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "reached pg", 2024-09-11T09:51:00.293 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:00.293 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4230", 2024-09-11T09:51:00.293 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.17:0/4101109576", 2024-09-11T09:51:00.293 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 2 2024-09-11T09:51:00.293 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:00.293 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:00.293 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:00.293 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:00.293 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.499576+0000", 2024-09-11T09:51:00.293 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:00.293 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:00.293 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:00.294 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:00.294 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.499576+0000", 2024-09-11T09:51:00.294 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:00.294 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:00.294 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:00.294 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:00.294 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.499577+0000", 2024-09-11T09:51:00.294 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.124e-06 2024-09-11T09:51:00.294 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:00.294 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:00.294 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:00.295 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.499584+0000", 2024-09-11T09:51:00.295 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.1949999999999997e-06 2024-09-11T09:51:00.295 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:00.295 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:00.295 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:00.296 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.499597+0000", 2024-09-11T09:51:00.296 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.366e-05 2024-09-11T09:51:00.296 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:00.296 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:00.296 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:00.296 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.499604+0000", 2024-09-11T09:51:00.296 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.2709999999999998e-06 2024-09-11T09:51:00.296 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:00.296 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:00.296 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:00.297 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.499660+0000", 2024-09-11T09:51:00.297 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.6270999999999997e-05 2024-09-11T09:51:00.297 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:00.297 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:00.297 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:00.297 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.499757+0000", 2024-09-11T09:51:00.297 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.7108999999999995e-05 2024-09-11T09:51:00.297 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:00.297 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:00.297 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:00.297 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:00.297 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:00.297 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4230.0:3 2.2 2:5cea7035:::rbd_info:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e16)", 2024-09-11T09:51:00.298 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:18.499937+0000", 2024-09-11T09:51:00.298 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 41.780969585999998, 2024-09-11T09:51:00.298 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000143318, 2024-09-11T09:51:00.298 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:00.298 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:00.298 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "reached pg", 2024-09-11T09:51:00.298 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:00.298 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4230", 2024-09-11T09:51:00.298 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.17:0/4101109576", 2024-09-11T09:51:00.298 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 3 2024-09-11T09:51:00.299 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:00.299 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:00.299 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:00.299 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:00.299 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.499937+0000", 2024-09-11T09:51:00.299 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:00.299 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:00.299 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:00.299 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:00.299 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.499937+0000", 2024-09-11T09:51:00.299 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:00.299 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:00.299 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:00.299 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:00.300 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.499938+0000", 2024-09-11T09:51:00.300 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.195e-06 2024-09-11T09:51:00.300 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:00.300 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:00.300 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:00.300 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.499943+0000", 2024-09-11T09:51:00.300 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.9080000000000003e-06 2024-09-11T09:51:00.300 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:00.300 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:00.300 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:00.300 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.499956+0000", 2024-09-11T09:51:00.300 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.2728000000000001e-05 2024-09-11T09:51:00.301 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:00.301 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:00.301 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:00.301 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.499964+0000", 2024-09-11T09:51:00.301 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.8220000000000007e-06 2024-09-11T09:51:00.301 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:00.301 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:00.301 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:00.301 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.500017+0000", 2024-09-11T09:51:00.301 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.2809000000000002e-05 2024-09-11T09:51:00.301 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:00.301 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:00.301 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:00.302 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.500081+0000", 2024-09-11T09:51:00.302 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.3856000000000007e-05 2024-09-11T09:51:00.302 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:00.302 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:00.302 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:00.302 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:00.302 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:00.302 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4230.0:5 2.2 2:5cea7035:::rbd_info:head [create,write 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-11T09:51:00.302 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:19.492662+0000", 2024-09-11T09:51:00.302 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 40.788244925999997, 2024-09-11T09:51:00.302 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0043824759999999997, 2024-09-11T09:51:00.302 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:00.302 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:00.302 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:00.303 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:00.303 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4230", 2024-09-11T09:51:00.303 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.17:0/4101109576", 2024-09-11T09:51:00.303 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 5 2024-09-11T09:51:00.303 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:00.303 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:00.303 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:00.303 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:00.303 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.492662+0000", 2024-09-11T09:51:00.303 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:00.303 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:00.303 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:00.303 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:00.303 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.492662+0000", 2024-09-11T09:51:00.304 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:00.304 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:00.304 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:00.304 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:00.304 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.492664+0000", 2024-09-11T09:51:00.304 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.3599999999999999e-06 2024-09-11T09:51:00.304 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:00.304 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:00.304 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:00.304 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.492675+0000", 2024-09-11T09:51:00.304 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1182000000000001e-05 2024-09-11T09:51:00.304 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:00.304 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:00.305 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:00.305 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.492693+0000", 2024-09-11T09:51:00.305 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.7031999999999999e-05 2024-09-11T09:51:00.305 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:00.305 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:00.305 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:00.305 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.492702+0000", 2024-09-11T09:51:00.305 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.5540000000000003e-06 2024-09-11T09:51:00.305 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:00.305 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:00.305 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:00.305 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.492767+0000", 2024-09-11T09:51:00.305 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.5411000000000001e-05 2024-09-11T09:51:00.306 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:00.306 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:00.306 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op must wait for map", 2024-09-11T09:51:00.306 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.492773+0000", 2024-09-11T09:51:00.306 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.1039999999999998e-06 2024-09-11T09:51:00.306 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:00.306 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:00.306 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:00.306 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.494354+0000", 2024-09-11T09:51:00.306 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0015814049999999999 2024-09-11T09:51:00.306 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:00.306 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:00.306 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:00.306 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.494476+0000", 2024-09-11T09:51:00.307 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000122089 2024-09-11T09:51:00.307 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:00.307 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:00.307 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 1", 2024-09-11T09:51:00.307 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.494577+0000", 2024-09-11T09:51:00.307 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000100536 2024-09-11T09:51:00.307 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:00.307 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:00.307 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:00.308 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.495770+0000", 2024-09-11T09:51:00.308 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0011932469999999999 2024-09-11T09:51:00.308 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:00.308 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:00.308 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:00.308 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.496966+0000", 2024-09-11T09:51:00.308 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001196003 2024-09-11T09:51:00.308 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:00.308 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:00.308 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:00.308 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.497011+0000", 2024-09-11T09:51:00.308 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.5229999999999999e-05 2024-09-11T09:51:00.308 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:00.308 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:00.309 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:00.309 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.497044+0000", 2024-09-11T09:51:00.309 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.3322999999999999e-05 2024-09-11T09:51:00.309 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:00.309 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:00.309 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:00.309 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:00.309 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:00.309 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_repop_reply(client.4230.0:5 2.2 e17/15 ondisk, result = 0)", 2024-09-11T09:51:00.309 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:19.495669+0000", 2024-09-11T09:51:00.309 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 40.785238137999997, 2024-09-11T09:51:00.309 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000106431, 2024-09-11T09:51:00.309 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:00.309 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:00.310 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "started", 2024-09-11T09:51:00.310 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:00.310 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:00.310 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:00.310 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.495669+0000", 2024-09-11T09:51:00.310 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:00.310 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:00.310 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:00.310 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:00.310 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.495669+0000", 2024-09-11T09:51:00.310 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:00.310 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:00.310 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:00.311 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:00.311 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.495670+0000", 2024-09-11T09:51:00.311 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.218e-06 2024-09-11T09:51:00.311 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:00.311 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:00.311 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:00.311 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.495676+0000", 2024-09-11T09:51:00.311 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.5119999999999997e-06 2024-09-11T09:51:00.311 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:00.311 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:00.311 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:00.311 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.495690+0000", 2024-09-11T09:51:00.311 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.3312000000000001e-05 2024-09-11T09:51:00.311 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:00.311 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:00.312 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:00.312 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.495706+0000", 2024-09-11T09:51:00.312 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.6560000000000001e-05 2024-09-11T09:51:00.312 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:00.312 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:00.312 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:00.312 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.495759+0000", 2024-09-11T09:51:00.312 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.2828000000000003e-05 2024-09-11T09:51:00.312 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:00.313 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:00.313 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:00.313 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.495765+0000", 2024-09-11T09:51:00.313 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.7869999999999998e-06 2024-09-11T09:51:00.313 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:00.313 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:00.313 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:00.313 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.495775+0000", 2024-09-11T09:51:00.313 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.0214000000000001e-05 2024-09-11T09:51:00.313 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:00.313 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:00.313 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:00.313 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:00.313 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:00.314 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4230.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-09-11T09:51:00.314 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:20.498634+0000", 2024-09-11T09:51:00.314 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 39.782272526, 2024-09-11T09:51:00.314 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0050648539999999997, 2024-09-11T09:51:00.314 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:00.314 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:00.314 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:00.314 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:00.314 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4230", 2024-09-11T09:51:00.314 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.17:0/4101109576", 2024-09-11T09:51:00.314 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 7 2024-09-11T09:51:00.314 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:00.314 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:00.314 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:00.315 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:00.315 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.498634+0000", 2024-09-11T09:51:00.315 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:00.315 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:00.315 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:00.315 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:00.315 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.498634+0000", 2024-09-11T09:51:00.315 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:00.315 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:00.315 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:00.315 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:00.315 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.498637+0000", 2024-09-11T09:51:00.315 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.1919999999999999e-06 2024-09-11T09:51:00.316 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:00.316 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:00.316 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:00.316 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.498648+0000", 2024-09-11T09:51:00.316 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1116e-05 2024-09-11T09:51:00.316 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:00.316 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:00.316 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:00.316 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.498666+0000", 2024-09-11T09:51:00.316 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.8411999999999999e-05 2024-09-11T09:51:00.316 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:00.317 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:00.317 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:00.317 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.498675+0000", 2024-09-11T09:51:00.317 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.4059999999999994e-06 2024-09-11T09:51:00.317 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:00.317 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:00.317 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:00.317 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.498745+0000", 2024-09-11T09:51:00.317 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.9764000000000005e-05 2024-09-11T09:51:00.318 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:00.318 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:00.318 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op must wait for map", 2024-09-11T09:51:00.318 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.498754+0000", 2024-09-11T09:51:00.318 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.5969999999999992e-06 2024-09-11T09:51:00.318 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:00.318 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:00.318 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:00.318 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.500385+0000", 2024-09-11T09:51:00.318 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0016308360000000001 2024-09-11T09:51:00.318 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:00.318 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:00.318 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:00.318 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.500473+0000", 2024-09-11T09:51:00.319 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.7966000000000004e-05 2024-09-11T09:51:00.319 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:00.319 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:00.319 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 1", 2024-09-11T09:51:00.319 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.500591+0000", 2024-09-11T09:51:00.319 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000117895 2024-09-11T09:51:00.319 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:00.319 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:00.319 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:00.319 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.501685+0000", 2024-09-11T09:51:00.319 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001094886 2024-09-11T09:51:00.319 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:00.319 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:00.319 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:00.320 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.503627+0000", 2024-09-11T09:51:00.320 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001941199 2024-09-11T09:51:00.320 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:00.320 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:00.320 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:00.320 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.503667+0000", 2024-09-11T09:51:00.320 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.0645000000000001e-05 2024-09-11T09:51:00.320 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:00.320 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:00.320 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:00.320 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.503699+0000", 2024-09-11T09:51:00.320 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.1940000000000003e-05 2024-09-11T09:51:00.320 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:00.321 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:00.321 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:00.321 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:00.321 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:00.321 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_repop_reply(client.4230.0:7 2.2 e18/15 ondisk, result = 0)", 2024-09-11T09:51:00.321 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:20.501608+0000", 2024-09-11T09:51:00.321 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 39.779299004000002, 2024-09-11T09:51:00.321 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.2459000000000005e-05, 2024-09-11T09:51:00.321 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:00.321 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:00.321 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "started", 2024-09-11T09:51:00.321 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:00.321 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:00.321 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:00.322 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.501608+0000", 2024-09-11T09:51:00.322 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:00.322 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:00.322 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:00.322 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:00.322 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.501608+0000", 2024-09-11T09:51:00.322 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:00.322 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:00.322 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:00.322 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:00.322 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.501609+0000", 2024-09-11T09:51:00.322 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.0160000000000001e-06 2024-09-11T09:51:00.322 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:00.323 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:00.323 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:00.323 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.501613+0000", 2024-09-11T09:51:00.323 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.5959999999999997e-06 2024-09-11T09:51:00.323 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:00.323 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:00.323 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:00.323 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.501625+0000", 2024-09-11T09:51:00.323 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1888e-05 2024-09-11T09:51:00.323 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:00.323 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:00.323 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:00.323 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.501634+0000", 2024-09-11T09:51:00.323 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.6139999999999992e-06 2024-09-11T09:51:00.324 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:00.324 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:00.324 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:00.324 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.501676+0000", 2024-09-11T09:51:00.324 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.1953999999999998e-05 2024-09-11T09:51:00.324 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:00.324 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:00.324 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:00.324 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.501682+0000", 2024-09-11T09:51:00.324 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.8000000000000004e-06 2024-09-11T09:51:00.324 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:00.324 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:00.324 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:00.324 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.501690+0000", 2024-09-11T09:51:00.325 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.5909999999999996e-06 2024-09-11T09:51:00.325 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:00.325 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:00.325 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:00.325 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:00.325 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:00.325 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-11T09:51:00.326 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-11T09:51:00.327 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:51:00.327+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:51:00.427 INFO:tasks.ceph.osd.2.smithi017.stderr:2024-09-11T09:51:00.428+0000 7fd20b170640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31672) UID: 0 2024-09-11T09:51:00.434 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:51:00.434 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [], 2024-09-11T09:51:00.434 INFO:teuthology.orchestra.run.smithi017.stdout: "num_ops": 0 2024-09-11T09:51:00.434 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:51:00.443 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-11T09:51:00.528 INFO:tasks.ceph.osd.2.smithi017.stderr:2024-09-11T09:51:00.529+0000 7fd20b170640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31672) UID: 0 2024-09-11T09:51:00.591 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:51:00.591 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [], 2024-09-11T09:51:00.591 INFO:teuthology.orchestra.run.smithi017.stdout: "complaint_time": 30, 2024-09-11T09:51:00.591 INFO:teuthology.orchestra.run.smithi017.stdout: "num_blocked_ops": 0 2024-09-11T09:51:00.591 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:51:00.601 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-11T09:51:00.629 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-09-11T09:51:00.630+0000 7f7b6c7d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31904) UID: 0 2024-09-11T09:51:00.639 INFO:teuthology.orchestra.run.smithi017.stderr:nodeep-scrub is unset 2024-09-11T09:51:00.730 INFO:tasks.ceph.osd.2.smithi017.stderr:2024-09-11T09:51:00.731+0000 7fd20b170640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31672) UID: 0 2024-09-11T09:51:00.747 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:51:00.747 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 20, 2024-09-11T09:51:00.747 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 600, 2024-09-11T09:51:00.747 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [ 2024-09-11T09:51:00.747 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.747 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4183.0:21 1.0 e13/12 1:30306672:devicehealth::main.db.0000000000000000:head v 13'11, mlcod=13'8)", 2024-09-11T09:51:00.748 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:11.491728+0000", 2024-09-11T09:51:00.748 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 49.255613093000001, 2024-09-11T09:51:00.748 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00093269800000000003, 2024-09-11T09:51:00.748 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:00.748 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:00.748 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:00.748 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:00.748 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.748 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:00.748 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.491728+0000", 2024-09-11T09:51:00.748 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:00.748 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:00.748 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.749 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:00.749 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.491728+0000", 2024-09-11T09:51:00.749 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:00.749 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:00.749 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.749 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:00.749 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.491729+0000", 2024-09-11T09:51:00.749 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.8599999999999997e-07 2024-09-11T09:51:00.749 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:00.749 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.749 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:00.749 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.491820+0000", 2024-09-11T09:51:00.749 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.0812999999999999e-05 2024-09-11T09:51:00.749 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:00.750 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.750 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:00.750 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.491833+0000", 2024-09-11T09:51:00.750 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.341e-05 2024-09-11T09:51:00.750 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:00.750 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.750 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:00.750 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.491841+0000", 2024-09-11T09:51:00.750 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.8730000000000007e-06 2024-09-11T09:51:00.750 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:00.750 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.750 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:00.750 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.491916+0000", 2024-09-11T09:51:00.750 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.4881000000000004e-05 2024-09-11T09:51:00.751 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:00.751 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.751 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:00.751 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.491931+0000", 2024-09-11T09:51:00.751 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.4825e-05 2024-09-11T09:51:00.751 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:00.751 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.751 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:00.751 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.492633+0000", 2024-09-11T09:51:00.752 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00070191600000000002 2024-09-11T09:51:00.752 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:00.752 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.752 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:00.752 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.492661+0000", 2024-09-11T09:51:00.752 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.8093999999999999e-05 2024-09-11T09:51:00.752 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:00.752 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:00.752 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:00.752 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:00.752 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.752 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4183.0:31 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'17, mlcod=13'14)", 2024-09-11T09:51:00.752 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:11.499319+0000", 2024-09-11T09:51:00.753 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 49.248022423999998, 2024-09-11T09:51:00.753 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0010073879999999999, 2024-09-11T09:51:00.753 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:00.753 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:00.753 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:00.753 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:00.753 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.753 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:00.753 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.499319+0000", 2024-09-11T09:51:00.753 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:00.753 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:00.753 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.753 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:00.753 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.499319+0000", 2024-09-11T09:51:00.754 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:00.754 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:00.754 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.754 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:00.754 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.499320+0000", 2024-09-11T09:51:00.754 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.7700000000000004e-07 2024-09-11T09:51:00.754 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:00.754 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.754 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:00.754 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.499326+0000", 2024-09-11T09:51:00.754 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.6420000000000001e-06 2024-09-11T09:51:00.754 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:00.754 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.754 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:00.755 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.499339+0000", 2024-09-11T09:51:00.755 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2537999999999999e-05 2024-09-11T09:51:00.755 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:00.755 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.755 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:00.755 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.499345+0000", 2024-09-11T09:51:00.755 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.8220000000000003e-06 2024-09-11T09:51:00.755 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:00.755 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.755 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:00.755 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.499403+0000", 2024-09-11T09:51:00.755 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.8814000000000003e-05 2024-09-11T09:51:00.755 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:00.755 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.756 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:00.756 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.499427+0000", 2024-09-11T09:51:00.756 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.3853999999999999e-05 2024-09-11T09:51:00.756 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:00.756 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.756 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:00.756 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.500294+0000", 2024-09-11T09:51:00.756 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00086622399999999997 2024-09-11T09:51:00.756 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:00.756 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.756 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:00.756 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.500326+0000", 2024-09-11T09:51:00.756 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.2716999999999999e-05 2024-09-11T09:51:00.757 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:00.757 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:00.757 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:00.757 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:00.757 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.757 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4183.0:32 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'18, mlcod=13'14)", 2024-09-11T09:51:00.757 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:11.499796+0000", 2024-09-11T09:51:00.757 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 49.247545217999999, 2024-09-11T09:51:00.757 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00092915100000000002, 2024-09-11T09:51:00.757 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:00.757 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:00.757 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:00.757 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:00.757 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.758 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:00.758 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.499796+0000", 2024-09-11T09:51:00.758 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:00.758 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:00.758 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.758 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:00.758 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.499796+0000", 2024-09-11T09:51:00.758 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:00.758 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:00.758 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.758 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:00.758 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.499797+0000", 2024-09-11T09:51:00.758 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.9400000000000004e-07 2024-09-11T09:51:00.758 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:00.759 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.759 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:00.759 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.499841+0000", 2024-09-11T09:51:00.759 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.3900000000000003e-05 2024-09-11T09:51:00.759 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:00.759 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.759 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:00.759 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.499852+0000", 2024-09-11T09:51:00.759 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1462e-05 2024-09-11T09:51:00.759 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:00.759 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.759 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:00.759 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.499858+0000", 2024-09-11T09:51:00.759 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.8320000000000002e-06 2024-09-11T09:51:00.760 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:00.760 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.760 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:00.760 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.499915+0000", 2024-09-11T09:51:00.760 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.7187e-05 2024-09-11T09:51:00.760 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:00.760 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.760 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:00.760 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.499930+0000", 2024-09-11T09:51:00.760 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.4928e-05 2024-09-11T09:51:00.760 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:00.760 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.760 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:00.760 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.500697+0000", 2024-09-11T09:51:00.761 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00076658399999999995 2024-09-11T09:51:00.761 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:00.761 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.761 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:00.761 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.500725+0000", 2024-09-11T09:51:00.761 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.8364000000000002e-05 2024-09-11T09:51:00.761 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:00.761 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:00.761 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:00.761 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:00.761 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.761 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4183.0:34 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'20, mlcod=13'15)", 2024-09-11T09:51:00.761 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:11.501016+0000", 2024-09-11T09:51:00.762 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 49.2463257, 2024-09-11T09:51:00.762 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00097815399999999996, 2024-09-11T09:51:00.762 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:00.762 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:00.762 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:00.762 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:00.762 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.762 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:00.762 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.501016+0000", 2024-09-11T09:51:00.762 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:00.762 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:00.762 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.762 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:00.762 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.501016+0000", 2024-09-11T09:51:00.763 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:00.763 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:00.763 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.763 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:00.763 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.501016+0000", 2024-09-11T09:51:00.763 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.3799999999999997e-07 2024-09-11T09:51:00.763 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:00.763 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.763 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:00.763 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.501023+0000", 2024-09-11T09:51:00.763 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.2180000000000002e-06 2024-09-11T09:51:00.763 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:00.763 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.763 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:00.764 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.501031+0000", 2024-09-11T09:51:00.764 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.5379999999999997e-06 2024-09-11T09:51:00.764 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:00.764 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.764 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:00.764 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.501035+0000", 2024-09-11T09:51:00.764 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.216e-06 2024-09-11T09:51:00.764 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:00.764 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.764 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:00.764 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.501105+0000", 2024-09-11T09:51:00.764 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.9757000000000003e-05 2024-09-11T09:51:00.764 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:00.765 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.765 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:00.765 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.501120+0000", 2024-09-11T09:51:00.765 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.5358000000000001e-05 2024-09-11T09:51:00.765 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:00.765 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.765 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:00.765 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.501955+0000", 2024-09-11T09:51:00.765 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00083505000000000005 2024-09-11T09:51:00.765 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:00.765 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.765 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:00.765 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.501994+0000", 2024-09-11T09:51:00.766 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.8479000000000002e-05 2024-09-11T09:51:00.766 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:00.766 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:00.766 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:00.766 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:00.766 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.766 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4183.0:38 1.0 e13/12 1:30306672:devicehealth::main.db.0000000000000000:head v 13'24, mlcod=13'22)", 2024-09-11T09:51:00.766 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:11.503891+0000", 2024-09-11T09:51:00.766 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 49.243450258000003, 2024-09-11T09:51:00.766 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000959351, 2024-09-11T09:51:00.766 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:00.766 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:00.767 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:00.767 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:00.767 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.767 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:00.767 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.503891+0000", 2024-09-11T09:51:00.767 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:00.767 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:00.767 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.767 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:00.767 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.503891+0000", 2024-09-11T09:51:00.767 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:00.767 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:00.767 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.767 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:00.768 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.503892+0000", 2024-09-11T09:51:00.768 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.5499999999999998e-07 2024-09-11T09:51:00.768 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:00.768 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.768 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:00.768 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.503967+0000", 2024-09-11T09:51:00.768 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.4993999999999994e-05 2024-09-11T09:51:00.768 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:00.768 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.768 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:00.768 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.503976+0000", 2024-09-11T09:51:00.768 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.4709999999999996e-06 2024-09-11T09:51:00.768 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:00.769 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.769 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:00.769 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.503981+0000", 2024-09-11T09:51:00.769 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.6789999999999996e-06 2024-09-11T09:51:00.769 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:00.769 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.769 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:00.769 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.504021+0000", 2024-09-11T09:51:00.769 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.0065999999999999e-05 2024-09-11T09:51:00.769 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:00.769 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.769 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:00.769 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.504036+0000", 2024-09-11T09:51:00.769 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.468e-05 2024-09-11T09:51:00.770 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:00.770 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.770 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:00.770 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.504813+0000", 2024-09-11T09:51:00.770 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00077718299999999998 2024-09-11T09:51:00.770 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:00.770 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.770 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:00.770 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.504850+0000", 2024-09-11T09:51:00.770 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.7722999999999998e-05 2024-09-11T09:51:00.770 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:00.770 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:00.771 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:00.771 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:00.771 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.771 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4183.0:54 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'32, mlcod=13'29)", 2024-09-11T09:51:00.771 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:11.515404+0000", 2024-09-11T09:51:00.771 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 49.231937653000003, 2024-09-11T09:51:00.771 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00089471799999999997, 2024-09-11T09:51:00.771 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:00.771 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:00.771 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:00.771 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:00.771 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.771 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:00.771 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.515404+0000", 2024-09-11T09:51:00.772 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:00.772 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:00.772 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.772 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:00.772 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.515404+0000", 2024-09-11T09:51:00.772 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:00.772 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:00.772 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.772 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:00.772 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.515404+0000", 2024-09-11T09:51:00.773 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.3799999999999996e-07 2024-09-11T09:51:00.773 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:00.773 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.773 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:00.773 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.515410+0000", 2024-09-11T09:51:00.773 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.1789999999999999e-06 2024-09-11T09:51:00.773 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:00.773 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.773 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:00.773 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.515422+0000", 2024-09-11T09:51:00.773 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1929999999999999e-05 2024-09-11T09:51:00.773 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:00.773 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.773 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:00.774 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.515427+0000", 2024-09-11T09:51:00.774 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.6060000000000002e-06 2024-09-11T09:51:00.774 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:00.774 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.774 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:00.774 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.515482+0000", 2024-09-11T09:51:00.774 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.4743000000000001e-05 2024-09-11T09:51:00.774 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:00.774 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.774 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:00.774 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.515498+0000", 2024-09-11T09:51:00.774 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.5778999999999999e-05 2024-09-11T09:51:00.774 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:00.775 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.775 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:00.775 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.516271+0000", 2024-09-11T09:51:00.775 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00077312599999999996 2024-09-11T09:51:00.775 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:00.775 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.775 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:00.775 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.516298+0000", 2024-09-11T09:51:00.775 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.7617e-05 2024-09-11T09:51:00.775 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:00.775 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:00.775 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:00.775 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:00.775 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.776 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4183.0:57 1.0 e13/12 1:30306672:devicehealth::main.db.0000000000000000:head v 13'35, mlcod=13'33)", 2024-09-11T09:51:00.776 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:11.519823+0000", 2024-09-11T09:51:00.776 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 49.227518596000003, 2024-09-11T09:51:00.776 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00090876100000000001, 2024-09-11T09:51:00.776 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:00.776 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:00.776 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:00.776 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:00.776 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.776 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:00.776 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.519823+0000", 2024-09-11T09:51:00.776 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:00.776 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:00.777 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.777 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:00.777 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.519823+0000", 2024-09-11T09:51:00.777 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:00.777 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:00.777 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.777 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:00.777 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.519823+0000", 2024-09-11T09:51:00.777 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.7000000000000004e-07 2024-09-11T09:51:00.777 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:00.777 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.777 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:00.777 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.519859+0000", 2024-09-11T09:51:00.777 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.5508e-05 2024-09-11T09:51:00.778 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:00.778 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.778 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:00.778 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.519871+0000", 2024-09-11T09:51:00.778 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.221e-05 2024-09-11T09:51:00.778 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:00.778 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.778 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:00.778 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.519877+0000", 2024-09-11T09:51:00.778 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.0410000000000003e-06 2024-09-11T09:51:00.778 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:00.778 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.778 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:00.778 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.519955+0000", 2024-09-11T09:51:00.779 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.8169000000000003e-05 2024-09-11T09:51:00.779 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:00.779 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.779 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:00.779 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.519972+0000", 2024-09-11T09:51:00.779 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6595999999999998e-05 2024-09-11T09:51:00.779 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:00.779 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.779 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:00.779 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.520691+0000", 2024-09-11T09:51:00.779 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000719405 2024-09-11T09:51:00.779 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:00.779 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.779 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:00.780 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.520731+0000", 2024-09-11T09:51:00.780 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.0061999999999997e-05 2024-09-11T09:51:00.780 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:00.780 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:00.780 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:00.780 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:00.780 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.780 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4183.0:58 1.0 e13/12 1:30306672:devicehealth::main.db.0000000000000000:head v 13'36, mlcod=13'33)", 2024-09-11T09:51:00.780 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:11.520285+0000", 2024-09-11T09:51:00.780 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 49.227056187999999, 2024-09-11T09:51:00.780 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00091014799999999995, 2024-09-11T09:51:00.780 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:00.780 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:00.781 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:00.781 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:00.781 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.781 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:00.781 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.520285+0000", 2024-09-11T09:51:00.781 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:00.781 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:00.781 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.781 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:00.781 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.520285+0000", 2024-09-11T09:51:00.781 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:00.781 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:00.781 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.782 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:00.782 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.520286+0000", 2024-09-11T09:51:00.782 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.4399999999999999e-07 2024-09-11T09:51:00.782 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:00.782 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.782 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:00.782 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.520348+0000", 2024-09-11T09:51:00.782 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.1760000000000005e-05 2024-09-11T09:51:00.782 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:00.782 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.782 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:00.782 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.520359+0000", 2024-09-11T09:51:00.782 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1362999999999999e-05 2024-09-11T09:51:00.782 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:00.783 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.783 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:00.783 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.520364+0000", 2024-09-11T09:51:00.783 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.3600000000000004e-06 2024-09-11T09:51:00.783 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:00.783 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.783 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:00.783 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.520413+0000", 2024-09-11T09:51:00.783 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.8226000000000002e-05 2024-09-11T09:51:00.783 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:00.783 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.783 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:00.783 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.520426+0000", 2024-09-11T09:51:00.784 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3560000000000001e-05 2024-09-11T09:51:00.784 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:00.784 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.784 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:00.784 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.521159+0000", 2024-09-11T09:51:00.784 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00073255499999999997 2024-09-11T09:51:00.784 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:00.784 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.784 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:00.784 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.521195+0000", 2024-09-11T09:51:00.784 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.6480000000000003e-05 2024-09-11T09:51:00.784 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:00.784 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:00.785 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:00.785 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:00.785 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.785 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4183.0:68 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'42, mlcod=13'39)", 2024-09-11T09:51:00.785 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:11.534957+0000", 2024-09-11T09:51:00.785 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 49.212384497000002, 2024-09-11T09:51:00.785 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.001065351, 2024-09-11T09:51:00.785 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:00.785 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:00.785 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:00.785 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:00.785 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.785 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:00.786 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.534957+0000", 2024-09-11T09:51:00.786 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:00.786 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:00.786 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.786 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:00.786 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.534957+0000", 2024-09-11T09:51:00.786 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:00.786 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:00.786 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.786 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:00.786 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.534957+0000", 2024-09-11T09:51:00.786 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.3799999999999998e-07 2024-09-11T09:51:00.786 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:00.786 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.787 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:00.787 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.534964+0000", 2024-09-11T09:51:00.787 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.9809999999999997e-06 2024-09-11T09:51:00.787 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:00.787 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.787 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:00.787 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.534976+0000", 2024-09-11T09:51:00.787 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2119999999999999e-05 2024-09-11T09:51:00.787 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:00.787 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.787 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:00.787 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.534980+0000", 2024-09-11T09:51:00.787 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.9890000000000003e-06 2024-09-11T09:51:00.787 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:00.788 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.788 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:00.788 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.535030+0000", 2024-09-11T09:51:00.788 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.9258000000000002e-05 2024-09-11T09:51:00.788 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:00.788 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.788 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:00.788 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.535044+0000", 2024-09-11T09:51:00.788 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3981e-05 2024-09-11T09:51:00.788 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:00.788 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.788 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:00.788 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.535995+0000", 2024-09-11T09:51:00.788 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000951804 2024-09-11T09:51:00.789 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:00.789 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.789 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:00.789 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.536022+0000", 2024-09-11T09:51:00.789 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.6780000000000001e-05 2024-09-11T09:51:00.789 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:00.789 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:00.789 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:00.789 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:00.789 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.789 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4183.0:69 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'43, mlcod=13'39)", 2024-09-11T09:51:00.789 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:11.535331+0000", 2024-09-11T09:51:00.789 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 49.212010174, 2024-09-11T09:51:00.789 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0011256090000000001, 2024-09-11T09:51:00.790 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:00.790 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:00.790 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:00.790 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:00.790 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.790 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:00.790 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.535331+0000", 2024-09-11T09:51:00.790 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:00.790 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:00.790 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.790 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:00.790 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.535331+0000", 2024-09-11T09:51:00.790 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:00.791 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:00.791 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.791 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:00.791 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.535332+0000", 2024-09-11T09:51:00.791 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.1600000000000001e-07 2024-09-11T09:51:00.791 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:00.791 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.791 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:00.791 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.535393+0000", 2024-09-11T09:51:00.791 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.1452999999999996e-05 2024-09-11T09:51:00.791 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:00.791 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.791 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:00.791 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.535402+0000", 2024-09-11T09:51:00.792 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.4300000000000006e-06 2024-09-11T09:51:00.792 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:00.792 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.792 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:00.792 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.535405+0000", 2024-09-11T09:51:00.792 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.512e-06 2024-09-11T09:51:00.792 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:00.792 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.792 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:00.792 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.535482+0000", 2024-09-11T09:51:00.792 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.7132999999999995e-05 2024-09-11T09:51:00.792 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:00.792 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.793 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:00.793 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.535506+0000", 2024-09-11T09:51:00.793 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.4070000000000002e-05 2024-09-11T09:51:00.793 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:00.793 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.793 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:00.793 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.536424+0000", 2024-09-11T09:51:00.793 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00091766700000000005 2024-09-11T09:51:00.793 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:00.793 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.793 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:00.793 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.536457+0000", 2024-09-11T09:51:00.793 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.2827999999999998e-05 2024-09-11T09:51:00.793 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:00.794 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:00.794 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:00.794 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:00.794 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.794 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4183.0:83 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'53, mlcod=13'50)", 2024-09-11T09:51:00.794 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:11.558836+0000", 2024-09-11T09:51:00.794 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 49.188504919000003, 2024-09-11T09:51:00.794 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00090313100000000005, 2024-09-11T09:51:00.794 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:00.794 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:00.794 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:00.794 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:00.794 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.794 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:00.795 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.558836+0000", 2024-09-11T09:51:00.795 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:00.795 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:00.795 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.795 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:00.795 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.558836+0000", 2024-09-11T09:51:00.795 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:00.795 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:00.795 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.795 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:00.795 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.558837+0000", 2024-09-11T09:51:00.795 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.4900000000000005e-07 2024-09-11T09:51:00.795 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:00.795 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.796 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:00.796 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.558857+0000", 2024-09-11T09:51:00.796 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.9502999999999999e-05 2024-09-11T09:51:00.796 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:00.796 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.796 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:00.796 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.558870+0000", 2024-09-11T09:51:00.796 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3020000000000001e-05 2024-09-11T09:51:00.796 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:00.796 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.796 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:00.796 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.558876+0000", 2024-09-11T09:51:00.796 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.7270000000000004e-06 2024-09-11T09:51:00.797 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:00.797 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.797 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:00.797 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.558934+0000", 2024-09-11T09:51:00.797 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.8100999999999998e-05 2024-09-11T09:51:00.797 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:00.797 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.797 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:00.797 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.558947+0000", 2024-09-11T09:51:00.797 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3269e-05 2024-09-11T09:51:00.797 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:00.797 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.797 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:00.797 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.559720+0000", 2024-09-11T09:51:00.798 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00077287600000000003 2024-09-11T09:51:00.798 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:00.798 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.798 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:00.798 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.559740+0000", 2024-09-11T09:51:00.798 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.9786e-05 2024-09-11T09:51:00.798 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:00.798 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:00.798 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:00.798 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:00.798 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.798 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4183.0:84 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'54, mlcod=13'50)", 2024-09-11T09:51:00.798 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:11.559267+0000", 2024-09-11T09:51:00.798 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 49.188074491999998, 2024-09-11T09:51:00.799 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00093990199999999995, 2024-09-11T09:51:00.799 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:00.799 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:00.799 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:00.799 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:00.799 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.799 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:00.799 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.559267+0000", 2024-09-11T09:51:00.799 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:00.799 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:00.799 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.799 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:00.799 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.559267+0000", 2024-09-11T09:51:00.800 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:00.800 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:00.800 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.800 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:00.800 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.559267+0000", 2024-09-11T09:51:00.800 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.2900000000000003e-07 2024-09-11T09:51:00.800 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:00.800 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.800 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:00.800 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.559322+0000", 2024-09-11T09:51:00.800 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.4429999999999999e-05 2024-09-11T09:51:00.800 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:00.800 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.800 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:00.801 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.559332+0000", 2024-09-11T09:51:00.801 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0473e-05 2024-09-11T09:51:00.801 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:00.801 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.801 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:00.801 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.559338+0000", 2024-09-11T09:51:00.801 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.0730000000000001e-06 2024-09-11T09:51:00.801 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:00.801 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.801 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:00.801 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.559373+0000", 2024-09-11T09:51:00.801 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.4626999999999999e-05 2024-09-11T09:51:00.801 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:00.801 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.802 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:00.802 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.559384+0000", 2024-09-11T09:51:00.802 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1307e-05 2024-09-11T09:51:00.802 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:00.802 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.802 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:00.802 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.560175+0000", 2024-09-11T09:51:00.802 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00079016399999999999 2024-09-11T09:51:00.802 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:00.802 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.802 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:00.802 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.560207+0000", 2024-09-11T09:51:00.802 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.2199000000000001e-05 2024-09-11T09:51:00.802 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:00.803 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:00.803 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:00.803 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:00.803 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.803 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4183.0:89 1.0 e13/12 1:30306672:devicehealth::main.db.0000000000000000:head v 13'59, mlcod=13'57)", 2024-09-11T09:51:00.803 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:11.563614+0000", 2024-09-11T09:51:00.803 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 49.183726872000001, 2024-09-11T09:51:00.803 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00091281399999999996, 2024-09-11T09:51:00.803 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:00.803 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:00.803 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:00.803 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:00.803 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.803 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:00.804 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.563614+0000", 2024-09-11T09:51:00.804 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:00.804 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:00.804 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.804 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:00.804 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.563614+0000", 2024-09-11T09:51:00.804 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:00.804 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:00.804 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.804 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:00.804 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.563615+0000", 2024-09-11T09:51:00.804 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.6899999999999998e-07 2024-09-11T09:51:00.804 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:00.804 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.805 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:00.805 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.563681+0000", 2024-09-11T09:51:00.805 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.5769e-05 2024-09-11T09:51:00.805 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:00.805 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.805 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:00.805 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.563690+0000", 2024-09-11T09:51:00.805 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.6800000000000005e-06 2024-09-11T09:51:00.805 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:00.805 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.805 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:00.805 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.563699+0000", 2024-09-11T09:51:00.805 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.3170000000000002e-06 2024-09-11T09:51:00.805 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:00.806 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.806 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:00.806 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.563734+0000", 2024-09-11T09:51:00.806 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.5023e-05 2024-09-11T09:51:00.806 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:00.806 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.806 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:00.806 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.563746+0000", 2024-09-11T09:51:00.806 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2661e-05 2024-09-11T09:51:00.806 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:00.806 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.806 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:00.806 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.564500+0000", 2024-09-11T09:51:00.807 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00075323100000000004 2024-09-11T09:51:00.807 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:00.807 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.807 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:00.807 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.564527+0000", 2024-09-11T09:51:00.807 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.7664000000000001e-05 2024-09-11T09:51:00.807 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:00.807 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:00.807 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:00.807 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:00.807 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.807 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4183.0:105 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'67, mlcod=13'64)", 2024-09-11T09:51:00.807 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:11.712146+0000", 2024-09-11T09:51:00.807 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 49.035195121999998, 2024-09-11T09:51:00.808 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0010916859999999999, 2024-09-11T09:51:00.808 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:00.808 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:00.808 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:00.808 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:00.808 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.808 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:00.808 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.712146+0000", 2024-09-11T09:51:00.808 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:00.808 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:00.808 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.808 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:00.808 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.712146+0000", 2024-09-11T09:51:00.808 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:00.809 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:00.809 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.809 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:00.809 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.712147+0000", 2024-09-11T09:51:00.809 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.8200000000000001e-07 2024-09-11T09:51:00.809 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:00.809 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.809 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:00.809 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.712155+0000", 2024-09-11T09:51:00.809 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.0309999999999998e-06 2024-09-11T09:51:00.809 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:00.809 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.809 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:00.809 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.712163+0000", 2024-09-11T09:51:00.810 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.5920000000000001e-06 2024-09-11T09:51:00.810 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:00.810 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.810 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:00.810 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.712169+0000", 2024-09-11T09:51:00.810 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.4299999999999997e-06 2024-09-11T09:51:00.810 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:00.810 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.810 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:00.810 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.712418+0000", 2024-09-11T09:51:00.810 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00024978499999999998 2024-09-11T09:51:00.810 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:00.810 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.811 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:00.811 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.712432+0000", 2024-09-11T09:51:00.811 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3190000000000001e-05 2024-09-11T09:51:00.811 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:00.811 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.811 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:00.811 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.713206+0000", 2024-09-11T09:51:00.811 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00077420199999999998 2024-09-11T09:51:00.811 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:00.811 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.811 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:00.811 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.713238+0000", 2024-09-11T09:51:00.811 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.2073999999999998e-05 2024-09-11T09:51:00.811 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:00.812 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:00.812 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:00.812 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:00.812 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.812 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4183.0:106 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'68, mlcod=13'64)", 2024-09-11T09:51:00.812 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:11.712624+0000", 2024-09-11T09:51:00.812 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 49.034717794000002, 2024-09-11T09:51:00.812 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0010009629999999999, 2024-09-11T09:51:00.812 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:00.812 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:00.812 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:00.812 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:00.812 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.813 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:00.813 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.712624+0000", 2024-09-11T09:51:00.813 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:00.813 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:00.813 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.813 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:00.813 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.712624+0000", 2024-09-11T09:51:00.813 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:00.813 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:00.813 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.813 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:00.813 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.712624+0000", 2024-09-11T09:51:00.813 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.4799999999999998e-07 2024-09-11T09:51:00.813 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:00.814 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.814 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:00.814 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.712689+0000", 2024-09-11T09:51:00.814 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.4488999999999999e-05 2024-09-11T09:51:00.814 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:00.814 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.814 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:00.814 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.712707+0000", 2024-09-11T09:51:00.814 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.8272000000000001e-05 2024-09-11T09:51:00.814 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:00.814 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.814 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:00.814 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.712713+0000", 2024-09-11T09:51:00.815 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.2040000000000002e-06 2024-09-11T09:51:00.815 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:00.815 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.815 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:00.815 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.712783+0000", 2024-09-11T09:51:00.815 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.9534000000000002e-05 2024-09-11T09:51:00.815 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:00.815 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.815 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:00.815 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.712811+0000", 2024-09-11T09:51:00.815 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.8620999999999999e-05 2024-09-11T09:51:00.815 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:00.815 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.815 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:00.815 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.713586+0000", 2024-09-11T09:51:00.816 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00077462600000000005 2024-09-11T09:51:00.816 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:00.816 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.816 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:00.816 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.713624+0000", 2024-09-11T09:51:00.816 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.8668999999999999e-05 2024-09-11T09:51:00.816 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:00.816 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:00.816 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:00.816 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:00.816 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.816 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4183.0:134 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'92, mlcod=13'86)", 2024-09-11T09:51:00.816 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:12.035698+0000", 2024-09-11T09:51:00.817 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 48.711643535, 2024-09-11T09:51:00.817 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00089550700000000003, 2024-09-11T09:51:00.817 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:00.817 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:00.817 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:00.817 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:00.817 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.817 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:00.817 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.035698+0000", 2024-09-11T09:51:00.817 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:00.817 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:00.817 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.817 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:00.818 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.035698+0000", 2024-09-11T09:51:00.818 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:00.818 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:00.818 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.818 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:00.818 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.035698+0000", 2024-09-11T09:51:00.818 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.0900000000000002e-07 2024-09-11T09:51:00.818 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:00.818 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.818 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:00.818 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.035758+0000", 2024-09-11T09:51:00.818 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.9481999999999997e-05 2024-09-11T09:51:00.818 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:00.818 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.819 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:00.819 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.035766+0000", 2024-09-11T09:51:00.819 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.3760000000000008e-06 2024-09-11T09:51:00.819 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:00.819 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.819 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:00.819 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.035772+0000", 2024-09-11T09:51:00.819 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.8930000000000002e-06 2024-09-11T09:51:00.819 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:00.819 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.819 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:00.819 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.035807+0000", 2024-09-11T09:51:00.819 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.4975999999999999e-05 2024-09-11T09:51:00.819 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:00.820 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.820 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:00.820 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.035817+0000", 2024-09-11T09:51:00.820 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0314e-05 2024-09-11T09:51:00.820 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:00.820 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.820 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:00.820 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.036542+0000", 2024-09-11T09:51:00.820 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00072434400000000003 2024-09-11T09:51:00.820 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:00.820 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.820 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:00.820 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.036593+0000", 2024-09-11T09:51:00.821 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.1613000000000002e-05 2024-09-11T09:51:00.821 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:00.821 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:00.821 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:00.821 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:00.821 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.821 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4183.0:136 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'94, mlcod=13'86)", 2024-09-11T09:51:00.821 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:12.036591+0000", 2024-09-11T09:51:00.821 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 48.710750324999999, 2024-09-11T09:51:00.821 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00090637900000000002, 2024-09-11T09:51:00.821 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:00.821 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:00.821 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:00.821 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:00.822 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.822 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:00.822 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.036591+0000", 2024-09-11T09:51:00.822 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:00.822 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:00.822 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.822 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:00.822 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.036591+0000", 2024-09-11T09:51:00.822 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:00.822 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:00.822 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.822 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:00.822 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.036592+0000", 2024-09-11T09:51:00.822 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.37e-07 2024-09-11T09:51:00.823 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:00.823 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.823 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:00.823 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.036600+0000", 2024-09-11T09:51:00.823 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.6910000000000003e-06 2024-09-11T09:51:00.823 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:00.823 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.823 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:00.823 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.036609+0000", 2024-09-11T09:51:00.823 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.0229999999999998e-06 2024-09-11T09:51:00.823 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:00.823 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.823 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:00.823 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.036617+0000", 2024-09-11T09:51:00.824 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.6979999999999998e-06 2024-09-11T09:51:00.824 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:00.824 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.824 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:00.824 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.036658+0000", 2024-09-11T09:51:00.824 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.0692000000000003e-05 2024-09-11T09:51:00.824 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:00.824 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.824 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:00.824 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.036670+0000", 2024-09-11T09:51:00.824 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2109e-05 2024-09-11T09:51:00.824 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:00.824 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.825 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:00.825 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.037475+0000", 2024-09-11T09:51:00.825 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00080487199999999999 2024-09-11T09:51:00.825 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:00.825 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.825 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:00.825 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.037497+0000", 2024-09-11T09:51:00.825 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.2557000000000001e-05 2024-09-11T09:51:00.825 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:00.825 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:00.825 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:00.825 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:00.825 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.826 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4183.0:158 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'112, mlcod=13'108)", 2024-09-11T09:51:00.826 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:12.455073+0000", 2024-09-11T09:51:00.826 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 48.292268071999999, 2024-09-11T09:51:00.826 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00090654500000000003, 2024-09-11T09:51:00.826 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:00.826 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:00.826 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:00.826 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:00.826 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.826 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:00.826 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.455073+0000", 2024-09-11T09:51:00.826 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:00.826 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:00.826 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.827 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:00.827 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.455073+0000", 2024-09-11T09:51:00.827 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:00.827 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:00.827 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.827 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:00.827 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.455074+0000", 2024-09-11T09:51:00.827 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.32e-07 2024-09-11T09:51:00.827 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:00.827 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.827 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:00.827 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.455126+0000", 2024-09-11T09:51:00.827 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.1898999999999999e-05 2024-09-11T09:51:00.828 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:00.828 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.828 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:00.828 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.455136+0000", 2024-09-11T09:51:00.828 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0651e-05 2024-09-11T09:51:00.828 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:00.828 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.828 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:00.828 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.455142+0000", 2024-09-11T09:51:00.828 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.0020000000000003e-06 2024-09-11T09:51:00.828 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:00.828 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.828 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:00.828 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.455192+0000", 2024-09-11T09:51:00.829 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.0164000000000003e-05 2024-09-11T09:51:00.829 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:00.829 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.829 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:00.829 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.455204+0000", 2024-09-11T09:51:00.829 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1178999999999999e-05 2024-09-11T09:51:00.829 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:00.829 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.829 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:00.829 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.455957+0000", 2024-09-11T09:51:00.829 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00075362300000000003 2024-09-11T09:51:00.829 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:00.829 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.830 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:00.830 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.455980+0000", 2024-09-11T09:51:00.830 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.2595e-05 2024-09-11T09:51:00.830 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:00.830 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:00.830 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:00.830 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:00.830 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.830 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "MOSDPGPush(1.0 34/33 [PushOp(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head, version: 13'116, data_included: [0~262688], data_size: 262688, omap_header_size: 0, omap_entries_size: 0, attrset_size: 9, recovery_info: ObjectRecoveryInfo(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head@13'116, size: 262688, copy_subset: [(0, 262688)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 262688, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-11T09:51:00.830 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:47.599030+0000", 2024-09-11T09:51:00.830 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 13.148311445999999, 2024-09-11T09:51:00.830 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0010572209999999999, 2024-09-11T09:51:00.830 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:00.830 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:00.830 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "started", 2024-09-11T09:51:00.831 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:00.831 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.831 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:00.831 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:47.599030+0000", 2024-09-11T09:51:00.831 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:00.831 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:00.831 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.831 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:00.831 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:47.599030+0000", 2024-09-11T09:51:00.831 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:00.831 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:00.831 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.831 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:00.831 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:47.599031+0000", 2024-09-11T09:51:00.832 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.92e-07 2024-09-11T09:51:00.832 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:00.832 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.832 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:00.832 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:47.599328+0000", 2024-09-11T09:51:00.832 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00029708399999999998 2024-09-11T09:51:00.832 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:00.832 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.832 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:00.832 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:47.599353+0000", 2024-09-11T09:51:00.832 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.5721e-05 2024-09-11T09:51:00.832 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:00.832 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.833 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:00.833 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:47.599375+0000", 2024-09-11T09:51:00.833 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.1990000000000001e-05 2024-09-11T09:51:00.833 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:00.833 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.833 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:00.833 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:47.599466+0000", 2024-09-11T09:51:00.833 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.0175999999999999e-05 2024-09-11T09:51:00.833 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:00.833 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.833 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:00.833 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:47.599480+0000", 2024-09-11T09:51:00.833 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.4873e-05 2024-09-11T09:51:00.833 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:00.834 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.834 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:00.834 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:47.600087+0000", 2024-09-11T09:51:00.834 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00060668500000000004 2024-09-11T09:51:00.834 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:00.834 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:00.834 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:00.834 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:00.834 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.834 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "MOSDPGPush(1.0 34/33 [PushOp(1:30306672:devicehealth::main.db.0000000000000000:head, version: 13'117, data_included: [0~327680], data_size: 327680, omap_header_size: 0, omap_entries_size: 0, attrset_size: 10, recovery_info: ObjectRecoveryInfo(1:30306672:devicehealth::main.db.0000000000000000:head@13'117, size: 327680, copy_subset: [(0, 327680)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 327680, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-11T09:51:00.834 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:47.601120+0000", 2024-09-11T09:51:00.834 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 13.146220958000001, 2024-09-11T09:51:00.834 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00099708000000000001, 2024-09-11T09:51:00.835 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:00.835 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:00.835 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "started", 2024-09-11T09:51:00.835 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:00.835 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.835 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:00.835 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:47.601120+0000", 2024-09-11T09:51:00.835 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:00.835 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:00.835 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.835 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:00.835 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:47.601120+0000", 2024-09-11T09:51:00.835 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:00.835 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:00.835 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.836 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:00.836 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:47.601121+0000", 2024-09-11T09:51:00.836 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.0200000000000001e-07 2024-09-11T09:51:00.836 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:00.836 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.836 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:00.836 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:47.601430+0000", 2024-09-11T09:51:00.836 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00030917899999999998 2024-09-11T09:51:00.836 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:00.836 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.836 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:00.836 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:47.601454+0000", 2024-09-11T09:51:00.836 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.4000999999999999e-05 2024-09-11T09:51:00.837 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:00.837 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.837 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:00.837 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:47.601475+0000", 2024-09-11T09:51:00.837 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.0305999999999998e-05 2024-09-11T09:51:00.837 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:00.837 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.837 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:00.837 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:47.601561+0000", 2024-09-11T09:51:00.837 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.6414000000000003e-05 2024-09-11T09:51:00.837 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:00.837 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.837 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:00.838 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:47.601573+0000", 2024-09-11T09:51:00.838 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1894e-05 2024-09-11T09:51:00.838 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:00.838 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:00.838 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:00.838 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:47.602117+0000", 2024-09-11T09:51:00.838 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00054458399999999998 2024-09-11T09:51:00.838 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:00.838 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:00.838 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:00.838 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:00.838 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:00.838 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:51:00.840 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-11T09:51:00.840 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-09-11T09:51:00.840+0000 7f7b6c7d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31904) UID: 0 2024-09-11T09:51:00.940 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-09-11T09:51:00.941+0000 7f7b6c7d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31904) UID: 0 2024-09-11T09:51:00.989 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:51:00.989 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [], 2024-09-11T09:51:00.989 INFO:teuthology.orchestra.run.smithi017.stdout: "num_ops": 0 2024-09-11T09:51:00.989 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:51:00.997 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-11T09:51:01.041 INFO:tasks.ceph.osd.0.smithi017.stderr:2024-09-11T09:51:01.042+0000 7fb73c0f1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31665) UID: 0 2024-09-11T09:51:01.104 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:51:01.105 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [], 2024-09-11T09:51:01.105 INFO:teuthology.orchestra.run.smithi017.stdout: "complaint_time": 30, 2024-09-11T09:51:01.105 INFO:teuthology.orchestra.run.smithi017.stdout: "num_blocked_ops": 0 2024-09-11T09:51:01.105 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:51:01.113 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-11T09:51:01.142 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-09-11T09:51:01.142+0000 7f7b6c7d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31904) UID: 0 2024-09-11T09:51:01.243 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:51:01.244+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:51:01.261 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:51:01.261 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 20, 2024-09-11T09:51:01.261 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 600, 2024-09-11T09:51:01.261 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [ 2024-09-11T09:51:01.261 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:01.261 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4230.0:5 2.2 e17/15 2:5cea7035:::rbd_info:head v 17'1, mlcod=0'0)", 2024-09-11T09:51:01.261 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:19.494757+0000", 2024-09-11T09:51:01.261 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 41.767684506999998, 2024-09-11T09:51:01.261 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00082098399999999997, 2024-09-11T09:51:01.261 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:01.261 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:01.261 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:01.262 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:01.262 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:01.262 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:01.262 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:19.494757+0000", 2024-09-11T09:51:01.262 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:01.262 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:01.262 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:01.262 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:01.262 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:19.494757+0000", 2024-09-11T09:51:01.262 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:01.262 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:01.262 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:01.263 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:01.263 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:19.494757+0000", 2024-09-11T09:51:01.263 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.97e-07 2024-09-11T09:51:01.263 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:01.263 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:01.263 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:01.263 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:19.494763+0000", 2024-09-11T09:51:01.263 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.3709999999999996e-06 2024-09-11T09:51:01.263 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:01.263 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:01.263 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:01.263 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:19.494778+0000", 2024-09-11T09:51:01.263 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.4960000000000001e-05 2024-09-11T09:51:01.263 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:01.264 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:01.264 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:01.264 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:19.494801+0000", 2024-09-11T09:51:01.264 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.2388999999999999e-05 2024-09-11T09:51:01.264 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:01.264 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:01.264 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:01.264 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:19.494859+0000", 2024-09-11T09:51:01.264 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.7961999999999998e-05 2024-09-11T09:51:01.264 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:01.264 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:01.264 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:01.264 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:19.494883+0000", 2024-09-11T09:51:01.265 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.4488999999999999e-05 2024-09-11T09:51:01.265 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:01.265 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:01.265 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:01.265 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:19.495534+0000", 2024-09-11T09:51:01.265 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00065075500000000004 2024-09-11T09:51:01.265 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:01.265 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:01.265 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:01.265 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:19.495578+0000", 2024-09-11T09:51:01.265 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.3860999999999999e-05 2024-09-11T09:51:01.265 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:01.265 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:01.266 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:01.266 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:01.266 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:01.266 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4230.0:7 2.2 e18/15 2:5cea7035:::rbd_info:head v 18'2, mlcod=0'0)", 2024-09-11T09:51:01.266 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:20.500782+0000", 2024-09-11T09:51:01.266 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 40.761659627, 2024-09-11T09:51:01.266 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000790185, 2024-09-11T09:51:01.266 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:01.266 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:01.266 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:01.266 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:01.266 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:01.266 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:01.267 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:20.500782+0000", 2024-09-11T09:51:01.267 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:01.267 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:01.267 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:01.267 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:01.267 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:20.500782+0000", 2024-09-11T09:51:01.267 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:01.267 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:01.267 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:01.267 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:01.268 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:20.500782+0000", 2024-09-11T09:51:01.268 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.3299999999999998e-07 2024-09-11T09:51:01.268 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:01.268 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:01.268 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:01.268 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:20.500799+0000", 2024-09-11T09:51:01.268 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6830999999999999e-05 2024-09-11T09:51:01.268 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:01.268 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:01.268 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:01.268 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:20.500812+0000", 2024-09-11T09:51:01.268 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3013999999999999e-05 2024-09-11T09:51:01.268 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:01.269 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:01.269 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:01.269 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:20.500824+0000", 2024-09-11T09:51:01.269 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2593e-05 2024-09-11T09:51:01.269 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:01.269 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:01.269 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:01.269 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:20.500895+0000", 2024-09-11T09:51:01.269 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.1018000000000003e-05 2024-09-11T09:51:01.269 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:01.269 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:01.269 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:01.269 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:20.500912+0000", 2024-09-11T09:51:01.269 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6484e-05 2024-09-11T09:51:01.270 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:01.270 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:01.270 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:01.270 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:20.501531+0000", 2024-09-11T09:51:01.270 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00061903399999999997 2024-09-11T09:51:01.270 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:01.270 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:01.270 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:01.270 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:20.501572+0000", 2024-09-11T09:51:01.270 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.0877999999999997e-05 2024-09-11T09:51:01.270 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:01.270 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:01.270 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:01.270 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:01.271 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:01.271 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:51:01.272 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-11T09:51:01.344 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:51:01.344+0000 7f3c96ec9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31909) UID: 0 2024-09-11T09:51:01.420 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:51:01.420 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [], 2024-09-11T09:51:01.420 INFO:teuthology.orchestra.run.smithi017.stdout: "num_ops": 0 2024-09-11T09:51:01.420 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:51:01.428 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-11T09:51:01.445 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:51:01.446+0000 7fdec5849640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31907) UID: 0 2024-09-11T09:51:01.535 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:51:01.535 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [], 2024-09-11T09:51:01.535 INFO:teuthology.orchestra.run.smithi017.stdout: "complaint_time": 30, 2024-09-11T09:51:01.535 INFO:teuthology.orchestra.run.smithi017.stdout: "num_blocked_ops": 0 2024-09-11T09:51:01.535 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:51:01.543 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-11T09:51:01.545 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:51:01.546+0000 7f3c96ec9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31909) UID: 0 2024-09-11T09:51:01.646 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:51:01.647+0000 7f3c96ec9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31909) UID: 0 2024-09-11T09:51:01.651 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:51:01.651 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 20, 2024-09-11T09:51:01.652 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 600, 2024-09-11T09:51:01.652 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [ 2024-09-11T09:51:01.652 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:01.652 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4230.0:1 2.3 e16/15 2:c4c92e5a:::rbd_trash:head v 16'1, mlcod=0'0)", 2024-09-11T09:51:01.652 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:18.494378+0000", 2024-09-11T09:51:01.652 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 43.158818537999998, 2024-09-11T09:51:01.652 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00089752199999999999, 2024-09-11T09:51:01.652 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:01.652 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:01.652 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:01.653 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:01.653 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:01.653 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:01.653 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:18.494378+0000", 2024-09-11T09:51:01.653 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:01.653 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:01.653 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:01.653 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:01.653 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:18.494378+0000", 2024-09-11T09:51:01.653 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:01.653 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:01.653 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:01.654 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:01.654 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:18.494378+0000", 2024-09-11T09:51:01.654 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.9099999999999999e-07 2024-09-11T09:51:01.654 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:01.654 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:01.654 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:01.654 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:18.494385+0000", 2024-09-11T09:51:01.654 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.8129999999999998e-06 2024-09-11T09:51:01.654 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:01.654 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:01.654 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:01.654 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:18.494416+0000", 2024-09-11T09:51:01.654 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.0654000000000002e-05 2024-09-11T09:51:01.655 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:01.655 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:01.655 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:01.655 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:18.494430+0000", 2024-09-11T09:51:01.655 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3754000000000001e-05 2024-09-11T09:51:01.655 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:01.655 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:01.655 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:01.655 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:18.494524+0000", 2024-09-11T09:51:01.655 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.4896999999999996e-05 2024-09-11T09:51:01.655 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:01.655 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:01.655 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:01.655 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:18.494558+0000", 2024-09-11T09:51:01.656 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.3900999999999999e-05 2024-09-11T09:51:01.656 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:01.656 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:01.656 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:01.656 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:18.495231+0000", 2024-09-11T09:51:01.656 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00067233500000000003 2024-09-11T09:51:01.656 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:01.656 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:01.656 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:01.656 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:18.495275+0000", 2024-09-11T09:51:01.656 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.4777000000000002e-05 2024-09-11T09:51:01.656 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:01.656 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:01.656 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:01.657 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:01.657 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:01.657 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:51:01.660 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-11T09:51:01.747 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:51:01.748+0000 7f3c96ec9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31909) UID: 0 2024-09-11T09:51:01.775 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-11T09:51:01.775 INFO:teuthology.orchestra.run.smithi096.stdout: "ops": [], 2024-09-11T09:51:01.775 INFO:teuthology.orchestra.run.smithi096.stdout: "num_ops": 0 2024-09-11T09:51:01.775 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-11T09:51:01.784 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-11T09:51:01.869 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-09-11T09:51:01.870+0000 7f7b6c7d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31904) UID: 0 2024-09-11T09:51:01.929 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-11T09:51:01.929 INFO:teuthology.orchestra.run.smithi096.stdout: "ops": [], 2024-09-11T09:51:01.929 INFO:teuthology.orchestra.run.smithi096.stdout: "complaint_time": 30, 2024-09-11T09:51:01.929 INFO:teuthology.orchestra.run.smithi096.stdout: "num_blocked_ops": 0 2024-09-11T09:51:01.929 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-11T09:51:01.938 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-11T09:51:01.949 INFO:tasks.ceph.osd.2.smithi017.stderr:2024-09-11T09:51:01.950+0000 7fd20b170640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31672) UID: 0 2024-09-11T09:51:02.050 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-09-11T09:51:02.050+0000 7f7b6c7d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31904) UID: 0 2024-09-11T09:51:02.084 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-11T09:51:02.084 INFO:teuthology.orchestra.run.smithi096.stdout: "size": 20, 2024-09-11T09:51:02.084 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 600, 2024-09-11T09:51:02.084 INFO:teuthology.orchestra.run.smithi096.stdout: "ops": [ 2024-09-11T09:51:02.084 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.084 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:74 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:51:02.085 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.539067+0000", 2024-09-11T09:51:02.085 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 50.544813589, 2024-09-11T09:51:02.085 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.006945957, 2024-09-11T09:51:02.085 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:02.085 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:02.085 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:02.085 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:02.085 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:51:02.085 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:51:02.085 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 74 2024-09-11T09:51:02.085 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.085 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:02.085 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.086 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:02.086 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.539067+0000", 2024-09-11T09:51:02.086 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:02.086 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.086 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.086 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:02.086 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.539067+0000", 2024-09-11T09:51:02.086 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:02.086 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.086 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.086 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:02.086 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.539068+0000", 2024-09-11T09:51:02.086 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.6000000000000003e-07 2024-09-11T09:51:02.086 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.086 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.087 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:02.087 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.539072+0000", 2024-09-11T09:51:02.087 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.0450000000000002e-06 2024-09-11T09:51:02.087 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.087 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.087 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:02.087 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.539078+0000", 2024-09-11T09:51:02.087 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.6889999999999998e-06 2024-09-11T09:51:02.087 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.087 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.087 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:02.087 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.539083+0000", 2024-09-11T09:51:02.087 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.7180000000000004e-06 2024-09-11T09:51:02.088 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.088 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.088 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:02.088 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.540230+0000", 2024-09-11T09:51:02.088 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0011466250000000001 2024-09-11T09:51:02.088 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.088 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.088 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:02.088 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.540279+0000", 2024-09-11T09:51:02.088 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.9667999999999999e-05 2024-09-11T09:51:02.088 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.088 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.088 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:02.088 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.540357+0000", 2024-09-11T09:51:02.089 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.7863999999999999e-05 2024-09-11T09:51:02.089 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.089 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.089 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:02.089 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.541086+0000", 2024-09-11T09:51:02.089 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00072921100000000001 2024-09-11T09:51:02.089 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.089 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.089 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:02.089 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.545949+0000", 2024-09-11T09:51:02.089 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0048626040000000004 2024-09-11T09:51:02.089 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.089 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.089 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:02.090 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.545989+0000", 2024-09-11T09:51:02.090 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.0259000000000002e-05 2024-09-11T09:51:02.090 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.090 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.090 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:02.090 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.546013+0000", 2024-09-11T09:51:02.090 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.3614000000000001e-05 2024-09-11T09:51:02.090 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:02.090 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:02.090 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:02.090 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.090 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.090 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:108 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:51:02.090 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.711510+0000", 2024-09-11T09:51:02.091 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 50.372370416999999, 2024-09-11T09:51:02.091 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0054615649999999998, 2024-09-11T09:51:02.091 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:02.091 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:02.091 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:02.091 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:02.091 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:51:02.091 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:51:02.091 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 108 2024-09-11T09:51:02.091 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.091 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:02.091 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.091 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:02.091 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711510+0000", 2024-09-11T09:51:02.092 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:02.092 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.092 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.092 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:02.092 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711510+0000", 2024-09-11T09:51:02.092 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:02.092 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.092 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.092 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:02.092 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711511+0000", 2024-09-11T09:51:02.092 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.1999999999999998e-07 2024-09-11T09:51:02.092 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.092 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.093 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:02.093 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711516+0000", 2024-09-11T09:51:02.093 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.4140000000000002e-06 2024-09-11T09:51:02.093 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.093 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.093 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:02.093 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711527+0000", 2024-09-11T09:51:02.093 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.0169e-05 2024-09-11T09:51:02.093 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.093 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.093 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:02.093 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711535+0000", 2024-09-11T09:51:02.093 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.3299999999999999e-06 2024-09-11T09:51:02.093 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.094 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.094 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:02.094 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.713763+0000", 2024-09-11T09:51:02.094 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0022281079999999999 2024-09-11T09:51:02.094 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.094 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.094 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:02.094 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.713812+0000", 2024-09-11T09:51:02.094 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.8544000000000001e-05 2024-09-11T09:51:02.094 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.094 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.094 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:02.094 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.713909+0000", 2024-09-11T09:51:02.095 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.7113000000000003e-05 2024-09-11T09:51:02.095 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.095 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.095 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:02.095 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.716873+0000", 2024-09-11T09:51:02.095 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0029642710000000001 2024-09-11T09:51:02.095 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.095 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.095 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:02.095 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.716918+0000", 2024-09-11T09:51:02.095 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.5447e-05 2024-09-11T09:51:02.095 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.095 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.095 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:02.096 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.716947+0000", 2024-09-11T09:51:02.096 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.8181e-05 2024-09-11T09:51:02.096 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.096 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.096 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:02.096 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.716972+0000", 2024-09-11T09:51:02.096 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.5068000000000001e-05 2024-09-11T09:51:02.096 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:02.096 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:02.096 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:02.096 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.096 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.096 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:109 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:51:02.096 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.711555+0000", 2024-09-11T09:51:02.097 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 50.372325168000003, 2024-09-11T09:51:02.097 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.005479902, 2024-09-11T09:51:02.097 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:02.097 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:02.097 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:02.097 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:02.097 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:51:02.097 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:51:02.097 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 109 2024-09-11T09:51:02.097 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.097 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:02.097 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.097 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:02.098 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711555+0000", 2024-09-11T09:51:02.098 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:02.098 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.098 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.098 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:02.098 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711555+0000", 2024-09-11T09:51:02.098 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:02.098 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.098 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.098 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:02.098 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711556+0000", 2024-09-11T09:51:02.098 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.3200000000000005e-07 2024-09-11T09:51:02.098 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.099 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.099 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:02.099 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711601+0000", 2024-09-11T09:51:02.099 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.5349000000000002e-05 2024-09-11T09:51:02.099 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.099 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.099 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:02.099 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711612+0000", 2024-09-11T09:51:02.099 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.0701e-05 2024-09-11T09:51:02.099 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.099 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.099 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:02.099 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711619+0000", 2024-09-11T09:51:02.100 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.7610000000000002e-06 2024-09-11T09:51:02.100 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.100 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.100 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:02.100 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.714371+0000", 2024-09-11T09:51:02.100 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0027526260000000002 2024-09-11T09:51:02.100 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.100 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.100 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:02.100 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.714417+0000", 2024-09-11T09:51:02.100 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.5893999999999998e-05 2024-09-11T09:51:02.100 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.100 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.101 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:02.101 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.714496+0000", 2024-09-11T09:51:02.101 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.8539000000000001e-05 2024-09-11T09:51:02.101 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.101 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.101 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:02.101 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.716910+0000", 2024-09-11T09:51:02.101 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0024137640000000001 2024-09-11T09:51:02.101 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.101 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.101 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:02.101 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.716989+0000", 2024-09-11T09:51:02.101 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.9198999999999998e-05 2024-09-11T09:51:02.102 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.102 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.102 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:02.102 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.717016+0000", 2024-09-11T09:51:02.102 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.6699e-05 2024-09-11T09:51:02.102 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.102 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.102 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:02.102 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.717035+0000", 2024-09-11T09:51:02.102 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.9738e-05 2024-09-11T09:51:02.102 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:02.102 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:02.102 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:02.102 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.103 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.103 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:110 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:51:02.103 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.711636+0000", 2024-09-11T09:51:02.103 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 50.372244707999997, 2024-09-11T09:51:02.103 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0067613070000000003, 2024-09-11T09:51:02.103 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:02.103 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:02.103 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:02.103 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:02.103 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:51:02.103 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:51:02.103 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 110 2024-09-11T09:51:02.103 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.104 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:02.104 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.104 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:02.104 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711636+0000", 2024-09-11T09:51:02.104 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:02.104 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.104 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.104 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:02.104 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711636+0000", 2024-09-11T09:51:02.104 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:02.104 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.104 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.104 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:02.105 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711637+0000", 2024-09-11T09:51:02.105 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.9999999999999997e-07 2024-09-11T09:51:02.105 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.105 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.105 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:02.105 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711641+0000", 2024-09-11T09:51:02.105 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.2570000000000001e-06 2024-09-11T09:51:02.105 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.105 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.105 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:02.105 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711651+0000", 2024-09-11T09:51:02.105 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.0285999999999999e-05 2024-09-11T09:51:02.105 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.105 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.106 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:02.106 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711658+0000", 2024-09-11T09:51:02.106 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.3540000000000002e-06 2024-09-11T09:51:02.106 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.106 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.106 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:02.106 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.716475+0000", 2024-09-11T09:51:02.106 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0048162819999999999 2024-09-11T09:51:02.106 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.106 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.106 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:02.106 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.716526+0000", 2024-09-11T09:51:02.106 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.0905000000000001e-05 2024-09-11T09:51:02.107 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.107 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.107 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:02.107 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.716601+0000", 2024-09-11T09:51:02.107 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.5283000000000004e-05 2024-09-11T09:51:02.107 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.107 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.107 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:02.107 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.717933+0000", 2024-09-11T09:51:02.107 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001331622 2024-09-11T09:51:02.107 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.107 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.107 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:02.107 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718332+0000", 2024-09-11T09:51:02.107 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00039909899999999998 2024-09-11T09:51:02.108 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.108 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.108 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:02.108 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718363+0000", 2024-09-11T09:51:02.108 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.1636e-05 2024-09-11T09:51:02.108 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.108 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.108 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:02.108 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718397+0000", 2024-09-11T09:51:02.108 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.3883000000000001e-05 2024-09-11T09:51:02.108 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:02.108 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:02.108 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:02.108 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.109 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.109 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:111 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:51:02.109 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.711668+0000", 2024-09-11T09:51:02.109 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 50.372212277999999, 2024-09-11T09:51:02.109 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0077145570000000004, 2024-09-11T09:51:02.109 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:02.109 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:02.109 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:02.109 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:02.109 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:51:02.109 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:51:02.109 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 111 2024-09-11T09:51:02.109 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.110 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:02.110 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.110 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:02.110 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711668+0000", 2024-09-11T09:51:02.110 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:02.110 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.110 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.110 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:02.110 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711668+0000", 2024-09-11T09:51:02.110 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:02.110 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.110 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.110 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:02.110 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711669+0000", 2024-09-11T09:51:02.110 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.99e-07 2024-09-11T09:51:02.111 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.111 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.111 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:02.111 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711673+0000", 2024-09-11T09:51:02.111 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.7440000000000001e-06 2024-09-11T09:51:02.111 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.111 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.111 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:02.111 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711686+0000", 2024-09-11T09:51:02.111 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.3317e-05 2024-09-11T09:51:02.111 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.111 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.111 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:02.112 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711697+0000", 2024-09-11T09:51:02.112 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1022000000000001e-05 2024-09-11T09:51:02.112 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.112 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.112 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:02.112 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.717065+0000", 2024-09-11T09:51:02.112 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.005367531 2024-09-11T09:51:02.112 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.112 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.112 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:02.112 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.717114+0000", 2024-09-11T09:51:02.112 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.9707999999999999e-05 2024-09-11T09:51:02.112 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.113 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.113 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:02.113 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.717190+0000", 2024-09-11T09:51:02.113 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.5394000000000004e-05 2024-09-11T09:51:02.113 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.113 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.113 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:02.113 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718793+0000", 2024-09-11T09:51:02.113 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001603653 2024-09-11T09:51:02.113 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.113 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.113 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:02.113 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719329+0000", 2024-09-11T09:51:02.114 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00053543799999999995 2024-09-11T09:51:02.114 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.114 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.114 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:02.114 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719359+0000", 2024-09-11T09:51:02.114 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.0420999999999999e-05 2024-09-11T09:51:02.114 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.114 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.114 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:02.114 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719383+0000", 2024-09-11T09:51:02.114 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.3730000000000001e-05 2024-09-11T09:51:02.114 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:02.114 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:02.114 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:02.115 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.115 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.115 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:112 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:51:02.115 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.711712+0000", 2024-09-11T09:51:02.115 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 50.372168717999998, 2024-09-11T09:51:02.115 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.008162084, 2024-09-11T09:51:02.115 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:02.115 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:02.115 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:02.115 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:02.115 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:51:02.115 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:51:02.115 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 112 2024-09-11T09:51:02.115 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.116 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:02.116 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.116 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:02.116 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711712+0000", 2024-09-11T09:51:02.116 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:02.116 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.116 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.116 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:02.116 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711712+0000", 2024-09-11T09:51:02.116 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:02.116 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.116 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.116 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:02.116 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711712+0000", 2024-09-11T09:51:02.117 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.6000000000000004e-07 2024-09-11T09:51:02.117 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.117 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.117 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:02.117 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711808+0000", 2024-09-11T09:51:02.117 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.5977999999999995e-05 2024-09-11T09:51:02.117 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.117 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.117 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:02.117 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711819+0000", 2024-09-11T09:51:02.117 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.0755e-05 2024-09-11T09:51:02.117 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.117 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.117 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:02.118 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711826+0000", 2024-09-11T09:51:02.118 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.7970000000000003e-06 2024-09-11T09:51:02.118 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.118 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.118 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:02.118 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.717435+0000", 2024-09-11T09:51:02.118 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0056090239999999998 2024-09-11T09:51:02.118 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.118 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.118 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:02.118 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.717496+0000", 2024-09-11T09:51:02.118 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.0543e-05 2024-09-11T09:51:02.118 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.118 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.119 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:02.119 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.717591+0000", 2024-09-11T09:51:02.119 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.5667000000000004e-05 2024-09-11T09:51:02.119 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.119 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.119 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:02.119 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718831+0000", 2024-09-11T09:51:02.119 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001240204 2024-09-11T09:51:02.119 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.119 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.119 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:02.119 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719819+0000", 2024-09-11T09:51:02.119 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00098751500000000001 2024-09-11T09:51:02.119 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.120 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.120 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:02.120 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719850+0000", 2024-09-11T09:51:02.120 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.1535999999999998e-05 2024-09-11T09:51:02.120 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.120 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.120 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:02.120 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719874+0000", 2024-09-11T09:51:02.120 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.3504999999999999e-05 2024-09-11T09:51:02.120 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:02.120 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:02.120 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:02.120 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.120 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.121 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:113 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:51:02.121 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.711842+0000", 2024-09-11T09:51:02.121 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 50.372038218999997, 2024-09-11T09:51:02.121 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0081160099999999999, 2024-09-11T09:51:02.121 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:02.121 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:02.121 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:02.121 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:02.121 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:51:02.121 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:51:02.121 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 113 2024-09-11T09:51:02.121 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.121 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:02.121 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.121 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:02.122 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711842+0000", 2024-09-11T09:51:02.122 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:02.122 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.122 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.122 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:02.122 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711842+0000", 2024-09-11T09:51:02.122 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:02.122 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.122 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.122 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:02.122 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711843+0000", 2024-09-11T09:51:02.122 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.4399999999999999e-07 2024-09-11T09:51:02.122 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.122 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.123 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:02.123 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711848+0000", 2024-09-11T09:51:02.123 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.4159999999999997e-06 2024-09-11T09:51:02.123 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.123 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.123 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:02.123 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711856+0000", 2024-09-11T09:51:02.123 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.7870000000000007e-06 2024-09-11T09:51:02.123 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.123 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.123 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:02.123 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711862+0000", 2024-09-11T09:51:02.123 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.3689999999999998e-06 2024-09-11T09:51:02.123 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.124 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.124 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:02.124 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.717965+0000", 2024-09-11T09:51:02.124 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0061028899999999997 2024-09-11T09:51:02.124 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.124 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.124 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:02.124 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718023+0000", 2024-09-11T09:51:02.124 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.8313000000000002e-05 2024-09-11T09:51:02.124 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.124 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.124 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:02.124 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718109+0000", 2024-09-11T09:51:02.124 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.6205000000000006e-05 2024-09-11T09:51:02.125 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.125 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.125 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:02.125 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719808+0000", 2024-09-11T09:51:02.125 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0016990060000000001 2024-09-11T09:51:02.125 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.125 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.125 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:02.125 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719883+0000", 2024-09-11T09:51:02.125 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.4624999999999998e-05 2024-09-11T09:51:02.125 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.125 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.125 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:02.125 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719910+0000", 2024-09-11T09:51:02.126 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.7512e-05 2024-09-11T09:51:02.126 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.126 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.126 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:02.126 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719958+0000", 2024-09-11T09:51:02.126 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.8143000000000003e-05 2024-09-11T09:51:02.126 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:02.126 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:02.126 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:02.126 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.126 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.126 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:51:02.126 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.711887+0000", 2024-09-11T09:51:02.126 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 50.371993605, 2024-09-11T09:51:02.127 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0086063589999999992, 2024-09-11T09:51:02.127 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:02.127 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:02.127 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:02.127 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:02.127 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:51:02.127 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:51:02.127 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 114 2024-09-11T09:51:02.127 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.127 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:02.127 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.127 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:02.127 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711887+0000", 2024-09-11T09:51:02.127 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:02.128 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.128 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.128 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:02.128 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711887+0000", 2024-09-11T09:51:02.128 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:02.128 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.128 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.128 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:02.128 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711887+0000", 2024-09-11T09:51:02.128 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.1500000000000005e-07 2024-09-11T09:51:02.128 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.128 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.128 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:02.128 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711897+0000", 2024-09-11T09:51:02.129 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.1900000000000001e-06 2024-09-11T09:51:02.129 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.129 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.129 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:02.129 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711902+0000", 2024-09-11T09:51:02.129 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.2349999999999999e-06 2024-09-11T09:51:02.129 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.129 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.129 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:02.129 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711906+0000", 2024-09-11T09:51:02.129 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.7129999999999999e-06 2024-09-11T09:51:02.129 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.129 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.129 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:02.130 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718429+0000", 2024-09-11T09:51:02.130 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0065231960000000002 2024-09-11T09:51:02.130 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.130 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.130 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:02.130 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718480+0000", 2024-09-11T09:51:02.130 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.1161e-05 2024-09-11T09:51:02.130 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.130 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.130 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:02.130 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718558+0000", 2024-09-11T09:51:02.130 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.7519000000000001e-05 2024-09-11T09:51:02.130 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.130 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.131 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:02.131 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720038+0000", 2024-09-11T09:51:02.131 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001480016 2024-09-11T09:51:02.131 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.131 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.131 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:02.131 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720417+0000", 2024-09-11T09:51:02.131 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00037943699999999999 2024-09-11T09:51:02.131 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.131 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.131 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:02.131 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720452+0000", 2024-09-11T09:51:02.131 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.4595000000000003e-05 2024-09-11T09:51:02.131 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.131 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.132 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:02.132 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720493+0000", 2024-09-11T09:51:02.132 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.1782000000000001e-05 2024-09-11T09:51:02.132 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:02.132 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:02.132 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:02.132 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.132 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.132 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:51:02.132 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.711912+0000", 2024-09-11T09:51:02.132 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 50.371968936000002, 2024-09-11T09:51:02.132 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0086864750000000008, 2024-09-11T09:51:02.132 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:02.133 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:02.133 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:02.133 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:02.133 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:51:02.133 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:51:02.133 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 115 2024-09-11T09:51:02.133 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.133 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:02.133 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.133 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:02.133 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711912+0000", 2024-09-11T09:51:02.133 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:02.133 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.134 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.134 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:02.134 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711912+0000", 2024-09-11T09:51:02.134 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:02.134 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.134 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.134 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:02.134 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711912+0000", 2024-09-11T09:51:02.134 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.3900000000000001e-07 2024-09-11T09:51:02.134 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.134 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.134 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:02.134 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711969+0000", 2024-09-11T09:51:02.135 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.7501999999999999e-05 2024-09-11T09:51:02.135 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.135 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.135 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:02.135 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711986+0000", 2024-09-11T09:51:02.135 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.6402999999999999e-05 2024-09-11T09:51:02.135 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.135 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.135 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:02.135 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711990+0000", 2024-09-11T09:51:02.135 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.929e-06 2024-09-11T09:51:02.135 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.135 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.135 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:02.136 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718861+0000", 2024-09-11T09:51:02.136 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0068710530000000002 2024-09-11T09:51:02.136 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.136 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.136 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:02.136 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718908+0000", 2024-09-11T09:51:02.136 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.7431e-05 2024-09-11T09:51:02.136 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.136 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.136 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:02.136 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718984+0000", 2024-09-11T09:51:02.136 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.5500999999999994e-05 2024-09-11T09:51:02.136 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.136 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.136 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:02.137 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720500+0000", 2024-09-11T09:51:02.137 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0015158820000000001 2024-09-11T09:51:02.137 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.137 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.137 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:02.137 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720536+0000", 2024-09-11T09:51:02.137 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.6118000000000002e-05 2024-09-11T09:51:02.137 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.137 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.137 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:02.137 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720570+0000", 2024-09-11T09:51:02.137 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.4387999999999997e-05 2024-09-11T09:51:02.138 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.138 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.138 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:02.138 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720598+0000", 2024-09-11T09:51:02.138 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.8028999999999999e-05 2024-09-11T09:51:02.138 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:02.138 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:02.138 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:02.138 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.138 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.138 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:51:02.138 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.711998+0000", 2024-09-11T09:51:02.138 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 50.371882757000002, 2024-09-11T09:51:02.138 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.010785813999999999, 2024-09-11T09:51:02.138 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:02.139 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:02.139 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:02.139 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:02.139 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:51:02.139 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:51:02.139 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 116 2024-09-11T09:51:02.139 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.139 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:02.139 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.139 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:02.139 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711998+0000", 2024-09-11T09:51:02.139 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:02.139 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.139 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.139 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:02.140 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711998+0000", 2024-09-11T09:51:02.140 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:02.140 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.140 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.140 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:02.140 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711998+0000", 2024-09-11T09:51:02.140 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.3300000000000003e-07 2024-09-11T09:51:02.140 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.140 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.140 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:02.140 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712001+0000", 2024-09-11T09:51:02.140 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.9349999999999999e-06 2024-09-11T09:51:02.140 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.140 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.141 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:02.141 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712006+0000", 2024-09-11T09:51:02.141 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.1630000000000004e-06 2024-09-11T09:51:02.141 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.141 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.141 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:02.141 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712010+0000", 2024-09-11T09:51:02.141 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.574e-06 2024-09-11T09:51:02.141 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.141 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.141 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:02.141 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719415+0000", 2024-09-11T09:51:02.141 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0074049099999999998 2024-09-11T09:51:02.141 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.141 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.142 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:02.142 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719465+0000", 2024-09-11T09:51:02.142 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.0129000000000001e-05 2024-09-11T09:51:02.142 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.142 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.142 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:02.142 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719545+0000", 2024-09-11T09:51:02.142 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.0253999999999994e-05 2024-09-11T09:51:02.142 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.142 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.142 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:02.142 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720636+0000", 2024-09-11T09:51:02.142 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0010909330000000001 2024-09-11T09:51:02.142 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.143 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.143 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:02.143 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.722707+0000", 2024-09-11T09:51:02.143 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0020713300000000001 2024-09-11T09:51:02.143 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.143 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.143 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:02.143 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.722743+0000", 2024-09-11T09:51:02.143 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.5351000000000001e-05 2024-09-11T09:51:02.143 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.143 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.143 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:02.143 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.722784+0000", 2024-09-11T09:51:02.143 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.0802e-05 2024-09-11T09:51:02.143 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:02.143 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:02.144 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:02.144 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.144 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.144 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:51:02.144 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.712014+0000", 2024-09-11T09:51:02.144 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 50.371866717000003, 2024-09-11T09:51:02.144 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.013420938, 2024-09-11T09:51:02.144 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:02.144 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:02.144 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:02.144 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:02.144 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:51:02.144 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:51:02.145 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 117 2024-09-11T09:51:02.145 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.145 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:02.145 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.145 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:02.145 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712014+0000", 2024-09-11T09:51:02.145 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:02.145 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.145 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.145 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:02.145 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712014+0000", 2024-09-11T09:51:02.145 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:02.145 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.146 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.146 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:02.146 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712014+0000", 2024-09-11T09:51:02.146 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.2000000000000001e-07 2024-09-11T09:51:02.146 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.146 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.146 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:02.146 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712016+0000", 2024-09-11T09:51:02.146 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.2309999999999999e-06 2024-09-11T09:51:02.146 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.146 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.146 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:02.146 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712020+0000", 2024-09-11T09:51:02.146 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.3830000000000001e-06 2024-09-11T09:51:02.146 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.147 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.147 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:02.147 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712023+0000", 2024-09-11T09:51:02.147 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.3340000000000002e-06 2024-09-11T09:51:02.147 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.147 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.147 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:02.147 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720083+0000", 2024-09-11T09:51:02.147 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0080601189999999993 2024-09-11T09:51:02.147 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.147 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.147 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:02.148 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720125+0000", 2024-09-11T09:51:02.148 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.2029e-05 2024-09-11T09:51:02.148 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.148 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.148 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:02.148 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720205+0000", 2024-09-11T09:51:02.148 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.0010000000000001e-05 2024-09-11T09:51:02.148 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.148 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.148 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:02.148 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720849+0000", 2024-09-11T09:51:02.148 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00064369899999999996 2024-09-11T09:51:02.148 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.149 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.149 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:02.149 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.725357+0000", 2024-09-11T09:51:02.149 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0045077379999999998 2024-09-11T09:51:02.149 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.149 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.149 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:02.149 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.725392+0000", 2024-09-11T09:51:02.149 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.5862999999999999e-05 2024-09-11T09:51:02.149 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.149 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.149 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:02.149 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.725435+0000", 2024-09-11T09:51:02.149 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.2311999999999997e-05 2024-09-11T09:51:02.150 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:02.150 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:02.150 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:02.150 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.150 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.150 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:119 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 131072~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:51:02.150 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.725673+0000", 2024-09-11T09:51:02.150 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 50.358208015000002, 2024-09-11T09:51:02.150 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0053927460000000003, 2024-09-11T09:51:02.150 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:02.150 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:02.150 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:02.150 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:02.150 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:51:02.151 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:51:02.151 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 119 2024-09-11T09:51:02.151 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.151 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:02.151 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.151 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:02.151 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.725673+0000", 2024-09-11T09:51:02.151 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:02.151 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.151 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.151 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:02.151 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.725673+0000", 2024-09-11T09:51:02.151 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:02.152 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.152 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.152 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:02.152 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.725673+0000", 2024-09-11T09:51:02.152 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.68e-07 2024-09-11T09:51:02.152 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.152 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.152 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:02.152 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.725720+0000", 2024-09-11T09:51:02.152 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.6526999999999997e-05 2024-09-11T09:51:02.152 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.152 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.152 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:02.152 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.725734+0000", 2024-09-11T09:51:02.153 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.4593000000000001e-05 2024-09-11T09:51:02.153 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.153 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.153 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:02.153 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.725741+0000", 2024-09-11T09:51:02.153 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.28e-06 2024-09-11T09:51:02.153 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.153 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.153 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:02.153 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.726290+0000", 2024-09-11T09:51:02.153 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00054892000000000005 2024-09-11T09:51:02.153 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.153 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.154 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:02.154 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.726338+0000", 2024-09-11T09:51:02.154 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.7982000000000002e-05 2024-09-11T09:51:02.154 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.154 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.154 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:02.154 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.726413+0000", 2024-09-11T09:51:02.154 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.5679000000000005e-05 2024-09-11T09:51:02.154 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.154 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.154 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:02.154 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.727555+0000", 2024-09-11T09:51:02.154 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0011420289999999999 2024-09-11T09:51:02.154 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.154 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.155 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:02.155 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.731004+0000", 2024-09-11T09:51:02.155 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0034484279999999999 2024-09-11T09:51:02.155 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.155 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.155 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:02.155 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.731040+0000", 2024-09-11T09:51:02.155 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.6315e-05 2024-09-11T09:51:02.155 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.155 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.155 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:02.155 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.731065+0000", 2024-09-11T09:51:02.155 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.5324999999999999e-05 2024-09-11T09:51:02.155 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:02.156 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:02.156 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:02.156 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.156 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.156 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:120 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 196608~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:51:02.156 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.725756+0000", 2024-09-11T09:51:02.156 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 50.358124105999998, 2024-09-11T09:51:02.156 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0053569100000000003, 2024-09-11T09:51:02.156 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:02.156 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:02.156 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:02.156 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:02.156 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:51:02.156 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:51:02.156 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 120 2024-09-11T09:51:02.157 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.157 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:02.157 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.157 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:02.157 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.725756+0000", 2024-09-11T09:51:02.157 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:02.157 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.157 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.157 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:02.157 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.725756+0000", 2024-09-11T09:51:02.157 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:02.157 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.157 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.157 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:02.158 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.725757+0000", 2024-09-11T09:51:02.158 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.3799999999999997e-07 2024-09-11T09:51:02.158 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.158 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.158 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:02.158 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.725802+0000", 2024-09-11T09:51:02.158 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.4790000000000003e-05 2024-09-11T09:51:02.158 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.158 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.158 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:02.158 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.725815+0000", 2024-09-11T09:51:02.158 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.2609000000000001e-05 2024-09-11T09:51:02.158 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.158 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.158 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:02.159 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.725822+0000", 2024-09-11T09:51:02.159 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.6450000000000005e-06 2024-09-11T09:51:02.159 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.159 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.159 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:02.159 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.726708+0000", 2024-09-11T09:51:02.159 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00088606500000000001 2024-09-11T09:51:02.159 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.159 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.159 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:02.159 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.726755+0000", 2024-09-11T09:51:02.159 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.6888000000000002e-05 2024-09-11T09:51:02.159 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.159 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.160 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:02.160 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.726828+0000", 2024-09-11T09:51:02.160 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.2510999999999997e-05 2024-09-11T09:51:02.160 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.160 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.160 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:02.160 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.727829+0000", 2024-09-11T09:51:02.160 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001001093 2024-09-11T09:51:02.160 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.160 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.160 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:02.160 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.731071+0000", 2024-09-11T09:51:02.160 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0032426009999999999 2024-09-11T09:51:02.160 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.160 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.161 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:02.161 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.731094+0000", 2024-09-11T09:51:02.161 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.2677e-05 2024-09-11T09:51:02.161 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.161 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.161 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:02.161 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.731113+0000", 2024-09-11T09:51:02.161 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.9392999999999999e-05 2024-09-11T09:51:02.161 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:02.161 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:02.161 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:02.161 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.161 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.161 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:138 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:51:02.162 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:12.034038+0000", 2024-09-11T09:51:02.162 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 50.049842775999998, 2024-09-11T09:51:02.162 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0054142749999999996, 2024-09-11T09:51:02.162 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:02.162 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:02.162 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:02.162 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:02.162 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:51:02.162 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:51:02.162 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 138 2024-09-11T09:51:02.162 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.162 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:02.162 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.162 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:02.163 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034038+0000", 2024-09-11T09:51:02.163 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:02.163 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.163 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.163 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:02.163 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034038+0000", 2024-09-11T09:51:02.163 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:02.163 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.163 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.163 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:02.163 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034038+0000", 2024-09-11T09:51:02.163 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.3499999999999996e-07 2024-09-11T09:51:02.163 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.163 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.163 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:02.164 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034043+0000", 2024-09-11T09:51:02.164 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.7310000000000001e-06 2024-09-11T09:51:02.164 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.164 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.164 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:02.164 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034050+0000", 2024-09-11T09:51:02.164 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.0419999999999997e-06 2024-09-11T09:51:02.164 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.164 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.164 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:02.164 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034055+0000", 2024-09-11T09:51:02.164 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.3299999999999998e-06 2024-09-11T09:51:02.164 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.164 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.165 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:02.165 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.037309+0000", 2024-09-11T09:51:02.165 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0032535440000000001 2024-09-11T09:51:02.165 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.165 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.165 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:02.165 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.037357+0000", 2024-09-11T09:51:02.165 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.8167999999999997e-05 2024-09-11T09:51:02.165 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.165 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.165 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:02.165 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.037430+0000", 2024-09-11T09:51:02.165 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.3275999999999994e-05 2024-09-11T09:51:02.165 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.166 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.166 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:02.166 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.039014+0000", 2024-09-11T09:51:02.166 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0015831350000000001 2024-09-11T09:51:02.166 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.166 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.166 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:02.166 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.039401+0000", 2024-09-11T09:51:02.166 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00038758500000000002 2024-09-11T09:51:02.166 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.166 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.166 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:02.166 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.039429+0000", 2024-09-11T09:51:02.166 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.7379e-05 2024-09-11T09:51:02.167 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.167 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.167 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:02.167 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.039452+0000", 2024-09-11T09:51:02.167 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.355e-05 2024-09-11T09:51:02.167 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:02.167 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:02.167 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:02.167 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.167 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.167 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:139 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:51:02.167 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:12.034064+0000", 2024-09-11T09:51:02.167 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 50.049816745000001, 2024-09-11T09:51:02.168 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0054736200000000002, 2024-09-11T09:51:02.168 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:02.168 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:02.168 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:02.168 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:02.168 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:51:02.168 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:51:02.168 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 139 2024-09-11T09:51:02.168 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.168 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:02.168 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.168 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:02.168 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034064+0000", 2024-09-11T09:51:02.168 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:02.169 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.169 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.169 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:02.169 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034064+0000", 2024-09-11T09:51:02.169 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:02.169 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.169 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.169 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:02.169 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034064+0000", 2024-09-11T09:51:02.169 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.89e-07 2024-09-11T09:51:02.169 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.169 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.169 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:02.170 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034069+0000", 2024-09-11T09:51:02.170 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.9799999999999998e-06 2024-09-11T09:51:02.170 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.170 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.170 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:02.170 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034079+0000", 2024-09-11T09:51:02.170 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.3549999999999994e-06 2024-09-11T09:51:02.170 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.170 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.170 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:02.170 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034084+0000", 2024-09-11T09:51:02.170 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.2209999999999996e-06 2024-09-11T09:51:02.170 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.170 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.170 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:02.171 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.038176+0000", 2024-09-11T09:51:02.171 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0040921610000000004 2024-09-11T09:51:02.171 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.171 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.171 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:02.171 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.038221+0000", 2024-09-11T09:51:02.171 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.5006000000000002e-05 2024-09-11T09:51:02.171 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.171 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.172 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:02.172 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.038293+0000", 2024-09-11T09:51:02.172 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.1952999999999994e-05 2024-09-11T09:51:02.172 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.172 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.172 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:02.172 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.039461+0000", 2024-09-11T09:51:02.172 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0011680340000000001 2024-09-11T09:51:02.172 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.172 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.172 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:02.172 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.039487+0000", 2024-09-11T09:51:02.172 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.5752e-05 2024-09-11T09:51:02.172 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.173 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.173 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:02.173 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.039515+0000", 2024-09-11T09:51:02.173 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.7763000000000002e-05 2024-09-11T09:51:02.173 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.173 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.173 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:02.173 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.039537+0000", 2024-09-11T09:51:02.173 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.2906000000000001e-05 2024-09-11T09:51:02.173 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:02.173 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:02.173 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:02.173 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.174 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.174 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:140 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:51:02.174 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:12.034093+0000", 2024-09-11T09:51:02.174 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 50.049787240999997, 2024-09-11T09:51:02.174 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0059112490000000004, 2024-09-11T09:51:02.174 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:02.174 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:02.174 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:02.174 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:02.174 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:51:02.174 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:51:02.174 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 140 2024-09-11T09:51:02.174 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.175 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:02.175 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.175 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:02.175 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034093+0000", 2024-09-11T09:51:02.175 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:02.175 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.175 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.175 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:02.175 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034093+0000", 2024-09-11T09:51:02.175 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:02.175 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.175 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.175 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:02.175 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034094+0000", 2024-09-11T09:51:02.175 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.4299999999999998e-07 2024-09-11T09:51:02.176 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.176 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.176 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:02.176 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034159+0000", 2024-09-11T09:51:02.176 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.567e-05 2024-09-11T09:51:02.176 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.176 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.176 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:02.176 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034167+0000", 2024-09-11T09:51:02.176 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.8490000000000006e-06 2024-09-11T09:51:02.176 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.176 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.176 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:02.176 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034173+0000", 2024-09-11T09:51:02.177 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.1770000000000003e-06 2024-09-11T09:51:02.177 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.177 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.177 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:02.177 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.038543+0000", 2024-09-11T09:51:02.177 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0043699150000000003 2024-09-11T09:51:02.177 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.177 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.177 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:02.177 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.038589+0000", 2024-09-11T09:51:02.177 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.5562999999999997e-05 2024-09-11T09:51:02.177 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.177 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.177 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:02.178 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.038662+0000", 2024-09-11T09:51:02.178 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.2748000000000002e-05 2024-09-11T09:51:02.178 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.178 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.178 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:02.178 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.039635+0000", 2024-09-11T09:51:02.178 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00097310799999999998 2024-09-11T09:51:02.178 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.178 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.178 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:02.178 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.039945+0000", 2024-09-11T09:51:02.178 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00031054100000000001 2024-09-11T09:51:02.178 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.178 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.178 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:02.179 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.039971+0000", 2024-09-11T09:51:02.179 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.5423999999999999e-05 2024-09-11T09:51:02.179 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.179 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.179 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:02.179 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.040005+0000", 2024-09-11T09:51:02.179 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.3810999999999999e-05 2024-09-11T09:51:02.179 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:02.179 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:02.179 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:02.179 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.179 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.179 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:51:02.180 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:12.034184+0000", 2024-09-11T09:51:02.180 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 50.049696543000003, 2024-09-11T09:51:02.180 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0061520380000000003, 2024-09-11T09:51:02.180 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:02.180 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:02.180 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:02.180 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:02.180 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:51:02.180 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:51:02.180 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 141 2024-09-11T09:51:02.180 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.180 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:02.180 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.180 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:02.181 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034184+0000", 2024-09-11T09:51:02.181 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:02.181 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.181 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.181 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:02.181 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034184+0000", 2024-09-11T09:51:02.181 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:02.181 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.181 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.181 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:02.181 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034185+0000", 2024-09-11T09:51:02.181 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.1099999999999995e-07 2024-09-11T09:51:02.181 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.182 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.182 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:02.182 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034190+0000", 2024-09-11T09:51:02.182 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.7309999999999998e-06 2024-09-11T09:51:02.182 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.182 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.182 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:02.182 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034197+0000", 2024-09-11T09:51:02.182 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.7440000000000001e-06 2024-09-11T09:51:02.182 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.182 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.182 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:02.182 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034204+0000", 2024-09-11T09:51:02.182 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.8970000000000002e-06 2024-09-11T09:51:02.182 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.183 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.183 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:02.183 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.039035+0000", 2024-09-11T09:51:02.183 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0048307039999999999 2024-09-11T09:51:02.183 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.183 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.183 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:02.183 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.039093+0000", 2024-09-11T09:51:02.183 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.8604000000000003e-05 2024-09-11T09:51:02.183 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.183 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.183 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:02.183 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.039184+0000", 2024-09-11T09:51:02.183 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.0196000000000002e-05 2024-09-11T09:51:02.183 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.184 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.184 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:02.184 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.039873+0000", 2024-09-11T09:51:02.184 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00068960600000000005 2024-09-11T09:51:02.184 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.184 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.184 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:02.184 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.040292+0000", 2024-09-11T09:51:02.184 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00041865199999999999 2024-09-11T09:51:02.184 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.184 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.184 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:02.184 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.040317+0000", 2024-09-11T09:51:02.185 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.5439000000000002e-05 2024-09-11T09:51:02.185 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.185 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.185 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:02.185 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.040336+0000", 2024-09-11T09:51:02.185 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.8754e-05 2024-09-11T09:51:02.185 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:02.185 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:02.185 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:02.185 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.185 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.185 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:157 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:51:02.185 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:12.452040+0000", 2024-09-11T09:51:02.185 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 49.631840689999997, 2024-09-11T09:51:02.186 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0068848269999999996, 2024-09-11T09:51:02.186 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:02.186 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:02.186 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:02.186 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:02.186 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:51:02.186 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:51:02.186 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 157 2024-09-11T09:51:02.186 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.186 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:02.186 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.186 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:02.186 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452040+0000", 2024-09-11T09:51:02.186 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:02.187 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.187 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.187 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:02.187 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452040+0000", 2024-09-11T09:51:02.187 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:02.187 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.187 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.187 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:02.187 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452040+0000", 2024-09-11T09:51:02.187 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.22e-07 2024-09-11T09:51:02.187 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.187 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.187 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:02.187 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452042+0000", 2024-09-11T09:51:02.188 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.7010000000000001e-06 2024-09-11T09:51:02.188 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.188 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.188 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:02.188 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452045+0000", 2024-09-11T09:51:02.188 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.1310000000000002e-06 2024-09-11T09:51:02.188 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.188 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.188 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:02.188 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452048+0000", 2024-09-11T09:51:02.188 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.65e-06 2024-09-11T09:51:02.188 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.188 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.189 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:02.189 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.454453+0000", 2024-09-11T09:51:02.189 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0024052959999999999 2024-09-11T09:51:02.189 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.189 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.189 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:02.189 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.454498+0000", 2024-09-11T09:51:02.189 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.4978999999999997e-05 2024-09-11T09:51:02.189 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.189 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.189 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:02.189 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.454567+0000", 2024-09-11T09:51:02.189 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.9110999999999996e-05 2024-09-11T09:51:02.190 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.190 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.190 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:02.190 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.455464+0000", 2024-09-11T09:51:02.190 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00089696199999999995 2024-09-11T09:51:02.190 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.190 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.190 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:02.190 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.458844+0000", 2024-09-11T09:51:02.190 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0033802469999999999 2024-09-11T09:51:02.190 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.190 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.190 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:02.191 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.458889+0000", 2024-09-11T09:51:02.191 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.4441999999999999e-05 2024-09-11T09:51:02.191 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.191 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.191 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:02.191 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.458925+0000", 2024-09-11T09:51:02.191 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.6086e-05 2024-09-11T09:51:02.191 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:02.191 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:02.191 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:02.191 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.191 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.191 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:158 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:51:02.191 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:12.452055+0000", 2024-09-11T09:51:02.191 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 49.631825296999999, 2024-09-11T09:51:02.192 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0069533829999999996, 2024-09-11T09:51:02.192 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:02.192 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:02.192 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:02.192 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:02.192 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:51:02.192 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:51:02.192 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 158 2024-09-11T09:51:02.192 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.192 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:02.192 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.192 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:02.192 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452055+0000", 2024-09-11T09:51:02.192 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:02.192 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.193 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.193 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:02.193 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452055+0000", 2024-09-11T09:51:02.193 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:02.193 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.193 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.193 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:02.193 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452056+0000", 2024-09-11T09:51:02.193 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.48e-07 2024-09-11T09:51:02.193 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.193 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.193 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:02.193 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452087+0000", 2024-09-11T09:51:02.193 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.1149000000000003e-05 2024-09-11T09:51:02.194 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.194 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.194 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:02.194 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452092+0000", 2024-09-11T09:51:02.194 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.8400000000000002e-06 2024-09-11T09:51:02.194 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.194 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.194 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:02.194 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452095+0000", 2024-09-11T09:51:02.194 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.0740000000000001e-06 2024-09-11T09:51:02.194 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.194 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.194 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:02.195 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.454796+0000", 2024-09-11T09:51:02.195 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0027009740000000001 2024-09-11T09:51:02.195 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.195 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.195 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:02.195 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.454869+0000", 2024-09-11T09:51:02.195 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.3678999999999997e-05 2024-09-11T09:51:02.195 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.195 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.195 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:02.195 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.454992+0000", 2024-09-11T09:51:02.195 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00012279000000000001 2024-09-11T09:51:02.195 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.196 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.196 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:02.196 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.456072+0000", 2024-09-11T09:51:02.196 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001080229 2024-09-11T09:51:02.196 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.196 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.196 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:02.196 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.458931+0000", 2024-09-11T09:51:02.196 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0028591789999999999 2024-09-11T09:51:02.196 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.196 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.196 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:02.196 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.458972+0000", 2024-09-11T09:51:02.196 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.0170999999999999e-05 2024-09-11T09:51:02.196 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.197 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.197 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:02.197 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.459009+0000", 2024-09-11T09:51:02.197 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.7049999999999999e-05 2024-09-11T09:51:02.197 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:02.197 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:02.197 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:02.197 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.197 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.197 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:159 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:51:02.197 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:12.452099+0000", 2024-09-11T09:51:02.197 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 49.631781128999997, 2024-09-11T09:51:02.197 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0069885950000000002, 2024-09-11T09:51:02.198 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:02.198 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:02.198 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:02.198 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:02.198 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:51:02.198 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:51:02.198 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 159 2024-09-11T09:51:02.198 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.198 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:02.198 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.198 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:02.198 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452099+0000", 2024-09-11T09:51:02.198 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:02.198 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.198 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.199 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:02.199 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452099+0000", 2024-09-11T09:51:02.199 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:02.199 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.199 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.199 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:02.199 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452100+0000", 2024-09-11T09:51:02.199 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.2800000000000003e-07 2024-09-11T09:51:02.199 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.199 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.199 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:02.199 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452102+0000", 2024-09-11T09:51:02.199 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.824e-06 2024-09-11T09:51:02.199 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.200 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.200 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:02.200 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452105+0000", 2024-09-11T09:51:02.200 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.0460000000000001e-06 2024-09-11T09:51:02.200 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.200 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.200 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:02.200 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452107+0000", 2024-09-11T09:51:02.200 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.543e-06 2024-09-11T09:51:02.200 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.200 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.200 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:02.200 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.455484+0000", 2024-09-11T09:51:02.200 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0033769759999999998 2024-09-11T09:51:02.201 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.201 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.201 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:02.201 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.455527+0000", 2024-09-11T09:51:02.201 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.2589999999999997e-05 2024-09-11T09:51:02.201 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.201 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.201 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:02.201 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.455594+0000", 2024-09-11T09:51:02.201 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.6775999999999999e-05 2024-09-11T09:51:02.201 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.201 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.201 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:02.201 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.456312+0000", 2024-09-11T09:51:02.202 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00071874700000000005 2024-09-11T09:51:02.202 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.202 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.202 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:02.202 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.459015+0000", 2024-09-11T09:51:02.202 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0027028540000000002 2024-09-11T09:51:02.202 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.202 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.202 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:02.202 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.459054+0000", 2024-09-11T09:51:02.202 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.8840000000000001e-05 2024-09-11T09:51:02.202 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:02.202 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:02.202 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:02.203 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.459088+0000", 2024-09-11T09:51:02.203 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.4071e-05 2024-09-11T09:51:02.203 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:02.203 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:02.203 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:02.203 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:02.203 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:02.203 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-11T09:51:02.204 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T09:51:02.205 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-11T09:51:02.205 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:51:02.205+0000 7fdec5849640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31907) UID: 0 2024-09-11T09:51:02.305 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:51:02.306+0000 7f3c96ec9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31909) UID: 0 2024-09-11T09:51:02.333 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-11T09:51:02.333 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T09:51:02.350 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-11T09:51:02.350 INFO:teuthology.orchestra.run.smithi096.stdout: "ops": [], 2024-09-11T09:51:02.350 INFO:teuthology.orchestra.run.smithi096.stdout: "num_ops": 0 2024-09-11T09:51:02.350 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-11T09:51:02.350 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-11T09:51:02.351 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T09:51:02.358 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-11T09:51:02.366 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-11T09:51:02.366 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T09:51:02.383 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-11T09:51:02.383 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T09:51:02.400 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-11T09:51:02.400 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T09:51:02.417 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-11T09:51:02.417 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T09:51:02.443 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:51:02.443+0000 7fdec5849640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31907) UID: 0 2024-09-11T09:51:02.500 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:51:02.501 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:51:02.507 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:51:02.508+0000 7f3c96ec9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31909) UID: 0 2024-09-11T09:51:02.509 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-11T09:51:02.608 INFO:tasks.ceph.osd.2.smithi017.stderr:2024-09-11T09:51:02.609+0000 7fd20b170640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31672) UID: 0 2024-09-11T09:51:02.652 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-11T09:51:02.654 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:51:02.654 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:51:02.663 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-11T09:51:02.709 INFO:tasks.ceph.osd.0.smithi017.stderr:2024-09-11T09:51:02.710+0000 7fb73c0f1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31665) UID: 0 2024-09-11T09:51:02.766 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:51:02.766 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:51:02.768 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 5, 0] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5] 2024-09-11T09:51:02.768 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-11T09:51:02.768 INFO:tasks.thrashosds.thrasher:inject_pause on osd.2 2024-09-11T09:51:02.768 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-11T09:51:02.768 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-11T09:51:02.768 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok config set filestore_inject_stall 3 2024-09-11T09:51:02.775 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-11T09:51:02.813 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:51:02.814+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:51:02.876 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:51:02.876 INFO:teuthology.orchestra.run.smithi017.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-11T09:51:02.876 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:51:02.878 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:51:02.878 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:51:02.886 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-11T09:51:02.930 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:51:02.931+0000 7f3c96ec9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31909) UID: 0 2024-09-11T09:51:02.989 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:51:02.989 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:51:02.998 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-11T09:51:03.039 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-09-11T09:51:03.040+0000 7f7b6c7d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31904) UID: 0 2024-09-11T09:51:03.105 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:51:03.105 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:51:03.111 INFO:tasks.ceph.osd.2.smithi017.stderr:2024-09-11T09:51:03.112+0000 7fd20b170640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31672) UID: 0 2024-09-11T09:51:03.114 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-11T09:51:03.212 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:51:03.213+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:51:03.261 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:51:03.261 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:51:03.270 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-11T09:51:03.313 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:51:03.314+0000 7f3c96ec9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31909) UID: 0 2024-09-11T09:51:03.414 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-09-11T09:51:03.414+0000 7f7b6c7d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31904) UID: 0 2024-09-11T09:51:03.418 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:51:03.418 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:51:03.426 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-11T09:51:03.515 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:51:03.516+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:51:03.533 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:51:03.534 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:51:03.542 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-11T09:51:03.616 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:51:03.617+0000 7fdec5849640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31907) UID: 0 2024-09-11T09:51:03.643 INFO:teuthology.orchestra.run.smithi017.stderr:noscrub is set 2024-09-11T09:51:03.691 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:51:03.691 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:51:03.700 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-11T09:51:03.741 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:51:03.742+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:51:03.819 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:51:03.820+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:51:03.847 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:51:03.847 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:51:03.856 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-11T09:51:03.920 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:51:03.921+0000 7f3c96ec9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31909) UID: 0 2024-09-11T09:51:04.005 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:51:04.006 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:51:04.014 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-11T09:51:04.058 INFO:tasks.ceph.osd.0.smithi017.stderr:2024-09-11T09:51:04.059+0000 7fb73c0f1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31665) UID: 0 2024-09-11T09:51:04.121 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:51:04.121 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:51:04.122 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:51:04.122+0000 7fdec5849640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31907) UID: 0 2024-09-11T09:51:04.130 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-11T09:51:04.223 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:51:04.224+0000 7f3c96ec9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31909) UID: 0 2024-09-11T09:51:04.279 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:51:04.279 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:51:04.290 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-11T09:51:04.324 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:51:04.325+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:51:04.396 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:51:04.397 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:51:04.405 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-11T09:51:04.430 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T09:51:04.449 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:51:04.450+0000 7f3c96ec9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31909) UID: 0 2024-09-11T09:51:04.508 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:51:04.508 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:51:04.517 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-11T09:51:04.526 INFO:tasks.ceph.osd.2.smithi017.stderr:2024-09-11T09:51:04.527+0000 7fd20b170640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31672) UID: 0 2024-09-11T09:51:04.552 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-11T09:51:04.553 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T09:51:04.570 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-11T09:51:04.570 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T09:51:04.586 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-11T09:51:04.587 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T09:51:04.603 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-11T09:51:04.603 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T09:51:04.620 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-11T09:51:04.620 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T09:51:04.627 INFO:tasks.ceph.osd.0.smithi017.stderr:2024-09-11T09:51:04.628+0000 7fb73c0f1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31665) UID: 0 2024-09-11T09:51:04.637 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-11T09:51:04.637 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T09:51:04.664 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-11T09:51:04.664 INFO:teuthology.orchestra.run.smithi096.stdout: "size": 20, 2024-09-11T09:51:04.665 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 600, 2024-09-11T09:51:04.665 INFO:teuthology.orchestra.run.smithi096.stdout: "ops": [ 2024-09-11T09:51:04.665 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.665 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:74 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:51:04.665 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.539067+0000", 2024-09-11T09:51:04.665 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 53.124058408000003, 2024-09-11T09:51:04.666 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.006945957, 2024-09-11T09:51:04.666 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:04.666 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:04.666 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:04.666 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:04.666 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:51:04.666 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:51:04.667 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 74 2024-09-11T09:51:04.667 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.667 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:04.667 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.667 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:04.667 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.539067+0000", 2024-09-11T09:51:04.667 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:04.668 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.668 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.668 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:04.668 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.539067+0000", 2024-09-11T09:51:04.668 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:04.668 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.668 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.669 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:04.669 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.539068+0000", 2024-09-11T09:51:04.669 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.6000000000000003e-07 2024-09-11T09:51:04.669 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.669 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.669 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:04.670 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.539072+0000", 2024-09-11T09:51:04.670 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.0450000000000002e-06 2024-09-11T09:51:04.670 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.670 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.670 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:04.670 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.539078+0000", 2024-09-11T09:51:04.670 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.6889999999999998e-06 2024-09-11T09:51:04.671 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.671 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.671 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:04.671 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.539083+0000", 2024-09-11T09:51:04.671 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.7180000000000004e-06 2024-09-11T09:51:04.671 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.671 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.671 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:04.672 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.540230+0000", 2024-09-11T09:51:04.672 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0011466250000000001 2024-09-11T09:51:04.672 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.672 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.672 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:04.672 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.540279+0000", 2024-09-11T09:51:04.672 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.9667999999999999e-05 2024-09-11T09:51:04.673 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.673 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.674 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:04.674 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.540357+0000", 2024-09-11T09:51:04.674 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.7863999999999999e-05 2024-09-11T09:51:04.674 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.675 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.675 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:04.675 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.541086+0000", 2024-09-11T09:51:04.675 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00072921100000000001 2024-09-11T09:51:04.675 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.675 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.675 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:04.676 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.545949+0000", 2024-09-11T09:51:04.676 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0048626040000000004 2024-09-11T09:51:04.676 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.677 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.677 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:04.677 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.545989+0000", 2024-09-11T09:51:04.677 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.0259000000000002e-05 2024-09-11T09:51:04.677 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.678 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.678 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:04.678 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.546013+0000", 2024-09-11T09:51:04.678 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.3614000000000001e-05 2024-09-11T09:51:04.678 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:04.678 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:04.679 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:04.679 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.679 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.679 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:108 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:51:04.680 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.711510+0000", 2024-09-11T09:51:04.680 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 52.951615236000002, 2024-09-11T09:51:04.680 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0054615649999999998, 2024-09-11T09:51:04.680 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:04.680 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:04.681 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:04.681 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:04.681 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:51:04.681 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:51:04.681 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 108 2024-09-11T09:51:04.681 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.681 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:04.682 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.682 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:04.682 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711510+0000", 2024-09-11T09:51:04.682 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:04.682 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.682 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.682 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:04.682 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711510+0000", 2024-09-11T09:51:04.683 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:04.683 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.683 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.683 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:04.683 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711511+0000", 2024-09-11T09:51:04.683 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.1999999999999998e-07 2024-09-11T09:51:04.683 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.683 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.684 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:04.684 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711516+0000", 2024-09-11T09:51:04.684 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.4140000000000002e-06 2024-09-11T09:51:04.684 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.684 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.684 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:04.684 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711527+0000", 2024-09-11T09:51:04.685 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.0169e-05 2024-09-11T09:51:04.685 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.685 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.685 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:04.685 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711535+0000", 2024-09-11T09:51:04.685 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.3299999999999999e-06 2024-09-11T09:51:04.685 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.686 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.686 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:04.686 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.713763+0000", 2024-09-11T09:51:04.686 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0022281079999999999 2024-09-11T09:51:04.686 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.687 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.687 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:04.687 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.713812+0000", 2024-09-11T09:51:04.687 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.8544000000000001e-05 2024-09-11T09:51:04.687 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.687 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.688 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:04.688 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.713909+0000", 2024-09-11T09:51:04.688 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.7113000000000003e-05 2024-09-11T09:51:04.688 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.688 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.688 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:04.688 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.716873+0000", 2024-09-11T09:51:04.688 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0029642710000000001 2024-09-11T09:51:04.689 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.689 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.689 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:04.689 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.716918+0000", 2024-09-11T09:51:04.689 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.5447e-05 2024-09-11T09:51:04.689 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.690 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.690 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:04.690 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.716947+0000", 2024-09-11T09:51:04.690 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.8181e-05 2024-09-11T09:51:04.690 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.690 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.690 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:04.690 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.716972+0000", 2024-09-11T09:51:04.690 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.5068000000000001e-05 2024-09-11T09:51:04.690 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:04.691 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:04.691 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:04.691 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.691 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.691 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:109 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:51:04.691 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.711555+0000", 2024-09-11T09:51:04.691 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 52.951569986999999, 2024-09-11T09:51:04.691 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.005479902, 2024-09-11T09:51:04.691 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:04.691 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:04.691 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:04.691 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:04.692 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:51:04.692 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:51:04.692 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 109 2024-09-11T09:51:04.692 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.692 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:04.692 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.692 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:04.692 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711555+0000", 2024-09-11T09:51:04.692 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:04.692 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.692 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.692 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:04.692 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711555+0000", 2024-09-11T09:51:04.693 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:04.693 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.693 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.693 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:04.693 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711556+0000", 2024-09-11T09:51:04.693 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.3200000000000005e-07 2024-09-11T09:51:04.693 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.693 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.693 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:04.693 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711601+0000", 2024-09-11T09:51:04.693 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.5349000000000002e-05 2024-09-11T09:51:04.693 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.694 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.694 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:04.694 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711612+0000", 2024-09-11T09:51:04.694 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.0701e-05 2024-09-11T09:51:04.694 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.694 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.694 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:04.694 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711619+0000", 2024-09-11T09:51:04.694 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.7610000000000002e-06 2024-09-11T09:51:04.694 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.694 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.694 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:04.695 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.714371+0000", 2024-09-11T09:51:04.695 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0027526260000000002 2024-09-11T09:51:04.695 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.695 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.695 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:04.695 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.714417+0000", 2024-09-11T09:51:04.695 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.5893999999999998e-05 2024-09-11T09:51:04.695 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.695 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.695 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:04.695 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.714496+0000", 2024-09-11T09:51:04.695 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.8539000000000001e-05 2024-09-11T09:51:04.695 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.696 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.696 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:04.696 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.716910+0000", 2024-09-11T09:51:04.696 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0024137640000000001 2024-09-11T09:51:04.696 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.696 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.696 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:04.696 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.716989+0000", 2024-09-11T09:51:04.696 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.9198999999999998e-05 2024-09-11T09:51:04.696 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.696 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.696 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:04.696 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.717016+0000", 2024-09-11T09:51:04.697 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.6699e-05 2024-09-11T09:51:04.697 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.697 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.697 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:04.697 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.717035+0000", 2024-09-11T09:51:04.697 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.9738e-05 2024-09-11T09:51:04.697 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:04.697 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:04.697 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:04.697 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.697 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.697 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:110 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:51:04.697 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.711636+0000", 2024-09-11T09:51:04.698 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 52.951489527, 2024-09-11T09:51:04.698 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0067613070000000003, 2024-09-11T09:51:04.698 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:04.698 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:04.698 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:04.698 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:04.698 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:51:04.698 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:51:04.698 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 110 2024-09-11T09:51:04.698 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.698 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:04.698 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.698 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:04.699 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711636+0000", 2024-09-11T09:51:04.699 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:04.699 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.699 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.699 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:04.699 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711636+0000", 2024-09-11T09:51:04.699 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:04.699 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.699 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.699 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:04.699 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711637+0000", 2024-09-11T09:51:04.699 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.9999999999999997e-07 2024-09-11T09:51:04.699 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.700 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.700 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:04.700 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711641+0000", 2024-09-11T09:51:04.700 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.2570000000000001e-06 2024-09-11T09:51:04.700 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.700 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.700 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:04.700 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711651+0000", 2024-09-11T09:51:04.700 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.0285999999999999e-05 2024-09-11T09:51:04.700 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.700 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.700 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:04.700 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711658+0000", 2024-09-11T09:51:04.701 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.3540000000000002e-06 2024-09-11T09:51:04.701 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.701 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.701 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:04.701 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.716475+0000", 2024-09-11T09:51:04.701 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0048162819999999999 2024-09-11T09:51:04.701 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.701 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.701 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:04.701 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.716526+0000", 2024-09-11T09:51:04.701 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.0905000000000001e-05 2024-09-11T09:51:04.701 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.701 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.701 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:04.702 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.716601+0000", 2024-09-11T09:51:04.702 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.5283000000000004e-05 2024-09-11T09:51:04.702 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.702 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.702 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:04.702 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.717933+0000", 2024-09-11T09:51:04.702 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001331622 2024-09-11T09:51:04.702 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.702 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.702 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:04.702 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718332+0000", 2024-09-11T09:51:04.702 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00039909899999999998 2024-09-11T09:51:04.702 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.703 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.703 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:04.703 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718363+0000", 2024-09-11T09:51:04.703 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.1636e-05 2024-09-11T09:51:04.703 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.703 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.703 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:04.703 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718397+0000", 2024-09-11T09:51:04.703 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.3883000000000001e-05 2024-09-11T09:51:04.703 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:04.703 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:04.703 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:04.703 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.704 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.704 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:111 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:51:04.704 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.711668+0000", 2024-09-11T09:51:04.704 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 52.951457097000002, 2024-09-11T09:51:04.704 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0077145570000000004, 2024-09-11T09:51:04.704 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:04.704 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:04.704 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:04.704 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:04.704 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:51:04.704 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:51:04.704 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 111 2024-09-11T09:51:04.704 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.705 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:04.705 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.705 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:04.705 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711668+0000", 2024-09-11T09:51:04.705 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:04.705 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.705 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.705 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:04.705 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711668+0000", 2024-09-11T09:51:04.705 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:04.705 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.705 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.705 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:04.705 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711669+0000", 2024-09-11T09:51:04.706 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.99e-07 2024-09-11T09:51:04.706 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.706 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.706 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:04.706 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711673+0000", 2024-09-11T09:51:04.706 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.7440000000000001e-06 2024-09-11T09:51:04.706 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.706 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.706 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:04.706 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711686+0000", 2024-09-11T09:51:04.706 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.3317e-05 2024-09-11T09:51:04.706 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.706 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.706 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:04.707 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711697+0000", 2024-09-11T09:51:04.707 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1022000000000001e-05 2024-09-11T09:51:04.707 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.707 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.707 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:04.707 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.717065+0000", 2024-09-11T09:51:04.707 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.005367531 2024-09-11T09:51:04.707 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.707 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.707 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:04.707 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.717114+0000", 2024-09-11T09:51:04.707 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.9707999999999999e-05 2024-09-11T09:51:04.707 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.708 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.708 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:04.708 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.717190+0000", 2024-09-11T09:51:04.708 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.5394000000000004e-05 2024-09-11T09:51:04.708 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.708 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.708 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:04.708 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718793+0000", 2024-09-11T09:51:04.708 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001603653 2024-09-11T09:51:04.708 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.708 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.708 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:04.708 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719329+0000", 2024-09-11T09:51:04.709 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00053543799999999995 2024-09-11T09:51:04.709 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.709 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.709 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:04.709 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719359+0000", 2024-09-11T09:51:04.709 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.0420999999999999e-05 2024-09-11T09:51:04.709 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.709 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.709 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:04.709 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719383+0000", 2024-09-11T09:51:04.709 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.3730000000000001e-05 2024-09-11T09:51:04.709 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:04.709 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:04.710 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:04.710 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.710 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.710 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:112 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:51:04.710 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.711712+0000", 2024-09-11T09:51:04.710 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 52.951413537000001, 2024-09-11T09:51:04.710 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.008162084, 2024-09-11T09:51:04.710 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:04.710 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:04.710 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:04.710 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:04.710 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:51:04.711 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:51:04.711 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 112 2024-09-11T09:51:04.711 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.711 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:04.711 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.711 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:04.711 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711712+0000", 2024-09-11T09:51:04.711 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:04.711 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.711 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.711 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:04.711 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711712+0000", 2024-09-11T09:51:04.711 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:04.711 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.712 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.712 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:04.712 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711712+0000", 2024-09-11T09:51:04.712 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.6000000000000004e-07 2024-09-11T09:51:04.712 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.712 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.712 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:04.712 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711808+0000", 2024-09-11T09:51:04.712 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.5977999999999995e-05 2024-09-11T09:51:04.712 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.712 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.712 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:04.712 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711819+0000", 2024-09-11T09:51:04.713 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.0755e-05 2024-09-11T09:51:04.713 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.713 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.713 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:04.713 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711826+0000", 2024-09-11T09:51:04.713 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.7970000000000003e-06 2024-09-11T09:51:04.713 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.713 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.713 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:04.713 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.717435+0000", 2024-09-11T09:51:04.713 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0056090239999999998 2024-09-11T09:51:04.713 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.713 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.714 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:04.714 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.717496+0000", 2024-09-11T09:51:04.714 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.0543e-05 2024-09-11T09:51:04.714 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.714 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.714 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:04.714 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.717591+0000", 2024-09-11T09:51:04.714 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.5667000000000004e-05 2024-09-11T09:51:04.714 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.714 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.714 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:04.714 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718831+0000", 2024-09-11T09:51:04.714 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001240204 2024-09-11T09:51:04.715 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.715 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.715 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:04.715 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719819+0000", 2024-09-11T09:51:04.715 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00098751500000000001 2024-09-11T09:51:04.715 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.715 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.715 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:04.715 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719850+0000", 2024-09-11T09:51:04.715 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.1535999999999998e-05 2024-09-11T09:51:04.715 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.715 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.715 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:04.716 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719874+0000", 2024-09-11T09:51:04.716 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.3504999999999999e-05 2024-09-11T09:51:04.716 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:04.716 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:04.716 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:04.716 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.716 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.716 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:113 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:51:04.716 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.711842+0000", 2024-09-11T09:51:04.716 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 52.951283038, 2024-09-11T09:51:04.716 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0081160099999999999, 2024-09-11T09:51:04.716 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:04.716 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:04.717 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:04.717 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:04.717 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:51:04.717 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:51:04.717 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 113 2024-09-11T09:51:04.717 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.717 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:04.717 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.717 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:04.717 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711842+0000", 2024-09-11T09:51:04.717 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:04.717 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.718 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.718 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:04.718 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711842+0000", 2024-09-11T09:51:04.718 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:04.718 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.718 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.718 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:04.718 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711843+0000", 2024-09-11T09:51:04.718 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.4399999999999999e-07 2024-09-11T09:51:04.718 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.718 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.718 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:04.718 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711848+0000", 2024-09-11T09:51:04.718 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.4159999999999997e-06 2024-09-11T09:51:04.719 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.719 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.719 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:04.719 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711856+0000", 2024-09-11T09:51:04.719 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.7870000000000007e-06 2024-09-11T09:51:04.719 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.719 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.719 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:04.719 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711862+0000", 2024-09-11T09:51:04.719 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.3689999999999998e-06 2024-09-11T09:51:04.719 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.719 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.719 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:04.719 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.717965+0000", 2024-09-11T09:51:04.720 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0061028899999999997 2024-09-11T09:51:04.720 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.720 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.720 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:04.720 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718023+0000", 2024-09-11T09:51:04.720 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.8313000000000002e-05 2024-09-11T09:51:04.720 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.720 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.720 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:04.720 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718109+0000", 2024-09-11T09:51:04.720 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.6205000000000006e-05 2024-09-11T09:51:04.720 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.720 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.721 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:04.721 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719808+0000", 2024-09-11T09:51:04.721 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0016990060000000001 2024-09-11T09:51:04.721 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.721 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.721 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:04.721 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719883+0000", 2024-09-11T09:51:04.721 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.4624999999999998e-05 2024-09-11T09:51:04.721 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.721 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.721 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:04.721 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719910+0000", 2024-09-11T09:51:04.721 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.7512e-05 2024-09-11T09:51:04.721 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.722 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.722 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:04.722 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719958+0000", 2024-09-11T09:51:04.722 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.8143000000000003e-05 2024-09-11T09:51:04.722 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:04.722 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:04.722 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:04.722 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.722 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.722 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:51:04.722 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.711887+0000", 2024-09-11T09:51:04.722 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 52.951238424000003, 2024-09-11T09:51:04.722 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0086063589999999992, 2024-09-11T09:51:04.723 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:04.723 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:04.723 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:04.723 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:04.723 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:51:04.723 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:51:04.723 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 114 2024-09-11T09:51:04.723 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.723 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:04.723 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.723 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:04.723 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711887+0000", 2024-09-11T09:51:04.723 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:04.723 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.724 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.724 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:04.724 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711887+0000", 2024-09-11T09:51:04.724 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:04.724 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.724 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.724 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:04.724 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711887+0000", 2024-09-11T09:51:04.724 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.1500000000000005e-07 2024-09-11T09:51:04.724 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.724 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.724 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:04.725 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711897+0000", 2024-09-11T09:51:04.725 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.1900000000000001e-06 2024-09-11T09:51:04.725 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.725 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.725 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:04.725 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711902+0000", 2024-09-11T09:51:04.725 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.2349999999999999e-06 2024-09-11T09:51:04.725 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.725 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.725 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:04.725 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711906+0000", 2024-09-11T09:51:04.725 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.7129999999999999e-06 2024-09-11T09:51:04.725 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.726 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.726 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:04.726 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718429+0000", 2024-09-11T09:51:04.726 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0065231960000000002 2024-09-11T09:51:04.726 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.726 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.726 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:04.726 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718480+0000", 2024-09-11T09:51:04.726 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.1161e-05 2024-09-11T09:51:04.726 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.726 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.726 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:04.726 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718558+0000", 2024-09-11T09:51:04.726 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.7519000000000001e-05 2024-09-11T09:51:04.727 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.727 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.727 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:04.727 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720038+0000", 2024-09-11T09:51:04.727 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001480016 2024-09-11T09:51:04.727 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.727 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.727 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:04.727 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720417+0000", 2024-09-11T09:51:04.727 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00037943699999999999 2024-09-11T09:51:04.727 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.727 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.727 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:04.728 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720452+0000", 2024-09-11T09:51:04.728 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.4595000000000003e-05 2024-09-11T09:51:04.728 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.728 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.728 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:04.728 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720493+0000", 2024-09-11T09:51:04.728 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.1782000000000001e-05 2024-09-11T09:51:04.728 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:04.728 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:04.728 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:04.728 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.728 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.728 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:51:04.729 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.711912+0000", 2024-09-11T09:51:04.729 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 52.951213754999998, 2024-09-11T09:51:04.729 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0086864750000000008, 2024-09-11T09:51:04.729 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:04.729 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:04.729 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:04.729 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:04.729 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:51:04.729 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:51:04.729 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 115 2024-09-11T09:51:04.729 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.729 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:04.729 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.730 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:04.730 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711912+0000", 2024-09-11T09:51:04.730 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:04.730 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.730 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.730 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:04.730 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711912+0000", 2024-09-11T09:51:04.730 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:04.730 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.730 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.730 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:04.730 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711912+0000", 2024-09-11T09:51:04.730 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.3900000000000001e-07 2024-09-11T09:51:04.730 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.731 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.731 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:04.731 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711969+0000", 2024-09-11T09:51:04.731 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.7501999999999999e-05 2024-09-11T09:51:04.731 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.731 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.731 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:04.731 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711986+0000", 2024-09-11T09:51:04.731 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.6402999999999999e-05 2024-09-11T09:51:04.731 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.731 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.731 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:04.731 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711990+0000", 2024-09-11T09:51:04.731 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.929e-06 2024-09-11T09:51:04.732 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.732 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.732 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:04.732 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718861+0000", 2024-09-11T09:51:04.732 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0068710530000000002 2024-09-11T09:51:04.732 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.732 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.732 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:04.732 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718908+0000", 2024-09-11T09:51:04.732 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.7431e-05 2024-09-11T09:51:04.732 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.732 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.732 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:04.733 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718984+0000", 2024-09-11T09:51:04.733 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.5500999999999994e-05 2024-09-11T09:51:04.733 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.733 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.733 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:04.733 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720500+0000", 2024-09-11T09:51:04.733 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0015158820000000001 2024-09-11T09:51:04.733 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.733 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.733 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:04.733 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720536+0000", 2024-09-11T09:51:04.733 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.6118000000000002e-05 2024-09-11T09:51:04.733 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.733 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.734 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:04.734 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720570+0000", 2024-09-11T09:51:04.734 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.4387999999999997e-05 2024-09-11T09:51:04.734 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.734 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.734 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:04.734 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720598+0000", 2024-09-11T09:51:04.734 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.8028999999999999e-05 2024-09-11T09:51:04.734 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:04.734 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:04.734 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:04.734 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.734 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.735 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:51:04.735 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.711998+0000", 2024-09-11T09:51:04.735 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 52.951127575999998, 2024-09-11T09:51:04.735 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.010785813999999999, 2024-09-11T09:51:04.735 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:04.735 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:04.735 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:04.735 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:04.735 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:51:04.735 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:51:04.735 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 116 2024-09-11T09:51:04.735 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.735 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:04.736 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.736 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:04.736 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711998+0000", 2024-09-11T09:51:04.736 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:04.736 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.736 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.736 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:04.736 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711998+0000", 2024-09-11T09:51:04.736 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:04.736 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.736 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.736 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:04.737 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711998+0000", 2024-09-11T09:51:04.737 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.3300000000000003e-07 2024-09-11T09:51:04.737 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.737 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.737 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:04.737 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712001+0000", 2024-09-11T09:51:04.737 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.9349999999999999e-06 2024-09-11T09:51:04.737 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.737 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.737 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:04.737 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712006+0000", 2024-09-11T09:51:04.737 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.1630000000000004e-06 2024-09-11T09:51:04.737 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.737 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.738 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:04.738 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712010+0000", 2024-09-11T09:51:04.738 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.574e-06 2024-09-11T09:51:04.738 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.738 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.738 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:04.738 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719415+0000", 2024-09-11T09:51:04.738 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0074049099999999998 2024-09-11T09:51:04.738 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.738 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.738 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:04.738 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719465+0000", 2024-09-11T09:51:04.738 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.0129000000000001e-05 2024-09-11T09:51:04.738 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.739 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.739 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:04.739 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719545+0000", 2024-09-11T09:51:04.739 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.0253999999999994e-05 2024-09-11T09:51:04.739 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.739 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.739 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:04.739 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720636+0000", 2024-09-11T09:51:04.739 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0010909330000000001 2024-09-11T09:51:04.739 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.739 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.739 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:04.739 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.722707+0000", 2024-09-11T09:51:04.739 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0020713300000000001 2024-09-11T09:51:04.740 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.740 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.740 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:04.740 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.722743+0000", 2024-09-11T09:51:04.740 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.5351000000000001e-05 2024-09-11T09:51:04.740 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.740 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.740 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:04.740 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.722784+0000", 2024-09-11T09:51:04.740 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.0802e-05 2024-09-11T09:51:04.740 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:04.740 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:04.741 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:04.741 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.741 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.741 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:51:04.741 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.712014+0000", 2024-09-11T09:51:04.741 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 52.951111535999999, 2024-09-11T09:51:04.741 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.013420938, 2024-09-11T09:51:04.741 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:04.741 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:04.741 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:04.741 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:04.741 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:51:04.741 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:51:04.742 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 117 2024-09-11T09:51:04.742 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.742 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:04.742 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.742 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:04.742 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712014+0000", 2024-09-11T09:51:04.742 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:04.742 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.742 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.742 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:04.742 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712014+0000", 2024-09-11T09:51:04.742 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:04.742 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.742 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.743 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:04.743 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712014+0000", 2024-09-11T09:51:04.743 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.2000000000000001e-07 2024-09-11T09:51:04.743 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.743 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.743 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:04.743 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712016+0000", 2024-09-11T09:51:04.743 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.2309999999999999e-06 2024-09-11T09:51:04.743 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.743 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.743 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:04.743 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712020+0000", 2024-09-11T09:51:04.743 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.3830000000000001e-06 2024-09-11T09:51:04.743 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.744 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.744 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:04.744 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712023+0000", 2024-09-11T09:51:04.744 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.3340000000000002e-06 2024-09-11T09:51:04.744 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.744 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.744 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:04.744 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720083+0000", 2024-09-11T09:51:04.744 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0080601189999999993 2024-09-11T09:51:04.744 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.744 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.744 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:04.744 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720125+0000", 2024-09-11T09:51:04.745 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.2029e-05 2024-09-11T09:51:04.745 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.745 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.745 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:04.745 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720205+0000", 2024-09-11T09:51:04.745 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.0010000000000001e-05 2024-09-11T09:51:04.745 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.745 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.745 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:04.745 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720849+0000", 2024-09-11T09:51:04.745 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00064369899999999996 2024-09-11T09:51:04.745 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.745 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.746 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:04.746 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.725357+0000", 2024-09-11T09:51:04.746 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0045077379999999998 2024-09-11T09:51:04.746 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.746 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.746 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:04.746 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.725392+0000", 2024-09-11T09:51:04.746 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.5862999999999999e-05 2024-09-11T09:51:04.746 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.746 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.746 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:04.746 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.725435+0000", 2024-09-11T09:51:04.746 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.2311999999999997e-05 2024-09-11T09:51:04.746 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:04.747 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:04.747 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:04.747 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.747 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.747 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:119 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 131072~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:51:04.747 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.725673+0000", 2024-09-11T09:51:04.747 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 52.937452833999998, 2024-09-11T09:51:04.747 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0053927460000000003, 2024-09-11T09:51:04.747 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:04.747 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:04.747 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:04.747 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:04.747 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:51:04.748 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:51:04.748 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 119 2024-09-11T09:51:04.748 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.748 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:04.748 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.748 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:04.748 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.725673+0000", 2024-09-11T09:51:04.748 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:04.748 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.748 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.748 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:04.748 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.725673+0000", 2024-09-11T09:51:04.748 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:04.748 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.749 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.749 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:04.749 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.725673+0000", 2024-09-11T09:51:04.749 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.68e-07 2024-09-11T09:51:04.749 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.749 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.749 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:04.749 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.725720+0000", 2024-09-11T09:51:04.749 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.6526999999999997e-05 2024-09-11T09:51:04.749 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.749 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.749 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:04.749 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.725734+0000", 2024-09-11T09:51:04.750 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.4593000000000001e-05 2024-09-11T09:51:04.750 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.750 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.750 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:04.750 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.725741+0000", 2024-09-11T09:51:04.750 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.28e-06 2024-09-11T09:51:04.750 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.750 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.750 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:04.750 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.726290+0000", 2024-09-11T09:51:04.750 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00054892000000000005 2024-09-11T09:51:04.750 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.750 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.751 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:04.751 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.726338+0000", 2024-09-11T09:51:04.751 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.7982000000000002e-05 2024-09-11T09:51:04.751 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.751 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.751 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:04.751 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.726413+0000", 2024-09-11T09:51:04.751 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.5679000000000005e-05 2024-09-11T09:51:04.751 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.751 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.751 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:04.751 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.727555+0000", 2024-09-11T09:51:04.751 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0011420289999999999 2024-09-11T09:51:04.752 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.752 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.752 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:04.752 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.731004+0000", 2024-09-11T09:51:04.752 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0034484279999999999 2024-09-11T09:51:04.752 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.752 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.752 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:04.752 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.731040+0000", 2024-09-11T09:51:04.752 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.6315e-05 2024-09-11T09:51:04.752 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.752 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.752 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:04.753 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.731065+0000", 2024-09-11T09:51:04.753 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.5324999999999999e-05 2024-09-11T09:51:04.753 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:04.753 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:04.753 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:04.753 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.753 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.753 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:120 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 196608~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:51:04.753 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.725756+0000", 2024-09-11T09:51:04.753 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 52.937368925000001, 2024-09-11T09:51:04.753 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0053569100000000003, 2024-09-11T09:51:04.753 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:04.754 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:04.754 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:04.754 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:04.754 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:51:04.754 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:51:04.754 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 120 2024-09-11T09:51:04.754 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.754 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:04.754 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.754 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:04.754 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.725756+0000", 2024-09-11T09:51:04.754 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:04.754 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.755 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.755 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:04.755 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.725756+0000", 2024-09-11T09:51:04.755 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:04.755 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.755 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.755 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:04.755 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.725757+0000", 2024-09-11T09:51:04.755 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.3799999999999997e-07 2024-09-11T09:51:04.755 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.755 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.755 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:04.755 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.725802+0000", 2024-09-11T09:51:04.755 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.4790000000000003e-05 2024-09-11T09:51:04.756 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.756 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.756 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:04.756 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.725815+0000", 2024-09-11T09:51:04.756 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.2609000000000001e-05 2024-09-11T09:51:04.756 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.756 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.756 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:04.756 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.725822+0000", 2024-09-11T09:51:04.756 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.6450000000000005e-06 2024-09-11T09:51:04.756 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.756 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.756 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:04.757 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.726708+0000", 2024-09-11T09:51:04.757 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00088606500000000001 2024-09-11T09:51:04.757 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.757 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.757 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:04.757 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.726755+0000", 2024-09-11T09:51:04.757 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.6888000000000002e-05 2024-09-11T09:51:04.757 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.757 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.757 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:04.757 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.726828+0000", 2024-09-11T09:51:04.757 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.2510999999999997e-05 2024-09-11T09:51:04.757 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.758 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.758 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:04.758 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.727829+0000", 2024-09-11T09:51:04.758 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001001093 2024-09-11T09:51:04.758 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.758 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.758 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:04.758 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.731071+0000", 2024-09-11T09:51:04.758 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0032426009999999999 2024-09-11T09:51:04.758 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.758 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.758 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:04.758 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.731094+0000", 2024-09-11T09:51:04.758 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.2677e-05 2024-09-11T09:51:04.759 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.759 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.759 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:04.759 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.731113+0000", 2024-09-11T09:51:04.759 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.9392999999999999e-05 2024-09-11T09:51:04.759 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:04.759 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:04.759 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:04.759 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.759 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.759 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:138 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:51:04.759 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:12.034038+0000", 2024-09-11T09:51:04.759 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 52.629087595000001, 2024-09-11T09:51:04.760 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0054142749999999996, 2024-09-11T09:51:04.760 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:04.760 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:04.760 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:04.760 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:04.760 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:51:04.760 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:51:04.760 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 138 2024-09-11T09:51:04.760 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.760 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:04.760 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.760 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:04.760 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034038+0000", 2024-09-11T09:51:04.761 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:04.761 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.761 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.761 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:04.761 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034038+0000", 2024-09-11T09:51:04.761 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:04.761 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.761 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.761 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:04.761 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034038+0000", 2024-09-11T09:51:04.761 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.3499999999999996e-07 2024-09-11T09:51:04.761 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.761 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.761 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:04.762 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034043+0000", 2024-09-11T09:51:04.762 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.7310000000000001e-06 2024-09-11T09:51:04.762 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.762 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.762 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:04.762 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034050+0000", 2024-09-11T09:51:04.762 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.0419999999999997e-06 2024-09-11T09:51:04.762 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.762 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.762 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:04.762 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034055+0000", 2024-09-11T09:51:04.762 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.3299999999999998e-06 2024-09-11T09:51:04.762 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.763 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.763 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:04.763 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.037309+0000", 2024-09-11T09:51:04.763 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0032535440000000001 2024-09-11T09:51:04.763 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.763 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.763 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:04.763 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.037357+0000", 2024-09-11T09:51:04.763 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.8167999999999997e-05 2024-09-11T09:51:04.763 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.763 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.763 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:04.763 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.037430+0000", 2024-09-11T09:51:04.763 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.3275999999999994e-05 2024-09-11T09:51:04.764 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.764 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.764 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:04.764 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.039014+0000", 2024-09-11T09:51:04.764 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0015831350000000001 2024-09-11T09:51:04.764 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.764 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.764 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:04.764 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.039401+0000", 2024-09-11T09:51:04.764 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00038758500000000002 2024-09-11T09:51:04.764 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.764 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.764 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:04.764 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.039429+0000", 2024-09-11T09:51:04.765 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.7379e-05 2024-09-11T09:51:04.765 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.765 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.765 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:04.765 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.039452+0000", 2024-09-11T09:51:04.765 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.355e-05 2024-09-11T09:51:04.765 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:04.765 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:04.765 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:04.765 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.765 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.765 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:139 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:51:04.765 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:12.034064+0000", 2024-09-11T09:51:04.766 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 52.629061563999997, 2024-09-11T09:51:04.766 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0054736200000000002, 2024-09-11T09:51:04.766 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:04.766 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:04.766 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:04.766 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:04.766 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:51:04.766 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:51:04.766 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 139 2024-09-11T09:51:04.766 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.766 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:04.766 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.766 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:04.766 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034064+0000", 2024-09-11T09:51:04.767 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:04.767 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.767 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.767 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:04.767 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034064+0000", 2024-09-11T09:51:04.767 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:04.767 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.767 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.767 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:04.767 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034064+0000", 2024-09-11T09:51:04.767 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.89e-07 2024-09-11T09:51:04.767 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.767 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.767 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:04.768 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034069+0000", 2024-09-11T09:51:04.768 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.9799999999999998e-06 2024-09-11T09:51:04.768 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.768 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.768 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:04.768 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034079+0000", 2024-09-11T09:51:04.768 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.3549999999999994e-06 2024-09-11T09:51:04.768 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.768 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.768 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:04.768 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034084+0000", 2024-09-11T09:51:04.768 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.2209999999999996e-06 2024-09-11T09:51:04.768 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.769 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.769 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:04.769 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.038176+0000", 2024-09-11T09:51:04.769 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0040921610000000004 2024-09-11T09:51:04.769 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.769 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.769 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:04.769 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.038221+0000", 2024-09-11T09:51:04.769 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.5006000000000002e-05 2024-09-11T09:51:04.769 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.769 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.769 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:04.769 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.038293+0000", 2024-09-11T09:51:04.770 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.1952999999999994e-05 2024-09-11T09:51:04.770 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.770 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.770 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:04.770 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.039461+0000", 2024-09-11T09:51:04.770 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0011680340000000001 2024-09-11T09:51:04.770 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.770 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.770 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:04.770 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.039487+0000", 2024-09-11T09:51:04.770 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.5752e-05 2024-09-11T09:51:04.770 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.770 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.771 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:04.771 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.039515+0000", 2024-09-11T09:51:04.771 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.7763000000000002e-05 2024-09-11T09:51:04.771 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.771 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.771 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:04.771 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.039537+0000", 2024-09-11T09:51:04.771 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.2906000000000001e-05 2024-09-11T09:51:04.771 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:04.771 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:04.771 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:04.771 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.771 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.772 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:140 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:51:04.772 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:12.034093+0000", 2024-09-11T09:51:04.772 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 52.62903206, 2024-09-11T09:51:04.772 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0059112490000000004, 2024-09-11T09:51:04.772 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:04.772 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:04.772 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:04.772 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:04.772 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:51:04.772 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:51:04.772 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 140 2024-09-11T09:51:04.772 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.772 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:04.773 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.773 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:04.773 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034093+0000", 2024-09-11T09:51:04.773 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:04.773 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.773 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.773 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:04.773 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034093+0000", 2024-09-11T09:51:04.773 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:04.773 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.773 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.773 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:04.773 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034094+0000", 2024-09-11T09:51:04.774 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.4299999999999998e-07 2024-09-11T09:51:04.774 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.774 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.774 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:04.774 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034159+0000", 2024-09-11T09:51:04.774 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.567e-05 2024-09-11T09:51:04.774 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.774 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.774 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:04.774 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034167+0000", 2024-09-11T09:51:04.774 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.8490000000000006e-06 2024-09-11T09:51:04.774 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.774 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.775 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:04.775 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034173+0000", 2024-09-11T09:51:04.775 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.1770000000000003e-06 2024-09-11T09:51:04.775 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.775 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.775 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:04.775 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.038543+0000", 2024-09-11T09:51:04.775 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0043699150000000003 2024-09-11T09:51:04.775 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.775 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.775 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:04.775 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.038589+0000", 2024-09-11T09:51:04.775 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.5562999999999997e-05 2024-09-11T09:51:04.776 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.776 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.776 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:04.776 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.038662+0000", 2024-09-11T09:51:04.776 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.2748000000000002e-05 2024-09-11T09:51:04.776 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.776 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.776 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:04.776 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.039635+0000", 2024-09-11T09:51:04.776 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00097310799999999998 2024-09-11T09:51:04.776 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.776 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.776 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:04.777 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.039945+0000", 2024-09-11T09:51:04.777 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00031054100000000001 2024-09-11T09:51:04.777 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.777 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.777 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:04.777 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.039971+0000", 2024-09-11T09:51:04.777 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.5423999999999999e-05 2024-09-11T09:51:04.777 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.777 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.777 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:04.777 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.040005+0000", 2024-09-11T09:51:04.777 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.3810999999999999e-05 2024-09-11T09:51:04.777 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:04.778 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:04.778 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:04.778 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.778 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.778 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:51:04.778 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:12.034184+0000", 2024-09-11T09:51:04.778 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 52.628941361999999, 2024-09-11T09:51:04.778 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0061520380000000003, 2024-09-11T09:51:04.778 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:04.778 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:04.778 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:04.778 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:04.778 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:51:04.779 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:51:04.779 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 141 2024-09-11T09:51:04.779 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.779 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:04.779 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.779 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:04.779 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034184+0000", 2024-09-11T09:51:04.779 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:04.779 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.779 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.779 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:04.779 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034184+0000", 2024-09-11T09:51:04.779 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:04.780 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.780 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.780 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:04.780 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034185+0000", 2024-09-11T09:51:04.780 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.1099999999999995e-07 2024-09-11T09:51:04.780 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.781 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.781 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:04.781 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034190+0000", 2024-09-11T09:51:04.781 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.7309999999999998e-06 2024-09-11T09:51:04.781 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.781 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.781 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:04.781 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034197+0000", 2024-09-11T09:51:04.781 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.7440000000000001e-06 2024-09-11T09:51:04.781 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.781 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.781 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:04.781 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034204+0000", 2024-09-11T09:51:04.781 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.8970000000000002e-06 2024-09-11T09:51:04.782 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.782 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.782 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:04.782 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.039035+0000", 2024-09-11T09:51:04.782 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0048307039999999999 2024-09-11T09:51:04.782 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.782 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.782 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:04.782 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.039093+0000", 2024-09-11T09:51:04.782 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.8604000000000003e-05 2024-09-11T09:51:04.782 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.782 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.782 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:04.783 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.039184+0000", 2024-09-11T09:51:04.783 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.0196000000000002e-05 2024-09-11T09:51:04.783 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.783 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.783 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:04.783 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.039873+0000", 2024-09-11T09:51:04.783 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00068960600000000005 2024-09-11T09:51:04.783 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.783 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.783 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:04.783 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.040292+0000", 2024-09-11T09:51:04.783 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00041865199999999999 2024-09-11T09:51:04.783 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.783 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.784 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:04.784 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.040317+0000", 2024-09-11T09:51:04.784 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.5439000000000002e-05 2024-09-11T09:51:04.784 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.784 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.784 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:04.784 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.040336+0000", 2024-09-11T09:51:04.784 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.8754e-05 2024-09-11T09:51:04.784 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:04.784 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:04.784 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:04.784 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.784 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.785 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:157 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:51:04.785 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:12.452040+0000", 2024-09-11T09:51:04.785 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 52.211085509, 2024-09-11T09:51:04.785 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0068848269999999996, 2024-09-11T09:51:04.785 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:04.785 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:04.785 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:04.785 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:04.785 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:51:04.785 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:51:04.786 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 157 2024-09-11T09:51:04.786 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.786 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:04.786 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.786 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:04.786 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452040+0000", 2024-09-11T09:51:04.786 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:04.786 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.786 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.786 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:04.786 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452040+0000", 2024-09-11T09:51:04.787 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:04.787 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.787 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.787 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:04.787 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452040+0000", 2024-09-11T09:51:04.787 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.22e-07 2024-09-11T09:51:04.787 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.787 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.787 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:04.787 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452042+0000", 2024-09-11T09:51:04.787 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.7010000000000001e-06 2024-09-11T09:51:04.787 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.787 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.787 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:04.788 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452045+0000", 2024-09-11T09:51:04.788 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.1310000000000002e-06 2024-09-11T09:51:04.788 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.788 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.788 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:04.788 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452048+0000", 2024-09-11T09:51:04.788 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.65e-06 2024-09-11T09:51:04.788 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.788 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.788 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:04.788 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.454453+0000", 2024-09-11T09:51:04.788 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0024052959999999999 2024-09-11T09:51:04.788 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.789 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.789 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:04.789 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.454498+0000", 2024-09-11T09:51:04.789 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.4978999999999997e-05 2024-09-11T09:51:04.789 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.789 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.789 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:04.789 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.454567+0000", 2024-09-11T09:51:04.789 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.9110999999999996e-05 2024-09-11T09:51:04.789 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.789 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.790 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:04.790 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.455464+0000", 2024-09-11T09:51:04.790 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00089696199999999995 2024-09-11T09:51:04.790 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.790 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.790 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:04.791 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.458844+0000", 2024-09-11T09:51:04.791 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0033802469999999999 2024-09-11T09:51:04.792 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.792 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.792 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:04.792 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.458889+0000", 2024-09-11T09:51:04.792 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.4441999999999999e-05 2024-09-11T09:51:04.792 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.792 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.792 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:04.792 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.458925+0000", 2024-09-11T09:51:04.792 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.6086e-05 2024-09-11T09:51:04.792 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:04.792 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:04.792 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:04.793 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.793 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.793 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:158 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:51:04.793 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:12.452055+0000", 2024-09-11T09:51:04.793 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 52.211070116000002, 2024-09-11T09:51:04.793 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0069533829999999996, 2024-09-11T09:51:04.793 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:04.793 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:04.793 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:04.793 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:04.793 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:51:04.793 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:51:04.793 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 158 2024-09-11T09:51:04.793 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.794 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:04.794 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.794 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:04.794 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452055+0000", 2024-09-11T09:51:04.794 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:04.794 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.794 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.794 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:04.794 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452055+0000", 2024-09-11T09:51:04.794 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:04.794 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.794 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.794 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:04.794 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452056+0000", 2024-09-11T09:51:04.795 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.48e-07 2024-09-11T09:51:04.795 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.795 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.795 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:04.795 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452087+0000", 2024-09-11T09:51:04.795 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.1149000000000003e-05 2024-09-11T09:51:04.795 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.795 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.795 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:04.795 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452092+0000", 2024-09-11T09:51:04.795 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.8400000000000002e-06 2024-09-11T09:51:04.795 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.795 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.795 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:04.796 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452095+0000", 2024-09-11T09:51:04.796 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.0740000000000001e-06 2024-09-11T09:51:04.796 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.796 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.796 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:04.796 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.454796+0000", 2024-09-11T09:51:04.796 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0027009740000000001 2024-09-11T09:51:04.796 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.796 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.796 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:04.796 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.454869+0000", 2024-09-11T09:51:04.796 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.3678999999999997e-05 2024-09-11T09:51:04.797 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.797 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.797 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:04.797 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.454992+0000", 2024-09-11T09:51:04.797 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00012279000000000001 2024-09-11T09:51:04.797 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.797 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.797 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:04.797 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.456072+0000", 2024-09-11T09:51:04.797 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001080229 2024-09-11T09:51:04.797 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.797 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.797 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:04.797 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.458931+0000", 2024-09-11T09:51:04.798 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0028591789999999999 2024-09-11T09:51:04.798 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.798 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.798 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:04.798 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.458972+0000", 2024-09-11T09:51:04.798 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.0170999999999999e-05 2024-09-11T09:51:04.798 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.798 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.798 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:04.798 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.459009+0000", 2024-09-11T09:51:04.798 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.7049999999999999e-05 2024-09-11T09:51:04.798 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:04.798 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:04.799 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:04.799 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.799 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.799 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:159 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:51:04.799 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:12.452099+0000", 2024-09-11T09:51:04.799 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 52.211025948, 2024-09-11T09:51:04.799 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0069885950000000002, 2024-09-11T09:51:04.799 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:04.799 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:04.799 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:04.799 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:04.799 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:51:04.799 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:51:04.799 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 159 2024-09-11T09:51:04.800 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.800 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:04.800 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.800 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:04.800 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452099+0000", 2024-09-11T09:51:04.800 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:04.800 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.800 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.800 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:04.800 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452099+0000", 2024-09-11T09:51:04.800 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:04.800 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.800 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.800 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:04.801 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452100+0000", 2024-09-11T09:51:04.801 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.2800000000000003e-07 2024-09-11T09:51:04.801 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.801 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.801 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:04.801 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452102+0000", 2024-09-11T09:51:04.801 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.824e-06 2024-09-11T09:51:04.801 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.801 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.801 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:04.801 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452105+0000", 2024-09-11T09:51:04.801 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.0460000000000001e-06 2024-09-11T09:51:04.801 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.802 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.802 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:04.802 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452107+0000", 2024-09-11T09:51:04.802 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.543e-06 2024-09-11T09:51:04.802 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.802 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.802 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:04.802 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.455484+0000", 2024-09-11T09:51:04.802 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0033769759999999998 2024-09-11T09:51:04.802 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.802 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.802 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:04.802 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.455527+0000", 2024-09-11T09:51:04.802 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.2589999999999997e-05 2024-09-11T09:51:04.803 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.803 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.803 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:04.803 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.455594+0000", 2024-09-11T09:51:04.803 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.6775999999999999e-05 2024-09-11T09:51:04.803 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.803 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.803 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:04.803 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.456312+0000", 2024-09-11T09:51:04.803 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00071874700000000005 2024-09-11T09:51:04.803 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.803 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.803 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:04.804 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.459015+0000", 2024-09-11T09:51:04.804 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0027028540000000002 2024-09-11T09:51:04.804 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.804 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.804 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:04.804 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.459054+0000", 2024-09-11T09:51:04.804 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.8840000000000001e-05 2024-09-11T09:51:04.804 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:04.804 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:04.804 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:04.804 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.459088+0000", 2024-09-11T09:51:04.804 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.4071e-05 2024-09-11T09:51:04.805 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:04.805 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:04.805 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:04.805 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:04.805 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:04.805 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-11T09:51:04.807 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-11T09:51:04.807 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-09-11T09:51:04.807+0000 7f7b6c7d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31904) UID: 0 2024-09-11T09:51:04.907 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:51:04.908+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:51:04.949 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-11T09:51:04.949 INFO:teuthology.orchestra.run.smithi096.stdout: "ops": [], 2024-09-11T09:51:04.949 INFO:teuthology.orchestra.run.smithi096.stdout: "num_ops": 0 2024-09-11T09:51:04.949 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-11T09:51:04.957 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-11T09:51:05.008 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:51:05.008+0000 7f3c96ec9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31909) UID: 0 2024-09-11T09:51:05.060 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-11T09:51:05.060 INFO:teuthology.orchestra.run.smithi096.stdout: "ops": [], 2024-09-11T09:51:05.060 INFO:teuthology.orchestra.run.smithi096.stdout: "complaint_time": 30, 2024-09-11T09:51:05.060 INFO:teuthology.orchestra.run.smithi096.stdout: "num_blocked_ops": 0 2024-09-11T09:51:05.060 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-11T09:51:05.068 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-11T09:51:05.109 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:51:05.110+0000 7fdec5849640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31907) UID: 0 2024-09-11T09:51:05.210 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:51:05.211+0000 7fdec5849640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31907) UID: 0 2024-09-11T09:51:05.211 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-11T09:51:05.211 INFO:teuthology.orchestra.run.smithi096.stdout: "size": 20, 2024-09-11T09:51:05.211 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 600, 2024-09-11T09:51:05.211 INFO:teuthology.orchestra.run.smithi096.stdout: "ops": [] 2024-09-11T09:51:05.211 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-11T09:51:05.220 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-11T09:51:05.311 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:51:05.312+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:51:05.363 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-11T09:51:05.363 INFO:teuthology.orchestra.run.smithi096.stdout: "ops": [], 2024-09-11T09:51:05.363 INFO:teuthology.orchestra.run.smithi096.stdout: "num_ops": 0 2024-09-11T09:51:05.363 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-11T09:51:05.372 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-11T09:51:05.411 INFO:tasks.ceph.osd.0.smithi017.stderr:2024-09-11T09:51:05.413+0000 7fb73c0f1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31665) UID: 0 2024-09-11T09:51:05.474 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-11T09:51:05.474 INFO:teuthology.orchestra.run.smithi096.stdout: "ops": [], 2024-09-11T09:51:05.474 INFO:teuthology.orchestra.run.smithi096.stdout: "complaint_time": 30, 2024-09-11T09:51:05.475 INFO:teuthology.orchestra.run.smithi096.stdout: "num_blocked_ops": 0 2024-09-11T09:51:05.475 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-11T09:51:05.483 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-11T09:51:05.513 INFO:tasks.ceph.osd.2.smithi017.stderr:2024-09-11T09:51:05.514+0000 7fd20b170640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31672) UID: 0 2024-09-11T09:51:05.587 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-11T09:51:05.587 INFO:teuthology.orchestra.run.smithi096.stdout: "size": 20, 2024-09-11T09:51:05.587 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 600, 2024-09-11T09:51:05.587 INFO:teuthology.orchestra.run.smithi096.stdout: "ops": [ 2024-09-11T09:51:05.587 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:05.587 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4230.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-11T09:51:05.587 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:18.490736+0000", 2024-09-11T09:51:05.587 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 47.096994369999997, 2024-09-11T09:51:05.587 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0059097569999999999, 2024-09-11T09:51:05.587 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:05.587 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:05.588 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:05.588 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:05.588 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4230", 2024-09-11T09:51:05.588 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.17:0/4101109576", 2024-09-11T09:51:05.588 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 1 2024-09-11T09:51:05.588 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:05.588 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:05.588 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:05.588 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:05.588 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.490736+0000", 2024-09-11T09:51:05.588 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:05.588 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:05.589 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:05.589 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:05.589 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.490736+0000", 2024-09-11T09:51:05.589 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:05.589 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:05.589 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:05.589 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:05.589 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.490742+0000", 2024-09-11T09:51:05.589 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.48e-06 2024-09-11T09:51:05.589 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:05.589 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:05.589 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:05.589 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.490749+0000", 2024-09-11T09:51:05.590 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.4569999999999999e-06 2024-09-11T09:51:05.590 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:05.590 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:05.590 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:05.590 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.490762+0000", 2024-09-11T09:51:05.590 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.2721000000000001e-05 2024-09-11T09:51:05.590 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:05.590 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:05.590 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:05.590 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.490774+0000", 2024-09-11T09:51:05.590 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1823e-05 2024-09-11T09:51:05.590 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:05.590 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:05.591 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:05.591 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.490846+0000", 2024-09-11T09:51:05.591 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.1878000000000005e-05 2024-09-11T09:51:05.591 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:05.591 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:05.591 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for peered", 2024-09-11T09:51:05.591 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.490857+0000", 2024-09-11T09:51:05.591 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1017e-05 2024-09-11T09:51:05.591 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:05.591 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:05.591 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:05.591 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.493856+0000", 2024-09-11T09:51:05.591 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00299885 2024-09-11T09:51:05.592 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:05.592 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:05.592 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:05.592 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.494020+0000", 2024-09-11T09:51:05.592 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00016436599999999999 2024-09-11T09:51:05.592 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:05.592 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:05.592 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 2", 2024-09-11T09:51:05.592 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.494186+0000", 2024-09-11T09:51:05.592 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00016562799999999999 2024-09-11T09:51:05.592 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:05.592 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:05.592 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:05.593 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.495407+0000", 2024-09-11T09:51:05.593 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0012217420000000001 2024-09-11T09:51:05.593 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:05.593 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:05.593 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:05.593 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.496579+0000", 2024-09-11T09:51:05.593 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001171844 2024-09-11T09:51:05.593 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:05.593 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:05.593 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:05.593 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.496620+0000", 2024-09-11T09:51:05.593 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.0479000000000003e-05 2024-09-11T09:51:05.593 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:05.594 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:05.594 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:05.594 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.496646+0000", 2024-09-11T09:51:05.594 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.6472e-05 2024-09-11T09:51:05.594 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:05.594 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:05.594 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:05.594 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:05.594 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:05.594 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_repop_reply(client.4230.0:1 2.3 e16/15 ondisk, result = 0)", 2024-09-11T09:51:05.594 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:18.495321+0000", 2024-09-11T09:51:05.594 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 47.092409705000001, 2024-09-11T09:51:05.594 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.4965999999999998e-05, 2024-09-11T09:51:05.595 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:05.595 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:05.595 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "started", 2024-09-11T09:51:05.595 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:05.595 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:05.595 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:05.595 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.495321+0000", 2024-09-11T09:51:05.595 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:05.595 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:05.595 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:05.595 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:05.595 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.495321+0000", 2024-09-11T09:51:05.595 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:05.596 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:05.596 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:05.596 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:05.596 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.495322+0000", 2024-09-11T09:51:05.596 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.7700000000000004e-07 2024-09-11T09:51:05.596 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:05.596 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:05.596 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:05.596 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.495325+0000", 2024-09-11T09:51:05.596 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.7780000000000001e-06 2024-09-11T09:51:05.596 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:05.596 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:05.596 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:05.597 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.495338+0000", 2024-09-11T09:51:05.597 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.2315e-05 2024-09-11T09:51:05.597 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:05.597 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:05.597 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:05.597 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.495345+0000", 2024-09-11T09:51:05.597 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.8889999999999996e-06 2024-09-11T09:51:05.597 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:05.597 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:05.597 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:05.597 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.495392+0000", 2024-09-11T09:51:05.598 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.7524000000000001e-05 2024-09-11T09:51:05.598 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:05.598 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:05.598 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:05.598 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.495402+0000", 2024-09-11T09:51:05.598 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.5840000000000001e-06 2024-09-11T09:51:05.598 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:05.598 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:05.598 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:05.598 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.495416+0000", 2024-09-11T09:51:05.598 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.4199000000000001e-05 2024-09-11T09:51:05.598 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:05.598 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:05.599 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:05.599 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:05.599 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:05.599 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4230.0:2 2.2 2:5cea7035:::rbd_info:head [call rbd.metadata_list in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e16)", 2024-09-11T09:51:05.599 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:18.499576+0000", 2024-09-11T09:51:05.599 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 47.088155198000003, 2024-09-11T09:51:05.599 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00018163000000000001, 2024-09-11T09:51:05.599 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:05.599 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:05.599 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "reached pg", 2024-09-11T09:51:05.599 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:05.599 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4230", 2024-09-11T09:51:05.599 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.17:0/4101109576", 2024-09-11T09:51:05.599 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 2 2024-09-11T09:51:05.600 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:05.600 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:05.600 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:05.600 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:05.600 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.499576+0000", 2024-09-11T09:51:05.600 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:05.600 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:05.600 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:05.600 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:05.600 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.499576+0000", 2024-09-11T09:51:05.600 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:05.600 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:05.600 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:05.601 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:05.601 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.499577+0000", 2024-09-11T09:51:05.601 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.124e-06 2024-09-11T09:51:05.601 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:05.601 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:05.601 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:05.601 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.499584+0000", 2024-09-11T09:51:05.601 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.1949999999999997e-06 2024-09-11T09:51:05.601 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:05.601 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:05.601 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:05.601 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.499597+0000", 2024-09-11T09:51:05.601 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.366e-05 2024-09-11T09:51:05.602 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:05.602 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:05.602 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:05.602 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.499604+0000", 2024-09-11T09:51:05.602 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.2709999999999998e-06 2024-09-11T09:51:05.602 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:05.602 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:05.602 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:05.602 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.499660+0000", 2024-09-11T09:51:05.602 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.6270999999999997e-05 2024-09-11T09:51:05.602 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:05.602 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:05.602 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:05.602 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.499757+0000", 2024-09-11T09:51:05.603 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.7108999999999995e-05 2024-09-11T09:51:05.603 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:05.603 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:05.603 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:05.603 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:05.603 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:05.603 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4230.0:3 2.2 2:5cea7035:::rbd_info:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e16)", 2024-09-11T09:51:05.603 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:18.499937+0000", 2024-09-11T09:51:05.603 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 47.087793409, 2024-09-11T09:51:05.603 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000143318, 2024-09-11T09:51:05.603 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:05.603 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:05.603 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "reached pg", 2024-09-11T09:51:05.604 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:05.604 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4230", 2024-09-11T09:51:05.604 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.17:0/4101109576", 2024-09-11T09:51:05.604 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 3 2024-09-11T09:51:05.604 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:05.604 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:05.604 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:05.604 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:05.604 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.499937+0000", 2024-09-11T09:51:05.604 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:05.604 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:05.604 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:05.604 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:05.605 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.499937+0000", 2024-09-11T09:51:05.605 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:05.605 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:05.605 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:05.605 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:05.605 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.499938+0000", 2024-09-11T09:51:05.605 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.195e-06 2024-09-11T09:51:05.605 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:05.605 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:05.605 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:05.605 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.499943+0000", 2024-09-11T09:51:05.605 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.9080000000000003e-06 2024-09-11T09:51:05.606 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:05.606 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:05.606 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:05.606 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.499956+0000", 2024-09-11T09:51:05.606 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.2728000000000001e-05 2024-09-11T09:51:05.606 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:05.606 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:05.606 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:05.606 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.499964+0000", 2024-09-11T09:51:05.606 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.8220000000000007e-06 2024-09-11T09:51:05.606 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:05.606 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:05.606 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:05.606 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.500017+0000", 2024-09-11T09:51:05.607 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.2809000000000002e-05 2024-09-11T09:51:05.607 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:05.607 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:05.607 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:05.607 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.500081+0000", 2024-09-11T09:51:05.607 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.3856000000000007e-05 2024-09-11T09:51:05.607 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:05.607 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:05.607 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:05.607 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:05.607 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:05.607 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4230.0:5 2.2 2:5cea7035:::rbd_info:head [create,write 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-11T09:51:05.608 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:19.492662+0000", 2024-09-11T09:51:05.608 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 46.095068748999999, 2024-09-11T09:51:05.608 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0043824759999999997, 2024-09-11T09:51:05.608 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:05.608 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:05.608 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:05.608 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:05.608 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4230", 2024-09-11T09:51:05.608 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.17:0/4101109576", 2024-09-11T09:51:05.608 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 5 2024-09-11T09:51:05.608 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:05.608 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:05.608 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:05.609 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:05.609 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.492662+0000", 2024-09-11T09:51:05.609 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:05.609 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:05.609 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:05.609 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:05.609 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.492662+0000", 2024-09-11T09:51:05.609 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:05.609 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:05.609 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:05.609 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:05.609 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.492664+0000", 2024-09-11T09:51:05.609 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.3599999999999999e-06 2024-09-11T09:51:05.610 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:05.610 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:05.610 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:05.610 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.492675+0000", 2024-09-11T09:51:05.610 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1182000000000001e-05 2024-09-11T09:51:05.610 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:05.611 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:05.611 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:05.611 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.492693+0000", 2024-09-11T09:51:05.612 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.7031999999999999e-05 2024-09-11T09:51:05.612 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:05.612 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:05.612 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:05.612 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.492702+0000", 2024-09-11T09:51:05.612 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.5540000000000003e-06 2024-09-11T09:51:05.612 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:05.612 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:05.612 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:05.612 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.492767+0000", 2024-09-11T09:51:05.612 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.5411000000000001e-05 2024-09-11T09:51:05.612 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:05.613 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:05.613 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op must wait for map", 2024-09-11T09:51:05.613 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.492773+0000", 2024-09-11T09:51:05.613 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.1039999999999998e-06 2024-09-11T09:51:05.613 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:05.613 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:05.613 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:05.613 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.494354+0000", 2024-09-11T09:51:05.613 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0015814049999999999 2024-09-11T09:51:05.613 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:05.613 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:05.613 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:05.613 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.494476+0000", 2024-09-11T09:51:05.614 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000122089 2024-09-11T09:51:05.614 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:05.614 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:05.614 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 1", 2024-09-11T09:51:05.614 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.494577+0000", 2024-09-11T09:51:05.614 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000100536 2024-09-11T09:51:05.614 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:05.614 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:05.614 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:05.614 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.495770+0000", 2024-09-11T09:51:05.614 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0011932469999999999 2024-09-11T09:51:05.614 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:05.614 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:05.615 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:05.615 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.496966+0000", 2024-09-11T09:51:05.615 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001196003 2024-09-11T09:51:05.615 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:05.615 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:05.615 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:05.615 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.497011+0000", 2024-09-11T09:51:05.615 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.5229999999999999e-05 2024-09-11T09:51:05.615 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:05.615 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:05.615 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:05.615 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.497044+0000", 2024-09-11T09:51:05.615 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.3322999999999999e-05 2024-09-11T09:51:05.615 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:05.616 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:05.616 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:05.616 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:05.616 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:05.616 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_repop_reply(client.4230.0:5 2.2 e17/15 ondisk, result = 0)", 2024-09-11T09:51:05.616 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:19.495669+0000", 2024-09-11T09:51:05.616 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 46.092061960999999, 2024-09-11T09:51:05.616 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000106431, 2024-09-11T09:51:05.616 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:05.616 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:05.616 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "started", 2024-09-11T09:51:05.616 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:05.616 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:05.617 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:05.617 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.495669+0000", 2024-09-11T09:51:05.617 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:05.617 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:05.617 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:05.617 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:05.617 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.495669+0000", 2024-09-11T09:51:05.617 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:05.617 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:05.617 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:05.617 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:05.617 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.495670+0000", 2024-09-11T09:51:05.617 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.218e-06 2024-09-11T09:51:05.618 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:05.618 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:05.618 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:05.618 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.495676+0000", 2024-09-11T09:51:05.618 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.5119999999999997e-06 2024-09-11T09:51:05.618 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:05.618 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:05.618 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:05.618 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.495690+0000", 2024-09-11T09:51:05.618 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.3312000000000001e-05 2024-09-11T09:51:05.618 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:05.618 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:05.618 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:05.619 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.495706+0000", 2024-09-11T09:51:05.619 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.6560000000000001e-05 2024-09-11T09:51:05.619 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:05.619 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:05.619 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:05.619 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.495759+0000", 2024-09-11T09:51:05.619 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.2828000000000003e-05 2024-09-11T09:51:05.619 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:05.619 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:05.619 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:05.619 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.495765+0000", 2024-09-11T09:51:05.619 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.7869999999999998e-06 2024-09-11T09:51:05.619 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:05.620 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:05.620 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:05.620 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.495775+0000", 2024-09-11T09:51:05.620 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.0214000000000001e-05 2024-09-11T09:51:05.620 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:05.620 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:05.620 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:05.620 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:05.620 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:05.620 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4230.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-09-11T09:51:05.620 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:20.498634+0000", 2024-09-11T09:51:05.620 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 45.089096349000002, 2024-09-11T09:51:05.621 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0050648539999999997, 2024-09-11T09:51:05.621 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:05.621 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:05.621 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:05.621 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:05.621 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4230", 2024-09-11T09:51:05.621 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.17:0/4101109576", 2024-09-11T09:51:05.621 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 7 2024-09-11T09:51:05.621 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:05.621 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:05.621 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:05.621 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:05.621 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.498634+0000", 2024-09-11T09:51:05.622 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:05.622 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:05.622 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:05.622 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:05.622 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.498634+0000", 2024-09-11T09:51:05.622 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:05.622 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:05.622 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:05.622 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:05.622 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.498637+0000", 2024-09-11T09:51:05.622 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.1919999999999999e-06 2024-09-11T09:51:05.622 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:05.622 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:05.623 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:05.623 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.498648+0000", 2024-09-11T09:51:05.623 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1116e-05 2024-09-11T09:51:05.623 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:05.623 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:05.623 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:05.623 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.498666+0000", 2024-09-11T09:51:05.623 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.8411999999999999e-05 2024-09-11T09:51:05.623 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:05.623 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:05.623 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:05.623 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.498675+0000", 2024-09-11T09:51:05.623 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.4059999999999994e-06 2024-09-11T09:51:05.623 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:05.624 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:05.624 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:05.624 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.498745+0000", 2024-09-11T09:51:05.624 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.9764000000000005e-05 2024-09-11T09:51:05.624 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:05.624 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:05.624 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op must wait for map", 2024-09-11T09:51:05.624 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.498754+0000", 2024-09-11T09:51:05.624 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.5969999999999992e-06 2024-09-11T09:51:05.624 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:05.624 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:05.624 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:05.624 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.500385+0000", 2024-09-11T09:51:05.625 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0016308360000000001 2024-09-11T09:51:05.625 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:05.625 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:05.625 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:05.625 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.500473+0000", 2024-09-11T09:51:05.625 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.7966000000000004e-05 2024-09-11T09:51:05.625 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:05.625 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:05.625 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 1", 2024-09-11T09:51:05.625 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.500591+0000", 2024-09-11T09:51:05.625 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000117895 2024-09-11T09:51:05.625 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:05.625 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:05.625 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:05.626 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.501685+0000", 2024-09-11T09:51:05.626 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001094886 2024-09-11T09:51:05.626 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:05.626 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:05.626 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:05.626 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.503627+0000", 2024-09-11T09:51:05.626 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001941199 2024-09-11T09:51:05.626 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:05.626 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:05.626 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:05.626 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.503667+0000", 2024-09-11T09:51:05.626 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.0645000000000001e-05 2024-09-11T09:51:05.626 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:05.627 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:05.627 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:05.627 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.503699+0000", 2024-09-11T09:51:05.627 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.1940000000000003e-05 2024-09-11T09:51:05.627 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:05.627 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:05.627 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:05.627 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:05.627 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:05.627 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_repop_reply(client.4230.0:7 2.2 e18/15 ondisk, result = 0)", 2024-09-11T09:51:05.627 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:20.501608+0000", 2024-09-11T09:51:05.627 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 45.086122826999997, 2024-09-11T09:51:05.627 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.2459000000000005e-05, 2024-09-11T09:51:05.627 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:05.628 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:05.628 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "started", 2024-09-11T09:51:05.628 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:05.628 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:05.628 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:05.628 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.501608+0000", 2024-09-11T09:51:05.628 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:05.628 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:05.628 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:05.628 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:05.628 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.501608+0000", 2024-09-11T09:51:05.628 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:05.628 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:05.628 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:05.629 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:05.629 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.501609+0000", 2024-09-11T09:51:05.629 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.0160000000000001e-06 2024-09-11T09:51:05.629 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:05.629 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:05.629 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:05.629 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.501613+0000", 2024-09-11T09:51:05.629 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.5959999999999997e-06 2024-09-11T09:51:05.629 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:05.629 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:05.629 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:05.629 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.501625+0000", 2024-09-11T09:51:05.629 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1888e-05 2024-09-11T09:51:05.630 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:05.630 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:05.630 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:05.630 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.501634+0000", 2024-09-11T09:51:05.630 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.6139999999999992e-06 2024-09-11T09:51:05.630 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:05.630 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:05.630 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:05.630 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.501676+0000", 2024-09-11T09:51:05.630 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.1953999999999998e-05 2024-09-11T09:51:05.630 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:05.630 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:05.630 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:05.630 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.501682+0000", 2024-09-11T09:51:05.630 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.8000000000000004e-06 2024-09-11T09:51:05.631 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:05.631 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:05.631 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:05.631 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.501690+0000", 2024-09-11T09:51:05.631 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.5909999999999996e-06 2024-09-11T09:51:05.631 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:05.631 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:05.631 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:05.631 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:05.631 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:05.631 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-11T09:51:05.633 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:51:05.633+0000 7fdec5849640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31907) UID: 0 2024-09-11T09:51:05.633 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-11T09:51:05.658 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-11T09:51:05.733 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:51:05.734+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:51:05.755 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:51:05.755 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [], 2024-09-11T09:51:05.755 INFO:teuthology.orchestra.run.smithi017.stdout: "num_ops": 0 2024-09-11T09:51:05.756 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:51:05.765 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-11T09:51:05.834 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:51:05.835+0000 7f3c96ec9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31909) UID: 0 2024-09-11T09:51:05.913 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:51:05.914 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [], 2024-09-11T09:51:05.914 INFO:teuthology.orchestra.run.smithi017.stdout: "complaint_time": 30, 2024-09-11T09:51:05.914 INFO:teuthology.orchestra.run.smithi017.stdout: "num_blocked_ops": 0 2024-09-11T09:51:05.914 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:51:05.922 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-11T09:51:05.966 INFO:tasks.ceph.osd.0.smithi017.stderr:2024-09-11T09:51:05.967+0000 7fb73c0f1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31665) UID: 0 2024-09-11T09:51:06.032 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:51:06.032 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 20, 2024-09-11T09:51:06.032 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 600, 2024-09-11T09:51:06.032 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [ 2024-09-11T09:51:06.032 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.032 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4183.0:21 1.0 e13/12 1:30306672:devicehealth::main.db.0000000000000000:head v 13'11, mlcod=13'8)", 2024-09-11T09:51:06.032 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:11.491728+0000", 2024-09-11T09:51:06.032 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 54.540228489, 2024-09-11T09:51:06.032 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00093269800000000003, 2024-09-11T09:51:06.033 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:06.033 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:06.033 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:06.033 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:06.033 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.033 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:06.033 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.491728+0000", 2024-09-11T09:51:06.033 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:06.033 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.033 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.033 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:06.033 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.491728+0000", 2024-09-11T09:51:06.033 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:06.034 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.034 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.034 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:06.034 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.491729+0000", 2024-09-11T09:51:06.034 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.8599999999999997e-07 2024-09-11T09:51:06.034 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.034 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.034 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:06.034 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.491820+0000", 2024-09-11T09:51:06.034 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.0812999999999999e-05 2024-09-11T09:51:06.034 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.034 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.034 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:06.035 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.491833+0000", 2024-09-11T09:51:06.035 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.341e-05 2024-09-11T09:51:06.035 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.035 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.035 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:06.035 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.491841+0000", 2024-09-11T09:51:06.035 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.8730000000000007e-06 2024-09-11T09:51:06.035 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.035 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.035 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:06.035 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.491916+0000", 2024-09-11T09:51:06.035 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.4881000000000004e-05 2024-09-11T09:51:06.036 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.036 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.036 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:06.036 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.491931+0000", 2024-09-11T09:51:06.036 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.4825e-05 2024-09-11T09:51:06.036 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.036 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.036 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:06.036 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.492633+0000", 2024-09-11T09:51:06.036 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00070191600000000002 2024-09-11T09:51:06.036 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.036 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.036 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:06.037 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.492661+0000", 2024-09-11T09:51:06.037 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.8093999999999999e-05 2024-09-11T09:51:06.037 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:06.037 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:06.037 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:06.037 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.037 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.037 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4183.0:31 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'17, mlcod=13'14)", 2024-09-11T09:51:06.037 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:11.499319+0000", 2024-09-11T09:51:06.037 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 54.532637819999998, 2024-09-11T09:51:06.037 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0010073879999999999, 2024-09-11T09:51:06.037 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:06.038 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:06.038 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:06.038 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:06.038 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.038 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:06.038 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.499319+0000", 2024-09-11T09:51:06.038 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:06.038 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.038 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.038 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:06.038 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.499319+0000", 2024-09-11T09:51:06.038 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:06.038 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.039 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.039 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:06.039 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.499320+0000", 2024-09-11T09:51:06.039 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.7700000000000004e-07 2024-09-11T09:51:06.039 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.039 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.039 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:06.039 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.499326+0000", 2024-09-11T09:51:06.039 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.6420000000000001e-06 2024-09-11T09:51:06.039 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.039 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.039 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:06.039 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.499339+0000", 2024-09-11T09:51:06.040 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2537999999999999e-05 2024-09-11T09:51:06.040 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.040 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.040 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:06.040 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.499345+0000", 2024-09-11T09:51:06.040 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.8220000000000003e-06 2024-09-11T09:51:06.040 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.040 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.040 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:06.041 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.499403+0000", 2024-09-11T09:51:06.041 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.8814000000000003e-05 2024-09-11T09:51:06.041 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.041 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.041 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:06.041 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.499427+0000", 2024-09-11T09:51:06.041 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.3853999999999999e-05 2024-09-11T09:51:06.041 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.041 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.041 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:06.041 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.500294+0000", 2024-09-11T09:51:06.041 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00086622399999999997 2024-09-11T09:51:06.042 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.042 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.042 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:06.042 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.500326+0000", 2024-09-11T09:51:06.042 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.2716999999999999e-05 2024-09-11T09:51:06.042 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:06.042 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:06.042 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:06.042 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.042 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.042 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4183.0:32 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'18, mlcod=13'14)", 2024-09-11T09:51:06.042 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:11.499796+0000", 2024-09-11T09:51:06.042 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 54.532160613999999, 2024-09-11T09:51:06.043 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00092915100000000002, 2024-09-11T09:51:06.043 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:06.043 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:06.043 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:06.043 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:06.043 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.043 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:06.043 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.499796+0000", 2024-09-11T09:51:06.043 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:06.043 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.043 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.043 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:06.043 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.499796+0000", 2024-09-11T09:51:06.044 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:06.044 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.044 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.044 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:06.044 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.499797+0000", 2024-09-11T09:51:06.044 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.9400000000000004e-07 2024-09-11T09:51:06.044 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.044 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.044 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:06.044 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.499841+0000", 2024-09-11T09:51:06.045 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.3900000000000003e-05 2024-09-11T09:51:06.045 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.045 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.045 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:06.045 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.499852+0000", 2024-09-11T09:51:06.045 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1462e-05 2024-09-11T09:51:06.045 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.045 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.045 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:06.045 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.499858+0000", 2024-09-11T09:51:06.045 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.8320000000000002e-06 2024-09-11T09:51:06.045 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.046 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.046 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:06.046 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.499915+0000", 2024-09-11T09:51:06.046 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.7187e-05 2024-09-11T09:51:06.046 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.046 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.046 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:06.046 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.499930+0000", 2024-09-11T09:51:06.046 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.4928e-05 2024-09-11T09:51:06.046 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.046 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.046 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:06.046 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.500697+0000", 2024-09-11T09:51:06.047 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00076658399999999995 2024-09-11T09:51:06.047 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.047 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.047 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:06.047 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.500725+0000", 2024-09-11T09:51:06.047 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.8364000000000002e-05 2024-09-11T09:51:06.047 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:06.047 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:06.047 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:06.047 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.047 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.047 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4183.0:34 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'20, mlcod=13'15)", 2024-09-11T09:51:06.047 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:11.501016+0000", 2024-09-11T09:51:06.047 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 54.530941095999999, 2024-09-11T09:51:06.048 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00097815399999999996, 2024-09-11T09:51:06.048 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:06.048 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:06.048 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:06.048 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:06.048 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.048 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:06.048 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.501016+0000", 2024-09-11T09:51:06.048 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:06.048 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.048 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.048 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:06.049 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.501016+0000", 2024-09-11T09:51:06.049 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:06.049 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.049 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.049 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:06.050 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.501016+0000", 2024-09-11T09:51:06.050 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.3799999999999997e-07 2024-09-11T09:51:06.050 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.050 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.050 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:06.050 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.501023+0000", 2024-09-11T09:51:06.050 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.2180000000000002e-06 2024-09-11T09:51:06.050 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.050 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.050 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:06.050 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.501031+0000", 2024-09-11T09:51:06.050 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.5379999999999997e-06 2024-09-11T09:51:06.050 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.051 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.051 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:06.051 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.501035+0000", 2024-09-11T09:51:06.051 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.216e-06 2024-09-11T09:51:06.051 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.051 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.051 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:06.051 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.501105+0000", 2024-09-11T09:51:06.051 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.9757000000000003e-05 2024-09-11T09:51:06.051 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.051 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.051 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:06.051 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.501120+0000", 2024-09-11T09:51:06.051 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.5358000000000001e-05 2024-09-11T09:51:06.052 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.052 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.052 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:06.052 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.501955+0000", 2024-09-11T09:51:06.052 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00083505000000000005 2024-09-11T09:51:06.052 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.052 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.052 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:06.052 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.501994+0000", 2024-09-11T09:51:06.052 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.8479000000000002e-05 2024-09-11T09:51:06.052 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:06.052 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:06.052 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:06.053 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.053 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.053 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4183.0:38 1.0 e13/12 1:30306672:devicehealth::main.db.0000000000000000:head v 13'24, mlcod=13'22)", 2024-09-11T09:51:06.053 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:11.503891+0000", 2024-09-11T09:51:06.053 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 54.528065654000002, 2024-09-11T09:51:06.053 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000959351, 2024-09-11T09:51:06.053 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:06.053 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:06.053 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:06.053 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:06.053 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.053 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:06.053 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.503891+0000", 2024-09-11T09:51:06.053 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:06.054 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.054 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.054 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:06.054 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.503891+0000", 2024-09-11T09:51:06.054 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:06.054 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.054 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.054 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:06.054 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.503892+0000", 2024-09-11T09:51:06.054 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.5499999999999998e-07 2024-09-11T09:51:06.054 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.054 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.054 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:06.055 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.503967+0000", 2024-09-11T09:51:06.055 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.4993999999999994e-05 2024-09-11T09:51:06.055 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.055 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.055 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:06.055 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.503976+0000", 2024-09-11T09:51:06.055 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.4709999999999996e-06 2024-09-11T09:51:06.055 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.055 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.055 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:06.055 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.503981+0000", 2024-09-11T09:51:06.055 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.6789999999999996e-06 2024-09-11T09:51:06.055 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.055 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.056 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:06.056 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.504021+0000", 2024-09-11T09:51:06.056 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.0065999999999999e-05 2024-09-11T09:51:06.056 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.056 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.056 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:06.056 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.504036+0000", 2024-09-11T09:51:06.057 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.468e-05 2024-09-11T09:51:06.057 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.057 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.057 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:06.057 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.504813+0000", 2024-09-11T09:51:06.057 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00077718299999999998 2024-09-11T09:51:06.057 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.057 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.057 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:06.057 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.504850+0000", 2024-09-11T09:51:06.057 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.7722999999999998e-05 2024-09-11T09:51:06.057 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:06.058 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:06.058 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:06.058 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.058 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.058 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4183.0:54 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'32, mlcod=13'29)", 2024-09-11T09:51:06.058 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:11.515404+0000", 2024-09-11T09:51:06.058 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 54.516553049000002, 2024-09-11T09:51:06.058 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00089471799999999997, 2024-09-11T09:51:06.058 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:06.058 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:06.058 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:06.058 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:06.058 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.059 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:06.059 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.515404+0000", 2024-09-11T09:51:06.059 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:06.059 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.059 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.059 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:06.059 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.515404+0000", 2024-09-11T09:51:06.059 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:06.060 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.060 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.060 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:06.060 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.515404+0000", 2024-09-11T09:51:06.060 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.3799999999999996e-07 2024-09-11T09:51:06.060 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.060 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.060 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:06.060 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.515410+0000", 2024-09-11T09:51:06.060 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.1789999999999999e-06 2024-09-11T09:51:06.061 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.061 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.061 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:06.061 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.515422+0000", 2024-09-11T09:51:06.061 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1929999999999999e-05 2024-09-11T09:51:06.061 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.061 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.061 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:06.061 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.515427+0000", 2024-09-11T09:51:06.061 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.6060000000000002e-06 2024-09-11T09:51:06.061 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.061 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.061 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:06.061 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.515482+0000", 2024-09-11T09:51:06.062 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.4743000000000001e-05 2024-09-11T09:51:06.062 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.062 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.062 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:06.062 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.515498+0000", 2024-09-11T09:51:06.062 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.5778999999999999e-05 2024-09-11T09:51:06.062 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.062 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.062 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:06.062 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.516271+0000", 2024-09-11T09:51:06.062 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00077312599999999996 2024-09-11T09:51:06.062 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.062 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.062 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:06.063 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.516298+0000", 2024-09-11T09:51:06.063 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.7617e-05 2024-09-11T09:51:06.063 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:06.063 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:06.063 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:06.063 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.063 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.063 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4183.0:57 1.0 e13/12 1:30306672:devicehealth::main.db.0000000000000000:head v 13'35, mlcod=13'33)", 2024-09-11T09:51:06.063 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:11.519823+0000", 2024-09-11T09:51:06.063 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 54.512133992000003, 2024-09-11T09:51:06.063 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00090876100000000001, 2024-09-11T09:51:06.063 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:06.064 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:06.064 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:06.064 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:06.064 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.064 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:06.064 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.519823+0000", 2024-09-11T09:51:06.064 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:06.064 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.064 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.064 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:06.064 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.519823+0000", 2024-09-11T09:51:06.065 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:06.065 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.065 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.065 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:06.065 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.519823+0000", 2024-09-11T09:51:06.065 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.7000000000000004e-07 2024-09-11T09:51:06.065 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.065 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.065 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:06.065 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.519859+0000", 2024-09-11T09:51:06.065 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.5508e-05 2024-09-11T09:51:06.065 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.065 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.066 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:06.066 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.519871+0000", 2024-09-11T09:51:06.066 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.221e-05 2024-09-11T09:51:06.066 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.066 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.066 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:06.066 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.519877+0000", 2024-09-11T09:51:06.066 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.0410000000000003e-06 2024-09-11T09:51:06.066 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.066 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.066 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:06.066 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.519955+0000", 2024-09-11T09:51:06.066 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.8169000000000003e-05 2024-09-11T09:51:06.067 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.067 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.067 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:06.067 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.519972+0000", 2024-09-11T09:51:06.067 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6595999999999998e-05 2024-09-11T09:51:06.067 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.067 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.067 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:06.067 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.520691+0000", 2024-09-11T09:51:06.067 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000719405 2024-09-11T09:51:06.067 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.067 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.067 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:06.068 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.520731+0000", 2024-09-11T09:51:06.068 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.0061999999999997e-05 2024-09-11T09:51:06.068 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:06.068 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:06.068 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:06.068 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.068 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.068 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4183.0:58 1.0 e13/12 1:30306672:devicehealth::main.db.0000000000000000:head v 13'36, mlcod=13'33)", 2024-09-11T09:51:06.068 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:11.520285+0000", 2024-09-11T09:51:06.068 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 54.511671583999998, 2024-09-11T09:51:06.068 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00091014799999999995, 2024-09-11T09:51:06.068 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:06.068 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:06.069 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:06.069 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:06.069 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.069 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:06.069 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.520285+0000", 2024-09-11T09:51:06.069 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:06.069 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.069 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.069 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:06.069 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.520285+0000", 2024-09-11T09:51:06.069 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:06.069 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.069 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.069 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:06.070 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.520286+0000", 2024-09-11T09:51:06.070 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.4399999999999999e-07 2024-09-11T09:51:06.070 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.070 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.070 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:06.070 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.520348+0000", 2024-09-11T09:51:06.070 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.1760000000000005e-05 2024-09-11T09:51:06.070 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.070 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.070 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:06.070 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.520359+0000", 2024-09-11T09:51:06.070 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1362999999999999e-05 2024-09-11T09:51:06.070 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.071 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.071 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:06.071 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.520364+0000", 2024-09-11T09:51:06.071 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.3600000000000004e-06 2024-09-11T09:51:06.071 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.071 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.071 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:06.071 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.520413+0000", 2024-09-11T09:51:06.071 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.8226000000000002e-05 2024-09-11T09:51:06.071 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.071 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.071 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:06.071 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.520426+0000", 2024-09-11T09:51:06.072 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3560000000000001e-05 2024-09-11T09:51:06.072 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.072 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.072 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:06.072 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.521159+0000", 2024-09-11T09:51:06.072 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00073255499999999997 2024-09-11T09:51:06.072 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.072 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.072 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:06.072 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.521195+0000", 2024-09-11T09:51:06.072 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.6480000000000003e-05 2024-09-11T09:51:06.072 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:06.072 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:06.072 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:06.073 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.073 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.073 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4183.0:68 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'42, mlcod=13'39)", 2024-09-11T09:51:06.073 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:11.534957+0000", 2024-09-11T09:51:06.073 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 54.496999893000002, 2024-09-11T09:51:06.073 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.001065351, 2024-09-11T09:51:06.073 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:06.073 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:06.073 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:06.073 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:06.073 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.073 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:06.073 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.534957+0000", 2024-09-11T09:51:06.074 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:06.074 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.074 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.074 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:06.074 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.534957+0000", 2024-09-11T09:51:06.074 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:06.074 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.074 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.074 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:06.074 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.534957+0000", 2024-09-11T09:51:06.074 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.3799999999999998e-07 2024-09-11T09:51:06.074 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.074 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.074 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:06.075 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.534964+0000", 2024-09-11T09:51:06.075 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.9809999999999997e-06 2024-09-11T09:51:06.075 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.075 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.075 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:06.075 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.534976+0000", 2024-09-11T09:51:06.075 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2119999999999999e-05 2024-09-11T09:51:06.075 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.075 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.075 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:06.075 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.534980+0000", 2024-09-11T09:51:06.075 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.9890000000000003e-06 2024-09-11T09:51:06.075 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.075 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.076 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:06.076 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.535030+0000", 2024-09-11T09:51:06.076 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.9258000000000002e-05 2024-09-11T09:51:06.076 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.076 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.076 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:06.076 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.535044+0000", 2024-09-11T09:51:06.076 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3981e-05 2024-09-11T09:51:06.076 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.076 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.076 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:06.076 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.535995+0000", 2024-09-11T09:51:06.076 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000951804 2024-09-11T09:51:06.077 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.077 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.077 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:06.077 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.536022+0000", 2024-09-11T09:51:06.077 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.6780000000000001e-05 2024-09-11T09:51:06.077 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:06.077 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:06.077 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:06.077 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.077 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.077 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4183.0:69 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'43, mlcod=13'39)", 2024-09-11T09:51:06.077 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:11.535331+0000", 2024-09-11T09:51:06.077 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 54.496625569999999, 2024-09-11T09:51:06.077 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0011256090000000001, 2024-09-11T09:51:06.078 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:06.078 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:06.078 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:06.078 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:06.078 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.078 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:06.078 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.535331+0000", 2024-09-11T09:51:06.078 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:06.078 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.078 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.078 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:06.078 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.535331+0000", 2024-09-11T09:51:06.078 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:06.078 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.079 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.079 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:06.079 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.535332+0000", 2024-09-11T09:51:06.079 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.1600000000000001e-07 2024-09-11T09:51:06.079 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.079 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.079 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:06.079 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.535393+0000", 2024-09-11T09:51:06.079 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.1452999999999996e-05 2024-09-11T09:51:06.079 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.079 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.080 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:06.080 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.535402+0000", 2024-09-11T09:51:06.080 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.4300000000000006e-06 2024-09-11T09:51:06.080 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.080 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.080 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:06.080 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.535405+0000", 2024-09-11T09:51:06.080 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.512e-06 2024-09-11T09:51:06.080 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.080 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.080 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:06.080 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.535482+0000", 2024-09-11T09:51:06.080 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.7132999999999995e-05 2024-09-11T09:51:06.081 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.081 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.081 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:06.081 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.535506+0000", 2024-09-11T09:51:06.081 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.4070000000000002e-05 2024-09-11T09:51:06.081 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.081 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.081 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:06.081 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.536424+0000", 2024-09-11T09:51:06.081 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00091766700000000005 2024-09-11T09:51:06.081 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.081 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.081 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:06.081 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.536457+0000", 2024-09-11T09:51:06.082 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.2827999999999998e-05 2024-09-11T09:51:06.082 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:06.082 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:06.082 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:06.082 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.082 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.082 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4183.0:83 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'53, mlcod=13'50)", 2024-09-11T09:51:06.082 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:11.558836+0000", 2024-09-11T09:51:06.082 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 54.473120315000003, 2024-09-11T09:51:06.082 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00090313100000000005, 2024-09-11T09:51:06.082 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:06.082 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:06.082 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:06.083 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:06.083 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.083 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:06.083 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.558836+0000", 2024-09-11T09:51:06.083 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:06.083 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.083 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.083 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:06.083 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.558836+0000", 2024-09-11T09:51:06.083 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:06.083 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.083 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.083 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:06.083 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.558837+0000", 2024-09-11T09:51:06.084 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.4900000000000005e-07 2024-09-11T09:51:06.084 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.084 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.084 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:06.084 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.558857+0000", 2024-09-11T09:51:06.084 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.9502999999999999e-05 2024-09-11T09:51:06.084 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.084 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.084 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:06.084 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.558870+0000", 2024-09-11T09:51:06.084 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3020000000000001e-05 2024-09-11T09:51:06.084 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.085 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.085 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:06.085 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.558876+0000", 2024-09-11T09:51:06.085 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.7270000000000004e-06 2024-09-11T09:51:06.085 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.085 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.085 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:06.085 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.558934+0000", 2024-09-11T09:51:06.085 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.8100999999999998e-05 2024-09-11T09:51:06.085 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.085 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.085 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:06.085 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.558947+0000", 2024-09-11T09:51:06.086 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3269e-05 2024-09-11T09:51:06.086 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.086 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.086 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:06.086 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.559720+0000", 2024-09-11T09:51:06.086 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00077287600000000003 2024-09-11T09:51:06.086 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.086 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.086 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:06.086 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.559740+0000", 2024-09-11T09:51:06.086 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.9786e-05 2024-09-11T09:51:06.086 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:06.086 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:06.086 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:06.087 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.087 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.087 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4183.0:84 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'54, mlcod=13'50)", 2024-09-11T09:51:06.087 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:11.559267+0000", 2024-09-11T09:51:06.087 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 54.472689887999998, 2024-09-11T09:51:06.087 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00093990199999999995, 2024-09-11T09:51:06.087 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:06.087 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:06.087 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:06.087 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:06.087 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.087 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:06.087 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.559267+0000", 2024-09-11T09:51:06.088 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:06.088 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.088 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.088 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:06.088 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.559267+0000", 2024-09-11T09:51:06.088 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:06.088 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.088 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.088 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:06.088 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.559267+0000", 2024-09-11T09:51:06.088 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.2900000000000003e-07 2024-09-11T09:51:06.088 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.088 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.089 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:06.089 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.559322+0000", 2024-09-11T09:51:06.089 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.4429999999999999e-05 2024-09-11T09:51:06.089 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.089 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.089 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:06.089 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.559332+0000", 2024-09-11T09:51:06.089 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0473e-05 2024-09-11T09:51:06.089 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.089 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.089 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:06.089 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.559338+0000", 2024-09-11T09:51:06.089 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.0730000000000001e-06 2024-09-11T09:51:06.090 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.090 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.090 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:06.090 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.559373+0000", 2024-09-11T09:51:06.090 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.4626999999999999e-05 2024-09-11T09:51:06.090 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.090 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.090 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:06.090 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.559384+0000", 2024-09-11T09:51:06.090 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1307e-05 2024-09-11T09:51:06.090 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.090 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.090 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:06.090 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.560175+0000", 2024-09-11T09:51:06.091 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00079016399999999999 2024-09-11T09:51:06.091 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.091 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.091 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:06.091 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.560207+0000", 2024-09-11T09:51:06.091 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.2199000000000001e-05 2024-09-11T09:51:06.091 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:06.091 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:06.091 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:06.091 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.091 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.091 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4183.0:89 1.0 e13/12 1:30306672:devicehealth::main.db.0000000000000000:head v 13'59, mlcod=13'57)", 2024-09-11T09:51:06.091 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:11.563614+0000", 2024-09-11T09:51:06.092 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 54.468342268000001, 2024-09-11T09:51:06.092 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00091281399999999996, 2024-09-11T09:51:06.092 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:06.092 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:06.092 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:06.092 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:06.092 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.092 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:06.092 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.563614+0000", 2024-09-11T09:51:06.092 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:06.092 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.092 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.092 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:06.092 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.563614+0000", 2024-09-11T09:51:06.093 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:06.093 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.093 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.093 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:06.093 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.563615+0000", 2024-09-11T09:51:06.093 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.6899999999999998e-07 2024-09-11T09:51:06.093 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.093 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.093 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:06.093 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.563681+0000", 2024-09-11T09:51:06.093 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.5769e-05 2024-09-11T09:51:06.093 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.093 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.093 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:06.094 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.563690+0000", 2024-09-11T09:51:06.094 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.6800000000000005e-06 2024-09-11T09:51:06.094 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.094 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.094 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:06.094 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.563699+0000", 2024-09-11T09:51:06.094 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.3170000000000002e-06 2024-09-11T09:51:06.094 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.094 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.094 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:06.094 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.563734+0000", 2024-09-11T09:51:06.094 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.5023e-05 2024-09-11T09:51:06.095 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.095 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.095 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:06.095 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.563746+0000", 2024-09-11T09:51:06.095 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2661e-05 2024-09-11T09:51:06.095 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.095 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.095 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:06.095 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.564500+0000", 2024-09-11T09:51:06.095 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00075323100000000004 2024-09-11T09:51:06.095 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.095 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.095 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:06.095 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.564527+0000", 2024-09-11T09:51:06.096 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.7664000000000001e-05 2024-09-11T09:51:06.096 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:06.096 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:06.096 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:06.096 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.096 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.096 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4183.0:105 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'67, mlcod=13'64)", 2024-09-11T09:51:06.096 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:11.712146+0000", 2024-09-11T09:51:06.096 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 54.319810517999997, 2024-09-11T09:51:06.096 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0010916859999999999, 2024-09-11T09:51:06.097 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:06.097 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:06.097 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:06.097 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:06.097 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.097 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:06.097 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.712146+0000", 2024-09-11T09:51:06.097 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:06.097 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.097 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.097 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:06.097 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.712146+0000", 2024-09-11T09:51:06.097 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:06.097 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.098 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.098 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:06.098 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.712147+0000", 2024-09-11T09:51:06.098 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.8200000000000001e-07 2024-09-11T09:51:06.098 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.098 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.098 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:06.098 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.712155+0000", 2024-09-11T09:51:06.098 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.0309999999999998e-06 2024-09-11T09:51:06.098 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.098 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.098 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:06.098 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.712163+0000", 2024-09-11T09:51:06.099 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.5920000000000001e-06 2024-09-11T09:51:06.099 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.099 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.099 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:06.099 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.712169+0000", 2024-09-11T09:51:06.099 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.4299999999999997e-06 2024-09-11T09:51:06.099 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.099 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.099 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:06.099 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.712418+0000", 2024-09-11T09:51:06.099 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00024978499999999998 2024-09-11T09:51:06.099 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.100 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.100 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:06.100 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.712432+0000", 2024-09-11T09:51:06.100 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3190000000000001e-05 2024-09-11T09:51:06.100 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.100 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.100 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:06.100 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.713206+0000", 2024-09-11T09:51:06.100 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00077420199999999998 2024-09-11T09:51:06.100 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.100 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.101 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:06.101 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.713238+0000", 2024-09-11T09:51:06.101 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.2073999999999998e-05 2024-09-11T09:51:06.101 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:06.101 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:06.101 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:06.101 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.101 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.101 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4183.0:106 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'68, mlcod=13'64)", 2024-09-11T09:51:06.101 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:11.712624+0000", 2024-09-11T09:51:06.101 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 54.319333190000002, 2024-09-11T09:51:06.101 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0010009629999999999, 2024-09-11T09:51:06.101 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:06.102 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:06.102 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:06.102 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:06.102 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.102 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:06.102 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.712624+0000", 2024-09-11T09:51:06.102 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:06.102 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.102 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.102 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:06.102 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.712624+0000", 2024-09-11T09:51:06.102 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:06.102 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.103 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.103 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:06.103 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.712624+0000", 2024-09-11T09:51:06.103 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.4799999999999998e-07 2024-09-11T09:51:06.103 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.103 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.103 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:06.103 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.712689+0000", 2024-09-11T09:51:06.103 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.4488999999999999e-05 2024-09-11T09:51:06.103 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.103 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.103 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:06.103 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.712707+0000", 2024-09-11T09:51:06.103 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.8272000000000001e-05 2024-09-11T09:51:06.104 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.104 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.104 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:06.104 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.712713+0000", 2024-09-11T09:51:06.104 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.2040000000000002e-06 2024-09-11T09:51:06.104 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.104 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.104 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:06.104 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.712783+0000", 2024-09-11T09:51:06.104 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.9534000000000002e-05 2024-09-11T09:51:06.104 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.104 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.104 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:06.105 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.712811+0000", 2024-09-11T09:51:06.105 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.8620999999999999e-05 2024-09-11T09:51:06.105 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.105 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.105 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:06.105 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.713586+0000", 2024-09-11T09:51:06.105 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00077462600000000005 2024-09-11T09:51:06.105 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.105 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.105 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:06.105 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.713624+0000", 2024-09-11T09:51:06.105 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.8668999999999999e-05 2024-09-11T09:51:06.105 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:06.106 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:06.106 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:06.106 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.106 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.106 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4183.0:134 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'92, mlcod=13'86)", 2024-09-11T09:51:06.106 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:12.035698+0000", 2024-09-11T09:51:06.106 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 53.996258931, 2024-09-11T09:51:06.106 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00089550700000000003, 2024-09-11T09:51:06.106 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:06.106 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:06.106 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:06.106 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:06.106 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.107 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:06.107 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.035698+0000", 2024-09-11T09:51:06.107 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:06.107 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.107 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.107 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:06.107 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.035698+0000", 2024-09-11T09:51:06.107 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:06.107 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.107 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.107 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:06.107 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.035698+0000", 2024-09-11T09:51:06.107 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.0900000000000002e-07 2024-09-11T09:51:06.108 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.108 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.108 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:06.108 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.035758+0000", 2024-09-11T09:51:06.108 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.9481999999999997e-05 2024-09-11T09:51:06.108 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.108 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.108 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:06.108 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.035766+0000", 2024-09-11T09:51:06.108 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.3760000000000008e-06 2024-09-11T09:51:06.108 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.108 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.108 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:06.109 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.035772+0000", 2024-09-11T09:51:06.109 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.8930000000000002e-06 2024-09-11T09:51:06.109 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.109 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.109 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:06.109 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.035807+0000", 2024-09-11T09:51:06.109 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.4975999999999999e-05 2024-09-11T09:51:06.109 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.109 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.109 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:06.109 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.035817+0000", 2024-09-11T09:51:06.109 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0314e-05 2024-09-11T09:51:06.109 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.110 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.110 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:06.110 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.036542+0000", 2024-09-11T09:51:06.110 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00072434400000000003 2024-09-11T09:51:06.110 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.110 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.110 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:06.110 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.036593+0000", 2024-09-11T09:51:06.110 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.1613000000000002e-05 2024-09-11T09:51:06.110 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:06.110 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:06.110 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:06.111 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.111 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.111 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4183.0:136 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'94, mlcod=13'86)", 2024-09-11T09:51:06.111 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:12.036591+0000", 2024-09-11T09:51:06.111 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 53.995365720999999, 2024-09-11T09:51:06.111 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00090637900000000002, 2024-09-11T09:51:06.111 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:06.111 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:06.111 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:06.111 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:06.111 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.111 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:06.111 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.036591+0000", 2024-09-11T09:51:06.111 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:06.112 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.112 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.112 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:06.112 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.036591+0000", 2024-09-11T09:51:06.112 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:06.112 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.112 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.112 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:06.112 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.036592+0000", 2024-09-11T09:51:06.112 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.37e-07 2024-09-11T09:51:06.112 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.112 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.113 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:06.113 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.036600+0000", 2024-09-11T09:51:06.113 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.6910000000000003e-06 2024-09-11T09:51:06.113 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.113 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.113 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:06.113 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.036609+0000", 2024-09-11T09:51:06.113 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.0229999999999998e-06 2024-09-11T09:51:06.113 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.113 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.113 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:06.113 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.036617+0000", 2024-09-11T09:51:06.113 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.6979999999999998e-06 2024-09-11T09:51:06.114 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.114 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.114 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:06.114 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.036658+0000", 2024-09-11T09:51:06.114 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.0692000000000003e-05 2024-09-11T09:51:06.114 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.114 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.114 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:06.114 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.036670+0000", 2024-09-11T09:51:06.114 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2109e-05 2024-09-11T09:51:06.114 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.114 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.114 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:06.115 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.037475+0000", 2024-09-11T09:51:06.115 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00080487199999999999 2024-09-11T09:51:06.115 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.115 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.115 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:06.115 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.037497+0000", 2024-09-11T09:51:06.115 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.2557000000000001e-05 2024-09-11T09:51:06.115 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:06.115 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:06.115 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:06.115 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.115 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.115 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4183.0:158 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'112, mlcod=13'108)", 2024-09-11T09:51:06.115 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:12.455073+0000", 2024-09-11T09:51:06.115 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 53.576883467999998, 2024-09-11T09:51:06.116 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00090654500000000003, 2024-09-11T09:51:06.116 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:06.116 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:06.116 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:06.116 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:06.116 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.116 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:06.116 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.455073+0000", 2024-09-11T09:51:06.116 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:06.116 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.116 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.116 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:06.116 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.455073+0000", 2024-09-11T09:51:06.117 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:06.117 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.117 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.117 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:06.117 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.455074+0000", 2024-09-11T09:51:06.117 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.32e-07 2024-09-11T09:51:06.117 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.117 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.117 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:06.117 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.455126+0000", 2024-09-11T09:51:06.117 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.1898999999999999e-05 2024-09-11T09:51:06.117 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.118 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.118 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:06.118 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.455136+0000", 2024-09-11T09:51:06.118 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0651e-05 2024-09-11T09:51:06.118 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.118 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.118 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:06.118 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.455142+0000", 2024-09-11T09:51:06.118 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.0020000000000003e-06 2024-09-11T09:51:06.118 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.118 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.118 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:06.119 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.455192+0000", 2024-09-11T09:51:06.119 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.0164000000000003e-05 2024-09-11T09:51:06.119 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.119 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.119 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:06.119 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.455204+0000", 2024-09-11T09:51:06.119 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1178999999999999e-05 2024-09-11T09:51:06.119 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.119 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.119 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:06.119 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.455957+0000", 2024-09-11T09:51:06.119 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00075362300000000003 2024-09-11T09:51:06.119 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.120 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.120 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:06.120 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.455980+0000", 2024-09-11T09:51:06.120 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.2595e-05 2024-09-11T09:51:06.120 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:06.120 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:06.120 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:06.120 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.120 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.120 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "MOSDPGPush(1.0 34/33 [PushOp(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head, version: 13'116, data_included: [0~262688], data_size: 262688, omap_header_size: 0, omap_entries_size: 0, attrset_size: 9, recovery_info: ObjectRecoveryInfo(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head@13'116, size: 262688, copy_subset: [(0, 262688)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 262688, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-11T09:51:06.120 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:47.599030+0000", 2024-09-11T09:51:06.120 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 18.432926842000001, 2024-09-11T09:51:06.120 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0010572209999999999, 2024-09-11T09:51:06.121 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:06.121 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:06.121 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "started", 2024-09-11T09:51:06.121 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:06.121 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.121 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:06.121 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:47.599030+0000", 2024-09-11T09:51:06.121 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:06.121 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.121 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.121 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:06.121 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:47.599030+0000", 2024-09-11T09:51:06.121 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:06.122 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.122 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.122 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:06.122 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:47.599031+0000", 2024-09-11T09:51:06.122 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.92e-07 2024-09-11T09:51:06.122 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.122 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.122 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:06.122 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:47.599328+0000", 2024-09-11T09:51:06.122 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00029708399999999998 2024-09-11T09:51:06.122 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.122 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.122 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:06.123 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:47.599353+0000", 2024-09-11T09:51:06.123 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.5721e-05 2024-09-11T09:51:06.123 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.123 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.123 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:06.123 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:47.599375+0000", 2024-09-11T09:51:06.123 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.1990000000000001e-05 2024-09-11T09:51:06.123 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.123 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.123 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:06.123 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:47.599466+0000", 2024-09-11T09:51:06.124 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.0175999999999999e-05 2024-09-11T09:51:06.124 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.124 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.124 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:06.124 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:47.599480+0000", 2024-09-11T09:51:06.124 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.4873e-05 2024-09-11T09:51:06.124 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.124 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.124 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:06.124 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:47.600087+0000", 2024-09-11T09:51:06.124 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00060668500000000004 2024-09-11T09:51:06.124 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:06.125 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:06.125 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:06.125 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.125 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.125 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "MOSDPGPush(1.0 34/33 [PushOp(1:30306672:devicehealth::main.db.0000000000000000:head, version: 13'117, data_included: [0~327680], data_size: 327680, omap_header_size: 0, omap_entries_size: 0, attrset_size: 10, recovery_info: ObjectRecoveryInfo(1:30306672:devicehealth::main.db.0000000000000000:head@13'117, size: 327680, copy_subset: [(0, 327680)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 327680, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-11T09:51:06.125 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:47.601120+0000", 2024-09-11T09:51:06.125 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 18.430836354, 2024-09-11T09:51:06.125 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00099708000000000001, 2024-09-11T09:51:06.125 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:06.125 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:06.125 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "started", 2024-09-11T09:51:06.125 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:06.125 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.126 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:06.126 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:47.601120+0000", 2024-09-11T09:51:06.126 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:06.126 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.126 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.126 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:06.126 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:47.601120+0000", 2024-09-11T09:51:06.126 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:06.126 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.126 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.126 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:06.126 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:47.601121+0000", 2024-09-11T09:51:06.126 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.0200000000000001e-07 2024-09-11T09:51:06.126 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.127 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.127 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:06.127 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:47.601430+0000", 2024-09-11T09:51:06.127 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00030917899999999998 2024-09-11T09:51:06.127 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.127 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.127 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:06.127 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:47.601454+0000", 2024-09-11T09:51:06.127 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.4000999999999999e-05 2024-09-11T09:51:06.127 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.127 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.127 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:06.127 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:47.601475+0000", 2024-09-11T09:51:06.127 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.0305999999999998e-05 2024-09-11T09:51:06.128 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.128 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.128 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:06.128 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:47.601561+0000", 2024-09-11T09:51:06.128 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.6414000000000003e-05 2024-09-11T09:51:06.128 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.128 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.128 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:06.128 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:47.601573+0000", 2024-09-11T09:51:06.128 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1894e-05 2024-09-11T09:51:06.128 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.128 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.128 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:06.128 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:47.602117+0000", 2024-09-11T09:51:06.129 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00054458399999999998 2024-09-11T09:51:06.129 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:06.129 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:06.129 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:06.129 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:06.129 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:06.129 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:51:06.130 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-11T09:51:06.131 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:51:06.131+0000 7fdec5849640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31907) UID: 0 2024-09-11T09:51:06.231 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:51:06.232+0000 7fdec5849640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31907) UID: 0 2024-09-11T09:51:06.279 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:51:06.279 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [], 2024-09-11T09:51:06.279 INFO:teuthology.orchestra.run.smithi017.stdout: "num_ops": 0 2024-09-11T09:51:06.279 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:51:06.288 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-11T09:51:06.332 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:51:06.333+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:51:06.394 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:51:06.394 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [], 2024-09-11T09:51:06.395 INFO:teuthology.orchestra.run.smithi017.stdout: "complaint_time": 30, 2024-09-11T09:51:06.395 INFO:teuthology.orchestra.run.smithi017.stdout: "num_blocked_ops": 0 2024-09-11T09:51:06.395 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:51:06.403 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-11T09:51:06.444 INFO:tasks.ceph.osd.0.smithi017.stderr:2024-09-11T09:51:06.445+0000 7fb73c0f1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31665) UID: 0 2024-09-11T09:51:06.535 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:51:06.536+0000 7fdec5849640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31907) UID: 0 2024-09-11T09:51:06.551 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:51:06.551 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 20, 2024-09-11T09:51:06.551 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 600, 2024-09-11T09:51:06.551 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [ 2024-09-11T09:51:06.551 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.551 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4230.0:5 2.2 e17/15 2:5cea7035:::rbd_info:head v 17'1, mlcod=0'0)", 2024-09-11T09:51:06.551 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:19.494757+0000", 2024-09-11T09:51:06.551 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 47.057700179999998, 2024-09-11T09:51:06.551 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00082098399999999997, 2024-09-11T09:51:06.551 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:06.552 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:06.552 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:06.552 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:06.552 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.552 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:06.552 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:19.494757+0000", 2024-09-11T09:51:06.552 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:06.552 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.552 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.552 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:06.552 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:19.494757+0000", 2024-09-11T09:51:06.552 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:06.553 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.553 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.553 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:06.553 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:19.494757+0000", 2024-09-11T09:51:06.553 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.97e-07 2024-09-11T09:51:06.553 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.553 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.553 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:06.553 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:19.494763+0000", 2024-09-11T09:51:06.553 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.3709999999999996e-06 2024-09-11T09:51:06.553 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.553 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.553 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:06.554 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:19.494778+0000", 2024-09-11T09:51:06.554 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.4960000000000001e-05 2024-09-11T09:51:06.554 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.554 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.554 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:06.554 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:19.494801+0000", 2024-09-11T09:51:06.554 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.2388999999999999e-05 2024-09-11T09:51:06.554 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.554 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.554 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:06.554 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:19.494859+0000", 2024-09-11T09:51:06.554 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.7961999999999998e-05 2024-09-11T09:51:06.554 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.555 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.555 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:06.555 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:19.494883+0000", 2024-09-11T09:51:06.555 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.4488999999999999e-05 2024-09-11T09:51:06.555 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.555 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.555 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:06.555 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:19.495534+0000", 2024-09-11T09:51:06.555 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00065075500000000004 2024-09-11T09:51:06.555 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.555 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.555 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:06.555 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:19.495578+0000", 2024-09-11T09:51:06.555 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.3860999999999999e-05 2024-09-11T09:51:06.556 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:06.556 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:06.556 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:06.556 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.556 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.556 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4230.0:7 2.2 e18/15 2:5cea7035:::rbd_info:head v 18'2, mlcod=0'0)", 2024-09-11T09:51:06.556 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:20.500782+0000", 2024-09-11T09:51:06.556 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 46.051675299999999, 2024-09-11T09:51:06.556 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000790185, 2024-09-11T09:51:06.556 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:06.556 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:06.557 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:06.557 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:06.557 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.557 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:06.557 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:20.500782+0000", 2024-09-11T09:51:06.557 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:06.557 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.557 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.557 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:06.557 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:20.500782+0000", 2024-09-11T09:51:06.557 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:06.557 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.557 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.557 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:06.558 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:20.500782+0000", 2024-09-11T09:51:06.558 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.3299999999999998e-07 2024-09-11T09:51:06.558 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.558 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.558 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:06.558 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:20.500799+0000", 2024-09-11T09:51:06.558 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6830999999999999e-05 2024-09-11T09:51:06.558 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.558 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.558 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:06.558 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:20.500812+0000", 2024-09-11T09:51:06.558 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3013999999999999e-05 2024-09-11T09:51:06.558 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.559 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.559 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:06.559 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:20.500824+0000", 2024-09-11T09:51:06.559 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2593e-05 2024-09-11T09:51:06.559 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.559 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.559 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:06.559 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:20.500895+0000", 2024-09-11T09:51:06.559 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.1018000000000003e-05 2024-09-11T09:51:06.559 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.559 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.559 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:06.560 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:20.500912+0000", 2024-09-11T09:51:06.560 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6484e-05 2024-09-11T09:51:06.560 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.560 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.560 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:06.560 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:20.501531+0000", 2024-09-11T09:51:06.560 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00061903399999999997 2024-09-11T09:51:06.560 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:06.560 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:06.560 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:06.560 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:20.501572+0000", 2024-09-11T09:51:06.560 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.0877999999999997e-05 2024-09-11T09:51:06.560 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:06.561 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:06.561 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:06.561 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:06.561 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:06.561 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:51:06.562 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-11T09:51:06.636 INFO:tasks.ceph.osd.2.smithi017.stderr:2024-09-11T09:51:06.637+0000 7fd20b170640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31672) UID: 0 2024-09-11T09:51:06.649 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T09:51:06.652 INFO:teuthology.orchestra.run.smithi017.stderr:nodeep-scrub is set 2024-09-11T09:51:06.678 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:51:06.678 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [], 2024-09-11T09:51:06.678 INFO:teuthology.orchestra.run.smithi017.stdout: "num_ops": 0 2024-09-11T09:51:06.678 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:51:06.688 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-11T09:51:06.738 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-09-11T09:51:06.738+0000 7f7b6c7d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31904) UID: 0 2024-09-11T09:51:06.773 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-11T09:51:06.774 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T09:51:06.791 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-11T09:51:06.791 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T09:51:06.808 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-11T09:51:06.808 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T09:51:06.825 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-11T09:51:06.825 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T09:51:06.840 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:51:06.840 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:51:06.841 INFO:tasks.ceph.osd.0.smithi017.stderr:2024-09-11T09:51:06.842+0000 7fb73c0f1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31665) UID: 0 2024-09-11T09:51:06.842 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-11T09:51:06.842 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T09:51:06.847 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-11T09:51:06.860 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-11T09:51:06.860 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T09:51:06.942 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:51:06.943+0000 7fdec5849640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31907) UID: 0 2024-09-11T09:51:06.967 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:51:06.967 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:51:06.976 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-11T09:51:07.043 INFO:tasks.ceph.osd.0.smithi017.stderr:2024-09-11T09:51:07.044+0000 7fb73c0f1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31665) UID: 0 2024-09-11T09:51:07.090 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:51:07.090 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:51:07.099 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-11T09:51:07.143 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:51:07.144+0000 7fdec5849640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31907) UID: 0 2024-09-11T09:51:07.202 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:51:07.202 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:51:07.211 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-11T09:51:07.244 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:51:07.245+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:51:07.345 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:51:07.346+0000 7fdec5849640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31907) UID: 0 2024-09-11T09:51:07.354 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:51:07.354 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:51:07.362 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-11T09:51:07.446 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:51:07.447+0000 7fdec5849640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31907) UID: 0 2024-09-11T09:51:07.505 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:51:07.506 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:51:07.514 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-11T09:51:07.555 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:51:07.556+0000 7fdec5849640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31907) UID: 0 2024-09-11T09:51:07.648 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-09-11T09:51:07.649+0000 7f7b6c7d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31904) UID: 0 2024-09-11T09:51:07.658 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:51:07.658 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:51:07.667 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-11T09:51:07.751 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:51:07.752+0000 7f3c96ec9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31909) UID: 0 2024-09-11T09:51:07.810 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:51:07.810 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:51:07.818 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-11T09:51:07.859 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-09-11T09:51:07.860+0000 7f7b6c7d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31904) UID: 0 2024-09-11T09:51:07.885 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 5, 0] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5] 2024-09-11T09:51:07.885 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-11T09:51:07.885 INFO:tasks.thrashosds.thrasher:inject_pause on osd.3 2024-09-11T09:51:07.886 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-11T09:51:07.886 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-11T09:51:07.886 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok config set filestore_inject_stall 3 2024-09-11T09:51:07.951 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:51:07.952+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:51:07.963 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:51:07.963 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:51:07.973 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-11T09:51:08.006 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-11T09:51:08.007 INFO:teuthology.orchestra.run.smithi096.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-11T09:51:08.007 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-11T09:51:08.056 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:51:08.057+0000 7fdec5849640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31907) UID: 0 2024-09-11T09:51:08.077 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:51:08.077 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:51:08.086 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-11T09:51:08.170 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:51:08.171+0000 7f3c96ec9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31909) UID: 0 2024-09-11T09:51:08.230 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:51:08.230 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:51:08.239 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-11T09:51:08.280 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:51:08.281+0000 7fdec5849640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31907) UID: 0 2024-09-11T09:51:08.355 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:51:08.356+0000 7f3c96ec9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31909) UID: 0 2024-09-11T09:51:08.360 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:51:08.360 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:51:08.369 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-11T09:51:08.456 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:51:08.457+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:51:08.477 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:51:08.477 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:51:08.486 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-11T09:51:08.570 INFO:tasks.ceph.osd.0.smithi017.stderr:2024-09-11T09:51:08.571+0000 7fb73c0f1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31665) UID: 0 2024-09-11T09:51:08.633 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:51:08.634 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:51:08.642 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-11T09:51:08.657 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-09-11T09:51:08.658+0000 7f7b6c7d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31904) UID: 0 2024-09-11T09:51:08.670 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-11T09:51:08.758 INFO:tasks.ceph.osd.2.smithi017.stderr:2024-09-11T09:51:08.759+0000 7fd20b170640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31672) UID: 0 2024-09-11T09:51:08.794 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:51:08.794 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:51:08.803 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-11T09:51:08.860 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:51:08.860+0000 7f3c96ec9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31909) UID: 0 2024-09-11T09:51:08.872 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T09:51:08.951 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:51:08.952 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:51:08.960 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-11T09:51:08.961 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:51:08.961+0000 7fdec5849640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31907) UID: 0 2024-09-11T09:51:09.013 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-11T09:51:09.013 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T09:51:09.030 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-11T09:51:09.030 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T09:51:09.047 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-11T09:51:09.047 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T09:51:09.061 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:51:09.062+0000 7f3c96ec9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31909) UID: 0 2024-09-11T09:51:09.064 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-11T09:51:09.064 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T09:51:09.070 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:51:09.071 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 20, 2024-09-11T09:51:09.071 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 600, 2024-09-11T09:51:09.071 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [ 2024-09-11T09:51:09.071 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:09.071 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4230.0:5 2.2 e17/15 2:5cea7035:::rbd_info:head v 17'1, mlcod=0'0)", 2024-09-11T09:51:09.071 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:19.494757+0000", 2024-09-11T09:51:09.071 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 49.577386842999999, 2024-09-11T09:51:09.071 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00082098399999999997, 2024-09-11T09:51:09.071 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:09.071 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:09.071 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:09.072 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:09.072 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:09.072 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:09.072 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:19.494757+0000", 2024-09-11T09:51:09.072 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:09.072 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:09.072 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:09.072 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:09.072 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:19.494757+0000", 2024-09-11T09:51:09.072 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:09.073 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:09.073 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:09.073 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:09.073 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:19.494757+0000", 2024-09-11T09:51:09.073 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.97e-07 2024-09-11T09:51:09.073 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:09.073 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:09.073 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:09.073 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:19.494763+0000", 2024-09-11T09:51:09.073 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.3709999999999996e-06 2024-09-11T09:51:09.073 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:09.073 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:09.073 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:09.074 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:19.494778+0000", 2024-09-11T09:51:09.074 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.4960000000000001e-05 2024-09-11T09:51:09.074 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:09.074 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:09.074 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:09.074 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:19.494801+0000", 2024-09-11T09:51:09.074 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.2388999999999999e-05 2024-09-11T09:51:09.074 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:09.074 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:09.074 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:09.074 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:19.494859+0000", 2024-09-11T09:51:09.074 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.7961999999999998e-05 2024-09-11T09:51:09.074 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:09.075 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:09.075 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:09.075 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:19.494883+0000", 2024-09-11T09:51:09.075 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.4488999999999999e-05 2024-09-11T09:51:09.075 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:09.075 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:09.075 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:09.075 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:19.495534+0000", 2024-09-11T09:51:09.075 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00065075500000000004 2024-09-11T09:51:09.075 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:09.075 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:09.075 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:09.075 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:19.495578+0000", 2024-09-11T09:51:09.076 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.3860999999999999e-05 2024-09-11T09:51:09.076 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:09.076 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:09.076 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:09.076 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:09.076 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:09.076 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4230.0:7 2.2 e18/15 2:5cea7035:::rbd_info:head v 18'2, mlcod=0'0)", 2024-09-11T09:51:09.076 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:20.500782+0000", 2024-09-11T09:51:09.076 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 48.571361963000001, 2024-09-11T09:51:09.076 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000790185, 2024-09-11T09:51:09.076 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:09.076 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:09.076 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:09.077 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:09.077 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:09.077 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:09.077 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:20.500782+0000", 2024-09-11T09:51:09.077 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:09.077 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:09.077 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:09.077 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:09.077 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:20.500782+0000", 2024-09-11T09:51:09.077 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:09.077 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:09.077 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:09.077 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:09.078 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:20.500782+0000", 2024-09-11T09:51:09.078 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.3299999999999998e-07 2024-09-11T09:51:09.078 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:09.078 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:09.078 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:09.078 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:20.500799+0000", 2024-09-11T09:51:09.078 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6830999999999999e-05 2024-09-11T09:51:09.078 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:09.078 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:09.078 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:09.078 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:20.500812+0000", 2024-09-11T09:51:09.078 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3013999999999999e-05 2024-09-11T09:51:09.078 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:09.078 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:09.079 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:09.079 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:20.500824+0000", 2024-09-11T09:51:09.079 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2593e-05 2024-09-11T09:51:09.079 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:09.079 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:09.079 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:09.079 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:20.500895+0000", 2024-09-11T09:51:09.079 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.1018000000000003e-05 2024-09-11T09:51:09.079 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:09.079 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:09.079 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:09.079 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:20.500912+0000", 2024-09-11T09:51:09.079 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6484e-05 2024-09-11T09:51:09.080 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:09.080 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:09.080 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:09.080 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:20.501531+0000", 2024-09-11T09:51:09.080 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00061903399999999997 2024-09-11T09:51:09.080 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:09.080 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:09.080 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:09.080 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:20.501572+0000", 2024-09-11T09:51:09.080 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.0877999999999997e-05 2024-09-11T09:51:09.080 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:09.080 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:09.080 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:09.081 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:09.081 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:09.081 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:51:09.082 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-11T09:51:09.082 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-11T09:51:09.083 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T09:51:09.101 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-11T09:51:09.101 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T09:51:09.162 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:51:09.163+0000 7fdec5849640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31907) UID: 0 2024-09-11T09:51:09.191 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:51:09.191 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [], 2024-09-11T09:51:09.191 INFO:teuthology.orchestra.run.smithi017.stdout: "num_ops": 0 2024-09-11T09:51:09.191 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:51:09.200 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-11T09:51:09.264 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:51:09.264+0000 7fdec5849640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31907) UID: 0 2024-09-11T09:51:09.307 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:51:09.307 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [], 2024-09-11T09:51:09.307 INFO:teuthology.orchestra.run.smithi017.stdout: "complaint_time": 30, 2024-09-11T09:51:09.307 INFO:teuthology.orchestra.run.smithi017.stdout: "num_blocked_ops": 0 2024-09-11T09:51:09.307 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:51:09.315 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-11T09:51:09.364 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:51:09.365+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:51:09.424 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:51:09.424 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 20, 2024-09-11T09:51:09.424 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 600, 2024-09-11T09:51:09.424 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [ 2024-09-11T09:51:09.424 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:09.424 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4230.0:1 2.3 e16/15 2:c4c92e5a:::rbd_trash:head v 16'1, mlcod=0'0)", 2024-09-11T09:51:09.424 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:18.494378+0000", 2024-09-11T09:51:09.425 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 50.931105455999997, 2024-09-11T09:51:09.425 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00089752199999999999, 2024-09-11T09:51:09.425 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:09.425 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:09.425 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:09.425 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:09.425 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:09.425 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:09.425 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:18.494378+0000", 2024-09-11T09:51:09.425 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:09.425 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:09.425 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:09.425 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:09.426 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:18.494378+0000", 2024-09-11T09:51:09.426 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:09.426 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:09.426 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:09.426 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:09.426 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:18.494378+0000", 2024-09-11T09:51:09.426 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.9099999999999999e-07 2024-09-11T09:51:09.426 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:09.426 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:09.426 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:09.426 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:18.494385+0000", 2024-09-11T09:51:09.426 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.8129999999999998e-06 2024-09-11T09:51:09.426 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:09.426 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:09.427 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:09.427 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:18.494416+0000", 2024-09-11T09:51:09.427 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.0654000000000002e-05 2024-09-11T09:51:09.427 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:09.427 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:09.427 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:09.427 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:18.494430+0000", 2024-09-11T09:51:09.427 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3754000000000001e-05 2024-09-11T09:51:09.427 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:09.427 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:09.427 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:09.427 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:18.494524+0000", 2024-09-11T09:51:09.427 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.4896999999999996e-05 2024-09-11T09:51:09.427 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:09.428 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:09.428 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:09.428 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:18.494558+0000", 2024-09-11T09:51:09.428 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.3900999999999999e-05 2024-09-11T09:51:09.428 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:09.428 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:09.428 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:09.428 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:18.495231+0000", 2024-09-11T09:51:09.428 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00067233500000000003 2024-09-11T09:51:09.428 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:09.428 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:09.428 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:09.428 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:18.495275+0000", 2024-09-11T09:51:09.428 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.4777000000000002e-05 2024-09-11T09:51:09.429 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:09.429 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:09.429 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:09.429 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:09.429 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:09.429 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:51:09.432 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-11T09:51:09.476 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:51:09.477+0000 7f3c96ec9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31909) UID: 0 2024-09-11T09:51:09.547 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-11T09:51:09.548 INFO:teuthology.orchestra.run.smithi096.stdout: "ops": [], 2024-09-11T09:51:09.548 INFO:teuthology.orchestra.run.smithi096.stdout: "num_ops": 0 2024-09-11T09:51:09.548 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-11T09:51:09.557 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-11T09:51:09.599 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-09-11T09:51:09.599+0000 7f7b6c7d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31904) UID: 0 2024-09-11T09:51:09.657 INFO:teuthology.orchestra.run.smithi017.stderr:noscrub is unset 2024-09-11T09:51:09.666 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-09-11T09:51:09.667+0000 7f7b6c7d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31904) UID: 0 2024-09-11T09:51:09.703 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-11T09:51:09.703 INFO:teuthology.orchestra.run.smithi096.stdout: "ops": [], 2024-09-11T09:51:09.703 INFO:teuthology.orchestra.run.smithi096.stdout: "complaint_time": 30, 2024-09-11T09:51:09.703 INFO:teuthology.orchestra.run.smithi096.stdout: "num_blocked_ops": 0 2024-09-11T09:51:09.703 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-11T09:51:09.712 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-11T09:51:09.767 INFO:tasks.ceph.osd.0.smithi017.stderr:2024-09-11T09:51:09.768+0000 7fb73c0f1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31665) UID: 0 2024-09-11T09:51:09.858 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-11T09:51:09.858 INFO:teuthology.orchestra.run.smithi096.stdout: "size": 20, 2024-09-11T09:51:09.858 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 600, 2024-09-11T09:51:09.858 INFO:teuthology.orchestra.run.smithi096.stdout: "ops": [ 2024-09-11T09:51:09.859 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.859 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:74 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:51:09.859 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.539067+0000", 2024-09-11T09:51:09.859 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 58.317548016000003, 2024-09-11T09:51:09.859 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.006945957, 2024-09-11T09:51:09.860 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:09.860 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:09.860 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:09.860 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:09.860 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:51:09.861 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:51:09.861 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 74 2024-09-11T09:51:09.861 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.861 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:09.861 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.862 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:09.862 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.539067+0000", 2024-09-11T09:51:09.862 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:09.862 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.862 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.863 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:09.863 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.539067+0000", 2024-09-11T09:51:09.863 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:09.863 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.863 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.863 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:09.864 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.539068+0000", 2024-09-11T09:51:09.864 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.6000000000000003e-07 2024-09-11T09:51:09.864 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.864 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.864 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:09.865 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.539072+0000", 2024-09-11T09:51:09.865 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.0450000000000002e-06 2024-09-11T09:51:09.865 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.865 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.865 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:09.866 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.539078+0000", 2024-09-11T09:51:09.866 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.6889999999999998e-06 2024-09-11T09:51:09.866 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.866 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.866 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:09.866 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.539083+0000", 2024-09-11T09:51:09.867 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.7180000000000004e-06 2024-09-11T09:51:09.867 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.867 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.867 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:09.867 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.540230+0000", 2024-09-11T09:51:09.868 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0011466250000000001 2024-09-11T09:51:09.868 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.868 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.868 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:09.868 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.540279+0000", 2024-09-11T09:51:09.869 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.9667999999999999e-05 2024-09-11T09:51:09.869 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.869 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.869 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:09.869 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.540357+0000", 2024-09-11T09:51:09.870 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.7863999999999999e-05 2024-09-11T09:51:09.870 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.870 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.870 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:09.870 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.541086+0000", 2024-09-11T09:51:09.870 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00072921100000000001 2024-09-11T09:51:09.871 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.871 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.871 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:09.871 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.545949+0000", 2024-09-11T09:51:09.871 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0048626040000000004 2024-09-11T09:51:09.872 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.872 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.872 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:09.872 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.545989+0000", 2024-09-11T09:51:09.872 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.0259000000000002e-05 2024-09-11T09:51:09.873 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.873 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.873 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:09.873 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.546013+0000", 2024-09-11T09:51:09.873 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.3614000000000001e-05 2024-09-11T09:51:09.873 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:09.874 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:09.874 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:09.874 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.874 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.874 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:108 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:51:09.875 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.711510+0000", 2024-09-11T09:51:09.875 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 58.145104844000002, 2024-09-11T09:51:09.875 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0054615649999999998, 2024-09-11T09:51:09.875 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:09.875 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:09.875 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:09.876 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:09.876 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:51:09.876 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:51:09.876 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 108 2024-09-11T09:51:09.876 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.877 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:09.877 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.877 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:09.877 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711510+0000", 2024-09-11T09:51:09.877 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:09.878 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.878 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.878 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:09.878 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711510+0000", 2024-09-11T09:51:09.878 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:09.878 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.879 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.879 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:09.879 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711511+0000", 2024-09-11T09:51:09.879 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.1999999999999998e-07 2024-09-11T09:51:09.879 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.880 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.880 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:09.880 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711516+0000", 2024-09-11T09:51:09.880 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.4140000000000002e-06 2024-09-11T09:51:09.880 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.880 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.881 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:09.881 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711527+0000", 2024-09-11T09:51:09.881 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.0169e-05 2024-09-11T09:51:09.881 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.881 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.882 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:09.882 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711535+0000", 2024-09-11T09:51:09.882 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.3299999999999999e-06 2024-09-11T09:51:09.882 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.882 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.883 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:09.883 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.713763+0000", 2024-09-11T09:51:09.883 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0022281079999999999 2024-09-11T09:51:09.883 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.883 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.883 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:09.884 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.713812+0000", 2024-09-11T09:51:09.884 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.8544000000000001e-05 2024-09-11T09:51:09.884 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.884 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.884 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:09.885 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.713909+0000", 2024-09-11T09:51:09.885 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.7113000000000003e-05 2024-09-11T09:51:09.885 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.885 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.885 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:09.885 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.716873+0000", 2024-09-11T09:51:09.885 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0029642710000000001 2024-09-11T09:51:09.886 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.886 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.886 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:09.886 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.716918+0000", 2024-09-11T09:51:09.886 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.5447e-05 2024-09-11T09:51:09.886 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.886 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.886 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:09.886 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.716947+0000", 2024-09-11T09:51:09.886 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.8181e-05 2024-09-11T09:51:09.887 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.887 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.887 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:09.887 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.716972+0000", 2024-09-11T09:51:09.887 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.5068000000000001e-05 2024-09-11T09:51:09.887 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:09.887 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:09.887 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:09.887 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.887 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.887 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:109 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:51:09.887 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.711555+0000", 2024-09-11T09:51:09.887 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 58.145059594999999, 2024-09-11T09:51:09.888 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.005479902, 2024-09-11T09:51:09.888 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:09.888 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:09.888 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:09.888 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:09.888 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:51:09.888 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:51:09.888 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 109 2024-09-11T09:51:09.888 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.888 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:09.888 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.888 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:09.888 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711555+0000", 2024-09-11T09:51:09.889 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:09.889 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.889 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.889 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:09.889 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711555+0000", 2024-09-11T09:51:09.889 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:09.889 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.889 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.889 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:09.889 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711556+0000", 2024-09-11T09:51:09.889 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.3200000000000005e-07 2024-09-11T09:51:09.889 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.889 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.890 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:09.890 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711601+0000", 2024-09-11T09:51:09.890 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.5349000000000002e-05 2024-09-11T09:51:09.890 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.890 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.890 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:09.890 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711612+0000", 2024-09-11T09:51:09.890 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.0701e-05 2024-09-11T09:51:09.890 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.890 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.890 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:09.890 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711619+0000", 2024-09-11T09:51:09.890 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.7610000000000002e-06 2024-09-11T09:51:09.891 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.891 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.891 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:09.891 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.714371+0000", 2024-09-11T09:51:09.891 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0027526260000000002 2024-09-11T09:51:09.891 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.891 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.891 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:09.891 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.714417+0000", 2024-09-11T09:51:09.891 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.5893999999999998e-05 2024-09-11T09:51:09.891 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.891 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.892 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:09.892 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.714496+0000", 2024-09-11T09:51:09.892 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.8539000000000001e-05 2024-09-11T09:51:09.892 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.892 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.892 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:09.892 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.716910+0000", 2024-09-11T09:51:09.892 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0024137640000000001 2024-09-11T09:51:09.892 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.892 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.892 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:09.892 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.716989+0000", 2024-09-11T09:51:09.892 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.9198999999999998e-05 2024-09-11T09:51:09.893 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.893 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.893 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:09.893 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.717016+0000", 2024-09-11T09:51:09.893 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.6699e-05 2024-09-11T09:51:09.893 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.893 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.893 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:09.893 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.717035+0000", 2024-09-11T09:51:09.893 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.9738e-05 2024-09-11T09:51:09.893 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:09.893 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:09.893 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:09.893 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.894 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.894 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:110 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:51:09.894 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.711636+0000", 2024-09-11T09:51:09.894 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 58.144979135, 2024-09-11T09:51:09.894 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0067613070000000003, 2024-09-11T09:51:09.894 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:09.894 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:09.894 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:09.894 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:09.894 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:51:09.894 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:51:09.894 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 110 2024-09-11T09:51:09.894 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.894 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:09.895 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.895 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:09.895 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711636+0000", 2024-09-11T09:51:09.895 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:09.895 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.895 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.895 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:09.895 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711636+0000", 2024-09-11T09:51:09.895 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:09.895 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.895 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.895 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:09.895 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711637+0000", 2024-09-11T09:51:09.896 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.9999999999999997e-07 2024-09-11T09:51:09.896 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.896 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.896 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:09.896 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711641+0000", 2024-09-11T09:51:09.896 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.2570000000000001e-06 2024-09-11T09:51:09.896 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.896 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.896 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:09.896 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711651+0000", 2024-09-11T09:51:09.896 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.0285999999999999e-05 2024-09-11T09:51:09.896 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.896 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.897 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:09.897 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711658+0000", 2024-09-11T09:51:09.897 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.3540000000000002e-06 2024-09-11T09:51:09.897 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.897 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.897 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:09.897 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.716475+0000", 2024-09-11T09:51:09.897 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0048162819999999999 2024-09-11T09:51:09.897 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.897 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.898 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:09.898 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.716526+0000", 2024-09-11T09:51:09.898 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.0905000000000001e-05 2024-09-11T09:51:09.898 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.898 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.898 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:09.898 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.716601+0000", 2024-09-11T09:51:09.898 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.5283000000000004e-05 2024-09-11T09:51:09.898 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.898 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.898 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:09.898 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.717933+0000", 2024-09-11T09:51:09.898 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001331622 2024-09-11T09:51:09.899 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.899 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.899 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:09.899 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718332+0000", 2024-09-11T09:51:09.899 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00039909899999999998 2024-09-11T09:51:09.899 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.899 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.899 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:09.899 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718363+0000", 2024-09-11T09:51:09.899 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.1636e-05 2024-09-11T09:51:09.899 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.899 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.899 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:09.900 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718397+0000", 2024-09-11T09:51:09.900 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.3883000000000001e-05 2024-09-11T09:51:09.900 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:09.900 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:09.900 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:09.900 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.900 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.900 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:111 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:51:09.900 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.711668+0000", 2024-09-11T09:51:09.900 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 58.144946705000002, 2024-09-11T09:51:09.900 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0077145570000000004, 2024-09-11T09:51:09.900 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:09.900 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:09.900 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:09.901 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:09.901 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:51:09.901 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:51:09.901 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 111 2024-09-11T09:51:09.901 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.901 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:09.901 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.901 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:09.901 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711668+0000", 2024-09-11T09:51:09.901 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:09.901 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.901 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.901 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:09.902 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711668+0000", 2024-09-11T09:51:09.902 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:09.902 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.902 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.902 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:09.902 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711669+0000", 2024-09-11T09:51:09.902 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.99e-07 2024-09-11T09:51:09.902 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.902 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.902 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:09.902 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711673+0000", 2024-09-11T09:51:09.902 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.7440000000000001e-06 2024-09-11T09:51:09.902 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.902 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.903 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:09.903 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711686+0000", 2024-09-11T09:51:09.903 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.3317e-05 2024-09-11T09:51:09.903 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.903 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.903 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:09.903 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711697+0000", 2024-09-11T09:51:09.903 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1022000000000001e-05 2024-09-11T09:51:09.903 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.903 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.903 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:09.903 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.717065+0000", 2024-09-11T09:51:09.903 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.005367531 2024-09-11T09:51:09.903 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.904 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.904 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:09.904 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.717114+0000", 2024-09-11T09:51:09.904 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.9707999999999999e-05 2024-09-11T09:51:09.904 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.904 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.904 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:09.904 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.717190+0000", 2024-09-11T09:51:09.904 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.5394000000000004e-05 2024-09-11T09:51:09.904 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.904 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.904 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:09.904 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718793+0000", 2024-09-11T09:51:09.905 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001603653 2024-09-11T09:51:09.905 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.905 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.905 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:09.905 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719329+0000", 2024-09-11T09:51:09.905 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00053543799999999995 2024-09-11T09:51:09.905 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.905 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.905 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:09.905 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719359+0000", 2024-09-11T09:51:09.905 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.0420999999999999e-05 2024-09-11T09:51:09.905 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.905 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.905 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:09.906 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719383+0000", 2024-09-11T09:51:09.906 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.3730000000000001e-05 2024-09-11T09:51:09.906 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:09.906 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:09.906 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:09.906 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.906 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.906 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:112 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:51:09.906 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.711712+0000", 2024-09-11T09:51:09.906 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 58.144903145000001, 2024-09-11T09:51:09.906 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.008162084, 2024-09-11T09:51:09.906 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:09.906 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:09.907 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:09.907 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:09.907 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:51:09.907 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:51:09.907 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 112 2024-09-11T09:51:09.907 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.907 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:09.907 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.907 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:09.907 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711712+0000", 2024-09-11T09:51:09.907 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:09.907 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.907 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.907 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:09.908 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711712+0000", 2024-09-11T09:51:09.908 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:09.908 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.908 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.908 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:09.908 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711712+0000", 2024-09-11T09:51:09.908 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.6000000000000004e-07 2024-09-11T09:51:09.908 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.908 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.908 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:09.909 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711808+0000", 2024-09-11T09:51:09.909 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.5977999999999995e-05 2024-09-11T09:51:09.909 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.909 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.909 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:09.909 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711819+0000", 2024-09-11T09:51:09.909 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.0755e-05 2024-09-11T09:51:09.909 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.910 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.910 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:09.910 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711826+0000", 2024-09-11T09:51:09.910 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.7970000000000003e-06 2024-09-11T09:51:09.910 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.910 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.910 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:09.910 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.717435+0000", 2024-09-11T09:51:09.910 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0056090239999999998 2024-09-11T09:51:09.910 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.910 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.910 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:09.911 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.717496+0000", 2024-09-11T09:51:09.911 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.0543e-05 2024-09-11T09:51:09.911 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.911 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.911 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:09.911 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.717591+0000", 2024-09-11T09:51:09.911 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.5667000000000004e-05 2024-09-11T09:51:09.911 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.911 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.911 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:09.911 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718831+0000", 2024-09-11T09:51:09.911 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001240204 2024-09-11T09:51:09.911 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.912 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.912 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:09.912 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719819+0000", 2024-09-11T09:51:09.912 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00098751500000000001 2024-09-11T09:51:09.912 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.912 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.912 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:09.912 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719850+0000", 2024-09-11T09:51:09.912 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.1535999999999998e-05 2024-09-11T09:51:09.912 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.912 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.912 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:09.913 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719874+0000", 2024-09-11T09:51:09.913 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.3504999999999999e-05 2024-09-11T09:51:09.913 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:09.913 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:09.913 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:09.913 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.913 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.913 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:113 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:51:09.913 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.711842+0000", 2024-09-11T09:51:09.913 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 58.144772646, 2024-09-11T09:51:09.913 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0081160099999999999, 2024-09-11T09:51:09.913 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:09.914 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:09.914 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:09.914 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:09.914 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:51:09.914 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:51:09.916 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 113 2024-09-11T09:51:09.916 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.916 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:09.916 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.917 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:09.917 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711842+0000", 2024-09-11T09:51:09.917 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:09.917 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.917 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.917 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:09.918 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711842+0000", 2024-09-11T09:51:09.918 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:09.918 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.918 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.918 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:09.918 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711843+0000", 2024-09-11T09:51:09.918 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.4399999999999999e-07 2024-09-11T09:51:09.919 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.919 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.919 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:09.919 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711848+0000", 2024-09-11T09:51:09.919 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.4159999999999997e-06 2024-09-11T09:51:09.919 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.919 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.919 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:09.919 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711856+0000", 2024-09-11T09:51:09.919 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.7870000000000007e-06 2024-09-11T09:51:09.919 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.919 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.920 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:09.920 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711862+0000", 2024-09-11T09:51:09.920 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.3689999999999998e-06 2024-09-11T09:51:09.920 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.920 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.920 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:09.920 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.717965+0000", 2024-09-11T09:51:09.920 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0061028899999999997 2024-09-11T09:51:09.920 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.920 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.920 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:09.920 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718023+0000", 2024-09-11T09:51:09.920 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.8313000000000002e-05 2024-09-11T09:51:09.920 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.921 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.921 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:09.921 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718109+0000", 2024-09-11T09:51:09.921 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.6205000000000006e-05 2024-09-11T09:51:09.921 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.921 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.921 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:09.921 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719808+0000", 2024-09-11T09:51:09.921 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0016990060000000001 2024-09-11T09:51:09.921 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.921 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.921 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:09.922 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719883+0000", 2024-09-11T09:51:09.922 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.4624999999999998e-05 2024-09-11T09:51:09.922 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.922 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.922 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:09.922 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719910+0000", 2024-09-11T09:51:09.922 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.7512e-05 2024-09-11T09:51:09.922 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.922 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.922 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:09.922 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719958+0000", 2024-09-11T09:51:09.922 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.8143000000000003e-05 2024-09-11T09:51:09.922 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:09.923 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:09.923 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:09.923 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.923 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.923 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:51:09.923 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.711887+0000", 2024-09-11T09:51:09.923 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 58.144728032000003, 2024-09-11T09:51:09.923 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0086063589999999992, 2024-09-11T09:51:09.923 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:09.923 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:09.923 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:09.923 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:09.923 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:51:09.923 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:51:09.924 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 114 2024-09-11T09:51:09.924 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.924 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:09.924 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.924 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:09.924 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711887+0000", 2024-09-11T09:51:09.924 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:09.924 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.924 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.924 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:09.924 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711887+0000", 2024-09-11T09:51:09.924 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:09.924 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.925 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.925 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:09.925 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711887+0000", 2024-09-11T09:51:09.925 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.1500000000000005e-07 2024-09-11T09:51:09.925 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.925 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.925 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:09.925 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711897+0000", 2024-09-11T09:51:09.925 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.1900000000000001e-06 2024-09-11T09:51:09.925 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.925 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.925 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:09.925 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711902+0000", 2024-09-11T09:51:09.925 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.2349999999999999e-06 2024-09-11T09:51:09.925 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.926 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.926 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:09.926 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711906+0000", 2024-09-11T09:51:09.926 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.7129999999999999e-06 2024-09-11T09:51:09.926 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.926 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.926 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:09.926 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718429+0000", 2024-09-11T09:51:09.926 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0065231960000000002 2024-09-11T09:51:09.926 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.926 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.927 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:09.927 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718480+0000", 2024-09-11T09:51:09.927 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.1161e-05 2024-09-11T09:51:09.927 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.927 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.927 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:09.927 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718558+0000", 2024-09-11T09:51:09.927 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.7519000000000001e-05 2024-09-11T09:51:09.927 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.927 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.927 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:09.927 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720038+0000", 2024-09-11T09:51:09.928 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001480016 2024-09-11T09:51:09.928 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.928 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.928 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:09.928 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720417+0000", 2024-09-11T09:51:09.928 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00037943699999999999 2024-09-11T09:51:09.928 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.929 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.929 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:09.929 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720452+0000", 2024-09-11T09:51:09.929 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.4595000000000003e-05 2024-09-11T09:51:09.929 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.929 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.930 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:09.930 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720493+0000", 2024-09-11T09:51:09.930 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.1782000000000001e-05 2024-09-11T09:51:09.930 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:09.930 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:09.930 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:09.931 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.931 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.931 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:51:09.931 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.711912+0000", 2024-09-11T09:51:09.931 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 58.144703362999998, 2024-09-11T09:51:09.931 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0086864750000000008, 2024-09-11T09:51:09.932 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:09.932 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:09.932 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:09.932 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:09.932 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:51:09.932 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:51:09.933 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 115 2024-09-11T09:51:09.933 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.933 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:09.933 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.933 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:09.934 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711912+0000", 2024-09-11T09:51:09.934 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:09.934 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.934 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.934 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:09.934 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711912+0000", 2024-09-11T09:51:09.935 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:09.935 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.935 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.935 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:09.935 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711912+0000", 2024-09-11T09:51:09.935 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.3900000000000001e-07 2024-09-11T09:51:09.936 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.936 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.936 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:09.936 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711969+0000", 2024-09-11T09:51:09.936 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.7501999999999999e-05 2024-09-11T09:51:09.936 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.937 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.937 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:09.937 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711986+0000", 2024-09-11T09:51:09.937 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.6402999999999999e-05 2024-09-11T09:51:09.937 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.937 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.938 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:09.938 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711990+0000", 2024-09-11T09:51:09.938 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.929e-06 2024-09-11T09:51:09.938 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.938 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.939 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:09.939 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718861+0000", 2024-09-11T09:51:09.939 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0068710530000000002 2024-09-11T09:51:09.939 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.939 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.939 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:09.940 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718908+0000", 2024-09-11T09:51:09.940 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.7431e-05 2024-09-11T09:51:09.940 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.940 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.940 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:09.941 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718984+0000", 2024-09-11T09:51:09.941 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.5500999999999994e-05 2024-09-11T09:51:09.941 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.941 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.941 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:09.941 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720500+0000", 2024-09-11T09:51:09.941 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0015158820000000001 2024-09-11T09:51:09.942 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.942 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.942 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:09.942 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720536+0000", 2024-09-11T09:51:09.942 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.6118000000000002e-05 2024-09-11T09:51:09.942 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.943 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.943 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:09.943 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720570+0000", 2024-09-11T09:51:09.943 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.4387999999999997e-05 2024-09-11T09:51:09.943 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.944 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.944 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:09.944 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720598+0000", 2024-09-11T09:51:09.944 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.8028999999999999e-05 2024-09-11T09:51:09.944 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:09.944 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:09.945 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:09.945 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.945 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.945 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:51:09.945 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.711998+0000", 2024-09-11T09:51:09.945 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 58.144617183999998, 2024-09-11T09:51:09.946 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.010785813999999999, 2024-09-11T09:51:09.946 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:09.946 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:09.946 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:09.946 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:09.947 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:51:09.947 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:51:09.947 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 116 2024-09-11T09:51:09.947 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.947 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:09.947 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.948 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:09.948 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711998+0000", 2024-09-11T09:51:09.948 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:09.948 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.948 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.949 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:09.949 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711998+0000", 2024-09-11T09:51:09.949 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:09.949 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.949 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.949 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:09.950 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711998+0000", 2024-09-11T09:51:09.950 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.3300000000000003e-07 2024-09-11T09:51:09.950 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.950 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.950 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:09.950 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712001+0000", 2024-09-11T09:51:09.951 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.9349999999999999e-06 2024-09-11T09:51:09.951 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.951 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.951 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:09.951 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712006+0000", 2024-09-11T09:51:09.952 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.1630000000000004e-06 2024-09-11T09:51:09.952 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.952 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.952 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:09.952 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712010+0000", 2024-09-11T09:51:09.952 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.574e-06 2024-09-11T09:51:09.953 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.953 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.953 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:09.953 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719415+0000", 2024-09-11T09:51:09.953 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0074049099999999998 2024-09-11T09:51:09.953 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.953 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.954 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:09.954 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719465+0000", 2024-09-11T09:51:09.954 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.0129000000000001e-05 2024-09-11T09:51:09.954 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.954 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.954 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:09.955 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719545+0000", 2024-09-11T09:51:09.955 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.0253999999999994e-05 2024-09-11T09:51:09.955 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.955 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.955 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:09.955 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720636+0000", 2024-09-11T09:51:09.955 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0010909330000000001 2024-09-11T09:51:09.955 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.956 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.956 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:09.956 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.722707+0000", 2024-09-11T09:51:09.956 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0020713300000000001 2024-09-11T09:51:09.956 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.956 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.956 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:09.956 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.722743+0000", 2024-09-11T09:51:09.956 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.5351000000000001e-05 2024-09-11T09:51:09.956 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.956 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.956 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:09.956 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.722784+0000", 2024-09-11T09:51:09.957 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.0802e-05 2024-09-11T09:51:09.957 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:09.957 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:09.957 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:09.957 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.957 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.957 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:51:09.957 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.712014+0000", 2024-09-11T09:51:09.957 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 58.144601143999999, 2024-09-11T09:51:09.957 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.013420938, 2024-09-11T09:51:09.957 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:09.957 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:09.958 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:09.958 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:09.958 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:51:09.958 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:51:09.958 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 117 2024-09-11T09:51:09.958 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.958 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:09.958 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.958 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:09.958 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712014+0000", 2024-09-11T09:51:09.958 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:09.958 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.958 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.958 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:09.959 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712014+0000", 2024-09-11T09:51:09.959 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:09.959 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.959 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.959 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:09.959 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712014+0000", 2024-09-11T09:51:09.959 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.2000000000000001e-07 2024-09-11T09:51:09.959 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.959 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.959 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:09.959 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712016+0000", 2024-09-11T09:51:09.959 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.2309999999999999e-06 2024-09-11T09:51:09.959 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.959 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.960 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:09.960 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712020+0000", 2024-09-11T09:51:09.960 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.3830000000000001e-06 2024-09-11T09:51:09.960 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.960 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.960 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:09.960 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712023+0000", 2024-09-11T09:51:09.960 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.3340000000000002e-06 2024-09-11T09:51:09.960 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.960 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.960 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:09.960 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720083+0000", 2024-09-11T09:51:09.961 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0080601189999999993 2024-09-11T09:51:09.961 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.961 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.961 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:09.961 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720125+0000", 2024-09-11T09:51:09.961 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.2029e-05 2024-09-11T09:51:09.961 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.961 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.961 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:09.961 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720205+0000", 2024-09-11T09:51:09.961 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.0010000000000001e-05 2024-09-11T09:51:09.961 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.961 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.962 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:09.962 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720849+0000", 2024-09-11T09:51:09.962 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00064369899999999996 2024-09-11T09:51:09.962 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.962 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.962 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:09.962 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.725357+0000", 2024-09-11T09:51:09.962 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0045077379999999998 2024-09-11T09:51:09.962 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.962 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.962 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:09.962 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.725392+0000", 2024-09-11T09:51:09.962 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.5862999999999999e-05 2024-09-11T09:51:09.963 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.963 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.963 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:09.963 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.725435+0000", 2024-09-11T09:51:09.963 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.2311999999999997e-05 2024-09-11T09:51:09.963 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:09.963 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:09.963 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:09.963 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.963 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.963 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:119 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 131072~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:51:09.963 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.725673+0000", 2024-09-11T09:51:09.963 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 58.130942441999998, 2024-09-11T09:51:09.963 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0053927460000000003, 2024-09-11T09:51:09.964 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:09.964 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:09.964 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:09.964 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:09.964 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:51:09.964 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:51:09.964 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 119 2024-09-11T09:51:09.964 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.964 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:09.964 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.964 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:09.964 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.725673+0000", 2024-09-11T09:51:09.965 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:09.965 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.965 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.965 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:09.965 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.725673+0000", 2024-09-11T09:51:09.965 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:09.965 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.965 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.965 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:09.965 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.725673+0000", 2024-09-11T09:51:09.965 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.68e-07 2024-09-11T09:51:09.965 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.965 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.966 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:09.966 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.725720+0000", 2024-09-11T09:51:09.966 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.6526999999999997e-05 2024-09-11T09:51:09.966 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.966 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.966 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:09.966 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.725734+0000", 2024-09-11T09:51:09.966 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.4593000000000001e-05 2024-09-11T09:51:09.966 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.966 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.966 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:09.966 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.725741+0000", 2024-09-11T09:51:09.966 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.28e-06 2024-09-11T09:51:09.967 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.967 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.967 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:09.967 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.726290+0000", 2024-09-11T09:51:09.967 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00054892000000000005 2024-09-11T09:51:09.967 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.967 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.967 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:09.967 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.726338+0000", 2024-09-11T09:51:09.967 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.7982000000000002e-05 2024-09-11T09:51:09.967 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.967 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.967 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:09.968 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.726413+0000", 2024-09-11T09:51:09.968 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.5679000000000005e-05 2024-09-11T09:51:09.968 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.968 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.968 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:09.968 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.727555+0000", 2024-09-11T09:51:09.968 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0011420289999999999 2024-09-11T09:51:09.968 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.968 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.968 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:09.968 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.731004+0000", 2024-09-11T09:51:09.968 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0034484279999999999 2024-09-11T09:51:09.968 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.969 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.969 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:09.969 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.731040+0000", 2024-09-11T09:51:09.969 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.6315e-05 2024-09-11T09:51:09.969 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.969 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.969 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:09.969 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.731065+0000", 2024-09-11T09:51:09.969 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.5324999999999999e-05 2024-09-11T09:51:09.969 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:09.969 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:09.969 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:09.969 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.970 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.970 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:120 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 196608~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:51:09.970 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.725756+0000", 2024-09-11T09:51:09.970 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 58.130858533000001, 2024-09-11T09:51:09.970 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0053569100000000003, 2024-09-11T09:51:09.970 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:09.970 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:09.970 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:09.970 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:09.970 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:51:09.970 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:51:09.970 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 120 2024-09-11T09:51:09.970 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.971 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:09.971 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.971 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:09.971 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.725756+0000", 2024-09-11T09:51:09.971 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:09.971 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.971 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.971 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:09.971 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.725756+0000", 2024-09-11T09:51:09.971 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:09.971 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.971 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.971 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:09.971 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.725757+0000", 2024-09-11T09:51:09.972 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.3799999999999997e-07 2024-09-11T09:51:09.972 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.972 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.972 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:09.972 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.725802+0000", 2024-09-11T09:51:09.972 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.4790000000000003e-05 2024-09-11T09:51:09.972 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.972 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.972 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:09.972 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.725815+0000", 2024-09-11T09:51:09.973 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.2609000000000001e-05 2024-09-11T09:51:09.973 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.973 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.973 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:09.973 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.725822+0000", 2024-09-11T09:51:09.973 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.6450000000000005e-06 2024-09-11T09:51:09.973 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.973 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.974 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:09.974 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.726708+0000", 2024-09-11T09:51:09.974 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00088606500000000001 2024-09-11T09:51:09.974 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.974 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.974 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:09.974 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.726755+0000", 2024-09-11T09:51:09.974 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.6888000000000002e-05 2024-09-11T09:51:09.974 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.974 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.974 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:09.974 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.726828+0000", 2024-09-11T09:51:09.975 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.2510999999999997e-05 2024-09-11T09:51:09.975 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.975 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.975 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:09.975 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.727829+0000", 2024-09-11T09:51:09.975 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001001093 2024-09-11T09:51:09.975 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.975 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.975 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:09.975 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.731071+0000", 2024-09-11T09:51:09.975 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0032426009999999999 2024-09-11T09:51:09.975 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.975 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.976 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:09.976 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.731094+0000", 2024-09-11T09:51:09.976 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.2677e-05 2024-09-11T09:51:09.976 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.976 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.976 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:09.976 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.731113+0000", 2024-09-11T09:51:09.976 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.9392999999999999e-05 2024-09-11T09:51:09.976 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:09.976 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:09.976 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:09.976 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.976 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.976 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:138 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:51:09.977 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:12.034038+0000", 2024-09-11T09:51:09.977 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 57.822577203000002, 2024-09-11T09:51:09.977 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0054142749999999996, 2024-09-11T09:51:09.977 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:09.977 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:09.977 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:09.977 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:09.977 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:51:09.977 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:51:09.978 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 138 2024-09-11T09:51:09.978 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.978 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:09.978 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.978 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:09.978 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034038+0000", 2024-09-11T09:51:09.978 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:09.978 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.978 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.978 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:09.978 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034038+0000", 2024-09-11T09:51:09.978 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:09.979 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.979 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.979 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:09.979 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034038+0000", 2024-09-11T09:51:09.979 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.3499999999999996e-07 2024-09-11T09:51:09.979 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.979 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.979 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:09.979 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034043+0000", 2024-09-11T09:51:09.979 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.7310000000000001e-06 2024-09-11T09:51:09.979 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.979 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.980 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:09.980 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034050+0000", 2024-09-11T09:51:09.980 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.0419999999999997e-06 2024-09-11T09:51:09.980 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.980 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.980 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:09.980 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034055+0000", 2024-09-11T09:51:09.980 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.3299999999999998e-06 2024-09-11T09:51:09.980 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.980 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.980 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:09.980 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.037309+0000", 2024-09-11T09:51:09.980 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0032535440000000001 2024-09-11T09:51:09.981 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.981 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.981 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:09.981 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.037357+0000", 2024-09-11T09:51:09.981 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.8167999999999997e-05 2024-09-11T09:51:09.981 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.981 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.981 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:09.981 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.037430+0000", 2024-09-11T09:51:09.981 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.3275999999999994e-05 2024-09-11T09:51:09.981 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.981 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.981 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:09.981 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.039014+0000", 2024-09-11T09:51:09.982 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0015831350000000001 2024-09-11T09:51:09.982 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.982 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.982 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:09.982 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.039401+0000", 2024-09-11T09:51:09.982 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00038758500000000002 2024-09-11T09:51:09.982 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.982 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.982 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:09.982 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.039429+0000", 2024-09-11T09:51:09.982 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.7379e-05 2024-09-11T09:51:09.982 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.983 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.983 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:09.983 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.039452+0000", 2024-09-11T09:51:09.983 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.355e-05 2024-09-11T09:51:09.983 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:09.983 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:09.983 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:09.983 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.983 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.983 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:139 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:51:09.983 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:12.034064+0000", 2024-09-11T09:51:09.983 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 57.822551171999997, 2024-09-11T09:51:09.983 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0054736200000000002, 2024-09-11T09:51:09.984 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:09.984 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:09.984 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:09.984 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:09.984 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:51:09.984 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:51:09.984 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 139 2024-09-11T09:51:09.984 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.984 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:09.984 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.984 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:09.984 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034064+0000", 2024-09-11T09:51:09.985 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:09.985 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.985 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.985 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:09.985 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034064+0000", 2024-09-11T09:51:09.985 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:09.985 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.985 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.985 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:09.985 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034064+0000", 2024-09-11T09:51:09.985 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.89e-07 2024-09-11T09:51:09.985 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.986 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.986 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:09.986 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034069+0000", 2024-09-11T09:51:09.986 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.9799999999999998e-06 2024-09-11T09:51:09.986 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.986 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.986 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:09.986 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034079+0000", 2024-09-11T09:51:09.986 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.3549999999999994e-06 2024-09-11T09:51:09.987 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.987 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.987 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:09.987 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034084+0000", 2024-09-11T09:51:09.987 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.2209999999999996e-06 2024-09-11T09:51:09.987 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.987 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.987 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:09.987 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.038176+0000", 2024-09-11T09:51:09.987 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0040921610000000004 2024-09-11T09:51:09.987 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.987 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.988 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:09.988 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.038221+0000", 2024-09-11T09:51:09.988 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.5006000000000002e-05 2024-09-11T09:51:09.988 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.988 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.988 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:09.988 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.038293+0000", 2024-09-11T09:51:09.988 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.1952999999999994e-05 2024-09-11T09:51:09.988 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.988 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.988 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:09.988 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.039461+0000", 2024-09-11T09:51:09.989 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0011680340000000001 2024-09-11T09:51:09.989 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.989 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.989 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:09.989 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.039487+0000", 2024-09-11T09:51:09.989 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.5752e-05 2024-09-11T09:51:09.989 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.989 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.990 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:09.990 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.039515+0000", 2024-09-11T09:51:09.990 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.7763000000000002e-05 2024-09-11T09:51:09.990 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.990 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.990 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:09.990 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.039537+0000", 2024-09-11T09:51:09.990 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.2906000000000001e-05 2024-09-11T09:51:09.990 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:09.990 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:09.990 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:09.991 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.991 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.991 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:140 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:51:09.991 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:12.034093+0000", 2024-09-11T09:51:09.991 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 57.822521668, 2024-09-11T09:51:09.991 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0059112490000000004, 2024-09-11T09:51:09.991 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:09.991 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:09.991 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:09.991 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:09.991 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:51:09.992 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:51:09.992 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 140 2024-09-11T09:51:09.992 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.992 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:09.992 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.992 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:09.992 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034093+0000", 2024-09-11T09:51:09.992 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:09.992 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.992 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.992 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:09.992 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034093+0000", 2024-09-11T09:51:09.993 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:09.993 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.993 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.993 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:09.993 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034094+0000", 2024-09-11T09:51:09.993 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.4299999999999998e-07 2024-09-11T09:51:09.993 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.993 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.993 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:09.993 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034159+0000", 2024-09-11T09:51:09.993 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.567e-05 2024-09-11T09:51:09.993 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.993 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.994 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:09.994 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034167+0000", 2024-09-11T09:51:09.994 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.8490000000000006e-06 2024-09-11T09:51:09.994 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.994 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.994 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:09.994 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034173+0000", 2024-09-11T09:51:09.994 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.1770000000000003e-06 2024-09-11T09:51:09.994 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.994 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.994 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:09.995 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.038543+0000", 2024-09-11T09:51:09.995 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0043699150000000003 2024-09-11T09:51:09.995 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.995 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.995 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:09.995 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.038589+0000", 2024-09-11T09:51:09.995 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.5562999999999997e-05 2024-09-11T09:51:09.995 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.995 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.995 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:09.995 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.038662+0000", 2024-09-11T09:51:09.995 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.2748000000000002e-05 2024-09-11T09:51:09.995 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.995 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.996 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:09.996 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.039635+0000", 2024-09-11T09:51:09.996 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00097310799999999998 2024-09-11T09:51:09.996 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.996 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.996 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:09.996 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.039945+0000", 2024-09-11T09:51:09.996 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00031054100000000001 2024-09-11T09:51:09.996 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.996 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.996 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:09.996 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.039971+0000", 2024-09-11T09:51:09.997 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.5423999999999999e-05 2024-09-11T09:51:09.997 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.997 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.997 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:09.997 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.040005+0000", 2024-09-11T09:51:09.997 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.3810999999999999e-05 2024-09-11T09:51:09.997 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:09.997 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:09.997 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:09.997 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.998 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.998 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:51:09.998 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:12.034184+0000", 2024-09-11T09:51:09.998 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 57.822430969999999, 2024-09-11T09:51:09.998 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0061520380000000003, 2024-09-11T09:51:09.998 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:09.998 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:09.998 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:09.998 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:09.998 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:51:09.998 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:51:09.999 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 141 2024-09-11T09:51:09.999 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.999 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:09.999 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.999 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:09.999 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034184+0000", 2024-09-11T09:51:09.999 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:09.999 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:09.999 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:09.999 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:10.000 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034184+0000", 2024-09-11T09:51:10.000 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:10.000 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:10.000 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:10.000 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:10.000 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034185+0000", 2024-09-11T09:51:10.000 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.1099999999999995e-07 2024-09-11T09:51:10.000 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:10.000 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:10.000 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:10.000 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034190+0000", 2024-09-11T09:51:10.000 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.7309999999999998e-06 2024-09-11T09:51:10.000 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:10.001 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:10.001 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:10.001 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034197+0000", 2024-09-11T09:51:10.001 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.7440000000000001e-06 2024-09-11T09:51:10.001 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:10.001 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:10.001 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:10.001 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034204+0000", 2024-09-11T09:51:10.001 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.8970000000000002e-06 2024-09-11T09:51:10.001 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:10.001 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:10.001 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:10.002 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.039035+0000", 2024-09-11T09:51:10.002 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0048307039999999999 2024-09-11T09:51:10.002 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:10.002 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:10.002 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:10.002 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.039093+0000", 2024-09-11T09:51:10.002 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.8604000000000003e-05 2024-09-11T09:51:10.002 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:10.002 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:10.002 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:10.002 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.039184+0000", 2024-09-11T09:51:10.002 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.0196000000000002e-05 2024-09-11T09:51:10.002 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:10.003 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:10.003 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:10.003 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.039873+0000", 2024-09-11T09:51:10.003 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00068960600000000005 2024-09-11T09:51:10.003 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:10.003 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:10.003 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:10.003 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.040292+0000", 2024-09-11T09:51:10.003 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00041865199999999999 2024-09-11T09:51:10.003 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:10.003 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:10.003 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:10.003 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.040317+0000", 2024-09-11T09:51:10.003 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.5439000000000002e-05 2024-09-11T09:51:10.004 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:10.004 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:10.004 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:10.004 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.040336+0000", 2024-09-11T09:51:10.004 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.8754e-05 2024-09-11T09:51:10.004 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:10.004 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:10.004 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:10.004 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:10.004 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:10.004 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:157 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:51:10.004 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:12.452040+0000", 2024-09-11T09:51:10.005 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 57.404575117, 2024-09-11T09:51:10.005 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0068848269999999996, 2024-09-11T09:51:10.005 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:10.005 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:10.005 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:10.005 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:10.005 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:51:10.005 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:51:10.005 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 157 2024-09-11T09:51:10.005 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:10.005 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:10.005 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:10.005 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:10.005 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452040+0000", 2024-09-11T09:51:10.006 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:10.006 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:10.006 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:10.006 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:10.006 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452040+0000", 2024-09-11T09:51:10.006 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:10.006 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:10.006 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:10.006 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:10.006 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452040+0000", 2024-09-11T09:51:10.006 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.22e-07 2024-09-11T09:51:10.006 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:10.006 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:10.006 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:10.007 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452042+0000", 2024-09-11T09:51:10.007 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.7010000000000001e-06 2024-09-11T09:51:10.007 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:10.007 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:10.007 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:10.007 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452045+0000", 2024-09-11T09:51:10.007 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.1310000000000002e-06 2024-09-11T09:51:10.007 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:10.007 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:10.007 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:10.007 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452048+0000", 2024-09-11T09:51:10.007 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.65e-06 2024-09-11T09:51:10.007 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:10.007 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:10.008 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:10.008 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.454453+0000", 2024-09-11T09:51:10.008 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0024052959999999999 2024-09-11T09:51:10.008 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:10.008 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:10.008 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:10.008 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.454498+0000", 2024-09-11T09:51:10.008 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.4978999999999997e-05 2024-09-11T09:51:10.008 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:10.009 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:10.009 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:10.009 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.454567+0000", 2024-09-11T09:51:10.009 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.9110999999999996e-05 2024-09-11T09:51:10.009 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:10.009 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:10.009 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:10.009 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.455464+0000", 2024-09-11T09:51:10.009 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00089696199999999995 2024-09-11T09:51:10.009 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:10.009 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:10.009 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:10.010 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.458844+0000", 2024-09-11T09:51:10.010 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0033802469999999999 2024-09-11T09:51:10.010 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:10.010 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:10.010 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:10.010 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.458889+0000", 2024-09-11T09:51:10.010 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.4441999999999999e-05 2024-09-11T09:51:10.010 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:10.010 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:10.010 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:10.011 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.458925+0000", 2024-09-11T09:51:10.011 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.6086e-05 2024-09-11T09:51:10.011 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:10.011 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:10.011 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:10.011 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:10.011 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:10.011 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:158 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:51:10.011 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:12.452055+0000", 2024-09-11T09:51:10.011 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 57.404559724000002, 2024-09-11T09:51:10.012 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0069533829999999996, 2024-09-11T09:51:10.012 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:10.012 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:10.012 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:10.012 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:10.012 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:51:10.012 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:51:10.012 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 158 2024-09-11T09:51:10.012 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:10.012 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:10.012 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:10.012 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:10.012 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452055+0000", 2024-09-11T09:51:10.013 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:10.013 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:10.013 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:10.013 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:10.013 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452055+0000", 2024-09-11T09:51:10.013 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:10.013 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:10.013 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:10.013 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:10.013 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452056+0000", 2024-09-11T09:51:10.013 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.48e-07 2024-09-11T09:51:10.013 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:10.013 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:10.014 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:10.014 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452087+0000", 2024-09-11T09:51:10.014 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.1149000000000003e-05 2024-09-11T09:51:10.014 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:10.014 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:10.014 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:10.014 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452092+0000", 2024-09-11T09:51:10.014 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.8400000000000002e-06 2024-09-11T09:51:10.014 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:10.014 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:10.014 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:10.014 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452095+0000", 2024-09-11T09:51:10.014 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.0740000000000001e-06 2024-09-11T09:51:10.015 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:10.015 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:10.015 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:10.015 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.454796+0000", 2024-09-11T09:51:10.015 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0027009740000000001 2024-09-11T09:51:10.015 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:10.015 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:10.015 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:10.015 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.454869+0000", 2024-09-11T09:51:10.015 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.3678999999999997e-05 2024-09-11T09:51:10.015 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:10.015 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:10.015 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:10.016 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.454992+0000", 2024-09-11T09:51:10.016 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00012279000000000001 2024-09-11T09:51:10.016 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:10.016 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:10.016 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:10.016 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.456072+0000", 2024-09-11T09:51:10.016 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001080229 2024-09-11T09:51:10.016 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:10.016 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:10.016 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:10.016 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.458931+0000", 2024-09-11T09:51:10.017 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0028591789999999999 2024-09-11T09:51:10.017 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:10.017 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:10.017 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:10.017 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.458972+0000", 2024-09-11T09:51:10.017 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.0170999999999999e-05 2024-09-11T09:51:10.017 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:10.017 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:10.017 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:10.017 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.459009+0000", 2024-09-11T09:51:10.017 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.7049999999999999e-05 2024-09-11T09:51:10.017 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:10.017 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:10.018 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:10.018 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:10.018 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:10.018 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:159 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:51:10.018 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:12.452099+0000", 2024-09-11T09:51:10.018 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 57.404515556, 2024-09-11T09:51:10.018 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0069885950000000002, 2024-09-11T09:51:10.018 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:10.018 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:10.018 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:10.018 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:10.019 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:51:10.019 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:51:10.019 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 159 2024-09-11T09:51:10.019 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:10.019 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:10.019 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:10.019 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:10.019 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452099+0000", 2024-09-11T09:51:10.019 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:10.019 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:10.019 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:10.019 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:10.020 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452099+0000", 2024-09-11T09:51:10.020 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:10.020 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:10.020 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:10.020 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:10.020 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452100+0000", 2024-09-11T09:51:10.020 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.2800000000000003e-07 2024-09-11T09:51:10.020 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:10.020 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:10.021 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:10.021 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452102+0000", 2024-09-11T09:51:10.021 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.824e-06 2024-09-11T09:51:10.021 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:10.021 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:10.021 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:10.021 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452105+0000", 2024-09-11T09:51:10.021 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.0460000000000001e-06 2024-09-11T09:51:10.021 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:10.021 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:10.021 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:10.021 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452107+0000", 2024-09-11T09:51:10.022 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.543e-06 2024-09-11T09:51:10.022 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:10.022 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:10.022 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:10.022 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.455484+0000", 2024-09-11T09:51:10.022 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0033769759999999998 2024-09-11T09:51:10.022 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:10.022 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:10.022 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:10.022 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.455527+0000", 2024-09-11T09:51:10.022 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.2589999999999997e-05 2024-09-11T09:51:10.023 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:10.023 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:10.023 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:10.023 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.455594+0000", 2024-09-11T09:51:10.023 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.6775999999999999e-05 2024-09-11T09:51:10.023 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:10.023 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:10.023 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:10.023 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.456312+0000", 2024-09-11T09:51:10.023 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00071874700000000005 2024-09-11T09:51:10.023 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:10.023 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:10.023 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:10.024 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.459015+0000", 2024-09-11T09:51:10.024 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0027028540000000002 2024-09-11T09:51:10.024 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:10.024 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:10.024 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:10.024 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.459054+0000", 2024-09-11T09:51:10.024 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.8840000000000001e-05 2024-09-11T09:51:10.024 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:10.024 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:10.024 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:10.024 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.459088+0000", 2024-09-11T09:51:10.024 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.4071e-05 2024-09-11T09:51:10.025 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:10.025 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:10.025 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:10.025 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:10.025 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:10.025 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-11T09:51:10.026 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:51:10.027+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:51:10.027 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-11T09:51:10.127 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-09-11T09:51:10.128+0000 7f7b6c7d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31904) UID: 0 2024-09-11T09:51:10.130 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-11T09:51:10.130 INFO:teuthology.orchestra.run.smithi096.stdout: "ops": [], 2024-09-11T09:51:10.130 INFO:teuthology.orchestra.run.smithi096.stdout: "num_ops": 0 2024-09-11T09:51:10.130 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-11T09:51:10.139 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-11T09:51:10.228 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-09-11T09:51:10.229+0000 7f7b6c7d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31904) UID: 0 2024-09-11T09:51:10.282 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-11T09:51:10.282 INFO:teuthology.orchestra.run.smithi096.stdout: "ops": [], 2024-09-11T09:51:10.282 INFO:teuthology.orchestra.run.smithi096.stdout: "complaint_time": 30, 2024-09-11T09:51:10.282 INFO:teuthology.orchestra.run.smithi096.stdout: "num_blocked_ops": 0 2024-09-11T09:51:10.282 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-11T09:51:10.291 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-11T09:51:10.336 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:51:10.334+0000 7f3c96ec9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31909) UID: 0 2024-09-11T09:51:10.430 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:51:10.431+0000 7f3c96ec9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31909) UID: 0 2024-09-11T09:51:10.439 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-11T09:51:10.439 INFO:teuthology.orchestra.run.smithi096.stdout: "size": 20, 2024-09-11T09:51:10.440 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 600, 2024-09-11T09:51:10.440 INFO:teuthology.orchestra.run.smithi096.stdout: "ops": [] 2024-09-11T09:51:10.440 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-11T09:51:10.449 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-11T09:51:10.531 INFO:tasks.ceph.osd.2.smithi017.stderr:2024-09-11T09:51:10.532+0000 7fd20b170640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31672) UID: 0 2024-09-11T09:51:10.591 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-11T09:51:10.591 INFO:teuthology.orchestra.run.smithi096.stdout: "ops": [], 2024-09-11T09:51:10.591 INFO:teuthology.orchestra.run.smithi096.stdout: "num_ops": 0 2024-09-11T09:51:10.591 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-11T09:51:10.600 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-11T09:51:10.645 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-09-11T09:51:10.646+0000 7f7b6c7d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31904) UID: 0 2024-09-11T09:51:10.703 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-11T09:51:10.703 INFO:teuthology.orchestra.run.smithi096.stdout: "ops": [], 2024-09-11T09:51:10.703 INFO:teuthology.orchestra.run.smithi096.stdout: "complaint_time": 30, 2024-09-11T09:51:10.703 INFO:teuthology.orchestra.run.smithi096.stdout: "num_blocked_ops": 0 2024-09-11T09:51:10.703 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-11T09:51:10.711 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-11T09:51:10.732 INFO:tasks.ceph.osd.0.smithi017.stderr:2024-09-11T09:51:10.733+0000 7fb73c0f1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31665) UID: 0 2024-09-11T09:51:10.834 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:51:10.835+0000 7fdec5849640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31907) UID: 0 2024-09-11T09:51:10.855 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-11T09:51:10.855 INFO:teuthology.orchestra.run.smithi096.stdout: "size": 20, 2024-09-11T09:51:10.855 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 600, 2024-09-11T09:51:10.855 INFO:teuthology.orchestra.run.smithi096.stdout: "ops": [ 2024-09-11T09:51:10.856 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:10.856 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4230.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-11T09:51:10.856 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:18.490736+0000", 2024-09-11T09:51:10.856 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 52.365738092000001, 2024-09-11T09:51:10.856 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0059097569999999999, 2024-09-11T09:51:10.856 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:10.856 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:10.856 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:10.857 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:10.857 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4230", 2024-09-11T09:51:10.857 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.17:0/4101109576", 2024-09-11T09:51:10.857 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 1 2024-09-11T09:51:10.857 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:10.857 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:10.857 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:10.857 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:10.857 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.490736+0000", 2024-09-11T09:51:10.857 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:10.857 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:10.857 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:10.857 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:10.858 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.490736+0000", 2024-09-11T09:51:10.858 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:10.858 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:10.858 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:10.858 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:10.858 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.490742+0000", 2024-09-11T09:51:10.858 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.48e-06 2024-09-11T09:51:10.858 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:10.858 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:10.858 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:10.858 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.490749+0000", 2024-09-11T09:51:10.858 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.4569999999999999e-06 2024-09-11T09:51:10.858 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:10.859 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:10.859 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:10.859 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.490762+0000", 2024-09-11T09:51:10.859 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.2721000000000001e-05 2024-09-11T09:51:10.859 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:10.859 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:10.859 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:10.859 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.490774+0000", 2024-09-11T09:51:10.859 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1823e-05 2024-09-11T09:51:10.859 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:10.859 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:10.859 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:10.859 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.490846+0000", 2024-09-11T09:51:10.860 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.1878000000000005e-05 2024-09-11T09:51:10.860 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:10.860 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:10.860 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for peered", 2024-09-11T09:51:10.860 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.490857+0000", 2024-09-11T09:51:10.860 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1017e-05 2024-09-11T09:51:10.860 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:10.860 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:10.860 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:10.860 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.493856+0000", 2024-09-11T09:51:10.860 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00299885 2024-09-11T09:51:10.860 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:10.860 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:10.861 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:10.861 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.494020+0000", 2024-09-11T09:51:10.861 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00016436599999999999 2024-09-11T09:51:10.861 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:10.861 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:10.861 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 2", 2024-09-11T09:51:10.862 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.494186+0000", 2024-09-11T09:51:10.862 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00016562799999999999 2024-09-11T09:51:10.862 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:10.862 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:10.862 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:10.862 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.495407+0000", 2024-09-11T09:51:10.862 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0012217420000000001 2024-09-11T09:51:10.862 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:10.863 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:10.863 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:10.863 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.496579+0000", 2024-09-11T09:51:10.863 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001171844 2024-09-11T09:51:10.863 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:10.863 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:10.863 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:10.863 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.496620+0000", 2024-09-11T09:51:10.863 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.0479000000000003e-05 2024-09-11T09:51:10.863 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:10.863 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:10.863 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:10.864 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.496646+0000", 2024-09-11T09:51:10.864 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.6472e-05 2024-09-11T09:51:10.864 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:10.864 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:10.864 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:10.864 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:10.864 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:10.864 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_repop_reply(client.4230.0:1 2.3 e16/15 ondisk, result = 0)", 2024-09-11T09:51:10.864 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:18.495321+0000", 2024-09-11T09:51:10.864 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 52.361153426999998, 2024-09-11T09:51:10.864 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.4965999999999998e-05, 2024-09-11T09:51:10.864 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:10.864 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:10.865 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "started", 2024-09-11T09:51:10.865 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:10.865 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:10.865 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:10.865 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.495321+0000", 2024-09-11T09:51:10.865 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:10.865 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:10.865 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:10.865 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:10.865 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.495321+0000", 2024-09-11T09:51:10.865 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:10.866 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:10.866 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:10.866 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:10.866 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.495322+0000", 2024-09-11T09:51:10.866 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.7700000000000004e-07 2024-09-11T09:51:10.866 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:10.866 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:10.866 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:10.866 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.495325+0000", 2024-09-11T09:51:10.866 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.7780000000000001e-06 2024-09-11T09:51:10.866 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:10.866 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:10.867 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:10.867 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.495338+0000", 2024-09-11T09:51:10.867 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.2315e-05 2024-09-11T09:51:10.867 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:10.867 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:10.867 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:10.867 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.495345+0000", 2024-09-11T09:51:10.867 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.8889999999999996e-06 2024-09-11T09:51:10.867 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:10.867 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:10.867 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:10.867 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.495392+0000", 2024-09-11T09:51:10.867 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.7524000000000001e-05 2024-09-11T09:51:10.867 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:10.867 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:10.868 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:10.868 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.495402+0000", 2024-09-11T09:51:10.868 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.5840000000000001e-06 2024-09-11T09:51:10.868 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:10.868 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:10.868 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:10.868 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.495416+0000", 2024-09-11T09:51:10.868 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.4199000000000001e-05 2024-09-11T09:51:10.868 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:10.868 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:10.868 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:10.869 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:10.869 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:10.869 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4230.0:2 2.2 2:5cea7035:::rbd_info:head [call rbd.metadata_list in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e16)", 2024-09-11T09:51:10.869 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:18.499576+0000", 2024-09-11T09:51:10.869 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 52.356898919999999, 2024-09-11T09:51:10.869 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00018163000000000001, 2024-09-11T09:51:10.869 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:10.869 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:10.869 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "reached pg", 2024-09-11T09:51:10.869 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:10.869 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4230", 2024-09-11T09:51:10.870 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.17:0/4101109576", 2024-09-11T09:51:10.870 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 2 2024-09-11T09:51:10.870 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:10.870 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:10.870 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:10.870 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:10.870 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.499576+0000", 2024-09-11T09:51:10.870 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:10.870 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:10.870 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:10.870 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:10.870 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.499576+0000", 2024-09-11T09:51:10.870 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:10.871 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:10.871 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:10.871 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:10.871 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.499577+0000", 2024-09-11T09:51:10.871 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.124e-06 2024-09-11T09:51:10.871 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:10.871 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:10.871 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:10.871 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.499584+0000", 2024-09-11T09:51:10.871 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.1949999999999997e-06 2024-09-11T09:51:10.872 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:10.872 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:10.872 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:10.872 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.499597+0000", 2024-09-11T09:51:10.872 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.366e-05 2024-09-11T09:51:10.872 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:10.872 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:10.872 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:10.872 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.499604+0000", 2024-09-11T09:51:10.873 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.2709999999999998e-06 2024-09-11T09:51:10.873 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:10.873 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:10.874 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:10.874 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.499660+0000", 2024-09-11T09:51:10.874 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.6270999999999997e-05 2024-09-11T09:51:10.874 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:10.874 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:10.874 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:10.874 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.499757+0000", 2024-09-11T09:51:10.874 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.7108999999999995e-05 2024-09-11T09:51:10.874 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:10.874 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:10.875 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:10.875 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:10.875 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:10.875 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4230.0:3 2.2 2:5cea7035:::rbd_info:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e16)", 2024-09-11T09:51:10.875 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:18.499937+0000", 2024-09-11T09:51:10.875 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 52.356537131000003, 2024-09-11T09:51:10.875 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000143318, 2024-09-11T09:51:10.875 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:10.875 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:10.876 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "reached pg", 2024-09-11T09:51:10.876 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:10.876 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4230", 2024-09-11T09:51:10.876 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.17:0/4101109576", 2024-09-11T09:51:10.876 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 3 2024-09-11T09:51:10.876 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:10.876 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:10.876 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:10.876 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:10.877 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.499937+0000", 2024-09-11T09:51:10.877 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:10.877 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:10.877 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:10.877 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:10.877 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.499937+0000", 2024-09-11T09:51:10.877 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:10.877 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:10.877 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:10.877 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:10.878 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.499938+0000", 2024-09-11T09:51:10.878 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.195e-06 2024-09-11T09:51:10.878 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:10.878 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:10.878 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:10.878 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.499943+0000", 2024-09-11T09:51:10.878 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.9080000000000003e-06 2024-09-11T09:51:10.878 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:10.878 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:10.879 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:10.879 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.499956+0000", 2024-09-11T09:51:10.879 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.2728000000000001e-05 2024-09-11T09:51:10.879 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:10.879 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:10.879 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:10.879 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.499964+0000", 2024-09-11T09:51:10.879 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.8220000000000007e-06 2024-09-11T09:51:10.880 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:10.880 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:10.880 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:10.880 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.500017+0000", 2024-09-11T09:51:10.880 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.2809000000000002e-05 2024-09-11T09:51:10.880 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:10.880 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:10.880 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:10.880 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.500081+0000", 2024-09-11T09:51:10.880 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.3856000000000007e-05 2024-09-11T09:51:10.880 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:10.880 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:10.880 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:10.881 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:10.881 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:10.881 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4230.0:5 2.2 2:5cea7035:::rbd_info:head [create,write 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-11T09:51:10.881 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:19.492662+0000", 2024-09-11T09:51:10.881 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 51.363812471000003, 2024-09-11T09:51:10.881 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0043824759999999997, 2024-09-11T09:51:10.881 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:10.881 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:10.881 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:10.881 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:10.882 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4230", 2024-09-11T09:51:10.882 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.17:0/4101109576", 2024-09-11T09:51:10.882 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 5 2024-09-11T09:51:10.882 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:10.882 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:10.882 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:10.882 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:10.882 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.492662+0000", 2024-09-11T09:51:10.882 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:10.882 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:10.882 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:10.882 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:10.882 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.492662+0000", 2024-09-11T09:51:10.883 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:10.883 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:10.883 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:10.883 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:10.883 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.492664+0000", 2024-09-11T09:51:10.883 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.3599999999999999e-06 2024-09-11T09:51:10.883 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:10.883 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:10.883 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:10.883 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.492675+0000", 2024-09-11T09:51:10.884 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1182000000000001e-05 2024-09-11T09:51:10.884 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:10.884 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:10.884 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:10.884 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.492693+0000", 2024-09-11T09:51:10.884 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.7031999999999999e-05 2024-09-11T09:51:10.884 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:10.884 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:10.884 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:10.884 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.492702+0000", 2024-09-11T09:51:10.884 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.5540000000000003e-06 2024-09-11T09:51:10.884 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:10.885 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:10.885 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:10.885 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.492767+0000", 2024-09-11T09:51:10.885 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.5411000000000001e-05 2024-09-11T09:51:10.885 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:10.885 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:10.885 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op must wait for map", 2024-09-11T09:51:10.885 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.492773+0000", 2024-09-11T09:51:10.885 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.1039999999999998e-06 2024-09-11T09:51:10.885 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:10.885 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:10.885 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:10.886 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.494354+0000", 2024-09-11T09:51:10.886 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0015814049999999999 2024-09-11T09:51:10.886 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:10.886 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:10.886 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:10.886 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.494476+0000", 2024-09-11T09:51:10.886 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000122089 2024-09-11T09:51:10.886 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:10.886 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:10.886 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 1", 2024-09-11T09:51:10.887 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.494577+0000", 2024-09-11T09:51:10.887 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000100536 2024-09-11T09:51:10.887 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:10.887 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:10.887 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:10.887 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.495770+0000", 2024-09-11T09:51:10.887 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0011932469999999999 2024-09-11T09:51:10.887 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:10.887 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:10.887 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:10.887 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.496966+0000", 2024-09-11T09:51:10.887 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001196003 2024-09-11T09:51:10.888 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:10.888 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:10.888 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:10.888 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.497011+0000", 2024-09-11T09:51:10.888 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.5229999999999999e-05 2024-09-11T09:51:10.888 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:10.888 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:10.888 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:10.888 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.497044+0000", 2024-09-11T09:51:10.888 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.3322999999999999e-05 2024-09-11T09:51:10.889 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:10.889 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:10.889 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:10.889 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:10.889 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:10.889 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_repop_reply(client.4230.0:5 2.2 e17/15 ondisk, result = 0)", 2024-09-11T09:51:10.889 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:19.495669+0000", 2024-09-11T09:51:10.889 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 51.360805683000002, 2024-09-11T09:51:10.889 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000106431, 2024-09-11T09:51:10.889 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:10.889 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:10.890 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "started", 2024-09-11T09:51:10.890 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:10.890 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:10.890 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:10.890 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.495669+0000", 2024-09-11T09:51:10.890 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:10.890 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:10.890 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:10.890 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:10.890 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.495669+0000", 2024-09-11T09:51:10.890 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:10.890 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:10.890 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:10.891 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:10.891 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.495670+0000", 2024-09-11T09:51:10.891 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.218e-06 2024-09-11T09:51:10.891 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:10.891 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:10.891 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:10.891 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.495676+0000", 2024-09-11T09:51:10.891 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.5119999999999997e-06 2024-09-11T09:51:10.891 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:10.891 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:10.891 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:10.891 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.495690+0000", 2024-09-11T09:51:10.891 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.3312000000000001e-05 2024-09-11T09:51:10.892 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:10.892 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:10.892 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:10.892 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.495706+0000", 2024-09-11T09:51:10.892 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.6560000000000001e-05 2024-09-11T09:51:10.892 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:10.892 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:10.892 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:10.892 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.495759+0000", 2024-09-11T09:51:10.893 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.2828000000000003e-05 2024-09-11T09:51:10.893 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:10.893 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:10.893 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:10.893 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.495765+0000", 2024-09-11T09:51:10.893 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.7869999999999998e-06 2024-09-11T09:51:10.893 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:10.893 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:10.893 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:10.894 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.495775+0000", 2024-09-11T09:51:10.894 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.0214000000000001e-05 2024-09-11T09:51:10.894 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:10.894 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:10.894 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:10.894 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:10.894 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:10.894 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4230.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-09-11T09:51:10.894 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:20.498634+0000", 2024-09-11T09:51:10.894 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 50.357840070999998, 2024-09-11T09:51:10.894 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0050648539999999997, 2024-09-11T09:51:10.894 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:10.895 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:10.895 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:10.895 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:10.895 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4230", 2024-09-11T09:51:10.895 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.17:0/4101109576", 2024-09-11T09:51:10.895 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 7 2024-09-11T09:51:10.895 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:10.895 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:10.895 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:10.895 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:10.895 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.498634+0000", 2024-09-11T09:51:10.895 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:10.895 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:10.896 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:10.896 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:10.896 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.498634+0000", 2024-09-11T09:51:10.896 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:10.896 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:10.896 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:10.896 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:10.896 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.498637+0000", 2024-09-11T09:51:10.896 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.1919999999999999e-06 2024-09-11T09:51:10.896 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:10.896 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:10.896 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:10.897 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.498648+0000", 2024-09-11T09:51:10.897 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1116e-05 2024-09-11T09:51:10.897 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:10.897 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:10.897 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:10.897 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.498666+0000", 2024-09-11T09:51:10.897 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.8411999999999999e-05 2024-09-11T09:51:10.897 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:10.897 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:10.897 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:10.897 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.498675+0000", 2024-09-11T09:51:10.897 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.4059999999999994e-06 2024-09-11T09:51:10.897 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:10.897 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:10.898 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:10.898 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.498745+0000", 2024-09-11T09:51:10.898 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.9764000000000005e-05 2024-09-11T09:51:10.898 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:10.898 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:10.898 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op must wait for map", 2024-09-11T09:51:10.898 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.498754+0000", 2024-09-11T09:51:10.898 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.5969999999999992e-06 2024-09-11T09:51:10.898 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:10.898 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:10.899 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:10.899 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.500385+0000", 2024-09-11T09:51:10.899 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0016308360000000001 2024-09-11T09:51:10.899 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:10.899 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:10.899 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:10.899 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.500473+0000", 2024-09-11T09:51:10.899 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.7966000000000004e-05 2024-09-11T09:51:10.899 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:10.900 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:10.900 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 1", 2024-09-11T09:51:10.900 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.500591+0000", 2024-09-11T09:51:10.900 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000117895 2024-09-11T09:51:10.900 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:10.900 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:10.900 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:10.900 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.501685+0000", 2024-09-11T09:51:10.901 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001094886 2024-09-11T09:51:10.901 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:10.901 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:10.901 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:10.901 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.503627+0000", 2024-09-11T09:51:10.901 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001941199 2024-09-11T09:51:10.901 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:10.901 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:10.902 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:10.902 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.503667+0000", 2024-09-11T09:51:10.902 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.0645000000000001e-05 2024-09-11T09:51:10.902 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:10.902 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:10.902 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:10.902 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.503699+0000", 2024-09-11T09:51:10.902 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.1940000000000003e-05 2024-09-11T09:51:10.903 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:10.903 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:10.903 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:10.903 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:10.903 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:10.903 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_repop_reply(client.4230.0:7 2.2 e18/15 ondisk, result = 0)", 2024-09-11T09:51:10.903 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:20.501608+0000", 2024-09-11T09:51:10.903 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 50.354866549, 2024-09-11T09:51:10.903 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.2459000000000005e-05, 2024-09-11T09:51:10.904 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:10.904 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:10.904 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "started", 2024-09-11T09:51:10.904 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:10.904 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:10.904 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:10.904 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.501608+0000", 2024-09-11T09:51:10.904 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:10.905 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:10.905 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:10.905 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:10.905 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.501608+0000", 2024-09-11T09:51:10.905 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:10.905 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:10.905 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:10.905 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:10.906 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.501609+0000", 2024-09-11T09:51:10.906 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.0160000000000001e-06 2024-09-11T09:51:10.906 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:10.906 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:10.906 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:10.906 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.501613+0000", 2024-09-11T09:51:10.906 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.5959999999999997e-06 2024-09-11T09:51:10.906 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:10.906 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:10.907 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:10.907 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.501625+0000", 2024-09-11T09:51:10.907 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1888e-05 2024-09-11T09:51:10.907 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:10.907 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:10.907 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:10.907 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.501634+0000", 2024-09-11T09:51:10.907 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.6139999999999992e-06 2024-09-11T09:51:10.908 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:10.908 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:10.908 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:10.908 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.501676+0000", 2024-09-11T09:51:10.908 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.1953999999999998e-05 2024-09-11T09:51:10.908 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:10.908 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:10.908 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:10.909 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.501682+0000", 2024-09-11T09:51:10.909 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.8000000000000004e-06 2024-09-11T09:51:10.909 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:10.909 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:10.909 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:10.909 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.501690+0000", 2024-09-11T09:51:10.909 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.5909999999999996e-06 2024-09-11T09:51:10.909 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:10.910 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:10.910 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:10.910 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:10.910 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:10.910 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-11T09:51:10.912 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-11T09:51:10.952 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-09-11T09:51:10.953+0000 7f7b6c7d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31904) UID: 0 2024-09-11T09:51:11.033 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:51:11.033 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [], 2024-09-11T09:51:11.033 INFO:teuthology.orchestra.run.smithi017.stdout: "num_ops": 0 2024-09-11T09:51:11.033 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:51:11.035 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:51:11.036+0000 7f3c96ec9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31909) UID: 0 2024-09-11T09:51:11.042 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-11T09:51:11.113 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T09:51:11.150 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:51:11.150 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [], 2024-09-11T09:51:11.150 INFO:teuthology.orchestra.run.smithi017.stdout: "complaint_time": 30, 2024-09-11T09:51:11.150 INFO:teuthology.orchestra.run.smithi017.stdout: "num_blocked_ops": 0 2024-09-11T09:51:11.150 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:51:11.151 INFO:tasks.ceph.osd.2.smithi017.stderr:2024-09-11T09:51:11.151+0000 7fd20b170640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31672) UID: 0 2024-09-11T09:51:11.159 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-11T09:51:11.237 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:51:11.238+0000 7fdec5849640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31907) UID: 0 2024-09-11T09:51:11.239 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-11T09:51:11.239 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T09:51:11.256 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-11T09:51:11.256 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T09:51:11.273 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-11T09:51:11.273 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T09:51:11.290 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-11T09:51:11.291 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T09:51:11.307 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-11T09:51:11.307 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T09:51:11.309 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:51:11.309 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:51:11.319 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-11T09:51:11.324 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-11T09:51:11.325 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T09:51:11.338 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:51:11.339+0000 7fdec5849640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31907) UID: 0 2024-09-11T09:51:11.432 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:51:11.432 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:51:11.439 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:51:11.440+0000 7fdec5849640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31907) UID: 0 2024-09-11T09:51:11.440 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-11T09:51:11.540 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:51:11.541+0000 7fdec5849640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31907) UID: 0 2024-09-11T09:51:11.548 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:51:11.548 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:51:11.556 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-11T09:51:11.641 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:51:11.642+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:51:11.665 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:51:11.665 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:51:11.671 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-11T09:51:11.674 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-11T09:51:11.758 INFO:tasks.ceph.osd.2.smithi017.stderr:2024-09-11T09:51:11.759+0000 7fd20b170640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31672) UID: 0 2024-09-11T09:51:11.824 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:51:11.825 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:51:11.834 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-11T09:51:11.843 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:51:11.844+0000 7f3c96ec9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31909) UID: 0 2024-09-11T09:51:11.945 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:51:11.945+0000 7fdec5849640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31907) UID: 0 2024-09-11T09:51:11.981 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:51:11.982 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:51:11.990 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-11T09:51:12.046 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:51:12.046+0000 7f3c96ec9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31909) UID: 0 2024-09-11T09:51:12.097 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:51:12.098 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:51:12.106 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-11T09:51:12.150 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:51:12.151+0000 7f3c96ec9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31909) UID: 0 2024-09-11T09:51:12.221 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:51:12.222 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:51:12.231 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-11T09:51:12.272 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-09-11T09:51:12.273+0000 7f7b6c7d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31904) UID: 0 2024-09-11T09:51:12.348 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:51:12.349+0000 7fdec5849640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31907) UID: 0 2024-09-11T09:51:12.375 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:51:12.375 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:51:12.384 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-11T09:51:12.467 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:51:12.468+0000 7fdec5849640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31907) UID: 0 2024-09-11T09:51:12.527 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:51:12.527 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:51:12.536 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-11T09:51:12.549 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:51:12.550+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:51:12.650 INFO:tasks.ceph.osd.2.smithi017.stderr:2024-09-11T09:51:12.651+0000 7fd20b170640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31672) UID: 0 2024-09-11T09:51:12.664 INFO:teuthology.orchestra.run.smithi017.stderr:nodeep-scrub is unset 2024-09-11T09:51:12.681 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:51:12.681 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:51:12.689 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-11T09:51:12.751 INFO:tasks.ceph.osd.2.smithi017.stderr:2024-09-11T09:51:12.752+0000 7fd20b170640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31672) UID: 0 2024-09-11T09:51:12.791 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:51:12.792 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:51:12.800 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-11T09:51:12.852 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:51:12.853+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:51:12.902 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:51:12.902 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:51:12.911 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-11T09:51:12.954 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-09-11T09:51:12.955+0000 7f7b6c7d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31904) UID: 0 2024-09-11T09:51:13.013 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:51:13.014 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:51:13.016 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 5, 0] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5] 2024-09-11T09:51:13.016 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-09-11T09:51:13.022 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-11T09:51:13.063 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:51:13.064+0000 7fdec5849640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31907) UID: 0 2024-09-11T09:51:13.155 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:51:13.156+0000 7fdec5849640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31907) UID: 0 2024-09-11T09:51:13.165 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:51:13.165 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:51:13.174 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-11T09:51:13.258 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:51:13.259+0000 7fdec5849640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31907) UID: 0 2024-09-11T09:51:13.317 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:51:13.317 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:51:13.325 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-11T09:51:13.336 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T09:51:13.366 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:51:13.367+0000 7fdec5849640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31907) UID: 0 2024-09-11T09:51:13.434 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:51:13.434 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:51:13.443 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-11T09:51:13.486 INFO:tasks.ceph.osd.2.smithi017.stderr:2024-09-11T09:51:13.487+0000 7fd20b170640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31672) UID: 0 2024-09-11T09:51:13.493 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-11T09:51:13.493 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T09:51:13.551 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:51:13.551 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [], 2024-09-11T09:51:13.552 INFO:teuthology.orchestra.run.smithi017.stdout: "complaint_time": 30, 2024-09-11T09:51:13.552 INFO:teuthology.orchestra.run.smithi017.stdout: "num_blocked_ops": 0 2024-09-11T09:51:13.552 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:51:13.559 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:51:13.560+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:51:13.560 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-11T09:51:13.660 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:51:13.660+0000 7fdec5849640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31907) UID: 0 2024-09-11T09:51:13.663 INFO:teuthology.orchestra.run.smithi017.stderr:reweighted osd.1 to 1 (10000) 2024-09-11T09:51:13.677 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-09-11T09:51:13.713 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:51:13.713 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 20, 2024-09-11T09:51:13.713 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 600, 2024-09-11T09:51:13.714 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [ 2024-09-11T09:51:13.714 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.714 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4183.0:21 1.0 e13/12 1:30306672:devicehealth::main.db.0000000000000000:head v 13'11, mlcod=13'8)", 2024-09-11T09:51:13.714 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:11.491728+0000", 2024-09-11T09:51:13.714 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 62.221585181000002, 2024-09-11T09:51:13.714 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00093269800000000003, 2024-09-11T09:51:13.714 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:13.714 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:13.714 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:13.714 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:13.714 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.714 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:13.714 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.491728+0000", 2024-09-11T09:51:13.714 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:13.714 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:13.715 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.715 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:13.715 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.491728+0000", 2024-09-11T09:51:13.715 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:13.715 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:13.715 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.715 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:13.715 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.491729+0000", 2024-09-11T09:51:13.715 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.8599999999999997e-07 2024-09-11T09:51:13.715 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:13.715 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.715 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:13.715 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.491820+0000", 2024-09-11T09:51:13.716 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.0812999999999999e-05 2024-09-11T09:51:13.716 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:13.716 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.716 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:13.716 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.491833+0000", 2024-09-11T09:51:13.716 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.341e-05 2024-09-11T09:51:13.716 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:13.716 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.716 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:13.716 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.491841+0000", 2024-09-11T09:51:13.716 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.8730000000000007e-06 2024-09-11T09:51:13.716 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:13.716 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.716 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:13.717 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.491916+0000", 2024-09-11T09:51:13.717 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.4881000000000004e-05 2024-09-11T09:51:13.717 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:13.717 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.717 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:13.717 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.491931+0000", 2024-09-11T09:51:13.717 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.4825e-05 2024-09-11T09:51:13.717 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:13.717 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.717 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:13.717 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.492633+0000", 2024-09-11T09:51:13.717 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00070191600000000002 2024-09-11T09:51:13.717 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:13.718 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.718 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:13.718 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.492661+0000", 2024-09-11T09:51:13.718 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.8093999999999999e-05 2024-09-11T09:51:13.718 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:13.718 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:13.718 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:13.718 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:13.718 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.718 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4183.0:31 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'17, mlcod=13'14)", 2024-09-11T09:51:13.718 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:11.499319+0000", 2024-09-11T09:51:13.718 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 62.213994511999999, 2024-09-11T09:51:13.718 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0010073879999999999, 2024-09-11T09:51:13.718 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:13.719 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:13.719 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:13.719 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:13.719 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.719 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:13.719 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.499319+0000", 2024-09-11T09:51:13.719 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:13.719 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:13.719 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.719 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:13.719 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.499319+0000", 2024-09-11T09:51:13.719 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:13.719 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:13.720 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.720 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:13.720 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.499320+0000", 2024-09-11T09:51:13.720 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.7700000000000004e-07 2024-09-11T09:51:13.720 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:13.720 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.720 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:13.720 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.499326+0000", 2024-09-11T09:51:13.720 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.6420000000000001e-06 2024-09-11T09:51:13.720 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:13.720 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.720 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:13.721 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.499339+0000", 2024-09-11T09:51:13.721 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2537999999999999e-05 2024-09-11T09:51:13.721 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:13.721 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.721 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:13.721 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.499345+0000", 2024-09-11T09:51:13.721 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.8220000000000003e-06 2024-09-11T09:51:13.721 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:13.721 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.721 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:13.721 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.499403+0000", 2024-09-11T09:51:13.721 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.8814000000000003e-05 2024-09-11T09:51:13.721 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:13.722 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.722 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:13.722 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.499427+0000", 2024-09-11T09:51:13.722 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.3853999999999999e-05 2024-09-11T09:51:13.723 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:13.723 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.723 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:13.723 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.500294+0000", 2024-09-11T09:51:13.723 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00086622399999999997 2024-09-11T09:51:13.723 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:13.724 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.724 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:13.724 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.500326+0000", 2024-09-11T09:51:13.724 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.2716999999999999e-05 2024-09-11T09:51:13.724 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:13.724 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:13.724 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:13.724 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:13.724 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.724 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4183.0:32 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'18, mlcod=13'14)", 2024-09-11T09:51:13.724 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:11.499796+0000", 2024-09-11T09:51:13.724 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 62.213517306, 2024-09-11T09:51:13.725 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00092915100000000002, 2024-09-11T09:51:13.725 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:13.725 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:13.725 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:13.725 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:13.725 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.725 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:13.725 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.499796+0000", 2024-09-11T09:51:13.725 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:13.725 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:13.725 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.725 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:13.725 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.499796+0000", 2024-09-11T09:51:13.726 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:13.726 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:13.726 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.726 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:13.726 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.499797+0000", 2024-09-11T09:51:13.726 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.9400000000000004e-07 2024-09-11T09:51:13.726 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:13.726 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.726 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:13.726 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.499841+0000", 2024-09-11T09:51:13.726 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.3900000000000003e-05 2024-09-11T09:51:13.726 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:13.726 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.726 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:13.727 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.499852+0000", 2024-09-11T09:51:13.727 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1462e-05 2024-09-11T09:51:13.727 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:13.727 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.727 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:13.727 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.499858+0000", 2024-09-11T09:51:13.727 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.8320000000000002e-06 2024-09-11T09:51:13.727 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:13.727 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.727 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:13.727 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.499915+0000", 2024-09-11T09:51:13.727 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.7187e-05 2024-09-11T09:51:13.727 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:13.728 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.728 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:13.728 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.499930+0000", 2024-09-11T09:51:13.728 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.4928e-05 2024-09-11T09:51:13.728 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:13.728 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.728 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:13.728 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.500697+0000", 2024-09-11T09:51:13.728 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00076658399999999995 2024-09-11T09:51:13.728 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:13.728 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.728 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:13.728 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.500725+0000", 2024-09-11T09:51:13.729 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.8364000000000002e-05 2024-09-11T09:51:13.729 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:13.729 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:13.729 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:13.729 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:13.729 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.729 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4183.0:34 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'20, mlcod=13'15)", 2024-09-11T09:51:13.729 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:11.501016+0000", 2024-09-11T09:51:13.729 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 62.212297788000001, 2024-09-11T09:51:13.729 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00097815399999999996, 2024-09-11T09:51:13.729 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:13.729 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:13.729 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:13.730 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:13.730 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.730 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:13.730 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.501016+0000", 2024-09-11T09:51:13.730 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:13.730 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:13.730 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.730 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:13.730 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.501016+0000", 2024-09-11T09:51:13.730 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:13.730 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:13.730 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.730 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:13.730 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.501016+0000", 2024-09-11T09:51:13.731 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.3799999999999997e-07 2024-09-11T09:51:13.731 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:13.731 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.731 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:13.731 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.501023+0000", 2024-09-11T09:51:13.731 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.2180000000000002e-06 2024-09-11T09:51:13.731 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:13.731 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.731 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:13.731 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.501031+0000", 2024-09-11T09:51:13.731 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.5379999999999997e-06 2024-09-11T09:51:13.731 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:13.731 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.732 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:13.732 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.501035+0000", 2024-09-11T09:51:13.732 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.216e-06 2024-09-11T09:51:13.732 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:13.732 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.732 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:13.732 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.501105+0000", 2024-09-11T09:51:13.732 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.9757000000000003e-05 2024-09-11T09:51:13.732 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:13.732 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.732 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:13.732 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.501120+0000", 2024-09-11T09:51:13.732 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.5358000000000001e-05 2024-09-11T09:51:13.733 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:13.733 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.733 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:13.733 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.501955+0000", 2024-09-11T09:51:13.733 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00083505000000000005 2024-09-11T09:51:13.733 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:13.733 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.733 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:13.733 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.501994+0000", 2024-09-11T09:51:13.733 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.8479000000000002e-05 2024-09-11T09:51:13.733 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:13.733 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:13.733 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:13.733 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:13.734 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.734 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4183.0:38 1.0 e13/12 1:30306672:devicehealth::main.db.0000000000000000:head v 13'24, mlcod=13'22)", 2024-09-11T09:51:13.734 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:11.503891+0000", 2024-09-11T09:51:13.734 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 62.209422345999997, 2024-09-11T09:51:13.734 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000959351, 2024-09-11T09:51:13.734 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:13.734 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:13.734 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:13.734 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:13.734 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.734 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:13.734 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.503891+0000", 2024-09-11T09:51:13.734 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:13.735 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:13.735 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.735 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:13.735 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.503891+0000", 2024-09-11T09:51:13.735 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:13.735 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:13.735 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.735 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:13.735 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.503892+0000", 2024-09-11T09:51:13.735 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.5499999999999998e-07 2024-09-11T09:51:13.735 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:13.735 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.735 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:13.736 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.503967+0000", 2024-09-11T09:51:13.736 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.4993999999999994e-05 2024-09-11T09:51:13.736 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:13.736 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.736 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:13.736 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.503976+0000", 2024-09-11T09:51:13.736 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.4709999999999996e-06 2024-09-11T09:51:13.736 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:13.736 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.736 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:13.736 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.503981+0000", 2024-09-11T09:51:13.736 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.6789999999999996e-06 2024-09-11T09:51:13.736 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:13.737 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.737 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:13.737 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.504021+0000", 2024-09-11T09:51:13.737 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.0065999999999999e-05 2024-09-11T09:51:13.737 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:13.737 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.737 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:13.737 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.504036+0000", 2024-09-11T09:51:13.737 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.468e-05 2024-09-11T09:51:13.737 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:13.737 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.737 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:13.737 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.504813+0000", 2024-09-11T09:51:13.738 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00077718299999999998 2024-09-11T09:51:13.738 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:13.738 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.738 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:13.738 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.504850+0000", 2024-09-11T09:51:13.738 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.7722999999999998e-05 2024-09-11T09:51:13.738 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:13.738 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:13.738 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:13.738 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:13.738 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.738 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4183.0:54 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'32, mlcod=13'29)", 2024-09-11T09:51:13.738 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:11.515404+0000", 2024-09-11T09:51:13.738 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 62.197909740999997, 2024-09-11T09:51:13.739 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00089471799999999997, 2024-09-11T09:51:13.739 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:13.739 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:13.739 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:13.739 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:13.739 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.739 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:13.739 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.515404+0000", 2024-09-11T09:51:13.739 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:13.739 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:13.739 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.739 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:13.739 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.515404+0000", 2024-09-11T09:51:13.740 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:13.740 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:13.740 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.740 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:13.740 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.515404+0000", 2024-09-11T09:51:13.740 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.3799999999999996e-07 2024-09-11T09:51:13.740 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:13.740 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.740 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:13.740 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.515410+0000", 2024-09-11T09:51:13.740 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.1789999999999999e-06 2024-09-11T09:51:13.740 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:13.740 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.741 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:13.741 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.515422+0000", 2024-09-11T09:51:13.741 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1929999999999999e-05 2024-09-11T09:51:13.741 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:13.741 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.741 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:13.741 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.515427+0000", 2024-09-11T09:51:13.741 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.6060000000000002e-06 2024-09-11T09:51:13.741 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:13.741 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.741 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:13.741 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.515482+0000", 2024-09-11T09:51:13.741 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.4743000000000001e-05 2024-09-11T09:51:13.741 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:13.742 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.742 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:13.742 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.515498+0000", 2024-09-11T09:51:13.742 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.5778999999999999e-05 2024-09-11T09:51:13.742 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:13.742 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.742 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:13.742 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.516271+0000", 2024-09-11T09:51:13.742 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00077312599999999996 2024-09-11T09:51:13.742 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:13.742 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.742 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:13.742 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.516298+0000", 2024-09-11T09:51:13.743 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.7617e-05 2024-09-11T09:51:13.743 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:13.743 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:13.743 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:13.743 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:13.743 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.743 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4183.0:57 1.0 e13/12 1:30306672:devicehealth::main.db.0000000000000000:head v 13'35, mlcod=13'33)", 2024-09-11T09:51:13.743 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:11.519823+0000", 2024-09-11T09:51:13.743 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 62.193490683999997, 2024-09-11T09:51:13.743 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00090876100000000001, 2024-09-11T09:51:13.743 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:13.743 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:13.743 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:13.744 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:13.744 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.744 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:13.744 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.519823+0000", 2024-09-11T09:51:13.744 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:13.744 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:13.744 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.744 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:13.744 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.519823+0000", 2024-09-11T09:51:13.744 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:13.744 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:13.744 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.745 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:13.745 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.519823+0000", 2024-09-11T09:51:13.745 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.7000000000000004e-07 2024-09-11T09:51:13.745 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:13.745 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.745 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:13.745 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.519859+0000", 2024-09-11T09:51:13.745 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.5508e-05 2024-09-11T09:51:13.745 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:13.745 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.745 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:13.745 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.519871+0000", 2024-09-11T09:51:13.745 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.221e-05 2024-09-11T09:51:13.746 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:13.746 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.746 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:13.746 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.519877+0000", 2024-09-11T09:51:13.746 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.0410000000000003e-06 2024-09-11T09:51:13.746 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:13.746 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.746 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:13.746 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.519955+0000", 2024-09-11T09:51:13.746 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.8169000000000003e-05 2024-09-11T09:51:13.746 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:13.746 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.746 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:13.747 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.519972+0000", 2024-09-11T09:51:13.747 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6595999999999998e-05 2024-09-11T09:51:13.747 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:13.747 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.747 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:13.747 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.520691+0000", 2024-09-11T09:51:13.747 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000719405 2024-09-11T09:51:13.747 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:13.747 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.747 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:13.747 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.520731+0000", 2024-09-11T09:51:13.747 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.0061999999999997e-05 2024-09-11T09:51:13.747 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:13.748 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:13.748 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:13.748 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:13.748 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.748 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4183.0:58 1.0 e13/12 1:30306672:devicehealth::main.db.0000000000000000:head v 13'36, mlcod=13'33)", 2024-09-11T09:51:13.748 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:11.520285+0000", 2024-09-11T09:51:13.748 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 62.193028276, 2024-09-11T09:51:13.748 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00091014799999999995, 2024-09-11T09:51:13.748 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:13.748 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:13.748 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:13.748 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:13.748 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.749 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:13.749 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.520285+0000", 2024-09-11T09:51:13.749 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:13.749 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:13.749 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.749 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:13.749 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.520285+0000", 2024-09-11T09:51:13.749 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:13.749 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:13.749 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.749 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:13.749 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.520286+0000", 2024-09-11T09:51:13.749 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.4399999999999999e-07 2024-09-11T09:51:13.750 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:13.750 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.750 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:13.750 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.520348+0000", 2024-09-11T09:51:13.750 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.1760000000000005e-05 2024-09-11T09:51:13.750 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:13.750 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.750 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:13.750 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.520359+0000", 2024-09-11T09:51:13.750 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1362999999999999e-05 2024-09-11T09:51:13.750 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:13.750 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.750 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:13.750 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.520364+0000", 2024-09-11T09:51:13.751 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.3600000000000004e-06 2024-09-11T09:51:13.751 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:13.751 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.751 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:13.751 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.520413+0000", 2024-09-11T09:51:13.751 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.8226000000000002e-05 2024-09-11T09:51:13.751 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:13.751 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.751 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:13.751 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.520426+0000", 2024-09-11T09:51:13.751 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3560000000000001e-05 2024-09-11T09:51:13.751 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:13.751 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.752 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:13.752 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.521159+0000", 2024-09-11T09:51:13.752 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00073255499999999997 2024-09-11T09:51:13.752 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:13.752 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.752 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:13.752 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.521195+0000", 2024-09-11T09:51:13.752 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.6480000000000003e-05 2024-09-11T09:51:13.752 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:13.752 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:13.752 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:13.752 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:13.752 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.753 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4183.0:68 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'42, mlcod=13'39)", 2024-09-11T09:51:13.753 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:11.534957+0000", 2024-09-11T09:51:13.753 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 62.178356585000003, 2024-09-11T09:51:13.753 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.001065351, 2024-09-11T09:51:13.753 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:13.753 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:13.753 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:13.753 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:13.753 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.753 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:13.753 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.534957+0000", 2024-09-11T09:51:13.753 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:13.753 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:13.753 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.754 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:13.754 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.534957+0000", 2024-09-11T09:51:13.754 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:13.754 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:13.754 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.754 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:13.754 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.534957+0000", 2024-09-11T09:51:13.754 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.3799999999999998e-07 2024-09-11T09:51:13.754 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:13.754 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.754 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:13.754 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.534964+0000", 2024-09-11T09:51:13.754 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.9809999999999997e-06 2024-09-11T09:51:13.754 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:13.755 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.755 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:13.755 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.534976+0000", 2024-09-11T09:51:13.755 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2119999999999999e-05 2024-09-11T09:51:13.755 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:13.755 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.755 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:13.755 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.534980+0000", 2024-09-11T09:51:13.755 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.9890000000000003e-06 2024-09-11T09:51:13.755 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:13.755 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.755 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:13.755 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.535030+0000", 2024-09-11T09:51:13.756 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.9258000000000002e-05 2024-09-11T09:51:13.756 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:13.756 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.756 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:13.756 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.535044+0000", 2024-09-11T09:51:13.756 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3981e-05 2024-09-11T09:51:13.756 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:13.756 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.756 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:13.756 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.535995+0000", 2024-09-11T09:51:13.756 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000951804 2024-09-11T09:51:13.756 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:13.756 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.756 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:13.757 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.536022+0000", 2024-09-11T09:51:13.757 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.6780000000000001e-05 2024-09-11T09:51:13.757 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:13.757 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:13.757 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:13.757 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:13.757 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.757 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4183.0:69 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'43, mlcod=13'39)", 2024-09-11T09:51:13.757 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:11.535331+0000", 2024-09-11T09:51:13.757 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 62.177982262, 2024-09-11T09:51:13.757 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0011256090000000001, 2024-09-11T09:51:13.757 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:13.757 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:13.758 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:13.758 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:13.758 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.758 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:13.758 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.535331+0000", 2024-09-11T09:51:13.758 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:13.758 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:13.758 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.758 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:13.758 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.535331+0000", 2024-09-11T09:51:13.758 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:13.758 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:13.758 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.758 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:13.759 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.535332+0000", 2024-09-11T09:51:13.759 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.1600000000000001e-07 2024-09-11T09:51:13.759 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:13.759 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.759 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:13.759 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.535393+0000", 2024-09-11T09:51:13.759 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.1452999999999996e-05 2024-09-11T09:51:13.759 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:13.759 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.759 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:13.759 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.535402+0000", 2024-09-11T09:51:13.759 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.4300000000000006e-06 2024-09-11T09:51:13.759 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:13.759 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.760 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:13.760 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.535405+0000", 2024-09-11T09:51:13.760 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.512e-06 2024-09-11T09:51:13.760 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:13.760 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.760 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:13.760 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.535482+0000", 2024-09-11T09:51:13.760 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.7132999999999995e-05 2024-09-11T09:51:13.760 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:13.760 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.760 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:13.760 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.535506+0000", 2024-09-11T09:51:13.760 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.4070000000000002e-05 2024-09-11T09:51:13.761 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:13.761 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.761 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:13.761 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.536424+0000", 2024-09-11T09:51:13.761 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00091766700000000005 2024-09-11T09:51:13.761 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:13.761 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.761 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:13.761 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.536457+0000", 2024-09-11T09:51:13.761 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.2827999999999998e-05 2024-09-11T09:51:13.761 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:13.761 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:13.761 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:13.762 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:13.762 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.762 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4183.0:83 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'53, mlcod=13'50)", 2024-09-11T09:51:13.762 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:11.558836+0000", 2024-09-11T09:51:13.762 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 62.154477006999997, 2024-09-11T09:51:13.762 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00090313100000000005, 2024-09-11T09:51:13.762 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:13.762 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:13.762 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:13.762 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:13.762 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.762 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:13.762 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.558836+0000", 2024-09-11T09:51:13.762 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:13.763 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:13.763 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.763 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:13.763 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.558836+0000", 2024-09-11T09:51:13.763 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:13.763 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:13.763 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.763 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:13.763 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.558837+0000", 2024-09-11T09:51:13.763 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.4900000000000005e-07 2024-09-11T09:51:13.763 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:13.763 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.763 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:13.763 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.558857+0000", 2024-09-11T09:51:13.764 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.9502999999999999e-05 2024-09-11T09:51:13.764 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:13.764 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.764 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:13.764 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.558870+0000", 2024-09-11T09:51:13.764 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3020000000000001e-05 2024-09-11T09:51:13.764 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:13.764 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.764 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:13.764 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.558876+0000", 2024-09-11T09:51:13.764 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.7270000000000004e-06 2024-09-11T09:51:13.764 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:13.764 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.765 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:13.765 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.558934+0000", 2024-09-11T09:51:13.765 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.8100999999999998e-05 2024-09-11T09:51:13.765 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:13.765 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.765 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:13.765 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.558947+0000", 2024-09-11T09:51:13.765 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3269e-05 2024-09-11T09:51:13.765 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:13.765 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.765 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:13.765 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.559720+0000", 2024-09-11T09:51:13.765 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00077287600000000003 2024-09-11T09:51:13.766 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:13.766 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.766 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:13.766 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.559740+0000", 2024-09-11T09:51:13.766 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.9786e-05 2024-09-11T09:51:13.766 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:13.766 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:13.766 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:13.766 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:13.766 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.766 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4183.0:84 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'54, mlcod=13'50)", 2024-09-11T09:51:13.766 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:11.559267+0000", 2024-09-11T09:51:13.766 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 62.154046579999999, 2024-09-11T09:51:13.766 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00093990199999999995, 2024-09-11T09:51:13.767 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:13.767 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:13.767 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:13.767 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:13.767 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.767 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:13.767 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.559267+0000", 2024-09-11T09:51:13.767 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:13.767 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:13.767 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.767 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:13.767 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.559267+0000", 2024-09-11T09:51:13.767 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:13.768 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:13.768 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.768 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:13.768 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.559267+0000", 2024-09-11T09:51:13.768 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.2900000000000003e-07 2024-09-11T09:51:13.768 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:13.768 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.768 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:13.768 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.559322+0000", 2024-09-11T09:51:13.768 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.4429999999999999e-05 2024-09-11T09:51:13.768 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:13.768 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.768 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:13.769 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.559332+0000", 2024-09-11T09:51:13.769 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0473e-05 2024-09-11T09:51:13.769 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:13.769 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.769 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:13.769 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.559338+0000", 2024-09-11T09:51:13.769 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.0730000000000001e-06 2024-09-11T09:51:13.769 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:13.769 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.769 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:13.769 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.559373+0000", 2024-09-11T09:51:13.769 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.4626999999999999e-05 2024-09-11T09:51:13.769 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:13.770 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.770 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:13.770 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.559384+0000", 2024-09-11T09:51:13.770 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1307e-05 2024-09-11T09:51:13.770 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:13.770 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.770 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:13.770 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.560175+0000", 2024-09-11T09:51:13.770 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00079016399999999999 2024-09-11T09:51:13.770 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:13.770 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.770 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:13.770 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.560207+0000", 2024-09-11T09:51:13.771 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.2199000000000001e-05 2024-09-11T09:51:13.771 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:13.771 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:13.771 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:13.771 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:13.771 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.771 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4183.0:89 1.0 e13/12 1:30306672:devicehealth::main.db.0000000000000000:head v 13'59, mlcod=13'57)", 2024-09-11T09:51:13.771 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:11.563614+0000", 2024-09-11T09:51:13.771 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 62.149698960000002, 2024-09-11T09:51:13.771 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00091281399999999996, 2024-09-11T09:51:13.771 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:13.771 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:13.771 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:13.771 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:13.772 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.772 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:13.772 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.563614+0000", 2024-09-11T09:51:13.772 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:13.772 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:13.772 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.772 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:13.772 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.563614+0000", 2024-09-11T09:51:13.772 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:13.772 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:13.772 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.772 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:13.772 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.563615+0000", 2024-09-11T09:51:13.773 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.6899999999999998e-07 2024-09-11T09:51:13.773 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:13.773 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.773 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:13.773 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.563681+0000", 2024-09-11T09:51:13.773 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.5769e-05 2024-09-11T09:51:13.773 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:13.773 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.773 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:13.773 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.563690+0000", 2024-09-11T09:51:13.773 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.6800000000000005e-06 2024-09-11T09:51:13.773 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:13.773 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.774 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:13.774 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.563699+0000", 2024-09-11T09:51:13.774 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.3170000000000002e-06 2024-09-11T09:51:13.774 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:13.774 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.774 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:13.774 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.563734+0000", 2024-09-11T09:51:13.774 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.5023e-05 2024-09-11T09:51:13.774 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:13.774 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.774 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:13.774 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.563746+0000", 2024-09-11T09:51:13.774 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2661e-05 2024-09-11T09:51:13.774 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:13.775 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.775 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:13.775 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.564500+0000", 2024-09-11T09:51:13.775 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00075323100000000004 2024-09-11T09:51:13.775 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:13.775 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.775 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:13.775 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.564527+0000", 2024-09-11T09:51:13.775 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.7664000000000001e-05 2024-09-11T09:51:13.775 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:13.775 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:13.775 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:13.775 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:13.776 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.776 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4183.0:105 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'67, mlcod=13'64)", 2024-09-11T09:51:13.776 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:11.712146+0000", 2024-09-11T09:51:13.776 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 62.001167209999998, 2024-09-11T09:51:13.776 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0010916859999999999, 2024-09-11T09:51:13.776 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:13.776 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:13.776 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:13.776 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:13.776 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.776 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:13.776 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.712146+0000", 2024-09-11T09:51:13.776 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:13.777 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:13.777 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.777 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:13.777 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.712146+0000", 2024-09-11T09:51:13.777 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:13.777 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:13.777 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.777 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:13.777 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.712147+0000", 2024-09-11T09:51:13.777 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.8200000000000001e-07 2024-09-11T09:51:13.777 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:13.777 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.777 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:13.777 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.712155+0000", 2024-09-11T09:51:13.778 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.0309999999999998e-06 2024-09-11T09:51:13.778 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:13.778 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.778 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:13.778 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.712163+0000", 2024-09-11T09:51:13.778 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.5920000000000001e-06 2024-09-11T09:51:13.778 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:13.778 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.778 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:13.778 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.712169+0000", 2024-09-11T09:51:13.779 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.4299999999999997e-06 2024-09-11T09:51:13.779 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:13.779 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.779 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:13.779 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.712418+0000", 2024-09-11T09:51:13.779 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00024978499999999998 2024-09-11T09:51:13.779 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:13.779 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.779 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:13.779 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.712432+0000", 2024-09-11T09:51:13.779 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3190000000000001e-05 2024-09-11T09:51:13.779 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:13.779 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.780 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:13.780 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.713206+0000", 2024-09-11T09:51:13.780 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00077420199999999998 2024-09-11T09:51:13.780 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:13.780 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.780 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:13.780 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.713238+0000", 2024-09-11T09:51:13.780 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.2073999999999998e-05 2024-09-11T09:51:13.780 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:13.780 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:13.780 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:13.780 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:13.780 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.781 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4183.0:106 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'68, mlcod=13'64)", 2024-09-11T09:51:13.781 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:11.712624+0000", 2024-09-11T09:51:13.781 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 62.000689882000003, 2024-09-11T09:51:13.781 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0010009629999999999, 2024-09-11T09:51:13.781 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:13.781 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:13.781 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:13.781 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:13.781 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.781 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:13.781 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.712624+0000", 2024-09-11T09:51:13.781 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:13.781 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:13.782 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.782 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:13.782 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.712624+0000", 2024-09-11T09:51:13.782 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:13.782 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:13.782 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.782 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:13.782 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.712624+0000", 2024-09-11T09:51:13.782 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.4799999999999998e-07 2024-09-11T09:51:13.782 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:13.782 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.782 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:13.782 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.712689+0000", 2024-09-11T09:51:13.783 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.4488999999999999e-05 2024-09-11T09:51:13.783 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:13.783 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.783 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:13.783 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.712707+0000", 2024-09-11T09:51:13.783 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.8272000000000001e-05 2024-09-11T09:51:13.783 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:13.783 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.783 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:13.783 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.712713+0000", 2024-09-11T09:51:13.783 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.2040000000000002e-06 2024-09-11T09:51:13.783 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:13.783 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.783 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:13.784 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.712783+0000", 2024-09-11T09:51:13.784 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.9534000000000002e-05 2024-09-11T09:51:13.784 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:13.784 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.784 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:13.784 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.712811+0000", 2024-09-11T09:51:13.784 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.8620999999999999e-05 2024-09-11T09:51:13.784 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:13.784 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.784 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:13.784 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.713586+0000", 2024-09-11T09:51:13.784 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00077462600000000005 2024-09-11T09:51:13.784 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:13.785 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.785 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:13.785 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.713624+0000", 2024-09-11T09:51:13.785 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.8668999999999999e-05 2024-09-11T09:51:13.785 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:13.785 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:13.785 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:13.785 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:13.785 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.785 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4183.0:134 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'92, mlcod=13'86)", 2024-09-11T09:51:13.785 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:12.035698+0000", 2024-09-11T09:51:13.785 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 61.677615623000001, 2024-09-11T09:51:13.785 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00089550700000000003, 2024-09-11T09:51:13.785 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:13.786 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:13.786 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:13.786 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:13.786 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.786 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:13.786 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.035698+0000", 2024-09-11T09:51:13.786 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:13.786 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:13.786 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.786 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:13.786 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.035698+0000", 2024-09-11T09:51:13.786 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:13.786 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:13.786 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.787 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:13.787 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.035698+0000", 2024-09-11T09:51:13.787 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.0900000000000002e-07 2024-09-11T09:51:13.787 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:13.787 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.787 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:13.787 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.035758+0000", 2024-09-11T09:51:13.787 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.9481999999999997e-05 2024-09-11T09:51:13.787 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:13.787 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.787 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:13.787 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.035766+0000", 2024-09-11T09:51:13.787 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.3760000000000008e-06 2024-09-11T09:51:13.788 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:13.788 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.788 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:13.788 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.035772+0000", 2024-09-11T09:51:13.788 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.8930000000000002e-06 2024-09-11T09:51:13.788 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:13.788 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.788 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:13.788 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.035807+0000", 2024-09-11T09:51:13.788 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.4975999999999999e-05 2024-09-11T09:51:13.788 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:13.788 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.788 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:13.789 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.035817+0000", 2024-09-11T09:51:13.789 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0314e-05 2024-09-11T09:51:13.789 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:13.789 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.789 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:13.789 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.036542+0000", 2024-09-11T09:51:13.789 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00072434400000000003 2024-09-11T09:51:13.789 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:13.789 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.789 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:13.789 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.036593+0000", 2024-09-11T09:51:13.789 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.1613000000000002e-05 2024-09-11T09:51:13.789 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:13.790 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:13.790 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:13.790 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:13.790 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.790 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4183.0:136 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'94, mlcod=13'86)", 2024-09-11T09:51:13.790 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:12.036591+0000", 2024-09-11T09:51:13.790 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 61.676722413, 2024-09-11T09:51:13.790 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00090637900000000002, 2024-09-11T09:51:13.790 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:13.790 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:13.790 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:13.790 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:13.790 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.790 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:13.791 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.036591+0000", 2024-09-11T09:51:13.791 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:13.791 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:13.791 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.791 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:13.791 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.036591+0000", 2024-09-11T09:51:13.791 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:13.791 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:13.791 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.791 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:13.791 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.036592+0000", 2024-09-11T09:51:13.791 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.37e-07 2024-09-11T09:51:13.791 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:13.792 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.792 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:13.792 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.036600+0000", 2024-09-11T09:51:13.792 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.6910000000000003e-06 2024-09-11T09:51:13.792 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:13.792 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.792 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:13.792 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.036609+0000", 2024-09-11T09:51:13.792 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.0229999999999998e-06 2024-09-11T09:51:13.792 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:13.792 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.792 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:13.792 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.036617+0000", 2024-09-11T09:51:13.792 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.6979999999999998e-06 2024-09-11T09:51:13.793 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:13.793 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.793 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:13.793 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.036658+0000", 2024-09-11T09:51:13.793 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.0692000000000003e-05 2024-09-11T09:51:13.793 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:13.793 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.793 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:13.793 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.036670+0000", 2024-09-11T09:51:13.793 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2109e-05 2024-09-11T09:51:13.793 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:13.793 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.794 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:13.794 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.037475+0000", 2024-09-11T09:51:13.794 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00080487199999999999 2024-09-11T09:51:13.794 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:13.794 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.794 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:13.794 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.037497+0000", 2024-09-11T09:51:13.794 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.2557000000000001e-05 2024-09-11T09:51:13.794 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:13.794 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:13.794 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:13.794 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:13.794 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.794 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4183.0:158 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'112, mlcod=13'108)", 2024-09-11T09:51:13.795 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:12.455073+0000", 2024-09-11T09:51:13.795 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 61.25824016, 2024-09-11T09:51:13.795 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00090654500000000003, 2024-09-11T09:51:13.795 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:13.795 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:13.795 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:13.795 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:13.795 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.795 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:13.795 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.455073+0000", 2024-09-11T09:51:13.795 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:13.795 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:13.795 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.796 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:13.796 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.455073+0000", 2024-09-11T09:51:13.796 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:13.796 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:13.796 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.796 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:13.796 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.455074+0000", 2024-09-11T09:51:13.796 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.32e-07 2024-09-11T09:51:13.796 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:13.796 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.796 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:13.796 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.455126+0000", 2024-09-11T09:51:13.796 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.1898999999999999e-05 2024-09-11T09:51:13.797 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:13.797 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.797 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:13.797 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.455136+0000", 2024-09-11T09:51:13.797 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0651e-05 2024-09-11T09:51:13.797 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:13.797 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.797 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:13.797 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.455142+0000", 2024-09-11T09:51:13.797 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.0020000000000003e-06 2024-09-11T09:51:13.797 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:13.797 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.797 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:13.797 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.455192+0000", 2024-09-11T09:51:13.798 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.0164000000000003e-05 2024-09-11T09:51:13.798 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:13.798 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.798 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:13.798 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.455204+0000", 2024-09-11T09:51:13.798 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1178999999999999e-05 2024-09-11T09:51:13.798 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:13.798 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.798 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:13.798 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.455957+0000", 2024-09-11T09:51:13.798 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00075362300000000003 2024-09-11T09:51:13.798 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:13.798 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.799 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:13.799 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.455980+0000", 2024-09-11T09:51:13.799 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.2595e-05 2024-09-11T09:51:13.799 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:13.799 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:13.799 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:13.799 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:13.799 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.799 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "MOSDPGPush(1.0 34/33 [PushOp(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head, version: 13'116, data_included: [0~262688], data_size: 262688, omap_header_size: 0, omap_entries_size: 0, attrset_size: 9, recovery_info: ObjectRecoveryInfo(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head@13'116, size: 262688, copy_subset: [(0, 262688)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 262688, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-11T09:51:13.799 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:47.599030+0000", 2024-09-11T09:51:13.799 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 26.114283533999998, 2024-09-11T09:51:13.799 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0010572209999999999, 2024-09-11T09:51:13.799 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:13.800 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:13.800 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "started", 2024-09-11T09:51:13.800 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:13.800 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.800 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:13.800 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:47.599030+0000", 2024-09-11T09:51:13.800 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:13.800 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:13.800 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.800 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:13.800 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:47.599030+0000", 2024-09-11T09:51:13.800 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:13.800 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:13.801 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.801 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:13.801 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:47.599031+0000", 2024-09-11T09:51:13.801 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.92e-07 2024-09-11T09:51:13.801 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:13.801 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.801 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:13.801 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:47.599328+0000", 2024-09-11T09:51:13.801 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00029708399999999998 2024-09-11T09:51:13.801 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:13.801 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.801 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:13.801 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:47.599353+0000", 2024-09-11T09:51:13.801 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.5721e-05 2024-09-11T09:51:13.802 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:13.802 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.802 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:13.802 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:47.599375+0000", 2024-09-11T09:51:13.802 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.1990000000000001e-05 2024-09-11T09:51:13.802 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:13.802 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.802 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:13.802 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:47.599466+0000", 2024-09-11T09:51:13.802 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.0175999999999999e-05 2024-09-11T09:51:13.802 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:13.802 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.802 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:13.803 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:47.599480+0000", 2024-09-11T09:51:13.803 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.4873e-05 2024-09-11T09:51:13.803 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:13.803 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.803 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:13.803 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:47.600087+0000", 2024-09-11T09:51:13.803 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00060668500000000004 2024-09-11T09:51:13.803 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:13.803 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:13.803 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:13.803 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:13.803 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.803 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "MOSDPGPush(1.0 34/33 [PushOp(1:30306672:devicehealth::main.db.0000000000000000:head, version: 13'117, data_included: [0~327680], data_size: 327680, omap_header_size: 0, omap_entries_size: 0, attrset_size: 10, recovery_info: ObjectRecoveryInfo(1:30306672:devicehealth::main.db.0000000000000000:head@13'117, size: 327680, copy_subset: [(0, 327680)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 327680, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-11T09:51:13.804 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:47.601120+0000", 2024-09-11T09:51:13.804 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 26.112193046000002, 2024-09-11T09:51:13.804 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00099708000000000001, 2024-09-11T09:51:13.804 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:13.804 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:13.804 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "started", 2024-09-11T09:51:13.804 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:13.804 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.804 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:13.804 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:47.601120+0000", 2024-09-11T09:51:13.804 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:13.804 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:13.804 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.805 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:13.805 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:47.601120+0000", 2024-09-11T09:51:13.805 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:13.805 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:13.805 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.805 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:13.805 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:47.601121+0000", 2024-09-11T09:51:13.805 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.0200000000000001e-07 2024-09-11T09:51:13.805 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:13.805 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.806 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:13.806 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:47.601430+0000", 2024-09-11T09:51:13.806 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00030917899999999998 2024-09-11T09:51:13.806 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:13.806 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.806 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:13.806 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:47.601454+0000", 2024-09-11T09:51:13.806 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.4000999999999999e-05 2024-09-11T09:51:13.806 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:13.806 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.806 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:13.806 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:47.601475+0000", 2024-09-11T09:51:13.806 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.0305999999999998e-05 2024-09-11T09:51:13.807 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:13.807 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.807 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:13.807 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:47.601561+0000", 2024-09-11T09:51:13.807 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.6414000000000003e-05 2024-09-11T09:51:13.807 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:13.807 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.807 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:13.807 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:47.601573+0000", 2024-09-11T09:51:13.807 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1894e-05 2024-09-11T09:51:13.807 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:13.807 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:13.807 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:13.807 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:47.602117+0000", 2024-09-11T09:51:13.808 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00054458399999999998 2024-09-11T09:51:13.808 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:13.808 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:13.808 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:13.808 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:13.808 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:13.808 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:51:13.808 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-11T09:51:13.809 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T09:51:13.810 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-11T09:51:13.810 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-11T09:51:13.811 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T09:51:13.811 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-11T09:51:13.811 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T09:51:13.811 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-11T09:51:13.812 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T09:51:13.812 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-11T09:51:13.813 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T09:51:13.813 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-09-11T09:51:13.814+0000 7f7b6c7d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31904) UID: 0 2024-09-11T09:51:13.914 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:51:13.915+0000 7fdec5849640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31907) UID: 0 2024-09-11T09:51:13.958 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:51:13.958 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [], 2024-09-11T09:51:13.958 INFO:teuthology.orchestra.run.smithi017.stdout: "num_ops": 0 2024-09-11T09:51:13.959 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:51:13.967 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-11T09:51:14.015 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:51:14.016+0000 7f3c96ec9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31909) UID: 0 2024-09-11T09:51:14.074 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:51:14.074 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [], 2024-09-11T09:51:14.074 INFO:teuthology.orchestra.run.smithi017.stdout: "complaint_time": 30, 2024-09-11T09:51:14.074 INFO:teuthology.orchestra.run.smithi017.stdout: "num_blocked_ops": 0 2024-09-11T09:51:14.075 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:51:14.083 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-11T09:51:14.127 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:51:14.128+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:51:14.190 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:51:14.191 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 20, 2024-09-11T09:51:14.191 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 600, 2024-09-11T09:51:14.191 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [ 2024-09-11T09:51:14.191 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:14.191 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4230.0:5 2.2 e17/15 2:5cea7035:::rbd_info:head v 17'1, mlcod=0'0)", 2024-09-11T09:51:14.191 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:19.494757+0000", 2024-09-11T09:51:14.191 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 54.697515574000001, 2024-09-11T09:51:14.191 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00082098399999999997, 2024-09-11T09:51:14.191 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:14.191 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:14.191 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:14.191 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:14.192 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:14.192 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:14.192 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:19.494757+0000", 2024-09-11T09:51:14.192 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:14.192 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:14.192 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:14.192 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:14.192 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:19.494757+0000", 2024-09-11T09:51:14.192 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:14.192 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:14.192 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:14.192 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:14.193 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:19.494757+0000", 2024-09-11T09:51:14.193 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.97e-07 2024-09-11T09:51:14.193 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:14.193 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:14.193 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:14.193 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:19.494763+0000", 2024-09-11T09:51:14.193 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.3709999999999996e-06 2024-09-11T09:51:14.193 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:14.193 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:14.193 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:14.193 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:19.494778+0000", 2024-09-11T09:51:14.193 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.4960000000000001e-05 2024-09-11T09:51:14.193 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:14.193 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:14.194 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:14.194 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:19.494801+0000", 2024-09-11T09:51:14.194 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.2388999999999999e-05 2024-09-11T09:51:14.194 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:14.194 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:14.194 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:14.194 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:19.494859+0000", 2024-09-11T09:51:14.194 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.7961999999999998e-05 2024-09-11T09:51:14.194 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:14.194 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:14.194 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:14.194 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:19.494883+0000", 2024-09-11T09:51:14.194 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.4488999999999999e-05 2024-09-11T09:51:14.195 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:14.195 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:14.195 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:14.195 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:19.495534+0000", 2024-09-11T09:51:14.195 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00065075500000000004 2024-09-11T09:51:14.195 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:14.195 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:14.195 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:14.195 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:19.495578+0000", 2024-09-11T09:51:14.195 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.3860999999999999e-05 2024-09-11T09:51:14.195 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:14.195 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:14.195 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:14.196 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:14.196 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:14.196 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4230.0:7 2.2 e18/15 2:5cea7035:::rbd_info:head v 18'2, mlcod=0'0)", 2024-09-11T09:51:14.196 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:20.500782+0000", 2024-09-11T09:51:14.196 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 53.691490694000002, 2024-09-11T09:51:14.196 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000790185, 2024-09-11T09:51:14.196 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:14.196 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:14.196 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:14.196 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:14.196 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:14.196 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:14.196 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:20.500782+0000", 2024-09-11T09:51:14.197 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:14.197 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:14.197 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:14.197 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:14.197 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:20.500782+0000", 2024-09-11T09:51:14.197 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:14.197 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:14.197 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:14.198 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:14.198 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:20.500782+0000", 2024-09-11T09:51:14.198 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.3299999999999998e-07 2024-09-11T09:51:14.198 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:14.198 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:14.198 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:14.198 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:20.500799+0000", 2024-09-11T09:51:14.198 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6830999999999999e-05 2024-09-11T09:51:14.198 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:14.199 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:14.199 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:14.199 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:20.500812+0000", 2024-09-11T09:51:14.199 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3013999999999999e-05 2024-09-11T09:51:14.199 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:14.199 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:14.199 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:14.201 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:20.500824+0000", 2024-09-11T09:51:14.201 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2593e-05 2024-09-11T09:51:14.201 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:14.201 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:14.201 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:14.201 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:20.500895+0000", 2024-09-11T09:51:14.202 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.1018000000000003e-05 2024-09-11T09:51:14.202 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:14.202 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:14.202 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:14.202 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:20.500912+0000", 2024-09-11T09:51:14.202 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6484e-05 2024-09-11T09:51:14.202 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:14.202 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:14.202 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:14.202 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:20.501531+0000", 2024-09-11T09:51:14.202 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00061903399999999997 2024-09-11T09:51:14.202 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:14.202 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:14.202 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:14.203 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:20.501572+0000", 2024-09-11T09:51:14.203 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.0877999999999997e-05 2024-09-11T09:51:14.203 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:14.203 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:14.203 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:14.203 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:14.203 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:14.203 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:51:14.204 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-11T09:51:14.217 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:51:14.218+0000 7f3c96ec9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31909) UID: 0 2024-09-11T09:51:14.318 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:51:14.319+0000 7f3c96ec9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31909) UID: 0 2024-09-11T09:51:14.351 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:51:14.352 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [], 2024-09-11T09:51:14.352 INFO:teuthology.orchestra.run.smithi017.stdout: "num_ops": 0 2024-09-11T09:51:14.352 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:51:14.360 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-11T09:51:14.419 INFO:tasks.ceph.osd.0.smithi017.stderr:2024-09-11T09:51:14.420+0000 7fb73c0f1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31665) UID: 0 2024-09-11T09:51:14.468 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:51:14.468 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [], 2024-09-11T09:51:14.469 INFO:teuthology.orchestra.run.smithi017.stdout: "complaint_time": 30, 2024-09-11T09:51:14.469 INFO:teuthology.orchestra.run.smithi017.stdout: "num_blocked_ops": 0 2024-09-11T09:51:14.469 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:51:14.477 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-11T09:51:14.520 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-09-11T09:51:14.521+0000 7f7b6c7d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31904) UID: 0 2024-09-11T09:51:14.621 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-09-11T09:51:14.622+0000 7f7b6c7d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31904) UID: 0 2024-09-11T09:51:14.625 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:51:14.625 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 20, 2024-09-11T09:51:14.625 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 600, 2024-09-11T09:51:14.625 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [ 2024-09-11T09:51:14.626 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:14.626 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4230.0:1 2.3 e16/15 2:c4c92e5a:::rbd_trash:head v 16'1, mlcod=0'0)", 2024-09-11T09:51:14.626 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:18.494378+0000", 2024-09-11T09:51:14.626 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 56.132123110000002, 2024-09-11T09:51:14.626 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00089752199999999999, 2024-09-11T09:51:14.626 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:14.627 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:14.627 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:14.627 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:14.627 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:14.627 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:14.627 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:18.494378+0000", 2024-09-11T09:51:14.628 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:14.628 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:14.628 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:14.628 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:14.628 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:18.494378+0000", 2024-09-11T09:51:14.628 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:14.629 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:14.629 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:14.629 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:14.629 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:18.494378+0000", 2024-09-11T09:51:14.629 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.9099999999999999e-07 2024-09-11T09:51:14.629 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:14.630 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:14.630 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:14.630 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:18.494385+0000", 2024-09-11T09:51:14.630 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.8129999999999998e-06 2024-09-11T09:51:14.630 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:14.631 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:14.631 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:14.631 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:18.494416+0000", 2024-09-11T09:51:14.631 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.0654000000000002e-05 2024-09-11T09:51:14.631 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:14.631 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:14.632 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:14.632 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:18.494430+0000", 2024-09-11T09:51:14.632 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3754000000000001e-05 2024-09-11T09:51:14.632 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:14.632 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:14.633 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:14.633 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:18.494524+0000", 2024-09-11T09:51:14.633 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.4896999999999996e-05 2024-09-11T09:51:14.633 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:14.633 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:14.633 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:14.633 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:18.494558+0000", 2024-09-11T09:51:14.634 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.3900999999999999e-05 2024-09-11T09:51:14.634 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:14.634 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:14.634 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:14.634 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:18.495231+0000", 2024-09-11T09:51:14.635 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00067233500000000003 2024-09-11T09:51:14.635 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:14.635 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:14.635 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:14.635 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:18.495275+0000", 2024-09-11T09:51:14.635 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.4777000000000002e-05 2024-09-11T09:51:14.636 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:14.636 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:14.636 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:14.636 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:14.636 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:14.637 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:51:14.638 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-11T09:51:14.667 INFO:teuthology.orchestra.run.smithi017.stderr:reweighted osd.2 to 1 (10000) 2024-09-11T09:51:14.680 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-09-11T09:51:14.680 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-11T09:51:14.721 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:51:14.722+0000 7fdec5849640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31907) UID: 0 2024-09-11T09:51:14.756 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-11T09:51:14.756 INFO:teuthology.orchestra.run.smithi096.stdout: "ops": [], 2024-09-11T09:51:14.756 INFO:teuthology.orchestra.run.smithi096.stdout: "num_ops": 0 2024-09-11T09:51:14.757 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-11T09:51:14.765 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-11T09:51:14.822 INFO:tasks.ceph.osd.0.smithi017.stderr:2024-09-11T09:51:14.823+0000 7fb73c0f1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31665) UID: 0 2024-09-11T09:51:14.909 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-11T09:51:14.909 INFO:teuthology.orchestra.run.smithi096.stdout: "ops": [], 2024-09-11T09:51:14.909 INFO:teuthology.orchestra.run.smithi096.stdout: "complaint_time": 30, 2024-09-11T09:51:14.909 INFO:teuthology.orchestra.run.smithi096.stdout: "num_blocked_ops": 0 2024-09-11T09:51:14.909 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-11T09:51:14.918 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-11T09:51:14.961 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:51:14.962+0000 7f3c96ec9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31909) UID: 0 2024-09-11T09:51:15.025 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-11T09:51:15.025 INFO:teuthology.orchestra.run.smithi096.stdout: "size": 20, 2024-09-11T09:51:15.025 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 600, 2024-09-11T09:51:15.025 INFO:teuthology.orchestra.run.smithi096.stdout: "ops": [ 2024-09-11T09:51:15.025 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.026 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:74 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:51:15.026 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.539067+0000", 2024-09-11T09:51:15.026 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 63.483052954000001, 2024-09-11T09:51:15.026 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.006945957, 2024-09-11T09:51:15.026 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:15.026 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:15.026 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:15.026 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:15.026 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:51:15.026 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:51:15.026 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 74 2024-09-11T09:51:15.026 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.026 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:15.027 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.027 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:15.027 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.539067+0000", 2024-09-11T09:51:15.027 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:15.027 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.027 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.027 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:15.027 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.539067+0000", 2024-09-11T09:51:15.027 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:15.027 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.027 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.027 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:15.027 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.539068+0000", 2024-09-11T09:51:15.028 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.6000000000000003e-07 2024-09-11T09:51:15.028 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.028 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.028 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:15.028 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.539072+0000", 2024-09-11T09:51:15.028 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.0450000000000002e-06 2024-09-11T09:51:15.028 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.028 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.028 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:15.028 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.539078+0000", 2024-09-11T09:51:15.028 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.6889999999999998e-06 2024-09-11T09:51:15.028 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.028 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.028 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:15.029 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.539083+0000", 2024-09-11T09:51:15.029 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.7180000000000004e-06 2024-09-11T09:51:15.029 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.029 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.029 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:15.029 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.540230+0000", 2024-09-11T09:51:15.029 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0011466250000000001 2024-09-11T09:51:15.029 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.029 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.029 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:15.029 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.540279+0000", 2024-09-11T09:51:15.029 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.9667999999999999e-05 2024-09-11T09:51:15.029 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.030 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.030 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:15.030 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.540357+0000", 2024-09-11T09:51:15.030 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.7863999999999999e-05 2024-09-11T09:51:15.030 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.030 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.030 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:15.030 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.541086+0000", 2024-09-11T09:51:15.030 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00072921100000000001 2024-09-11T09:51:15.030 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.030 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.030 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:15.030 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.545949+0000", 2024-09-11T09:51:15.030 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0048626040000000004 2024-09-11T09:51:15.031 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.031 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.031 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:15.031 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.545989+0000", 2024-09-11T09:51:15.031 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.0259000000000002e-05 2024-09-11T09:51:15.031 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.031 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.031 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:15.031 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.546013+0000", 2024-09-11T09:51:15.031 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.3614000000000001e-05 2024-09-11T09:51:15.031 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:15.031 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:15.031 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:15.032 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.032 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.032 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:108 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:51:15.032 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.711510+0000", 2024-09-11T09:51:15.032 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 63.310609782, 2024-09-11T09:51:15.032 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0054615649999999998, 2024-09-11T09:51:15.032 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:15.032 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:15.032 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:15.032 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:15.032 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:51:15.032 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:51:15.033 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 108 2024-09-11T09:51:15.033 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.033 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:15.033 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.033 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:15.033 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711510+0000", 2024-09-11T09:51:15.033 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:15.033 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.033 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.033 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:15.033 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711510+0000", 2024-09-11T09:51:15.033 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:15.033 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.033 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.034 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:15.034 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711511+0000", 2024-09-11T09:51:15.034 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.1999999999999998e-07 2024-09-11T09:51:15.034 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.034 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.034 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:15.034 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711516+0000", 2024-09-11T09:51:15.034 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.4140000000000002e-06 2024-09-11T09:51:15.034 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.034 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.034 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:15.034 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711527+0000", 2024-09-11T09:51:15.034 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.0169e-05 2024-09-11T09:51:15.034 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.035 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.035 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:15.035 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711535+0000", 2024-09-11T09:51:15.035 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.3299999999999999e-06 2024-09-11T09:51:15.035 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.035 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.035 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:15.035 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.713763+0000", 2024-09-11T09:51:15.035 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0022281079999999999 2024-09-11T09:51:15.035 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.035 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.035 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:15.035 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.713812+0000", 2024-09-11T09:51:15.036 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.8544000000000001e-05 2024-09-11T09:51:15.036 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.036 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.036 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:15.036 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.713909+0000", 2024-09-11T09:51:15.036 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.7113000000000003e-05 2024-09-11T09:51:15.036 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.036 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.036 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:15.036 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.716873+0000", 2024-09-11T09:51:15.036 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0029642710000000001 2024-09-11T09:51:15.036 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.036 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.037 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:15.037 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.716918+0000", 2024-09-11T09:51:15.037 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.5447e-05 2024-09-11T09:51:15.037 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.037 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.037 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:15.037 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.716947+0000", 2024-09-11T09:51:15.037 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.8181e-05 2024-09-11T09:51:15.037 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.037 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.037 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:15.037 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.716972+0000", 2024-09-11T09:51:15.037 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.5068000000000001e-05 2024-09-11T09:51:15.037 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:15.038 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:15.038 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:15.038 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.038 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.038 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:109 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:51:15.038 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.711555+0000", 2024-09-11T09:51:15.038 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 63.310564532999997, 2024-09-11T09:51:15.038 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.005479902, 2024-09-11T09:51:15.038 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:15.038 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:15.038 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:15.038 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:15.038 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:51:15.038 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:51:15.039 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 109 2024-09-11T09:51:15.039 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.039 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:15.039 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.039 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:15.039 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711555+0000", 2024-09-11T09:51:15.039 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:15.039 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.039 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.039 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:15.039 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711555+0000", 2024-09-11T09:51:15.039 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:15.039 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.040 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.040 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:15.040 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711556+0000", 2024-09-11T09:51:15.040 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.3200000000000005e-07 2024-09-11T09:51:15.040 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.040 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.040 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:15.040 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711601+0000", 2024-09-11T09:51:15.040 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.5349000000000002e-05 2024-09-11T09:51:15.040 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.040 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.040 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:15.040 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711612+0000", 2024-09-11T09:51:15.040 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.0701e-05 2024-09-11T09:51:15.041 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.041 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.041 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:15.041 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711619+0000", 2024-09-11T09:51:15.041 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.7610000000000002e-06 2024-09-11T09:51:15.041 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.041 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.041 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:15.041 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.714371+0000", 2024-09-11T09:51:15.041 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0027526260000000002 2024-09-11T09:51:15.041 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.041 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.041 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:15.042 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.714417+0000", 2024-09-11T09:51:15.042 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.5893999999999998e-05 2024-09-11T09:51:15.042 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.042 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.042 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:15.042 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.714496+0000", 2024-09-11T09:51:15.042 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.8539000000000001e-05 2024-09-11T09:51:15.042 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.042 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.042 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:15.042 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.716910+0000", 2024-09-11T09:51:15.042 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0024137640000000001 2024-09-11T09:51:15.042 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.043 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.043 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:15.043 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.716989+0000", 2024-09-11T09:51:15.043 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.9198999999999998e-05 2024-09-11T09:51:15.043 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.043 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.043 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:15.043 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.717016+0000", 2024-09-11T09:51:15.043 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.6699e-05 2024-09-11T09:51:15.043 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.043 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.043 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:15.043 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.717035+0000", 2024-09-11T09:51:15.043 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.9738e-05 2024-09-11T09:51:15.044 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:15.044 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:15.044 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:15.044 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.044 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.044 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:110 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:51:15.044 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.711636+0000", 2024-09-11T09:51:15.044 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 63.310484072999998, 2024-09-11T09:51:15.044 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0067613070000000003, 2024-09-11T09:51:15.044 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:15.044 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:15.044 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:15.044 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:15.044 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:51:15.045 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:51:15.045 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 110 2024-09-11T09:51:15.045 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.045 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:15.045 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.045 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:15.045 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711636+0000", 2024-09-11T09:51:15.045 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:15.045 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.045 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.045 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:15.045 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711636+0000", 2024-09-11T09:51:15.045 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:15.046 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.046 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.046 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:15.046 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711637+0000", 2024-09-11T09:51:15.046 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.9999999999999997e-07 2024-09-11T09:51:15.046 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.046 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.046 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:15.046 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711641+0000", 2024-09-11T09:51:15.046 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.2570000000000001e-06 2024-09-11T09:51:15.046 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.046 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.046 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:15.046 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711651+0000", 2024-09-11T09:51:15.047 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.0285999999999999e-05 2024-09-11T09:51:15.047 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.047 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.047 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:15.047 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711658+0000", 2024-09-11T09:51:15.047 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.3540000000000002e-06 2024-09-11T09:51:15.047 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.047 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.047 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:15.047 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.716475+0000", 2024-09-11T09:51:15.047 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0048162819999999999 2024-09-11T09:51:15.047 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.047 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.047 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:15.048 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.716526+0000", 2024-09-11T09:51:15.048 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.0905000000000001e-05 2024-09-11T09:51:15.048 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.048 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.048 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:15.048 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.716601+0000", 2024-09-11T09:51:15.048 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.5283000000000004e-05 2024-09-11T09:51:15.048 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.048 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.048 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:15.048 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.717933+0000", 2024-09-11T09:51:15.048 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001331622 2024-09-11T09:51:15.048 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.049 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.049 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:15.049 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718332+0000", 2024-09-11T09:51:15.049 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00039909899999999998 2024-09-11T09:51:15.049 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.049 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.049 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:15.049 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718363+0000", 2024-09-11T09:51:15.049 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.1636e-05 2024-09-11T09:51:15.049 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.049 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.049 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:15.049 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718397+0000", 2024-09-11T09:51:15.049 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.3883000000000001e-05 2024-09-11T09:51:15.050 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:15.050 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:15.050 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:15.050 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.050 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.050 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:111 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:51:15.050 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.711668+0000", 2024-09-11T09:51:15.050 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 63.310451643, 2024-09-11T09:51:15.050 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0077145570000000004, 2024-09-11T09:51:15.050 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:15.050 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:15.050 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:15.050 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:15.050 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:51:15.051 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:51:15.051 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 111 2024-09-11T09:51:15.051 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.051 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:15.051 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.051 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:15.051 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711668+0000", 2024-09-11T09:51:15.051 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:15.051 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.051 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.051 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:15.051 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711668+0000", 2024-09-11T09:51:15.051 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:15.051 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.052 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.052 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:15.052 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711669+0000", 2024-09-11T09:51:15.052 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.99e-07 2024-09-11T09:51:15.052 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.052 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.052 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:15.052 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711673+0000", 2024-09-11T09:51:15.052 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.7440000000000001e-06 2024-09-11T09:51:15.052 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.052 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.052 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:15.052 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711686+0000", 2024-09-11T09:51:15.052 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.3317e-05 2024-09-11T09:51:15.053 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.053 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.053 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:15.053 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711697+0000", 2024-09-11T09:51:15.053 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1022000000000001e-05 2024-09-11T09:51:15.053 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.053 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.053 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:15.053 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.717065+0000", 2024-09-11T09:51:15.053 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.005367531 2024-09-11T09:51:15.053 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.053 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.053 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:15.053 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.717114+0000", 2024-09-11T09:51:15.054 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.9707999999999999e-05 2024-09-11T09:51:15.054 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.054 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.054 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:15.054 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.717190+0000", 2024-09-11T09:51:15.054 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.5394000000000004e-05 2024-09-11T09:51:15.054 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.054 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.054 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:15.054 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718793+0000", 2024-09-11T09:51:15.054 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001603653 2024-09-11T09:51:15.054 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.054 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.055 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:15.055 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719329+0000", 2024-09-11T09:51:15.055 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00053543799999999995 2024-09-11T09:51:15.055 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.055 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.055 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:15.055 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719359+0000", 2024-09-11T09:51:15.055 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.0420999999999999e-05 2024-09-11T09:51:15.055 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.055 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.055 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:15.055 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719383+0000", 2024-09-11T09:51:15.055 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.3730000000000001e-05 2024-09-11T09:51:15.056 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:15.056 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:15.056 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:15.056 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.056 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.056 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:112 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:51:15.056 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.711712+0000", 2024-09-11T09:51:15.056 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 63.310408082999999, 2024-09-11T09:51:15.056 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.008162084, 2024-09-11T09:51:15.056 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:15.056 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:15.056 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:15.056 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:15.056 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:51:15.057 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:51:15.057 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 112 2024-09-11T09:51:15.057 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.057 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:15.057 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.057 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:15.057 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711712+0000", 2024-09-11T09:51:15.057 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:15.057 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.057 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.057 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:15.057 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711712+0000", 2024-09-11T09:51:15.057 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:15.058 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.058 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.058 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:15.058 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711712+0000", 2024-09-11T09:51:15.058 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.6000000000000004e-07 2024-09-11T09:51:15.058 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.058 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.058 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:15.058 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711808+0000", 2024-09-11T09:51:15.058 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.5977999999999995e-05 2024-09-11T09:51:15.058 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.058 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.058 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:15.058 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711819+0000", 2024-09-11T09:51:15.059 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.0755e-05 2024-09-11T09:51:15.059 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.059 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.059 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:15.059 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711826+0000", 2024-09-11T09:51:15.059 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.7970000000000003e-06 2024-09-11T09:51:15.059 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.059 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.059 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:15.059 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.717435+0000", 2024-09-11T09:51:15.059 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0056090239999999998 2024-09-11T09:51:15.059 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.059 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.060 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:15.060 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.717496+0000", 2024-09-11T09:51:15.060 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.0543e-05 2024-09-11T09:51:15.060 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.060 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.060 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:15.060 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.717591+0000", 2024-09-11T09:51:15.060 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.5667000000000004e-05 2024-09-11T09:51:15.060 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.060 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.060 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:15.060 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718831+0000", 2024-09-11T09:51:15.060 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001240204 2024-09-11T09:51:15.061 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.061 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.061 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:15.061 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719819+0000", 2024-09-11T09:51:15.061 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00098751500000000001 2024-09-11T09:51:15.061 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.061 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.061 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:15.061 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719850+0000", 2024-09-11T09:51:15.061 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.1535999999999998e-05 2024-09-11T09:51:15.061 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.061 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.061 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:15.062 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719874+0000", 2024-09-11T09:51:15.062 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.3504999999999999e-05 2024-09-11T09:51:15.062 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:15.062 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:15.062 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:15.062 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.062 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.062 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:113 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:51:15.062 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.711842+0000", 2024-09-11T09:51:15.062 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 63.310277583999998, 2024-09-11T09:51:15.062 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0081160099999999999, 2024-09-11T09:51:15.062 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:15.062 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:15.063 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:15.063 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:15.063 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:51:15.063 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:51:15.063 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 113 2024-09-11T09:51:15.063 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.063 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:15.063 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.063 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:15.063 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711842+0000", 2024-09-11T09:51:15.063 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:15.063 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.063 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.063 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:15.064 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711842+0000", 2024-09-11T09:51:15.064 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:15.064 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.064 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.064 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:15.064 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711843+0000", 2024-09-11T09:51:15.064 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.4399999999999999e-07 2024-09-11T09:51:15.064 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.064 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.064 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:15.064 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711848+0000", 2024-09-11T09:51:15.064 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.4159999999999997e-06 2024-09-11T09:51:15.064 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.064 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.065 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:15.065 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711856+0000", 2024-09-11T09:51:15.065 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.7870000000000007e-06 2024-09-11T09:51:15.065 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.065 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.065 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:15.065 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711862+0000", 2024-09-11T09:51:15.065 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.3689999999999998e-06 2024-09-11T09:51:15.065 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.065 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.065 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:15.065 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.717965+0000", 2024-09-11T09:51:15.065 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0061028899999999997 2024-09-11T09:51:15.065 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.066 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.066 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:15.066 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718023+0000", 2024-09-11T09:51:15.066 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.8313000000000002e-05 2024-09-11T09:51:15.066 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.066 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.066 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:15.066 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718109+0000", 2024-09-11T09:51:15.066 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.6205000000000006e-05 2024-09-11T09:51:15.066 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.066 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.066 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:15.066 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719808+0000", 2024-09-11T09:51:15.067 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0016990060000000001 2024-09-11T09:51:15.067 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.067 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.067 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:15.067 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719883+0000", 2024-09-11T09:51:15.067 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.4624999999999998e-05 2024-09-11T09:51:15.067 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.067 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.067 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:15.067 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719910+0000", 2024-09-11T09:51:15.067 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.7512e-05 2024-09-11T09:51:15.067 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.067 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.067 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:15.068 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719958+0000", 2024-09-11T09:51:15.068 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.8143000000000003e-05 2024-09-11T09:51:15.068 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:15.068 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:15.068 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:15.068 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.068 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.068 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:51:15.068 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.711887+0000", 2024-09-11T09:51:15.068 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 63.310232970000001, 2024-09-11T09:51:15.068 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0086063589999999992, 2024-09-11T09:51:15.068 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:15.069 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:15.069 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:15.069 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:15.069 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:51:15.069 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:51:15.069 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 114 2024-09-11T09:51:15.069 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.069 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:15.069 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.069 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:15.069 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711887+0000", 2024-09-11T09:51:15.069 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:15.069 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.069 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.070 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:15.070 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711887+0000", 2024-09-11T09:51:15.070 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:15.070 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.070 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.070 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:15.070 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711887+0000", 2024-09-11T09:51:15.070 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.1500000000000005e-07 2024-09-11T09:51:15.070 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.070 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.070 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:15.070 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711897+0000", 2024-09-11T09:51:15.070 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.1900000000000001e-06 2024-09-11T09:51:15.070 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.071 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.071 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:15.071 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711902+0000", 2024-09-11T09:51:15.071 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.2349999999999999e-06 2024-09-11T09:51:15.071 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.071 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.071 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:15.071 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711906+0000", 2024-09-11T09:51:15.071 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.7129999999999999e-06 2024-09-11T09:51:15.071 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.071 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.071 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:15.071 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718429+0000", 2024-09-11T09:51:15.072 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0065231960000000002 2024-09-11T09:51:15.072 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.072 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.072 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:15.072 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718480+0000", 2024-09-11T09:51:15.072 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.1161e-05 2024-09-11T09:51:15.072 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.072 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.072 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:15.072 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718558+0000", 2024-09-11T09:51:15.072 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.7519000000000001e-05 2024-09-11T09:51:15.072 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.072 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.072 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:15.073 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720038+0000", 2024-09-11T09:51:15.073 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001480016 2024-09-11T09:51:15.073 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.073 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.073 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:15.073 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720417+0000", 2024-09-11T09:51:15.073 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00037943699999999999 2024-09-11T09:51:15.073 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.073 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.073 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:15.073 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720452+0000", 2024-09-11T09:51:15.073 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.4595000000000003e-05 2024-09-11T09:51:15.073 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.074 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.074 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:15.074 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720493+0000", 2024-09-11T09:51:15.074 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.1782000000000001e-05 2024-09-11T09:51:15.074 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:15.074 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:15.074 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:15.074 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.074 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.074 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:51:15.074 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.711912+0000", 2024-09-11T09:51:15.074 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 63.310208301000003, 2024-09-11T09:51:15.074 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0086864750000000008, 2024-09-11T09:51:15.074 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:15.075 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:15.075 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:15.075 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:15.075 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:51:15.075 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:51:15.075 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 115 2024-09-11T09:51:15.075 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.075 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:15.075 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.075 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:15.075 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711912+0000", 2024-09-11T09:51:15.075 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:15.075 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.076 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.076 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:15.076 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711912+0000", 2024-09-11T09:51:15.076 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:15.076 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.076 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.076 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:15.076 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711912+0000", 2024-09-11T09:51:15.076 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.3900000000000001e-07 2024-09-11T09:51:15.076 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.076 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.076 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:15.076 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711969+0000", 2024-09-11T09:51:15.077 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.7501999999999999e-05 2024-09-11T09:51:15.077 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.077 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.077 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:15.077 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711986+0000", 2024-09-11T09:51:15.077 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.6402999999999999e-05 2024-09-11T09:51:15.077 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.077 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.077 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:15.077 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711990+0000", 2024-09-11T09:51:15.077 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.929e-06 2024-09-11T09:51:15.077 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.077 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.077 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:15.078 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718861+0000", 2024-09-11T09:51:15.078 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0068710530000000002 2024-09-11T09:51:15.078 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.078 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.078 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:15.078 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718908+0000", 2024-09-11T09:51:15.078 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.7431e-05 2024-09-11T09:51:15.078 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.078 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.078 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:15.078 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718984+0000", 2024-09-11T09:51:15.078 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.5500999999999994e-05 2024-09-11T09:51:15.078 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.078 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.079 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:15.079 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720500+0000", 2024-09-11T09:51:15.079 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0015158820000000001 2024-09-11T09:51:15.079 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.079 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.079 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:15.079 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720536+0000", 2024-09-11T09:51:15.079 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.6118000000000002e-05 2024-09-11T09:51:15.079 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.079 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.079 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:15.079 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720570+0000", 2024-09-11T09:51:15.079 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.4387999999999997e-05 2024-09-11T09:51:15.080 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.080 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.080 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:15.080 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720598+0000", 2024-09-11T09:51:15.080 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.8028999999999999e-05 2024-09-11T09:51:15.080 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:15.080 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:15.080 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:15.080 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.080 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.080 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:51:15.080 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.711998+0000", 2024-09-11T09:51:15.080 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 63.310122122000003, 2024-09-11T09:51:15.080 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.010785813999999999, 2024-09-11T09:51:15.081 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:15.081 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:15.081 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:15.081 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:15.081 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:51:15.081 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:51:15.081 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 116 2024-09-11T09:51:15.081 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.081 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:15.081 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.081 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:15.081 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711998+0000", 2024-09-11T09:51:15.081 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:15.082 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.082 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.082 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:15.082 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711998+0000", 2024-09-11T09:51:15.082 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:15.082 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.082 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.082 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:15.082 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711998+0000", 2024-09-11T09:51:15.082 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.3300000000000003e-07 2024-09-11T09:51:15.082 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.082 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.082 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:15.082 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712001+0000", 2024-09-11T09:51:15.083 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.9349999999999999e-06 2024-09-11T09:51:15.083 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.083 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.083 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:15.083 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712006+0000", 2024-09-11T09:51:15.083 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.1630000000000004e-06 2024-09-11T09:51:15.083 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.083 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.083 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:15.083 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712010+0000", 2024-09-11T09:51:15.083 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.574e-06 2024-09-11T09:51:15.083 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.083 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.084 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:15.084 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719415+0000", 2024-09-11T09:51:15.084 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0074049099999999998 2024-09-11T09:51:15.084 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.084 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.084 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:15.084 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719465+0000", 2024-09-11T09:51:15.084 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.0129000000000001e-05 2024-09-11T09:51:15.084 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.084 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.084 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:15.084 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719545+0000", 2024-09-11T09:51:15.084 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.0253999999999994e-05 2024-09-11T09:51:15.084 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.085 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.085 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:15.085 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720636+0000", 2024-09-11T09:51:15.085 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0010909330000000001 2024-09-11T09:51:15.085 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.085 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.085 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:15.085 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.722707+0000", 2024-09-11T09:51:15.085 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0020713300000000001 2024-09-11T09:51:15.085 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.085 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.085 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:15.085 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.722743+0000", 2024-09-11T09:51:15.085 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.5351000000000001e-05 2024-09-11T09:51:15.086 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.086 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.086 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:15.086 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.722784+0000", 2024-09-11T09:51:15.086 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.0802e-05 2024-09-11T09:51:15.086 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:15.086 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:15.086 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:15.086 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.086 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.086 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:51:15.086 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.712014+0000", 2024-09-11T09:51:15.086 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 63.310106081999997, 2024-09-11T09:51:15.086 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.013420938, 2024-09-11T09:51:15.087 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:15.087 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:15.087 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:15.087 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:15.087 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:51:15.087 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:51:15.087 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 117 2024-09-11T09:51:15.087 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.087 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:15.087 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.087 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:15.087 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712014+0000", 2024-09-11T09:51:15.087 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:15.087 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.088 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.088 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:15.088 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712014+0000", 2024-09-11T09:51:15.088 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:15.088 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.088 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.088 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:15.088 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712014+0000", 2024-09-11T09:51:15.088 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.2000000000000001e-07 2024-09-11T09:51:15.088 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.088 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.088 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:15.088 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712016+0000", 2024-09-11T09:51:15.089 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.2309999999999999e-06 2024-09-11T09:51:15.089 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.089 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.089 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:15.089 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712020+0000", 2024-09-11T09:51:15.089 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.3830000000000001e-06 2024-09-11T09:51:15.089 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.089 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.089 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:15.089 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712023+0000", 2024-09-11T09:51:15.089 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.3340000000000002e-06 2024-09-11T09:51:15.089 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.089 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.090 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:15.090 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720083+0000", 2024-09-11T09:51:15.090 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0080601189999999993 2024-09-11T09:51:15.090 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.090 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.090 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:15.090 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720125+0000", 2024-09-11T09:51:15.090 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.2029e-05 2024-09-11T09:51:15.090 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.090 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.090 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:15.090 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720205+0000", 2024-09-11T09:51:15.090 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.0010000000000001e-05 2024-09-11T09:51:15.090 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.091 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.091 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:15.091 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720849+0000", 2024-09-11T09:51:15.091 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00064369899999999996 2024-09-11T09:51:15.091 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.091 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.091 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:15.091 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.725357+0000", 2024-09-11T09:51:15.091 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0045077379999999998 2024-09-11T09:51:15.091 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.091 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.091 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:15.091 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.725392+0000", 2024-09-11T09:51:15.091 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.5862999999999999e-05 2024-09-11T09:51:15.092 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.092 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.092 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:15.092 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.725435+0000", 2024-09-11T09:51:15.092 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.2311999999999997e-05 2024-09-11T09:51:15.092 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:15.092 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:15.092 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:15.092 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.092 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.092 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:119 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 131072~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:51:15.092 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.725673+0000", 2024-09-11T09:51:15.092 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 63.296447379999996, 2024-09-11T09:51:15.093 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0053927460000000003, 2024-09-11T09:51:15.093 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:15.093 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:15.093 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:15.093 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:15.093 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:51:15.093 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:51:15.093 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 119 2024-09-11T09:51:15.093 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.093 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:15.093 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.093 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:15.093 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.725673+0000", 2024-09-11T09:51:15.093 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:15.094 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.094 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.094 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:15.094 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.725673+0000", 2024-09-11T09:51:15.094 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:15.094 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.094 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.094 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:15.094 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.725673+0000", 2024-09-11T09:51:15.094 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.68e-07 2024-09-11T09:51:15.094 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.094 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.094 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:15.095 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.725720+0000", 2024-09-11T09:51:15.095 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.6526999999999997e-05 2024-09-11T09:51:15.095 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.095 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.095 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:15.095 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.725734+0000", 2024-09-11T09:51:15.095 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.4593000000000001e-05 2024-09-11T09:51:15.095 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.095 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.095 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:15.095 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.725741+0000", 2024-09-11T09:51:15.095 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.28e-06 2024-09-11T09:51:15.095 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.095 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.096 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:15.096 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.726290+0000", 2024-09-11T09:51:15.096 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00054892000000000005 2024-09-11T09:51:15.096 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.096 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.096 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:15.096 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.726338+0000", 2024-09-11T09:51:15.096 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.7982000000000002e-05 2024-09-11T09:51:15.096 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.096 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.096 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:15.096 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.726413+0000", 2024-09-11T09:51:15.096 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.5679000000000005e-05 2024-09-11T09:51:15.097 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.097 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.097 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:15.097 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.727555+0000", 2024-09-11T09:51:15.097 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0011420289999999999 2024-09-11T09:51:15.097 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.097 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.097 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:15.097 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.731004+0000", 2024-09-11T09:51:15.097 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0034484279999999999 2024-09-11T09:51:15.097 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.097 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.097 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:15.098 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.731040+0000", 2024-09-11T09:51:15.098 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.6315e-05 2024-09-11T09:51:15.098 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.098 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.098 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:15.098 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.731065+0000", 2024-09-11T09:51:15.098 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.5324999999999999e-05 2024-09-11T09:51:15.098 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:15.098 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:15.098 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:15.099 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.099 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.099 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:120 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 196608~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:51:15.099 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.725756+0000", 2024-09-11T09:51:15.099 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 63.296363470999999, 2024-09-11T09:51:15.099 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0053569100000000003, 2024-09-11T09:51:15.099 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:15.099 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:15.099 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:15.099 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:15.099 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:51:15.099 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:51:15.099 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 120 2024-09-11T09:51:15.099 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.100 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:15.100 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.100 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:15.100 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.725756+0000", 2024-09-11T09:51:15.100 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:15.100 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.100 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.100 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:15.100 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.725756+0000", 2024-09-11T09:51:15.100 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:15.100 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.100 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.100 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:15.101 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.725757+0000", 2024-09-11T09:51:15.101 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.3799999999999997e-07 2024-09-11T09:51:15.101 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.101 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.101 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:15.101 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.725802+0000", 2024-09-11T09:51:15.101 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.4790000000000003e-05 2024-09-11T09:51:15.101 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.101 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.101 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:15.101 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.725815+0000", 2024-09-11T09:51:15.101 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.2609000000000001e-05 2024-09-11T09:51:15.101 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.101 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.102 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:15.102 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.725822+0000", 2024-09-11T09:51:15.102 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.6450000000000005e-06 2024-09-11T09:51:15.102 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.102 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.102 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:15.102 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.726708+0000", 2024-09-11T09:51:15.102 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00088606500000000001 2024-09-11T09:51:15.102 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.102 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.102 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:15.102 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.726755+0000", 2024-09-11T09:51:15.102 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.6888000000000002e-05 2024-09-11T09:51:15.103 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.103 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.103 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:15.103 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.726828+0000", 2024-09-11T09:51:15.103 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.2510999999999997e-05 2024-09-11T09:51:15.103 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.103 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.103 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:15.103 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.727829+0000", 2024-09-11T09:51:15.103 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001001093 2024-09-11T09:51:15.103 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.103 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.103 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:15.103 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.731071+0000", 2024-09-11T09:51:15.104 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0032426009999999999 2024-09-11T09:51:15.104 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.104 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.104 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:15.104 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.731094+0000", 2024-09-11T09:51:15.104 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.2677e-05 2024-09-11T09:51:15.104 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.104 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.104 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:15.104 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.731113+0000", 2024-09-11T09:51:15.104 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.9392999999999999e-05 2024-09-11T09:51:15.104 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:15.104 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:15.105 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:15.105 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.105 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.105 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:138 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:51:15.105 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:12.034038+0000", 2024-09-11T09:51:15.105 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 62.988082141, 2024-09-11T09:51:15.105 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0054142749999999996, 2024-09-11T09:51:15.105 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:15.105 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:15.105 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:15.105 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:15.105 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:51:15.105 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:51:15.105 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 138 2024-09-11T09:51:15.106 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.106 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:15.106 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.106 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:15.106 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034038+0000", 2024-09-11T09:51:15.106 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:15.106 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.106 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.106 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:15.106 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034038+0000", 2024-09-11T09:51:15.106 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:15.106 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.106 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.106 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:15.107 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034038+0000", 2024-09-11T09:51:15.107 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.3499999999999996e-07 2024-09-11T09:51:15.107 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.107 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.107 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:15.107 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034043+0000", 2024-09-11T09:51:15.107 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.7310000000000001e-06 2024-09-11T09:51:15.107 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.107 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.107 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:15.107 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034050+0000", 2024-09-11T09:51:15.107 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.0419999999999997e-06 2024-09-11T09:51:15.107 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.107 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.107 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:15.108 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034055+0000", 2024-09-11T09:51:15.108 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.3299999999999998e-06 2024-09-11T09:51:15.108 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.108 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.108 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:15.108 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.037309+0000", 2024-09-11T09:51:15.108 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0032535440000000001 2024-09-11T09:51:15.108 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.108 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.108 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:15.108 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.037357+0000", 2024-09-11T09:51:15.108 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.8167999999999997e-05 2024-09-11T09:51:15.108 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.109 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.109 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:15.109 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.037430+0000", 2024-09-11T09:51:15.109 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.3275999999999994e-05 2024-09-11T09:51:15.109 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.109 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.109 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:15.109 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.039014+0000", 2024-09-11T09:51:15.109 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0015831350000000001 2024-09-11T09:51:15.109 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.109 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.109 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:15.109 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.039401+0000", 2024-09-11T09:51:15.109 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00038758500000000002 2024-09-11T09:51:15.110 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.110 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.110 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:15.110 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.039429+0000", 2024-09-11T09:51:15.110 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.7379e-05 2024-09-11T09:51:15.110 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.110 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.110 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:15.110 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.039452+0000", 2024-09-11T09:51:15.110 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.355e-05 2024-09-11T09:51:15.110 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:15.110 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:15.110 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:15.110 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.111 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.111 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:139 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:51:15.111 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:12.034064+0000", 2024-09-11T09:51:15.111 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 62.988056110000002, 2024-09-11T09:51:15.111 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0054736200000000002, 2024-09-11T09:51:15.111 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:15.111 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:15.111 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:15.111 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:15.111 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:51:15.111 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:51:15.111 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 139 2024-09-11T09:51:15.111 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.111 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:15.112 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.112 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:15.112 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034064+0000", 2024-09-11T09:51:15.112 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:15.112 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.112 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.112 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:15.112 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034064+0000", 2024-09-11T09:51:15.112 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:15.112 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.112 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.112 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:15.112 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034064+0000", 2024-09-11T09:51:15.112 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.89e-07 2024-09-11T09:51:15.113 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.113 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.113 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:15.113 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034069+0000", 2024-09-11T09:51:15.113 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.9799999999999998e-06 2024-09-11T09:51:15.113 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.113 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.113 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:15.113 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034079+0000", 2024-09-11T09:51:15.113 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.3549999999999994e-06 2024-09-11T09:51:15.113 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.113 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.113 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:15.114 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034084+0000", 2024-09-11T09:51:15.114 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.2209999999999996e-06 2024-09-11T09:51:15.114 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.114 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.114 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:15.114 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.038176+0000", 2024-09-11T09:51:15.114 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0040921610000000004 2024-09-11T09:51:15.114 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.114 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.114 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:15.114 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.038221+0000", 2024-09-11T09:51:15.114 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.5006000000000002e-05 2024-09-11T09:51:15.114 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.114 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.115 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:15.115 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.038293+0000", 2024-09-11T09:51:15.115 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.1952999999999994e-05 2024-09-11T09:51:15.115 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.115 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.115 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:15.115 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.039461+0000", 2024-09-11T09:51:15.115 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0011680340000000001 2024-09-11T09:51:15.115 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.115 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.115 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:15.115 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.039487+0000", 2024-09-11T09:51:15.115 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.5752e-05 2024-09-11T09:51:15.115 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.116 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.116 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:15.116 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.039515+0000", 2024-09-11T09:51:15.116 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.7763000000000002e-05 2024-09-11T09:51:15.116 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.116 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.116 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:15.116 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.039537+0000", 2024-09-11T09:51:15.116 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.2906000000000001e-05 2024-09-11T09:51:15.116 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:15.116 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:15.116 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:15.116 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.117 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.117 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:140 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:51:15.117 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:12.034093+0000", 2024-09-11T09:51:15.117 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 62.988026605999998, 2024-09-11T09:51:15.117 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0059112490000000004, 2024-09-11T09:51:15.117 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:15.117 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:15.117 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:15.117 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:15.117 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:51:15.117 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:51:15.117 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 140 2024-09-11T09:51:15.117 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.117 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:15.118 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.118 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:15.118 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034093+0000", 2024-09-11T09:51:15.118 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:15.118 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.118 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.118 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:15.118 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034093+0000", 2024-09-11T09:51:15.118 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:15.118 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.118 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.118 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:15.118 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034094+0000", 2024-09-11T09:51:15.118 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.4299999999999998e-07 2024-09-11T09:51:15.119 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.119 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.119 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:15.119 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034159+0000", 2024-09-11T09:51:15.119 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.567e-05 2024-09-11T09:51:15.119 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.119 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.119 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:15.119 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034167+0000", 2024-09-11T09:51:15.119 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.8490000000000006e-06 2024-09-11T09:51:15.119 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.119 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.119 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:15.119 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034173+0000", 2024-09-11T09:51:15.120 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.1770000000000003e-06 2024-09-11T09:51:15.120 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.120 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.120 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:15.120 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.038543+0000", 2024-09-11T09:51:15.120 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0043699150000000003 2024-09-11T09:51:15.120 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.120 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.120 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:15.120 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.038589+0000", 2024-09-11T09:51:15.120 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.5562999999999997e-05 2024-09-11T09:51:15.120 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.120 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.120 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:15.121 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.038662+0000", 2024-09-11T09:51:15.121 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.2748000000000002e-05 2024-09-11T09:51:15.121 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.121 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.121 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:15.121 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.039635+0000", 2024-09-11T09:51:15.121 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00097310799999999998 2024-09-11T09:51:15.121 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.121 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.121 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:15.121 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.039945+0000", 2024-09-11T09:51:15.121 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00031054100000000001 2024-09-11T09:51:15.121 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.121 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.122 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:15.122 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.039971+0000", 2024-09-11T09:51:15.122 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.5423999999999999e-05 2024-09-11T09:51:15.122 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.122 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.122 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:15.122 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.040005+0000", 2024-09-11T09:51:15.122 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.3810999999999999e-05 2024-09-11T09:51:15.122 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:15.122 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:15.122 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:15.122 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.122 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.122 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:51:15.123 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:12.034184+0000", 2024-09-11T09:51:15.123 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 62.987935907999997, 2024-09-11T09:51:15.123 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0061520380000000003, 2024-09-11T09:51:15.123 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:15.123 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:15.123 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:15.123 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:15.123 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:51:15.123 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:51:15.123 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 141 2024-09-11T09:51:15.123 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.123 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:15.123 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.123 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:15.124 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034184+0000", 2024-09-11T09:51:15.124 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:15.124 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.124 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.124 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:15.124 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034184+0000", 2024-09-11T09:51:15.124 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:15.124 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.124 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.124 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:15.124 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034185+0000", 2024-09-11T09:51:15.124 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.1099999999999995e-07 2024-09-11T09:51:15.124 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.125 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.125 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:15.125 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034190+0000", 2024-09-11T09:51:15.125 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.7309999999999998e-06 2024-09-11T09:51:15.125 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.125 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.125 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:15.125 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034197+0000", 2024-09-11T09:51:15.125 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.7440000000000001e-06 2024-09-11T09:51:15.125 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.125 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.125 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:15.125 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034204+0000", 2024-09-11T09:51:15.125 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.8970000000000002e-06 2024-09-11T09:51:15.126 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.126 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.126 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:15.126 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.039035+0000", 2024-09-11T09:51:15.126 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0048307039999999999 2024-09-11T09:51:15.126 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.126 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.126 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:15.126 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.039093+0000", 2024-09-11T09:51:15.126 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.8604000000000003e-05 2024-09-11T09:51:15.126 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.126 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.126 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:15.126 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.039184+0000", 2024-09-11T09:51:15.127 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.0196000000000002e-05 2024-09-11T09:51:15.127 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.127 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.127 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:15.127 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.039873+0000", 2024-09-11T09:51:15.127 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00068960600000000005 2024-09-11T09:51:15.127 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.127 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.127 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:15.127 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.040292+0000", 2024-09-11T09:51:15.127 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00041865199999999999 2024-09-11T09:51:15.127 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.127 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.127 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:15.128 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.040317+0000", 2024-09-11T09:51:15.128 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.5439000000000002e-05 2024-09-11T09:51:15.128 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.128 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.128 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:15.128 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.040336+0000", 2024-09-11T09:51:15.128 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.8754e-05 2024-09-11T09:51:15.128 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:15.128 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:15.128 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:15.128 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.128 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.128 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:157 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:51:15.129 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:12.452040+0000", 2024-09-11T09:51:15.129 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 62.570080054999998, 2024-09-11T09:51:15.129 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0068848269999999996, 2024-09-11T09:51:15.129 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:15.129 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:15.129 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:15.129 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:15.129 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:51:15.129 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:51:15.129 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 157 2024-09-11T09:51:15.129 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.129 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:15.129 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.129 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:15.130 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452040+0000", 2024-09-11T09:51:15.130 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:15.130 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.130 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.130 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:15.130 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452040+0000", 2024-09-11T09:51:15.130 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:15.130 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.130 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.130 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:15.130 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452040+0000", 2024-09-11T09:51:15.130 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.22e-07 2024-09-11T09:51:15.130 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.131 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.131 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:15.131 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452042+0000", 2024-09-11T09:51:15.131 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.7010000000000001e-06 2024-09-11T09:51:15.131 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.131 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.131 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:15.131 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452045+0000", 2024-09-11T09:51:15.131 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.1310000000000002e-06 2024-09-11T09:51:15.131 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.131 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.131 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:15.131 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452048+0000", 2024-09-11T09:51:15.132 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.65e-06 2024-09-11T09:51:15.132 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.132 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.132 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:15.132 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.454453+0000", 2024-09-11T09:51:15.132 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0024052959999999999 2024-09-11T09:51:15.132 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.132 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.132 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:15.132 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.454498+0000", 2024-09-11T09:51:15.132 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.4978999999999997e-05 2024-09-11T09:51:15.132 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.132 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.133 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:15.133 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.454567+0000", 2024-09-11T09:51:15.133 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.9110999999999996e-05 2024-09-11T09:51:15.133 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.133 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.133 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:15.133 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.455464+0000", 2024-09-11T09:51:15.133 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00089696199999999995 2024-09-11T09:51:15.133 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.133 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.133 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:15.133 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.458844+0000", 2024-09-11T09:51:15.133 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0033802469999999999 2024-09-11T09:51:15.133 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.134 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.134 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:15.134 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.458889+0000", 2024-09-11T09:51:15.134 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.4441999999999999e-05 2024-09-11T09:51:15.134 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.134 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.134 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:15.134 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.458925+0000", 2024-09-11T09:51:15.134 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.6086e-05 2024-09-11T09:51:15.134 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:15.134 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:15.134 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:15.134 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.134 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.135 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:158 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:51:15.135 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:12.452055+0000", 2024-09-11T09:51:15.135 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 62.570064662, 2024-09-11T09:51:15.135 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0069533829999999996, 2024-09-11T09:51:15.135 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:15.135 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:15.135 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:15.135 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:15.135 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:51:15.135 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:51:15.135 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 158 2024-09-11T09:51:15.135 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.135 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:15.135 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.136 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:15.136 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452055+0000", 2024-09-11T09:51:15.136 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:15.136 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.136 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.136 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:15.136 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452055+0000", 2024-09-11T09:51:15.136 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:15.136 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.136 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.136 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:15.136 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452056+0000", 2024-09-11T09:51:15.136 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.48e-07 2024-09-11T09:51:15.137 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.137 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.137 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:15.137 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452087+0000", 2024-09-11T09:51:15.137 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.1149000000000003e-05 2024-09-11T09:51:15.137 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.137 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.137 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:15.137 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452092+0000", 2024-09-11T09:51:15.137 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.8400000000000002e-06 2024-09-11T09:51:15.137 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.137 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.137 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:15.137 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452095+0000", 2024-09-11T09:51:15.138 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.0740000000000001e-06 2024-09-11T09:51:15.138 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.138 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.138 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:15.138 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.454796+0000", 2024-09-11T09:51:15.138 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0027009740000000001 2024-09-11T09:51:15.138 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.138 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.138 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:15.138 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.454869+0000", 2024-09-11T09:51:15.138 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.3678999999999997e-05 2024-09-11T09:51:15.138 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.138 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.139 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:15.139 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.454992+0000", 2024-09-11T09:51:15.139 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00012279000000000001 2024-09-11T09:51:15.139 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.139 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.139 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:15.139 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.456072+0000", 2024-09-11T09:51:15.139 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001080229 2024-09-11T09:51:15.139 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.139 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.139 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:15.139 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.458931+0000", 2024-09-11T09:51:15.139 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0028591789999999999 2024-09-11T09:51:15.139 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.140 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.140 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:15.140 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.458972+0000", 2024-09-11T09:51:15.140 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.0170999999999999e-05 2024-09-11T09:51:15.140 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.140 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.140 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:15.140 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.459009+0000", 2024-09-11T09:51:15.140 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.7049999999999999e-05 2024-09-11T09:51:15.140 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:15.140 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:15.140 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:15.140 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.140 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.141 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:159 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:51:15.141 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:12.452099+0000", 2024-09-11T09:51:15.141 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 62.570020493999998, 2024-09-11T09:51:15.141 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0069885950000000002, 2024-09-11T09:51:15.141 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:15.141 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:15.141 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:15.141 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:15.141 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:51:15.141 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:51:15.141 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 159 2024-09-11T09:51:15.141 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.141 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:15.142 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.142 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:15.142 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452099+0000", 2024-09-11T09:51:15.142 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:15.142 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.142 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.142 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:15.142 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452099+0000", 2024-09-11T09:51:15.142 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:15.142 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.142 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.142 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:15.142 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452100+0000", 2024-09-11T09:51:15.142 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.2800000000000003e-07 2024-09-11T09:51:15.143 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.143 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.143 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:15.143 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452102+0000", 2024-09-11T09:51:15.143 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.824e-06 2024-09-11T09:51:15.143 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.143 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.143 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:15.143 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452105+0000", 2024-09-11T09:51:15.143 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.0460000000000001e-06 2024-09-11T09:51:15.143 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.143 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.143 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:15.143 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452107+0000", 2024-09-11T09:51:15.144 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.543e-06 2024-09-11T09:51:15.144 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.144 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.144 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:15.144 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.455484+0000", 2024-09-11T09:51:15.144 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0033769759999999998 2024-09-11T09:51:15.144 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.144 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.144 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:15.144 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.455527+0000", 2024-09-11T09:51:15.144 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.2589999999999997e-05 2024-09-11T09:51:15.144 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.144 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.145 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:15.145 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.455594+0000", 2024-09-11T09:51:15.145 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.6775999999999999e-05 2024-09-11T09:51:15.145 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.145 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.145 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:15.145 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.456312+0000", 2024-09-11T09:51:15.145 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00071874700000000005 2024-09-11T09:51:15.145 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.145 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.145 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:15.145 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.459015+0000", 2024-09-11T09:51:15.145 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0027028540000000002 2024-09-11T09:51:15.145 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.146 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.146 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:15.146 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.459054+0000", 2024-09-11T09:51:15.146 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.8840000000000001e-05 2024-09-11T09:51:15.146 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:15.146 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:15.146 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:15.146 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.459088+0000", 2024-09-11T09:51:15.146 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.4071e-05 2024-09-11T09:51:15.146 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:15.146 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:15.146 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:15.146 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:15.147 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:15.147 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-11T09:51:15.148 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-11T09:51:15.149 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-09-11T09:51:15.149+0000 7f7b6c7d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31904) UID: 0 2024-09-11T09:51:15.249 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:51:15.250+0000 7f3c96ec9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31909) UID: 0 2024-09-11T09:51:15.292 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-11T09:51:15.293 INFO:teuthology.orchestra.run.smithi096.stdout: "ops": [], 2024-09-11T09:51:15.293 INFO:teuthology.orchestra.run.smithi096.stdout: "num_ops": 0 2024-09-11T09:51:15.293 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-11T09:51:15.301 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-11T09:51:15.350 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:51:15.351+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:51:15.444 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-11T09:51:15.445 INFO:teuthology.orchestra.run.smithi096.stdout: "ops": [], 2024-09-11T09:51:15.445 INFO:teuthology.orchestra.run.smithi096.stdout: "complaint_time": 30, 2024-09-11T09:51:15.445 INFO:teuthology.orchestra.run.smithi096.stdout: "num_blocked_ops": 0 2024-09-11T09:51:15.445 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-11T09:51:15.451 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:51:15.452+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:51:15.453 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-11T09:51:15.552 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:51:15.553+0000 7fdec5849640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31907) UID: 0 2024-09-11T09:51:15.556 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-11T09:51:15.556 INFO:teuthology.orchestra.run.smithi096.stdout: "size": 20, 2024-09-11T09:51:15.556 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 600, 2024-09-11T09:51:15.556 INFO:teuthology.orchestra.run.smithi096.stdout: "ops": [] 2024-09-11T09:51:15.556 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-11T09:51:15.565 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-11T09:51:15.653 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-09-11T09:51:15.653+0000 7f7b6c7d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31904) UID: 0 2024-09-11T09:51:15.671 INFO:teuthology.orchestra.run.smithi017.stderr:reweighted osd.3 to 1 (10000) 2024-09-11T09:51:15.674 INFO:teuthology.orchestra.run.smithi017.stderr:noscrub is set 2024-09-11T09:51:15.690 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 4 1 2024-09-11T09:51:15.710 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-11T09:51:15.710 INFO:teuthology.orchestra.run.smithi096.stdout: "ops": [], 2024-09-11T09:51:15.710 INFO:teuthology.orchestra.run.smithi096.stdout: "num_ops": 0 2024-09-11T09:51:15.711 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-11T09:51:15.719 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-11T09:51:15.760 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:51:15.761+0000 7f3c96ec9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31909) UID: 0 2024-09-11T09:51:15.814 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T09:51:15.854 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:51:15.855+0000 7fdec5849640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31907) UID: 0 2024-09-11T09:51:15.862 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-11T09:51:15.862 INFO:teuthology.orchestra.run.smithi096.stdout: "ops": [], 2024-09-11T09:51:15.862 INFO:teuthology.orchestra.run.smithi096.stdout: "complaint_time": 30, 2024-09-11T09:51:15.862 INFO:teuthology.orchestra.run.smithi096.stdout: "num_blocked_ops": 0 2024-09-11T09:51:15.862 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-11T09:51:15.871 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-11T09:51:15.946 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-11T09:51:15.946 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T09:51:15.954 INFO:tasks.ceph.osd.2.smithi017.stderr:2024-09-11T09:51:15.955+0000 7fd20b170640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31672) UID: 0 2024-09-11T09:51:15.963 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-11T09:51:15.963 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T09:51:15.979 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-11T09:51:15.979 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T09:51:15.997 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-11T09:51:15.997 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T09:51:16.014 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-11T09:51:16.014 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T09:51:16.017 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-11T09:51:16.017 INFO:teuthology.orchestra.run.smithi096.stdout: "size": 20, 2024-09-11T09:51:16.017 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 600, 2024-09-11T09:51:16.017 INFO:teuthology.orchestra.run.smithi096.stdout: "ops": [ 2024-09-11T09:51:16.017 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:16.017 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4230.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-11T09:51:16.017 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:18.490736+0000", 2024-09-11T09:51:16.017 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 57.527267043000002, 2024-09-11T09:51:16.017 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0059097569999999999, 2024-09-11T09:51:16.017 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:16.018 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:16.018 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:16.018 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:16.018 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4230", 2024-09-11T09:51:16.018 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.17:0/4101109576", 2024-09-11T09:51:16.018 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 1 2024-09-11T09:51:16.018 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:16.018 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:16.018 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:16.018 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:16.018 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.490736+0000", 2024-09-11T09:51:16.018 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:16.019 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:16.019 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:16.019 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:16.019 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.490736+0000", 2024-09-11T09:51:16.019 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:16.019 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:16.019 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:16.019 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:16.019 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.490742+0000", 2024-09-11T09:51:16.019 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.48e-06 2024-09-11T09:51:16.019 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:16.019 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:16.019 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:16.020 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.490749+0000", 2024-09-11T09:51:16.020 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.4569999999999999e-06 2024-09-11T09:51:16.020 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:16.020 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:16.020 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:16.020 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.490762+0000", 2024-09-11T09:51:16.020 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.2721000000000001e-05 2024-09-11T09:51:16.020 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:16.020 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:16.020 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:16.020 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.490774+0000", 2024-09-11T09:51:16.020 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1823e-05 2024-09-11T09:51:16.020 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:16.021 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:16.021 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:16.021 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.490846+0000", 2024-09-11T09:51:16.021 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.1878000000000005e-05 2024-09-11T09:51:16.021 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:16.021 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:16.021 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for peered", 2024-09-11T09:51:16.021 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.490857+0000", 2024-09-11T09:51:16.021 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1017e-05 2024-09-11T09:51:16.021 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:16.021 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:16.021 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:16.021 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.493856+0000", 2024-09-11T09:51:16.022 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00299885 2024-09-11T09:51:16.022 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:16.022 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:16.022 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:16.022 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.494020+0000", 2024-09-11T09:51:16.022 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00016436599999999999 2024-09-11T09:51:16.022 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:16.022 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:16.022 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 2", 2024-09-11T09:51:16.022 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.494186+0000", 2024-09-11T09:51:16.022 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00016562799999999999 2024-09-11T09:51:16.022 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:16.022 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:16.023 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:16.023 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.495407+0000", 2024-09-11T09:51:16.023 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0012217420000000001 2024-09-11T09:51:16.023 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:16.023 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:16.023 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:16.023 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.496579+0000", 2024-09-11T09:51:16.023 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001171844 2024-09-11T09:51:16.023 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:16.023 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:16.023 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:16.023 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.496620+0000", 2024-09-11T09:51:16.023 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.0479000000000003e-05 2024-09-11T09:51:16.024 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:16.024 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:16.024 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:16.024 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.496646+0000", 2024-09-11T09:51:16.024 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.6472e-05 2024-09-11T09:51:16.024 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:16.024 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:16.024 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:16.024 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:16.024 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:16.024 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_repop_reply(client.4230.0:1 2.3 e16/15 ondisk, result = 0)", 2024-09-11T09:51:16.024 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:18.495321+0000", 2024-09-11T09:51:16.024 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 57.522682377999999, 2024-09-11T09:51:16.025 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.4965999999999998e-05, 2024-09-11T09:51:16.025 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:16.025 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:16.025 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "started", 2024-09-11T09:51:16.025 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:16.025 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:16.025 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:16.025 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.495321+0000", 2024-09-11T09:51:16.025 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:16.025 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:16.025 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:16.025 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:16.025 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.495321+0000", 2024-09-11T09:51:16.025 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:16.026 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:16.026 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:16.026 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:16.026 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.495322+0000", 2024-09-11T09:51:16.026 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.7700000000000004e-07 2024-09-11T09:51:16.026 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:16.026 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:16.026 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:16.026 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.495325+0000", 2024-09-11T09:51:16.026 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.7780000000000001e-06 2024-09-11T09:51:16.026 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:16.026 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:16.026 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:16.027 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.495338+0000", 2024-09-11T09:51:16.027 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.2315e-05 2024-09-11T09:51:16.027 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:16.027 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:16.027 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:16.027 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.495345+0000", 2024-09-11T09:51:16.027 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.8889999999999996e-06 2024-09-11T09:51:16.027 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:16.027 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:16.027 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:16.027 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.495392+0000", 2024-09-11T09:51:16.027 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.7524000000000001e-05 2024-09-11T09:51:16.027 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:16.028 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:16.028 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:16.028 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.495402+0000", 2024-09-11T09:51:16.028 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.5840000000000001e-06 2024-09-11T09:51:16.028 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:16.028 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:16.028 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:16.028 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.495416+0000", 2024-09-11T09:51:16.028 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.4199000000000001e-05 2024-09-11T09:51:16.028 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:16.028 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:16.028 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:16.028 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:16.029 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:16.029 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4230.0:2 2.2 2:5cea7035:::rbd_info:head [call rbd.metadata_list in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e16)", 2024-09-11T09:51:16.029 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:18.499576+0000", 2024-09-11T09:51:16.029 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 57.518427871, 2024-09-11T09:51:16.029 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00018163000000000001, 2024-09-11T09:51:16.029 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:16.029 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:16.029 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "reached pg", 2024-09-11T09:51:16.029 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:16.029 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4230", 2024-09-11T09:51:16.029 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.17:0/4101109576", 2024-09-11T09:51:16.029 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 2 2024-09-11T09:51:16.029 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:16.030 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:16.030 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:16.030 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:16.030 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.499576+0000", 2024-09-11T09:51:16.030 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:16.030 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:16.030 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:16.030 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:16.030 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.499576+0000", 2024-09-11T09:51:16.030 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:16.030 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:16.030 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:16.030 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:16.031 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.499577+0000", 2024-09-11T09:51:16.031 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.124e-06 2024-09-11T09:51:16.031 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:16.031 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:16.031 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:16.031 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.499584+0000", 2024-09-11T09:51:16.031 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.1949999999999997e-06 2024-09-11T09:51:16.031 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:16.031 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:16.031 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:16.031 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.499597+0000", 2024-09-11T09:51:16.031 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.366e-05 2024-09-11T09:51:16.031 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:16.031 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:16.032 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:16.032 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.499604+0000", 2024-09-11T09:51:16.032 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.2709999999999998e-06 2024-09-11T09:51:16.032 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:16.032 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:16.032 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:16.032 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.499660+0000", 2024-09-11T09:51:16.032 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.6270999999999997e-05 2024-09-11T09:51:16.032 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:16.032 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:16.032 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:16.032 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.499757+0000", 2024-09-11T09:51:16.032 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.7108999999999995e-05 2024-09-11T09:51:16.033 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:16.033 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:16.033 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:16.033 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:16.033 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:16.033 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4230.0:3 2.2 2:5cea7035:::rbd_info:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e16)", 2024-09-11T09:51:16.033 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:18.499937+0000", 2024-09-11T09:51:16.033 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 57.518066081999997, 2024-09-11T09:51:16.033 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000143318, 2024-09-11T09:51:16.033 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:16.033 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:16.033 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "reached pg", 2024-09-11T09:51:16.033 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:16.034 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4230", 2024-09-11T09:51:16.034 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.17:0/4101109576", 2024-09-11T09:51:16.034 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 3 2024-09-11T09:51:16.034 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:16.034 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:16.034 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:16.034 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:16.034 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.499937+0000", 2024-09-11T09:51:16.034 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:16.034 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:16.034 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:16.034 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:16.034 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.499937+0000", 2024-09-11T09:51:16.035 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:16.035 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:16.035 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:16.035 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:16.035 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.499938+0000", 2024-09-11T09:51:16.035 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.195e-06 2024-09-11T09:51:16.035 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:16.035 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:16.035 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:16.035 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.499943+0000", 2024-09-11T09:51:16.035 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.9080000000000003e-06 2024-09-11T09:51:16.035 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:16.036 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:16.036 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:16.036 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.499956+0000", 2024-09-11T09:51:16.036 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.2728000000000001e-05 2024-09-11T09:51:16.036 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:16.036 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:16.036 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:16.036 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.499964+0000", 2024-09-11T09:51:16.036 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.8220000000000007e-06 2024-09-11T09:51:16.036 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:16.036 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:16.036 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:16.036 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.500017+0000", 2024-09-11T09:51:16.037 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.2809000000000002e-05 2024-09-11T09:51:16.037 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:16.037 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:16.037 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:16.037 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.500081+0000", 2024-09-11T09:51:16.037 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.3856000000000007e-05 2024-09-11T09:51:16.037 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:16.037 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:16.037 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:16.037 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:16.037 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:16.037 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4230.0:5 2.2 2:5cea7035:::rbd_info:head [create,write 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-11T09:51:16.038 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:19.492662+0000", 2024-09-11T09:51:16.038 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 56.525341421999997, 2024-09-11T09:51:16.038 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0043824759999999997, 2024-09-11T09:51:16.038 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:16.038 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:16.038 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:16.038 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:16.038 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4230", 2024-09-11T09:51:16.038 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.17:0/4101109576", 2024-09-11T09:51:16.038 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 5 2024-09-11T09:51:16.038 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:16.038 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:16.038 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:16.039 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:16.039 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.492662+0000", 2024-09-11T09:51:16.039 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:16.039 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:16.039 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:16.039 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:16.039 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.492662+0000", 2024-09-11T09:51:16.039 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:16.039 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:16.039 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:16.039 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:16.039 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.492664+0000", 2024-09-11T09:51:16.039 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.3599999999999999e-06 2024-09-11T09:51:16.040 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:16.040 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:16.040 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:16.040 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.492675+0000", 2024-09-11T09:51:16.040 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1182000000000001e-05 2024-09-11T09:51:16.040 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:16.040 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:16.040 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:16.040 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.492693+0000", 2024-09-11T09:51:16.040 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.7031999999999999e-05 2024-09-11T09:51:16.040 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:16.040 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:16.040 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:16.041 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.492702+0000", 2024-09-11T09:51:16.041 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.5540000000000003e-06 2024-09-11T09:51:16.041 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:16.041 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:16.041 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:16.041 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.492767+0000", 2024-09-11T09:51:16.041 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.5411000000000001e-05 2024-09-11T09:51:16.041 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:16.041 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:16.041 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op must wait for map", 2024-09-11T09:51:16.041 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.492773+0000", 2024-09-11T09:51:16.041 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.1039999999999998e-06 2024-09-11T09:51:16.042 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:16.042 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:16.042 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:16.042 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.494354+0000", 2024-09-11T09:51:16.042 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0015814049999999999 2024-09-11T09:51:16.042 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:16.042 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:16.042 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:16.042 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.494476+0000", 2024-09-11T09:51:16.042 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000122089 2024-09-11T09:51:16.042 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:16.042 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:16.042 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 1", 2024-09-11T09:51:16.043 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.494577+0000", 2024-09-11T09:51:16.043 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000100536 2024-09-11T09:51:16.043 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:16.043 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:16.043 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:16.043 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.495770+0000", 2024-09-11T09:51:16.043 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0011932469999999999 2024-09-11T09:51:16.043 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:16.043 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:16.043 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:16.043 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.496966+0000", 2024-09-11T09:51:16.043 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001196003 2024-09-11T09:51:16.043 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:16.044 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:16.044 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:16.044 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.497011+0000", 2024-09-11T09:51:16.044 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.5229999999999999e-05 2024-09-11T09:51:16.044 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:16.044 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:16.044 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:16.044 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.497044+0000", 2024-09-11T09:51:16.044 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.3322999999999999e-05 2024-09-11T09:51:16.044 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:16.044 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:16.044 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:16.045 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:16.045 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:16.045 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_repop_reply(client.4230.0:5 2.2 e17/15 ondisk, result = 0)", 2024-09-11T09:51:16.045 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:19.495669+0000", 2024-09-11T09:51:16.045 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 56.522334634000003, 2024-09-11T09:51:16.045 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000106431, 2024-09-11T09:51:16.045 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:16.045 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:16.045 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "started", 2024-09-11T09:51:16.045 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:16.048 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:16.048 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:16.048 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.495669+0000", 2024-09-11T09:51:16.048 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:16.048 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:16.048 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:16.048 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:16.048 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.495669+0000", 2024-09-11T09:51:16.048 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:16.048 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:16.049 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:16.049 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:16.049 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.495670+0000", 2024-09-11T09:51:16.049 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.218e-06 2024-09-11T09:51:16.049 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:16.049 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:16.049 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:16.049 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.495676+0000", 2024-09-11T09:51:16.049 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.5119999999999997e-06 2024-09-11T09:51:16.049 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:16.049 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:16.049 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:16.050 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.495690+0000", 2024-09-11T09:51:16.050 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.3312000000000001e-05 2024-09-11T09:51:16.050 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:16.050 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:16.050 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:16.050 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.495706+0000", 2024-09-11T09:51:16.050 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.6560000000000001e-05 2024-09-11T09:51:16.050 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:16.050 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:16.050 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:16.050 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.495759+0000", 2024-09-11T09:51:16.050 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.2828000000000003e-05 2024-09-11T09:51:16.051 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:16.051 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:16.051 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:16.051 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.495765+0000", 2024-09-11T09:51:16.051 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.7869999999999998e-06 2024-09-11T09:51:16.051 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:16.051 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:16.051 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:16.051 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.495775+0000", 2024-09-11T09:51:16.051 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.0214000000000001e-05 2024-09-11T09:51:16.051 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:16.051 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:16.051 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:16.052 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:16.052 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:16.052 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4230.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-09-11T09:51:16.052 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:20.498634+0000", 2024-09-11T09:51:16.052 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 55.519369021999999, 2024-09-11T09:51:16.052 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0050648539999999997, 2024-09-11T09:51:16.052 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:16.052 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:16.052 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:16.052 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:16.052 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4230", 2024-09-11T09:51:16.052 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.17:0/4101109576", 2024-09-11T09:51:16.053 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 7 2024-09-11T09:51:16.053 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:16.053 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:16.053 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:16.053 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:16.053 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.498634+0000", 2024-09-11T09:51:16.053 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:16.053 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:16.053 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:16.053 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:16.053 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.498634+0000", 2024-09-11T09:51:16.053 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:16.054 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:16.054 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:16.054 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:16.054 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.498637+0000", 2024-09-11T09:51:16.054 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.1919999999999999e-06 2024-09-11T09:51:16.054 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:16.054 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:16.054 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:16.054 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.498648+0000", 2024-09-11T09:51:16.054 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1116e-05 2024-09-11T09:51:16.054 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:16.054 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:16.054 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:16.055 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.498666+0000", 2024-09-11T09:51:16.055 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.8411999999999999e-05 2024-09-11T09:51:16.055 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:16.055 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:16.055 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:16.055 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.498675+0000", 2024-09-11T09:51:16.055 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.4059999999999994e-06 2024-09-11T09:51:16.055 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:16.055 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:16.055 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:16.055 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.498745+0000", 2024-09-11T09:51:16.055 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.9764000000000005e-05 2024-09-11T09:51:16.055 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:16.056 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:16.056 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op must wait for map", 2024-09-11T09:51:16.056 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.498754+0000", 2024-09-11T09:51:16.056 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.5969999999999992e-06 2024-09-11T09:51:16.056 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:16.056 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:16.056 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:16.056 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.500385+0000", 2024-09-11T09:51:16.056 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0016308360000000001 2024-09-11T09:51:16.056 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:16.056 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:16.056 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:16.057 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.500473+0000", 2024-09-11T09:51:16.057 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.7966000000000004e-05 2024-09-11T09:51:16.057 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:16.057 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:16.057 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 1", 2024-09-11T09:51:16.057 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.500591+0000", 2024-09-11T09:51:16.057 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000117895 2024-09-11T09:51:16.057 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:16.057 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:16.057 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:16.057 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.501685+0000", 2024-09-11T09:51:16.057 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001094886 2024-09-11T09:51:16.058 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:16.058 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:16.058 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:16.058 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.503627+0000", 2024-09-11T09:51:16.058 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001941199 2024-09-11T09:51:16.058 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:16.058 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:16.058 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:16.058 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.503667+0000", 2024-09-11T09:51:16.058 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.0645000000000001e-05 2024-09-11T09:51:16.058 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:16.058 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:16.058 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:16.059 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.503699+0000", 2024-09-11T09:51:16.059 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.1940000000000003e-05 2024-09-11T09:51:16.059 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:16.059 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:16.059 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:16.059 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:16.059 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:16.059 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_repop_reply(client.4230.0:7 2.2 e18/15 ondisk, result = 0)", 2024-09-11T09:51:16.059 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:20.501608+0000", 2024-09-11T09:51:16.059 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 55.516395500000002, 2024-09-11T09:51:16.059 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.2459000000000005e-05, 2024-09-11T09:51:16.059 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:16.059 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:16.060 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "started", 2024-09-11T09:51:16.060 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:16.060 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:16.060 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:16.060 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.501608+0000", 2024-09-11T09:51:16.060 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:16.060 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:16.060 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:16.060 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:16.060 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.501608+0000", 2024-09-11T09:51:16.060 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:16.060 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:16.060 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:16.061 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:16.061 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.501609+0000", 2024-09-11T09:51:16.061 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.0160000000000001e-06 2024-09-11T09:51:16.061 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:16.061 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:16.061 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:16.061 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.501613+0000", 2024-09-11T09:51:16.061 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.5959999999999997e-06 2024-09-11T09:51:16.061 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:16.061 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:16.061 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:16.061 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.501625+0000", 2024-09-11T09:51:16.062 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1888e-05 2024-09-11T09:51:16.062 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:16.062 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:16.062 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:16.062 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.501634+0000", 2024-09-11T09:51:16.062 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.6139999999999992e-06 2024-09-11T09:51:16.062 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:16.062 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:16.062 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:16.062 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.501676+0000", 2024-09-11T09:51:16.062 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.1953999999999998e-05 2024-09-11T09:51:16.062 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:16.062 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:16.063 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:16.063 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.501682+0000", 2024-09-11T09:51:16.063 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.8000000000000004e-06 2024-09-11T09:51:16.063 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:16.063 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:16.063 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:16.063 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.501690+0000", 2024-09-11T09:51:16.063 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.5909999999999996e-06 2024-09-11T09:51:16.063 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:16.063 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:16.063 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:16.063 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:16.063 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:16.063 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-11T09:51:16.064 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-11T09:51:16.066 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:51:16.065+0000 7fdec5849640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31907) UID: 0 2024-09-11T09:51:16.066 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-11T09:51:16.066 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T09:51:16.165 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:51:16.166+0000 7f3c96ec9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31909) UID: 0 2024-09-11T09:51:16.214 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:51:16.214 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:51:16.223 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-11T09:51:16.266 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:51:16.267+0000 7fdec5849640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31907) UID: 0 2024-09-11T09:51:16.332 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:51:16.332 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:51:16.341 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-11T09:51:16.385 INFO:tasks.ceph.osd.2.smithi017.stderr:2024-09-11T09:51:16.386+0000 7fd20b170640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31672) UID: 0 2024-09-11T09:51:16.448 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:51:16.448 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:51:16.457 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-11T09:51:16.467 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:51:16.468+0000 7fdec5849640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31907) UID: 0 2024-09-11T09:51:16.569 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-09-11T09:51:16.570+0000 7f7b6c7d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31904) UID: 0 2024-09-11T09:51:16.604 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:51:16.605 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:51:16.613 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-11T09:51:16.670 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:51:16.670+0000 7fdec5849640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31907) UID: 0 2024-09-11T09:51:16.677 INFO:teuthology.orchestra.run.smithi017.stderr:reweighted osd.4 to 1 (10000) 2024-09-11T09:51:16.691 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 5 1 2024-09-11T09:51:16.725 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:51:16.725 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:51:16.734 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-11T09:51:16.775 INFO:tasks.ceph.osd.2.smithi017.stderr:2024-09-11T09:51:16.776+0000 7fd20b170640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31672) UID: 0 2024-09-11T09:51:16.872 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:51:16.872+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:51:16.883 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:51:16.883 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:51:16.892 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-11T09:51:16.977 INFO:tasks.ceph.osd.2.smithi017.stderr:2024-09-11T09:51:16.978+0000 7fd20b170640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31672) UID: 0 2024-09-11T09:51:17.041 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:51:17.041 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:51:17.050 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-11T09:51:17.073 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:51:17.074+0000 7f3c96ec9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31909) UID: 0 2024-09-11T09:51:17.174 INFO:tasks.ceph.osd.0.smithi017.stderr:2024-09-11T09:51:17.175+0000 7fb73c0f1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31665) UID: 0 2024-09-11T09:51:17.198 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:51:17.198 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:51:17.206 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-11T09:51:17.291 INFO:tasks.ceph.osd.2.smithi017.stderr:2024-09-11T09:51:17.292+0000 7fd20b170640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31672) UID: 0 2024-09-11T09:51:17.354 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:51:17.354 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:51:17.363 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-11T09:51:17.404 INFO:tasks.ceph.osd.0.smithi017.stderr:2024-09-11T09:51:17.405+0000 7fb73c0f1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31665) UID: 0 2024-09-11T09:51:17.477 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:51:17.478+0000 7f3c96ec9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31909) UID: 0 2024-09-11T09:51:17.478 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:51:17.478 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:51:17.487 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-11T09:51:17.578 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-09-11T09:51:17.579+0000 7f7b6c7d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31904) UID: 0 2024-09-11T09:51:17.631 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:51:17.632 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:51:17.641 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-11T09:51:17.677 INFO:teuthology.orchestra.run.smithi017.stderr:reweighted osd.5 to 1 (10000) 2024-09-11T09:51:17.679 INFO:tasks.ceph.osd.0.smithi017.stderr:2024-09-11T09:51:17.680+0000 7fb73c0f1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31665) UID: 0 2024-09-11T09:51:17.691 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-11T09:51:17.693 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-09-11T09:51:17.781 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:51:17.781+0000 7f3c96ec9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31909) UID: 0 2024-09-11T09:51:17.783 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:51:17.783 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:51:17.792 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-11T09:51:17.881 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:51:17.882+0000 7fdec5849640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31907) UID: 0 2024-09-11T09:51:17.936 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:51:17.936 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:51:17.945 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-11T09:51:17.982 INFO:tasks.ceph.osd.0.smithi017.stderr:2024-09-11T09:51:17.983+0000 7fb73c0f1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31665) UID: 0 2024-09-11T09:51:18.067 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T09:51:18.088 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:51:18.089 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:51:18.097 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-11T09:51:18.113 INFO:tasks.ceph.osd.2.smithi017.stderr:2024-09-11T09:51:18.114+0000 7fd20b170640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31672) UID: 0 2024-09-11T09:51:18.184 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-09-11T09:51:18.185+0000 7f7b6c7d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31904) UID: 0 2024-09-11T09:51:18.191 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-11T09:51:18.191 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T09:51:18.200 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:51:18.201 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:51:18.208 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-11T09:51:18.208 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T09:51:18.209 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-11T09:51:18.224 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-11T09:51:18.224 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T09:51:18.241 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-11T09:51:18.242 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T09:51:18.259 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-11T09:51:18.259 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T09:51:18.275 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-11T09:51:18.275 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T09:51:18.285 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:51:18.286+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:51:18.351 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-11T09:51:18.351 INFO:teuthology.orchestra.run.smithi096.stdout: "ops": [], 2024-09-11T09:51:18.351 INFO:teuthology.orchestra.run.smithi096.stdout: "num_ops": 0 2024-09-11T09:51:18.351 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-11T09:51:18.360 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-11T09:51:18.403 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:51:18.404+0000 7f3c96ec9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31909) UID: 0 2024-09-11T09:51:18.462 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-11T09:51:18.463 INFO:teuthology.orchestra.run.smithi096.stdout: "ops": [], 2024-09-11T09:51:18.463 INFO:teuthology.orchestra.run.smithi096.stdout: "complaint_time": 30, 2024-09-11T09:51:18.463 INFO:teuthology.orchestra.run.smithi096.stdout: "num_blocked_ops": 0 2024-09-11T09:51:18.463 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-11T09:51:18.471 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-11T09:51:18.487 INFO:tasks.ceph.osd.0.smithi017.stderr:2024-09-11T09:51:18.489+0000 7fb73c0f1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31665) UID: 0 2024-09-11T09:51:18.588 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:51:18.589+0000 7fdec5849640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31907) UID: 0 2024-09-11T09:51:18.614 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-11T09:51:18.615 INFO:teuthology.orchestra.run.smithi096.stdout: "size": 20, 2024-09-11T09:51:18.615 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 600, 2024-09-11T09:51:18.615 INFO:teuthology.orchestra.run.smithi096.stdout: "ops": [ 2024-09-11T09:51:18.615 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:18.615 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4230.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-11T09:51:18.615 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:18.490736+0000", 2024-09-11T09:51:18.615 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 60.125000813, 2024-09-11T09:51:18.615 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0059097569999999999, 2024-09-11T09:51:18.615 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:18.615 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:18.615 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:18.616 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:18.616 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4230", 2024-09-11T09:51:18.616 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.17:0/4101109576", 2024-09-11T09:51:18.616 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 1 2024-09-11T09:51:18.616 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:18.616 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:18.616 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:18.616 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:18.616 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.490736+0000", 2024-09-11T09:51:18.616 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:18.616 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:18.616 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:18.617 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:18.617 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.490736+0000", 2024-09-11T09:51:18.617 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:18.617 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:18.617 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:18.617 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:18.617 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.490742+0000", 2024-09-11T09:51:18.617 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.48e-06 2024-09-11T09:51:18.617 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:18.617 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:18.617 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:18.617 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.490749+0000", 2024-09-11T09:51:18.617 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.4569999999999999e-06 2024-09-11T09:51:18.618 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:18.618 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:18.618 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:18.618 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.490762+0000", 2024-09-11T09:51:18.618 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.2721000000000001e-05 2024-09-11T09:51:18.618 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:18.618 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:18.618 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:18.618 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.490774+0000", 2024-09-11T09:51:18.618 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1823e-05 2024-09-11T09:51:18.618 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:18.618 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:18.618 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:18.619 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.490846+0000", 2024-09-11T09:51:18.619 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.1878000000000005e-05 2024-09-11T09:51:18.619 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:18.619 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:18.619 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for peered", 2024-09-11T09:51:18.619 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.490857+0000", 2024-09-11T09:51:18.619 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1017e-05 2024-09-11T09:51:18.619 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:18.619 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:18.619 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:18.619 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.493856+0000", 2024-09-11T09:51:18.619 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00299885 2024-09-11T09:51:18.619 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:18.620 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:18.620 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:18.620 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.494020+0000", 2024-09-11T09:51:18.620 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00016436599999999999 2024-09-11T09:51:18.620 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:18.620 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:18.620 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 2", 2024-09-11T09:51:18.620 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.494186+0000", 2024-09-11T09:51:18.620 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00016562799999999999 2024-09-11T09:51:18.620 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:18.620 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:18.620 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:18.621 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.495407+0000", 2024-09-11T09:51:18.621 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0012217420000000001 2024-09-11T09:51:18.621 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:18.621 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:18.621 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:18.621 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.496579+0000", 2024-09-11T09:51:18.621 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001171844 2024-09-11T09:51:18.621 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:18.621 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:18.621 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:18.621 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.496620+0000", 2024-09-11T09:51:18.621 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.0479000000000003e-05 2024-09-11T09:51:18.622 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:18.622 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:18.622 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:18.622 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.496646+0000", 2024-09-11T09:51:18.622 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.6472e-05 2024-09-11T09:51:18.622 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:18.622 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:18.622 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:18.622 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:18.622 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:18.622 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_repop_reply(client.4230.0:1 2.3 e16/15 ondisk, result = 0)", 2024-09-11T09:51:18.622 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:18.495321+0000", 2024-09-11T09:51:18.622 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 60.120416147999997, 2024-09-11T09:51:18.623 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.4965999999999998e-05, 2024-09-11T09:51:18.623 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:18.623 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:18.623 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "started", 2024-09-11T09:51:18.623 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:18.623 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:18.623 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:18.623 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.495321+0000", 2024-09-11T09:51:18.623 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:18.623 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:18.623 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:18.623 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:18.624 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.495321+0000", 2024-09-11T09:51:18.624 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:18.624 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:18.624 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:18.624 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:18.624 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.495322+0000", 2024-09-11T09:51:18.624 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.7700000000000004e-07 2024-09-11T09:51:18.624 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:18.624 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:18.624 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:18.624 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.495325+0000", 2024-09-11T09:51:18.624 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.7780000000000001e-06 2024-09-11T09:51:18.624 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:18.625 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:18.625 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:18.625 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.495338+0000", 2024-09-11T09:51:18.625 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.2315e-05 2024-09-11T09:51:18.625 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:18.625 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:18.625 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:18.625 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.495345+0000", 2024-09-11T09:51:18.625 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.8889999999999996e-06 2024-09-11T09:51:18.625 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:18.625 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:18.625 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:18.625 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.495392+0000", 2024-09-11T09:51:18.625 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.7524000000000001e-05 2024-09-11T09:51:18.626 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:18.626 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:18.626 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:18.626 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.495402+0000", 2024-09-11T09:51:18.626 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.5840000000000001e-06 2024-09-11T09:51:18.626 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:18.626 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:18.626 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:18.626 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.495416+0000", 2024-09-11T09:51:18.626 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.4199000000000001e-05 2024-09-11T09:51:18.626 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:18.626 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:18.626 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:18.627 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:18.627 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:18.627 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4230.0:2 2.2 2:5cea7035:::rbd_info:head [call rbd.metadata_list in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e16)", 2024-09-11T09:51:18.627 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:18.499576+0000", 2024-09-11T09:51:18.627 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 60.116161640999998, 2024-09-11T09:51:18.627 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00018163000000000001, 2024-09-11T09:51:18.627 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:18.627 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:18.627 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "reached pg", 2024-09-11T09:51:18.627 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:18.627 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4230", 2024-09-11T09:51:18.627 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.17:0/4101109576", 2024-09-11T09:51:18.627 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 2 2024-09-11T09:51:18.628 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:18.628 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:18.628 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:18.628 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:18.628 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.499576+0000", 2024-09-11T09:51:18.628 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:18.628 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:18.628 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:18.628 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:18.628 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.499576+0000", 2024-09-11T09:51:18.628 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:18.628 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:18.628 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:18.629 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:18.629 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.499577+0000", 2024-09-11T09:51:18.629 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.124e-06 2024-09-11T09:51:18.629 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:18.629 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:18.629 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:18.629 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.499584+0000", 2024-09-11T09:51:18.629 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.1949999999999997e-06 2024-09-11T09:51:18.629 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:18.629 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:18.629 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:18.629 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.499597+0000", 2024-09-11T09:51:18.629 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.366e-05 2024-09-11T09:51:18.630 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:18.630 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:18.630 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:18.630 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.499604+0000", 2024-09-11T09:51:18.630 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.2709999999999998e-06 2024-09-11T09:51:18.630 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:18.630 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:18.630 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:18.630 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.499660+0000", 2024-09-11T09:51:18.630 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.6270999999999997e-05 2024-09-11T09:51:18.630 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:18.630 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:18.630 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:18.631 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.499757+0000", 2024-09-11T09:51:18.631 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.7108999999999995e-05 2024-09-11T09:51:18.631 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:18.631 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:18.631 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:18.631 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:18.631 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:18.631 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4230.0:3 2.2 2:5cea7035:::rbd_info:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e16)", 2024-09-11T09:51:18.631 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:18.499937+0000", 2024-09-11T09:51:18.631 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 60.115799852000002, 2024-09-11T09:51:18.631 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000143318, 2024-09-11T09:51:18.631 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:18.631 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:18.631 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "reached pg", 2024-09-11T09:51:18.632 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:18.632 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4230", 2024-09-11T09:51:18.632 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.17:0/4101109576", 2024-09-11T09:51:18.632 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 3 2024-09-11T09:51:18.632 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:18.632 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:18.632 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:18.632 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:18.632 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.499937+0000", 2024-09-11T09:51:18.632 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:18.632 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:18.632 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:18.632 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:18.633 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.499937+0000", 2024-09-11T09:51:18.633 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:18.633 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:18.633 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:18.633 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:18.633 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.499938+0000", 2024-09-11T09:51:18.633 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.195e-06 2024-09-11T09:51:18.633 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:18.633 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:18.633 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:18.633 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.499943+0000", 2024-09-11T09:51:18.633 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.9080000000000003e-06 2024-09-11T09:51:18.633 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:18.634 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:18.634 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:18.634 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.499956+0000", 2024-09-11T09:51:18.634 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.2728000000000001e-05 2024-09-11T09:51:18.634 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:18.634 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:18.634 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:18.634 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.499964+0000", 2024-09-11T09:51:18.634 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.8220000000000007e-06 2024-09-11T09:51:18.634 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:18.634 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:18.634 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:18.634 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.500017+0000", 2024-09-11T09:51:18.634 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.2809000000000002e-05 2024-09-11T09:51:18.634 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:18.635 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:18.635 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:18.635 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.500081+0000", 2024-09-11T09:51:18.635 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.3856000000000007e-05 2024-09-11T09:51:18.635 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:18.635 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:18.635 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:18.635 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:18.635 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:18.635 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4230.0:5 2.2 2:5cea7035:::rbd_info:head [create,write 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-11T09:51:18.635 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:19.492662+0000", 2024-09-11T09:51:18.635 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 59.123075192000002, 2024-09-11T09:51:18.635 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0043824759999999997, 2024-09-11T09:51:18.635 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:18.636 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:18.636 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:18.636 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:18.636 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4230", 2024-09-11T09:51:18.636 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.17:0/4101109576", 2024-09-11T09:51:18.636 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 5 2024-09-11T09:51:18.636 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:18.636 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:18.636 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:18.636 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:18.636 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.492662+0000", 2024-09-11T09:51:18.636 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:18.636 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:18.636 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:18.637 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:18.637 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.492662+0000", 2024-09-11T09:51:18.637 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:18.637 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:18.637 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:18.637 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:18.637 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.492664+0000", 2024-09-11T09:51:18.637 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.3599999999999999e-06 2024-09-11T09:51:18.637 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:18.637 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:18.637 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:18.637 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.492675+0000", 2024-09-11T09:51:18.637 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1182000000000001e-05 2024-09-11T09:51:18.638 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:18.638 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:18.638 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:18.638 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.492693+0000", 2024-09-11T09:51:18.638 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.7031999999999999e-05 2024-09-11T09:51:18.638 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:18.638 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:18.638 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:18.638 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.492702+0000", 2024-09-11T09:51:18.638 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.5540000000000003e-06 2024-09-11T09:51:18.638 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:18.638 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:18.639 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:18.639 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.492767+0000", 2024-09-11T09:51:18.639 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.5411000000000001e-05 2024-09-11T09:51:18.639 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:18.639 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:18.639 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op must wait for map", 2024-09-11T09:51:18.639 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.492773+0000", 2024-09-11T09:51:18.639 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.1039999999999998e-06 2024-09-11T09:51:18.639 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:18.639 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:18.639 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:18.639 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.494354+0000", 2024-09-11T09:51:18.639 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0015814049999999999 2024-09-11T09:51:18.640 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:18.640 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:18.640 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:18.640 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.494476+0000", 2024-09-11T09:51:18.640 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000122089 2024-09-11T09:51:18.640 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:18.640 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:18.640 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 1", 2024-09-11T09:51:18.640 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.494577+0000", 2024-09-11T09:51:18.640 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000100536 2024-09-11T09:51:18.640 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:18.640 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:18.640 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:18.641 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.495770+0000", 2024-09-11T09:51:18.641 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0011932469999999999 2024-09-11T09:51:18.641 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:18.641 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:18.641 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:18.641 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.496966+0000", 2024-09-11T09:51:18.641 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001196003 2024-09-11T09:51:18.641 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:18.641 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:18.641 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:18.641 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.497011+0000", 2024-09-11T09:51:18.641 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.5229999999999999e-05 2024-09-11T09:51:18.641 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:18.642 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:18.642 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:18.642 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.497044+0000", 2024-09-11T09:51:18.642 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.3322999999999999e-05 2024-09-11T09:51:18.642 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:18.642 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:18.642 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:18.642 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:18.642 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:18.642 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_repop_reply(client.4230.0:5 2.2 e17/15 ondisk, result = 0)", 2024-09-11T09:51:18.642 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:19.495669+0000", 2024-09-11T09:51:18.642 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 59.120068404000001, 2024-09-11T09:51:18.642 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000106431, 2024-09-11T09:51:18.643 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:18.643 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:18.643 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "started", 2024-09-11T09:51:18.643 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:18.643 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:18.643 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:18.643 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.495669+0000", 2024-09-11T09:51:18.643 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:18.643 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:18.643 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:18.643 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:18.643 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.495669+0000", 2024-09-11T09:51:18.643 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:18.644 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:18.644 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:18.644 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:18.644 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.495670+0000", 2024-09-11T09:51:18.644 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.218e-06 2024-09-11T09:51:18.644 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:18.644 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:18.644 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:18.644 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.495676+0000", 2024-09-11T09:51:18.644 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.5119999999999997e-06 2024-09-11T09:51:18.644 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:18.644 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:18.644 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:18.644 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.495690+0000", 2024-09-11T09:51:18.645 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.3312000000000001e-05 2024-09-11T09:51:18.645 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:18.645 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:18.645 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:18.645 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.495706+0000", 2024-09-11T09:51:18.645 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.6560000000000001e-05 2024-09-11T09:51:18.645 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:18.645 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:18.645 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:18.645 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.495759+0000", 2024-09-11T09:51:18.645 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.2828000000000003e-05 2024-09-11T09:51:18.645 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:18.645 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:18.646 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:18.646 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.495765+0000", 2024-09-11T09:51:18.646 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.7869999999999998e-06 2024-09-11T09:51:18.646 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:18.646 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:18.646 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:18.646 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.495775+0000", 2024-09-11T09:51:18.646 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.0214000000000001e-05 2024-09-11T09:51:18.646 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:18.646 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:18.646 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:18.646 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:18.646 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:18.647 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4230.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-09-11T09:51:18.647 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:20.498634+0000", 2024-09-11T09:51:18.647 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 58.117102791999997, 2024-09-11T09:51:18.647 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0050648539999999997, 2024-09-11T09:51:18.647 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:18.647 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:18.647 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:18.647 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:18.647 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4230", 2024-09-11T09:51:18.647 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.17:0/4101109576", 2024-09-11T09:51:18.647 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 7 2024-09-11T09:51:18.647 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:18.647 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:18.647 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:18.648 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:18.648 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.498634+0000", 2024-09-11T09:51:18.648 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:18.648 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:18.648 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:18.648 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:18.648 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.498634+0000", 2024-09-11T09:51:18.648 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:18.648 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:18.648 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:18.648 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:18.648 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.498637+0000", 2024-09-11T09:51:18.648 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.1919999999999999e-06 2024-09-11T09:51:18.649 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:18.649 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:18.649 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:18.649 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.498648+0000", 2024-09-11T09:51:18.649 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1116e-05 2024-09-11T09:51:18.649 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:18.649 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:18.649 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:18.649 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.498666+0000", 2024-09-11T09:51:18.649 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.8411999999999999e-05 2024-09-11T09:51:18.649 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:18.649 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:18.649 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:18.649 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.498675+0000", 2024-09-11T09:51:18.650 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.4059999999999994e-06 2024-09-11T09:51:18.650 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:18.650 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:18.650 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:18.650 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.498745+0000", 2024-09-11T09:51:18.650 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.9764000000000005e-05 2024-09-11T09:51:18.650 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:18.650 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:18.650 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op must wait for map", 2024-09-11T09:51:18.650 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.498754+0000", 2024-09-11T09:51:18.650 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.5969999999999992e-06 2024-09-11T09:51:18.650 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:18.650 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:18.650 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:18.651 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.500385+0000", 2024-09-11T09:51:18.651 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0016308360000000001 2024-09-11T09:51:18.651 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:18.651 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:18.651 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:18.651 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.500473+0000", 2024-09-11T09:51:18.651 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.7966000000000004e-05 2024-09-11T09:51:18.651 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:18.651 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:18.651 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 1", 2024-09-11T09:51:18.651 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.500591+0000", 2024-09-11T09:51:18.651 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000117895 2024-09-11T09:51:18.651 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:18.652 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:18.652 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:18.652 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.501685+0000", 2024-09-11T09:51:18.652 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001094886 2024-09-11T09:51:18.652 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:18.652 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:18.652 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:18.652 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.503627+0000", 2024-09-11T09:51:18.652 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001941199 2024-09-11T09:51:18.652 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:18.652 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:18.652 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:18.653 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.503667+0000", 2024-09-11T09:51:18.653 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.0645000000000001e-05 2024-09-11T09:51:18.653 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:18.653 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:18.653 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:18.653 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.503699+0000", 2024-09-11T09:51:18.653 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.1940000000000003e-05 2024-09-11T09:51:18.653 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:18.653 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:18.653 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:18.653 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:18.653 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:18.653 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_repop_reply(client.4230.0:7 2.2 e18/15 ondisk, result = 0)", 2024-09-11T09:51:18.654 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:20.501608+0000", 2024-09-11T09:51:18.654 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 58.114129269999999, 2024-09-11T09:51:18.654 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.2459000000000005e-05, 2024-09-11T09:51:18.654 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:18.654 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:18.654 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "started", 2024-09-11T09:51:18.654 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:18.654 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:18.654 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:18.654 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.501608+0000", 2024-09-11T09:51:18.654 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:18.654 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:18.655 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:18.655 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:18.655 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.501608+0000", 2024-09-11T09:51:18.655 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:18.655 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:18.655 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:18.655 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:18.655 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.501609+0000", 2024-09-11T09:51:18.655 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.0160000000000001e-06 2024-09-11T09:51:18.655 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:18.655 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:18.655 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:18.656 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.501613+0000", 2024-09-11T09:51:18.656 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.5959999999999997e-06 2024-09-11T09:51:18.656 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:18.656 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:18.656 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:18.656 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.501625+0000", 2024-09-11T09:51:18.656 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1888e-05 2024-09-11T09:51:18.656 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:18.656 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:18.656 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:18.656 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.501634+0000", 2024-09-11T09:51:18.656 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.6139999999999992e-06 2024-09-11T09:51:18.656 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:18.657 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:18.657 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:18.657 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.501676+0000", 2024-09-11T09:51:18.657 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.1953999999999998e-05 2024-09-11T09:51:18.657 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:18.657 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:18.657 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:18.657 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.501682+0000", 2024-09-11T09:51:18.657 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.8000000000000004e-06 2024-09-11T09:51:18.657 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:18.657 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:18.657 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:18.658 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.501690+0000", 2024-09-11T09:51:18.658 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.5909999999999996e-06 2024-09-11T09:51:18.658 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:18.658 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:18.658 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:18.658 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:18.658 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:18.658 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-11T09:51:18.659 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-11T09:51:18.684 INFO:teuthology.orchestra.run.smithi017.stderr:reweighted osd.0 to 1 (10000) 2024-09-11T09:51:18.685 INFO:teuthology.orchestra.run.smithi017.stderr:nodeep-scrub is set 2024-09-11T09:51:18.700 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-09-11T09:51:18.700 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T09:51:18.701 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:51:18.701+0000 7fdec5849640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31907) UID: 0 2024-09-11T09:51:18.772 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:51:18.773 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [], 2024-09-11T09:51:18.773 INFO:teuthology.orchestra.run.smithi017.stdout: "num_ops": 0 2024-09-11T09:51:18.773 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:51:18.782 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-11T09:51:18.823 INFO:tasks.ceph.osd.0.smithi017.stderr:2024-09-11T09:51:18.824+0000 7fb73c0f1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31665) UID: 0 2024-09-11T09:51:18.890 INFO:tasks.ceph.osd.0.smithi017.stderr:2024-09-11T09:51:18.891+0000 7fb73c0f1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31665) UID: 0 2024-09-11T09:51:18.930 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:51:18.930 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [], 2024-09-11T09:51:18.930 INFO:teuthology.orchestra.run.smithi017.stdout: "complaint_time": 30, 2024-09-11T09:51:18.931 INFO:teuthology.orchestra.run.smithi017.stdout: "num_blocked_ops": 0 2024-09-11T09:51:18.931 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:51:18.939 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-11T09:51:18.990 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:51:18.991+0000 7fdec5849640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31907) UID: 0 2024-09-11T09:51:19.032 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-11T09:51:19.033 INFO:teuthology.orchestra.run.smithi017.stderr:dumped all 2024-09-11T09:51:19.045 INFO:teuthology.orchestra.run.smithi017.stdout:{"pg_ready":true,"pg_map":{"version":81,"stamp":"2024-09-11T09:51:18.684089+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":161,"num_write_kb":2418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120,"ondisk_log_size":120,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":5,"kb":566231040,"kb_used":182100,"kb_used_data":3264,"kb_used_omap":9,"kb_used_meta":178678,"kb_avail":566048940,"statfs":{"total":579820584960,"available":579634114560,"internally_reserved":0,"allocated":3342336,"data_stored":2342234,"data_compressed":66628,"data_compressed_allocated":90112,"data_compressed_original":1163264,"omap_allocated":9566,"internal_metadata":182966946},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.017072"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":30,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-11T09:51:15.675632+0000","last_change":"2024-09-11T09:50:54.231326+0000","last_active":"2024-09-11T09:51:15.675632+0000","last_peered":"2024-09-11T09:51:15.675632+0000","last_clean":"2024-09-11T09:51:15.675632+0000","last_became_active":"2024-09-11T09:50:54.231067+0000","last_became_peered":"2024-09-11T09:50:54.231067+0000","last_unstale":"2024-09-11T09:51:15.675632+0000","last_undegraded":"2024-09-11T09:51:15.675632+0000","last_fullsized":"2024-09-11T09:51:15.675632+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_clean_scrub_stamp":"2024-09-11T09:50:17.477890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:15:30.630386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00046094899999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":79,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-11T09:51:14.668069+0000","last_change":"2024-09-11T09:50:20.497446+0000","last_active":"2024-09-11T09:51:14.668069+0000","last_peered":"2024-09-11T09:51:14.668069+0000","last_clean":"2024-09-11T09:51:14.668069+0000","last_became_active":"2024-09-11T09:50:18.497481+0000","last_became_peered":"2024-09-11T09:50:18.497481+0000","last_unstale":"2024-09-11T09:51:14.668069+0000","last_undegraded":"2024-09-11T09:51:14.668069+0000","last_fullsized":"2024-09-11T09:51:14.668069+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_clean_scrub_stamp":"2024-09-11T09:50:17.477890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T14:12:09.857529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00050697600000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":91,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-11T09:51:15.675063+0000","last_change":"2024-09-11T09:50:46.593530+0000","last_active":"2024-09-11T09:51:15.675063+0000","last_peered":"2024-09-11T09:51:15.675063+0000","last_clean":"2024-09-11T09:51:15.675063+0000","last_became_active":"2024-09-11T09:50:46.593241+0000","last_became_peered":"2024-09-11T09:50:46.593241+0000","last_unstale":"2024-09-11T09:51:15.675063+0000","last_undegraded":"2024-09-11T09:51:15.675063+0000","last_fullsized":"2024-09-11T09:51:15.675063+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_clean_scrub_stamp":"2024-09-11T09:50:17.477890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T14:12:35.116334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045546699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":89,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-11T09:51:14.668031+0000","last_change":"2024-09-11T09:50:46.593637+0000","last_active":"2024-09-11T09:51:14.668031+0000","last_peered":"2024-09-11T09:51:14.668031+0000","last_clean":"2024-09-11T09:51:14.668031+0000","last_became_active":"2024-09-11T09:50:46.593286+0000","last_became_peered":"2024-09-11T09:50:46.593286+0000","last_unstale":"2024-09-11T09:51:14.668031+0000","last_undegraded":"2024-09-11T09:51:14.668031+0000","last_fullsized":"2024-09-11T09:51:14.668031+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_clean_scrub_stamp":"2024-09-11T09:50:17.477890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T19:11:42.500266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00050337500000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"18'2","reported_seq":80,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-11T09:51:15.221143+0000","last_change":"2024-09-11T09:50:20.501218+0000","last_active":"2024-09-11T09:51:15.221143+0000","last_peered":"2024-09-11T09:51:15.221143+0000","last_clean":"2024-09-11T09:51:15.221143+0000","last_became_active":"2024-09-11T09:50:18.493165+0000","last_became_peered":"2024-09-11T09:50:18.493165+0000","last_unstale":"2024-09-11T09:51:15.221143+0000","last_undegraded":"2024-09-11T09:51:15.221143+0000","last_fullsized":"2024-09-11T09:51:15.221143+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_clean_scrub_stamp":"2024-09-11T09:50:17.477890+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T18:04:31.250555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00094463900000000005,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":76,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-11T09:51:14.727737+0000","last_change":"2024-09-11T09:50:21.030566+0000","last_active":"2024-09-11T09:51:14.727737+0000","last_peered":"2024-09-11T09:51:14.727737+0000","last_clean":"2024-09-11T09:51:14.727737+0000","last_became_active":"2024-09-11T09:50:18.493109+0000","last_became_peered":"2024-09-11T09:50:18.493109+0000","last_unstale":"2024-09-11T09:51:14.727737+0000","last_undegraded":"2024-09-11T09:51:14.727737+0000","last_fullsized":"2024-09-11T09:51:14.727737+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_clean_scrub_stamp":"2024-09-11T09:50:17.477890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:37:16.328368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041008499999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":81,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-11T09:51:15.675063+0000","last_change":"2024-09-11T09:50:20.498284+0000","last_active":"2024-09-11T09:51:15.675063+0000","last_peered":"2024-09-11T09:51:15.675063+0000","last_clean":"2024-09-11T09:51:15.675063+0000","last_became_active":"2024-09-11T09:50:18.493210+0000","last_became_peered":"2024-09-11T09:50:18.493210+0000","last_unstale":"2024-09-11T09:51:15.675063+0000","last_undegraded":"2024-09-11T09:51:15.675063+0000","last_fullsized":"2024-09-11T09:51:15.675063+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_clean_scrub_stamp":"2024-09-11T09:50:17.477890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T18:16:13.887681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079749499999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"16'1","reported_seq":79,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-11T09:51:15.221152+0000","last_change":"2024-09-11T09:50:20.500687+0000","last_active":"2024-09-11T09:51:15.221152+0000","last_peered":"2024-09-11T09:51:15.221152+0000","last_clean":"2024-09-11T09:51:15.221152+0000","last_became_active":"2024-09-11T09:50:18.493650+0000","last_became_peered":"2024-09-11T09:50:18.493650+0000","last_unstale":"2024-09-11T09:51:15.221152+0000","last_undegraded":"2024-09-11T09:51:15.221152+0000","last_fullsized":"2024-09-11T09:51:15.221152+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_clean_scrub_stamp":"2024-09-11T09:50:17.477890+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T14:26:32.838143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041261100000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"13'117","reported_seq":258,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-11T09:51:15.675073+0000","last_change":"2024-09-11T09:50:49.606072+0000","last_active":"2024-09-11T09:51:15.675073+0000","last_peered":"2024-09-11T09:51:15.675073+0000","last_clean":"2024-09-11T09:51:15.675073+0000","last_became_active":"2024-09-11T09:50:49.605824+0000","last_became_peered":"2024-09-11T09:50:49.605824+0000","last_unstale":"2024-09-11T09:51:15.675073+0000","last_undegraded":"2024-09-11T09:51:15.675073+0000","last_fullsized":"2024-09-11T09:51:15.675073+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:50:10.457262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:50:10.457262+0000","last_clean_scrub_stamp":"2024-09-11T09:50:10.457262+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:02:29.081582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1771104,"data_compressed":100760,"data_compressed_allocated":135168,"data_compressed_original":1753088,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":12,"seq":51539607569,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31800,"kb_used_data":604,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340040,"statfs":{"total":96636764160,"available":96604200960,"internally_reserved":0,"allocated":618496,"data_stored":790098,"data_compressed":32496,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":12,"seq":51539607568,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31784,"kb_used_data":588,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340056,"statfs":{"total":96636764160,"available":96604217344,"internally_reserved":0,"allocated":602112,"data_stored":790098,"data_compressed":34132,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":11,"seq":47244640273,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27536,"kb_used_data":508,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344304,"statfs":{"total":96636764160,"available":96608567296,"internally_reserved":0,"allocated":520192,"data_stored":185904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":11,"seq":47244640273,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27548,"kb_used_data":516,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344292,"statfs":{"total":96636764160,"available":96608555008,"internally_reserved":0,"allocated":528384,"data_stored":190500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640272,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31720,"kb_used_data":528,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340120,"statfs":{"total":96636764160,"available":96604282880,"internally_reserved":0,"allocated":540672,"data_stored":195115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640272,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31712,"kb_used_data":520,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340128,"statfs":{"total":96636764160,"available":96604291072,"internally_reserved":0,"allocated":532480,"data_stored":190519,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":34132,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":34132,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32496,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T09:51:19.046 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T09:51:19.093 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:51:19.093 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 20, 2024-09-11T09:51:19.094 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 600, 2024-09-11T09:51:19.094 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [ 2024-09-11T09:51:19.094 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.094 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4183.0:21 1.0 e13/12 1:30306672:devicehealth::main.db.0000000000000000:head v 13'11, mlcod=13'8)", 2024-09-11T09:51:19.095 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:11.491728+0000", 2024-09-11T09:51:19.095 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 67.599733588000007, 2024-09-11T09:51:19.095 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00093269800000000003, 2024-09-11T09:51:19.095 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:19.096 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:19.096 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:19.096 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:19.096 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.097 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:19.097 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.491728+0000", 2024-09-11T09:51:19.097 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:19.097 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.098 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.098 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:19.098 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.491728+0000", 2024-09-11T09:51:19.098 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:19.099 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.099 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.099 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:19.099 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.491729+0000", 2024-09-11T09:51:19.099 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.8599999999999997e-07 2024-09-11T09:51:19.100 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.100 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.100 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:19.100 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.491820+0000", 2024-09-11T09:51:19.101 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.0812999999999999e-05 2024-09-11T09:51:19.101 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.101 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.101 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:19.102 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.491833+0000", 2024-09-11T09:51:19.102 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.341e-05 2024-09-11T09:51:19.102 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.102 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.102 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:19.103 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.491841+0000", 2024-09-11T09:51:19.103 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.8730000000000007e-06 2024-09-11T09:51:19.103 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.103 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.103 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:19.104 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.491916+0000", 2024-09-11T09:51:19.104 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.4881000000000004e-05 2024-09-11T09:51:19.104 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.104 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.104 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:19.104 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.491931+0000", 2024-09-11T09:51:19.105 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.4825e-05 2024-09-11T09:51:19.105 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.105 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.105 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:19.105 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.492633+0000", 2024-09-11T09:51:19.105 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00070191600000000002 2024-09-11T09:51:19.106 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.106 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.106 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:19.106 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.492661+0000", 2024-09-11T09:51:19.106 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.8093999999999999e-05 2024-09-11T09:51:19.106 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:19.107 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:19.107 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:19.107 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.107 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.107 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4183.0:31 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'17, mlcod=13'14)", 2024-09-11T09:51:19.108 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:11.499319+0000", 2024-09-11T09:51:19.108 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 67.592142918999997, 2024-09-11T09:51:19.108 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0010073879999999999, 2024-09-11T09:51:19.108 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:19.108 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:19.108 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:19.109 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:19.109 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.109 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:19.109 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.499319+0000", 2024-09-11T09:51:19.109 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:19.109 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.110 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.110 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:19.110 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.499319+0000", 2024-09-11T09:51:19.110 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:19.110 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.110 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.111 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:19.111 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.499320+0000", 2024-09-11T09:51:19.111 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.7700000000000004e-07 2024-09-11T09:51:19.111 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.111 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.111 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:19.111 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.499326+0000", 2024-09-11T09:51:19.112 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.6420000000000001e-06 2024-09-11T09:51:19.112 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.112 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.112 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:19.112 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.499339+0000", 2024-09-11T09:51:19.112 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2537999999999999e-05 2024-09-11T09:51:19.113 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.113 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.113 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:19.113 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.499345+0000", 2024-09-11T09:51:19.113 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.8220000000000003e-06 2024-09-11T09:51:19.113 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.114 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.114 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:19.114 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.499403+0000", 2024-09-11T09:51:19.114 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.8814000000000003e-05 2024-09-11T09:51:19.114 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.114 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.115 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:19.115 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.499427+0000", 2024-09-11T09:51:19.115 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.3853999999999999e-05 2024-09-11T09:51:19.115 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.115 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.115 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:19.115 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.500294+0000", 2024-09-11T09:51:19.116 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00086622399999999997 2024-09-11T09:51:19.116 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.116 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.116 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:19.116 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.500326+0000", 2024-09-11T09:51:19.116 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.2716999999999999e-05 2024-09-11T09:51:19.117 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:19.117 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:19.117 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:19.117 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.117 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.117 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4183.0:32 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'18, mlcod=13'14)", 2024-09-11T09:51:19.118 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:11.499796+0000", 2024-09-11T09:51:19.118 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 67.591665712999998, 2024-09-11T09:51:19.118 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00092915100000000002, 2024-09-11T09:51:19.118 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:19.118 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:19.118 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:19.119 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:19.119 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.119 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:19.119 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.499796+0000", 2024-09-11T09:51:19.119 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:19.119 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.119 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.120 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:19.120 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.499796+0000", 2024-09-11T09:51:19.120 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:19.120 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.120 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.120 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:19.121 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.499797+0000", 2024-09-11T09:51:19.121 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.9400000000000004e-07 2024-09-11T09:51:19.121 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.121 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.121 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:19.121 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.499841+0000", 2024-09-11T09:51:19.122 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.3900000000000003e-05 2024-09-11T09:51:19.122 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.122 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.122 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:19.122 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.499852+0000", 2024-09-11T09:51:19.122 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1462e-05 2024-09-11T09:51:19.123 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.123 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.123 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:19.123 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.499858+0000", 2024-09-11T09:51:19.123 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.8320000000000002e-06 2024-09-11T09:51:19.123 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.123 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.124 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:19.124 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.499915+0000", 2024-09-11T09:51:19.124 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.7187e-05 2024-09-11T09:51:19.124 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.124 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.124 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:19.125 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.499930+0000", 2024-09-11T09:51:19.125 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.4928e-05 2024-09-11T09:51:19.125 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.125 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.125 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:19.125 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.500697+0000", 2024-09-11T09:51:19.125 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00076658399999999995 2024-09-11T09:51:19.126 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.126 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.126 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:19.126 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.500725+0000", 2024-09-11T09:51:19.126 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.8364000000000002e-05 2024-09-11T09:51:19.126 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:19.127 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:19.127 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:19.127 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.127 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.127 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4183.0:34 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'20, mlcod=13'15)", 2024-09-11T09:51:19.127 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:11.501016+0000", 2024-09-11T09:51:19.127 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 67.590446194999998, 2024-09-11T09:51:19.128 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00097815399999999996, 2024-09-11T09:51:19.128 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:19.128 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:19.128 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:19.128 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:19.129 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.129 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:19.129 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.501016+0000", 2024-09-11T09:51:19.129 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:19.129 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.129 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.129 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:19.130 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.501016+0000", 2024-09-11T09:51:19.130 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:19.130 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.130 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.130 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:19.130 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.501016+0000", 2024-09-11T09:51:19.131 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.3799999999999997e-07 2024-09-11T09:51:19.131 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.131 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.131 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:19.131 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.501023+0000", 2024-09-11T09:51:19.131 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.2180000000000002e-06 2024-09-11T09:51:19.132 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.132 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.132 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:19.133 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.501031+0000", 2024-09-11T09:51:19.133 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.5379999999999997e-06 2024-09-11T09:51:19.133 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.133 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.133 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:19.134 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.501035+0000", 2024-09-11T09:51:19.134 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.216e-06 2024-09-11T09:51:19.134 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.134 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.135 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:19.135 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.501105+0000", 2024-09-11T09:51:19.135 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.9757000000000003e-05 2024-09-11T09:51:19.135 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.136 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.136 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:19.136 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.501120+0000", 2024-09-11T09:51:19.136 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.5358000000000001e-05 2024-09-11T09:51:19.136 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.137 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.137 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:19.137 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.501955+0000", 2024-09-11T09:51:19.137 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00083505000000000005 2024-09-11T09:51:19.138 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.138 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.138 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:19.138 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.501994+0000", 2024-09-11T09:51:19.139 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.8479000000000002e-05 2024-09-11T09:51:19.139 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:19.139 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:19.139 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:19.139 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.140 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.140 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4183.0:38 1.0 e13/12 1:30306672:devicehealth::main.db.0000000000000000:head v 13'24, mlcod=13'22)", 2024-09-11T09:51:19.140 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:11.503891+0000", 2024-09-11T09:51:19.140 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 67.587570752999994, 2024-09-11T09:51:19.141 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000959351, 2024-09-11T09:51:19.141 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:19.141 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:19.141 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:19.141 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:19.142 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.142 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:19.142 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.503891+0000", 2024-09-11T09:51:19.142 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:19.143 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.143 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.143 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:19.143 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.503891+0000", 2024-09-11T09:51:19.143 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:19.144 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.144 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.144 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:19.144 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.503892+0000", 2024-09-11T09:51:19.145 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.5499999999999998e-07 2024-09-11T09:51:19.145 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.145 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.145 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:19.145 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.503967+0000", 2024-09-11T09:51:19.146 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.4993999999999994e-05 2024-09-11T09:51:19.146 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.146 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.146 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:19.147 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.503976+0000", 2024-09-11T09:51:19.147 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.4709999999999996e-06 2024-09-11T09:51:19.147 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.147 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.147 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:19.148 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.503981+0000", 2024-09-11T09:51:19.148 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.6789999999999996e-06 2024-09-11T09:51:19.148 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.148 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.149 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:19.149 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.504021+0000", 2024-09-11T09:51:19.149 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.0065999999999999e-05 2024-09-11T09:51:19.149 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.149 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.150 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:19.150 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.504036+0000", 2024-09-11T09:51:19.150 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.468e-05 2024-09-11T09:51:19.150 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.150 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.151 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:19.151 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.504813+0000", 2024-09-11T09:51:19.151 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00077718299999999998 2024-09-11T09:51:19.151 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.151 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.152 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:19.152 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.504850+0000", 2024-09-11T09:51:19.152 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.7722999999999998e-05 2024-09-11T09:51:19.152 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:19.152 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:19.152 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:19.153 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.153 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.153 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4183.0:54 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'32, mlcod=13'29)", 2024-09-11T09:51:19.153 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:11.515404+0000", 2024-09-11T09:51:19.153 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 67.576058148000001, 2024-09-11T09:51:19.153 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00089471799999999997, 2024-09-11T09:51:19.153 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:19.153 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:19.153 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:19.153 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:19.153 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.154 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:19.154 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.515404+0000", 2024-09-11T09:51:19.154 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:19.154 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.154 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.154 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:19.154 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.515404+0000", 2024-09-11T09:51:19.154 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:19.154 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.154 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.154 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:19.154 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.515404+0000", 2024-09-11T09:51:19.154 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.3799999999999996e-07 2024-09-11T09:51:19.154 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.155 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.155 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:19.155 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.515410+0000", 2024-09-11T09:51:19.155 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.1789999999999999e-06 2024-09-11T09:51:19.155 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.155 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.155 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:19.155 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.515422+0000", 2024-09-11T09:51:19.155 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1929999999999999e-05 2024-09-11T09:51:19.155 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.155 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.155 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:19.155 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.515427+0000", 2024-09-11T09:51:19.155 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.6060000000000002e-06 2024-09-11T09:51:19.156 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.156 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.156 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:19.156 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.515482+0000", 2024-09-11T09:51:19.156 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.4743000000000001e-05 2024-09-11T09:51:19.156 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.156 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.156 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:19.156 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.515498+0000", 2024-09-11T09:51:19.156 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.5778999999999999e-05 2024-09-11T09:51:19.156 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.156 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.157 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:19.157 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.516271+0000", 2024-09-11T09:51:19.157 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00077312599999999996 2024-09-11T09:51:19.157 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.157 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.157 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:19.157 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.516298+0000", 2024-09-11T09:51:19.157 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.7617e-05 2024-09-11T09:51:19.157 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:19.157 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:19.157 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:19.157 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.157 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.157 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4183.0:57 1.0 e13/12 1:30306672:devicehealth::main.db.0000000000000000:head v 13'35, mlcod=13'33)", 2024-09-11T09:51:19.158 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:11.519823+0000", 2024-09-11T09:51:19.158 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 67.571639090999994, 2024-09-11T09:51:19.158 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00090876100000000001, 2024-09-11T09:51:19.158 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:19.158 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:19.158 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:19.158 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:19.158 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.158 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:19.158 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.519823+0000", 2024-09-11T09:51:19.158 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:19.158 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.158 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.159 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:19.159 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.519823+0000", 2024-09-11T09:51:19.159 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:19.159 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.159 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.159 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:19.159 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.519823+0000", 2024-09-11T09:51:19.159 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.7000000000000004e-07 2024-09-11T09:51:19.159 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.159 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.159 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:19.159 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.519859+0000", 2024-09-11T09:51:19.159 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.5508e-05 2024-09-11T09:51:19.160 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.160 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.160 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:19.160 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.519871+0000", 2024-09-11T09:51:19.160 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.221e-05 2024-09-11T09:51:19.160 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.160 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.160 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:19.160 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.519877+0000", 2024-09-11T09:51:19.160 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.0410000000000003e-06 2024-09-11T09:51:19.160 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.160 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.160 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:19.161 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.519955+0000", 2024-09-11T09:51:19.161 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.8169000000000003e-05 2024-09-11T09:51:19.161 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.161 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.161 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:19.161 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.519972+0000", 2024-09-11T09:51:19.161 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6595999999999998e-05 2024-09-11T09:51:19.161 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.161 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.161 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:19.161 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.520691+0000", 2024-09-11T09:51:19.161 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000719405 2024-09-11T09:51:19.161 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.161 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.162 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:19.162 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.520731+0000", 2024-09-11T09:51:19.162 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.0061999999999997e-05 2024-09-11T09:51:19.162 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:19.162 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:19.162 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:19.162 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.162 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.162 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4183.0:58 1.0 e13/12 1:30306672:devicehealth::main.db.0000000000000000:head v 13'36, mlcod=13'33)", 2024-09-11T09:51:19.162 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:11.520285+0000", 2024-09-11T09:51:19.162 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 67.571176683000004, 2024-09-11T09:51:19.162 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00091014799999999995, 2024-09-11T09:51:19.162 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:19.162 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:19.163 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:19.163 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:19.163 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.163 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:19.163 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.520285+0000", 2024-09-11T09:51:19.163 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:19.163 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.163 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.163 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:19.163 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.520285+0000", 2024-09-11T09:51:19.163 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:19.164 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.164 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.164 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:19.164 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.520286+0000", 2024-09-11T09:51:19.164 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.4399999999999999e-07 2024-09-11T09:51:19.164 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.164 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.164 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:19.164 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.520348+0000", 2024-09-11T09:51:19.164 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.1760000000000005e-05 2024-09-11T09:51:19.164 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.164 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.164 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:19.165 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.520359+0000", 2024-09-11T09:51:19.165 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1362999999999999e-05 2024-09-11T09:51:19.165 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.165 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.165 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:19.165 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.520364+0000", 2024-09-11T09:51:19.165 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.3600000000000004e-06 2024-09-11T09:51:19.165 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.165 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.165 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:19.165 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.520413+0000", 2024-09-11T09:51:19.165 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.8226000000000002e-05 2024-09-11T09:51:19.165 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.166 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.166 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:19.166 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.520426+0000", 2024-09-11T09:51:19.166 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3560000000000001e-05 2024-09-11T09:51:19.166 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.166 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.166 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:19.166 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.521159+0000", 2024-09-11T09:51:19.166 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00073255499999999997 2024-09-11T09:51:19.166 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.166 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.166 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:19.166 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.521195+0000", 2024-09-11T09:51:19.166 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.6480000000000003e-05 2024-09-11T09:51:19.167 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:19.167 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:19.167 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:19.167 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.167 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.167 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4183.0:68 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'42, mlcod=13'39)", 2024-09-11T09:51:19.167 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:11.534957+0000", 2024-09-11T09:51:19.167 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 67.556504992000001, 2024-09-11T09:51:19.167 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.001065351, 2024-09-11T09:51:19.167 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:19.167 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:19.167 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:19.167 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:19.168 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.168 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:19.168 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.534957+0000", 2024-09-11T09:51:19.168 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:19.168 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.168 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.168 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:19.168 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.534957+0000", 2024-09-11T09:51:19.168 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:19.168 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.168 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.168 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:19.168 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.534957+0000", 2024-09-11T09:51:19.169 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.3799999999999998e-07 2024-09-11T09:51:19.169 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.169 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.169 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:19.169 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.534964+0000", 2024-09-11T09:51:19.169 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.9809999999999997e-06 2024-09-11T09:51:19.169 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.169 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.169 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:19.169 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.534976+0000", 2024-09-11T09:51:19.169 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2119999999999999e-05 2024-09-11T09:51:19.169 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.169 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.169 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:19.170 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.534980+0000", 2024-09-11T09:51:19.170 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.9890000000000003e-06 2024-09-11T09:51:19.170 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.170 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.170 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:19.170 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.535030+0000", 2024-09-11T09:51:19.170 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.9258000000000002e-05 2024-09-11T09:51:19.170 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.170 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.170 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:19.170 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.535044+0000", 2024-09-11T09:51:19.170 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3981e-05 2024-09-11T09:51:19.170 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.170 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.170 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:19.171 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.535995+0000", 2024-09-11T09:51:19.171 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000951804 2024-09-11T09:51:19.171 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.171 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.171 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:19.171 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.536022+0000", 2024-09-11T09:51:19.171 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.6780000000000001e-05 2024-09-11T09:51:19.171 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:19.171 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:19.171 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:19.171 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.171 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.171 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4183.0:69 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'43, mlcod=13'39)", 2024-09-11T09:51:19.171 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:11.535331+0000", 2024-09-11T09:51:19.172 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 67.556130668999998, 2024-09-11T09:51:19.172 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0011256090000000001, 2024-09-11T09:51:19.172 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:19.172 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:19.172 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:19.172 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:19.172 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.172 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:19.172 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.535331+0000", 2024-09-11T09:51:19.172 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:19.172 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.172 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.172 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:19.173 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.535331+0000", 2024-09-11T09:51:19.173 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:19.173 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.173 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.173 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:19.173 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.535332+0000", 2024-09-11T09:51:19.173 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.1600000000000001e-07 2024-09-11T09:51:19.173 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.173 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.173 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:19.173 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.535393+0000", 2024-09-11T09:51:19.173 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.1452999999999996e-05 2024-09-11T09:51:19.173 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.174 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.174 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:19.174 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.535402+0000", 2024-09-11T09:51:19.174 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.4300000000000006e-06 2024-09-11T09:51:19.174 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.174 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.174 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:19.174 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.535405+0000", 2024-09-11T09:51:19.174 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.512e-06 2024-09-11T09:51:19.174 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.174 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.174 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:19.174 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.535482+0000", 2024-09-11T09:51:19.175 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.7132999999999995e-05 2024-09-11T09:51:19.175 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.175 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.175 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:19.175 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.535506+0000", 2024-09-11T09:51:19.175 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.4070000000000002e-05 2024-09-11T09:51:19.175 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.175 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.175 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:19.175 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.536424+0000", 2024-09-11T09:51:19.175 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00091766700000000005 2024-09-11T09:51:19.175 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.175 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.175 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:19.176 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.536457+0000", 2024-09-11T09:51:19.176 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.2827999999999998e-05 2024-09-11T09:51:19.176 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:19.176 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:19.176 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:19.176 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.176 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.176 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4183.0:83 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'53, mlcod=13'50)", 2024-09-11T09:51:19.176 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:11.558836+0000", 2024-09-11T09:51:19.176 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 67.532625413999995, 2024-09-11T09:51:19.176 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00090313100000000005, 2024-09-11T09:51:19.176 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:19.176 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:19.177 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:19.177 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:19.177 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.177 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:19.177 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.558836+0000", 2024-09-11T09:51:19.177 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:19.177 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.177 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.177 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:19.177 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.558836+0000", 2024-09-11T09:51:19.177 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:19.177 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.177 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.177 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:19.178 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.558837+0000", 2024-09-11T09:51:19.178 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.4900000000000005e-07 2024-09-11T09:51:19.178 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.178 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.178 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:19.178 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.558857+0000", 2024-09-11T09:51:19.178 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.9502999999999999e-05 2024-09-11T09:51:19.178 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.178 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.178 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:19.178 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.558870+0000", 2024-09-11T09:51:19.178 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3020000000000001e-05 2024-09-11T09:51:19.178 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.178 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.179 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:19.179 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.558876+0000", 2024-09-11T09:51:19.179 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.7270000000000004e-06 2024-09-11T09:51:19.179 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.179 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.179 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:19.179 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.558934+0000", 2024-09-11T09:51:19.179 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.8100999999999998e-05 2024-09-11T09:51:19.179 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.179 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.179 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:19.179 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.558947+0000", 2024-09-11T09:51:19.179 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3269e-05 2024-09-11T09:51:19.179 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.180 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.180 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:19.180 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.559720+0000", 2024-09-11T09:51:19.180 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00077287600000000003 2024-09-11T09:51:19.180 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.180 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.180 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:19.180 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.559740+0000", 2024-09-11T09:51:19.180 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.9786e-05 2024-09-11T09:51:19.180 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:19.180 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:19.180 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:19.180 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.180 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.181 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4183.0:84 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'54, mlcod=13'50)", 2024-09-11T09:51:19.181 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:11.559267+0000", 2024-09-11T09:51:19.181 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 67.532194986999997, 2024-09-11T09:51:19.181 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00093990199999999995, 2024-09-11T09:51:19.181 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:19.181 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:19.181 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:19.181 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:19.181 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.181 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:19.181 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.559267+0000", 2024-09-11T09:51:19.181 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:19.181 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.181 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.182 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:19.182 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.559267+0000", 2024-09-11T09:51:19.182 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:19.182 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.182 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.182 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:19.182 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.559267+0000", 2024-09-11T09:51:19.182 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.2900000000000003e-07 2024-09-11T09:51:19.182 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.182 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.182 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:19.182 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.559322+0000", 2024-09-11T09:51:19.182 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.4429999999999999e-05 2024-09-11T09:51:19.183 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.183 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.183 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:19.183 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.559332+0000", 2024-09-11T09:51:19.183 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0473e-05 2024-09-11T09:51:19.183 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.183 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.183 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:19.183 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.559338+0000", 2024-09-11T09:51:19.183 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.0730000000000001e-06 2024-09-11T09:51:19.183 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.183 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.183 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:19.183 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.559373+0000", 2024-09-11T09:51:19.183 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.4626999999999999e-05 2024-09-11T09:51:19.184 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.184 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.184 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:19.184 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.559384+0000", 2024-09-11T09:51:19.184 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1307e-05 2024-09-11T09:51:19.184 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.184 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.184 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:19.184 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.560175+0000", 2024-09-11T09:51:19.184 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00079016399999999999 2024-09-11T09:51:19.184 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.184 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.185 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:19.185 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.560207+0000", 2024-09-11T09:51:19.185 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.2199000000000001e-05 2024-09-11T09:51:19.185 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:19.185 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:19.185 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:19.185 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.185 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.185 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4183.0:89 1.0 e13/12 1:30306672:devicehealth::main.db.0000000000000000:head v 13'59, mlcod=13'57)", 2024-09-11T09:51:19.185 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:11.563614+0000", 2024-09-11T09:51:19.185 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 67.527847367000007, 2024-09-11T09:51:19.185 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00091281399999999996, 2024-09-11T09:51:19.185 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:19.186 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:19.186 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:19.186 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:19.186 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.186 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:19.186 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.563614+0000", 2024-09-11T09:51:19.186 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:19.186 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.186 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.186 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:19.186 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.563614+0000", 2024-09-11T09:51:19.186 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:19.186 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.186 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.187 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:19.187 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.563615+0000", 2024-09-11T09:51:19.187 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.6899999999999998e-07 2024-09-11T09:51:19.187 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.187 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.187 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:19.187 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.563681+0000", 2024-09-11T09:51:19.187 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.5769e-05 2024-09-11T09:51:19.187 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.187 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.187 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:19.187 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.563690+0000", 2024-09-11T09:51:19.187 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.6800000000000005e-06 2024-09-11T09:51:19.188 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.188 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.188 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:19.188 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.563699+0000", 2024-09-11T09:51:19.188 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.3170000000000002e-06 2024-09-11T09:51:19.188 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.188 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.188 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:19.188 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.563734+0000", 2024-09-11T09:51:19.188 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.5023e-05 2024-09-11T09:51:19.188 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.188 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.188 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:19.189 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.563746+0000", 2024-09-11T09:51:19.189 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2661e-05 2024-09-11T09:51:19.189 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.189 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.189 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:19.189 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.564500+0000", 2024-09-11T09:51:19.189 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00075323100000000004 2024-09-11T09:51:19.189 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.189 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.189 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:19.189 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.564527+0000", 2024-09-11T09:51:19.189 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.7664000000000001e-05 2024-09-11T09:51:19.189 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:19.190 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:19.190 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:19.190 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.190 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.190 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4183.0:105 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'67, mlcod=13'64)", 2024-09-11T09:51:19.190 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:11.712146+0000", 2024-09-11T09:51:19.190 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 67.379315617000003, 2024-09-11T09:51:19.190 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0010916859999999999, 2024-09-11T09:51:19.190 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:19.190 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:19.190 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:19.190 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:19.190 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.191 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:19.191 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.712146+0000", 2024-09-11T09:51:19.191 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:19.191 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.191 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.191 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:19.191 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.712146+0000", 2024-09-11T09:51:19.191 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:19.191 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.191 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.191 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:19.191 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.712147+0000", 2024-09-11T09:51:19.191 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.8200000000000001e-07 2024-09-11T09:51:19.192 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.192 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.192 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:19.192 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.712155+0000", 2024-09-11T09:51:19.192 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.0309999999999998e-06 2024-09-11T09:51:19.192 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.192 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.192 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:19.192 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.712163+0000", 2024-09-11T09:51:19.192 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.5920000000000001e-06 2024-09-11T09:51:19.192 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.192 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.192 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:19.193 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.712169+0000", 2024-09-11T09:51:19.193 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.4299999999999997e-06 2024-09-11T09:51:19.193 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.193 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.193 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:19.193 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.712418+0000", 2024-09-11T09:51:19.193 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00024978499999999998 2024-09-11T09:51:19.193 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.193 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.193 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:19.193 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.712432+0000", 2024-09-11T09:51:19.193 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3190000000000001e-05 2024-09-11T09:51:19.193 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.193 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.194 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:19.194 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.713206+0000", 2024-09-11T09:51:19.194 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00077420199999999998 2024-09-11T09:51:19.194 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.194 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.194 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:19.194 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.713238+0000", 2024-09-11T09:51:19.194 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.2073999999999998e-05 2024-09-11T09:51:19.194 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:19.194 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:19.194 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:19.194 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.194 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.194 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4183.0:106 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'68, mlcod=13'64)", 2024-09-11T09:51:19.194 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:11.712624+0000", 2024-09-11T09:51:19.195 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 67.378838289000001, 2024-09-11T09:51:19.195 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0010009629999999999, 2024-09-11T09:51:19.195 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:19.195 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:19.195 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:19.195 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:19.195 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.195 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:19.195 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.712624+0000", 2024-09-11T09:51:19.195 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:19.195 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.195 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.195 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:19.196 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.712624+0000", 2024-09-11T09:51:19.196 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:19.196 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.196 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.196 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:19.196 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.712624+0000", 2024-09-11T09:51:19.196 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.4799999999999998e-07 2024-09-11T09:51:19.196 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.196 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.196 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:19.196 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.712689+0000", 2024-09-11T09:51:19.196 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.4488999999999999e-05 2024-09-11T09:51:19.196 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.196 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.197 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:19.197 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.712707+0000", 2024-09-11T09:51:19.197 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.8272000000000001e-05 2024-09-11T09:51:19.197 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.197 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.197 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:19.197 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.712713+0000", 2024-09-11T09:51:19.197 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.2040000000000002e-06 2024-09-11T09:51:19.197 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.197 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.197 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:19.197 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.712783+0000", 2024-09-11T09:51:19.197 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.9534000000000002e-05 2024-09-11T09:51:19.197 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.198 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.198 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:19.198 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.712811+0000", 2024-09-11T09:51:19.198 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.8620999999999999e-05 2024-09-11T09:51:19.198 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.198 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.198 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:19.198 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.713586+0000", 2024-09-11T09:51:19.198 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00077462600000000005 2024-09-11T09:51:19.198 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.198 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.198 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:19.198 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:11.713624+0000", 2024-09-11T09:51:19.198 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.8668999999999999e-05 2024-09-11T09:51:19.199 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:19.199 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:19.199 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:19.199 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.199 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.199 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4183.0:134 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'92, mlcod=13'86)", 2024-09-11T09:51:19.199 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:12.035698+0000", 2024-09-11T09:51:19.199 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 67.055764030000006, 2024-09-11T09:51:19.199 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00089550700000000003, 2024-09-11T09:51:19.199 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:19.199 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:19.199 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:19.199 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:19.199 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.200 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:19.200 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.035698+0000", 2024-09-11T09:51:19.200 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:19.200 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.200 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.200 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:19.200 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.035698+0000", 2024-09-11T09:51:19.200 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:19.200 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.200 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.200 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:19.200 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.035698+0000", 2024-09-11T09:51:19.200 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.0900000000000002e-07 2024-09-11T09:51:19.201 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.201 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.201 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:19.201 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.035758+0000", 2024-09-11T09:51:19.201 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.9481999999999997e-05 2024-09-11T09:51:19.201 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.201 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.201 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:19.201 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.035766+0000", 2024-09-11T09:51:19.201 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.3760000000000008e-06 2024-09-11T09:51:19.201 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.201 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.201 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:19.201 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.035772+0000", 2024-09-11T09:51:19.202 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.8930000000000002e-06 2024-09-11T09:51:19.202 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.202 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.202 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:19.202 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.035807+0000", 2024-09-11T09:51:19.202 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.4975999999999999e-05 2024-09-11T09:51:19.202 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.202 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.202 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:19.202 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.035817+0000", 2024-09-11T09:51:19.202 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0314e-05 2024-09-11T09:51:19.202 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.202 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.202 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:19.203 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.036542+0000", 2024-09-11T09:51:19.203 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00072434400000000003 2024-09-11T09:51:19.203 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.203 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.203 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:19.203 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.036593+0000", 2024-09-11T09:51:19.203 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.1613000000000002e-05 2024-09-11T09:51:19.203 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:19.203 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:19.203 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:19.203 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.203 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.203 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4183.0:136 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'94, mlcod=13'86)", 2024-09-11T09:51:19.203 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:12.036591+0000", 2024-09-11T09:51:19.204 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 67.054870820000005, 2024-09-11T09:51:19.204 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00090637900000000002, 2024-09-11T09:51:19.204 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:19.204 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:19.204 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:19.204 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:19.204 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.204 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:19.204 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.036591+0000", 2024-09-11T09:51:19.204 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:19.204 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.204 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.204 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:19.205 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.036591+0000", 2024-09-11T09:51:19.205 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:19.205 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.205 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.205 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:19.205 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.036592+0000", 2024-09-11T09:51:19.205 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.37e-07 2024-09-11T09:51:19.205 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.205 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.205 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:19.205 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.036600+0000", 2024-09-11T09:51:19.205 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.6910000000000003e-06 2024-09-11T09:51:19.205 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.206 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.206 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:19.206 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.036609+0000", 2024-09-11T09:51:19.206 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.0229999999999998e-06 2024-09-11T09:51:19.206 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.206 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.206 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:19.206 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.036617+0000", 2024-09-11T09:51:19.206 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.6979999999999998e-06 2024-09-11T09:51:19.206 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.206 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.206 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:19.206 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.036658+0000", 2024-09-11T09:51:19.206 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.0692000000000003e-05 2024-09-11T09:51:19.207 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.207 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.207 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:19.207 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.036670+0000", 2024-09-11T09:51:19.207 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2109e-05 2024-09-11T09:51:19.207 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.207 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.207 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:19.207 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.037475+0000", 2024-09-11T09:51:19.207 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00080487199999999999 2024-09-11T09:51:19.207 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.207 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.207 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:19.208 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.037497+0000", 2024-09-11T09:51:19.208 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.2557000000000001e-05 2024-09-11T09:51:19.208 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:19.208 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:19.208 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:19.208 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.208 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.208 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4183.0:158 1.0 e13/12 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 13'112, mlcod=13'108)", 2024-09-11T09:51:19.208 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:12.455073+0000", 2024-09-11T09:51:19.208 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 66.636388566999997, 2024-09-11T09:51:19.208 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00090654500000000003, 2024-09-11T09:51:19.208 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:19.208 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:19.209 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:19.209 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:19.209 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.209 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:19.209 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.455073+0000", 2024-09-11T09:51:19.209 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:19.209 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.209 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.209 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:19.209 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.455073+0000", 2024-09-11T09:51:19.209 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:19.209 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.209 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.210 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:19.210 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.455074+0000", 2024-09-11T09:51:19.210 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.32e-07 2024-09-11T09:51:19.210 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.210 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.210 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:19.210 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.455126+0000", 2024-09-11T09:51:19.210 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.1898999999999999e-05 2024-09-11T09:51:19.210 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.210 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.210 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:19.210 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.455136+0000", 2024-09-11T09:51:19.210 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0651e-05 2024-09-11T09:51:19.211 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.211 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.211 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:19.211 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.455142+0000", 2024-09-11T09:51:19.211 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.0020000000000003e-06 2024-09-11T09:51:19.211 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.211 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.211 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:19.211 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.455192+0000", 2024-09-11T09:51:19.211 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.0164000000000003e-05 2024-09-11T09:51:19.211 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.211 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.211 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:19.212 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.455204+0000", 2024-09-11T09:51:19.212 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1178999999999999e-05 2024-09-11T09:51:19.212 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.212 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.212 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:19.212 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.455957+0000", 2024-09-11T09:51:19.212 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00075362300000000003 2024-09-11T09:51:19.212 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.212 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.212 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:19.212 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:12.455980+0000", 2024-09-11T09:51:19.212 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.2595e-05 2024-09-11T09:51:19.212 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:19.213 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:19.213 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:19.213 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.213 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.213 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "MOSDPGPush(1.0 34/33 [PushOp(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head, version: 13'116, data_included: [0~262688], data_size: 262688, omap_header_size: 0, omap_entries_size: 0, attrset_size: 9, recovery_info: ObjectRecoveryInfo(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head@13'116, size: 262688, copy_subset: [(0, 262688)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 262688, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-11T09:51:19.213 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:47.599030+0000", 2024-09-11T09:51:19.213 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 31.492431941, 2024-09-11T09:51:19.213 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0010572209999999999, 2024-09-11T09:51:19.213 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:19.213 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:19.213 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "started", 2024-09-11T09:51:19.213 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:19.213 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.213 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:19.214 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:47.599030+0000", 2024-09-11T09:51:19.214 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:19.214 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.214 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.214 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:19.214 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:47.599030+0000", 2024-09-11T09:51:19.214 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:19.214 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.214 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.214 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:19.214 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:47.599031+0000", 2024-09-11T09:51:19.214 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.92e-07 2024-09-11T09:51:19.214 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.215 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.215 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:19.215 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:47.599328+0000", 2024-09-11T09:51:19.215 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00029708399999999998 2024-09-11T09:51:19.215 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.215 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.215 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:19.215 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:47.599353+0000", 2024-09-11T09:51:19.215 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.5721e-05 2024-09-11T09:51:19.215 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.215 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.215 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:19.215 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:47.599375+0000", 2024-09-11T09:51:19.216 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.1990000000000001e-05 2024-09-11T09:51:19.216 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.216 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.216 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:19.216 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:47.599466+0000", 2024-09-11T09:51:19.216 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.0175999999999999e-05 2024-09-11T09:51:19.216 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.216 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.216 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:19.216 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:47.599480+0000", 2024-09-11T09:51:19.216 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.4873e-05 2024-09-11T09:51:19.216 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.216 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.217 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:19.217 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:47.600087+0000", 2024-09-11T09:51:19.217 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00060668500000000004 2024-09-11T09:51:19.217 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:19.217 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:19.217 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:19.217 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.217 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.217 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "MOSDPGPush(1.0 34/33 [PushOp(1:30306672:devicehealth::main.db.0000000000000000:head, version: 13'117, data_included: [0~327680], data_size: 327680, omap_header_size: 0, omap_entries_size: 0, attrset_size: 10, recovery_info: ObjectRecoveryInfo(1:30306672:devicehealth::main.db.0000000000000000:head@13'117, size: 327680, copy_subset: [(0, 327680)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 327680, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-11T09:51:19.217 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:47.601120+0000", 2024-09-11T09:51:19.217 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 31.490341452999999, 2024-09-11T09:51:19.217 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00099708000000000001, 2024-09-11T09:51:19.218 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:19.218 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:19.218 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "started", 2024-09-11T09:51:19.218 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:19.218 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.218 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:19.218 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:47.601120+0000", 2024-09-11T09:51:19.218 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:19.218 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.218 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.218 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:19.218 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:47.601120+0000", 2024-09-11T09:51:19.218 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:19.218 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.219 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.219 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:19.219 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:47.601121+0000", 2024-09-11T09:51:19.219 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.0200000000000001e-07 2024-09-11T09:51:19.219 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.219 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.219 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:19.219 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:47.601430+0000", 2024-09-11T09:51:19.219 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00030917899999999998 2024-09-11T09:51:19.219 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.219 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.219 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:19.219 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:47.601454+0000", 2024-09-11T09:51:19.220 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.4000999999999999e-05 2024-09-11T09:51:19.220 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.220 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.220 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:19.220 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:47.601475+0000", 2024-09-11T09:51:19.220 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.0305999999999998e-05 2024-09-11T09:51:19.220 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.220 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.220 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:19.220 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:47.601561+0000", 2024-09-11T09:51:19.220 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.6414000000000003e-05 2024-09-11T09:51:19.220 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.220 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.220 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:19.221 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:47.601573+0000", 2024-09-11T09:51:19.221 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1894e-05 2024-09-11T09:51:19.221 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.221 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.221 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:19.221 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:47.602117+0000", 2024-09-11T09:51:19.221 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00054458399999999998 2024-09-11T09:51:19.221 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:19.221 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:19.221 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:19.221 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:19.221 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:19.221 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:51:19.222 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:51:19.092+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:51:19.223 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-11T09:51:19.224 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:51:19.224+0000 7f3c96ec9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31909) UID: 0 2024-09-11T09:51:19.325 INFO:tasks.ceph.osd.2.smithi017.stderr:2024-09-11T09:51:19.326+0000 7fd20b170640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31672) UID: 0 2024-09-11T09:51:19.334 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-11T09:51:19.335 INFO:teuthology.orchestra.run.smithi017.stderr:dumped all 2024-09-11T09:51:19.347 INFO:teuthology.orchestra.run.smithi017.stdout:{"pg_ready":true,"pg_map":{"version":81,"stamp":"2024-09-11T09:51:18.684089+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":161,"num_write_kb":2418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120,"ondisk_log_size":120,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":5,"kb":566231040,"kb_used":182100,"kb_used_data":3264,"kb_used_omap":9,"kb_used_meta":178678,"kb_avail":566048940,"statfs":{"total":579820584960,"available":579634114560,"internally_reserved":0,"allocated":3342336,"data_stored":2342234,"data_compressed":66628,"data_compressed_allocated":90112,"data_compressed_original":1163264,"omap_allocated":9566,"internal_metadata":182966946},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.017072"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":30,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-11T09:51:15.675632+0000","last_change":"2024-09-11T09:50:54.231326+0000","last_active":"2024-09-11T09:51:15.675632+0000","last_peered":"2024-09-11T09:51:15.675632+0000","last_clean":"2024-09-11T09:51:15.675632+0000","last_became_active":"2024-09-11T09:50:54.231067+0000","last_became_peered":"2024-09-11T09:50:54.231067+0000","last_unstale":"2024-09-11T09:51:15.675632+0000","last_undegraded":"2024-09-11T09:51:15.675632+0000","last_fullsized":"2024-09-11T09:51:15.675632+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_clean_scrub_stamp":"2024-09-11T09:50:17.477890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:15:30.630386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00046094899999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":79,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-11T09:51:14.668069+0000","last_change":"2024-09-11T09:50:20.497446+0000","last_active":"2024-09-11T09:51:14.668069+0000","last_peered":"2024-09-11T09:51:14.668069+0000","last_clean":"2024-09-11T09:51:14.668069+0000","last_became_active":"2024-09-11T09:50:18.497481+0000","last_became_peered":"2024-09-11T09:50:18.497481+0000","last_unstale":"2024-09-11T09:51:14.668069+0000","last_undegraded":"2024-09-11T09:51:14.668069+0000","last_fullsized":"2024-09-11T09:51:14.668069+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_clean_scrub_stamp":"2024-09-11T09:50:17.477890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T14:12:09.857529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00050697600000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":91,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-11T09:51:15.675063+0000","last_change":"2024-09-11T09:50:46.593530+0000","last_active":"2024-09-11T09:51:15.675063+0000","last_peered":"2024-09-11T09:51:15.675063+0000","last_clean":"2024-09-11T09:51:15.675063+0000","last_became_active":"2024-09-11T09:50:46.593241+0000","last_became_peered":"2024-09-11T09:50:46.593241+0000","last_unstale":"2024-09-11T09:51:15.675063+0000","last_undegraded":"2024-09-11T09:51:15.675063+0000","last_fullsized":"2024-09-11T09:51:15.675063+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_clean_scrub_stamp":"2024-09-11T09:50:17.477890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T14:12:35.116334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045546699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":89,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-11T09:51:14.668031+0000","last_change":"2024-09-11T09:50:46.593637+0000","last_active":"2024-09-11T09:51:14.668031+0000","last_peered":"2024-09-11T09:51:14.668031+0000","last_clean":"2024-09-11T09:51:14.668031+0000","last_became_active":"2024-09-11T09:50:46.593286+0000","last_became_peered":"2024-09-11T09:50:46.593286+0000","last_unstale":"2024-09-11T09:51:14.668031+0000","last_undegraded":"2024-09-11T09:51:14.668031+0000","last_fullsized":"2024-09-11T09:51:14.668031+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_clean_scrub_stamp":"2024-09-11T09:50:17.477890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T19:11:42.500266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00050337500000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"18'2","reported_seq":80,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-11T09:51:15.221143+0000","last_change":"2024-09-11T09:50:20.501218+0000","last_active":"2024-09-11T09:51:15.221143+0000","last_peered":"2024-09-11T09:51:15.221143+0000","last_clean":"2024-09-11T09:51:15.221143+0000","last_became_active":"2024-09-11T09:50:18.493165+0000","last_became_peered":"2024-09-11T09:50:18.493165+0000","last_unstale":"2024-09-11T09:51:15.221143+0000","last_undegraded":"2024-09-11T09:51:15.221143+0000","last_fullsized":"2024-09-11T09:51:15.221143+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_clean_scrub_stamp":"2024-09-11T09:50:17.477890+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T18:04:31.250555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00094463900000000005,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":76,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-11T09:51:14.727737+0000","last_change":"2024-09-11T09:50:21.030566+0000","last_active":"2024-09-11T09:51:14.727737+0000","last_peered":"2024-09-11T09:51:14.727737+0000","last_clean":"2024-09-11T09:51:14.727737+0000","last_became_active":"2024-09-11T09:50:18.493109+0000","last_became_peered":"2024-09-11T09:50:18.493109+0000","last_unstale":"2024-09-11T09:51:14.727737+0000","last_undegraded":"2024-09-11T09:51:14.727737+0000","last_fullsized":"2024-09-11T09:51:14.727737+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_clean_scrub_stamp":"2024-09-11T09:50:17.477890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:37:16.328368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041008499999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":81,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-11T09:51:15.675063+0000","last_change":"2024-09-11T09:50:20.498284+0000","last_active":"2024-09-11T09:51:15.675063+0000","last_peered":"2024-09-11T09:51:15.675063+0000","last_clean":"2024-09-11T09:51:15.675063+0000","last_became_active":"2024-09-11T09:50:18.493210+0000","last_became_peered":"2024-09-11T09:50:18.493210+0000","last_unstale":"2024-09-11T09:51:15.675063+0000","last_undegraded":"2024-09-11T09:51:15.675063+0000","last_fullsized":"2024-09-11T09:51:15.675063+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_clean_scrub_stamp":"2024-09-11T09:50:17.477890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T18:16:13.887681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079749499999999995,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"16'1","reported_seq":79,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-11T09:51:15.221152+0000","last_change":"2024-09-11T09:50:20.500687+0000","last_active":"2024-09-11T09:51:15.221152+0000","last_peered":"2024-09-11T09:51:15.221152+0000","last_clean":"2024-09-11T09:51:15.221152+0000","last_became_active":"2024-09-11T09:50:18.493650+0000","last_became_peered":"2024-09-11T09:50:18.493650+0000","last_unstale":"2024-09-11T09:51:15.221152+0000","last_undegraded":"2024-09-11T09:51:15.221152+0000","last_fullsized":"2024-09-11T09:51:15.221152+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_clean_scrub_stamp":"2024-09-11T09:50:17.477890+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T14:26:32.838143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041261100000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"13'117","reported_seq":258,"reported_epoch":49,"state":"active+clean","last_fresh":"2024-09-11T09:51:15.675073+0000","last_change":"2024-09-11T09:50:49.606072+0000","last_active":"2024-09-11T09:51:15.675073+0000","last_peered":"2024-09-11T09:51:15.675073+0000","last_clean":"2024-09-11T09:51:15.675073+0000","last_became_active":"2024-09-11T09:50:49.605824+0000","last_became_peered":"2024-09-11T09:50:49.605824+0000","last_unstale":"2024-09-11T09:51:15.675073+0000","last_undegraded":"2024-09-11T09:51:15.675073+0000","last_fullsized":"2024-09-11T09:51:15.675073+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:50:10.457262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:50:10.457262+0000","last_clean_scrub_stamp":"2024-09-11T09:50:10.457262+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:02:29.081582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1771104,"data_compressed":100760,"data_compressed_allocated":135168,"data_compressed_original":1753088,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":12,"seq":51539607569,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31800,"kb_used_data":604,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340040,"statfs":{"total":96636764160,"available":96604200960,"internally_reserved":0,"allocated":618496,"data_stored":790098,"data_compressed":32496,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":12,"seq":51539607568,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31784,"kb_used_data":588,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340056,"statfs":{"total":96636764160,"available":96604217344,"internally_reserved":0,"allocated":602112,"data_stored":790098,"data_compressed":34132,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":11,"seq":47244640273,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27536,"kb_used_data":508,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344304,"statfs":{"total":96636764160,"available":96608567296,"internally_reserved":0,"allocated":520192,"data_stored":185904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":11,"seq":47244640273,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27548,"kb_used_data":516,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344292,"statfs":{"total":96636764160,"available":96608555008,"internally_reserved":0,"allocated":528384,"data_stored":190500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640272,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31720,"kb_used_data":528,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340120,"statfs":{"total":96636764160,"available":96604282880,"internally_reserved":0,"allocated":540672,"data_stored":195115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640272,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31712,"kb_used_data":520,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340128,"statfs":{"total":96636764160,"available":96604291072,"internally_reserved":0,"allocated":532480,"data_stored":190519,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":34132,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":34132,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32496,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T09:51:19.348 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-09-11T09:51:19.348 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-11T09:51:19.348 INFO:tasks.thrashosds.thrasher:inject_pause on osd.2 2024-09-11T09:51:19.348 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-11T09:51:19.348 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-11T09:51:19.349 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok config set filestore_inject_stall 3 2024-09-11T09:51:19.372 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:51:19.372 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [], 2024-09-11T09:51:19.372 INFO:teuthology.orchestra.run.smithi017.stdout: "num_ops": 0 2024-09-11T09:51:19.372 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:51:19.381 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-11T09:51:19.465 INFO:tasks.ceph.osd.0.smithi017.stderr:2024-09-11T09:51:19.466+0000 7fb73c0f1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31665) UID: 0 2024-09-11T09:51:19.480 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:51:19.480 INFO:teuthology.orchestra.run.smithi017.stdout: "success": "" 2024-09-11T09:51:19.480 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:51:19.527 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:51:19.527+0000 7fdec5849640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31907) UID: 0 2024-09-11T09:51:19.529 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:51:19.529 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [], 2024-09-11T09:51:19.529 INFO:teuthology.orchestra.run.smithi017.stdout: "complaint_time": 30, 2024-09-11T09:51:19.529 INFO:teuthology.orchestra.run.smithi017.stdout: "num_blocked_ops": 0 2024-09-11T09:51:19.529 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:51:19.539 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-11T09:51:19.627 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:51:19.628+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:51:19.687 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:51:19.687 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 20, 2024-09-11T09:51:19.687 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 600, 2024-09-11T09:51:19.687 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [ 2024-09-11T09:51:19.687 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.688 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4230.0:5 2.2 e17/15 2:5cea7035:::rbd_info:head v 17'1, mlcod=0'0)", 2024-09-11T09:51:19.688 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:19.494757+0000", 2024-09-11T09:51:19.688 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 60.194013579999996, 2024-09-11T09:51:19.688 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00082098399999999997, 2024-09-11T09:51:19.688 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:19.688 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:19.688 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:19.688 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:19.688 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.688 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:19.688 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:19.494757+0000", 2024-09-11T09:51:19.688 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:19.689 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.689 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.689 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:19.689 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:19.494757+0000", 2024-09-11T09:51:19.689 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:19.689 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.689 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.689 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:19.689 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:19.494757+0000", 2024-09-11T09:51:19.689 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.97e-07 2024-09-11T09:51:19.689 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.689 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.689 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:19.690 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:19.494763+0000", 2024-09-11T09:51:19.690 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.3709999999999996e-06 2024-09-11T09:51:19.690 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.690 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.690 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:19.690 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:19.494778+0000", 2024-09-11T09:51:19.690 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.4960000000000001e-05 2024-09-11T09:51:19.690 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.690 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.690 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:19.690 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:19.494801+0000", 2024-09-11T09:51:19.690 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.2388999999999999e-05 2024-09-11T09:51:19.690 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.690 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.691 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:19.691 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:19.494859+0000", 2024-09-11T09:51:19.691 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.7961999999999998e-05 2024-09-11T09:51:19.691 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.691 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.691 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:19.691 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:19.494883+0000", 2024-09-11T09:51:19.691 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.4488999999999999e-05 2024-09-11T09:51:19.691 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.691 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.691 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:19.691 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:19.495534+0000", 2024-09-11T09:51:19.691 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00065075500000000004 2024-09-11T09:51:19.691 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.692 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.692 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:19.692 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:19.495578+0000", 2024-09-11T09:51:19.692 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.3860999999999999e-05 2024-09-11T09:51:19.692 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:19.692 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:19.692 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:19.692 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.692 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.692 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4230.0:7 2.2 e18/15 2:5cea7035:::rbd_info:head v 18'2, mlcod=0'0)", 2024-09-11T09:51:19.692 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:20.500782+0000", 2024-09-11T09:51:19.692 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 59.187988699999998, 2024-09-11T09:51:19.692 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000790185, 2024-09-11T09:51:19.693 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:19.693 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:19.693 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:19.693 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:19.693 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.693 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:19.693 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:20.500782+0000", 2024-09-11T09:51:19.693 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:19.693 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.693 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.693 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:19.693 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:20.500782+0000", 2024-09-11T09:51:19.693 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:19.693 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.694 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.694 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:19.694 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:20.500782+0000", 2024-09-11T09:51:19.694 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.3299999999999998e-07 2024-09-11T09:51:19.694 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.694 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.694 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:19.694 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:20.500799+0000", 2024-09-11T09:51:19.694 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6830999999999999e-05 2024-09-11T09:51:19.694 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.694 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.694 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:19.694 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:20.500812+0000", 2024-09-11T09:51:19.695 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3013999999999999e-05 2024-09-11T09:51:19.695 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.695 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.695 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:19.695 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:20.500824+0000", 2024-09-11T09:51:19.695 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2593e-05 2024-09-11T09:51:19.695 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.695 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.695 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:19.695 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:20.500895+0000", 2024-09-11T09:51:19.695 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.1018000000000003e-05 2024-09-11T09:51:19.695 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.696 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.696 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:19.696 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:20.500912+0000", 2024-09-11T09:51:19.696 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6484e-05 2024-09-11T09:51:19.696 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.696 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.696 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:19.696 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:20.501531+0000", 2024-09-11T09:51:19.696 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00061903399999999997 2024-09-11T09:51:19.696 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:19.696 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:19.696 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:19.696 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:20.501572+0000", 2024-09-11T09:51:19.697 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.0877999999999997e-05 2024-09-11T09:51:19.697 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:19.697 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:19.697 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:19.697 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:19.697 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:19.697 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:51:19.698 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-11T09:51:19.728 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:51:19.728+0000 7f3c96ec9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31909) UID: 0 2024-09-11T09:51:19.829 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:51:19.830+0000 7f3c96ec9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31909) UID: 0 2024-09-11T09:51:19.845 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:51:19.845 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [], 2024-09-11T09:51:19.845 INFO:teuthology.orchestra.run.smithi017.stdout: "num_ops": 0 2024-09-11T09:51:19.846 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:51:19.854 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-11T09:51:19.930 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:51:19.930+0000 7f3c96ec9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31909) UID: 0 2024-09-11T09:51:19.961 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:51:19.961 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [], 2024-09-11T09:51:19.962 INFO:teuthology.orchestra.run.smithi017.stdout: "complaint_time": 30, 2024-09-11T09:51:19.962 INFO:teuthology.orchestra.run.smithi017.stdout: "num_blocked_ops": 0 2024-09-11T09:51:19.962 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:51:19.970 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-11T09:51:20.031 INFO:tasks.ceph.osd.2.smithi017.stderr:2024-09-11T09:51:20.032+0000 7fd20b170640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31672) UID: 0 2024-09-11T09:51:20.078 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:51:20.078 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 20, 2024-09-11T09:51:20.078 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 600, 2024-09-11T09:51:20.078 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [ 2024-09-11T09:51:20.078 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:20.078 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4230.0:1 2.3 e16/15 2:c4c92e5a:::rbd_trash:head v 16'1, mlcod=0'0)", 2024-09-11T09:51:20.078 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:18.494378+0000", 2024-09-11T09:51:20.079 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 61.585212536999997, 2024-09-11T09:51:20.079 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00089752199999999999, 2024-09-11T09:51:20.079 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:20.079 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:20.079 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:20.079 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:20.079 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:20.079 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:20.079 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:18.494378+0000", 2024-09-11T09:51:20.079 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:20.079 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:20.079 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:20.079 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:20.080 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:18.494378+0000", 2024-09-11T09:51:20.080 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:20.080 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:20.080 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:20.080 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:20.080 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:18.494378+0000", 2024-09-11T09:51:20.080 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.9099999999999999e-07 2024-09-11T09:51:20.080 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:20.080 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:20.080 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:20.080 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:18.494385+0000", 2024-09-11T09:51:20.080 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.8129999999999998e-06 2024-09-11T09:51:20.081 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:20.081 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:20.081 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:20.081 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:18.494416+0000", 2024-09-11T09:51:20.081 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.0654000000000002e-05 2024-09-11T09:51:20.081 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:20.081 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:20.081 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:20.081 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:18.494430+0000", 2024-09-11T09:51:20.081 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3754000000000001e-05 2024-09-11T09:51:20.081 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:20.081 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:20.081 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:20.081 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:18.494524+0000", 2024-09-11T09:51:20.082 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.4896999999999996e-05 2024-09-11T09:51:20.082 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:20.082 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:20.082 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:20.082 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:18.494558+0000", 2024-09-11T09:51:20.082 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.3900999999999999e-05 2024-09-11T09:51:20.082 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:20.082 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:20.082 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:20.082 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:18.495231+0000", 2024-09-11T09:51:20.082 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00067233500000000003 2024-09-11T09:51:20.082 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:20.082 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:20.083 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:20.083 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:18.495275+0000", 2024-09-11T09:51:20.083 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.4777000000000002e-05 2024-09-11T09:51:20.083 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:20.083 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:20.083 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:20.083 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:20.083 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:20.083 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:51:20.087 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-11T09:51:20.132 INFO:tasks.ceph.osd.2.smithi017.stderr:2024-09-11T09:51:20.133+0000 7fd20b170640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31672) UID: 0 2024-09-11T09:51:20.203 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-11T09:51:20.203 INFO:teuthology.orchestra.run.smithi096.stdout: "ops": [], 2024-09-11T09:51:20.203 INFO:teuthology.orchestra.run.smithi096.stdout: "num_ops": 0 2024-09-11T09:51:20.203 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-11T09:51:20.212 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-11T09:51:20.233 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:51:20.234+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:51:20.287 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T09:51:20.316 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-11T09:51:20.316 INFO:teuthology.orchestra.run.smithi096.stdout: "ops": [], 2024-09-11T09:51:20.317 INFO:teuthology.orchestra.run.smithi096.stdout: "complaint_time": 30, 2024-09-11T09:51:20.317 INFO:teuthology.orchestra.run.smithi096.stdout: "num_blocked_ops": 0 2024-09-11T09:51:20.317 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-11T09:51:20.325 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-11T09:51:20.368 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-09-11T09:51:20.369+0000 7f7b6c7d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31904) UID: 0 2024-09-11T09:51:20.412 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-11T09:51:20.412 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T09:51:20.428 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-11T09:51:20.429 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T09:51:20.430 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-11T09:51:20.430 INFO:teuthology.orchestra.run.smithi096.stdout: "size": 20, 2024-09-11T09:51:20.430 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 600, 2024-09-11T09:51:20.430 INFO:teuthology.orchestra.run.smithi096.stdout: "ops": [ 2024-09-11T09:51:20.430 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.430 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:74 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:51:20.430 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.539067+0000", 2024-09-11T09:51:20.431 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 68.890737556000005, 2024-09-11T09:51:20.431 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.006945957, 2024-09-11T09:51:20.431 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:20.431 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:20.431 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:20.431 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:20.431 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:51:20.431 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:51:20.431 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 74 2024-09-11T09:51:20.431 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.431 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:20.431 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.431 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:20.432 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.539067+0000", 2024-09-11T09:51:20.432 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:20.432 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.432 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.432 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:20.432 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.539067+0000", 2024-09-11T09:51:20.432 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:20.432 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.432 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.432 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:20.432 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.539068+0000", 2024-09-11T09:51:20.432 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.6000000000000003e-07 2024-09-11T09:51:20.432 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.433 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.433 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:20.433 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.539072+0000", 2024-09-11T09:51:20.433 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.0450000000000002e-06 2024-09-11T09:51:20.433 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.433 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.433 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:20.433 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.539078+0000", 2024-09-11T09:51:20.433 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.6889999999999998e-06 2024-09-11T09:51:20.433 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.433 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.434 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:20.434 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.539083+0000", 2024-09-11T09:51:20.434 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.7180000000000004e-06 2024-09-11T09:51:20.434 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.434 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.434 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:20.434 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.540230+0000", 2024-09-11T09:51:20.434 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0011466250000000001 2024-09-11T09:51:20.434 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.434 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.434 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:20.434 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.540279+0000", 2024-09-11T09:51:20.434 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.9667999999999999e-05 2024-09-11T09:51:20.435 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.435 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.435 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:20.435 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.540357+0000", 2024-09-11T09:51:20.435 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.7863999999999999e-05 2024-09-11T09:51:20.435 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.435 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.435 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:20.435 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.541086+0000", 2024-09-11T09:51:20.435 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00072921100000000001 2024-09-11T09:51:20.435 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.435 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.435 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:20.435 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.545949+0000", 2024-09-11T09:51:20.436 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0048626040000000004 2024-09-11T09:51:20.436 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.436 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.436 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:20.436 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.545989+0000", 2024-09-11T09:51:20.436 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.0259000000000002e-05 2024-09-11T09:51:20.436 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.437 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.437 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:20.437 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.546013+0000", 2024-09-11T09:51:20.437 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.3614000000000001e-05 2024-09-11T09:51:20.437 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:20.437 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:20.437 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:20.437 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.437 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.437 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:108 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:51:20.437 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.711510+0000", 2024-09-11T09:51:20.438 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 68.718294384000004, 2024-09-11T09:51:20.438 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0054615649999999998, 2024-09-11T09:51:20.438 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:20.438 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:20.438 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:20.438 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:20.438 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:51:20.438 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:51:20.438 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 108 2024-09-11T09:51:20.438 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.438 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:20.438 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.439 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:20.439 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711510+0000", 2024-09-11T09:51:20.439 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:20.439 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.439 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.439 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:20.439 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711510+0000", 2024-09-11T09:51:20.439 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:20.439 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.439 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.439 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:20.439 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711511+0000", 2024-09-11T09:51:20.439 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.1999999999999998e-07 2024-09-11T09:51:20.440 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.440 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.440 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:20.440 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711516+0000", 2024-09-11T09:51:20.440 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.4140000000000002e-06 2024-09-11T09:51:20.440 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.440 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.440 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:20.440 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711527+0000", 2024-09-11T09:51:20.440 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.0169e-05 2024-09-11T09:51:20.440 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.440 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.440 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:20.441 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711535+0000", 2024-09-11T09:51:20.441 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.3299999999999999e-06 2024-09-11T09:51:20.441 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.441 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.441 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:20.441 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.713763+0000", 2024-09-11T09:51:20.441 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0022281079999999999 2024-09-11T09:51:20.441 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.441 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.441 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:20.441 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.713812+0000", 2024-09-11T09:51:20.441 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.8544000000000001e-05 2024-09-11T09:51:20.441 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.442 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.442 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:20.442 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.713909+0000", 2024-09-11T09:51:20.442 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.7113000000000003e-05 2024-09-11T09:51:20.442 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.442 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.442 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:20.442 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.716873+0000", 2024-09-11T09:51:20.442 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0029642710000000001 2024-09-11T09:51:20.442 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.442 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.442 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:20.443 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.716918+0000", 2024-09-11T09:51:20.443 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.5447e-05 2024-09-11T09:51:20.443 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.443 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.443 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:20.443 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.716947+0000", 2024-09-11T09:51:20.443 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.8181e-05 2024-09-11T09:51:20.443 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.443 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.443 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:20.443 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.716972+0000", 2024-09-11T09:51:20.443 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.5068000000000001e-05 2024-09-11T09:51:20.443 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:20.444 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:20.444 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:20.444 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.444 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.444 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:109 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:51:20.444 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.711555+0000", 2024-09-11T09:51:20.444 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 68.718249134999994, 2024-09-11T09:51:20.444 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.005479902, 2024-09-11T09:51:20.444 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:20.444 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:20.444 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:20.444 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:20.445 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:51:20.445 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:51:20.445 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 109 2024-09-11T09:51:20.445 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.445 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:20.445 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.445 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:20.445 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711555+0000", 2024-09-11T09:51:20.445 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:20.445 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.445 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.445 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:20.445 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711555+0000", 2024-09-11T09:51:20.446 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:20.446 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.446 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.446 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:20.446 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711556+0000", 2024-09-11T09:51:20.446 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.3200000000000005e-07 2024-09-11T09:51:20.446 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.446 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.446 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:20.446 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711601+0000", 2024-09-11T09:51:20.446 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.5349000000000002e-05 2024-09-11T09:51:20.446 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.446 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.447 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:20.447 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711612+0000", 2024-09-11T09:51:20.447 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.0701e-05 2024-09-11T09:51:20.447 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.447 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.447 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:20.447 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711619+0000", 2024-09-11T09:51:20.447 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.7610000000000002e-06 2024-09-11T09:51:20.447 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.447 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.447 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:20.447 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.714371+0000", 2024-09-11T09:51:20.447 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0027526260000000002 2024-09-11T09:51:20.447 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.448 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.448 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:20.448 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.714417+0000", 2024-09-11T09:51:20.448 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.5893999999999998e-05 2024-09-11T09:51:20.448 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.448 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.448 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:20.448 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.714496+0000", 2024-09-11T09:51:20.448 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.8539000000000001e-05 2024-09-11T09:51:20.448 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.448 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.448 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:20.448 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.716910+0000", 2024-09-11T09:51:20.449 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0024137640000000001 2024-09-11T09:51:20.449 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.449 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.449 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:20.449 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.716989+0000", 2024-09-11T09:51:20.449 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.9198999999999998e-05 2024-09-11T09:51:20.449 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.449 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.449 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:20.449 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.717016+0000", 2024-09-11T09:51:20.449 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.6699e-05 2024-09-11T09:51:20.449 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.449 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.450 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:20.450 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.717035+0000", 2024-09-11T09:51:20.450 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.9738e-05 2024-09-11T09:51:20.450 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:20.450 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:20.450 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:20.450 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.450 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.450 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:110 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:51:20.450 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.711636+0000", 2024-09-11T09:51:20.450 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 68.718168675000001, 2024-09-11T09:51:20.450 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0067613070000000003, 2024-09-11T09:51:20.450 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:20.450 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:20.451 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:20.451 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:20.451 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:51:20.451 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:51:20.451 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 110 2024-09-11T09:51:20.451 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.451 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:20.451 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.451 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:20.451 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711636+0000", 2024-09-11T09:51:20.451 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:20.451 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.451 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.452 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:20.452 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711636+0000", 2024-09-11T09:51:20.452 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:20.452 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.452 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.452 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:20.452 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711637+0000", 2024-09-11T09:51:20.452 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.9999999999999997e-07 2024-09-11T09:51:20.452 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.452 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.452 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:20.452 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711641+0000", 2024-09-11T09:51:20.453 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.2570000000000001e-06 2024-09-11T09:51:20.453 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.453 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.453 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:20.453 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711651+0000", 2024-09-11T09:51:20.453 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.0285999999999999e-05 2024-09-11T09:51:20.453 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.453 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.453 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:20.453 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711658+0000", 2024-09-11T09:51:20.453 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.3540000000000002e-06 2024-09-11T09:51:20.453 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.453 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.454 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:20.454 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.716475+0000", 2024-09-11T09:51:20.454 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0048162819999999999 2024-09-11T09:51:20.454 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.454 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.454 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:20.454 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.716526+0000", 2024-09-11T09:51:20.454 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.0905000000000001e-05 2024-09-11T09:51:20.454 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.454 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.454 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:20.454 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.716601+0000", 2024-09-11T09:51:20.454 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.5283000000000004e-05 2024-09-11T09:51:20.454 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.455 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.455 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:20.455 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.717933+0000", 2024-09-11T09:51:20.455 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001331622 2024-09-11T09:51:20.455 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.455 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.455 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:20.455 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718332+0000", 2024-09-11T09:51:20.455 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00039909899999999998 2024-09-11T09:51:20.455 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.455 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.455 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:20.456 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718363+0000", 2024-09-11T09:51:20.456 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.1636e-05 2024-09-11T09:51:20.456 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.456 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.456 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:20.456 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718397+0000", 2024-09-11T09:51:20.456 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.3883000000000001e-05 2024-09-11T09:51:20.456 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:20.456 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:20.456 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:20.456 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.456 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.456 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:111 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:51:20.456 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.711668+0000", 2024-09-11T09:51:20.457 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 68.718136244999997, 2024-09-11T09:51:20.457 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0077145570000000004, 2024-09-11T09:51:20.457 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:20.457 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:20.457 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:20.457 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:20.457 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:51:20.457 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:51:20.457 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 111 2024-09-11T09:51:20.457 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.457 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:20.457 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.458 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:20.458 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711668+0000", 2024-09-11T09:51:20.458 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:20.458 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.458 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.458 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:20.458 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711668+0000", 2024-09-11T09:51:20.458 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:20.458 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.458 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.458 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:20.458 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711669+0000", 2024-09-11T09:51:20.458 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.99e-07 2024-09-11T09:51:20.459 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.459 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.459 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:20.459 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711673+0000", 2024-09-11T09:51:20.459 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.7440000000000001e-06 2024-09-11T09:51:20.459 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.459 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.459 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:20.459 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711686+0000", 2024-09-11T09:51:20.459 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.3317e-05 2024-09-11T09:51:20.459 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.459 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.459 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:20.459 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711697+0000", 2024-09-11T09:51:20.460 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1022000000000001e-05 2024-09-11T09:51:20.460 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.460 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.460 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:20.460 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.717065+0000", 2024-09-11T09:51:20.460 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.005367531 2024-09-11T09:51:20.460 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.460 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.460 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:20.460 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.717114+0000", 2024-09-11T09:51:20.460 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.9707999999999999e-05 2024-09-11T09:51:20.460 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.460 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.461 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:20.461 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.717190+0000", 2024-09-11T09:51:20.461 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.5394000000000004e-05 2024-09-11T09:51:20.461 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.461 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.461 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:20.461 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718793+0000", 2024-09-11T09:51:20.461 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001603653 2024-09-11T09:51:20.461 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.461 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.461 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:20.461 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719329+0000", 2024-09-11T09:51:20.461 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00053543799999999995 2024-09-11T09:51:20.462 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.462 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.462 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:20.462 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719359+0000", 2024-09-11T09:51:20.462 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.0420999999999999e-05 2024-09-11T09:51:20.462 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.462 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.462 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:20.462 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719383+0000", 2024-09-11T09:51:20.462 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.3730000000000001e-05 2024-09-11T09:51:20.462 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:20.462 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:20.463 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:20.463 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.463 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.463 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:112 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:51:20.463 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.711712+0000", 2024-09-11T09:51:20.463 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 68.718092685000002, 2024-09-11T09:51:20.463 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.008162084, 2024-09-11T09:51:20.463 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:20.463 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:20.463 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:20.463 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:20.463 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:51:20.463 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:51:20.463 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 112 2024-09-11T09:51:20.464 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.464 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:20.464 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.464 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:20.464 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711712+0000", 2024-09-11T09:51:20.464 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:20.464 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.464 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.464 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:20.464 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711712+0000", 2024-09-11T09:51:20.464 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:20.464 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.464 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.464 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:20.465 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711712+0000", 2024-09-11T09:51:20.465 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.6000000000000004e-07 2024-09-11T09:51:20.465 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.465 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.465 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:20.465 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711808+0000", 2024-09-11T09:51:20.465 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.5977999999999995e-05 2024-09-11T09:51:20.465 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.465 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.465 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:20.465 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711819+0000", 2024-09-11T09:51:20.465 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.0755e-05 2024-09-11T09:51:20.466 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.466 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.466 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:20.466 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711826+0000", 2024-09-11T09:51:20.466 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.7970000000000003e-06 2024-09-11T09:51:20.466 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.466 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.466 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:20.466 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.717435+0000", 2024-09-11T09:51:20.466 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0056090239999999998 2024-09-11T09:51:20.466 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.466 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.467 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:20.467 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.717496+0000", 2024-09-11T09:51:20.467 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.0543e-05 2024-09-11T09:51:20.467 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.467 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.467 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:20.467 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.717591+0000", 2024-09-11T09:51:20.467 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.5667000000000004e-05 2024-09-11T09:51:20.467 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.467 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.467 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:20.467 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718831+0000", 2024-09-11T09:51:20.467 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001240204 2024-09-11T09:51:20.467 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.468 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.468 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:20.468 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719819+0000", 2024-09-11T09:51:20.468 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00098751500000000001 2024-09-11T09:51:20.468 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.468 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.468 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:20.468 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719850+0000", 2024-09-11T09:51:20.468 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.1535999999999998e-05 2024-09-11T09:51:20.468 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.468 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.468 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:20.468 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719874+0000", 2024-09-11T09:51:20.469 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.3504999999999999e-05 2024-09-11T09:51:20.469 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:20.469 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:20.469 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:20.469 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.469 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.469 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:113 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:51:20.469 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.711842+0000", 2024-09-11T09:51:20.469 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 68.717962185999994, 2024-09-11T09:51:20.469 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0081160099999999999, 2024-09-11T09:51:20.469 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:20.469 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:20.469 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:20.469 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:20.470 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:51:20.470 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:51:20.470 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 113 2024-09-11T09:51:20.470 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.470 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:20.470 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.470 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:20.470 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711842+0000", 2024-09-11T09:51:20.470 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:20.470 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.470 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.470 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:20.470 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711842+0000", 2024-09-11T09:51:20.470 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:20.471 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.471 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.471 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:20.471 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711843+0000", 2024-09-11T09:51:20.471 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.4399999999999999e-07 2024-09-11T09:51:20.471 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.471 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.471 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:20.471 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711848+0000", 2024-09-11T09:51:20.471 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.4159999999999997e-06 2024-09-11T09:51:20.471 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.471 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.471 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:20.472 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711856+0000", 2024-09-11T09:51:20.472 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.7870000000000007e-06 2024-09-11T09:51:20.472 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.472 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.472 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:20.472 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711862+0000", 2024-09-11T09:51:20.472 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.3689999999999998e-06 2024-09-11T09:51:20.472 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.472 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.472 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:20.472 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.717965+0000", 2024-09-11T09:51:20.472 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0061028899999999997 2024-09-11T09:51:20.472 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.473 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.473 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:20.473 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718023+0000", 2024-09-11T09:51:20.473 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.8313000000000002e-05 2024-09-11T09:51:20.473 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.473 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.473 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:20.473 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718109+0000", 2024-09-11T09:51:20.473 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.6205000000000006e-05 2024-09-11T09:51:20.473 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.473 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.473 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:20.473 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719808+0000", 2024-09-11T09:51:20.474 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0016990060000000001 2024-09-11T09:51:20.474 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.474 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.474 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:20.474 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719883+0000", 2024-09-11T09:51:20.474 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.4624999999999998e-05 2024-09-11T09:51:20.474 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.474 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.474 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:20.474 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719910+0000", 2024-09-11T09:51:20.474 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.7512e-05 2024-09-11T09:51:20.474 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.474 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.474 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:20.475 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719958+0000", 2024-09-11T09:51:20.475 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.8143000000000003e-05 2024-09-11T09:51:20.475 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:20.475 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:20.475 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:20.475 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.475 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.475 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:51:20.475 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.711887+0000", 2024-09-11T09:51:20.475 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 68.717917572000005, 2024-09-11T09:51:20.475 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0086063589999999992, 2024-09-11T09:51:20.475 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:20.475 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:20.476 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:20.476 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:20.476 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:51:20.476 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:51:20.476 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 114 2024-09-11T09:51:20.476 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.476 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:20.476 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.476 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:20.476 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711887+0000", 2024-09-11T09:51:20.476 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:20.476 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.476 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.477 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:20.477 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711887+0000", 2024-09-11T09:51:20.477 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:20.477 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.477 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.477 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:20.477 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711887+0000", 2024-09-11T09:51:20.477 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.1500000000000005e-07 2024-09-11T09:51:20.477 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.477 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.477 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:20.477 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711897+0000", 2024-09-11T09:51:20.477 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.1900000000000001e-06 2024-09-11T09:51:20.477 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.478 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.478 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:20.478 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711902+0000", 2024-09-11T09:51:20.478 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.2349999999999999e-06 2024-09-11T09:51:20.478 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.478 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.478 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:20.478 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711906+0000", 2024-09-11T09:51:20.478 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.7129999999999999e-06 2024-09-11T09:51:20.478 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.478 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.478 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:20.478 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718429+0000", 2024-09-11T09:51:20.479 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0065231960000000002 2024-09-11T09:51:20.479 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.479 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.479 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:20.479 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718480+0000", 2024-09-11T09:51:20.479 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.1161e-05 2024-09-11T09:51:20.479 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.479 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.479 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:20.479 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718558+0000", 2024-09-11T09:51:20.479 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.7519000000000001e-05 2024-09-11T09:51:20.479 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.479 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.479 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:20.480 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720038+0000", 2024-09-11T09:51:20.480 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001480016 2024-09-11T09:51:20.480 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.480 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.480 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:20.480 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720417+0000", 2024-09-11T09:51:20.480 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00037943699999999999 2024-09-11T09:51:20.480 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.480 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.480 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:20.480 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720452+0000", 2024-09-11T09:51:20.480 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.4595000000000003e-05 2024-09-11T09:51:20.480 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.480 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.481 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:20.481 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720493+0000", 2024-09-11T09:51:20.481 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.1782000000000001e-05 2024-09-11T09:51:20.481 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:20.481 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:20.481 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:20.481 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.481 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.481 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:51:20.481 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.711912+0000", 2024-09-11T09:51:20.481 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 68.717892903000006, 2024-09-11T09:51:20.481 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0086864750000000008, 2024-09-11T09:51:20.481 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:20.482 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:20.482 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:20.482 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:20.482 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:51:20.482 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:51:20.482 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 115 2024-09-11T09:51:20.482 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.482 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:20.482 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.482 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:20.482 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711912+0000", 2024-09-11T09:51:20.482 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:20.482 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.483 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.483 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:20.483 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711912+0000", 2024-09-11T09:51:20.483 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:20.483 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.483 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.483 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:20.483 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711912+0000", 2024-09-11T09:51:20.483 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.3900000000000001e-07 2024-09-11T09:51:20.483 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.483 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.483 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:20.483 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711969+0000", 2024-09-11T09:51:20.483 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.7501999999999999e-05 2024-09-11T09:51:20.484 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.484 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.484 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:20.484 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711986+0000", 2024-09-11T09:51:20.484 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.6402999999999999e-05 2024-09-11T09:51:20.484 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.484 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.484 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:20.484 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711990+0000", 2024-09-11T09:51:20.484 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.929e-06 2024-09-11T09:51:20.484 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.484 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.484 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:20.485 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718861+0000", 2024-09-11T09:51:20.485 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0068710530000000002 2024-09-11T09:51:20.485 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.485 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.485 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:20.485 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718908+0000", 2024-09-11T09:51:20.485 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.7431e-05 2024-09-11T09:51:20.485 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.485 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.485 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:20.485 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718984+0000", 2024-09-11T09:51:20.485 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.5500999999999994e-05 2024-09-11T09:51:20.486 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.486 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.486 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:20.486 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720500+0000", 2024-09-11T09:51:20.486 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0015158820000000001 2024-09-11T09:51:20.486 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.486 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.486 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:20.486 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720536+0000", 2024-09-11T09:51:20.486 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.6118000000000002e-05 2024-09-11T09:51:20.486 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.486 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.486 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:20.486 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720570+0000", 2024-09-11T09:51:20.487 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.4387999999999997e-05 2024-09-11T09:51:20.487 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.487 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.487 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:20.487 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720598+0000", 2024-09-11T09:51:20.487 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.8028999999999999e-05 2024-09-11T09:51:20.487 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:20.487 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:20.487 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:20.487 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.487 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.487 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:51:20.487 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.711998+0000", 2024-09-11T09:51:20.487 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 68.717806723999999, 2024-09-11T09:51:20.487 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.010785813999999999, 2024-09-11T09:51:20.488 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:20.488 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:20.488 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:20.488 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:20.488 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:51:20.488 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:51:20.488 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 116 2024-09-11T09:51:20.488 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.488 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:20.488 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.488 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:20.488 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711998+0000", 2024-09-11T09:51:20.489 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:20.489 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.489 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.489 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:20.489 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711998+0000", 2024-09-11T09:51:20.489 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:20.489 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.489 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.489 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:20.489 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711998+0000", 2024-09-11T09:51:20.489 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.3300000000000003e-07 2024-09-11T09:51:20.489 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.490 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.490 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:20.490 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712001+0000", 2024-09-11T09:51:20.490 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.9349999999999999e-06 2024-09-11T09:51:20.490 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.490 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.490 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:20.490 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712006+0000", 2024-09-11T09:51:20.490 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.1630000000000004e-06 2024-09-11T09:51:20.490 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.490 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.490 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:20.490 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712010+0000", 2024-09-11T09:51:20.491 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.574e-06 2024-09-11T09:51:20.491 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.491 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.491 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:20.491 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719415+0000", 2024-09-11T09:51:20.491 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0074049099999999998 2024-09-11T09:51:20.491 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.491 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.491 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:20.491 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719465+0000", 2024-09-11T09:51:20.491 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.0129000000000001e-05 2024-09-11T09:51:20.491 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.491 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.492 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:20.492 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719545+0000", 2024-09-11T09:51:20.492 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.0253999999999994e-05 2024-09-11T09:51:20.492 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.492 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.492 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:20.492 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720636+0000", 2024-09-11T09:51:20.492 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0010909330000000001 2024-09-11T09:51:20.492 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.492 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.492 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:20.492 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.722707+0000", 2024-09-11T09:51:20.493 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0020713300000000001 2024-09-11T09:51:20.493 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.493 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.493 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:20.493 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.722743+0000", 2024-09-11T09:51:20.493 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.5351000000000001e-05 2024-09-11T09:51:20.493 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.493 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.493 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:20.493 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.722784+0000", 2024-09-11T09:51:20.493 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.0802e-05 2024-09-11T09:51:20.493 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:20.494 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:20.494 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:20.494 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.494 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.494 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:51:20.494 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.712014+0000", 2024-09-11T09:51:20.494 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 68.717790683999993, 2024-09-11T09:51:20.494 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.013420938, 2024-09-11T09:51:20.494 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:20.494 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:20.494 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:20.494 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:20.494 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:51:20.495 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:51:20.495 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 117 2024-09-11T09:51:20.495 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.495 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:20.495 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.495 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:20.495 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712014+0000", 2024-09-11T09:51:20.495 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:20.495 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.495 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.495 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:20.495 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712014+0000", 2024-09-11T09:51:20.495 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:20.495 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.496 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.496 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:20.496 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712014+0000", 2024-09-11T09:51:20.496 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.2000000000000001e-07 2024-09-11T09:51:20.496 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.496 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.496 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:20.496 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712016+0000", 2024-09-11T09:51:20.496 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.2309999999999999e-06 2024-09-11T09:51:20.496 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.496 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.496 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:20.497 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712020+0000", 2024-09-11T09:51:20.497 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.3830000000000001e-06 2024-09-11T09:51:20.497 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.497 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.497 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:20.497 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712023+0000", 2024-09-11T09:51:20.497 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.3340000000000002e-06 2024-09-11T09:51:20.497 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.497 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.497 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:20.497 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720083+0000", 2024-09-11T09:51:20.497 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0080601189999999993 2024-09-11T09:51:20.497 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.498 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.498 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:20.498 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720125+0000", 2024-09-11T09:51:20.498 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.2029e-05 2024-09-11T09:51:20.498 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.498 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.498 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:20.498 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720205+0000", 2024-09-11T09:51:20.498 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.0010000000000001e-05 2024-09-11T09:51:20.498 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.498 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.498 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:20.498 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720849+0000", 2024-09-11T09:51:20.498 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00064369899999999996 2024-09-11T09:51:20.499 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.499 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.499 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:20.499 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.725357+0000", 2024-09-11T09:51:20.499 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0045077379999999998 2024-09-11T09:51:20.499 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.499 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.499 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:20.499 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.725392+0000", 2024-09-11T09:51:20.499 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.5862999999999999e-05 2024-09-11T09:51:20.499 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.499 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.499 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:20.500 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.725435+0000", 2024-09-11T09:51:20.500 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.2311999999999997e-05 2024-09-11T09:51:20.500 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:20.500 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:20.500 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:20.500 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.500 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.500 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:119 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 131072~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:51:20.500 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.725673+0000", 2024-09-11T09:51:20.501 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 68.704131982000007, 2024-09-11T09:51:20.501 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0053927460000000003, 2024-09-11T09:51:20.501 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:20.501 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:20.501 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:20.501 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:20.501 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:51:20.501 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:51:20.501 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 119 2024-09-11T09:51:20.501 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.501 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:20.501 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.501 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:20.502 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.725673+0000", 2024-09-11T09:51:20.502 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:20.502 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.502 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.502 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:20.502 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.725673+0000", 2024-09-11T09:51:20.502 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:20.502 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.502 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.502 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:20.502 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.725673+0000", 2024-09-11T09:51:20.502 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.68e-07 2024-09-11T09:51:20.502 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.503 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.503 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:20.503 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.725720+0000", 2024-09-11T09:51:20.503 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.6526999999999997e-05 2024-09-11T09:51:20.503 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.503 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.503 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:20.503 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.725734+0000", 2024-09-11T09:51:20.503 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.4593000000000001e-05 2024-09-11T09:51:20.503 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.503 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.503 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:20.503 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.725741+0000", 2024-09-11T09:51:20.503 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.28e-06 2024-09-11T09:51:20.504 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.504 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.504 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:20.504 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.726290+0000", 2024-09-11T09:51:20.504 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00054892000000000005 2024-09-11T09:51:20.504 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.504 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.504 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:20.504 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.726338+0000", 2024-09-11T09:51:20.504 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.7982000000000002e-05 2024-09-11T09:51:20.504 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.504 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.505 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:20.505 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.726413+0000", 2024-09-11T09:51:20.505 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.5679000000000005e-05 2024-09-11T09:51:20.505 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.505 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.505 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:20.505 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.727555+0000", 2024-09-11T09:51:20.505 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0011420289999999999 2024-09-11T09:51:20.505 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.505 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.505 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:20.505 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.731004+0000", 2024-09-11T09:51:20.505 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0034484279999999999 2024-09-11T09:51:20.506 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.506 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.506 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:20.506 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.731040+0000", 2024-09-11T09:51:20.506 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.6315e-05 2024-09-11T09:51:20.506 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.506 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.506 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:20.506 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.731065+0000", 2024-09-11T09:51:20.506 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.5324999999999999e-05 2024-09-11T09:51:20.506 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:20.506 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:20.506 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:20.507 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.507 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.507 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:120 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 196608~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:51:20.507 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.725756+0000", 2024-09-11T09:51:20.507 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 68.704048072999996, 2024-09-11T09:51:20.507 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0053569100000000003, 2024-09-11T09:51:20.507 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:20.507 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:20.507 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:20.507 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:20.507 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:51:20.507 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:51:20.507 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 120 2024-09-11T09:51:20.507 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.508 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:20.508 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.508 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:20.508 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.725756+0000", 2024-09-11T09:51:20.508 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:20.508 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.508 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.508 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:20.508 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.725756+0000", 2024-09-11T09:51:20.508 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:20.508 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.508 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.509 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:20.509 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.725757+0000", 2024-09-11T09:51:20.509 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.3799999999999997e-07 2024-09-11T09:51:20.509 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.509 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.509 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:20.509 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.725802+0000", 2024-09-11T09:51:20.509 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.4790000000000003e-05 2024-09-11T09:51:20.509 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.509 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.509 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:20.509 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.725815+0000", 2024-09-11T09:51:20.509 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.2609000000000001e-05 2024-09-11T09:51:20.509 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.510 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.510 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:20.510 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.725822+0000", 2024-09-11T09:51:20.510 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.6450000000000005e-06 2024-09-11T09:51:20.510 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.511 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.511 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:20.511 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.726708+0000", 2024-09-11T09:51:20.511 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00088606500000000001 2024-09-11T09:51:20.511 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.511 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.511 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:20.511 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.726755+0000", 2024-09-11T09:51:20.511 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.6888000000000002e-05 2024-09-11T09:51:20.511 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.511 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.511 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:20.511 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.726828+0000", 2024-09-11T09:51:20.512 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.2510999999999997e-05 2024-09-11T09:51:20.512 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.512 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.512 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:20.512 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.727829+0000", 2024-09-11T09:51:20.512 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001001093 2024-09-11T09:51:20.512 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.512 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.512 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:20.512 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.731071+0000", 2024-09-11T09:51:20.512 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0032426009999999999 2024-09-11T09:51:20.512 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.512 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.513 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:20.513 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.731094+0000", 2024-09-11T09:51:20.513 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.2677e-05 2024-09-11T09:51:20.513 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.513 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.513 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:20.513 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.731113+0000", 2024-09-11T09:51:20.513 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.9392999999999999e-05 2024-09-11T09:51:20.513 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:20.513 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:20.513 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:20.513 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.514 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.514 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:138 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:51:20.514 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:12.034038+0000", 2024-09-11T09:51:20.514 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 68.395766742999996, 2024-09-11T09:51:20.514 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0054142749999999996, 2024-09-11T09:51:20.514 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:20.514 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:20.514 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:20.514 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:20.514 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:51:20.514 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:51:20.515 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 138 2024-09-11T09:51:20.515 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.515 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:20.515 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.515 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:20.515 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034038+0000", 2024-09-11T09:51:20.515 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:20.515 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.515 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.515 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:20.515 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034038+0000", 2024-09-11T09:51:20.515 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:20.516 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.516 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.516 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:20.516 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034038+0000", 2024-09-11T09:51:20.516 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.3499999999999996e-07 2024-09-11T09:51:20.516 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.516 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.516 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:20.516 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034043+0000", 2024-09-11T09:51:20.516 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.7310000000000001e-06 2024-09-11T09:51:20.516 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.516 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.516 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:20.517 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034050+0000", 2024-09-11T09:51:20.517 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.0419999999999997e-06 2024-09-11T09:51:20.517 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.517 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.517 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:20.517 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034055+0000", 2024-09-11T09:51:20.517 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.3299999999999998e-06 2024-09-11T09:51:20.517 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.517 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.517 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:20.517 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.037309+0000", 2024-09-11T09:51:20.517 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0032535440000000001 2024-09-11T09:51:20.518 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.518 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.518 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:20.518 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.037357+0000", 2024-09-11T09:51:20.518 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.8167999999999997e-05 2024-09-11T09:51:20.518 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.518 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.518 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:20.518 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.037430+0000", 2024-09-11T09:51:20.518 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.3275999999999994e-05 2024-09-11T09:51:20.518 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.518 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.518 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:20.518 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.039014+0000", 2024-09-11T09:51:20.519 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0015831350000000001 2024-09-11T09:51:20.519 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.519 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.519 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:20.519 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.039401+0000", 2024-09-11T09:51:20.519 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00038758500000000002 2024-09-11T09:51:20.519 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.519 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.519 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:20.519 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.039429+0000", 2024-09-11T09:51:20.519 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.7379e-05 2024-09-11T09:51:20.519 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.520 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.520 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:20.520 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.039452+0000", 2024-09-11T09:51:20.520 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.355e-05 2024-09-11T09:51:20.520 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:20.520 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:20.520 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:20.520 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.520 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.520 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:139 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:51:20.520 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:12.034064+0000", 2024-09-11T09:51:20.520 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 68.395740712000006, 2024-09-11T09:51:20.520 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0054736200000000002, 2024-09-11T09:51:20.521 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:20.521 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:20.521 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:20.521 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:20.521 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:51:20.521 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:51:20.521 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 139 2024-09-11T09:51:20.521 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.521 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:20.521 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.521 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:20.521 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034064+0000", 2024-09-11T09:51:20.521 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:20.521 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.522 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.522 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:20.522 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034064+0000", 2024-09-11T09:51:20.522 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:20.522 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.522 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.522 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:20.522 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034064+0000", 2024-09-11T09:51:20.522 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.89e-07 2024-09-11T09:51:20.522 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.522 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.522 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:20.522 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034069+0000", 2024-09-11T09:51:20.523 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.9799999999999998e-06 2024-09-11T09:51:20.523 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.523 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.523 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:20.523 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034079+0000", 2024-09-11T09:51:20.523 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.3549999999999994e-06 2024-09-11T09:51:20.523 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.523 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.523 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:20.523 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034084+0000", 2024-09-11T09:51:20.523 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.2209999999999996e-06 2024-09-11T09:51:20.523 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.523 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.524 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:20.524 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.038176+0000", 2024-09-11T09:51:20.524 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0040921610000000004 2024-09-11T09:51:20.524 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.524 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.524 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:20.524 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.038221+0000", 2024-09-11T09:51:20.524 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.5006000000000002e-05 2024-09-11T09:51:20.524 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.524 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.524 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:20.524 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.038293+0000", 2024-09-11T09:51:20.525 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.1952999999999994e-05 2024-09-11T09:51:20.525 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.525 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.525 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:20.525 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.039461+0000", 2024-09-11T09:51:20.525 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0011680340000000001 2024-09-11T09:51:20.525 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.525 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.525 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:20.525 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.039487+0000", 2024-09-11T09:51:20.525 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.5752e-05 2024-09-11T09:51:20.525 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.525 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.526 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:20.526 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.039515+0000", 2024-09-11T09:51:20.526 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.7763000000000002e-05 2024-09-11T09:51:20.526 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.526 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.526 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:20.526 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.039537+0000", 2024-09-11T09:51:20.526 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.2906000000000001e-05 2024-09-11T09:51:20.526 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:20.526 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:20.526 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:20.526 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.527 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.527 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:140 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:51:20.527 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:12.034093+0000", 2024-09-11T09:51:20.527 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 68.395711207999994, 2024-09-11T09:51:20.527 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0059112490000000004, 2024-09-11T09:51:20.527 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:20.527 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:20.527 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:20.527 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:20.527 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:51:20.527 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:51:20.527 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 140 2024-09-11T09:51:20.527 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.528 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:20.528 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.528 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:20.528 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034093+0000", 2024-09-11T09:51:20.528 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:20.528 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.528 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.528 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:20.528 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034093+0000", 2024-09-11T09:51:20.528 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:20.528 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.529 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.529 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:20.529 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034094+0000", 2024-09-11T09:51:20.529 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.4299999999999998e-07 2024-09-11T09:51:20.529 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.529 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.529 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:20.529 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034159+0000", 2024-09-11T09:51:20.529 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.567e-05 2024-09-11T09:51:20.529 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.529 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.529 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:20.529 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034167+0000", 2024-09-11T09:51:20.530 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.8490000000000006e-06 2024-09-11T09:51:20.530 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.530 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.530 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:20.530 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034173+0000", 2024-09-11T09:51:20.530 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.1770000000000003e-06 2024-09-11T09:51:20.530 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.530 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.530 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:20.530 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.038543+0000", 2024-09-11T09:51:20.530 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0043699150000000003 2024-09-11T09:51:20.530 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.530 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.531 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:20.531 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.038589+0000", 2024-09-11T09:51:20.531 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.5562999999999997e-05 2024-09-11T09:51:20.531 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.531 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.531 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:20.531 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.038662+0000", 2024-09-11T09:51:20.531 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.2748000000000002e-05 2024-09-11T09:51:20.531 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.531 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.531 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:20.531 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.039635+0000", 2024-09-11T09:51:20.531 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00097310799999999998 2024-09-11T09:51:20.532 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.532 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.532 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:20.532 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.039945+0000", 2024-09-11T09:51:20.532 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00031054100000000001 2024-09-11T09:51:20.532 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.532 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.532 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:20.532 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.039971+0000", 2024-09-11T09:51:20.532 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.5423999999999999e-05 2024-09-11T09:51:20.532 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.532 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.533 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:20.533 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.040005+0000", 2024-09-11T09:51:20.533 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.3810999999999999e-05 2024-09-11T09:51:20.533 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:20.533 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:20.533 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:20.533 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.533 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.533 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:51:20.533 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:12.034184+0000", 2024-09-11T09:51:20.533 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 68.395620510000001, 2024-09-11T09:51:20.533 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0061520380000000003, 2024-09-11T09:51:20.534 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:20.534 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:20.534 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:20.534 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:20.534 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:51:20.534 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:51:20.534 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 141 2024-09-11T09:51:20.534 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.534 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:20.534 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.534 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:20.534 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034184+0000", 2024-09-11T09:51:20.534 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:20.535 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.535 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.535 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:20.535 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034184+0000", 2024-09-11T09:51:20.535 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:20.535 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.535 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.535 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:20.535 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034185+0000", 2024-09-11T09:51:20.535 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.1099999999999995e-07 2024-09-11T09:51:20.535 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.535 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.535 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:20.536 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034190+0000", 2024-09-11T09:51:20.536 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.7309999999999998e-06 2024-09-11T09:51:20.536 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.536 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.536 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:20.536 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034197+0000", 2024-09-11T09:51:20.536 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.7440000000000001e-06 2024-09-11T09:51:20.536 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.536 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.536 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:20.536 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034204+0000", 2024-09-11T09:51:20.536 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.8970000000000002e-06 2024-09-11T09:51:20.536 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.537 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.537 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:20.537 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.039035+0000", 2024-09-11T09:51:20.537 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0048307039999999999 2024-09-11T09:51:20.537 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.537 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.537 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:20.537 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.039093+0000", 2024-09-11T09:51:20.537 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.8604000000000003e-05 2024-09-11T09:51:20.538 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.538 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.538 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:20.538 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.039184+0000", 2024-09-11T09:51:20.538 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.0196000000000002e-05 2024-09-11T09:51:20.538 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.538 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.538 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:20.538 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.039873+0000", 2024-09-11T09:51:20.538 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00068960600000000005 2024-09-11T09:51:20.538 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.538 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.538 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:20.539 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.040292+0000", 2024-09-11T09:51:20.539 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00041865199999999999 2024-09-11T09:51:20.539 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.539 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.539 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:20.539 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.040317+0000", 2024-09-11T09:51:20.539 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.5439000000000002e-05 2024-09-11T09:51:20.539 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.539 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.539 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:20.539 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.040336+0000", 2024-09-11T09:51:20.539 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.8754e-05 2024-09-11T09:51:20.539 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:20.539 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:20.540 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:20.540 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.540 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.540 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:157 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:51:20.540 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:12.452040+0000", 2024-09-11T09:51:20.540 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 67.977764656999994, 2024-09-11T09:51:20.540 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0068848269999999996, 2024-09-11T09:51:20.540 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:20.540 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:20.540 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:20.540 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:20.540 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:51:20.540 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:51:20.541 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 157 2024-09-11T09:51:20.541 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.541 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:20.541 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.541 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:20.541 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452040+0000", 2024-09-11T09:51:20.541 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:20.541 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.541 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.541 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:20.541 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452040+0000", 2024-09-11T09:51:20.541 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:20.541 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.542 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.542 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:20.542 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452040+0000", 2024-09-11T09:51:20.542 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.22e-07 2024-09-11T09:51:20.542 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.542 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.542 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:20.542 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452042+0000", 2024-09-11T09:51:20.542 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.7010000000000001e-06 2024-09-11T09:51:20.542 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.542 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.542 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:20.542 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452045+0000", 2024-09-11T09:51:20.543 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.1310000000000002e-06 2024-09-11T09:51:20.543 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.543 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.543 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:20.543 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452048+0000", 2024-09-11T09:51:20.543 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.65e-06 2024-09-11T09:51:20.543 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.543 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.543 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:20.543 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.454453+0000", 2024-09-11T09:51:20.543 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0024052959999999999 2024-09-11T09:51:20.543 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.543 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.543 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:20.544 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.454498+0000", 2024-09-11T09:51:20.544 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.4978999999999997e-05 2024-09-11T09:51:20.544 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.544 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.544 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:20.544 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.454567+0000", 2024-09-11T09:51:20.544 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.9110999999999996e-05 2024-09-11T09:51:20.544 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.544 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.544 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:20.544 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.455464+0000", 2024-09-11T09:51:20.544 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00089696199999999995 2024-09-11T09:51:20.544 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.545 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.545 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:20.545 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.458844+0000", 2024-09-11T09:51:20.545 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0033802469999999999 2024-09-11T09:51:20.545 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.545 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.545 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:20.545 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.458889+0000", 2024-09-11T09:51:20.545 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.4441999999999999e-05 2024-09-11T09:51:20.545 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.545 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.545 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:20.545 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.458925+0000", 2024-09-11T09:51:20.546 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.6086e-05 2024-09-11T09:51:20.546 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:20.546 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:20.546 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:20.546 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.546 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.546 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:158 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:51:20.546 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:12.452055+0000", 2024-09-11T09:51:20.546 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 67.977749263999996, 2024-09-11T09:51:20.546 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0069533829999999996, 2024-09-11T09:51:20.546 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:20.546 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:20.547 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:20.547 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:20.547 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:51:20.547 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:51:20.547 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 158 2024-09-11T09:51:20.547 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.547 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:20.547 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.547 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:20.547 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452055+0000", 2024-09-11T09:51:20.547 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:20.547 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.547 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.548 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:20.548 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452055+0000", 2024-09-11T09:51:20.548 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:20.548 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.548 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.548 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:20.548 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452056+0000", 2024-09-11T09:51:20.548 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.48e-07 2024-09-11T09:51:20.548 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.548 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.548 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:20.548 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452087+0000", 2024-09-11T09:51:20.548 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.1149000000000003e-05 2024-09-11T09:51:20.549 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.549 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.549 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:20.549 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452092+0000", 2024-09-11T09:51:20.549 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.8400000000000002e-06 2024-09-11T09:51:20.549 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.549 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.549 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:20.549 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452095+0000", 2024-09-11T09:51:20.549 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.0740000000000001e-06 2024-09-11T09:51:20.549 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.549 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.550 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:20.550 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.454796+0000", 2024-09-11T09:51:20.550 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0027009740000000001 2024-09-11T09:51:20.550 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.550 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.550 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:20.550 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.454869+0000", 2024-09-11T09:51:20.550 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.3678999999999997e-05 2024-09-11T09:51:20.550 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.550 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.550 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:20.550 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.454992+0000", 2024-09-11T09:51:20.551 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00012279000000000001 2024-09-11T09:51:20.551 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.551 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.551 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:20.551 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.456072+0000", 2024-09-11T09:51:20.551 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001080229 2024-09-11T09:51:20.551 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.551 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.551 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:20.551 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.458931+0000", 2024-09-11T09:51:20.551 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0028591789999999999 2024-09-11T09:51:20.551 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.551 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.552 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:20.552 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.458972+0000", 2024-09-11T09:51:20.552 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.0170999999999999e-05 2024-09-11T09:51:20.552 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.552 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.552 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:20.552 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.459009+0000", 2024-09-11T09:51:20.552 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.7049999999999999e-05 2024-09-11T09:51:20.552 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:20.552 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:20.552 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:20.552 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.553 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.553 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:159 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:51:20.553 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:12.452099+0000", 2024-09-11T09:51:20.553 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 67.977705095999994, 2024-09-11T09:51:20.553 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0069885950000000002, 2024-09-11T09:51:20.553 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:20.553 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:20.553 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:20.553 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:20.553 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:51:20.553 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:51:20.553 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 159 2024-09-11T09:51:20.553 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.554 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:20.554 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.554 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:20.554 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452099+0000", 2024-09-11T09:51:20.554 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:20.554 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.554 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.554 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:20.554 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452099+0000", 2024-09-11T09:51:20.554 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:20.554 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.554 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.555 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:20.555 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452100+0000", 2024-09-11T09:51:20.555 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.2800000000000003e-07 2024-09-11T09:51:20.555 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.555 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.555 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:20.555 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452102+0000", 2024-09-11T09:51:20.555 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.824e-06 2024-09-11T09:51:20.555 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.555 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.555 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:20.555 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452105+0000", 2024-09-11T09:51:20.555 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.0460000000000001e-06 2024-09-11T09:51:20.555 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.556 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.556 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:20.556 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452107+0000", 2024-09-11T09:51:20.556 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.543e-06 2024-09-11T09:51:20.556 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.556 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.556 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:20.556 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.455484+0000", 2024-09-11T09:51:20.556 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0033769759999999998 2024-09-11T09:51:20.556 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.556 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.556 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:20.556 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.455527+0000", 2024-09-11T09:51:20.556 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.2589999999999997e-05 2024-09-11T09:51:20.557 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.557 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.557 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:20.557 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.455594+0000", 2024-09-11T09:51:20.557 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.6775999999999999e-05 2024-09-11T09:51:20.557 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.557 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.557 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:20.557 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.456312+0000", 2024-09-11T09:51:20.557 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00071874700000000005 2024-09-11T09:51:20.557 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.557 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.557 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:20.558 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.459015+0000", 2024-09-11T09:51:20.558 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0027028540000000002 2024-09-11T09:51:20.558 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.558 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.558 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:20.558 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.459054+0000", 2024-09-11T09:51:20.558 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.8840000000000001e-05 2024-09-11T09:51:20.558 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:20.558 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:20.558 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:20.558 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.459088+0000", 2024-09-11T09:51:20.558 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.4071e-05 2024-09-11T09:51:20.558 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:20.558 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:20.559 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:20.559 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:20.559 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:20.559 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-11T09:51:20.560 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-11T09:51:20.560 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T09:51:20.561 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-11T09:51:20.561 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T09:51:20.561 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-11T09:51:20.562 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-11T09:51:20.562 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T09:51:20.563 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-11T09:51:20.563 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T09:51:20.564 INFO:tasks.ceph.osd.0.smithi017.stderr:2024-09-11T09:51:20.561+0000 7fb73c0f1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31665) UID: 0 2024-09-11T09:51:20.661 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-09-11T09:51:20.662+0000 7f7b6c7d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31904) UID: 0 2024-09-11T09:51:20.702 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-11T09:51:20.707 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:51:20.707 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:51:20.716 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-11T09:51:20.800 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:51:20.801+0000 7fdec5849640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31907) UID: 0 2024-09-11T09:51:20.860 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:51:20.860 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:51:20.863 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-09-11T09:51:20.864+0000 7f7b6c7d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31904) UID: 0 2024-09-11T09:51:20.868 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-11T09:51:20.964 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-09-11T09:51:20.965+0000 7f7b6c7d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31904) UID: 0 2024-09-11T09:51:21.010 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:51:21.011 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:51:21.019 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-11T09:51:21.103 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-09-11T09:51:21.104+0000 7f7b6c7d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31904) UID: 0 2024-09-11T09:51:21.163 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:51:21.163 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:51:21.166 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:51:21.167+0000 7fdec5849640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31907) UID: 0 2024-09-11T09:51:21.171 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-11T09:51:21.267 INFO:tasks.ceph.osd.2.smithi017.stderr:2024-09-11T09:51:21.268+0000 7fd20b170640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31672) UID: 0 2024-09-11T09:51:21.313 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:51:21.313 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:51:21.322 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-11T09:51:21.368 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:51:21.369+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:51:21.424 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:51:21.445 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:51:21.446 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-11T09:51:21.477 INFO:tasks.workunit.client.0.smithi107.stderr:Note: switching to 'daaf78de334647fc0c800f4ac495cc9475a26bae'. 2024-09-11T09:51:21.477 INFO:tasks.workunit.client.0.smithi107.stderr: 2024-09-11T09:51:21.477 INFO:tasks.workunit.client.0.smithi107.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-09-11T09:51:21.477 INFO:tasks.workunit.client.0.smithi107.stderr:changes and commit them, and you can discard any commits you make in this 2024-09-11T09:51:21.477 INFO:tasks.workunit.client.0.smithi107.stderr:state without impacting any branches by switching back to a branch. 2024-09-11T09:51:21.477 INFO:tasks.workunit.client.0.smithi107.stderr: 2024-09-11T09:51:21.477 INFO:tasks.workunit.client.0.smithi107.stderr:If you want to create a new branch to retain commits you create, you may 2024-09-11T09:51:21.477 INFO:tasks.workunit.client.0.smithi107.stderr:do so (now or later) by using -c with the switch command. Example: 2024-09-11T09:51:21.477 INFO:tasks.workunit.client.0.smithi107.stderr: 2024-09-11T09:51:21.477 INFO:tasks.workunit.client.0.smithi107.stderr: git switch -c 2024-09-11T09:51:21.477 INFO:tasks.workunit.client.0.smithi107.stderr: 2024-09-11T09:51:21.478 INFO:tasks.workunit.client.0.smithi107.stderr:Or undo this operation with: 2024-09-11T09:51:21.478 INFO:tasks.workunit.client.0.smithi107.stderr: 2024-09-11T09:51:21.478 INFO:tasks.workunit.client.0.smithi107.stderr: git switch - 2024-09-11T09:51:21.478 INFO:tasks.workunit.client.0.smithi107.stderr: 2024-09-11T09:51:21.478 INFO:tasks.workunit.client.0.smithi107.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-09-11T09:51:21.478 INFO:tasks.workunit.client.0.smithi107.stderr: 2024-09-11T09:51:21.478 INFO:tasks.workunit.client.0.smithi107.stderr:HEAD is now at daaf78de334 Merge pull request #59682 from zdover23/wip-doc-2024-09-10-backport-59677-to-squid 2024-09-11T09:51:21.481 DEBUG:teuthology.orchestra.run.smithi107:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-09-11T09:51:21.490 INFO:tasks.ceph.osd.2.smithi017.stderr:2024-09-11T09:51:21.491+0000 7fd20b170640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31672) UID: 0 2024-09-11T09:51:21.548 INFO:tasks.workunit.client.0.smithi107.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-09-11T09:51:21.550 INFO:tasks.workunit.client.0.smithi107.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-09-11T09:51:21.550 INFO:tasks.workunit.client.0.smithi107.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-09-11T09:51:21.561 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:51:21.561 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:51:21.568 INFO:tasks.ceph.osd.2.smithi017.stderr:2024-09-11T09:51:21.569+0000 7fd20b170640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31672) UID: 0 2024-09-11T09:51:21.570 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-11T09:51:21.670 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:51:21.670+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:51:21.695 INFO:teuthology.orchestra.run.smithi017.stderr:noscrub is unset 2024-09-11T09:51:21.721 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:51:21.721 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:51:21.730 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-11T09:51:21.771 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-09-11T09:51:21.771+0000 7f7b6c7d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31904) UID: 0 2024-09-11T09:51:21.872 INFO:tasks.ceph.osd.0.smithi017.stderr:2024-09-11T09:51:21.873+0000 7fb73c0f1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31665) UID: 0 2024-09-11T09:51:21.877 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:51:21.877 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:51:21.886 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-11T09:51:21.973 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:51:21.974+0000 7f3c96ec9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31909) UID: 0 2024-09-11T09:51:22.034 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:51:22.034 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:51:22.042 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-11T09:51:22.087 INFO:tasks.ceph.osd.0.smithi017.stderr:2024-09-11T09:51:22.088+0000 7fb73c0f1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31665) UID: 0 2024-09-11T09:51:22.149 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:51:22.149 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:51:22.158 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-11T09:51:22.199 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:51:22.200+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:51:22.278 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-09-11T09:51:22.279+0000 7f7b6c7d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31904) UID: 0 2024-09-11T09:51:22.298 INFO:tasks.workunit.client.0.smithi107.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-09-11T09:51:22.305 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:51:22.305 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:51:22.313 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-11T09:51:22.368 INFO:tasks.workunit.client.0.smithi107.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-09-11T09:51:22.379 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:51:22.380+0000 7f3c96ec9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31909) UID: 0 2024-09-11T09:51:22.408 INFO:tasks.workunit.client.0.smithi107.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-09-11T09:51:22.410 INFO:tasks.workunit.client.0.smithi107.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-09-11T09:51:22.410 INFO:tasks.workunit.client.0.smithi107.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-09-11T09:51:22.421 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:51:22.421 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:51:22.433 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-11T09:51:22.448 INFO:tasks.workunit.client.0.smithi107.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-09-11T09:51:22.451 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-09-11T09:51:22.452 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-09-11T09:51:22.480 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:51:22.481+0000 7fdec5849640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31907) UID: 0 2024-09-11T09:51:22.507 INFO:tasks.workunit:Running workunits matching rbd/rbd-nbd.sh on client.0... 2024-09-11T09:51:22.509 INFO:tasks.workunit:Running workunit rbd/rbd-nbd.sh... 2024-09-11T09:51:22.509 DEBUG:teuthology.orchestra.run.smithi107:workunit test rbd/rbd-nbd.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=daaf78de334647fc0c800f4ac495cc9475a26bae TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/rbd-nbd.sh 2024-09-11T09:51:22.540 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:51:22.541 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:51:22.549 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-11T09:51:22.565 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T09:51:22.566 INFO:tasks.workunit.client.0.smithi107.stderr:++ dirname /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/rbd-nbd.sh 2024-09-11T09:51:22.568 INFO:tasks.workunit.client.0.smithi107.stderr:+ . /home/ubuntu/cephtest/clone.client.0/qa/workunits/rbd/../../standalone/ceph-helpers.sh 2024-09-11T09:51:22.568 INFO:tasks.workunit.client.0.smithi107.stderr:++ TIMEOUT=300 2024-09-11T09:51:22.568 INFO:tasks.workunit.client.0.smithi107.stderr:++ WAIT_FOR_CLEAN_TIMEOUT=90 2024-09-11T09:51:22.568 INFO:tasks.workunit.client.0.smithi107.stderr:++ MAX_TIMEOUT=15 2024-09-11T09:51:22.568 INFO:tasks.workunit.client.0.smithi107.stderr:++ PG_NUM=4 2024-09-11T09:51:22.568 INFO:tasks.workunit.client.0.smithi107.stderr:++ TMPDIR=/tmp 2024-09-11T09:51:22.568 INFO:tasks.workunit.client.0.smithi107.stderr:++ CEPH_BUILD_VIRTUALENV=/tmp 2024-09-11T09:51:22.568 INFO:tasks.workunit.client.0.smithi107.stderr:++ TESTDIR=/home/ubuntu/cephtest 2024-09-11T09:51:22.568 INFO:tasks.workunit.client.0.smithi107.stderr:++ type xmlstarlet 2024-09-11T09:51:22.569 INFO:tasks.workunit.client.0.smithi107.stderr:++ XMLSTARLET=xmlstarlet 2024-09-11T09:51:22.569 INFO:tasks.workunit.client.0.smithi107.stderr:+++ uname 2024-09-11T09:51:22.570 INFO:tasks.workunit.client.0.smithi107.stderr:++ '[' Linux = FreeBSD ']' 2024-09-11T09:51:22.570 INFO:tasks.workunit.client.0.smithi107.stderr:++ SED=sed 2024-09-11T09:51:22.570 INFO:tasks.workunit.client.0.smithi107.stderr:++ AWK=awk 2024-09-11T09:51:22.570 INFO:tasks.workunit.client.0.smithi107.stderr:+++ stty -a 2024-09-11T09:51:22.570 INFO:tasks.workunit.client.0.smithi107.stderr:+++ head -1 2024-09-11T09:51:22.570 INFO:tasks.workunit.client.0.smithi107.stderr:+++ sed -e 's/.*columns \([0-9]*\).*/\1/' 2024-09-11T09:51:22.570 INFO:tasks.workunit.client.0.smithi107.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-09-11T09:51:22.570 INFO:tasks.workunit.client.0.smithi107.stderr:++ termwidth= 2024-09-11T09:51:22.570 INFO:tasks.workunit.client.0.smithi107.stderr:++ '[' -n '' -a '' '!=' 0 ']' 2024-09-11T09:51:22.571 INFO:tasks.workunit.client.0.smithi107.stderr:++ DIFFCOLOPTS='-y ' 2024-09-11T09:51:22.571 INFO:tasks.workunit.client.0.smithi107.stderr:++ KERNCORE=kernel.core_pattern 2024-09-11T09:51:22.571 INFO:tasks.workunit.client.0.smithi107.stderr:++ EXTRA_OPTS= 2024-09-11T09:51:22.574 INFO:tasks.workunit.client.0.smithi107.stderr:++ test '' = TESTS 2024-09-11T09:51:22.574 INFO:tasks.workunit.client.0.smithi107.stderr:+ POOL=rbd 2024-09-11T09:51:22.574 INFO:tasks.workunit.client.0.smithi107.stderr:+ ANOTHER_POOL=new_default_pool27121 2024-09-11T09:51:22.574 INFO:tasks.workunit.client.0.smithi107.stderr:+ NS=ns 2024-09-11T09:51:22.574 INFO:tasks.workunit.client.0.smithi107.stderr:+ IMAGE=testrbdnbd27121 2024-09-11T09:51:22.574 INFO:tasks.workunit.client.0.smithi107.stderr:+ SIZE=64 2024-09-11T09:51:22.574 INFO:tasks.workunit.client.0.smithi107.stderr:+ DATA= 2024-09-11T09:51:22.574 INFO:tasks.workunit.client.0.smithi107.stderr:+ DEV= 2024-09-11T09:51:22.574 INFO:tasks.workunit.client.0.smithi107.stderr:+ setup 2024-09-11T09:51:22.574 INFO:tasks.workunit.client.0.smithi107.stderr:+ local ns x 2024-09-11T09:51:22.575 INFO:tasks.workunit.client.0.smithi107.stderr:+ '[' -e CMakeCache.txt ']' 2024-09-11T09:51:22.575 INFO:tasks.workunit.client.0.smithi107.stderr:+ _sudo echo test sudo 2024-09-11T09:51:22.575 INFO:tasks.workunit.client.0.smithi107.stderr:+ local cmd 2024-09-11T09:51:22.575 INFO:tasks.workunit.client.0.smithi107.stderr:++ id -u 2024-09-11T09:51:22.575 INFO:tasks.workunit.client.0.smithi107.stderr:+ '[' 1000 -eq 0 ']' 2024-09-11T09:51:22.575 INFO:tasks.workunit.client.0.smithi107.stderr:++ which echo 2024-09-11T09:51:22.591 INFO:tasks.workunit.client.0.smithi107.stderr:+ cmd=/usr/bin/echo 2024-09-11T09:51:22.591 INFO:tasks.workunit.client.0.smithi107.stderr:+ shift 2024-09-11T09:51:22.592 INFO:tasks.workunit.client.0.smithi107.stderr:+ sudo -nE /usr/bin/echo test sudo 2024-09-11T09:51:22.594 INFO:tasks.ceph.osd.2.smithi017.stderr:2024-09-11T09:51:22.595+0000 7fd20b170640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31672) UID: 0 2024-09-11T09:51:22.601 INFO:tasks.workunit.client.0.smithi107.stdout:test sudo 2024-09-11T09:51:22.602 INFO:tasks.workunit.client.0.smithi107.stderr:+ trap cleanup INT TERM EXIT 2024-09-11T09:51:22.602 INFO:tasks.workunit.client.0.smithi107.stderr:++ mktemp -d 2024-09-11T09:51:22.603 INFO:tasks.workunit.client.0.smithi107.stderr:+ TEMPDIR=/tmp/tmp.rvrcDirNue 2024-09-11T09:51:22.603 INFO:tasks.workunit.client.0.smithi107.stderr:+ DATA=/tmp/tmp.rvrcDirNue/data 2024-09-11T09:51:22.604 INFO:tasks.workunit.client.0.smithi107.stderr:+ dd if=/dev/urandom of=/tmp/tmp.rvrcDirNue/data bs=1M count=64 2024-09-11T09:51:22.658 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:51:22.658 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:51:22.668 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-11T09:51:22.712 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-09-11T09:51:22.713+0000 7f7b6c7d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31904) UID: 0 2024-09-11T09:51:22.717 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-11T09:51:22.718 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T09:51:22.734 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-11T09:51:22.735 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T09:51:22.751 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-11T09:51:22.751 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T09:51:22.767 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-11T09:51:22.767 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T09:51:22.781 INFO:tasks.workunit.client.0.smithi107.stderr:64+0 records in 2024-09-11T09:51:22.781 INFO:tasks.workunit.client.0.smithi107.stderr:64+0 records out 2024-09-11T09:51:22.781 INFO:tasks.workunit.client.0.smithi107.stderr:67108864 bytes (67 MB, 64 MiB) copied, 0.176732 s, 380 MB/s 2024-09-11T09:51:22.781 INFO:tasks.workunit.client.0.smithi107.stderr:+ rbd namespace create rbd/ns 2024-09-11T09:51:22.782 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:51:22.783+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:51:22.784 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-11T09:51:22.785 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T09:51:22.785 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-11T09:51:22.786 INFO:teuthology.orchestra.run.smithi096.stdout: "ops": [], 2024-09-11T09:51:22.786 INFO:teuthology.orchestra.run.smithi096.stdout: "num_ops": 0 2024-09-11T09:51:22.786 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-11T09:51:22.795 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-11T09:51:22.801 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-11T09:51:22.802 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T09:51:22.830 INFO:tasks.workunit.client.0.smithi107.stderr:+ for ns in '' ${NS} 2024-09-11T09:51:22.830 INFO:tasks.workunit.client.0.smithi107.stderr:+ rbd --dest-pool rbd --dest-namespace '' --no-progress import /tmp/tmp.rvrcDirNue/data testrbdnbd27121 2024-09-11T09:51:22.883 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-09-11T09:51:22.884+0000 7f7b6c7d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31904) UID: 0 2024-09-11T09:51:22.953 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-11T09:51:22.953 INFO:teuthology.orchestra.run.smithi096.stdout: "ops": [], 2024-09-11T09:51:22.953 INFO:teuthology.orchestra.run.smithi096.stdout: "complaint_time": 30, 2024-09-11T09:51:22.953 INFO:teuthology.orchestra.run.smithi096.stdout: "num_blocked_ops": 0 2024-09-11T09:51:22.953 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-11T09:51:22.963 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-11T09:51:22.968 INFO:tasks.workunit.client.0.smithi107.stderr:2024-09-11T09:51:22.969+0000 7fb73c889640 0 -- 172.21.15.107:0/3325461587 >> [v2:172.21.15.17:6800/4139511644,v1:172.21.15.17:6801/4139511644] conn(0x7fb72003d440 msgr2=0x7fb72003f830 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-09-11T09:51:23.005 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:51:23.005+0000 7fdec5849640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31907) UID: 0 2024-09-11T09:51:23.015 INFO:tasks.workunit.client.0.smithi107.stderr:+ for ns in '' ${NS} 2024-09-11T09:51:23.015 INFO:tasks.workunit.client.0.smithi107.stderr:+ rbd --dest-pool rbd --dest-namespace ns --no-progress import /tmp/tmp.rvrcDirNue/data testrbdnbd27121 2024-09-11T09:51:23.085 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:51:23.086+0000 7fdec5849640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31907) UID: 0 2024-09-11T09:51:23.113 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-11T09:51:23.114 INFO:teuthology.orchestra.run.smithi096.stdout: "size": 20, 2024-09-11T09:51:23.114 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 600, 2024-09-11T09:51:23.114 INFO:teuthology.orchestra.run.smithi096.stdout: "ops": [ 2024-09-11T09:51:23.114 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.114 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:74 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:51:23.114 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.539067+0000", 2024-09-11T09:51:23.114 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 71.573811383000006, 2024-09-11T09:51:23.114 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.006945957, 2024-09-11T09:51:23.114 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:23.114 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:23.115 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:23.115 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:23.115 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:51:23.115 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:51:23.115 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 74 2024-09-11T09:51:23.115 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.115 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:23.115 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.115 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:23.115 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.539067+0000", 2024-09-11T09:51:23.115 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:23.115 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.116 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.116 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:23.116 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.539067+0000", 2024-09-11T09:51:23.116 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:23.116 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.116 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.116 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:23.116 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.539068+0000", 2024-09-11T09:51:23.116 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.6000000000000003e-07 2024-09-11T09:51:23.116 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.116 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.116 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:23.117 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.539072+0000", 2024-09-11T09:51:23.117 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.0450000000000002e-06 2024-09-11T09:51:23.117 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.117 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.117 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:23.117 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.539078+0000", 2024-09-11T09:51:23.117 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.6889999999999998e-06 2024-09-11T09:51:23.117 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.117 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.117 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:23.117 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.539083+0000", 2024-09-11T09:51:23.117 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.7180000000000004e-06 2024-09-11T09:51:23.118 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.118 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.118 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:23.118 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.540230+0000", 2024-09-11T09:51:23.118 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0011466250000000001 2024-09-11T09:51:23.118 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.118 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.118 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:23.118 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.540279+0000", 2024-09-11T09:51:23.118 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.9667999999999999e-05 2024-09-11T09:51:23.118 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.118 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.118 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:23.119 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.540357+0000", 2024-09-11T09:51:23.119 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.7863999999999999e-05 2024-09-11T09:51:23.119 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.119 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.119 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:23.119 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.541086+0000", 2024-09-11T09:51:23.119 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00072921100000000001 2024-09-11T09:51:23.119 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.119 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.119 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:23.119 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.545949+0000", 2024-09-11T09:51:23.119 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0048626040000000004 2024-09-11T09:51:23.119 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.120 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.120 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:23.120 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.545989+0000", 2024-09-11T09:51:23.120 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.0259000000000002e-05 2024-09-11T09:51:23.120 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.120 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.120 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:23.120 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.546013+0000", 2024-09-11T09:51:23.120 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.3614000000000001e-05 2024-09-11T09:51:23.120 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:23.121 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:23.121 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:23.121 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.121 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.121 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:110 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:51:23.121 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.711636+0000", 2024-09-11T09:51:23.121 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 71.401242502000002, 2024-09-11T09:51:23.121 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0067613070000000003, 2024-09-11T09:51:23.121 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:23.121 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:23.121 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:23.122 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:23.122 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:51:23.122 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:51:23.122 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 110 2024-09-11T09:51:23.122 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.122 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:23.122 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.122 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:23.122 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711636+0000", 2024-09-11T09:51:23.122 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:23.122 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.123 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.123 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:23.123 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711636+0000", 2024-09-11T09:51:23.123 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:23.123 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.123 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.123 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:23.123 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711637+0000", 2024-09-11T09:51:23.123 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.9999999999999997e-07 2024-09-11T09:51:23.123 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.123 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.124 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:23.124 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711641+0000", 2024-09-11T09:51:23.124 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.2570000000000001e-06 2024-09-11T09:51:23.124 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.124 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.124 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:23.124 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711651+0000", 2024-09-11T09:51:23.124 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.0285999999999999e-05 2024-09-11T09:51:23.124 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.124 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.124 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:23.124 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711658+0000", 2024-09-11T09:51:23.125 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.3540000000000002e-06 2024-09-11T09:51:23.125 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.125 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.125 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:23.125 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.716475+0000", 2024-09-11T09:51:23.125 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0048162819999999999 2024-09-11T09:51:23.125 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.125 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.125 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:23.125 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.716526+0000", 2024-09-11T09:51:23.125 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.0905000000000001e-05 2024-09-11T09:51:23.126 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.126 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.126 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:23.126 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.716601+0000", 2024-09-11T09:51:23.126 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.5283000000000004e-05 2024-09-11T09:51:23.126 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.126 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.126 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:23.126 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.717933+0000", 2024-09-11T09:51:23.126 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001331622 2024-09-11T09:51:23.126 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.126 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.127 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:23.127 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718332+0000", 2024-09-11T09:51:23.127 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00039909899999999998 2024-09-11T09:51:23.127 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.127 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.127 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:23.127 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718363+0000", 2024-09-11T09:51:23.127 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.1636e-05 2024-09-11T09:51:23.127 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.127 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.127 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:23.128 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718397+0000", 2024-09-11T09:51:23.128 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.3883000000000001e-05 2024-09-11T09:51:23.128 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:23.128 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:23.128 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:23.128 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.128 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.128 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:111 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:51:23.128 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.711668+0000", 2024-09-11T09:51:23.128 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 71.401210071999998, 2024-09-11T09:51:23.128 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0077145570000000004, 2024-09-11T09:51:23.129 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:23.129 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:23.129 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:23.129 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:23.129 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:51:23.129 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:51:23.129 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 111 2024-09-11T09:51:23.129 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.129 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:23.129 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.129 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:23.130 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711668+0000", 2024-09-11T09:51:23.130 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:23.130 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.130 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.130 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:23.130 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711668+0000", 2024-09-11T09:51:23.130 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:23.130 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.130 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.130 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:23.131 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711669+0000", 2024-09-11T09:51:23.131 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.99e-07 2024-09-11T09:51:23.131 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.131 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.131 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:23.131 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711673+0000", 2024-09-11T09:51:23.131 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.7440000000000001e-06 2024-09-11T09:51:23.131 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.131 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.131 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:23.131 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711686+0000", 2024-09-11T09:51:23.132 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.3317e-05 2024-09-11T09:51:23.132 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.132 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.132 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:23.132 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711697+0000", 2024-09-11T09:51:23.132 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1022000000000001e-05 2024-09-11T09:51:23.132 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.132 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.132 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:23.132 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.717065+0000", 2024-09-11T09:51:23.132 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.005367531 2024-09-11T09:51:23.133 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.133 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.133 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:23.133 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.717114+0000", 2024-09-11T09:51:23.133 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.9707999999999999e-05 2024-09-11T09:51:23.133 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.133 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.133 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:23.133 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.717190+0000", 2024-09-11T09:51:23.134 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.5394000000000004e-05 2024-09-11T09:51:23.134 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.134 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.134 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:23.134 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718793+0000", 2024-09-11T09:51:23.134 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001603653 2024-09-11T09:51:23.134 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.134 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.135 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:23.135 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719329+0000", 2024-09-11T09:51:23.135 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00053543799999999995 2024-09-11T09:51:23.135 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.135 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.135 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:23.135 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719359+0000", 2024-09-11T09:51:23.135 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.0420999999999999e-05 2024-09-11T09:51:23.136 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.136 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.136 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:23.136 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719383+0000", 2024-09-11T09:51:23.136 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.3730000000000001e-05 2024-09-11T09:51:23.136 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:23.136 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:23.136 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:23.136 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.137 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.137 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:112 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:51:23.137 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.711712+0000", 2024-09-11T09:51:23.137 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 71.401166512000003, 2024-09-11T09:51:23.137 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.008162084, 2024-09-11T09:51:23.137 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:23.137 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:23.137 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:23.138 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:23.138 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:51:23.138 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:51:23.138 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 112 2024-09-11T09:51:23.138 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.138 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:23.138 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.138 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:23.138 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711712+0000", 2024-09-11T09:51:23.139 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:23.139 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.139 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.139 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:23.139 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711712+0000", 2024-09-11T09:51:23.139 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:23.139 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.139 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.139 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:23.139 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711712+0000", 2024-09-11T09:51:23.140 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.6000000000000004e-07 2024-09-11T09:51:23.140 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.140 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.140 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:23.140 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711808+0000", 2024-09-11T09:51:23.140 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.5977999999999995e-05 2024-09-11T09:51:23.140 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.140 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.140 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:23.140 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711819+0000", 2024-09-11T09:51:23.140 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.0755e-05 2024-09-11T09:51:23.141 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.141 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.141 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:23.141 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711826+0000", 2024-09-11T09:51:23.141 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.7970000000000003e-06 2024-09-11T09:51:23.141 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.141 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.141 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:23.141 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.717435+0000", 2024-09-11T09:51:23.141 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0056090239999999998 2024-09-11T09:51:23.141 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.141 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.142 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:23.142 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.717496+0000", 2024-09-11T09:51:23.142 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.0543e-05 2024-09-11T09:51:23.142 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.142 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.142 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:23.142 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.717591+0000", 2024-09-11T09:51:23.142 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.5667000000000004e-05 2024-09-11T09:51:23.142 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.142 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.142 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:23.143 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718831+0000", 2024-09-11T09:51:23.143 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001240204 2024-09-11T09:51:23.143 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.143 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.143 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:23.143 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719819+0000", 2024-09-11T09:51:23.143 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00098751500000000001 2024-09-11T09:51:23.143 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.143 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.143 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:23.143 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719850+0000", 2024-09-11T09:51:23.143 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.1535999999999998e-05 2024-09-11T09:51:23.144 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.144 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.144 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:23.144 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719874+0000", 2024-09-11T09:51:23.144 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.3504999999999999e-05 2024-09-11T09:51:23.144 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:23.144 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:23.144 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:23.144 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.144 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.144 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:113 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:51:23.145 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.711842+0000", 2024-09-11T09:51:23.145 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 71.401036012999995, 2024-09-11T09:51:23.145 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0081160099999999999, 2024-09-11T09:51:23.145 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:23.145 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:23.145 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:23.145 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:23.145 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:51:23.145 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:51:23.145 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 113 2024-09-11T09:51:23.145 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.146 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:23.146 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.146 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:23.146 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711842+0000", 2024-09-11T09:51:23.146 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:23.146 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.146 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.146 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:23.146 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711842+0000", 2024-09-11T09:51:23.146 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:23.146 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.146 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.146 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:23.147 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711843+0000", 2024-09-11T09:51:23.147 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.4399999999999999e-07 2024-09-11T09:51:23.147 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.147 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.147 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:23.147 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711848+0000", 2024-09-11T09:51:23.147 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.4159999999999997e-06 2024-09-11T09:51:23.147 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.147 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.147 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:23.148 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711856+0000", 2024-09-11T09:51:23.148 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.7870000000000007e-06 2024-09-11T09:51:23.148 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.148 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.148 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:23.148 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711862+0000", 2024-09-11T09:51:23.148 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.3689999999999998e-06 2024-09-11T09:51:23.148 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.148 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.148 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:23.148 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.717965+0000", 2024-09-11T09:51:23.149 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0061028899999999997 2024-09-11T09:51:23.149 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.149 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.149 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:23.149 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718023+0000", 2024-09-11T09:51:23.149 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.8313000000000002e-05 2024-09-11T09:51:23.149 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.149 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.150 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:23.150 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718109+0000", 2024-09-11T09:51:23.150 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.6205000000000006e-05 2024-09-11T09:51:23.150 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.150 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.150 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:23.150 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719808+0000", 2024-09-11T09:51:23.150 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0016990060000000001 2024-09-11T09:51:23.150 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.150 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.151 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:23.151 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719883+0000", 2024-09-11T09:51:23.151 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.4624999999999998e-05 2024-09-11T09:51:23.151 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.151 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.151 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:23.151 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719910+0000", 2024-09-11T09:51:23.151 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.7512e-05 2024-09-11T09:51:23.152 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.152 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.152 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:23.152 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719958+0000", 2024-09-11T09:51:23.152 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.8143000000000003e-05 2024-09-11T09:51:23.152 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:23.152 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:23.152 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:23.152 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.153 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.153 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:51:23.153 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.711887+0000", 2024-09-11T09:51:23.153 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 71.400991399000006, 2024-09-11T09:51:23.153 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0086063589999999992, 2024-09-11T09:51:23.153 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:23.153 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:23.153 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:23.153 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:23.153 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:51:23.153 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:51:23.154 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 114 2024-09-11T09:51:23.154 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.154 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:23.154 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.154 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:23.154 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711887+0000", 2024-09-11T09:51:23.154 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:23.154 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.154 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.154 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:23.154 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711887+0000", 2024-09-11T09:51:23.155 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:23.155 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.155 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.155 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:23.155 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711887+0000", 2024-09-11T09:51:23.155 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.1500000000000005e-07 2024-09-11T09:51:23.155 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.155 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.155 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:23.155 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711897+0000", 2024-09-11T09:51:23.155 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.1900000000000001e-06 2024-09-11T09:51:23.156 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.156 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.156 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:23.156 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711902+0000", 2024-09-11T09:51:23.156 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.2349999999999999e-06 2024-09-11T09:51:23.156 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.156 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.156 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:23.156 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711906+0000", 2024-09-11T09:51:23.156 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.7129999999999999e-06 2024-09-11T09:51:23.156 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.156 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.157 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:23.157 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718429+0000", 2024-09-11T09:51:23.157 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0065231960000000002 2024-09-11T09:51:23.157 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.157 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.157 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:23.157 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718480+0000", 2024-09-11T09:51:23.157 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.1161e-05 2024-09-11T09:51:23.157 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.157 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.157 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:23.158 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718558+0000", 2024-09-11T09:51:23.158 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.7519000000000001e-05 2024-09-11T09:51:23.158 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.158 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.158 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:23.158 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720038+0000", 2024-09-11T09:51:23.158 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001480016 2024-09-11T09:51:23.158 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.158 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.158 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:23.158 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720417+0000", 2024-09-11T09:51:23.159 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00037943699999999999 2024-09-11T09:51:23.159 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.159 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.159 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:23.159 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720452+0000", 2024-09-11T09:51:23.159 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.4595000000000003e-05 2024-09-11T09:51:23.159 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.159 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.159 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:23.159 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720493+0000", 2024-09-11T09:51:23.159 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.1782000000000001e-05 2024-09-11T09:51:23.160 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:23.160 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:23.160 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:23.160 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.160 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.160 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:51:23.160 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.711912+0000", 2024-09-11T09:51:23.160 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 71.400966729999993, 2024-09-11T09:51:23.160 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0086864750000000008, 2024-09-11T09:51:23.160 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:23.160 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:23.161 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:23.161 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:23.161 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:51:23.161 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:51:23.161 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 115 2024-09-11T09:51:23.161 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.161 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:23.161 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.161 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:23.161 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711912+0000", 2024-09-11T09:51:23.161 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:23.161 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.162 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.162 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:23.162 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711912+0000", 2024-09-11T09:51:23.162 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:23.162 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.162 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.162 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:23.162 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711912+0000", 2024-09-11T09:51:23.162 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.3900000000000001e-07 2024-09-11T09:51:23.162 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.162 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.163 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:23.163 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711969+0000", 2024-09-11T09:51:23.163 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.7501999999999999e-05 2024-09-11T09:51:23.163 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.163 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.163 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:23.163 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711986+0000", 2024-09-11T09:51:23.163 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.6402999999999999e-05 2024-09-11T09:51:23.163 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.163 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.164 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:23.164 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711990+0000", 2024-09-11T09:51:23.164 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.929e-06 2024-09-11T09:51:23.164 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.164 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.164 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:23.164 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718861+0000", 2024-09-11T09:51:23.164 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0068710530000000002 2024-09-11T09:51:23.164 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.164 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.164 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:23.165 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718908+0000", 2024-09-11T09:51:23.165 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.7431e-05 2024-09-11T09:51:23.165 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.165 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.165 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:23.165 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718984+0000", 2024-09-11T09:51:23.165 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.5500999999999994e-05 2024-09-11T09:51:23.165 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.165 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.165 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:23.165 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720500+0000", 2024-09-11T09:51:23.165 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0015158820000000001 2024-09-11T09:51:23.166 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.166 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.166 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:23.166 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720536+0000", 2024-09-11T09:51:23.166 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.6118000000000002e-05 2024-09-11T09:51:23.166 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.166 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.166 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:23.166 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720570+0000", 2024-09-11T09:51:23.166 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.4387999999999997e-05 2024-09-11T09:51:23.166 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.167 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.167 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:23.167 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720598+0000", 2024-09-11T09:51:23.167 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.8028999999999999e-05 2024-09-11T09:51:23.167 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:23.167 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:23.167 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:23.167 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.167 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.167 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:51:23.167 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.711998+0000", 2024-09-11T09:51:23.167 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 71.400880551, 2024-09-11T09:51:23.168 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.010785813999999999, 2024-09-11T09:51:23.168 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:23.168 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:23.168 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:23.168 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:23.168 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:51:23.168 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:51:23.168 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 116 2024-09-11T09:51:23.168 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.168 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:23.168 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.169 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:23.169 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711998+0000", 2024-09-11T09:51:23.169 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:23.169 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.169 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.169 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:23.169 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711998+0000", 2024-09-11T09:51:23.169 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:23.169 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.169 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.169 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:23.169 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711998+0000", 2024-09-11T09:51:23.170 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.3300000000000003e-07 2024-09-11T09:51:23.170 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.170 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.170 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:23.170 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712001+0000", 2024-09-11T09:51:23.170 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.9349999999999999e-06 2024-09-11T09:51:23.170 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.170 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.170 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:23.170 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712006+0000", 2024-09-11T09:51:23.170 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.1630000000000004e-06 2024-09-11T09:51:23.170 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.171 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.171 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:23.171 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712010+0000", 2024-09-11T09:51:23.171 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.574e-06 2024-09-11T09:51:23.171 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.171 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.171 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:23.171 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719415+0000", 2024-09-11T09:51:23.171 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0074049099999999998 2024-09-11T09:51:23.171 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.171 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.172 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:23.172 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719465+0000", 2024-09-11T09:51:23.172 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.0129000000000001e-05 2024-09-11T09:51:23.172 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.172 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.172 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:23.172 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719545+0000", 2024-09-11T09:51:23.172 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.0253999999999994e-05 2024-09-11T09:51:23.172 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.172 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.172 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:23.172 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720636+0000", 2024-09-11T09:51:23.173 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0010909330000000001 2024-09-11T09:51:23.173 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.173 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.173 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:23.173 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.722707+0000", 2024-09-11T09:51:23.173 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0020713300000000001 2024-09-11T09:51:23.173 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.173 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.173 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:23.173 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.722743+0000", 2024-09-11T09:51:23.173 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.5351000000000001e-05 2024-09-11T09:51:23.174 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.174 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.174 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:23.174 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.722784+0000", 2024-09-11T09:51:23.174 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.0802e-05 2024-09-11T09:51:23.174 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:23.174 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:23.174 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:23.174 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.174 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.174 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:51:23.175 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.712014+0000", 2024-09-11T09:51:23.175 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 71.400864510999995, 2024-09-11T09:51:23.175 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.013420938, 2024-09-11T09:51:23.175 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:23.175 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:23.175 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:23.175 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:23.175 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:51:23.175 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:51:23.175 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 117 2024-09-11T09:51:23.175 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.175 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:23.175 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.176 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:23.176 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712014+0000", 2024-09-11T09:51:23.176 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:23.176 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.176 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.176 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:23.176 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712014+0000", 2024-09-11T09:51:23.176 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:23.176 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.176 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.176 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:23.177 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712014+0000", 2024-09-11T09:51:23.177 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.2000000000000001e-07 2024-09-11T09:51:23.177 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.177 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.177 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:23.177 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712016+0000", 2024-09-11T09:51:23.177 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.2309999999999999e-06 2024-09-11T09:51:23.177 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.177 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.177 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:23.177 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712020+0000", 2024-09-11T09:51:23.177 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.3830000000000001e-06 2024-09-11T09:51:23.178 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.178 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.178 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:23.178 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712023+0000", 2024-09-11T09:51:23.178 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.3340000000000002e-06 2024-09-11T09:51:23.178 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.178 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.178 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:23.178 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720083+0000", 2024-09-11T09:51:23.178 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0080601189999999993 2024-09-11T09:51:23.178 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.178 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.179 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:23.179 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720125+0000", 2024-09-11T09:51:23.179 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.2029e-05 2024-09-11T09:51:23.179 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.179 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.179 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:23.179 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720205+0000", 2024-09-11T09:51:23.179 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.0010000000000001e-05 2024-09-11T09:51:23.179 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.179 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.179 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:23.180 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720849+0000", 2024-09-11T09:51:23.180 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00064369899999999996 2024-09-11T09:51:23.180 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.180 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.180 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:23.180 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.725357+0000", 2024-09-11T09:51:23.180 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0045077379999999998 2024-09-11T09:51:23.180 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.180 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.180 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:23.180 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.725392+0000", 2024-09-11T09:51:23.180 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.5862999999999999e-05 2024-09-11T09:51:23.181 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.181 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.181 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:23.181 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.725435+0000", 2024-09-11T09:51:23.181 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.2311999999999997e-05 2024-09-11T09:51:23.181 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:23.181 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:23.181 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:23.181 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.181 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.181 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:51:23.181 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:12.034184+0000", 2024-09-11T09:51:23.182 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 71.078694337000002, 2024-09-11T09:51:23.182 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0061520380000000003, 2024-09-11T09:51:23.182 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:23.182 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:23.182 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:23.182 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:23.182 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:51:23.182 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:51:23.182 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 141 2024-09-11T09:51:23.182 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.182 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:23.182 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.183 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:23.183 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034184+0000", 2024-09-11T09:51:23.183 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:23.183 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.183 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.183 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:23.183 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034184+0000", 2024-09-11T09:51:23.183 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:23.183 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.183 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.183 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:23.183 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034185+0000", 2024-09-11T09:51:23.184 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.1099999999999995e-07 2024-09-11T09:51:23.184 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.184 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.184 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:23.184 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034190+0000", 2024-09-11T09:51:23.184 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.7309999999999998e-06 2024-09-11T09:51:23.184 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.184 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.184 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:23.184 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034197+0000", 2024-09-11T09:51:23.184 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.7440000000000001e-06 2024-09-11T09:51:23.184 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.185 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.185 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:23.185 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.034204+0000", 2024-09-11T09:51:23.185 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.8970000000000002e-06 2024-09-11T09:51:23.185 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.185 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.185 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:23.185 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.039035+0000", 2024-09-11T09:51:23.185 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0048307039999999999 2024-09-11T09:51:23.185 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.185 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.186 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:23.186 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.039093+0000", 2024-09-11T09:51:23.186 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.8604000000000003e-05 2024-09-11T09:51:23.186 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.186 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.186 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:23.186 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.039184+0000", 2024-09-11T09:51:23.186 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.0196000000000002e-05 2024-09-11T09:51:23.186 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.186 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.186 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:23.187 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.039873+0000", 2024-09-11T09:51:23.187 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00068960600000000005 2024-09-11T09:51:23.187 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.187 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.187 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:23.187 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.040292+0000", 2024-09-11T09:51:23.187 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00041865199999999999 2024-09-11T09:51:23.187 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.187 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.187 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:23.187 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.040317+0000", 2024-09-11T09:51:23.187 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.5439000000000002e-05 2024-09-11T09:51:23.188 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.188 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.188 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:23.188 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.040336+0000", 2024-09-11T09:51:23.188 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.8754e-05 2024-09-11T09:51:23.188 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:23.188 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:23.188 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:23.188 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.188 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.188 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:157 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:51:23.189 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:12.452040+0000", 2024-09-11T09:51:23.189 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 70.660838483999996, 2024-09-11T09:51:23.189 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0068848269999999996, 2024-09-11T09:51:23.189 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:23.189 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:23.189 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:23.189 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:23.189 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:51:23.189 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:51:23.189 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 157 2024-09-11T09:51:23.189 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.190 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:23.190 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.190 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:23.190 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452040+0000", 2024-09-11T09:51:23.190 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:23.190 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.190 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.190 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:23.190 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452040+0000", 2024-09-11T09:51:23.190 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:23.190 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.191 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.191 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:23.191 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452040+0000", 2024-09-11T09:51:23.191 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.22e-07 2024-09-11T09:51:23.191 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.191 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.191 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:23.191 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452042+0000", 2024-09-11T09:51:23.191 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.7010000000000001e-06 2024-09-11T09:51:23.191 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.191 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.192 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:23.192 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452045+0000", 2024-09-11T09:51:23.192 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.1310000000000002e-06 2024-09-11T09:51:23.192 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.192 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.192 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:23.192 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452048+0000", 2024-09-11T09:51:23.192 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.65e-06 2024-09-11T09:51:23.192 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.192 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.192 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:23.193 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.454453+0000", 2024-09-11T09:51:23.193 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0024052959999999999 2024-09-11T09:51:23.196 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.196 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.196 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:23.196 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.454498+0000", 2024-09-11T09:51:23.196 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.4978999999999997e-05 2024-09-11T09:51:23.196 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.196 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.197 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:23.197 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.454567+0000", 2024-09-11T09:51:23.197 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.9110999999999996e-05 2024-09-11T09:51:23.197 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.197 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.197 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:23.197 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.455464+0000", 2024-09-11T09:51:23.197 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00089696199999999995 2024-09-11T09:51:23.197 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.197 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.197 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:23.197 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.458844+0000", 2024-09-11T09:51:23.198 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0033802469999999999 2024-09-11T09:51:23.198 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.198 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.198 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:23.198 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.458889+0000", 2024-09-11T09:51:23.198 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.4441999999999999e-05 2024-09-11T09:51:23.198 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.198 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.198 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:23.198 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.458925+0000", 2024-09-11T09:51:23.198 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.6086e-05 2024-09-11T09:51:23.198 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:23.198 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:23.199 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:23.199 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.199 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.199 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:158 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:51:23.199 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:12.452055+0000", 2024-09-11T09:51:23.199 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 70.660823090999997, 2024-09-11T09:51:23.199 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0069533829999999996, 2024-09-11T09:51:23.199 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:23.199 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:23.199 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:23.199 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:23.200 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:51:23.200 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:51:23.200 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 158 2024-09-11T09:51:23.200 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.200 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:23.200 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.200 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:23.200 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452055+0000", 2024-09-11T09:51:23.200 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:23.200 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.200 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.200 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:23.201 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452055+0000", 2024-09-11T09:51:23.201 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:23.201 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.201 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.201 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:23.201 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452056+0000", 2024-09-11T09:51:23.201 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.48e-07 2024-09-11T09:51:23.201 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.201 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.201 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:23.201 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452087+0000", 2024-09-11T09:51:23.202 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.1149000000000003e-05 2024-09-11T09:51:23.202 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.202 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.202 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:23.202 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452092+0000", 2024-09-11T09:51:23.202 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.8400000000000002e-06 2024-09-11T09:51:23.202 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.202 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.202 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:23.202 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452095+0000", 2024-09-11T09:51:23.202 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.0740000000000001e-06 2024-09-11T09:51:23.202 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.203 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.203 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:23.203 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.454796+0000", 2024-09-11T09:51:23.203 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0027009740000000001 2024-09-11T09:51:23.203 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.203 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.203 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:23.203 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.454869+0000", 2024-09-11T09:51:23.203 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.3678999999999997e-05 2024-09-11T09:51:23.203 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.203 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.204 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:23.204 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.454992+0000", 2024-09-11T09:51:23.204 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00012279000000000001 2024-09-11T09:51:23.204 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.204 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.204 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:23.204 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.456072+0000", 2024-09-11T09:51:23.204 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001080229 2024-09-11T09:51:23.204 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.204 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.204 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:23.205 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.458931+0000", 2024-09-11T09:51:23.205 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0028591789999999999 2024-09-11T09:51:23.205 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.205 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.205 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:23.205 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.458972+0000", 2024-09-11T09:51:23.205 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.0170999999999999e-05 2024-09-11T09:51:23.205 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.205 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.205 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:23.205 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.459009+0000", 2024-09-11T09:51:23.206 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.7049999999999999e-05 2024-09-11T09:51:23.206 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:23.206 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:23.206 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:23.206 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.206 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.206 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:159 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:51:23.206 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:12.452099+0000", 2024-09-11T09:51:23.206 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 70.660778922999995, 2024-09-11T09:51:23.206 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0069885950000000002, 2024-09-11T09:51:23.206 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:23.206 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:23.207 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:23.207 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:23.207 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:51:23.207 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:51:23.207 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 159 2024-09-11T09:51:23.207 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.207 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:23.207 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.207 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:23.207 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452099+0000", 2024-09-11T09:51:23.207 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:23.208 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.208 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.208 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:23.208 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452099+0000", 2024-09-11T09:51:23.208 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:23.208 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.208 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.208 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:23.208 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452100+0000", 2024-09-11T09:51:23.208 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.2800000000000003e-07 2024-09-11T09:51:23.208 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.209 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.209 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:23.209 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452102+0000", 2024-09-11T09:51:23.209 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.824e-06 2024-09-11T09:51:23.209 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.209 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.209 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:23.209 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452105+0000", 2024-09-11T09:51:23.209 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.0460000000000001e-06 2024-09-11T09:51:23.209 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.209 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.210 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:23.210 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452107+0000", 2024-09-11T09:51:23.210 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.543e-06 2024-09-11T09:51:23.210 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.210 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.210 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:23.210 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.455484+0000", 2024-09-11T09:51:23.210 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0033769759999999998 2024-09-11T09:51:23.210 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.210 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.210 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:23.211 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.455527+0000", 2024-09-11T09:51:23.211 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.2589999999999997e-05 2024-09-11T09:51:23.211 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.211 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.211 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:23.211 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.455594+0000", 2024-09-11T09:51:23.211 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.6775999999999999e-05 2024-09-11T09:51:23.211 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.211 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.211 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:23.211 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.456312+0000", 2024-09-11T09:51:23.211 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00071874700000000005 2024-09-11T09:51:23.212 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.212 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.212 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:23.212 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.459015+0000", 2024-09-11T09:51:23.212 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0027028540000000002 2024-09-11T09:51:23.212 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.212 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.212 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:23.212 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.459054+0000", 2024-09-11T09:51:23.212 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.8840000000000001e-05 2024-09-11T09:51:23.212 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.213 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.213 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:23.213 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.459088+0000", 2024-09-11T09:51:23.213 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.4071e-05 2024-09-11T09:51:23.213 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:23.213 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:23.213 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:23.213 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.213 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.213 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4688.0:38 2.5 2:abde2694:::rbd_data.1250f5e25b80.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:23.213 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:22.903926+0000", 2024-09-11T09:51:23.213 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 0.20895224700000001, 2024-09-11T09:51:23.214 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.028303676, 2024-09-11T09:51:23.214 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:23.214 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:23.214 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:23.214 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:23.214 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4688", 2024-09-11T09:51:23.214 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:51:23.214 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 38 2024-09-11T09:51:23.214 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.214 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:23.214 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.214 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:23.215 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.903926+0000", 2024-09-11T09:51:23.215 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:23.215 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.215 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.215 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:23.215 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.903926+0000", 2024-09-11T09:51:23.215 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:23.215 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.215 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.215 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:23.215 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.903928+0000", 2024-09-11T09:51:23.215 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.4249999999999999e-06 2024-09-11T09:51:23.216 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.216 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.216 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:23.216 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.914774+0000", 2024-09-11T09:51:23.216 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.010846613 2024-09-11T09:51:23.216 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.216 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.216 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:23.216 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.914793+0000", 2024-09-11T09:51:23.216 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.8879999999999999e-05 2024-09-11T09:51:23.216 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.216 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.217 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:23.217 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.914801+0000", 2024-09-11T09:51:23.217 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.2230000000000007e-06 2024-09-11T09:51:23.217 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.217 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.217 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:23.217 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.914854+0000", 2024-09-11T09:51:23.217 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.2883999999999997e-05 2024-09-11T09:51:23.217 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.217 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.217 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:23.217 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.914951+0000", 2024-09-11T09:51:23.218 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.6731000000000006e-05 2024-09-11T09:51:23.218 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.218 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.218 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:23.218 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.915080+0000", 2024-09-11T09:51:23.218 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00012926999999999999 2024-09-11T09:51:23.218 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.218 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.218 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:23.218 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.931469+0000", 2024-09-11T09:51:23.219 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.016389107 2024-09-11T09:51:23.219 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.219 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.219 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:23.219 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.932118+0000", 2024-09-11T09:51:23.219 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00064915600000000002 2024-09-11T09:51:23.219 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.219 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.219 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:23.219 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.932177+0000", 2024-09-11T09:51:23.219 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.9032e-05 2024-09-11T09:51:23.219 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.220 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.220 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:23.220 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.932230+0000", 2024-09-11T09:51:23.220 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.2355000000000003e-05 2024-09-11T09:51:23.220 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:23.220 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:23.220 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:23.220 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.220 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.220 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_repop(client.4688.0:36 2.6 e53/15 2:61e1b442:::rbd_data.1250f5e25b80.0000000000000003:head v 53'2, mlcod=0'0)", 2024-09-11T09:51:23.221 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:22.915597+0000", 2024-09-11T09:51:23.221 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 0.19728169700000001, 2024-09-11T09:51:23.221 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.020430798, 2024-09-11T09:51:23.221 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:23.221 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:23.221 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:23.221 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:23.221 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.221 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:23.221 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.915597+0000", 2024-09-11T09:51:23.221 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:23.221 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.222 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.222 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:23.222 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.915597+0000", 2024-09-11T09:51:23.222 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:23.222 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.222 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.222 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:23.222 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.915598+0000", 2024-09-11T09:51:23.222 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.7150000000000001e-06 2024-09-11T09:51:23.222 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.222 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.223 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:23.223 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921741+0000", 2024-09-11T09:51:23.223 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0061424339999999996 2024-09-11T09:51:23.223 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.223 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.223 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:23.223 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921757+0000", 2024-09-11T09:51:23.223 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.6144e-05 2024-09-11T09:51:23.223 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.223 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.224 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:23.224 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921765+0000", 2024-09-11T09:51:23.224 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.0809999999999994e-06 2024-09-11T09:51:23.224 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.224 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.224 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:23.224 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921826+0000", 2024-09-11T09:51:23.224 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.0878999999999998e-05 2024-09-11T09:51:23.224 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.224 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.224 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:23.225 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921849+0000", 2024-09-11T09:51:23.225 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.2966999999999999e-05 2024-09-11T09:51:23.225 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.225 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.225 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:23.225 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.935968+0000", 2024-09-11T09:51:23.225 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.014119191 2024-09-11T09:51:23.225 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.226 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.226 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:23.226 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.936027+0000", 2024-09-11T09:51:23.226 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.9386999999999999e-05 2024-09-11T09:51:23.226 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:23.226 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:23.226 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:23.226 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.226 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.227 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_repop(client.4688.0:39 2.6 e53/15 2:743e7616:::rbd_data.1250f5e25b80.0000000000000005:head v 53'3, mlcod=0'0)", 2024-09-11T09:51:23.227 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:22.921806+0000", 2024-09-11T09:51:23.227 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 0.19107264700000001, 2024-09-11T09:51:23.227 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.015275018, 2024-09-11T09:51:23.227 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:23.227 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:23.227 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:23.227 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:23.228 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.228 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:23.228 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921806+0000", 2024-09-11T09:51:23.228 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:23.228 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.228 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.228 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:23.228 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921806+0000", 2024-09-11T09:51:23.228 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:23.229 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.229 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.229 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:23.229 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921806+0000", 2024-09-11T09:51:23.229 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.2200000000000003e-07 2024-09-11T09:51:23.229 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.229 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.229 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:23.229 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.925590+0000", 2024-09-11T09:51:23.230 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0037838949999999998 2024-09-11T09:51:23.230 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.230 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.230 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:23.230 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.925605+0000", 2024-09-11T09:51:23.230 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.4321999999999999e-05 2024-09-11T09:51:23.230 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.230 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.230 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:23.230 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.925613+0000", 2024-09-11T09:51:23.230 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.1980000000000001e-06 2024-09-11T09:51:23.230 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.231 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.231 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:23.231 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.925857+0000", 2024-09-11T09:51:23.231 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000244064 2024-09-11T09:51:23.231 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.231 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.231 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:23.231 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.925875+0000", 2024-09-11T09:51:23.231 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.838e-05 2024-09-11T09:51:23.231 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.231 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.231 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:23.232 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.937042+0000", 2024-09-11T09:51:23.232 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.011166416 2024-09-11T09:51:23.232 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.232 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.232 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:23.232 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.937081+0000", 2024-09-11T09:51:23.232 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.9020999999999998e-05 2024-09-11T09:51:23.232 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:23.232 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:23.232 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:23.232 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.233 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.233 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4688.0:49 2.5 2:b6dd9fc9:::rbd_data.1250f5e25b80.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:23.233 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:22.938548+0000", 2024-09-11T09:51:23.233 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 0.17433064400000001, 2024-09-11T09:51:23.233 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.022766950000000001, 2024-09-11T09:51:23.233 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:23.233 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:23.233 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:23.233 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:23.233 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4688", 2024-09-11T09:51:23.233 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:51:23.234 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 49 2024-09-11T09:51:23.234 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.234 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:23.234 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.234 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:23.234 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.938548+0000", 2024-09-11T09:51:23.234 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:23.234 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.234 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.235 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:23.235 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.938548+0000", 2024-09-11T09:51:23.235 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:23.235 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.235 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.235 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:23.235 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.938550+0000", 2024-09-11T09:51:23.235 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.7710000000000001e-06 2024-09-11T09:51:23.235 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.235 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.235 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:23.236 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.944237+0000", 2024-09-11T09:51:23.236 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0056879790000000001 2024-09-11T09:51:23.236 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.236 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.236 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:23.236 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.944262+0000", 2024-09-11T09:51:23.236 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.4247e-05 2024-09-11T09:51:23.236 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.236 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.236 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:23.236 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.944274+0000", 2024-09-11T09:51:23.236 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.2639999999999999e-05 2024-09-11T09:51:23.237 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.237 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.237 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:23.237 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.944344+0000", 2024-09-11T09:51:23.237 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.9574999999999997e-05 2024-09-11T09:51:23.237 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.237 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.237 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:23.237 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.944474+0000", 2024-09-11T09:51:23.237 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000130464 2024-09-11T09:51:23.237 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.237 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.237 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:23.238 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.944585+0000", 2024-09-11T09:51:23.238 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000110096 2024-09-11T09:51:23.238 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.238 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.238 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:23.238 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.952724+0000", 2024-09-11T09:51:23.238 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0081398900000000003 2024-09-11T09:51:23.238 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.238 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.238 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:23.238 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.961219+0000", 2024-09-11T09:51:23.238 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0084949789999999997 2024-09-11T09:51:23.239 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.239 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.239 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:23.239 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.961280+0000", 2024-09-11T09:51:23.239 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.0404e-05 2024-09-11T09:51:23.239 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.239 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.239 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:23.239 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.961315+0000", 2024-09-11T09:51:23.239 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.4904999999999999e-05 2024-09-11T09:51:23.239 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:23.239 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:23.240 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:23.240 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.240 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.240 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_repop(client.4688.0:44 2.6 e53/15 2:67246a22:::rbd_data.1250f5e25b80.0000000000000007:head v 53'4, mlcod=53'2)", 2024-09-11T09:51:23.240 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:22.951721+0000", 2024-09-11T09:51:23.240 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 0.16115737199999999, 2024-09-11T09:51:23.240 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.01080981, 2024-09-11T09:51:23.240 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:23.240 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:23.240 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:23.240 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:23.240 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.241 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:23.241 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.951721+0000", 2024-09-11T09:51:23.241 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:23.241 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.241 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.241 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:23.241 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.951721+0000", 2024-09-11T09:51:23.241 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:23.241 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.241 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.242 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:23.242 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.951722+0000", 2024-09-11T09:51:23.242 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.0279999999999999e-06 2024-09-11T09:51:23.242 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.242 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.242 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:23.242 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.955612+0000", 2024-09-11T09:51:23.242 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0038896920000000001 2024-09-11T09:51:23.242 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.242 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.243 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:23.243 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.955628+0000", 2024-09-11T09:51:23.243 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.5945000000000001e-05 2024-09-11T09:51:23.243 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.243 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.243 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:23.243 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.955635+0000", 2024-09-11T09:51:23.243 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.8080000000000005e-06 2024-09-11T09:51:23.243 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.243 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.243 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:23.244 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.955700+0000", 2024-09-11T09:51:23.244 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.4114000000000003e-05 2024-09-11T09:51:23.244 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.244 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.244 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:23.244 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.955724+0000", 2024-09-11T09:51:23.244 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.3989999999999999e-05 2024-09-11T09:51:23.244 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.244 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.244 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:23.244 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.962490+0000", 2024-09-11T09:51:23.244 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0067662599999999996 2024-09-11T09:51:23.245 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.245 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.245 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:23.245 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.962531+0000", 2024-09-11T09:51:23.245 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.0973000000000002e-05 2024-09-11T09:51:23.245 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:23.245 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:23.245 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:23.245 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.245 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.245 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_repop(client.4688.0:51 2.6 e53/15 2:6b5894fc:::rbd_data.1250f5e25b80.000000000000000a:head v 53'5, mlcod=53'2)", 2024-09-11T09:51:23.245 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:22.959030+0000", 2024-09-11T09:51:23.246 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 0.15384819799999999, 2024-09-11T09:51:23.246 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.011503638, 2024-09-11T09:51:23.246 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:23.246 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:23.246 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:23.246 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:23.246 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.246 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:23.246 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.959030+0000", 2024-09-11T09:51:23.246 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:23.246 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.247 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.247 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:23.247 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.959030+0000", 2024-09-11T09:51:23.247 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:23.247 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.247 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.247 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:23.247 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.959032+0000", 2024-09-11T09:51:23.247 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.356e-06 2024-09-11T09:51:23.247 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.247 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.247 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:23.248 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.962899+0000", 2024-09-11T09:51:23.248 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.003867639 2024-09-11T09:51:23.248 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.248 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.248 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:23.248 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.962909+0000", 2024-09-11T09:51:23.248 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.0105999999999999e-05 2024-09-11T09:51:23.248 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.248 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.248 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:23.248 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.962915+0000", 2024-09-11T09:51:23.249 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.8869999999999997e-06 2024-09-11T09:51:23.249 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.249 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.249 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:23.249 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.962962+0000", 2024-09-11T09:51:23.249 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.6511999999999998e-05 2024-09-11T09:51:23.249 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.249 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.249 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:23.249 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.962989+0000", 2024-09-11T09:51:23.249 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.7702999999999999e-05 2024-09-11T09:51:23.249 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.250 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.250 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:23.250 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.970495+0000", 2024-09-11T09:51:23.250 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0075052449999999998 2024-09-11T09:51:23.250 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.250 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.250 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:23.250 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.970534+0000", 2024-09-11T09:51:23.250 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.9190000000000003e-05 2024-09-11T09:51:23.250 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:23.250 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:23.250 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:23.251 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.251 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.251 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_repop(client.4688.0:58 2.6 e53/15 2:6277c644:::rbd_data.1250f5e25b80.000000000000000e:head v 53'6, mlcod=53'4)", 2024-09-11T09:51:23.251 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:22.996199+0000", 2024-09-11T09:51:23.251 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 0.116679278, 2024-09-11T09:51:23.251 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.010145573999999999, 2024-09-11T09:51:23.251 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:23.251 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:23.251 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:23.251 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:23.251 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.252 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:23.252 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.996199+0000", 2024-09-11T09:51:23.252 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:23.252 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.252 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.252 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:23.252 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.996199+0000", 2024-09-11T09:51:23.252 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:23.252 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.252 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.252 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:23.252 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.996200+0000", 2024-09-11T09:51:23.253 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.85e-07 2024-09-11T09:51:23.253 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.253 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.253 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:23.253 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.999914+0000", 2024-09-11T09:51:23.253 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.003714406 2024-09-11T09:51:23.253 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.253 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.253 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:23.253 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.999928+0000", 2024-09-11T09:51:23.253 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.3149000000000001e-05 2024-09-11T09:51:23.254 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.254 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.254 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:23.254 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.999951+0000", 2024-09-11T09:51:23.254 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.3116e-05 2024-09-11T09:51:23.254 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.254 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.254 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:23.254 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.000003+0000", 2024-09-11T09:51:23.254 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.2169000000000001e-05 2024-09-11T09:51:23.254 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.255 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.255 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:23.255 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.000018+0000", 2024-09-11T09:51:23.255 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.5500000000000001e-05 2024-09-11T09:51:23.255 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.255 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.255 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:23.255 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.006303+0000", 2024-09-11T09:51:23.255 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0062846710000000004 2024-09-11T09:51:23.255 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:23.255 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:23.255 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:23.256 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.006345+0000", 2024-09-11T09:51:23.256 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.1678000000000003e-05 2024-09-11T09:51:23.256 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:23.256 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:23.256 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:23.256 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:23.256 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:23.256 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-11T09:51:23.258 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:51:23.259+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:51:23.258 INFO:tasks.workunit.client.0.smithi107.stderr:2024-09-11T09:51:23.184+0000 7f275c8ad640 0 -- 172.21.15.107:0/1790305122 >> [v2:172.21.15.96:6816/332854807,v1:172.21.15.96:6817/332854807] conn(0x7f274003d3a0 msgr2=0x7f274003f790 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-11T09:51:23.258 INFO:tasks.workunit.client.0.smithi107.stderr:+ ceph osd pool create new_default_pool27121 8 2024-09-11T09:51:23.259 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-11T09:51:23.358 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-09-11T09:51:23.359+0000 7f7b6c7d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31904) UID: 0 2024-09-11T09:51:23.403 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-11T09:51:23.404 INFO:teuthology.orchestra.run.smithi096.stdout: "ops": [], 2024-09-11T09:51:23.404 INFO:teuthology.orchestra.run.smithi096.stdout: "num_ops": 0 2024-09-11T09:51:23.404 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-11T09:51:23.412 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-11T09:51:23.460 INFO:tasks.ceph.osd.0.smithi017.stderr:2024-09-11T09:51:23.461+0000 7fb73c0f1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31665) UID: 0 2024-09-11T09:51:23.555 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-11T09:51:23.555 INFO:teuthology.orchestra.run.smithi096.stdout: "ops": [], 2024-09-11T09:51:23.555 INFO:teuthology.orchestra.run.smithi096.stdout: "complaint_time": 30, 2024-09-11T09:51:23.555 INFO:teuthology.orchestra.run.smithi096.stdout: "num_blocked_ops": 0 2024-09-11T09:51:23.556 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-11T09:51:23.560 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:51:23.561+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:51:23.564 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-11T09:51:23.660 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-09-11T09:51:23.661+0000 7f7b6c7d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31904) UID: 0 2024-09-11T09:51:23.666 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-11T09:51:23.666 INFO:teuthology.orchestra.run.smithi096.stdout: "size": 20, 2024-09-11T09:51:23.666 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 600, 2024-09-11T09:51:23.666 INFO:teuthology.orchestra.run.smithi096.stdout: "ops": [] 2024-09-11T09:51:23.666 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-11T09:51:23.675 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-11T09:51:23.709 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-11T09:51:23.761 INFO:tasks.ceph.osd.2.smithi017.stderr:2024-09-11T09:51:23.762+0000 7fd20b170640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31672) UID: 0 2024-09-11T09:51:23.803 INFO:tasks.workunit.client.0.smithi107.stderr:pool 'new_default_pool27121' already exists 2024-09-11T09:51:23.814 INFO:tasks.workunit.client.0.smithi107.stderr:+ rbd pool init new_default_pool27121 2024-09-11T09:51:23.819 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-11T09:51:23.819 INFO:teuthology.orchestra.run.smithi096.stdout: "ops": [], 2024-09-11T09:51:23.819 INFO:teuthology.orchestra.run.smithi096.stdout: "num_ops": 0 2024-09-11T09:51:23.819 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-11T09:51:23.828 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-11T09:51:23.861 INFO:tasks.ceph.osd.2.smithi017.stderr:2024-09-11T09:51:23.862+0000 7fd20b170640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31672) UID: 0 2024-09-11T09:51:23.930 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-11T09:51:23.930 INFO:teuthology.orchestra.run.smithi096.stdout: "ops": [], 2024-09-11T09:51:23.930 INFO:teuthology.orchestra.run.smithi096.stdout: "complaint_time": 30, 2024-09-11T09:51:23.930 INFO:teuthology.orchestra.run.smithi096.stdout: "num_blocked_ops": 0 2024-09-11T09:51:23.930 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-11T09:51:23.938 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-11T09:51:23.984 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:51:23.985+0000 7fdec5849640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 31907) UID: 0 2024-09-11T09:51:24.045 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-11T09:51:24.045 INFO:teuthology.orchestra.run.smithi096.stdout: "size": 20, 2024-09-11T09:51:24.045 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 600, 2024-09-11T09:51:24.045 INFO:teuthology.orchestra.run.smithi096.stdout: "ops": [ 2024-09-11T09:51:24.046 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.046 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4230.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-11T09:51:24.046 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:18.490736+0000", 2024-09-11T09:51:24.046 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 65.553261677999998, 2024-09-11T09:51:24.046 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0059097569999999999, 2024-09-11T09:51:24.047 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:24.047 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:24.047 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:24.047 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:24.047 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4230", 2024-09-11T09:51:24.047 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.17:0/4101109576", 2024-09-11T09:51:24.047 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 1 2024-09-11T09:51:24.047 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.047 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:24.048 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.048 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:24.048 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.490736+0000", 2024-09-11T09:51:24.048 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:24.048 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.048 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.048 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:24.048 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.490736+0000", 2024-09-11T09:51:24.049 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:24.049 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.049 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.049 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:24.049 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.490742+0000", 2024-09-11T09:51:24.049 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.48e-06 2024-09-11T09:51:24.049 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.049 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.050 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:24.050 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.490749+0000", 2024-09-11T09:51:24.050 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.4569999999999999e-06 2024-09-11T09:51:24.050 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.050 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.050 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:24.050 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.490762+0000", 2024-09-11T09:51:24.050 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.2721000000000001e-05 2024-09-11T09:51:24.050 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.050 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.050 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:24.050 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.490774+0000", 2024-09-11T09:51:24.050 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1823e-05 2024-09-11T09:51:24.051 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.051 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.051 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:24.051 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.490846+0000", 2024-09-11T09:51:24.051 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.1878000000000005e-05 2024-09-11T09:51:24.051 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.051 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.051 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for peered", 2024-09-11T09:51:24.051 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.490857+0000", 2024-09-11T09:51:24.051 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1017e-05 2024-09-11T09:51:24.051 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.051 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.051 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:24.052 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.493856+0000", 2024-09-11T09:51:24.052 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00299885 2024-09-11T09:51:24.052 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.052 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.052 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:24.052 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.494020+0000", 2024-09-11T09:51:24.052 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00016436599999999999 2024-09-11T09:51:24.052 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.052 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.052 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 2", 2024-09-11T09:51:24.052 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.494186+0000", 2024-09-11T09:51:24.053 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00016562799999999999 2024-09-11T09:51:24.053 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.053 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.053 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:24.053 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.495407+0000", 2024-09-11T09:51:24.053 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0012217420000000001 2024-09-11T09:51:24.053 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.053 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.053 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:24.054 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.496579+0000", 2024-09-11T09:51:24.054 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001171844 2024-09-11T09:51:24.054 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.054 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.054 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:24.054 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.496620+0000", 2024-09-11T09:51:24.054 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.0479000000000003e-05 2024-09-11T09:51:24.054 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.054 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.054 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:24.055 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.496646+0000", 2024-09-11T09:51:24.055 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.6472e-05 2024-09-11T09:51:24.055 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:24.055 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:24.055 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:24.055 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.055 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.055 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4230.0:5 2.2 2:5cea7035:::rbd_info:head [create,write 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-11T09:51:24.055 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:19.492662+0000", 2024-09-11T09:51:24.055 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 64.551336057, 2024-09-11T09:51:24.056 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0043824759999999997, 2024-09-11T09:51:24.056 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:24.056 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:24.056 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:24.056 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:24.056 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4230", 2024-09-11T09:51:24.056 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.17:0/4101109576", 2024-09-11T09:51:24.056 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 5 2024-09-11T09:51:24.056 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.056 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:24.056 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.056 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:24.056 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.492662+0000", 2024-09-11T09:51:24.057 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:24.057 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.057 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.057 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:24.057 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.492662+0000", 2024-09-11T09:51:24.057 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:24.057 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.057 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.057 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:24.057 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.492664+0000", 2024-09-11T09:51:24.057 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.3599999999999999e-06 2024-09-11T09:51:24.058 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.058 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.058 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:24.058 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.492675+0000", 2024-09-11T09:51:24.058 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1182000000000001e-05 2024-09-11T09:51:24.058 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.058 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.058 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:24.058 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.492693+0000", 2024-09-11T09:51:24.058 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.7031999999999999e-05 2024-09-11T09:51:24.059 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.059 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.059 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:24.059 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.492702+0000", 2024-09-11T09:51:24.059 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.5540000000000003e-06 2024-09-11T09:51:24.059 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.059 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.059 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:24.059 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.492767+0000", 2024-09-11T09:51:24.059 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.5411000000000001e-05 2024-09-11T09:51:24.059 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.059 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.060 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op must wait for map", 2024-09-11T09:51:24.060 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.492773+0000", 2024-09-11T09:51:24.060 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.1039999999999998e-06 2024-09-11T09:51:24.060 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.060 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.060 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:24.060 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.494354+0000", 2024-09-11T09:51:24.060 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0015814049999999999 2024-09-11T09:51:24.060 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.060 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.060 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:24.061 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.494476+0000", 2024-09-11T09:51:24.061 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000122089 2024-09-11T09:51:24.061 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.061 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.061 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 1", 2024-09-11T09:51:24.061 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.494577+0000", 2024-09-11T09:51:24.061 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000100536 2024-09-11T09:51:24.061 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.061 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.061 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:24.061 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.495770+0000", 2024-09-11T09:51:24.061 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0011932469999999999 2024-09-11T09:51:24.062 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.062 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.062 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:24.062 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.496966+0000", 2024-09-11T09:51:24.062 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001196003 2024-09-11T09:51:24.062 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.062 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.062 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:24.062 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.497011+0000", 2024-09-11T09:51:24.062 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.5229999999999999e-05 2024-09-11T09:51:24.062 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.062 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.062 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:24.063 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.497044+0000", 2024-09-11T09:51:24.063 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.3322999999999999e-05 2024-09-11T09:51:24.063 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:24.063 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:24.063 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:24.063 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.063 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.063 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4230.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-09-11T09:51:24.063 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:20.498634+0000", 2024-09-11T09:51:24.063 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 63.545363657000003, 2024-09-11T09:51:24.063 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0050648539999999997, 2024-09-11T09:51:24.064 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:24.064 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:24.064 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:24.064 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:24.064 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4230", 2024-09-11T09:51:24.064 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.17:0/4101109576", 2024-09-11T09:51:24.064 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 7 2024-09-11T09:51:24.064 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.064 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:24.064 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.064 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:24.065 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.498634+0000", 2024-09-11T09:51:24.065 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:24.065 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.065 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.065 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:24.065 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.498634+0000", 2024-09-11T09:51:24.065 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:24.065 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.065 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.066 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:24.066 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.498637+0000", 2024-09-11T09:51:24.066 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.1919999999999999e-06 2024-09-11T09:51:24.066 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.066 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.066 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:24.066 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.498648+0000", 2024-09-11T09:51:24.066 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1116e-05 2024-09-11T09:51:24.066 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.066 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.066 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:24.066 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.498666+0000", 2024-09-11T09:51:24.067 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.8411999999999999e-05 2024-09-11T09:51:24.067 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.067 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.067 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:24.067 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.498675+0000", 2024-09-11T09:51:24.067 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.4059999999999994e-06 2024-09-11T09:51:24.067 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.067 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.067 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:24.067 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.498745+0000", 2024-09-11T09:51:24.067 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.9764000000000005e-05 2024-09-11T09:51:24.067 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.067 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.067 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op must wait for map", 2024-09-11T09:51:24.068 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.498754+0000", 2024-09-11T09:51:24.068 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.5969999999999992e-06 2024-09-11T09:51:24.068 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.068 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.068 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:24.068 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.500385+0000", 2024-09-11T09:51:24.068 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0016308360000000001 2024-09-11T09:51:24.068 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.068 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.068 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:24.068 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.500473+0000", 2024-09-11T09:51:24.068 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.7966000000000004e-05 2024-09-11T09:51:24.069 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.069 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.069 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 1", 2024-09-11T09:51:24.069 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.500591+0000", 2024-09-11T09:51:24.069 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000117895 2024-09-11T09:51:24.069 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.069 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.069 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:24.069 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.501685+0000", 2024-09-11T09:51:24.069 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001094886 2024-09-11T09:51:24.069 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.069 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.069 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:24.070 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.503627+0000", 2024-09-11T09:51:24.070 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001941199 2024-09-11T09:51:24.070 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.070 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.070 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:24.070 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.503667+0000", 2024-09-11T09:51:24.070 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.0645000000000001e-05 2024-09-11T09:51:24.070 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.070 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.070 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:24.070 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.503699+0000", 2024-09-11T09:51:24.070 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.1940000000000003e-05 2024-09-11T09:51:24.070 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:24.071 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:24.071 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:24.071 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.071 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.071 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4688.0:32 2.2 2:45941fd2:::rbd_data.1250f5e25b80.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:24.071 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:22.896607+0000", 2024-09-11T09:51:24.071 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 1.1473910540000001, 2024-09-11T09:51:24.071 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.038655179999999997, 2024-09-11T09:51:24.071 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:24.071 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:24.071 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:24.071 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:24.071 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4688", 2024-09-11T09:51:24.072 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:51:24.072 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 32 2024-09-11T09:51:24.072 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.072 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:24.072 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.072 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:24.072 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.896607+0000", 2024-09-11T09:51:24.072 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:24.072 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.072 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.072 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:24.072 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.896607+0000", 2024-09-11T09:51:24.072 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:24.073 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.073 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.073 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:24.073 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.896608+0000", 2024-09-11T09:51:24.073 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.0690000000000001e-06 2024-09-11T09:51:24.073 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.073 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.073 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:24.073 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.914461+0000", 2024-09-11T09:51:24.073 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.017852903 2024-09-11T09:51:24.073 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.073 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.074 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:24.074 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.914495+0000", 2024-09-11T09:51:24.074 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.4301000000000002e-05 2024-09-11T09:51:24.074 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.074 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.074 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:24.074 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.914522+0000", 2024-09-11T09:51:24.074 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.7127e-05 2024-09-11T09:51:24.074 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.074 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.074 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:24.074 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.914583+0000", 2024-09-11T09:51:24.074 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.0730999999999998e-05 2024-09-11T09:51:24.075 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.075 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.075 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:24.075 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.914709+0000", 2024-09-11T09:51:24.075 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00012590400000000001 2024-09-11T09:51:24.075 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.075 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.075 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 1", 2024-09-11T09:51:24.075 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.914809+0000", 2024-09-11T09:51:24.075 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.9882000000000004e-05 2024-09-11T09:51:24.075 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.075 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.075 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:24.075 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.932551+0000", 2024-09-11T09:51:24.076 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.017742048999999999 2024-09-11T09:51:24.076 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.076 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.076 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:24.076 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.935144+0000", 2024-09-11T09:51:24.076 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0025927630000000001 2024-09-11T09:51:24.076 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.076 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.076 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:24.076 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.935205+0000", 2024-09-11T09:51:24.076 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.1634e-05 2024-09-11T09:51:24.076 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.076 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.077 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:24.077 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.935262+0000", 2024-09-11T09:51:24.077 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.6817000000000002e-05 2024-09-11T09:51:24.077 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:24.077 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:24.077 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:24.077 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.077 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.077 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_repop(client.4688.0:40 2.7 e53/38 2:fe41411c:::rbd_object_map.1250f5e25b80:head v 53'9, mlcod=53'7)", 2024-09-11T09:51:24.077 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:22.920457+0000", 2024-09-11T09:51:24.077 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 1.1235409629999999, 2024-09-11T09:51:24.077 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.011132817, 2024-09-11T09:51:24.077 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:24.078 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:24.078 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:24.078 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:24.078 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.078 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:24.078 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.920457+0000", 2024-09-11T09:51:24.078 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:24.078 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.078 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.078 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:24.078 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.920457+0000", 2024-09-11T09:51:24.078 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:24.078 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.079 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.079 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:24.079 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.920459+0000", 2024-09-11T09:51:24.079 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.6729999999999999e-06 2024-09-11T09:51:24.079 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.079 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.079 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:24.079 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.920469+0000", 2024-09-11T09:51:24.079 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.944e-06 2024-09-11T09:51:24.079 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.079 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.079 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:24.079 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.920495+0000", 2024-09-11T09:51:24.080 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.6401999999999999e-05 2024-09-11T09:51:24.080 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.080 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.080 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:24.080 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.920506+0000", 2024-09-11T09:51:24.080 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.133e-05 2024-09-11T09:51:24.080 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.080 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.080 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:24.080 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.920581+0000", 2024-09-11T09:51:24.080 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.4932999999999996e-05 2024-09-11T09:51:24.080 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.081 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.081 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:24.081 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.920612+0000", 2024-09-11T09:51:24.081 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.0537999999999998e-05 2024-09-11T09:51:24.081 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.081 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.081 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:24.081 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.931537+0000", 2024-09-11T09:51:24.081 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.010925239 2024-09-11T09:51:24.081 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.081 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.081 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:24.081 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.931590+0000", 2024-09-11T09:51:24.082 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.2757999999999999e-05 2024-09-11T09:51:24.082 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:24.082 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:24.082 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:24.082 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.082 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.082 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_repop(client.4688.0:41 2.7 e53/38 2:fe41411c:::rbd_object_map.1250f5e25b80:head v 53'10, mlcod=53'7)", 2024-09-11T09:51:24.082 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:22.921422+0000", 2024-09-11T09:51:24.082 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 1.1225755470000001, 2024-09-11T09:51:24.082 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.013734942, 2024-09-11T09:51:24.082 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:24.082 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:24.083 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:24.083 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:24.083 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.083 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:24.083 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921422+0000", 2024-09-11T09:51:24.083 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:24.083 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.083 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.083 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:24.083 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921422+0000", 2024-09-11T09:51:24.083 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:24.083 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.083 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.083 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:24.084 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921426+0000", 2024-09-11T09:51:24.084 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.6150000000000001e-06 2024-09-11T09:51:24.084 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.084 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.084 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:24.084 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921435+0000", 2024-09-11T09:51:24.084 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.9840000000000007e-06 2024-09-11T09:51:24.084 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.084 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.084 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:24.084 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921444+0000", 2024-09-11T09:51:24.084 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.6000000000000007e-06 2024-09-11T09:51:24.085 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.085 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.085 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:24.085 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921450+0000", 2024-09-11T09:51:24.085 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.4690000000000002e-06 2024-09-11T09:51:24.085 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.085 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.085 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:24.085 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921511+0000", 2024-09-11T09:51:24.085 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.0779999999999997e-05 2024-09-11T09:51:24.085 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.085 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.085 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:24.086 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921531+0000", 2024-09-11T09:51:24.086 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.0302999999999999e-05 2024-09-11T09:51:24.086 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.086 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.086 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:24.086 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.935115+0000", 2024-09-11T09:51:24.086 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.013583471 2024-09-11T09:51:24.086 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.086 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.086 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:24.086 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.935157+0000", 2024-09-11T09:51:24.086 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.2719999999999998e-05 2024-09-11T09:51:24.087 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:24.087 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:24.087 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:24.087 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.087 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.087 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4688.0:42 2.3 2:c54742d8:::rbd_data.1250f5e25b80.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:24.087 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:22.932560+0000", 2024-09-11T09:51:24.087 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 1.1114378039999999, 2024-09-11T09:51:24.087 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.017798198000000001, 2024-09-11T09:51:24.087 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:24.087 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:24.087 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:24.087 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:24.087 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4688", 2024-09-11T09:51:24.088 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:51:24.088 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 42 2024-09-11T09:51:24.088 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.088 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:24.088 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.088 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:24.088 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.932560+0000", 2024-09-11T09:51:24.088 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:24.088 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.088 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.088 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:24.088 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.932560+0000", 2024-09-11T09:51:24.089 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:24.089 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.089 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.089 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:24.089 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.932562+0000", 2024-09-11T09:51:24.089 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.17e-06 2024-09-11T09:51:24.089 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.089 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.089 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:24.089 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.938851+0000", 2024-09-11T09:51:24.089 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.006288971 2024-09-11T09:51:24.089 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.089 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.089 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:24.090 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.938865+0000", 2024-09-11T09:51:24.090 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.365e-05 2024-09-11T09:51:24.090 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.090 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.090 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:24.090 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.938871+0000", 2024-09-11T09:51:24.090 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.8850000000000004e-06 2024-09-11T09:51:24.090 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.090 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.090 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:24.090 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.938926+0000", 2024-09-11T09:51:24.090 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.4691999999999997e-05 2024-09-11T09:51:24.090 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.091 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.091 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:24.091 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.939059+0000", 2024-09-11T09:51:24.091 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00013310899999999999 2024-09-11T09:51:24.091 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.091 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.091 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 2", 2024-09-11T09:51:24.091 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.939171+0000", 2024-09-11T09:51:24.091 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000112718 2024-09-11T09:51:24.091 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.091 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.091 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:24.091 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.950262+0000", 2024-09-11T09:51:24.091 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.011091063 2024-09-11T09:51:24.092 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.092 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.092 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:24.092 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.950274+0000", 2024-09-11T09:51:24.092 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1963e-05 2024-09-11T09:51:24.092 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.092 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.092 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:24.092 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.950318+0000", 2024-09-11T09:51:24.092 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.3519999999999997e-05 2024-09-11T09:51:24.092 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.092 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.092 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:24.093 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.950358+0000", 2024-09-11T09:51:24.093 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.0457000000000002e-05 2024-09-11T09:51:24.093 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:24.093 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:24.093 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:24.093 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.093 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.093 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_repop(client.4694.0:34 2.7 e53/38 2:f80bfc0c:ns::rbd_data.1256a0c02879.0000000000000002:head v 53'25, mlcod=53'23)", 2024-09-11T09:51:24.093 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.094944+0000", 2024-09-11T09:51:24.093 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 0.94905441199999996, 2024-09-11T09:51:24.093 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.010896785000000001, 2024-09-11T09:51:24.093 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:24.093 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:24.094 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:24.094 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:24.094 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.094 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:24.094 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.094944+0000", 2024-09-11T09:51:24.094 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:24.094 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.094 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.094 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:24.094 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.094944+0000", 2024-09-11T09:51:24.094 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:24.094 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.094 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.095 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:24.095 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.094944+0000", 2024-09-11T09:51:24.095 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.0699999999999996e-07 2024-09-11T09:51:24.095 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.095 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.095 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:24.095 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.099060+0000", 2024-09-11T09:51:24.095 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0041155580000000001 2024-09-11T09:51:24.095 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.095 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.095 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:24.095 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.099082+0000", 2024-09-11T09:51:24.096 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.2323999999999998e-05 2024-09-11T09:51:24.096 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.096 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.096 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:24.096 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.099094+0000", 2024-09-11T09:51:24.096 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1535000000000001e-05 2024-09-11T09:51:24.096 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.096 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.096 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:24.096 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.099160+0000", 2024-09-11T09:51:24.096 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.6449e-05 2024-09-11T09:51:24.096 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.096 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.096 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:24.097 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.099183+0000", 2024-09-11T09:51:24.097 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.3108e-05 2024-09-11T09:51:24.097 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.097 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.097 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:24.097 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.105776+0000", 2024-09-11T09:51:24.097 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0065922760000000002 2024-09-11T09:51:24.097 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.097 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.097 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:24.097 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.105840+0000", 2024-09-11T09:51:24.097 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.4727999999999994e-05 2024-09-11T09:51:24.098 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:24.098 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:24.098 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:24.098 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.098 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.098 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_repop(client.4694.0:35 2.7 e53/38 2:e8766325:ns::rbd_object_map.1256a0c02879:head v 53'26, mlcod=53'23)", 2024-09-11T09:51:24.098 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.100964+0000", 2024-09-11T09:51:24.098 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 0.94303388399999999, 2024-09-11T09:51:24.098 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0049649359999999997, 2024-09-11T09:51:24.098 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:24.098 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:24.098 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:24.098 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:24.099 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.099 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:24.099 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.100964+0000", 2024-09-11T09:51:24.099 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:24.099 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.099 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.099 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:24.099 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.100964+0000", 2024-09-11T09:51:24.099 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:24.099 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.099 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.099 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:24.099 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.100966+0000", 2024-09-11T09:51:24.100 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.6330000000000001e-06 2024-09-11T09:51:24.100 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.100 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.100 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:24.100 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.100979+0000", 2024-09-11T09:51:24.100 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.2813000000000001e-05 2024-09-11T09:51:24.100 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.100 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.100 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:24.100 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.100992+0000", 2024-09-11T09:51:24.100 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.3414999999999999e-05 2024-09-11T09:51:24.100 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.100 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.101 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:24.101 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.101002+0000", 2024-09-11T09:51:24.101 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.0159e-05 2024-09-11T09:51:24.101 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.101 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.101 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:24.101 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.102365+0000", 2024-09-11T09:51:24.101 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001362912 2024-09-11T09:51:24.101 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.101 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.101 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:24.101 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.102382+0000", 2024-09-11T09:51:24.101 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.7433000000000001e-05 2024-09-11T09:51:24.101 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.102 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.102 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:24.102 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.105901+0000", 2024-09-11T09:51:24.102 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.003518476 2024-09-11T09:51:24.102 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.102 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.102 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:24.102 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.105929+0000", 2024-09-11T09:51:24.102 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.8095000000000001e-05 2024-09-11T09:51:24.102 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:24.102 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:24.102 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:24.102 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.103 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.103 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_repop(client.4694.0:36 2.7 e53/38 2:e8766325:ns::rbd_object_map.1256a0c02879:head v 53'27, mlcod=53'23)", 2024-09-11T09:51:24.103 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.101688+0000", 2024-09-11T09:51:24.103 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 0.94231020799999998, 2024-09-11T09:51:24.103 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0042633289999999997, 2024-09-11T09:51:24.103 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:24.103 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:24.103 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:24.103 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:24.103 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.103 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:24.103 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.101688+0000", 2024-09-11T09:51:24.103 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:24.103 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.104 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.104 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:24.104 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.101688+0000", 2024-09-11T09:51:24.104 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:24.104 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.104 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.104 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:24.104 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.101689+0000", 2024-09-11T09:51:24.104 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.9299999999999997e-07 2024-09-11T09:51:24.104 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.104 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.104 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:24.104 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.101694+0000", 2024-09-11T09:51:24.105 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.6779999999999997e-06 2024-09-11T09:51:24.105 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.105 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.105 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:24.105 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.101707+0000", 2024-09-11T09:51:24.105 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.2549e-05 2024-09-11T09:51:24.105 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.105 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.105 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:24.105 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.101714+0000", 2024-09-11T09:51:24.105 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.9210000000000003e-06 2024-09-11T09:51:24.105 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.105 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.105 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:24.106 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.102635+0000", 2024-09-11T09:51:24.106 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00092109400000000001 2024-09-11T09:51:24.106 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.106 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.106 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:24.106 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.102649+0000", 2024-09-11T09:51:24.106 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.3913e-05 2024-09-11T09:51:24.106 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.106 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.106 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:24.106 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.105931+0000", 2024-09-11T09:51:24.106 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.003281876 2024-09-11T09:51:24.106 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.107 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.107 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:24.107 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.105951+0000", 2024-09-11T09:51:24.107 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.0505000000000001e-05 2024-09-11T09:51:24.107 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:24.107 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:24.107 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:24.107 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.107 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.107 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_repop(client.4694.0:37 2.7 e53/38 2:e8766325:ns::rbd_object_map.1256a0c02879:head v 53'28, mlcod=53'23)", 2024-09-11T09:51:24.107 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.102274+0000", 2024-09-11T09:51:24.107 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 0.94172415799999998, 2024-09-11T09:51:24.107 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0036897850000000001, 2024-09-11T09:51:24.107 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:24.108 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:24.108 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:24.108 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:24.108 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.108 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:24.108 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.102274+0000", 2024-09-11T09:51:24.108 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:24.108 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.108 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.108 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:24.108 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.102274+0000", 2024-09-11T09:51:24.108 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:24.108 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.108 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.109 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:24.109 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.102274+0000", 2024-09-11T09:51:24.109 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.1699999999999998e-07 2024-09-11T09:51:24.109 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.109 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.109 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:24.109 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.102280+0000", 2024-09-11T09:51:24.109 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.1200000000000001e-06 2024-09-11T09:51:24.109 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.109 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.109 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:24.109 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.102291+0000", 2024-09-11T09:51:24.109 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1032e-05 2024-09-11T09:51:24.110 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.110 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.110 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:24.110 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.102297+0000", 2024-09-11T09:51:24.110 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.4919999999999998e-06 2024-09-11T09:51:24.110 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.110 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.110 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:24.110 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.102902+0000", 2024-09-11T09:51:24.110 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00060473799999999996 2024-09-11T09:51:24.110 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.110 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.111 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:24.111 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.102913+0000", 2024-09-11T09:51:24.111 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.131e-05 2024-09-11T09:51:24.111 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.111 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.111 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:24.111 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.105952+0000", 2024-09-11T09:51:24.111 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0030386200000000001 2024-09-11T09:51:24.111 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.111 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.111 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:24.111 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.105964+0000", 2024-09-11T09:51:24.111 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1856e-05 2024-09-11T09:51:24.111 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:24.112 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:24.112 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:24.112 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.112 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.112 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4694.0:41 2.2 2:5e2505c4:ns::rbd_data.1256a0c02879.0000000000000003:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:24.112 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.107511+0000", 2024-09-11T09:51:24.112 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 0.93648673100000002, 2024-09-11T09:51:24.112 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.018571066000000001, 2024-09-11T09:51:24.112 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:24.113 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:24.113 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:24.113 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:24.113 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4694", 2024-09-11T09:51:24.113 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:51:24.114 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 41 2024-09-11T09:51:24.114 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.114 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:24.114 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.114 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:24.114 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.107511+0000", 2024-09-11T09:51:24.114 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:24.114 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.114 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.114 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:24.114 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.107511+0000", 2024-09-11T09:51:24.114 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:24.114 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.115 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.115 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:24.115 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.107513+0000", 2024-09-11T09:51:24.115 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.3489999999999999e-06 2024-09-11T09:51:24.115 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.115 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.115 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:24.115 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.112704+0000", 2024-09-11T09:51:24.115 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0051918509999999999 2024-09-11T09:51:24.115 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.115 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.115 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:24.115 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.112718+0000", 2024-09-11T09:51:24.115 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.359e-05 2024-09-11T09:51:24.116 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.116 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.116 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:24.116 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.112725+0000", 2024-09-11T09:51:24.116 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.9940000000000003e-06 2024-09-11T09:51:24.116 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.116 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.116 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:24.116 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.112788+0000", 2024-09-11T09:51:24.116 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.2447999999999996e-05 2024-09-11T09:51:24.116 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.116 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.116 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:24.117 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.112937+0000", 2024-09-11T09:51:24.117 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00014921499999999999 2024-09-11T09:51:24.117 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.117 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.117 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 1", 2024-09-11T09:51:24.117 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.113085+0000", 2024-09-11T09:51:24.117 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000148085 2024-09-11T09:51:24.117 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.117 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.117 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:24.117 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.124318+0000", 2024-09-11T09:51:24.117 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.011233158 2024-09-11T09:51:24.117 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.117 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.118 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:24.118 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.125957+0000", 2024-09-11T09:51:24.118 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001638701 2024-09-11T09:51:24.118 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.118 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.118 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:24.118 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.126029+0000", 2024-09-11T09:51:24.118 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.2125000000000005e-05 2024-09-11T09:51:24.118 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.118 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.118 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:24.118 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.126082+0000", 2024-09-11T09:51:24.118 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.3550000000000001e-05 2024-09-11T09:51:24.119 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:24.119 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:24.119 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:24.119 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.119 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.119 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4694.0:44 2.3 2:cd1a7d1c:ns::rbd_data.1256a0c02879.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:24.119 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.112744+0000", 2024-09-11T09:51:24.119 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 0.93125416400000005, 2024-09-11T09:51:24.119 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.021672212999999999, 2024-09-11T09:51:24.119 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:24.119 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:24.119 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:24.119 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:24.119 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4694", 2024-09-11T09:51:24.120 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:51:24.120 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 44 2024-09-11T09:51:24.120 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.120 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:24.120 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.120 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:24.120 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.112744+0000", 2024-09-11T09:51:24.120 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:24.120 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.120 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.120 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:24.120 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.112744+0000", 2024-09-11T09:51:24.120 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:24.120 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.121 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.121 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:24.121 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.112745+0000", 2024-09-11T09:51:24.121 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.2900000000000002e-07 2024-09-11T09:51:24.121 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.121 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.121 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:24.121 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.119929+0000", 2024-09-11T09:51:24.121 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0071846230000000002 2024-09-11T09:51:24.121 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.121 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.121 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:24.121 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.119949+0000", 2024-09-11T09:51:24.121 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.9264000000000001e-05 2024-09-11T09:51:24.122 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.122 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.122 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:24.122 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.119959+0000", 2024-09-11T09:51:24.122 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.0465999999999999e-05 2024-09-11T09:51:24.122 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.122 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.122 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:24.122 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.120028+0000", 2024-09-11T09:51:24.122 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.9318000000000003e-05 2024-09-11T09:51:24.122 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.122 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.122 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:24.123 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.120160+0000", 2024-09-11T09:51:24.123 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000132015 2024-09-11T09:51:24.123 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.123 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.123 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 2", 2024-09-11T09:51:24.123 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.120260+0000", 2024-09-11T09:51:24.123 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.9397999999999999e-05 2024-09-11T09:51:24.123 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.123 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.123 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:24.123 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.134246+0000", 2024-09-11T09:51:24.123 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.013986721000000001 2024-09-11T09:51:24.123 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.123 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.124 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:24.124 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.134295+0000", 2024-09-11T09:51:24.124 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.8649000000000001e-05 2024-09-11T09:51:24.124 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.124 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.124 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:24.124 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.134361+0000", 2024-09-11T09:51:24.124 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.6081999999999995e-05 2024-09-11T09:51:24.124 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.124 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.124 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:24.124 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.134416+0000", 2024-09-11T09:51:24.124 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.4848000000000001e-05 2024-09-11T09:51:24.125 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:24.125 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:24.125 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:24.125 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.125 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.125 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4694.0:46 2.3 2:ce0e1469:ns::rbd_data.1256a0c02879.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:24.125 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.119993+0000", 2024-09-11T09:51:24.125 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 0.92400468800000002, 2024-09-11T09:51:24.125 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.025840610999999999, 2024-09-11T09:51:24.125 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:24.125 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:24.125 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:24.125 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:24.125 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4694", 2024-09-11T09:51:24.126 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:51:24.126 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 46 2024-09-11T09:51:24.126 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.126 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:24.126 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.126 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:24.126 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.119993+0000", 2024-09-11T09:51:24.126 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:24.126 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.126 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.126 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:24.126 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.119993+0000", 2024-09-11T09:51:24.126 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:24.127 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.127 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.127 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:24.127 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.119995+0000", 2024-09-11T09:51:24.127 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1930000000000001e-06 2024-09-11T09:51:24.127 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.127 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.127 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:24.127 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.128458+0000", 2024-09-11T09:51:24.127 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0084632630000000004 2024-09-11T09:51:24.127 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.127 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.127 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:24.128 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.128478+0000", 2024-09-11T09:51:24.128 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.0353e-05 2024-09-11T09:51:24.128 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.128 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.128 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:24.128 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.128491+0000", 2024-09-11T09:51:24.128 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.2954e-05 2024-09-11T09:51:24.128 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.128 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.128 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:24.128 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.128558+0000", 2024-09-11T09:51:24.128 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.7076999999999995e-05 2024-09-11T09:51:24.128 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.128 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.129 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:24.129 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.128708+0000", 2024-09-11T09:51:24.129 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00014939599999999999 2024-09-11T09:51:24.129 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.129 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.129 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 2", 2024-09-11T09:51:24.129 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.128866+0000", 2024-09-11T09:51:24.129 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000158145 2024-09-11T09:51:24.129 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.129 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.129 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:24.129 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.141045+0000", 2024-09-11T09:51:24.129 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.012179331999999999 2024-09-11T09:51:24.130 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.130 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.130 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:24.130 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.145742+0000", 2024-09-11T09:51:24.130 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0046972050000000003 2024-09-11T09:51:24.130 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.130 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.130 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:24.130 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.145779+0000", 2024-09-11T09:51:24.130 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.6544000000000001e-05 2024-09-11T09:51:24.130 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.130 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.130 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:24.130 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.145834+0000", 2024-09-11T09:51:24.131 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.5148999999999997e-05 2024-09-11T09:51:24.131 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:24.131 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:24.131 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:24.131 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.131 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.131 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4694.0:48 2.3 2:d2090b70:ns::rbd_data.1256a0c02879.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:24.131 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.130540+0000", 2024-09-11T09:51:24.131 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 0.91345842200000005, 2024-09-11T09:51:24.131 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.017167789999999999, 2024-09-11T09:51:24.131 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:24.131 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:24.131 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:24.132 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:24.132 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4694", 2024-09-11T09:51:24.132 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:51:24.132 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 48 2024-09-11T09:51:24.132 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.132 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:24.132 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.132 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:24.132 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.130540+0000", 2024-09-11T09:51:24.132 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:24.132 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.132 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.133 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:24.133 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.130540+0000", 2024-09-11T09:51:24.133 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:24.133 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.133 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.133 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:24.133 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.130542+0000", 2024-09-11T09:51:24.133 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.9209999999999999e-06 2024-09-11T09:51:24.133 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.133 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.133 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:24.133 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.134691+0000", 2024-09-11T09:51:24.133 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0041498739999999996 2024-09-11T09:51:24.134 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.134 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.134 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:24.134 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.134705+0000", 2024-09-11T09:51:24.134 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.3669000000000001e-05 2024-09-11T09:51:24.134 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.134 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.134 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:24.134 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.134712+0000", 2024-09-11T09:51:24.134 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.8410000000000002e-06 2024-09-11T09:51:24.134 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.134 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.134 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:24.134 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.134754+0000", 2024-09-11T09:51:24.135 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.2524999999999997e-05 2024-09-11T09:51:24.135 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.135 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.135 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:24.135 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.134875+0000", 2024-09-11T09:51:24.135 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000120332 2024-09-11T09:51:24.135 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.135 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.135 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 2", 2024-09-11T09:51:24.135 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.135005+0000", 2024-09-11T09:51:24.135 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00013024900000000001 2024-09-11T09:51:24.135 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.136 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.136 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:24.136 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.145971+0000", 2024-09-11T09:51:24.136 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.010966203000000001 2024-09-11T09:51:24.136 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.136 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.136 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:24.136 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.147600+0000", 2024-09-11T09:51:24.136 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001628382 2024-09-11T09:51:24.136 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.136 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.136 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:24.136 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.147635+0000", 2024-09-11T09:51:24.137 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.5420000000000003e-05 2024-09-11T09:51:24.137 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.137 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.137 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:24.137 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.147707+0000", 2024-09-11T09:51:24.137 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.2373999999999995e-05 2024-09-11T09:51:24.137 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:24.137 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:24.137 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:24.137 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.137 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.137 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_repop(client.4694.0:49 2.7 e53/38 2:e8766325:ns::rbd_object_map.1256a0c02879:head v 53'33, mlcod=53'31)", 2024-09-11T09:51:24.137 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.135391+0000", 2024-09-11T09:51:24.138 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 0.90860685699999999, 2024-09-11T09:51:24.138 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0051189859999999998, 2024-09-11T09:51:24.138 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:24.138 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:24.138 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:24.138 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:24.138 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.138 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:24.138 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.135391+0000", 2024-09-11T09:51:24.138 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:24.138 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.138 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.138 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:24.139 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.135391+0000", 2024-09-11T09:51:24.139 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:24.139 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.139 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.139 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:24.139 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.135393+0000", 2024-09-11T09:51:24.139 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.4020000000000001e-06 2024-09-11T09:51:24.139 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.139 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.139 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:24.139 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.135400+0000", 2024-09-11T09:51:24.139 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.5129999999999999e-06 2024-09-11T09:51:24.139 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.140 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.140 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:24.140 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.135411+0000", 2024-09-11T09:51:24.140 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1038e-05 2024-09-11T09:51:24.140 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.140 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.140 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:24.140 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.135418+0000", 2024-09-11T09:51:24.140 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.1489999999999997e-06 2024-09-11T09:51:24.140 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.140 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.140 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:24.141 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.135470+0000", 2024-09-11T09:51:24.141 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.2021999999999998e-05 2024-09-11T09:51:24.141 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.141 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.141 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:24.141 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.135501+0000", 2024-09-11T09:51:24.141 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.0349999999999999e-05 2024-09-11T09:51:24.141 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.141 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.141 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:24.141 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.140453+0000", 2024-09-11T09:51:24.141 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0049522409999999996 2024-09-11T09:51:24.141 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.142 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.142 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:24.142 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.140510+0000", 2024-09-11T09:51:24.142 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.7271000000000001e-05 2024-09-11T09:51:24.142 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:24.142 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:24.142 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:24.142 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.142 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.142 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4694.0:54 2.3 2:c44f0253:ns::rbd_data.1256a0c02879.000000000000000b:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:24.142 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.150151+0000", 2024-09-11T09:51:24.142 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 0.893847006, 2024-09-11T09:51:24.142 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.015103484, 2024-09-11T09:51:24.143 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:24.143 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:24.143 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:24.143 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:24.143 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4694", 2024-09-11T09:51:24.143 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:51:24.143 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 54 2024-09-11T09:51:24.143 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.143 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:24.143 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.143 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:24.143 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.150151+0000", 2024-09-11T09:51:24.143 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:24.144 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.144 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.144 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:24.144 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.150151+0000", 2024-09-11T09:51:24.144 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:24.144 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.144 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.144 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:24.144 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.150152+0000", 2024-09-11T09:51:24.144 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.2550000000000001e-06 2024-09-11T09:51:24.144 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.144 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.144 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:24.145 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.154528+0000", 2024-09-11T09:51:24.145 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0043758219999999997 2024-09-11T09:51:24.145 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.145 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.145 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:24.145 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.154541+0000", 2024-09-11T09:51:24.145 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.3414999999999999e-05 2024-09-11T09:51:24.145 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.145 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.145 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:24.145 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.154551+0000", 2024-09-11T09:51:24.145 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.4350000000000003e-06 2024-09-11T09:51:24.145 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.146 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.146 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:24.146 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.154615+0000", 2024-09-11T09:51:24.146 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.3627000000000006e-05 2024-09-11T09:51:24.146 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.146 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.146 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:24.146 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.154717+0000", 2024-09-11T09:51:24.146 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000102729 2024-09-11T09:51:24.146 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.146 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.146 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 2", 2024-09-11T09:51:24.146 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.154826+0000", 2024-09-11T09:51:24.147 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000108409 2024-09-11T09:51:24.147 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.147 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.147 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:24.147 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.165094+0000", 2024-09-11T09:51:24.147 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.010268210999999999 2024-09-11T09:51:24.147 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.147 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.147 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:24.147 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.165138+0000", 2024-09-11T09:51:24.147 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.4078e-05 2024-09-11T09:51:24.147 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.147 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.148 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:24.148 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.165204+0000", 2024-09-11T09:51:24.148 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.5716000000000006e-05 2024-09-11T09:51:24.148 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.148 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.148 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:24.148 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.165254+0000", 2024-09-11T09:51:24.148 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.0787e-05 2024-09-11T09:51:24.148 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:24.148 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:24.148 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:24.148 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.148 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.148 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_repop(client.4694.0:55 2.7 e53/38 2:e8766325:ns::rbd_object_map.1256a0c02879:head v 53'37, mlcod=53'34)", 2024-09-11T09:51:24.149 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.150221+0000", 2024-09-11T09:51:24.149 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 0.89377714100000005, 2024-09-11T09:51:24.149 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00530015, 2024-09-11T09:51:24.149 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:24.149 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:24.149 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:24.149 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:24.149 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.149 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:24.149 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.150221+0000", 2024-09-11T09:51:24.149 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:24.149 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.149 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.150 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:24.150 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.150221+0000", 2024-09-11T09:51:24.150 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:24.150 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.150 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.150 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:24.150 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.150221+0000", 2024-09-11T09:51:24.150 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.9100000000000004e-07 2024-09-11T09:51:24.150 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.150 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.150 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:24.150 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.150226+0000", 2024-09-11T09:51:24.150 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.2250000000000002e-06 2024-09-11T09:51:24.151 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.151 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.151 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:24.151 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.150251+0000", 2024-09-11T09:51:24.151 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.4987e-05 2024-09-11T09:51:24.151 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.151 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.151 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:24.151 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.150258+0000", 2024-09-11T09:51:24.151 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.0790000000000003e-06 2024-09-11T09:51:24.151 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.151 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.151 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:24.152 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.150305+0000", 2024-09-11T09:51:24.152 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.6879000000000003e-05 2024-09-11T09:51:24.152 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.152 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.152 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:24.152 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.150320+0000", 2024-09-11T09:51:24.152 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.5268999999999999e-05 2024-09-11T09:51:24.152 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.152 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.152 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:24.152 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.155493+0000", 2024-09-11T09:51:24.152 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0051731390000000002 2024-09-11T09:51:24.152 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.152 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.153 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:24.153 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.155521+0000", 2024-09-11T09:51:24.153 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.8081000000000001e-05 2024-09-11T09:51:24.153 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:24.153 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:24.153 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:24.153 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.153 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.153 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4694.0:56 2.3 2:c462f029:ns::rbd_data.1256a0c02879.000000000000000c:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:24.153 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.154576+0000", 2024-09-11T09:51:24.153 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 0.88942216500000004, 2024-09-11T09:51:24.153 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.020781621, 2024-09-11T09:51:24.154 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:24.154 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:24.154 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:24.154 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:24.154 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4694", 2024-09-11T09:51:24.154 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:51:24.154 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 56 2024-09-11T09:51:24.154 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.154 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:24.154 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.154 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:24.154 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.154576+0000", 2024-09-11T09:51:24.155 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:24.155 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.155 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.155 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:24.155 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.154576+0000", 2024-09-11T09:51:24.155 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:24.155 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.155 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.155 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:24.155 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.154577+0000", 2024-09-11T09:51:24.155 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.2249999999999999e-06 2024-09-11T09:51:24.155 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.155 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.156 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:24.156 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.158783+0000", 2024-09-11T09:51:24.156 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0042063889999999996 2024-09-11T09:51:24.156 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.156 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.156 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:24.156 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.158803+0000", 2024-09-11T09:51:24.156 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.9576999999999999e-05 2024-09-11T09:51:24.156 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.156 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.156 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:24.156 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.158815+0000", 2024-09-11T09:51:24.156 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.2221999999999999e-05 2024-09-11T09:51:24.157 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.157 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.157 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:24.157 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.158902+0000", 2024-09-11T09:51:24.157 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.6390000000000005e-05 2024-09-11T09:51:24.157 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.157 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.157 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:24.157 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.159141+0000", 2024-09-11T09:51:24.157 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000238886 2024-09-11T09:51:24.157 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.157 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.157 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 2", 2024-09-11T09:51:24.158 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.159279+0000", 2024-09-11T09:51:24.158 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000138073 2024-09-11T09:51:24.158 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.158 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.158 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:24.158 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.173297+0000", 2024-09-11T09:51:24.158 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.014018113 2024-09-11T09:51:24.158 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.158 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.158 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:24.158 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.175278+0000", 2024-09-11T09:51:24.158 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0019817390000000002 2024-09-11T09:51:24.158 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.158 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.159 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:24.159 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.175316+0000", 2024-09-11T09:51:24.159 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.7829e-05 2024-09-11T09:51:24.159 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.159 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.159 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:24.159 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.175357+0000", 2024-09-11T09:51:24.159 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.1177999999999998e-05 2024-09-11T09:51:24.159 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:24.159 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:24.159 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:24.159 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.159 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.160 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4694.0:59 2.2 2:41adefc9:ns::rbd_data.1256a0c02879.000000000000000e:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:24.160 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.158843+0000", 2024-09-11T09:51:24.160 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 0.88515459900000004, 2024-09-11T09:51:24.160 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.017874207, 2024-09-11T09:51:24.160 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:24.160 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:24.160 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:24.160 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:24.160 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4694", 2024-09-11T09:51:24.160 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:51:24.160 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 59 2024-09-11T09:51:24.160 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.160 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:24.161 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.161 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:24.161 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.158843+0000", 2024-09-11T09:51:24.161 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:24.161 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.161 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.161 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:24.161 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.158843+0000", 2024-09-11T09:51:24.161 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:24.161 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.161 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.161 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:24.161 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.158845+0000", 2024-09-11T09:51:24.161 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.7439999999999999e-06 2024-09-11T09:51:24.162 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.162 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.162 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:24.162 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.162713+0000", 2024-09-11T09:51:24.162 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0038678829999999999 2024-09-11T09:51:24.162 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.162 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.162 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:24.162 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.162728+0000", 2024-09-11T09:51:24.162 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.5114e-05 2024-09-11T09:51:24.162 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.162 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.162 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:24.162 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.162737+0000", 2024-09-11T09:51:24.163 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.1099999999999992e-06 2024-09-11T09:51:24.163 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.163 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.163 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:24.163 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.162799+0000", 2024-09-11T09:51:24.163 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.1934999999999996e-05 2024-09-11T09:51:24.163 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.163 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.163 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:24.163 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.162911+0000", 2024-09-11T09:51:24.163 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000111867 2024-09-11T09:51:24.163 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.163 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.164 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 1", 2024-09-11T09:51:24.164 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.163030+0000", 2024-09-11T09:51:24.164 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000119108 2024-09-11T09:51:24.164 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.164 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.164 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:24.164 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.174086+0000", 2024-09-11T09:51:24.164 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.01105543 2024-09-11T09:51:24.164 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.164 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.164 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:24.164 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.176614+0000", 2024-09-11T09:51:24.164 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0025284940000000001 2024-09-11T09:51:24.164 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.165 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.165 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:24.165 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.176672+0000", 2024-09-11T09:51:24.165 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.7892000000000001e-05 2024-09-11T09:51:24.165 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:24.165 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:24.165 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:24.165 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.176718+0000", 2024-09-11T09:51:24.165 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.5630000000000002e-05 2024-09-11T09:51:24.165 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:24.165 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:24.165 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:24.165 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:24.166 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:24.166 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-11T09:51:24.167 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-09-11T09:51:24.167+0000 7f7b6c7d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31904) UID: 0 2024-09-11T09:51:24.167 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-11T09:51:24.268 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:51:24.269+0000 7f3c96ec9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31909) UID: 0 2024-09-11T09:51:24.276 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:51:24.276 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [], 2024-09-11T09:51:24.276 INFO:teuthology.orchestra.run.smithi017.stdout: "num_ops": 0 2024-09-11T09:51:24.276 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:51:24.285 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-11T09:51:24.369 INFO:tasks.ceph.osd.2.smithi017.stderr:2024-09-11T09:51:24.370+0000 7fd20b170640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31672) UID: 0 2024-09-11T09:51:24.392 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:51:24.392 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [], 2024-09-11T09:51:24.392 INFO:teuthology.orchestra.run.smithi017.stdout: "complaint_time": 30, 2024-09-11T09:51:24.392 INFO:teuthology.orchestra.run.smithi017.stdout: "num_blocked_ops": 0 2024-09-11T09:51:24.392 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:51:24.401 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-11T09:51:24.485 INFO:tasks.ceph.osd.2.smithi017.stderr:2024-09-11T09:51:24.486+0000 7fd20b170640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31672) UID: 0 2024-09-11T09:51:24.490 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 5, 0] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5] 2024-09-11T09:51:24.490 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-11T09:51:24.491 INFO:tasks.thrashosds.thrasher:Killing osd 4, live_osds are [0, 1, 2, 3, 4, 5] 2024-09-11T09:51:24.491 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-09-11T09:51:24.492 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T09:51:24.517 INFO:tasks.ceph.osd.4:Stopped 2024-09-11T09:51:24.517 INFO:tasks.thrashosds.thrasher:Testing ceph-objectstore-tool on down osd.4 2024-09-11T09:51:24.517 INFO:tasks.thrashosds.thrasher:remote for osd 4 is ubuntu@smithi096.front.sepia.ceph.com 2024-09-11T09:51:24.517 DEBUG:teuthology.orchestra.run.smithi096:> type ceph-objectstore-tool 2024-09-11T09:51:24.551 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:51:24.551 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 20, 2024-09-11T09:51:24.552 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 600, 2024-09-11T09:51:24.552 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [ 2024-09-11T09:51:24.552 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.552 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4688.0:34 2.4 e53/32 2:2367385a:::rbd_data.1250f5e25b80.0000000000000002:head v 53'6, mlcod=53'4)", 2024-09-11T09:51:24.552 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.907415+0000", 2024-09-11T09:51:24.552 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 1.6436917439999998, 2024-09-11T09:51:24.552 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.011285778999999999, 2024-09-11T09:51:24.552 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:24.552 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:24.552 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:24.553 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:24.553 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.553 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:24.553 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.907415+0000", 2024-09-11T09:51:24.553 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:24.553 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.553 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.553 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:24.553 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.907415+0000", 2024-09-11T09:51:24.553 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:24.553 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.554 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.554 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:24.554 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.907417+0000", 2024-09-11T09:51:24.554 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.2910000000000002e-06 2024-09-11T09:51:24.554 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.554 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.554 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:24.554 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.910477+0000", 2024-09-11T09:51:24.554 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0030597829999999999 2024-09-11T09:51:24.554 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.555 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.555 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:24.555 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.910501+0000", 2024-09-11T09:51:24.555 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.3332e-05 2024-09-11T09:51:24.555 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.555 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.555 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:24.555 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.910514+0000", 2024-09-11T09:51:24.555 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3957e-05 2024-09-11T09:51:24.555 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.555 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.556 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:24.556 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.910612+0000", 2024-09-11T09:51:24.556 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.7075000000000001e-05 2024-09-11T09:51:24.556 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.556 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.556 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:24.556 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.910640+0000", 2024-09-11T09:51:24.556 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.8768999999999998e-05 2024-09-11T09:51:24.556 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.556 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.556 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:24.557 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.918640+0000", 2024-09-11T09:51:24.557 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0079991899999999998 2024-09-11T09:51:24.557 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.557 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.557 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:24.557 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.918701+0000", 2024-09-11T09:51:24.557 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.1382000000000003e-05 2024-09-11T09:51:24.557 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:24.557 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:24.557 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:24.557 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.558 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.558 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4688.0:38 2.5 e53/32 2:abde2694:::rbd_data.1250f5e25b80.0000000000000004:head v 53'1, mlcod=0'0)", 2024-09-11T09:51:24.558 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.915432+0000", 2024-09-11T09:51:24.558 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 1.6356753639999999, 2024-09-11T09:51:24.558 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.015849652999999998, 2024-09-11T09:51:24.558 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:24.558 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:24.558 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:24.558 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:24.558 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.558 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:24.559 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.915432+0000", 2024-09-11T09:51:24.559 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:24.559 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.559 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.559 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:24.559 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.915432+0000", 2024-09-11T09:51:24.559 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:24.559 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.559 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.559 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:24.560 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.915432+0000", 2024-09-11T09:51:24.560 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.91e-07 2024-09-11T09:51:24.560 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.560 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.560 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:24.560 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.924943+0000", 2024-09-11T09:51:24.560 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0095110630000000002 2024-09-11T09:51:24.560 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.560 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.560 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:24.560 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.924960+0000", 2024-09-11T09:51:24.560 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6915999999999999e-05 2024-09-11T09:51:24.561 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.561 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.561 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:24.561 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.924968+0000", 2024-09-11T09:51:24.561 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.6999999999999997e-06 2024-09-11T09:51:24.561 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.561 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.561 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:24.561 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.925025+0000", 2024-09-11T09:51:24.561 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.6949e-05 2024-09-11T09:51:24.561 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.561 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.561 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:24.562 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.925043+0000", 2024-09-11T09:51:24.562 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7784000000000001e-05 2024-09-11T09:51:24.562 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.562 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.562 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:24.562 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.931237+0000", 2024-09-11T09:51:24.562 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0061935389999999996 2024-09-11T09:51:24.562 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.562 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.562 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:24.562 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.931281+0000", 2024-09-11T09:51:24.562 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.4511000000000001e-05 2024-09-11T09:51:24.562 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:24.562 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:24.563 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:24.563 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.563 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.563 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:40 2.7 2:fe41411c:::rbd_object_map.1250f5e25b80:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:24.563 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.919652+0000", 2024-09-11T09:51:24.563 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 1.6314546970000001, 2024-09-11T09:51:24.563 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.012187073, 2024-09-11T09:51:24.563 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:24.563 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:24.563 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:24.563 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:24.563 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:51:24.563 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:51:24.563 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 40 2024-09-11T09:51:24.564 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.564 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:24.564 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.564 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:24.564 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919652+0000", 2024-09-11T09:51:24.564 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:24.564 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.564 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.564 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:24.564 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919652+0000", 2024-09-11T09:51:24.564 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:24.564 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.564 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.565 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:24.565 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919654+0000", 2024-09-11T09:51:24.565 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.88e-06 2024-09-11T09:51:24.565 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.565 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.565 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:24.565 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919669+0000", 2024-09-11T09:51:24.565 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.5271e-05 2024-09-11T09:51:24.565 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.565 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.565 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:24.565 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919687+0000", 2024-09-11T09:51:24.565 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7448999999999998e-05 2024-09-11T09:51:24.565 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.566 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.566 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:24.566 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919695+0000", 2024-09-11T09:51:24.566 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.1149999999999994e-06 2024-09-11T09:51:24.566 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.566 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.566 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:24.566 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919757+0000", 2024-09-11T09:51:24.566 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.2348999999999995e-05 2024-09-11T09:51:24.566 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.566 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.566 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:24.566 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919875+0000", 2024-09-11T09:51:24.566 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00011750499999999999 2024-09-11T09:51:24.567 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.567 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.567 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 5", 2024-09-11T09:51:24.567 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.920201+0000", 2024-09-11T09:51:24.567 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00032628099999999999 2024-09-11T09:51:24.567 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.567 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.567 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:24.567 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.922089+0000", 2024-09-11T09:51:24.567 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.001887978 2024-09-11T09:51:24.567 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.567 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.567 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:24.568 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.931752+0000", 2024-09-11T09:51:24.568 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0096626309999999997 2024-09-11T09:51:24.568 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.568 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.568 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:24.568 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.931794+0000", 2024-09-11T09:51:24.568 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.2849999999999998e-05 2024-09-11T09:51:24.568 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.568 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.568 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:24.568 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.931839+0000", 2024-09-11T09:51:24.568 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.4764e-05 2024-09-11T09:51:24.568 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:24.568 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:24.569 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:24.569 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.569 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.569 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:41 2.7 2:fe41411c:::rbd_object_map.1250f5e25b80:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:24.569 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.919716+0000", 2024-09-11T09:51:24.569 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 1.6313908530000001, 2024-09-11T09:51:24.569 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.015641605999999999, 2024-09-11T09:51:24.569 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:24.569 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:24.569 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:24.569 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:24.569 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:51:24.569 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:51:24.570 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 41 2024-09-11T09:51:24.570 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.570 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:24.570 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.570 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:24.570 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919716+0000", 2024-09-11T09:51:24.570 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:24.570 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.570 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.570 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:24.570 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919716+0000", 2024-09-11T09:51:24.570 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:24.570 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.570 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.571 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:24.571 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919716+0000", 2024-09-11T09:51:24.571 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.9999999999999998e-07 2024-09-11T09:51:24.571 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.571 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.571 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:24.571 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919720+0000", 2024-09-11T09:51:24.571 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.6440000000000002e-06 2024-09-11T09:51:24.571 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.571 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.571 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:24.571 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919729+0000", 2024-09-11T09:51:24.571 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.1759999999999999e-06 2024-09-11T09:51:24.572 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.572 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.572 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:24.572 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919736+0000", 2024-09-11T09:51:24.572 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.5390000000000004e-06 2024-09-11T09:51:24.572 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.572 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.572 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:24.572 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.920669+0000", 2024-09-11T09:51:24.572 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00093275599999999997 2024-09-11T09:51:24.572 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.572 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.572 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:24.573 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.920797+0000", 2024-09-11T09:51:24.573 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00012893400000000001 2024-09-11T09:51:24.573 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.573 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.573 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 5", 2024-09-11T09:51:24.573 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.921269+0000", 2024-09-11T09:51:24.573 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00047128600000000001 2024-09-11T09:51:24.573 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.573 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.573 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:24.573 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.923198+0000", 2024-09-11T09:51:24.573 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0019289369999999999 2024-09-11T09:51:24.573 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.573 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.574 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:24.574 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.935300+0000", 2024-09-11T09:51:24.574 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.012102392 2024-09-11T09:51:24.574 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.574 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.574 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:24.574 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.935332+0000", 2024-09-11T09:51:24.574 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.2292000000000002e-05 2024-09-11T09:51:24.574 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.574 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.574 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:24.574 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.935358+0000", 2024-09-11T09:51:24.574 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.525e-05 2024-09-11T09:51:24.575 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:24.575 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:24.575 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:24.575 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.575 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.575 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:48 2.7 2:fe41411c:::rbd_object_map.1250f5e25b80:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:24.575 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.938799+0000", 2024-09-11T09:51:24.575 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 1.612307744, 2024-09-11T09:51:24.575 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0041642340000000002, 2024-09-11T09:51:24.575 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:24.575 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:24.575 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:24.575 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:24.575 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:51:24.576 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:51:24.576 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 48 2024-09-11T09:51:24.576 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.576 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:24.576 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.576 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:24.576 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.938799+0000", 2024-09-11T09:51:24.576 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:24.576 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.576 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.576 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:24.576 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.938799+0000", 2024-09-11T09:51:24.576 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:24.577 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.577 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.577 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:24.577 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.938800+0000", 2024-09-11T09:51:24.577 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.71e-07 2024-09-11T09:51:24.577 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.577 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.577 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:24.577 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.938805+0000", 2024-09-11T09:51:24.577 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.3140000000000003e-06 2024-09-11T09:51:24.577 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.577 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.577 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:24.577 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.938816+0000", 2024-09-11T09:51:24.578 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0231000000000001e-05 2024-09-11T09:51:24.578 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.578 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.578 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:24.578 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.938822+0000", 2024-09-11T09:51:24.578 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.9550000000000003e-06 2024-09-11T09:51:24.578 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.578 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.578 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:24.578 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.938868+0000", 2024-09-11T09:51:24.578 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.5562999999999997e-05 2024-09-11T09:51:24.578 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.578 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.579 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:24.579 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.938918+0000", 2024-09-11T09:51:24.579 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.0317e-05 2024-09-11T09:51:24.579 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.579 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.579 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 5", 2024-09-11T09:51:24.579 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.939114+0000", 2024-09-11T09:51:24.579 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00019538300000000001 2024-09-11T09:51:24.579 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.579 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.579 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:24.579 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.939663+0000", 2024-09-11T09:51:24.579 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00054932199999999998 2024-09-11T09:51:24.579 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.580 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.580 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:24.580 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.942910+0000", 2024-09-11T09:51:24.580 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0032473839999999999 2024-09-11T09:51:24.580 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.580 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.580 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:24.580 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.942940+0000", 2024-09-11T09:51:24.580 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.9306999999999999e-05 2024-09-11T09:51:24.580 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.580 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.580 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:24.580 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.942963+0000", 2024-09-11T09:51:24.581 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.3587e-05 2024-09-11T09:51:24.581 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:24.581 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:24.581 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:24.581 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.581 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.581 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4688.0:49 2.5 e53/32 2:b6dd9fc9:::rbd_data.1250f5e25b80.0000000000000009:head v 53'2, mlcod=0'0)", 2024-09-11T09:51:24.581 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.944712+0000", 2024-09-11T09:51:24.581 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 1.6063949050000002, 2024-09-11T09:51:24.581 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.016185069999999999, 2024-09-11T09:51:24.581 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:24.581 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:24.581 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:24.581 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:24.582 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.582 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:24.582 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.944712+0000", 2024-09-11T09:51:24.582 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:24.582 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.582 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.582 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:24.582 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.944712+0000", 2024-09-11T09:51:24.582 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:24.582 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.582 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.582 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:24.582 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.944713+0000", 2024-09-11T09:51:24.583 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0139999999999999e-06 2024-09-11T09:51:24.583 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.583 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.583 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:24.583 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.955518+0000", 2024-09-11T09:51:24.583 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.01080542 2024-09-11T09:51:24.583 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.583 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.583 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:24.583 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.955536+0000", 2024-09-11T09:51:24.583 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7818000000000001e-05 2024-09-11T09:51:24.583 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.583 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.583 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:24.584 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.955544+0000", 2024-09-11T09:51:24.584 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.1699999999999997e-06 2024-09-11T09:51:24.584 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.584 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.584 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:24.584 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.955607+0000", 2024-09-11T09:51:24.584 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.2451000000000002e-05 2024-09-11T09:51:24.584 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.584 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.584 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:24.584 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.955632+0000", 2024-09-11T09:51:24.584 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.5335000000000001e-05 2024-09-11T09:51:24.584 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.585 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.585 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:24.585 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.960826+0000", 2024-09-11T09:51:24.585 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0051938560000000002 2024-09-11T09:51:24.585 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.585 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.585 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:24.585 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.960897+0000", 2024-09-11T09:51:24.585 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.1006000000000004e-05 2024-09-11T09:51:24.585 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:24.585 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:24.585 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:24.585 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.586 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.586 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:53 2.7 2:fe41411c:::rbd_object_map.1250f5e25b80:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:24.586 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.951291+0000", 2024-09-11T09:51:24.586 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 1.599815915, 2024-09-11T09:51:24.586 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0044964999999999996, 2024-09-11T09:51:24.586 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:24.586 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:24.586 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:24.586 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:24.586 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:51:24.586 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:51:24.586 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 53 2024-09-11T09:51:24.586 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.587 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:24.587 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.587 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:24.587 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951291+0000", 2024-09-11T09:51:24.587 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:24.587 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.587 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.587 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:24.587 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951291+0000", 2024-09-11T09:51:24.587 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:24.587 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.587 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.587 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:24.587 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951293+0000", 2024-09-11T09:51:24.588 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7889999999999999e-06 2024-09-11T09:51:24.588 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.588 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.588 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:24.588 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951302+0000", 2024-09-11T09:51:24.588 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.9819999999999997e-06 2024-09-11T09:51:24.588 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.588 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.588 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:24.588 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951324+0000", 2024-09-11T09:51:24.588 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.2141e-05 2024-09-11T09:51:24.588 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.588 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.589 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:24.589 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951338+0000", 2024-09-11T09:51:24.589 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3750999999999999e-05 2024-09-11T09:51:24.589 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.589 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.589 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:24.589 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951409+0000", 2024-09-11T09:51:24.589 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.1658999999999999e-05 2024-09-11T09:51:24.589 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.589 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.589 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:24.589 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951517+0000", 2024-09-11T09:51:24.589 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00010746799999999999 2024-09-11T09:51:24.590 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.590 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.590 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 5", 2024-09-11T09:51:24.590 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951942+0000", 2024-09-11T09:51:24.590 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00042524300000000001 2024-09-11T09:51:24.590 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.590 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.590 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:24.590 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.952872+0000", 2024-09-11T09:51:24.590 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00092976300000000001 2024-09-11T09:51:24.590 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.590 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.590 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:24.591 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.955682+0000", 2024-09-11T09:51:24.591 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0028099459999999998 2024-09-11T09:51:24.591 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.591 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.591 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:24.591 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.955746+0000", 2024-09-11T09:51:24.591 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.4084999999999994e-05 2024-09-11T09:51:24.591 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.591 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.591 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:24.591 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.955787+0000", 2024-09-11T09:51:24.591 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.1672999999999999e-05 2024-09-11T09:51:24.592 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:24.592 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:24.592 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:24.592 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.592 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.592 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4688.0:52 2.4 e53/32 2:26d5c2f0:::rbd_data.1250f5e25b80.000000000000000b:head v 53'8, mlcod=53'6)", 2024-09-11T09:51:24.592 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.963677+0000", 2024-09-11T09:51:24.592 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 1.5874303599999999, 2024-09-11T09:51:24.592 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0089437969999999999, 2024-09-11T09:51:24.592 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:24.592 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:24.592 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:24.593 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:24.593 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.593 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:24.593 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.963677+0000", 2024-09-11T09:51:24.593 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:24.593 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.593 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.593 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:24.593 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.963677+0000", 2024-09-11T09:51:24.593 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:24.593 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.594 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.594 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:24.594 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.963677+0000", 2024-09-11T09:51:24.594 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.7199999999999998e-07 2024-09-11T09:51:24.594 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.594 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.594 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:24.594 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.965506+0000", 2024-09-11T09:51:24.594 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.001829064 2024-09-11T09:51:24.594 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.594 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.595 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:24.595 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.965538+0000", 2024-09-11T09:51:24.595 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.1306999999999997e-05 2024-09-11T09:51:24.595 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.595 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.595 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:24.595 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.965553+0000", 2024-09-11T09:51:24.595 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.4924e-05 2024-09-11T09:51:24.595 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.595 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.596 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:24.596 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.965663+0000", 2024-09-11T09:51:24.596 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000110315 2024-09-11T09:51:24.596 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.596 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.596 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:24.596 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.965692+0000", 2024-09-11T09:51:24.596 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.934e-05 2024-09-11T09:51:24.596 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.596 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.596 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:24.597 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.972537+0000", 2024-09-11T09:51:24.597 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0068444969999999997 2024-09-11T09:51:24.597 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.597 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.597 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:24.597 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.972620+0000", 2024-09-11T09:51:24.597 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.3578000000000005e-05 2024-09-11T09:51:24.597 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:24.597 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:24.597 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:24.597 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.597 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.597 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4688.0:54 2.4 e53/32 2:239e915b:::rbd_data.1250f5e25b80.000000000000000c:head v 53'9, mlcod=53'7)", 2024-09-11T09:51:24.598 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.987028+0000", 2024-09-11T09:51:24.598 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 1.5640784430000001, 2024-09-11T09:51:24.598 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0094155650000000007, 2024-09-11T09:51:24.598 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:24.598 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:24.598 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:24.598 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:24.598 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.598 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:24.598 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.987028+0000", 2024-09-11T09:51:24.598 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:24.598 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.598 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.598 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:24.599 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.987028+0000", 2024-09-11T09:51:24.599 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:24.599 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.599 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.599 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:24.599 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.987030+0000", 2024-09-11T09:51:24.599 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6130000000000001e-06 2024-09-11T09:51:24.599 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.599 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.599 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:24.599 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.988575+0000", 2024-09-11T09:51:24.599 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.001545193 2024-09-11T09:51:24.599 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.599 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.600 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:24.600 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.988604+0000", 2024-09-11T09:51:24.600 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.9173e-05 2024-09-11T09:51:24.600 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.600 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.600 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:24.600 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.988620+0000", 2024-09-11T09:51:24.600 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.5517999999999999e-05 2024-09-11T09:51:24.600 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.600 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.600 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:24.600 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.988704+0000", 2024-09-11T09:51:24.600 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.3588e-05 2024-09-11T09:51:24.601 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.601 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.601 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:24.601 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.988735+0000", 2024-09-11T09:51:24.601 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.1643000000000002e-05 2024-09-11T09:51:24.601 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.601 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.601 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:24.601 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.996383+0000", 2024-09-11T09:51:24.601 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0076478099999999997 2024-09-11T09:51:24.601 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.601 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.602 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:24.602 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.996444+0000", 2024-09-11T09:51:24.602 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.1026999999999997e-05 2024-09-11T09:51:24.602 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:24.602 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:24.602 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:24.602 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.602 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.602 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4688.0:56 2.4 e53/32 2:2167012f:::rbd_data.1250f5e25b80.000000000000000d:head v 53'10, mlcod=53'7)", 2024-09-11T09:51:24.602 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.993894+0000", 2024-09-11T09:51:24.602 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 1.557212901, 2024-09-11T09:51:24.602 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0099268559999999995, 2024-09-11T09:51:24.602 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:24.603 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:24.603 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:24.603 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:24.603 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.603 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:24.603 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.993894+0000", 2024-09-11T09:51:24.603 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:24.603 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.603 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.603 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:24.603 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.993894+0000", 2024-09-11T09:51:24.603 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:24.603 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.604 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.604 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:24.604 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.993896+0000", 2024-09-11T09:51:24.604 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6249999999999999e-06 2024-09-11T09:51:24.604 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.604 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.604 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:24.604 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.997606+0000", 2024-09-11T09:51:24.604 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.003710737 2024-09-11T09:51:24.604 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.604 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.604 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:24.604 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.997628+0000", 2024-09-11T09:51:24.604 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.1971e-05 2024-09-11T09:51:24.605 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.605 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.605 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:24.605 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.997640+0000", 2024-09-11T09:51:24.605 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1481000000000001e-05 2024-09-11T09:51:24.605 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.605 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.605 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:24.605 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.997712+0000", 2024-09-11T09:51:24.605 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.2255000000000005e-05 2024-09-11T09:51:24.605 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.606 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.606 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:24.606 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.997738+0000", 2024-09-11T09:51:24.606 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.6050999999999998e-05 2024-09-11T09:51:24.606 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.606 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.606 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:24.606 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.003779+0000", 2024-09-11T09:51:24.606 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0060410960000000001 2024-09-11T09:51:24.606 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.606 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.606 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:24.607 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.003821+0000", 2024-09-11T09:51:24.607 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.1640000000000001e-05 2024-09-11T09:51:24.607 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:24.607 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:24.607 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:24.607 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.607 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.607 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "pg_update_log_missing(2.4 epoch 53/32 rep_tid 16 entries 53'11 (0'0) error 2:2906e6a7:::rbd_header.1250f5e25b80:head by client.4688.0:59 0.000000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 53'7)", 2024-09-11T09:51:24.607 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.998956+0000", 2024-09-11T09:51:24.607 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 1.5521507319999999, 2024-09-11T09:51:24.607 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.004934127, 2024-09-11T09:51:24.608 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:24.608 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:24.608 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "reached pg", 2024-09-11T09:51:24.608 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:24.608 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.608 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:24.608 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.998956+0000", 2024-09-11T09:51:24.608 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:24.608 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.608 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.608 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:24.608 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.998956+0000", 2024-09-11T09:51:24.609 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:24.609 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.609 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.609 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:24.609 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.998957+0000", 2024-09-11T09:51:24.609 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.139e-06 2024-09-11T09:51:24.609 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.609 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.609 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:24.609 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.998985+0000", 2024-09-11T09:51:24.610 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.7912e-05 2024-09-11T09:51:24.610 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.610 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.610 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:24.610 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.999011+0000", 2024-09-11T09:51:24.610 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.6061e-05 2024-09-11T09:51:24.610 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.610 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.610 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:24.610 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.999028+0000", 2024-09-11T09:51:24.610 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7065999999999999e-05 2024-09-11T09:51:24.610 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.611 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.611 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:24.611 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.001919+0000", 2024-09-11T09:51:24.611 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.002891167 2024-09-11T09:51:24.611 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.611 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.611 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:24.611 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.003890+0000", 2024-09-11T09:51:24.611 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0019707819999999999 2024-09-11T09:51:24.611 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:24.611 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:24.612 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:24.612 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.612 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.612 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4694.0:34 2.7 2:f80bfc0c:ns::rbd_data.1256a0c02879.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:24.612 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.085183+0000", 2024-09-11T09:51:24.612 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 1.4659242189999999, 2024-09-11T09:51:24.612 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.020908599, 2024-09-11T09:51:24.612 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:24.612 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:24.612 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:24.612 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:24.613 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4694", 2024-09-11T09:51:24.613 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:51:24.613 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 34 2024-09-11T09:51:24.613 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.613 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:24.613 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.613 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:24.613 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.085183+0000", 2024-09-11T09:51:24.613 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:24.613 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.613 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.614 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:24.614 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.085183+0000", 2024-09-11T09:51:24.614 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:24.614 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.614 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.614 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:24.614 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.085183+0000", 2024-09-11T09:51:24.614 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.4300000000000002e-07 2024-09-11T09:51:24.614 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.614 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.615 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:24.615 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094337+0000", 2024-09-11T09:51:24.615 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0091532769999999996 2024-09-11T09:51:24.615 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.615 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.615 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:24.615 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094362+0000", 2024-09-11T09:51:24.615 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.5287999999999999e-05 2024-09-11T09:51:24.615 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.615 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.615 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:24.616 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094375+0000", 2024-09-11T09:51:24.616 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2446000000000001e-05 2024-09-11T09:51:24.616 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.616 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.616 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:24.616 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094455+0000", 2024-09-11T09:51:24.616 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.0092999999999994e-05 2024-09-11T09:51:24.616 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.616 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.616 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:24.617 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094618+0000", 2024-09-11T09:51:24.617 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000163747 2024-09-11T09:51:24.617 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.617 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.617 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 5", 2024-09-11T09:51:24.617 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094783+0000", 2024-09-11T09:51:24.617 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000165038 2024-09-11T09:51:24.617 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.617 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.617 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:24.617 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.105853+0000", 2024-09-11T09:51:24.618 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.011069294 2024-09-11T09:51:24.618 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.618 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.618 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:24.618 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.106004+0000", 2024-09-11T09:51:24.618 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00015123900000000001 2024-09-11T09:51:24.618 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.618 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.618 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:24.618 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.106062+0000", 2024-09-11T09:51:24.619 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.8205000000000003e-05 2024-09-11T09:51:24.619 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.619 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.619 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:24.619 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.106091+0000", 2024-09-11T09:51:24.619 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.9128999999999998e-05 2024-09-11T09:51:24.619 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:24.619 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:24.619 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:24.619 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.619 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.620 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4694.0:35 2.7 2:e8766325:ns::rbd_object_map.1256a0c02879:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:24.620 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.094405+0000", 2024-09-11T09:51:24.620 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 1.4567015830000001, 2024-09-11T09:51:24.620 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.012623564, 2024-09-11T09:51:24.620 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:24.620 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:24.620 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:24.620 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:24.620 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4694", 2024-09-11T09:51:24.620 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:51:24.620 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 35 2024-09-11T09:51:24.621 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.621 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:24.621 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.621 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:24.621 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094405+0000", 2024-09-11T09:51:24.621 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:24.621 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.621 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.621 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:24.621 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094405+0000", 2024-09-11T09:51:24.621 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:24.622 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.622 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.622 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:24.622 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094407+0000", 2024-09-11T09:51:24.622 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.739e-06 2024-09-11T09:51:24.622 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.622 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.622 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:24.622 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094411+0000", 2024-09-11T09:51:24.622 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.3340000000000003e-06 2024-09-11T09:51:24.622 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.623 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.623 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:24.623 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094419+0000", 2024-09-11T09:51:24.623 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.8909999999999995e-06 2024-09-11T09:51:24.623 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.623 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.623 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:24.623 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094424+0000", 2024-09-11T09:51:24.623 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.7639999999999996e-06 2024-09-11T09:51:24.623 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.623 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.623 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:24.624 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.100209+0000", 2024-09-11T09:51:24.624 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0057850489999999996 2024-09-11T09:51:24.624 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.624 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.624 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:24.624 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.100341+0000", 2024-09-11T09:51:24.624 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000131438 2024-09-11T09:51:24.624 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.624 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.624 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 5", 2024-09-11T09:51:24.624 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.100805+0000", 2024-09-11T09:51:24.624 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00046471499999999998 2024-09-11T09:51:24.624 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.625 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.625 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:24.625 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.106148+0000", 2024-09-11T09:51:24.625 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0053426460000000004 2024-09-11T09:51:24.625 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.625 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.625 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:24.625 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.106929+0000", 2024-09-11T09:51:24.625 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00078161300000000001 2024-09-11T09:51:24.625 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.625 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.625 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:24.626 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.106985+0000", 2024-09-11T09:51:24.626 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.5223999999999998e-05 2024-09-11T09:51:24.626 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.626 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.626 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:24.626 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.107029+0000", 2024-09-11T09:51:24.626 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.4150999999999998e-05 2024-09-11T09:51:24.626 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:24.626 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:24.626 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:24.626 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.626 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.626 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4694.0:36 2.7 2:e8766325:ns::rbd_object_map.1256a0c02879:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:24.626 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.094452+0000", 2024-09-11T09:51:24.627 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 1.456654573, 2024-09-11T09:51:24.627 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.012680383, 2024-09-11T09:51:24.627 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:24.627 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:24.627 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:24.627 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:24.627 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4694", 2024-09-11T09:51:24.627 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:51:24.627 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 36 2024-09-11T09:51:24.627 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.627 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:24.627 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.627 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:24.628 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094452+0000", 2024-09-11T09:51:24.628 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:24.628 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.628 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.628 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:24.628 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094452+0000", 2024-09-11T09:51:24.628 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:24.628 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.628 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.628 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:24.628 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094453+0000", 2024-09-11T09:51:24.628 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.1800000000000002e-07 2024-09-11T09:51:24.628 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.628 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.629 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:24.629 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094455+0000", 2024-09-11T09:51:24.629 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.7020000000000001e-06 2024-09-11T09:51:24.629 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.629 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.629 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:24.629 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094463+0000", 2024-09-11T09:51:24.629 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.1999999999999997e-06 2024-09-11T09:51:24.629 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.629 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.629 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:24.629 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094467+0000", 2024-09-11T09:51:24.629 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.729e-06 2024-09-11T09:51:24.630 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.630 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.630 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:24.630 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.101269+0000", 2024-09-11T09:51:24.630 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0068018799999999997 2024-09-11T09:51:24.630 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.630 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.630 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:24.630 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.101353+0000", 2024-09-11T09:51:24.630 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.3827999999999998e-05 2024-09-11T09:51:24.630 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.630 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.630 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 5", 2024-09-11T09:51:24.630 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.101588+0000", 2024-09-11T09:51:24.630 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00023467099999999999 2024-09-11T09:51:24.631 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.631 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.631 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:24.631 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.106182+0000", 2024-09-11T09:51:24.631 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0045939489999999999 2024-09-11T09:51:24.631 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.631 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.631 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:24.631 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.107041+0000", 2024-09-11T09:51:24.631 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00085965800000000002 2024-09-11T09:51:24.631 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.631 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.631 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:24.631 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.107084+0000", 2024-09-11T09:51:24.632 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.261e-05 2024-09-11T09:51:24.632 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.632 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.632 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:24.632 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.107133+0000", 2024-09-11T09:51:24.632 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.8838000000000002e-05 2024-09-11T09:51:24.632 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:24.632 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:24.632 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:24.632 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.632 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.632 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4694.0:37 2.7 2:e8766325:ns::rbd_object_map.1256a0c02879:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:24.632 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.094473+0000", 2024-09-11T09:51:24.633 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 1.4566334219999999, 2024-09-11T09:51:24.633 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.012778484999999999, 2024-09-11T09:51:24.633 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:24.633 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:24.633 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:24.633 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:24.633 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4694", 2024-09-11T09:51:24.633 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:51:24.633 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 37 2024-09-11T09:51:24.633 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.633 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:24.633 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.633 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:24.634 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094473+0000", 2024-09-11T09:51:24.634 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:24.634 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.634 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.634 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:24.634 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094473+0000", 2024-09-11T09:51:24.634 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:24.634 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.634 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.634 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:24.634 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094474+0000", 2024-09-11T09:51:24.634 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.9699999999999996e-07 2024-09-11T09:51:24.634 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.634 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.635 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:24.635 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094477+0000", 2024-09-11T09:51:24.635 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.151e-06 2024-09-11T09:51:24.635 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.635 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.635 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:24.635 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094483+0000", 2024-09-11T09:51:24.636 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.6330000000000001e-06 2024-09-11T09:51:24.636 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.636 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.636 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:24.636 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094487+0000", 2024-09-11T09:51:24.636 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.5399999999999997e-06 2024-09-11T09:51:24.636 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.636 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.636 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:24.636 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.101866+0000", 2024-09-11T09:51:24.636 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.007378768 2024-09-11T09:51:24.636 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.636 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.637 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:24.637 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.101960+0000", 2024-09-11T09:51:24.637 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.3861000000000001e-05 2024-09-11T09:51:24.637 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.637 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.637 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 5", 2024-09-11T09:51:24.637 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.102197+0000", 2024-09-11T09:51:24.637 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00023723200000000001 2024-09-11T09:51:24.637 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.637 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.637 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:24.637 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.106222+0000", 2024-09-11T09:51:24.638 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0040249689999999998 2024-09-11T09:51:24.638 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.638 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.638 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:24.638 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.107141+0000", 2024-09-11T09:51:24.638 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00091913900000000002 2024-09-11T09:51:24.638 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.638 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.638 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:24.638 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.107198+0000", 2024-09-11T09:51:24.638 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.6564999999999998e-05 2024-09-11T09:51:24.638 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.638 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.639 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:24.639 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.107252+0000", 2024-09-11T09:51:24.639 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.4030000000000003e-05 2024-09-11T09:51:24.639 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:24.639 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:24.639 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:24.639 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.639 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.639 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4694.0:33 2.4 e53/32 2:3bfc818c:ns::rbd_data.1256a0c02879.0000000000000001:head v 53'14, mlcod=53'12)", 2024-09-11T09:51:24.639 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.096496+0000", 2024-09-11T09:51:24.639 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 1.454610849, 2024-09-11T09:51:24.639 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0099139310000000008, 2024-09-11T09:51:24.640 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:24.640 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:24.640 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:24.640 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:24.640 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.640 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:24.640 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.096496+0000", 2024-09-11T09:51:24.640 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:24.640 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.640 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.640 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:24.640 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.096496+0000", 2024-09-11T09:51:24.640 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:24.641 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.641 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.641 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:24.641 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.096498+0000", 2024-09-11T09:51:24.641 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6309999999999999e-06 2024-09-11T09:51:24.641 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.641 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.641 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:24.641 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.098087+0000", 2024-09-11T09:51:24.641 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0015889019999999999 2024-09-11T09:51:24.641 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.641 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.641 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:24.642 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.098112+0000", 2024-09-11T09:51:24.642 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.5369000000000001e-05 2024-09-11T09:51:24.642 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.642 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.642 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:24.642 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.098127+0000", 2024-09-11T09:51:24.642 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.5218999999999999e-05 2024-09-11T09:51:24.642 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.642 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.642 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:24.642 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.098209+0000", 2024-09-11T09:51:24.642 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.2120999999999995e-05 2024-09-11T09:51:24.642 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.643 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.643 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:24.643 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.098237+0000", 2024-09-11T09:51:24.643 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.7657999999999998e-05 2024-09-11T09:51:24.643 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.643 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.643 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:24.643 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.106376+0000", 2024-09-11T09:51:24.643 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0081386749999999997 2024-09-11T09:51:24.643 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.643 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.643 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:24.643 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.106410+0000", 2024-09-11T09:51:24.643 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.4356000000000001e-05 2024-09-11T09:51:24.644 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:24.644 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:24.644 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:24.644 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.644 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.644 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4694.0:49 2.7 2:e8766325:ns::rbd_object_map.1256a0c02879:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:24.644 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.134806+0000", 2024-09-11T09:51:24.644 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 1.41630078, 2024-09-11T09:51:24.644 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0061629049999999998, 2024-09-11T09:51:24.644 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:24.644 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:24.644 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:24.645 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:24.645 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4694", 2024-09-11T09:51:24.645 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:51:24.645 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 49 2024-09-11T09:51:24.645 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.645 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:24.645 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.645 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:24.645 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.134806+0000", 2024-09-11T09:51:24.645 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:24.645 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.645 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.645 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:24.646 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.134806+0000", 2024-09-11T09:51:24.646 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:24.646 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.646 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.646 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:24.646 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.134808+0000", 2024-09-11T09:51:24.646 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.9870000000000002e-06 2024-09-11T09:51:24.646 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.646 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.646 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:24.646 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.134818+0000", 2024-09-11T09:51:24.646 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.5599999999999999e-06 2024-09-11T09:51:24.646 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.647 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.647 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:24.647 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.134835+0000", 2024-09-11T09:51:24.647 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7399000000000001e-05 2024-09-11T09:51:24.647 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.647 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.647 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:24.647 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.134845+0000", 2024-09-11T09:51:24.647 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0285000000000001e-05 2024-09-11T09:51:24.647 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.647 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.647 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:24.647 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.134911+0000", 2024-09-11T09:51:24.648 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.5654999999999994e-05 2024-09-11T09:51:24.648 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.648 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.648 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:24.648 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.135023+0000", 2024-09-11T09:51:24.648 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00011186799999999999 2024-09-11T09:51:24.648 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.648 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.648 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 5", 2024-09-11T09:51:24.648 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.135290+0000", 2024-09-11T09:51:24.648 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00026725300000000001 2024-09-11T09:51:24.648 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.648 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.649 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:24.649 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.136506+0000", 2024-09-11T09:51:24.649 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0012161139999999999 2024-09-11T09:51:24.649 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.649 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.649 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:24.649 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.140884+0000", 2024-09-11T09:51:24.649 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0043781990000000002 2024-09-11T09:51:24.649 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.649 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.649 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:24.649 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.140923+0000", 2024-09-11T09:51:24.649 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.8813999999999998e-05 2024-09-11T09:51:24.650 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.650 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.650 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:24.650 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.140969+0000", 2024-09-11T09:51:24.650 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.5770999999999999e-05 2024-09-11T09:51:24.650 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:24.650 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:24.650 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:24.650 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.650 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.650 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4694.0:42 2.5 e53/32 2:a4fc00ec:ns::rbd_data.1256a0c02879.0000000000000004:head v 53'3, mlcod=53'1)", 2024-09-11T09:51:24.650 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.136646+0000", 2024-09-11T09:51:24.650 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 1.414460474, 2024-09-11T09:51:24.650 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.012545501000000001, 2024-09-11T09:51:24.651 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:24.651 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:24.651 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:24.651 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:24.651 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.651 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:24.651 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.136646+0000", 2024-09-11T09:51:24.651 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:24.651 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.651 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.651 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:24.651 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.136646+0000", 2024-09-11T09:51:24.651 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:24.652 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.652 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.652 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:24.652 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.136647+0000", 2024-09-11T09:51:24.652 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.2900000000000002e-07 2024-09-11T09:51:24.652 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.652 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.652 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:24.652 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.143620+0000", 2024-09-11T09:51:24.652 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0069727440000000003 2024-09-11T09:51:24.652 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.652 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.652 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:24.653 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.143642+0000", 2024-09-11T09:51:24.653 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.1846000000000001e-05 2024-09-11T09:51:24.653 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.653 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.653 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:24.653 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.143654+0000", 2024-09-11T09:51:24.653 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2081e-05 2024-09-11T09:51:24.653 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.653 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.653 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:24.653 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.143717+0000", 2024-09-11T09:51:24.653 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.3443999999999998e-05 2024-09-11T09:51:24.653 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.653 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.654 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:24.654 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.143740+0000", 2024-09-11T09:51:24.654 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.2626999999999999e-05 2024-09-11T09:51:24.654 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.654 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.654 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:24.654 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.149142+0000", 2024-09-11T09:51:24.654 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0054017630000000004 2024-09-11T09:51:24.654 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.654 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.654 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:24.654 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.149192+0000", 2024-09-11T09:51:24.654 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.0067e-05 2024-09-11T09:51:24.654 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:24.655 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:24.655 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:24.655 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.655 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.655 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4694.0:53 2.7 2:e8766325:ns::rbd_object_map.1256a0c02879:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:24.655 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.147978+0000", 2024-09-11T09:51:24.655 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 1.4031289899999999, 2024-09-11T09:51:24.655 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0044549769999999997, 2024-09-11T09:51:24.655 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:24.655 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:24.655 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:24.655 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:24.655 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4694", 2024-09-11T09:51:24.655 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:51:24.656 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 53 2024-09-11T09:51:24.656 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.656 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:24.656 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.656 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:24.656 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.147978+0000", 2024-09-11T09:51:24.656 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:24.656 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.656 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.656 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:24.656 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.147978+0000", 2024-09-11T09:51:24.656 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:24.656 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.656 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.657 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:24.657 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.147980+0000", 2024-09-11T09:51:24.657 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.8309999999999999e-06 2024-09-11T09:51:24.657 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.657 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.657 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:24.657 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.147989+0000", 2024-09-11T09:51:24.657 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.9639999999999992e-06 2024-09-11T09:51:24.657 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.657 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.657 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:24.657 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.148007+0000", 2024-09-11T09:51:24.657 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7884999999999999e-05 2024-09-11T09:51:24.658 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.658 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.658 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:24.658 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.148017+0000", 2024-09-11T09:51:24.658 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0895e-05 2024-09-11T09:51:24.658 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.658 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.658 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:24.658 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.148091+0000", 2024-09-11T09:51:24.658 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.3709999999999997e-05 2024-09-11T09:51:24.658 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.658 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.658 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:24.658 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.148189+0000", 2024-09-11T09:51:24.659 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.7403000000000002e-05 2024-09-11T09:51:24.659 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.659 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.659 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 5", 2024-09-11T09:51:24.659 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.148504+0000", 2024-09-11T09:51:24.659 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00031589900000000002 2024-09-11T09:51:24.659 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.659 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.659 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:24.659 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.149514+0000", 2024-09-11T09:51:24.659 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0010096 2024-09-11T09:51:24.659 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.659 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.659 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:24.660 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.152367+0000", 2024-09-11T09:51:24.660 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0028532309999999999 2024-09-11T09:51:24.660 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.660 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.660 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:24.660 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.152402+0000", 2024-09-11T09:51:24.660 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.4232e-05 2024-09-11T09:51:24.660 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.660 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.660 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:24.660 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.152433+0000", 2024-09-11T09:51:24.660 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.1327e-05 2024-09-11T09:51:24.660 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:24.661 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:24.661 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:24.661 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.661 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.661 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4694.0:55 2.7 2:e8766325:ns::rbd_object_map.1256a0c02879:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:24.661 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.149838+0000", 2024-09-11T09:51:24.661 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 1.401269257, 2024-09-11T09:51:24.661 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0060695159999999996, 2024-09-11T09:51:24.661 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:24.661 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:24.661 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:24.661 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:24.661 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4694", 2024-09-11T09:51:24.662 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:51:24.662 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 55 2024-09-11T09:51:24.662 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.662 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:24.662 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.662 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:24.662 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.149838+0000", 2024-09-11T09:51:24.662 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:24.662 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.662 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.662 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:24.662 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.149838+0000", 2024-09-11T09:51:24.662 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:24.663 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.663 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.663 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:24.663 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.149839+0000", 2024-09-11T09:51:24.663 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.119e-06 2024-09-11T09:51:24.663 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.663 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.663 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:24.663 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.149845+0000", 2024-09-11T09:51:24.663 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.7810000000000002e-06 2024-09-11T09:51:24.663 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.663 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.663 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:24.663 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.149856+0000", 2024-09-11T09:51:24.664 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1199999999999999e-05 2024-09-11T09:51:24.664 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.664 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.664 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:24.664 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.149863+0000", 2024-09-11T09:51:24.664 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.4109999999999999e-06 2024-09-11T09:51:24.664 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.664 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.664 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:24.664 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.149908+0000", 2024-09-11T09:51:24.664 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.5184e-05 2024-09-11T09:51:24.664 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.664 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.665 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:24.665 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.149964+0000", 2024-09-11T09:51:24.665 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.5395000000000001e-05 2024-09-11T09:51:24.665 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.665 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.665 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 5", 2024-09-11T09:51:24.665 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.150157+0000", 2024-09-11T09:51:24.665 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000193489 2024-09-11T09:51:24.665 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.665 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.665 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:24.665 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.152238+0000", 2024-09-11T09:51:24.665 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0020803480000000001 2024-09-11T09:51:24.666 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.666 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.666 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:24.666 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.155828+0000", 2024-09-11T09:51:24.666 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0035902210000000002 2024-09-11T09:51:24.666 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.666 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.666 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:24.666 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.155873+0000", 2024-09-11T09:51:24.666 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.5145999999999998e-05 2024-09-11T09:51:24.666 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:24.666 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:24.666 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:24.666 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.155907+0000", 2024-09-11T09:51:24.667 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.4221999999999999e-05 2024-09-11T09:51:24.667 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:24.667 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:24.667 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:24.667 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:24.667 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:24.667 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:51:24.669 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-11T09:51:24.669 INFO:tasks.ceph.osd.2.smithi017.stderr:2024-09-11T09:51:24.670+0000 7fd20b170640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31672) UID: 0 2024-09-11T09:51:24.684 INFO:teuthology.orchestra.run.smithi096.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-09-11T09:51:24.684 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-4 --journal-path /var/lib/ceph/osd/ceph-4/journal --op list-pgs 2024-09-11T09:51:24.710 INFO:teuthology.orchestra.run.smithi017.stderr:nodeep-scrub is unset 2024-09-11T09:51:24.769 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:51:24.770+0000 7f3c96ec9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31909) UID: 0 2024-09-11T09:51:24.814 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T09:51:24.819 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:51:24.819 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [], 2024-09-11T09:51:24.819 INFO:teuthology.orchestra.run.smithi017.stdout: "num_ops": 0 2024-09-11T09:51:24.819 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:51:24.828 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-11T09:51:24.870 INFO:tasks.ceph.osd.2.smithi017.stderr:2024-09-11T09:51:24.870+0000 7fd20b170640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31672) UID: 0 2024-09-11T09:51:24.939 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-11T09:51:24.939 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T09:51:24.957 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-11T09:51:24.957 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T09:51:24.970 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:51:24.971+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:51:24.973 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-11T09:51:24.974 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T09:51:24.979 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:51:24.979 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:51:24.988 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-11T09:51:24.991 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-11T09:51:24.991 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T09:51:24.991 INFO:teuthology.orchestra.run.smithi017.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-09-11T09:51:25.009 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-11T09:51:25.009 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T09:51:25.013 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-11T09:51:25.013 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-11T09:51:25.072 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-09-11T09:51:25.072+0000 7f7b6c7d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31904) UID: 0 2024-09-11T09:51:25.098 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:51:25.098 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:51:25.107 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-11T09:51:25.173 INFO:tasks.ceph.osd.2.smithi017.stderr:2024-09-11T09:51:25.174+0000 7fd20b170640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31672) UID: 0 2024-09-11T09:51:25.215 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:51:25.215 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:51:25.223 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-11T09:51:25.308 INFO:tasks.ceph.osd.0.smithi017.stderr:2024-09-11T09:51:25.309+0000 7fb73c0f1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31665) UID: 0 2024-09-11T09:51:25.371 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:51:25.372 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:51:25.375 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:51:25.376+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:51:25.380 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-11T09:51:25.476 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:51:25.477+0000 7f3c96ec9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31909) UID: 0 2024-09-11T09:51:25.528 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:51:25.528 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:51:25.536 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-11T09:51:25.580 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-09-11T09:51:25.581+0000 7f7b6c7d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31904) UID: 0 2024-09-11T09:51:25.641 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:51:25.641 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:51:25.649 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-11T09:51:25.678 INFO:tasks.ceph.osd.2.smithi017.stderr:2024-09-11T09:51:25.679+0000 7fd20b170640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31672) UID: 0 2024-09-11T09:51:25.779 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:51:25.780+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:51:25.792 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:51:25.792 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:51:25.800 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-11T09:51:25.880 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-09-11T09:51:25.881+0000 7f7b6c7d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31904) UID: 0 2024-09-11T09:51:25.902 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:51:25.903 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:51:25.911 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-11T09:51:25.981 INFO:tasks.ceph.osd.0.smithi017.stderr:2024-09-11T09:51:25.982+0000 7fb73c0f1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31665) UID: 0 2024-09-11T09:51:26.053 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:51:26.053 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:51:26.062 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-11T09:51:26.081 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:51:26.082+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:51:26.166 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:51:26.166 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:51:26.175 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-11T09:51:26.182 INFO:tasks.ceph.osd.2.smithi017.stderr:2024-09-11T09:51:26.183+0000 7fd20b170640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31672) UID: 0 2024-09-11T09:51:26.277 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:51:26.277 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:51:26.282 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-09-11T09:51:26.283+0000 7f7b6c7d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31904) UID: 0 2024-09-11T09:51:26.286 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-11T09:51:26.384 INFO:tasks.ceph.osd.2.smithi017.stderr:2024-09-11T09:51:26.385+0000 7fd20b170640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31672) UID: 0 2024-09-11T09:51:26.402 INFO:teuthology.orchestra.run.smithi096.stdout:3.3 2024-09-11T09:51:26.406 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:51:26.406 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:51:26.416 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-11T09:51:26.500 INFO:tasks.ceph.osd.2.smithi017.stderr:2024-09-11T09:51:26.501+0000 7fd20b170640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31672) UID: 0 2024-09-11T09:51:26.565 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:51:26.565 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:51:26.574 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-11T09:51:26.615 INFO:tasks.ceph.osd.0.smithi017.stderr:2024-09-11T09:51:26.616+0000 7fb73c0f1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31665) UID: 0 2024-09-11T09:51:26.687 INFO:tasks.ceph.osd.0.smithi017.stderr:2024-09-11T09:51:26.688+0000 7fb73c0f1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31665) UID: 0 2024-09-11T09:51:26.724 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:51:26.724 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:51:26.726 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-11T09:51:26.733 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-11T09:51:26.818 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:51:26.819+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:51:26.883 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:51:26.883 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:51:26.889 INFO:tasks.ceph.osd.2.smithi017.stderr:2024-09-11T09:51:26.890+0000 7fd20b170640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31672) UID: 0 2024-09-11T09:51:26.893 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-11T09:51:26.919 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-4 --journal-path /var/lib/ceph/osd/ceph-4/journal --op export --pgid 3.3 --file /var/log/ceph/exp.3.3.4 2024-09-11T09:51:26.990 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-09-11T09:51:26.991+0000 7f7b6c7d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31904) UID: 0 2024-09-11T09:51:27.014 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T09:51:27.042 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:51:27.042 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:51:27.051 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-11T09:51:27.092 INFO:tasks.ceph.osd.2.smithi017.stderr:2024-09-11T09:51:27.093+0000 7fd20b170640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31672) UID: 0 2024-09-11T09:51:27.139 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-11T09:51:27.140 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T09:51:27.157 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-11T09:51:27.157 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T09:51:27.174 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-11T09:51:27.174 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T09:51:27.191 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-11T09:51:27.192 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:51:27.192+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:51:27.192 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T09:51:27.192 INFO:teuthology.orchestra.run.smithi017.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-09-11T09:51:27.203 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:51:27.203 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 20, 2024-09-11T09:51:27.203 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 600, 2024-09-11T09:51:27.203 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [ 2024-09-11T09:51:27.203 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.203 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:34 2.4 2:2367385a:::rbd_data.1250f5e25b80.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:27.203 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.897717+0000", 2024-09-11T09:51:27.203 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 4.3048673109999998, 2024-09-11T09:51:27.203 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.021323661000000001, 2024-09-11T09:51:27.203 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:27.203 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:27.203 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:27.204 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:27.204 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:51:27.204 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:51:27.204 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 34 2024-09-11T09:51:27.204 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.204 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:27.204 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.204 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:27.204 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.897717+0000", 2024-09-11T09:51:27.204 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:27.204 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.204 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.205 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:27.205 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.897717+0000", 2024-09-11T09:51:27.205 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:27.205 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.205 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.205 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:27.205 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.897718+0000", 2024-09-11T09:51:27.205 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.1800000000000004e-07 2024-09-11T09:51:27.205 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.205 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.205 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:27.205 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.905339+0000", 2024-09-11T09:51:27.206 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0076210239999999997 2024-09-11T09:51:27.206 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.206 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.206 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:27.206 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.905366+0000", 2024-09-11T09:51:27.206 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.6959000000000001e-05 2024-09-11T09:51:27.206 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.206 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.206 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:27.206 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.905375+0000", 2024-09-11T09:51:27.206 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.516e-06 2024-09-11T09:51:27.206 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.206 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.207 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:27.207 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.905475+0000", 2024-09-11T09:51:27.207 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.9738e-05 2024-09-11T09:51:27.207 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.207 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.207 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:27.207 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.905660+0000", 2024-09-11T09:51:27.207 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00018554199999999999 2024-09-11T09:51:27.207 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.207 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.207 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:27.207 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.905808+0000", 2024-09-11T09:51:27.207 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000147386 2024-09-11T09:51:27.208 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.208 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.208 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:27.208 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.918230+0000", 2024-09-11T09:51:27.208 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.012421690000000001 2024-09-11T09:51:27.208 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.208 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.208 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:27.208 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.918932+0000", 2024-09-11T09:51:27.208 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00070267300000000001 2024-09-11T09:51:27.208 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.208 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.209 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:27.209 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.918989+0000", 2024-09-11T09:51:27.209 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.7268999999999997e-05 2024-09-11T09:51:27.209 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.209 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.209 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:27.209 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919040+0000", 2024-09-11T09:51:27.209 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.0946000000000003e-05 2024-09-11T09:51:27.209 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:27.209 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:27.209 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:27.209 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.209 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.210 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4688.0:31 2.1 e53/15 2:8654721b:::rbd_data.1250f5e25b80.0000000000000000:head v 53'2, mlcod=0'0)", 2024-09-11T09:51:27.210 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.904661+0000", 2024-09-11T09:51:27.210 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 4.2979228220000003, 2024-09-11T09:51:27.210 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.014152804999999999, 2024-09-11T09:51:27.210 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:27.210 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:27.210 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:27.210 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:27.210 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.210 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:27.210 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.904661+0000", 2024-09-11T09:51:27.210 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:27.210 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.210 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.211 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:27.211 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.904661+0000", 2024-09-11T09:51:27.211 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:27.211 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.211 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.211 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:27.211 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.904662+0000", 2024-09-11T09:51:27.211 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0759999999999999e-06 2024-09-11T09:51:27.211 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.211 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.211 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:27.211 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.907298+0000", 2024-09-11T09:51:27.211 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0026361240000000001 2024-09-11T09:51:27.212 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.212 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.212 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:27.212 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.907320+0000", 2024-09-11T09:51:27.212 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.1268999999999999e-05 2024-09-11T09:51:27.212 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.212 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.212 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:27.212 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.907331+0000", 2024-09-11T09:51:27.212 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1031e-05 2024-09-11T09:51:27.212 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.212 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.212 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:27.213 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.907403+0000", 2024-09-11T09:51:27.213 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.2571999999999996e-05 2024-09-11T09:51:27.213 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.213 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.213 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:27.213 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.907428+0000", 2024-09-11T09:51:27.213 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.4380999999999999e-05 2024-09-11T09:51:27.213 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.213 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.213 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:27.213 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.918750+0000", 2024-09-11T09:51:27.213 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.011322211 2024-09-11T09:51:27.213 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.213 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.214 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:27.214 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.918814+0000", 2024-09-11T09:51:27.214 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.4140999999999995e-05 2024-09-11T09:51:27.214 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:27.214 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:27.214 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:27.214 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.214 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.214 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:36 2.6 2:61e1b442:::rbd_data.1250f5e25b80.0000000000000003:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:27.214 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.905404+0000", 2024-09-11T09:51:27.214 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 4.2971799510000004, 2024-09-11T09:51:27.214 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.030912479999999999, 2024-09-11T09:51:27.214 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:27.214 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:27.215 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:27.215 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:27.215 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:51:27.215 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:51:27.215 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 36 2024-09-11T09:51:27.215 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.215 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:27.215 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.215 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:27.215 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.905404+0000", 2024-09-11T09:51:27.215 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:27.215 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.215 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.215 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:27.216 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.905404+0000", 2024-09-11T09:51:27.216 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:27.216 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.216 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.216 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:27.216 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.905405+0000", 2024-09-11T09:51:27.216 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2270000000000001e-06 2024-09-11T09:51:27.216 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.216 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.216 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:27.216 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.914859+0000", 2024-09-11T09:51:27.216 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0094541190000000004 2024-09-11T09:51:27.216 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.217 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.217 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:27.217 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.914879+0000", 2024-09-11T09:51:27.217 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.0052999999999999e-05 2024-09-11T09:51:27.217 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.217 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.217 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:27.217 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.914897+0000", 2024-09-11T09:51:27.217 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7331000000000001e-05 2024-09-11T09:51:27.217 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.217 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.217 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:27.217 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.915019+0000", 2024-09-11T09:51:27.218 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00012168 2024-09-11T09:51:27.218 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.218 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.218 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:27.218 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.915269+0000", 2024-09-11T09:51:27.218 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00025054899999999998 2024-09-11T09:51:27.218 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.218 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.218 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 3", 2024-09-11T09:51:27.218 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.915479+0000", 2024-09-11T09:51:27.218 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00021017799999999999 2024-09-11T09:51:27.218 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.218 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.219 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:27.219 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.926070+0000", 2024-09-11T09:51:27.219 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.010590624999999999 2024-09-11T09:51:27.219 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.219 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.219 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:27.219 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.936198+0000", 2024-09-11T09:51:27.219 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.010128194 2024-09-11T09:51:27.219 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.219 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.219 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:27.219 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.936263+0000", 2024-09-11T09:51:27.219 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.5366999999999999e-05 2024-09-11T09:51:27.219 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.220 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.220 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:27.220 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.936317+0000", 2024-09-11T09:51:27.220 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.3156999999999999e-05 2024-09-11T09:51:27.220 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:27.220 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:27.220 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:27.220 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.220 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.220 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:39 2.6 2:743e7616:::rbd_data.1250f5e25b80.0000000000000005:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:27.220 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.914960+0000", 2024-09-11T09:51:27.220 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 4.2876236490000004, 2024-09-11T09:51:27.221 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.022371518, 2024-09-11T09:51:27.221 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:27.221 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:27.221 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:27.221 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:27.221 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:51:27.221 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:51:27.221 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 39 2024-09-11T09:51:27.221 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.221 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:27.221 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.221 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:27.221 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.914960+0000", 2024-09-11T09:51:27.221 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:27.222 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.222 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.222 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:27.222 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.914960+0000", 2024-09-11T09:51:27.222 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:27.222 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.222 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.222 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:27.222 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.914963+0000", 2024-09-11T09:51:27.222 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.1119999999999999e-06 2024-09-11T09:51:27.222 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.222 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.223 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:27.223 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.920417+0000", 2024-09-11T09:51:27.223 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0054543969999999997 2024-09-11T09:51:27.223 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.223 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.223 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:27.223 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.920430+0000", 2024-09-11T09:51:27.223 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3414000000000001e-05 2024-09-11T09:51:27.223 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.223 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.223 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:27.223 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.920437+0000", 2024-09-11T09:51:27.223 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.6159999999999998e-06 2024-09-11T09:51:27.223 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.224 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.224 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:27.224 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.921325+0000", 2024-09-11T09:51:27.224 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000887731 2024-09-11T09:51:27.224 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.224 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.224 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:27.224 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.921422+0000", 2024-09-11T09:51:27.224 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.7201999999999995e-05 2024-09-11T09:51:27.224 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.224 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.224 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 3", 2024-09-11T09:51:27.224 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.921511+0000", 2024-09-11T09:51:27.225 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.8880000000000003e-05 2024-09-11T09:51:27.225 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.225 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.225 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:27.225 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.928092+0000", 2024-09-11T09:51:27.225 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0065816290000000003 2024-09-11T09:51:27.225 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.225 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.225 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:27.225 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.937240+0000", 2024-09-11T09:51:27.225 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0091481070000000008 2024-09-11T09:51:27.225 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.226 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.226 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:27.226 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.937294+0000", 2024-09-11T09:51:27.226 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.3100000000000003e-05 2024-09-11T09:51:27.226 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.226 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.226 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:27.226 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.937332+0000", 2024-09-11T09:51:27.226 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.8330000000000001e-05 2024-09-11T09:51:27.226 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:27.226 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:27.226 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:27.226 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.227 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.227 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4688.0:32 2.2 e53/15 2:45941fd2:::rbd_data.1250f5e25b80.0000000000000001:head v 53'3, mlcod=17'1)", 2024-09-11T09:51:27.227 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.916509+0000", 2024-09-11T09:51:27.227 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 4.286074567, 2024-09-11T09:51:27.227 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.015885318999999998, 2024-09-11T09:51:27.227 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:27.227 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:27.227 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:27.227 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:27.227 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.227 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:27.227 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.916509+0000", 2024-09-11T09:51:27.227 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:27.228 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.228 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.228 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:27.228 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.916509+0000", 2024-09-11T09:51:27.228 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:27.228 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.228 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.228 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:27.228 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.916511+0000", 2024-09-11T09:51:27.228 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.052e-06 2024-09-11T09:51:27.228 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.228 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.228 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:27.229 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.927019+0000", 2024-09-11T09:51:27.229 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.010508224 2024-09-11T09:51:27.229 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.229 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.229 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:27.229 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.927038+0000", 2024-09-11T09:51:27.229 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.9590999999999999e-05 2024-09-11T09:51:27.229 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.229 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.229 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:27.229 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.927050+0000", 2024-09-11T09:51:27.229 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1439e-05 2024-09-11T09:51:27.229 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.230 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.230 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:27.230 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.927115+0000", 2024-09-11T09:51:27.230 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.5301000000000004e-05 2024-09-11T09:51:27.230 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.230 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.230 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:27.230 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.927132+0000", 2024-09-11T09:51:27.230 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7297000000000001e-05 2024-09-11T09:51:27.230 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.230 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.230 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:27.230 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.932354+0000", 2024-09-11T09:51:27.231 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0052211870000000004 2024-09-11T09:51:27.231 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.231 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.231 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:27.231 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.932395+0000", 2024-09-11T09:51:27.231 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.1227999999999999e-05 2024-09-11T09:51:27.231 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:27.231 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:27.231 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:27.231 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.231 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.231 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:44 2.6 2:67246a22:::rbd_data.1250f5e25b80.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:27.231 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.943181+0000", 2024-09-11T09:51:27.232 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 4.259403356, 2024-09-11T09:51:27.232 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.022629371999999998, 2024-09-11T09:51:27.232 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:27.232 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:27.232 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:27.232 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:27.232 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:51:27.232 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:51:27.232 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 44 2024-09-11T09:51:27.232 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.232 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:27.232 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.233 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:27.233 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.943181+0000", 2024-09-11T09:51:27.233 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:27.233 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.233 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.233 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:27.233 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.943181+0000", 2024-09-11T09:51:27.233 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:27.233 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.233 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.233 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:27.233 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.943182+0000", 2024-09-11T09:51:27.233 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.0500000000000002e-07 2024-09-11T09:51:27.234 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.234 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.234 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:27.234 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951240+0000", 2024-09-11T09:51:27.234 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.008058529 2024-09-11T09:51:27.234 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.234 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.234 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:27.234 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951260+0000", 2024-09-11T09:51:27.234 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.9570000000000001e-05 2024-09-11T09:51:27.234 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.234 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.234 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:27.235 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951270+0000", 2024-09-11T09:51:27.235 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0018e-05 2024-09-11T09:51:27.235 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.235 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.235 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:27.235 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951349+0000", 2024-09-11T09:51:27.235 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.8801000000000007e-05 2024-09-11T09:51:27.235 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.235 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.235 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:27.235 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951483+0000", 2024-09-11T09:51:27.235 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00013427 2024-09-11T09:51:27.236 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.236 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.236 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 3", 2024-09-11T09:51:27.236 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951593+0000", 2024-09-11T09:51:27.236 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00011031000000000001 2024-09-11T09:51:27.236 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.236 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.236 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:27.236 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.957303+0000", 2024-09-11T09:51:27.236 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0057097959999999996 2024-09-11T09:51:27.236 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.236 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.236 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:27.237 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.965690+0000", 2024-09-11T09:51:27.237 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0083867420000000008 2024-09-11T09:51:27.237 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.237 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.237 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:27.237 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.965749+0000", 2024-09-11T09:51:27.237 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.8981999999999999e-05 2024-09-11T09:51:27.237 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.237 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.237 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:27.237 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.965810+0000", 2024-09-11T09:51:27.237 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.1449000000000001e-05 2024-09-11T09:51:27.238 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:27.238 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:27.238 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:27.238 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.238 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.238 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:51 2.6 2:6b5894fc:::rbd_data.1250f5e25b80.000000000000000a:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:27.238 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.951324+0000", 2024-09-11T09:51:27.238 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 4.2512602829999997, 2024-09-11T09:51:27.238 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.021375273, 2024-09-11T09:51:27.238 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:27.238 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:27.238 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:27.238 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:27.239 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:51:27.239 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:51:27.239 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 51 2024-09-11T09:51:27.239 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.239 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:27.239 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.239 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:27.239 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951324+0000", 2024-09-11T09:51:27.239 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:27.239 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.239 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.239 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:27.239 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951324+0000", 2024-09-11T09:51:27.240 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:27.240 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.240 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.240 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:27.240 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951324+0000", 2024-09-11T09:51:27.240 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.39e-07 2024-09-11T09:51:27.240 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.240 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.240 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:27.240 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.958537+0000", 2024-09-11T09:51:27.240 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0072126769999999998 2024-09-11T09:51:27.240 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.240 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.241 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:27.241 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.958554+0000", 2024-09-11T09:51:27.241 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7232e-05 2024-09-11T09:51:27.241 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.241 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.241 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:27.241 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.958570+0000", 2024-09-11T09:51:27.241 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.5747999999999999e-05 2024-09-11T09:51:27.241 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.241 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.241 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:27.241 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.958638+0000", 2024-09-11T09:51:27.241 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.7810000000000003e-05 2024-09-11T09:51:27.242 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.242 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.242 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:27.242 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.958758+0000", 2024-09-11T09:51:27.242 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000120689 2024-09-11T09:51:27.242 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.242 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.242 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 3", 2024-09-11T09:51:27.242 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.958865+0000", 2024-09-11T09:51:27.242 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00010692 2024-09-11T09:51:27.242 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.242 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.242 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:27.242 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.970712+0000", 2024-09-11T09:51:27.243 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.011846327 2024-09-11T09:51:27.243 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.243 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.243 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:27.243 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.972603+0000", 2024-09-11T09:51:27.243 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0018918610000000001 2024-09-11T09:51:27.243 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.243 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.243 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:27.243 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.972653+0000", 2024-09-11T09:51:27.243 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.9762999999999998e-05 2024-09-11T09:51:27.243 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.243 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.244 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:27.244 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.972699+0000", 2024-09-11T09:51:27.244 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.5806999999999997e-05 2024-09-11T09:51:27.244 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:27.244 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:27.244 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:27.244 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.244 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.244 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:52 2.4 2:26d5c2f0:::rbd_data.1250f5e25b80.000000000000000b:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:27.244 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.958596+0000", 2024-09-11T09:51:27.244 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 4.243988088, 2024-09-11T09:51:27.244 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.016477849999999999, 2024-09-11T09:51:27.244 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:27.244 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:27.245 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:27.245 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:27.245 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:51:27.245 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:51:27.245 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 52 2024-09-11T09:51:27.245 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.245 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:27.245 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.245 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:27.245 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.958596+0000", 2024-09-11T09:51:27.245 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:27.245 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.245 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.246 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:27.246 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.958596+0000", 2024-09-11T09:51:27.246 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:27.246 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.246 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.246 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:27.246 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.958597+0000", 2024-09-11T09:51:27.246 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1960000000000001e-06 2024-09-11T09:51:27.246 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.246 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.246 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:27.246 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.963157+0000", 2024-09-11T09:51:27.246 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0045603029999999999 2024-09-11T09:51:27.247 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.247 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.247 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:27.247 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.963171+0000", 2024-09-11T09:51:27.247 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3777999999999999e-05 2024-09-11T09:51:27.247 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.247 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.247 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:27.247 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.963181+0000", 2024-09-11T09:51:27.247 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.9010000000000006e-06 2024-09-11T09:51:27.247 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.247 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.247 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:27.248 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.963246+0000", 2024-09-11T09:51:27.248 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.5351000000000005e-05 2024-09-11T09:51:27.248 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.248 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.248 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:27.248 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.963419+0000", 2024-09-11T09:51:27.248 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00017299700000000001 2024-09-11T09:51:27.248 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.248 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.248 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:27.248 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.963570+0000", 2024-09-11T09:51:27.248 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00015066999999999999 2024-09-11T09:51:27.248 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.249 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.249 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:27.249 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.973881+0000", 2024-09-11T09:51:27.249 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.010311268 2024-09-11T09:51:27.249 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.249 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.249 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:27.249 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.974963+0000", 2024-09-11T09:51:27.249 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.001081493 2024-09-11T09:51:27.249 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.249 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.249 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:27.249 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.975012+0000", 2024-09-11T09:51:27.250 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.9471000000000001e-05 2024-09-11T09:51:27.250 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.250 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.250 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:27.250 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.975074+0000", 2024-09-11T09:51:27.250 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.1421999999999996e-05 2024-09-11T09:51:27.250 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:27.250 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:27.250 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:27.250 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.250 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.250 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4688.0:47 2.1 e53/15 2:95f53b3e:::rbd_data.1250f5e25b80.0000000000000008:head v 53'3, mlcod=53'1)", 2024-09-11T09:51:27.250 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.960861+0000", 2024-09-11T09:51:27.250 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 4.2417226699999997, 2024-09-11T09:51:27.251 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.012346107, 2024-09-11T09:51:27.251 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:27.251 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:27.251 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:27.251 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:27.251 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.251 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:27.251 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.960861+0000", 2024-09-11T09:51:27.251 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:27.251 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.251 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.251 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:27.251 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.960861+0000", 2024-09-11T09:51:27.252 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:27.252 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.252 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.252 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:27.252 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.960863+0000", 2024-09-11T09:51:27.252 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7150000000000001e-06 2024-09-11T09:51:27.252 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.252 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.252 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:27.252 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.962428+0000", 2024-09-11T09:51:27.252 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.001564668 2024-09-11T09:51:27.252 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.252 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.253 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:27.253 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.962446+0000", 2024-09-11T09:51:27.253 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.8476999999999999e-05 2024-09-11T09:51:27.253 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.253 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.253 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:27.253 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.962458+0000", 2024-09-11T09:51:27.253 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1263999999999999e-05 2024-09-11T09:51:27.253 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.253 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.253 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:27.253 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.962575+0000", 2024-09-11T09:51:27.253 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000117025 2024-09-11T09:51:27.254 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.254 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.254 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:27.254 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.962601+0000", 2024-09-11T09:51:27.254 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.5984999999999999e-05 2024-09-11T09:51:27.254 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.254 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.254 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:27.254 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.973065+0000", 2024-09-11T09:51:27.254 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.010464753 2024-09-11T09:51:27.254 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.254 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.254 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:27.255 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.973207+0000", 2024-09-11T09:51:27.255 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00014222 2024-09-11T09:51:27.255 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:27.255 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:27.255 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:27.255 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.255 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.255 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:54 2.4 2.da8979c4 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:27.255 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.963207+0000", 2024-09-11T09:51:27.255 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 4.2393772319999998, 2024-09-11T09:51:27.255 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.010641575, 2024-09-11T09:51:27.255 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:27.255 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:27.256 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "reached pg", 2024-09-11T09:51:27.256 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:27.256 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:51:27.256 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:51:27.256 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 54 2024-09-11T09:51:27.256 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.256 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:27.256 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.256 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:27.256 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.963207+0000", 2024-09-11T09:51:27.256 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:27.256 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.257 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.257 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:27.257 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.963207+0000", 2024-09-11T09:51:27.257 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:27.257 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.257 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.257 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:27.257 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.963208+0000", 2024-09-11T09:51:27.257 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2979999999999999e-06 2024-09-11T09:51:27.257 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.257 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.257 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:27.257 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.967419+0000", 2024-09-11T09:51:27.258 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0042111750000000002 2024-09-11T09:51:27.258 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.258 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.258 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:27.258 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.967439+0000", 2024-09-11T09:51:27.258 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.0129999999999999e-05 2024-09-11T09:51:27.258 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.258 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.258 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:27.258 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.967453+0000", 2024-09-11T09:51:27.258 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3781000000000001e-05 2024-09-11T09:51:27.258 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.258 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.259 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:27.259 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.973840+0000", 2024-09-11T09:51:27.259 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0063872269999999997 2024-09-11T09:51:27.259 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.259 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.259 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:27.259 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.973848+0000", 2024-09-11T09:51:27.259 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.9640000000000003e-06 2024-09-11T09:51:27.259 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:27.259 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:27.259 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:27.259 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.260 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.260 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:54 2.4 2:239e915b:::rbd_data.1250f5e25b80.000000000000000c:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:27.260 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.982401+0000", 2024-09-11T09:51:27.260 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 4.2201833049999999, 2024-09-11T09:51:27.260 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.016817347, 2024-09-11T09:51:27.260 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:27.260 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:27.260 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:27.260 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:27.260 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:51:27.260 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:51:27.260 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 54 2024-09-11T09:51:27.260 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.261 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:27.261 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.261 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:27.261 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.982401+0000", 2024-09-11T09:51:27.261 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:27.261 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.261 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.261 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:27.261 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.982401+0000", 2024-09-11T09:51:27.261 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:27.261 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.261 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.262 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:27.262 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.982402+0000", 2024-09-11T09:51:27.262 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.8999999999999995e-07 2024-09-11T09:51:27.262 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.262 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.262 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:27.262 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.986535+0000", 2024-09-11T09:51:27.262 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0041333689999999996 2024-09-11T09:51:27.262 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.262 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.262 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:27.262 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.986555+0000", 2024-09-11T09:51:27.262 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.9851e-05 2024-09-11T09:51:27.262 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.263 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.263 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:27.263 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.986569+0000", 2024-09-11T09:51:27.263 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3784000000000001e-05 2024-09-11T09:51:27.263 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.263 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.263 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:27.263 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.986683+0000", 2024-09-11T09:51:27.263 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000114474 2024-09-11T09:51:27.263 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.263 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.263 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:27.263 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.986815+0000", 2024-09-11T09:51:27.264 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00013217799999999999 2024-09-11T09:51:27.264 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.264 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.264 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:27.264 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.986918+0000", 2024-09-11T09:51:27.264 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00010235599999999999 2024-09-11T09:51:27.264 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.264 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.264 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:27.264 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.998983+0000", 2024-09-11T09:51:27.264 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.012065352 2024-09-11T09:51:27.264 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.265 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.265 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:27.265 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.999081+0000", 2024-09-11T09:51:27.265 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.8091000000000007e-05 2024-09-11T09:51:27.265 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.265 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.265 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:27.265 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.999159+0000", 2024-09-11T09:51:27.265 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.7823000000000004e-05 2024-09-11T09:51:27.265 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.265 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.265 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:27.265 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.999218+0000", 2024-09-11T09:51:27.266 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.9178999999999997e-05 2024-09-11T09:51:27.266 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:27.266 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:27.266 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:27.266 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.266 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.266 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:56 2.4 2:2167012f:::rbd_data.1250f5e25b80.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:27.266 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.986661+0000", 2024-09-11T09:51:27.266 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 4.2159228520000003, 2024-09-11T09:51:27.266 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.017353539000000001, 2024-09-11T09:51:27.266 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:27.266 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:27.266 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:27.267 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:27.267 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:51:27.267 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:51:27.267 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 56 2024-09-11T09:51:27.267 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.267 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:27.267 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.267 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:27.267 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.986661+0000", 2024-09-11T09:51:27.267 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:27.267 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.267 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.268 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:27.268 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.986661+0000", 2024-09-11T09:51:27.268 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:27.268 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.268 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.268 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:27.268 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.986663+0000", 2024-09-11T09:51:27.268 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.8989999999999999e-06 2024-09-11T09:51:27.268 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.268 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.268 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:27.268 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.990444+0000", 2024-09-11T09:51:27.269 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0037808569999999999 2024-09-11T09:51:27.269 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.269 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.269 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:27.269 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.990462+0000", 2024-09-11T09:51:27.269 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.8450000000000001e-05 2024-09-11T09:51:27.269 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.269 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.269 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:27.269 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.990476+0000", 2024-09-11T09:51:27.269 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3565e-05 2024-09-11T09:51:27.269 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.269 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.270 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:27.270 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.993582+0000", 2024-09-11T09:51:27.270 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0031059109999999998 2024-09-11T09:51:27.270 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.270 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.270 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:27.270 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.993686+0000", 2024-09-11T09:51:27.270 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000104424 2024-09-11T09:51:27.270 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.270 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.270 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:27.271 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.993799+0000", 2024-09-11T09:51:27.271 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000112985 2024-09-11T09:51:27.271 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.271 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.271 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:27.271 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.002449+0000", 2024-09-11T09:51:27.271 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.008650151 2024-09-11T09:51:27.271 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.271 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.271 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:27.271 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.003948+0000", 2024-09-11T09:51:27.271 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.001498636 2024-09-11T09:51:27.271 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.272 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.272 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:27.272 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.003985+0000", 2024-09-11T09:51:27.272 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.7054999999999997e-05 2024-09-11T09:51:27.272 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.272 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.272 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:27.272 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.004015+0000", 2024-09-11T09:51:27.272 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.9606000000000001e-05 2024-09-11T09:51:27.272 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:27.272 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:27.272 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:27.272 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.273 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.273 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:58 2.6 2:6277c644:::rbd_data.1250f5e25b80.000000000000000e:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:27.273 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.990540+0000", 2024-09-11T09:51:27.273 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 4.212044057, 2024-09-11T09:51:27.273 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.016232068999999998, 2024-09-11T09:51:27.273 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:27.273 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:27.273 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:27.273 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:27.273 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:51:27.273 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:51:27.274 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 58 2024-09-11T09:51:27.274 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.274 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:27.274 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.274 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:27.274 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.990540+0000", 2024-09-11T09:51:27.274 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:27.274 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.274 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.274 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:27.275 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.990540+0000", 2024-09-11T09:51:27.275 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:27.275 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.275 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.275 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:27.275 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.990541+0000", 2024-09-11T09:51:27.275 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2359999999999999e-06 2024-09-11T09:51:27.275 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.275 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.275 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:27.275 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.994626+0000", 2024-09-11T09:51:27.275 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.004084865 2024-09-11T09:51:27.275 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.276 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.276 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:27.276 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.994644+0000", 2024-09-11T09:51:27.276 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7866000000000001e-05 2024-09-11T09:51:27.276 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.276 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.276 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:27.276 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.994653+0000", 2024-09-11T09:51:27.276 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.2159999999999995e-06 2024-09-11T09:51:27.276 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.276 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.276 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:27.276 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.994732+0000", 2024-09-11T09:51:27.277 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.8954000000000003e-05 2024-09-11T09:51:27.277 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.277 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.277 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:27.277 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.994906+0000", 2024-09-11T09:51:27.277 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0001741 2024-09-11T09:51:27.277 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.277 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.277 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 3", 2024-09-11T09:51:27.277 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.995083+0000", 2024-09-11T09:51:27.277 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00017637100000000001 2024-09-11T09:51:27.277 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.277 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.278 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:27.278 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.003066+0000", 2024-09-11T09:51:27.278 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0079833360000000006 2024-09-11T09:51:27.278 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.278 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.278 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:27.278 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.006634+0000", 2024-09-11T09:51:27.278 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.003568309 2024-09-11T09:51:27.278 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.278 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.278 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:27.278 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.006716+0000", 2024-09-11T09:51:27.278 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.1377000000000004e-05 2024-09-11T09:51:27.279 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.279 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.279 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:27.279 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.006772+0000", 2024-09-11T09:51:27.279 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.6439e-05 2024-09-11T09:51:27.279 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:27.279 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:27.279 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:27.279 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.279 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.279 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:59 2.4 2:2906e6a7:::rbd_header.1250f5e25b80:head [watch reconnect cookie 140424358411408 gen 2] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:27.279 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.994684+0000", 2024-09-11T09:51:27.279 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 4.2079004260000001, 2024-09-11T09:51:27.280 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0094134349999999995, 2024-09-11T09:51:27.280 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:27.280 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:27.280 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "started", 2024-09-11T09:51:27.280 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:27.280 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:51:27.280 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:51:27.280 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 59 2024-09-11T09:51:27.280 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.280 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:27.280 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.280 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:27.280 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.994684+0000", 2024-09-11T09:51:27.281 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:27.281 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.281 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.281 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:27.281 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.994684+0000", 2024-09-11T09:51:27.281 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:27.281 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.281 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.281 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:27.281 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.994686+0000", 2024-09-11T09:51:27.281 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.8159999999999998e-06 2024-09-11T09:51:27.281 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.281 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.281 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:27.282 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.994691+0000", 2024-09-11T09:51:27.282 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.5009999999999998e-06 2024-09-11T09:51:27.282 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.282 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.282 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:27.282 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.994698+0000", 2024-09-11T09:51:27.282 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.3039999999999999e-06 2024-09-11T09:51:27.282 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.282 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.282 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:27.282 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.994703+0000", 2024-09-11T09:51:27.282 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.5190000000000003e-06 2024-09-11T09:51:27.282 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.283 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.283 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:27.283 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.998678+0000", 2024-09-11T09:51:27.283 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0039752340000000002 2024-09-11T09:51:27.283 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.283 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.283 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:27.283 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.998748+0000", 2024-09-11T09:51:27.283 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.0114e-05 2024-09-11T09:51:27.283 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.283 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.283 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:27.283 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.004097+0000", 2024-09-11T09:51:27.284 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0053489469999999997 2024-09-11T09:51:27.284 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:27.284 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:27.284 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:27.284 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.284 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.284 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4694.0:33 2.4 2:3bfc818c:ns::rbd_data.1256a0c02879.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:27.284 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.084537+0000", 2024-09-11T09:51:27.284 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 4.1180469979999996, 2024-09-11T09:51:27.284 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.022075773999999999, 2024-09-11T09:51:27.284 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:27.284 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:27.284 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:27.285 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:27.285 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4694", 2024-09-11T09:51:27.285 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:51:27.285 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 33 2024-09-11T09:51:27.285 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.285 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:27.285 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.285 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:27.285 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.084537+0000", 2024-09-11T09:51:27.285 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:27.285 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.285 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.285 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:27.286 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.084537+0000", 2024-09-11T09:51:27.286 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:27.286 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.286 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.286 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:27.286 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.084538+0000", 2024-09-11T09:51:27.286 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.5000000000000001e-07 2024-09-11T09:51:27.286 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.286 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.286 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:27.286 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.095984+0000", 2024-09-11T09:51:27.286 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.011446543 2024-09-11T09:51:27.286 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.287 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.287 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:27.287 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.096014+0000", 2024-09-11T09:51:27.287 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.9745000000000001e-05 2024-09-11T09:51:27.287 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.287 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.287 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:27.287 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.096030+0000", 2024-09-11T09:51:27.287 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6152999999999999e-05 2024-09-11T09:51:27.287 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.287 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.287 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:27.287 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.096113+0000", 2024-09-11T09:51:27.288 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.2731999999999993e-05 2024-09-11T09:51:27.288 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.288 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.288 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:27.288 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.096273+0000", 2024-09-11T09:51:27.288 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00015973699999999999 2024-09-11T09:51:27.288 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.288 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.288 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:27.288 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.096394+0000", 2024-09-11T09:51:27.288 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000120883 2024-09-11T09:51:27.288 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.289 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.289 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:27.289 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.106048+0000", 2024-09-11T09:51:27.289 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0096548020000000005 2024-09-11T09:51:27.289 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.289 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.289 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:27.289 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.106531+0000", 2024-09-11T09:51:27.289 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00048264299999999999 2024-09-11T09:51:27.289 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.289 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.289 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:27.289 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.106565+0000", 2024-09-11T09:51:27.289 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.4343e-05 2024-09-11T09:51:27.290 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.290 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.290 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:27.290 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.106613+0000", 2024-09-11T09:51:27.290 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.7342999999999997e-05 2024-09-11T09:51:27.290 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:27.290 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:27.290 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:27.290 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.290 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.292 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4694.0:32 2.1 e53/15 2:8a862b48:ns::rbd_data.1256a0c02879.0000000000000000:head v 53'8, mlcod=53'6)", 2024-09-11T09:51:27.292 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.091418+0000", 2024-09-11T09:51:27.292 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 4.1111658289999999, 2024-09-11T09:51:27.292 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.013027372000000001, 2024-09-11T09:51:27.292 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:27.292 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:27.292 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:27.292 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:27.292 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.292 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:27.293 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.091418+0000", 2024-09-11T09:51:27.293 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:27.293 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.293 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.293 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:27.293 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.091418+0000", 2024-09-11T09:51:27.293 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:27.293 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.293 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.293 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:27.293 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.091420+0000", 2024-09-11T09:51:27.293 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6899999999999999e-06 2024-09-11T09:51:27.293 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.294 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.294 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:27.294 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.093086+0000", 2024-09-11T09:51:27.294 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0016662510000000001 2024-09-11T09:51:27.294 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.294 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.294 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:27.294 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.093107+0000", 2024-09-11T09:51:27.294 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.0999e-05 2024-09-11T09:51:27.294 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.294 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.294 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:27.295 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.093120+0000", 2024-09-11T09:51:27.295 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2544999999999999e-05 2024-09-11T09:51:27.295 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.295 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.295 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:27.295 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.093186+0000", 2024-09-11T09:51:27.295 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.6595999999999997e-05 2024-09-11T09:51:27.295 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.295 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.295 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:27.295 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.093208+0000", 2024-09-11T09:51:27.295 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.1624000000000002e-05 2024-09-11T09:51:27.295 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.296 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.296 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:27.296 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.104392+0000", 2024-09-11T09:51:27.296 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.011184265000000001 2024-09-11T09:51:27.296 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.296 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.296 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:27.296 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.104446+0000", 2024-09-11T09:51:27.296 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.3402000000000001e-05 2024-09-11T09:51:27.296 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:27.296 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:27.296 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:27.296 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.297 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.297 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4694.0:45 2.6 2:671fb830:ns::rbd_data.1256a0c02879.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:27.297 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.109629+0000", 2024-09-11T09:51:27.297 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 4.0929549879999998, 2024-09-11T09:51:27.297 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.036403143999999998, 2024-09-11T09:51:27.297 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:27.297 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:27.297 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:27.297 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:27.297 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4694", 2024-09-11T09:51:27.297 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:51:27.297 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 45 2024-09-11T09:51:27.297 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.298 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:27.298 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.298 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:27.298 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.109629+0000", 2024-09-11T09:51:27.298 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:27.298 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.298 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.298 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:27.298 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.109629+0000", 2024-09-11T09:51:27.298 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:27.298 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.298 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.299 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:27.299 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.109630+0000", 2024-09-11T09:51:27.299 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1200000000000001e-06 2024-09-11T09:51:27.299 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.299 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.299 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:27.299 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.121370+0000", 2024-09-11T09:51:27.299 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.011739571000000001 2024-09-11T09:51:27.299 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.299 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.299 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:27.299 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.121391+0000", 2024-09-11T09:51:27.299 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.1175e-05 2024-09-11T09:51:27.300 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.300 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.300 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:27.300 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.121404+0000", 2024-09-11T09:51:27.300 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2807e-05 2024-09-11T09:51:27.300 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.300 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.300 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:27.300 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.121498+0000", 2024-09-11T09:51:27.300 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.3770999999999994e-05 2024-09-11T09:51:27.300 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.300 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.300 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:27.301 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.121612+0000", 2024-09-11T09:51:27.301 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000114666 2024-09-11T09:51:27.301 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.301 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.301 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 3", 2024-09-11T09:51:27.301 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.121711+0000", 2024-09-11T09:51:27.301 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.8647999999999995e-05 2024-09-11T09:51:27.301 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.301 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.301 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:27.301 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.126558+0000", 2024-09-11T09:51:27.301 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0048473769999999999 2024-09-11T09:51:27.301 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.302 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.302 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:27.302 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.145912+0000", 2024-09-11T09:51:27.302 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.019354281000000001 2024-09-11T09:51:27.302 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.302 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.302 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:27.302 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.145977+0000", 2024-09-11T09:51:27.302 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.4135999999999997e-05 2024-09-11T09:51:27.302 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.302 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.302 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:27.302 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.146032+0000", 2024-09-11T09:51:27.303 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.5591999999999999e-05 2024-09-11T09:51:27.303 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:27.303 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:27.303 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:27.303 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.303 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.303 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4694.0:41 2.2 e53/15 2:5e2505c4:ns::rbd_data.1256a0c02879.0000000000000003:head v 53'4, mlcod=18'2)", 2024-09-11T09:51:27.303 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.113199+0000", 2024-09-11T09:51:27.303 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 4.0893845779999998, 2024-09-11T09:51:27.303 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.011404512, 2024-09-11T09:51:27.303 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:27.303 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:27.303 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:27.304 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:27.304 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.304 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:27.304 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.113199+0000", 2024-09-11T09:51:27.304 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:27.304 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.304 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.304 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:27.304 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.113199+0000", 2024-09-11T09:51:27.304 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:27.304 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.304 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.304 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:27.304 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.113200+0000", 2024-09-11T09:51:27.305 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.2799999999999995e-07 2024-09-11T09:51:27.305 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.305 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.305 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:27.305 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.118872+0000", 2024-09-11T09:51:27.305 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0056718089999999999 2024-09-11T09:51:27.305 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.305 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.305 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:27.305 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.118889+0000", 2024-09-11T09:51:27.305 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7428999999999999e-05 2024-09-11T09:51:27.305 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.305 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.306 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:27.306 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.118900+0000", 2024-09-11T09:51:27.306 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0243e-05 2024-09-11T09:51:27.306 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.306 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.306 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:27.306 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.118959+0000", 2024-09-11T09:51:27.306 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.9364000000000003e-05 2024-09-11T09:51:27.306 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.306 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.306 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:27.306 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.118976+0000", 2024-09-11T09:51:27.306 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7124000000000001e-05 2024-09-11T09:51:27.306 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.307 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.307 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:27.307 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.124557+0000", 2024-09-11T09:51:27.307 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0055812659999999997 2024-09-11T09:51:27.307 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.307 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.307 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:27.307 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.124604+0000", 2024-09-11T09:51:27.307 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.6548999999999998e-05 2024-09-11T09:51:27.307 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:27.307 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:27.307 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:27.308 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.308 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.308 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4694.0:43 2.0 e53/15 2:16c8ae84:ns::rbd_data.1256a0c02879.0000000000000005:head v 53'2, mlcod=0'0)", 2024-09-11T09:51:27.308 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.158559+0000", 2024-09-11T09:51:27.308 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 4.044025252, 2024-09-11T09:51:27.308 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.016713677999999999, 2024-09-11T09:51:27.308 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:27.308 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:27.308 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:27.308 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:27.308 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.308 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:27.308 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.158559+0000", 2024-09-11T09:51:27.309 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:27.309 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.309 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.309 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:27.309 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.158559+0000", 2024-09-11T09:51:27.309 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:27.309 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.309 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.309 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:27.309 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.158560+0000", 2024-09-11T09:51:27.309 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1459999999999999e-06 2024-09-11T09:51:27.309 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.309 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.310 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:27.310 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.166465+0000", 2024-09-11T09:51:27.310 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.007904767 2024-09-11T09:51:27.310 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.310 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.310 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:27.310 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.166491+0000", 2024-09-11T09:51:27.310 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.6308e-05 2024-09-11T09:51:27.310 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.310 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.310 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:27.310 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.166502+0000", 2024-09-11T09:51:27.310 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0736e-05 2024-09-11T09:51:27.311 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.311 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.311 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:27.311 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.166564+0000", 2024-09-11T09:51:27.311 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.2578999999999998e-05 2024-09-11T09:51:27.311 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.311 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.311 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:27.311 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.166585+0000", 2024-09-11T09:51:27.311 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.0395999999999999e-05 2024-09-11T09:51:27.311 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.311 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.311 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:27.312 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.175227+0000", 2024-09-11T09:51:27.312 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0086426420000000007 2024-09-11T09:51:27.312 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.312 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.312 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:27.312 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.175272+0000", 2024-09-11T09:51:27.312 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.5104e-05 2024-09-11T09:51:27.312 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:27.312 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:27.312 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:27.312 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.312 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.312 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4694.0:59 2.2 e53/15 2:41adefc9:ns::rbd_data.1256a0c02879.000000000000000e:head v 53'5, mlcod=53'3)", 2024-09-11T09:51:27.313 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.163389+0000", 2024-09-11T09:51:27.313 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 4.0391952760000001, 2024-09-11T09:51:27.313 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.013095719, 2024-09-11T09:51:27.313 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:27.313 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:27.313 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:27.313 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:27.313 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.313 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:27.313 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.163389+0000", 2024-09-11T09:51:27.313 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:27.313 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.313 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.314 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:27.314 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.163389+0000", 2024-09-11T09:51:27.314 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:27.314 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.314 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.314 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:27.314 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.163390+0000", 2024-09-11T09:51:27.314 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.336e-06 2024-09-11T09:51:27.314 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.314 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.314 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:27.314 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.170335+0000", 2024-09-11T09:51:27.314 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0069451080000000002 2024-09-11T09:51:27.314 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.315 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.315 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:27.315 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.170354+0000", 2024-09-11T09:51:27.315 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.8423e-05 2024-09-11T09:51:27.315 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.315 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.315 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:27.315 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.170366+0000", 2024-09-11T09:51:27.315 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2235999999999999e-05 2024-09-11T09:51:27.315 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.315 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.315 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:27.315 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.170439+0000", 2024-09-11T09:51:27.316 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.2989000000000002e-05 2024-09-11T09:51:27.316 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.316 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.316 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:27.316 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.170460+0000", 2024-09-11T09:51:27.316 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.1211000000000001e-05 2024-09-11T09:51:27.316 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.316 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.316 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:27.316 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.176442+0000", 2024-09-11T09:51:27.316 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0059821969999999999 2024-09-11T09:51:27.316 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.317 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.317 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:27.317 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.176484+0000", 2024-09-11T09:51:27.317 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.2218999999999997e-05 2024-09-11T09:51:27.317 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:27.317 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:27.317 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:27.317 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:27.317 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:27.317 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:51:27.318 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-11T09:51:27.318 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T09:51:27.319 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-11T09:51:27.320 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-11T09:51:27.320 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-11T09:51:27.321 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:51:27.320+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:51:27.419 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-09-11T09:51:27.420+0000 7f7b6c7d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31904) UID: 0 2024-09-11T09:51:27.467 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:51:27.467 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [], 2024-09-11T09:51:27.467 INFO:teuthology.orchestra.run.smithi017.stdout: "num_ops": 0 2024-09-11T09:51:27.467 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:51:27.476 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-11T09:51:27.520 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:51:27.521+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:51:27.582 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:51:27.582 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [], 2024-09-11T09:51:27.582 INFO:teuthology.orchestra.run.smithi017.stdout: "complaint_time": 30, 2024-09-11T09:51:27.583 INFO:teuthology.orchestra.run.smithi017.stdout: "num_blocked_ops": 0 2024-09-11T09:51:27.583 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:51:27.591 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-11T09:51:27.621 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:51:27.622+0000 7f3c96ec9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31909) UID: 0 2024-09-11T09:51:27.715 INFO:teuthology.orchestra.run.smithi017.stderr:noscrub is set 2024-09-11T09:51:27.722 INFO:tasks.ceph.osd.0.smithi017.stderr:2024-09-11T09:51:27.723+0000 7fb73c0f1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31665) UID: 0 2024-09-11T09:51:27.728 INFO:tasks.workunit.client.0.smithi107.stderr:+ expect_false rbd-nbd 2024-09-11T09:51:27.729 INFO:tasks.workunit.client.0.smithi107.stderr:+ rbd-nbd 2024-09-11T09:51:27.744 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:51:27.744 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 20, 2024-09-11T09:51:27.744 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 600, 2024-09-11T09:51:27.744 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [ 2024-09-11T09:51:27.744 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.744 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4230.0:1 2.3 e16/15 2:c4c92e5a:::rbd_trash:head v 16'1, mlcod=0'0)", 2024-09-11T09:51:27.744 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:50:18.494378+0000", 2024-09-11T09:51:27.744 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 69.249308833000001, 2024-09-11T09:51:27.744 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00089752199999999999, 2024-09-11T09:51:27.744 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:27.744 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:27.745 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:27.745 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:27.745 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.745 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:27.745 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:18.494378+0000", 2024-09-11T09:51:27.745 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:27.745 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.745 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.745 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:27.745 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:18.494378+0000", 2024-09-11T09:51:27.745 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:27.746 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.746 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.746 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:27.746 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:18.494378+0000", 2024-09-11T09:51:27.746 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.9099999999999999e-07 2024-09-11T09:51:27.746 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.746 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.746 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:27.746 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:18.494385+0000", 2024-09-11T09:51:27.746 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.8129999999999998e-06 2024-09-11T09:51:27.747 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.747 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.747 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:27.747 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:18.494416+0000", 2024-09-11T09:51:27.747 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.0654000000000002e-05 2024-09-11T09:51:27.747 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.747 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.747 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:27.747 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:18.494430+0000", 2024-09-11T09:51:27.747 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3754000000000001e-05 2024-09-11T09:51:27.747 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.747 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.747 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:27.748 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:18.494524+0000", 2024-09-11T09:51:27.748 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.4896999999999996e-05 2024-09-11T09:51:27.748 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.748 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.748 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:27.748 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:18.494558+0000", 2024-09-11T09:51:27.748 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.3900999999999999e-05 2024-09-11T09:51:27.748 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.748 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.748 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:27.748 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:18.495231+0000", 2024-09-11T09:51:27.748 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00067233500000000003 2024-09-11T09:51:27.748 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.749 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.749 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:27.749 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:50:18.495275+0000", 2024-09-11T09:51:27.749 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.4777000000000002e-05 2024-09-11T09:51:27.749 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:27.749 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:27.749 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:27.749 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.749 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.749 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:6 2.1 2:92878cd2:::rbd_id.testrbdnbd27121:head [create,call rbd.set_id in=9b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:27.749 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.873158+0000", 2024-09-11T09:51:27.749 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 4.870528363, 2024-09-11T09:51:27.750 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.001212411, 2024-09-11T09:51:27.750 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:27.750 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:27.750 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:27.750 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:27.750 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:51:27.750 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:51:27.750 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 6 2024-09-11T09:51:27.750 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.750 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:27.750 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.750 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:27.750 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.873158+0000", 2024-09-11T09:51:27.750 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:27.751 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.751 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.751 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:27.751 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.873158+0000", 2024-09-11T09:51:27.751 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:27.751 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.751 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.751 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:27.751 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.873159+0000", 2024-09-11T09:51:27.751 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.6400000000000001e-07 2024-09-11T09:51:27.751 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.751 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.751 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:27.752 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.873164+0000", 2024-09-11T09:51:27.752 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.7419999999999997e-06 2024-09-11T09:51:27.752 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.752 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.752 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:27.752 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.873175+0000", 2024-09-11T09:51:27.752 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0774999999999999e-05 2024-09-11T09:51:27.752 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.752 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.752 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:27.752 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.873182+0000", 2024-09-11T09:51:27.752 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.5240000000000003e-06 2024-09-11T09:51:27.752 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.753 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.753 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:27.753 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.873223+0000", 2024-09-11T09:51:27.753 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.0479000000000003e-05 2024-09-11T09:51:27.753 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.753 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.753 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:27.753 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.873299+0000", 2024-09-11T09:51:27.753 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.6774999999999996e-05 2024-09-11T09:51:27.753 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.753 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.753 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1", 2024-09-11T09:51:27.753 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.873412+0000", 2024-09-11T09:51:27.754 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000112489 2024-09-11T09:51:27.754 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.754 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.754 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:27.754 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.874213+0000", 2024-09-11T09:51:27.754 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00080068299999999995 2024-09-11T09:51:27.754 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.754 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.754 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:27.754 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.874311+0000", 2024-09-11T09:51:27.754 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.8301999999999995e-05 2024-09-11T09:51:27.754 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.754 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.754 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:27.755 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.874344+0000", 2024-09-11T09:51:27.755 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.3441000000000001e-05 2024-09-11T09:51:27.755 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.755 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.755 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:27.755 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.874371+0000", 2024-09-11T09:51:27.755 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.6437000000000001e-05 2024-09-11T09:51:27.755 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:27.755 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:27.755 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:27.755 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.755 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.755 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:31 2.1 2:8654721b:::rbd_data.1250f5e25b80.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:27.756 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.896024+0000", 2024-09-11T09:51:27.756 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 4.847662755, 2024-09-11T09:51:27.756 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.023148401999999998, 2024-09-11T09:51:27.756 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:27.756 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:27.756 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:27.756 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:27.756 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:51:27.756 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:51:27.756 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 31 2024-09-11T09:51:27.756 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.756 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:27.756 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.757 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:27.757 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.896024+0000", 2024-09-11T09:51:27.757 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:27.757 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.757 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.757 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:27.757 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.896024+0000", 2024-09-11T09:51:27.757 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:27.757 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.757 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.757 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:27.757 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.896025+0000", 2024-09-11T09:51:27.757 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.54e-07 2024-09-11T09:51:27.758 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.758 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.758 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:27.758 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.904287+0000", 2024-09-11T09:51:27.758 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0082624729999999993 2024-09-11T09:51:27.758 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.758 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.758 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:27.758 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.904305+0000", 2024-09-11T09:51:27.758 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7912999999999999e-05 2024-09-11T09:51:27.758 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.758 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.758 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:27.759 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.904312+0000", 2024-09-11T09:51:27.759 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.0230000000000004e-06 2024-09-11T09:51:27.759 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.759 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.759 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:27.759 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.904395+0000", 2024-09-11T09:51:27.759 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.2355999999999996e-05 2024-09-11T09:51:27.759 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.759 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.759 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:27.759 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.904499+0000", 2024-09-11T09:51:27.760 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000104672 2024-09-11T09:51:27.760 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.760 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.760 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1", 2024-09-11T09:51:27.760 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.904594+0000", 2024-09-11T09:51:27.760 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.5129000000000003e-05 2024-09-11T09:51:27.760 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.760 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.760 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:27.760 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.918183+0000", 2024-09-11T09:51:27.760 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.013588355999999999 2024-09-11T09:51:27.760 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.760 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.761 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:27.761 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919053+0000", 2024-09-11T09:51:27.761 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00087066800000000001 2024-09-11T09:51:27.761 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.761 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.761 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:27.761 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919125+0000", 2024-09-11T09:51:27.761 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.1333000000000003e-05 2024-09-11T09:51:27.761 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.761 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.761 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:27.761 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919172+0000", 2024-09-11T09:51:27.762 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.7725000000000001e-05 2024-09-11T09:51:27.762 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:27.762 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:27.762 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:27.762 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.762 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.762 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4688.0:42 2.3 e53/15 2:c54742d8:::rbd_data.1250f5e25b80.0000000000000006:head v 53'2, mlcod=0'0)", 2024-09-11T09:51:27.762 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.939272+0000", 2024-09-11T09:51:27.762 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 4.8044147710000003, 2024-09-11T09:51:27.762 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.010821958, 2024-09-11T09:51:27.762 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:27.762 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:27.762 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:27.763 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:27.763 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.763 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:27.763 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.939272+0000", 2024-09-11T09:51:27.763 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:27.763 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.763 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.763 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:27.763 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.939272+0000", 2024-09-11T09:51:27.763 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:27.763 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.763 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.763 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:27.764 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.939273+0000", 2024-09-11T09:51:27.764 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.7899999999999997e-07 2024-09-11T09:51:27.764 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.764 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.764 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:27.764 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.944403+0000", 2024-09-11T09:51:27.764 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0051302129999999998 2024-09-11T09:51:27.764 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.764 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.764 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:27.765 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.944413+0000", 2024-09-11T09:51:27.765 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0182e-05 2024-09-11T09:51:27.765 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.765 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.765 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:27.765 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.944420+0000", 2024-09-11T09:51:27.765 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.7000000000000002e-06 2024-09-11T09:51:27.765 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.765 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.765 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:27.766 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.944482+0000", 2024-09-11T09:51:27.766 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.1911999999999999e-05 2024-09-11T09:51:27.766 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.766 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.766 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:27.766 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.944508+0000", 2024-09-11T09:51:27.766 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.6582000000000001e-05 2024-09-11T09:51:27.766 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.766 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.766 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:27.766 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.950050+0000", 2024-09-11T09:51:27.766 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0055414080000000003 2024-09-11T09:51:27.766 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.767 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.767 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:27.767 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.950094+0000", 2024-09-11T09:51:27.767 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.4082000000000002e-05 2024-09-11T09:51:27.767 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:27.767 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:27.767 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:27.767 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.767 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.767 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:47 2.1 2:95f53b3e:::rbd_data.1250f5e25b80.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:27.767 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.948330+0000", 2024-09-11T09:51:27.767 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 4.7953564269999998, 2024-09-11T09:51:27.768 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.025156603999999999, 2024-09-11T09:51:27.768 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:27.768 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:27.768 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:27.768 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:27.768 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:51:27.768 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:51:27.768 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 47 2024-09-11T09:51:27.768 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.768 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:27.768 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.768 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:27.768 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.948330+0000", 2024-09-11T09:51:27.769 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:27.769 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.769 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.769 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:27.769 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.948330+0000", 2024-09-11T09:51:27.769 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:27.769 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.769 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.769 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:27.769 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.948333+0000", 2024-09-11T09:51:27.769 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.481e-06 2024-09-11T09:51:27.769 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.769 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.770 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:27.770 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.960263+0000", 2024-09-11T09:51:27.770 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.011930671 2024-09-11T09:51:27.770 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.770 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.770 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:27.770 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.960289+0000", 2024-09-11T09:51:27.770 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.5573000000000001e-05 2024-09-11T09:51:27.770 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.770 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.770 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:27.770 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.960304+0000", 2024-09-11T09:51:27.770 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.4647000000000001e-05 2024-09-11T09:51:27.770 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.771 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.771 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:27.771 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.960390+0000", 2024-09-11T09:51:27.771 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.6088999999999995e-05 2024-09-11T09:51:27.771 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.771 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.771 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:27.771 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.960562+0000", 2024-09-11T09:51:27.771 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00017237400000000001 2024-09-11T09:51:27.771 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.771 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.771 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1", 2024-09-11T09:51:27.772 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.960725+0000", 2024-09-11T09:51:27.772 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00016321599999999999 2024-09-11T09:51:27.772 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.772 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.772 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:27.772 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.969465+0000", 2024-09-11T09:51:27.772 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0087396899999999996 2024-09-11T09:51:27.772 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.772 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.772 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:27.772 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.973393+0000", 2024-09-11T09:51:27.772 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0039274669999999996 2024-09-11T09:51:27.772 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.773 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.773 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:27.773 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.973443+0000", 2024-09-11T09:51:27.773 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.0340000000000003e-05 2024-09-11T09:51:27.773 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.773 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.773 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:27.773 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.973487+0000", 2024-09-11T09:51:27.773 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.4056e-05 2024-09-11T09:51:27.773 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:27.773 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:27.773 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:27.773 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.774 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.774 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:61 2.1 2:9af038b1:::rbd_data.1250f5e25b80.000000000000000f:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:27.774 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.975764+0000", 2024-09-11T09:51:27.774 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 4.7679229029999997, 2024-09-11T09:51:27.774 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.016158891000000002, 2024-09-11T09:51:27.774 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:27.774 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:27.774 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:27.774 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:27.774 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:51:27.774 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:51:27.774 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 61 2024-09-11T09:51:27.774 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.775 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:27.775 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.775 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:27.775 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.975764+0000", 2024-09-11T09:51:27.775 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:27.775 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.775 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.775 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:27.775 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.975764+0000", 2024-09-11T09:51:27.775 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:27.775 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.775 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.775 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:27.776 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.975766+0000", 2024-09-11T09:51:27.776 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.767e-06 2024-09-11T09:51:27.776 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.776 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.776 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:27.776 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.983204+0000", 2024-09-11T09:51:27.776 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0074388400000000004 2024-09-11T09:51:27.776 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.776 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.776 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:27.776 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.983226+0000", 2024-09-11T09:51:27.776 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.1279000000000001e-05 2024-09-11T09:51:27.776 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.777 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.777 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:27.777 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.983238+0000", 2024-09-11T09:51:27.777 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2563e-05 2024-09-11T09:51:27.777 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.777 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.777 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:27.777 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.983318+0000", 2024-09-11T09:51:27.777 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.0075999999999997e-05 2024-09-11T09:51:27.777 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.777 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.777 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:27.777 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.983429+0000", 2024-09-11T09:51:27.778 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000110618 2024-09-11T09:51:27.778 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.778 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.778 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1", 2024-09-11T09:51:27.778 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.983525+0000", 2024-09-11T09:51:27.778 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.603e-05 2024-09-11T09:51:27.778 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.778 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.778 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:27.778 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.989004+0000", 2024-09-11T09:51:27.778 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0054785429999999998 2024-09-11T09:51:27.778 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.779 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.779 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:27.779 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.991788+0000", 2024-09-11T09:51:27.779 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0027845729999999998 2024-09-11T09:51:27.779 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.779 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.779 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:27.779 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.991870+0000", 2024-09-11T09:51:27.779 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.1567e-05 2024-09-11T09:51:27.779 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.779 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.779 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:27.779 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.991923+0000", 2024-09-11T09:51:27.779 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.3035000000000003e-05 2024-09-11T09:51:27.780 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:27.780 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:27.780 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:27.780 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.780 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.780 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4694.0:9 2.1 2:839846fa:ns::rbd_header.1256a0c02879:head [create,call rbd.create in=9b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:27.780 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.062019+0000", 2024-09-11T09:51:27.780 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 4.6816681429999996, 2024-09-11T09:51:27.780 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.001293296, 2024-09-11T09:51:27.780 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:27.780 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:27.780 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:27.780 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:27.781 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4694", 2024-09-11T09:51:27.781 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:51:27.781 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 9 2024-09-11T09:51:27.781 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.781 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:27.781 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.781 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:27.781 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.062019+0000", 2024-09-11T09:51:27.781 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:27.781 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.781 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.781 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:27.781 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.062019+0000", 2024-09-11T09:51:27.782 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:27.782 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.782 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.782 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:27.782 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.062020+0000", 2024-09-11T09:51:27.782 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2610000000000001e-06 2024-09-11T09:51:27.782 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.782 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.782 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:27.782 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.062025+0000", 2024-09-11T09:51:27.782 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.7879999999999997e-06 2024-09-11T09:51:27.782 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.782 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.783 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:27.783 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.062033+0000", 2024-09-11T09:51:27.783 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.9489999999999996e-06 2024-09-11T09:51:27.783 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.783 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.783 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:27.783 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.062039+0000", 2024-09-11T09:51:27.783 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.781e-06 2024-09-11T09:51:27.783 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.783 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.783 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:27.783 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.062099+0000", 2024-09-11T09:51:27.783 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.9905999999999999e-05 2024-09-11T09:51:27.784 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.784 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.784 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:27.784 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.062247+0000", 2024-09-11T09:51:27.784 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000147472 2024-09-11T09:51:27.784 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.784 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.784 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1", 2024-09-11T09:51:27.784 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.062468+0000", 2024-09-11T09:51:27.784 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00022112999999999999 2024-09-11T09:51:27.784 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.784 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.785 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:27.785 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.063076+0000", 2024-09-11T09:51:27.785 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00060857100000000003 2024-09-11T09:51:27.785 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.785 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.785 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:27.785 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.063251+0000", 2024-09-11T09:51:27.785 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00017496499999999999 2024-09-11T09:51:27.785 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.785 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.785 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:27.785 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.063285+0000", 2024-09-11T09:51:27.785 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.3633999999999997e-05 2024-09-11T09:51:27.785 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.786 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.786 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:27.786 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.063312+0000", 2024-09-11T09:51:27.786 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.6838999999999998e-05 2024-09-11T09:51:27.786 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:27.786 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:27.786 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:27.786 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.786 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.786 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4694.0:17 2.1 2:839846fa:ns::rbd_header.1256a0c02879:head [call rbd.get_size in=11b,call rbd.get_features in=15b,call rbd.get_flags in=12b,call,call lock.get_info in=12b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:27.786 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.070382+0000", 2024-09-11T09:51:27.786 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 4.6733049019999999, 2024-09-11T09:51:27.786 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00036771800000000001, 2024-09-11T09:51:27.787 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:27.787 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:27.787 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "started", 2024-09-11T09:51:27.787 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:27.787 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4694", 2024-09-11T09:51:27.787 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:51:27.787 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 17 2024-09-11T09:51:27.787 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.787 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:27.787 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.787 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:27.787 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.070382+0000", 2024-09-11T09:51:27.787 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:27.788 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.788 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.788 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:27.788 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.070382+0000", 2024-09-11T09:51:27.788 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:27.788 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.788 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.788 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:27.788 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.070383+0000", 2024-09-11T09:51:27.788 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.5000000000000002e-07 2024-09-11T09:51:27.788 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.788 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.788 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:27.789 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.070387+0000", 2024-09-11T09:51:27.789 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.0609999999999997e-06 2024-09-11T09:51:27.789 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.789 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.789 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:27.789 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.070397+0000", 2024-09-11T09:51:27.789 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0736e-05 2024-09-11T09:51:27.789 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.789 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.789 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:27.789 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.070405+0000", 2024-09-11T09:51:27.789 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.7950000000000008e-06 2024-09-11T09:51:27.789 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.790 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.790 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:27.790 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.070472+0000", 2024-09-11T09:51:27.790 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.6643999999999994e-05 2024-09-11T09:51:27.790 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.790 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.790 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:27.790 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.070525+0000", 2024-09-11T09:51:27.790 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.3341999999999999e-05 2024-09-11T09:51:27.790 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.790 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.790 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:27.790 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.070750+0000", 2024-09-11T09:51:27.791 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00022439000000000001 2024-09-11T09:51:27.791 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:27.791 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:27.791 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:27.791 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.791 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.791 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4694.0:22 2.1 2:839846fa:ns::rbd_header.1256a0c02879:head [watch watch cookie 139806419992736] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:27.791 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.073210+0000", 2024-09-11T09:51:27.791 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 4.6704765649999995, 2024-09-11T09:51:27.791 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00098618800000000004, 2024-09-11T09:51:27.791 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:27.791 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:27.791 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:27.791 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:27.792 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4694", 2024-09-11T09:51:27.792 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:51:27.792 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 22 2024-09-11T09:51:27.792 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.792 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:27.792 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.792 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:27.792 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.073210+0000", 2024-09-11T09:51:27.792 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:27.792 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.792 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.792 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:27.792 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.073210+0000", 2024-09-11T09:51:27.793 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:27.793 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.793 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.793 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:27.793 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.073211+0000", 2024-09-11T09:51:27.793 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.54e-07 2024-09-11T09:51:27.793 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.793 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.793 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:27.793 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.073216+0000", 2024-09-11T09:51:27.793 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.7840000000000003e-06 2024-09-11T09:51:27.793 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.793 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.794 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:27.794 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.073227+0000", 2024-09-11T09:51:27.794 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1032e-05 2024-09-11T09:51:27.794 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.794 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.794 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:27.794 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.073232+0000", 2024-09-11T09:51:27.794 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.9069999999999998e-06 2024-09-11T09:51:27.794 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.794 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.794 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:27.794 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.073297+0000", 2024-09-11T09:51:27.794 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.5691999999999994e-05 2024-09-11T09:51:27.795 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.795 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.795 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:27.795 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.073344+0000", 2024-09-11T09:51:27.795 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.6366000000000003e-05 2024-09-11T09:51:27.795 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.795 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.795 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1", 2024-09-11T09:51:27.795 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.073424+0000", 2024-09-11T09:51:27.795 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.9974000000000004e-05 2024-09-11T09:51:27.795 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.795 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.795 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:27.795 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.073909+0000", 2024-09-11T09:51:27.796 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00048529500000000001 2024-09-11T09:51:27.796 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.796 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.796 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:27.796 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.074131+0000", 2024-09-11T09:51:27.796 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00022189200000000001 2024-09-11T09:51:27.796 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.796 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.796 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:27.796 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.074160+0000", 2024-09-11T09:51:27.796 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.9577000000000002e-05 2024-09-11T09:51:27.796 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.796 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.796 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:27.797 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.074196+0000", 2024-09-11T09:51:27.797 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.5914999999999997e-05 2024-09-11T09:51:27.797 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:27.797 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:27.797 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:27.797 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.797 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.797 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4694.0:24 2.1 2:839846fa:ns::rbd_header.1256a0c02879:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:27.797 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.078154+0000", 2024-09-11T09:51:27.797 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 4.6655323050000002, 2024-09-11T09:51:27.797 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00091518800000000005, 2024-09-11T09:51:27.797 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:27.798 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:27.798 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:27.798 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:27.798 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4694", 2024-09-11T09:51:27.798 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:51:27.798 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 24 2024-09-11T09:51:27.798 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.798 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:27.798 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.798 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:27.798 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.078154+0000", 2024-09-11T09:51:27.798 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:27.798 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.799 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.799 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:27.799 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.078154+0000", 2024-09-11T09:51:27.799 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:27.799 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.799 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.799 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:27.799 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.078155+0000", 2024-09-11T09:51:27.799 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.4300000000000003e-07 2024-09-11T09:51:27.799 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.799 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.799 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:27.799 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.078158+0000", 2024-09-11T09:51:27.800 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.8789999999999999e-06 2024-09-11T09:51:27.800 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.800 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.800 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:27.800 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.078166+0000", 2024-09-11T09:51:27.800 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.8939999999999993e-06 2024-09-11T09:51:27.800 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.800 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.800 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:27.800 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.078171+0000", 2024-09-11T09:51:27.800 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.7570000000000004e-06 2024-09-11T09:51:27.800 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.800 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.801 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:27.801 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.078201+0000", 2024-09-11T09:51:27.801 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.0502999999999999e-05 2024-09-11T09:51:27.801 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.801 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.801 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:27.801 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.078244+0000", 2024-09-11T09:51:27.801 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.2738000000000003e-05 2024-09-11T09:51:27.801 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.801 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.801 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1", 2024-09-11T09:51:27.801 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.078344+0000", 2024-09-11T09:51:27.801 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.9998e-05 2024-09-11T09:51:27.802 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.802 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.802 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:27.802 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.078718+0000", 2024-09-11T09:51:27.802 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00037423799999999999 2024-09-11T09:51:27.802 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.802 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.802 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:27.802 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.079018+0000", 2024-09-11T09:51:27.802 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00029968699999999998 2024-09-11T09:51:27.802 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.802 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.802 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:27.802 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.079048+0000", 2024-09-11T09:51:27.803 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.0069999999999998e-05 2024-09-11T09:51:27.803 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.803 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.803 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:27.803 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.079070+0000", 2024-09-11T09:51:27.803 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.1880999999999999e-05 2024-09-11T09:51:27.803 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:27.803 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:27.803 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:27.803 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.803 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.803 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4694.0:32 2.1 2:8a862b48:ns::rbd_data.1256a0c02879.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:27.803 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.084089+0000", 2024-09-11T09:51:27.804 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 4.6595974289999997, 2024-09-11T09:51:27.804 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.020630553999999999, 2024-09-11T09:51:27.804 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:27.804 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:27.804 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:27.804 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:27.804 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4694", 2024-09-11T09:51:27.804 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:51:27.804 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 32 2024-09-11T09:51:27.804 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.804 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:27.804 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.804 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:27.805 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.084089+0000", 2024-09-11T09:51:27.805 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:27.805 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.805 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.805 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:27.805 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.084089+0000", 2024-09-11T09:51:27.805 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:27.805 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.805 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.805 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:27.805 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.084090+0000", 2024-09-11T09:51:27.805 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.2900000000000003e-07 2024-09-11T09:51:27.805 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.806 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.806 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:27.806 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.090866+0000", 2024-09-11T09:51:27.806 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0067756609999999997 2024-09-11T09:51:27.806 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.806 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.806 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:27.806 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.090891+0000", 2024-09-11T09:51:27.806 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.5119000000000001e-05 2024-09-11T09:51:27.806 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.806 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.806 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:27.806 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.090900+0000", 2024-09-11T09:51:27.807 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.4110000000000002e-06 2024-09-11T09:51:27.807 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.807 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.807 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:27.807 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.090968+0000", 2024-09-11T09:51:27.807 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.8041999999999997e-05 2024-09-11T09:51:27.807 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.807 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.807 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:27.807 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.091115+0000", 2024-09-11T09:51:27.807 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00014633499999999999 2024-09-11T09:51:27.807 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.807 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.807 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1", 2024-09-11T09:51:27.808 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.091284+0000", 2024-09-11T09:51:27.808 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000169836 2024-09-11T09:51:27.808 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.808 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.808 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:27.808 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.100896+0000", 2024-09-11T09:51:27.808 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0096118639999999995 2024-09-11T09:51:27.808 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.808 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.808 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:27.808 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.104604+0000", 2024-09-11T09:51:27.808 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0037074619999999999 2024-09-11T09:51:27.809 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.809 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.809 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:27.809 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.104669+0000", 2024-09-11T09:51:27.809 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.4836000000000001e-05 2024-09-11T09:51:27.809 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.809 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.809 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:27.809 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.104720+0000", 2024-09-11T09:51:27.809 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.1258999999999998e-05 2024-09-11T09:51:27.809 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:27.809 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:27.809 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:27.810 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.810 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.810 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4694.0:44 2.3 e53/15 2:cd1a7d1c:ns::rbd_data.1256a0c02879.0000000000000006:head v 53'3, mlcod=16'1)", 2024-09-11T09:51:27.810 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.120387+0000", 2024-09-11T09:51:27.810 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 4.6232997759999996, 2024-09-11T09:51:27.810 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.011988468, 2024-09-11T09:51:27.810 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:27.810 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:27.810 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:27.810 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:27.810 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.810 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:27.810 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.120387+0000", 2024-09-11T09:51:27.811 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:27.811 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.811 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.811 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:27.811 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.120387+0000", 2024-09-11T09:51:27.811 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:27.811 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.811 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.811 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:27.811 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.120389+0000", 2024-09-11T09:51:27.811 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.018e-06 2024-09-11T09:51:27.811 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.811 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.812 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:27.812 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.127403+0000", 2024-09-11T09:51:27.812 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0070144769999999999 2024-09-11T09:51:27.812 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.812 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.812 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:27.812 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.127414+0000", 2024-09-11T09:51:27.812 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0991e-05 2024-09-11T09:51:27.812 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.812 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.812 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:27.812 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.127423+0000", 2024-09-11T09:51:27.812 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.8580000000000006e-06 2024-09-11T09:51:27.812 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.813 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.813 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:27.813 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.127468+0000", 2024-09-11T09:51:27.813 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.4830999999999998e-05 2024-09-11T09:51:27.813 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.813 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.813 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:27.813 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.127485+0000", 2024-09-11T09:51:27.813 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6838999999999999e-05 2024-09-11T09:51:27.813 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.813 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.813 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:27.813 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.132341+0000", 2024-09-11T09:51:27.814 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0048557959999999999 2024-09-11T09:51:27.814 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.814 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.814 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:27.814 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.132375+0000", 2024-09-11T09:51:27.814 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.4657999999999999e-05 2024-09-11T09:51:27.814 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:27.814 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:27.814 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:27.814 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.814 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.814 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4694.0:46 2.3 e53/15 2:ce0e1469:ns::rbd_data.1256a0c02879.0000000000000008:head v 53'4, mlcod=16'1)", 2024-09-11T09:51:27.814 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.129008+0000", 2024-09-11T09:51:27.814 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 4.6146791900000004, 2024-09-11T09:51:27.815 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0087151220000000005, 2024-09-11T09:51:27.815 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:27.815 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:27.815 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:27.815 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:27.815 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.815 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:27.815 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.129008+0000", 2024-09-11T09:51:27.815 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:27.815 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.815 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.815 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:27.815 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.129008+0000", 2024-09-11T09:51:27.816 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:27.816 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.816 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.816 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:27.816 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.129009+0000", 2024-09-11T09:51:27.816 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.367e-06 2024-09-11T09:51:27.816 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.816 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.816 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:27.816 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.132839+0000", 2024-09-11T09:51:27.816 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0038297610000000001 2024-09-11T09:51:27.816 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.816 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.817 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:27.817 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.132849+0000", 2024-09-11T09:51:27.817 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0672e-05 2024-09-11T09:51:27.817 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.817 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.817 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:27.817 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.132856+0000", 2024-09-11T09:51:27.817 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.1380000000000004e-06 2024-09-11T09:51:27.817 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.817 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.817 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:27.817 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.132891+0000", 2024-09-11T09:51:27.817 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.5712e-05 2024-09-11T09:51:27.817 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.818 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.818 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:27.818 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.132907+0000", 2024-09-11T09:51:27.818 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6166000000000001e-05 2024-09-11T09:51:27.818 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.818 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.818 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:27.818 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.137681+0000", 2024-09-11T09:51:27.818 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0047735299999999998 2024-09-11T09:51:27.818 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.818 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.818 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:27.818 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.137723+0000", 2024-09-11T09:51:27.819 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.1776000000000001e-05 2024-09-11T09:51:27.819 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:27.819 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:27.819 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:27.819 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.819 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.819 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4694.0:48 2.3 e53/15 2:d2090b70:ns::rbd_data.1256a0c02879.0000000000000009:head v 53'5, mlcod=53'2)", 2024-09-11T09:51:27.819 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.135108+0000", 2024-09-11T09:51:27.819 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 4.6085782929999999, 2024-09-11T09:51:27.819 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.010710679000000001, 2024-09-11T09:51:27.819 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:27.819 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:27.819 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:27.820 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:27.820 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.820 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:27.820 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.135108+0000", 2024-09-11T09:51:27.820 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:27.820 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.820 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.820 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:27.820 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.135108+0000", 2024-09-11T09:51:27.820 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:27.820 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.820 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.820 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:27.821 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.135110+0000", 2024-09-11T09:51:27.821 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.122e-06 2024-09-11T09:51:27.821 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.821 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.821 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:27.821 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.140876+0000", 2024-09-11T09:51:27.821 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.005766162 2024-09-11T09:51:27.821 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.821 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.821 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:27.821 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.140888+0000", 2024-09-11T09:51:27.821 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.201e-05 2024-09-11T09:51:27.822 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.822 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.822 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:27.822 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.140894+0000", 2024-09-11T09:51:27.822 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.1739999999999997e-06 2024-09-11T09:51:27.822 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.822 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.822 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:27.822 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.140971+0000", 2024-09-11T09:51:27.822 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.7495000000000003e-05 2024-09-11T09:51:27.822 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.822 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.822 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:27.822 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.140995+0000", 2024-09-11T09:51:27.823 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.3204999999999999e-05 2024-09-11T09:51:27.823 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.823 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.823 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:27.823 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.145757+0000", 2024-09-11T09:51:27.823 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0047627190000000003 2024-09-11T09:51:27.823 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.823 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.823 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:27.823 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.145819+0000", 2024-09-11T09:51:27.823 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.1791999999999994e-05 2024-09-11T09:51:27.823 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:27.823 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:27.824 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:27.824 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.824 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.824 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4694.0:50 2.1 2:83078786:ns::rbd_data.1256a0c02879.000000000000000a:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:27.824 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.141467+0000", 2024-09-11T09:51:27.824 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 4.6022195689999998, 2024-09-11T09:51:27.824 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.012785889, 2024-09-11T09:51:27.824 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:27.824 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:27.824 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:27.824 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:27.824 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4694", 2024-09-11T09:51:27.825 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:51:27.825 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 50 2024-09-11T09:51:27.825 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.825 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:27.825 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.825 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:27.825 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.141467+0000", 2024-09-11T09:51:27.825 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:27.825 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.825 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.825 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:27.825 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.141467+0000", 2024-09-11T09:51:27.825 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:27.826 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.826 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.826 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:27.826 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.141468+0000", 2024-09-11T09:51:27.826 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1009999999999999e-06 2024-09-11T09:51:27.826 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.826 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.826 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:27.826 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.146432+0000", 2024-09-11T09:51:27.826 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0049640539999999999 2024-09-11T09:51:27.826 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.826 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.826 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:27.826 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.146446+0000", 2024-09-11T09:51:27.827 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3254e-05 2024-09-11T09:51:27.827 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.827 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.827 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:27.827 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.146452+0000", 2024-09-11T09:51:27.827 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.207e-06 2024-09-11T09:51:27.827 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.827 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.827 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:27.827 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.146510+0000", 2024-09-11T09:51:27.827 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.8322999999999997e-05 2024-09-11T09:51:27.827 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.827 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.828 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:27.828 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.146650+0000", 2024-09-11T09:51:27.828 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00013947100000000001 2024-09-11T09:51:27.828 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.828 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.828 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1", 2024-09-11T09:51:27.828 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.146779+0000", 2024-09-11T09:51:27.828 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000129059 2024-09-11T09:51:27.828 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.828 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.828 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:27.828 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.153778+0000", 2024-09-11T09:51:27.828 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0069997130000000003 2024-09-11T09:51:27.829 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.829 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.829 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:27.829 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.154180+0000", 2024-09-11T09:51:27.829 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00040159800000000003 2024-09-11T09:51:27.829 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.829 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.829 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:27.829 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.154218+0000", 2024-09-11T09:51:27.829 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.7795e-05 2024-09-11T09:51:27.829 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.829 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.829 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:27.830 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.154253+0000", 2024-09-11T09:51:27.830 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.5314000000000001e-05 2024-09-11T09:51:27.830 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:27.830 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:27.830 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:27.830 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.830 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.830 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4694.0:54 2.3 e53/15 2:c44f0253:ns::rbd_data.1256a0c02879.000000000000000b:head v 53'6, mlcod=53'4)", 2024-09-11T09:51:27.830 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.154926+0000", 2024-09-11T09:51:27.830 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 4.5887611939999999, 2024-09-11T09:51:27.830 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0089497099999999996, 2024-09-11T09:51:27.830 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:27.830 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:27.830 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:27.831 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:27.831 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.831 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:27.831 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.154926+0000", 2024-09-11T09:51:27.831 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:27.831 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.831 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.831 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:27.831 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.154926+0000", 2024-09-11T09:51:27.831 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:27.831 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.831 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.832 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:27.832 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.154927+0000", 2024-09-11T09:51:27.832 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.7000000000000003e-07 2024-09-11T09:51:27.832 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.832 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.832 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:27.832 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.158712+0000", 2024-09-11T09:51:27.832 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0037855469999999998 2024-09-11T09:51:27.832 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.832 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.832 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:27.832 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.158726+0000", 2024-09-11T09:51:27.833 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3621000000000001e-05 2024-09-11T09:51:27.833 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.833 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.833 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:27.833 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.158732+0000", 2024-09-11T09:51:27.833 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.9089999999999997e-06 2024-09-11T09:51:27.833 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.833 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.833 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:27.833 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.158781+0000", 2024-09-11T09:51:27.833 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.8900000000000003e-05 2024-09-11T09:51:27.833 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.833 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.834 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:27.834 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.158805+0000", 2024-09-11T09:51:27.834 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.44e-05 2024-09-11T09:51:27.834 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.834 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.834 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:27.834 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.163826+0000", 2024-09-11T09:51:27.834 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0050206399999999998 2024-09-11T09:51:27.834 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.834 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.834 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:27.834 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.163875+0000", 2024-09-11T09:51:27.834 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.9722999999999998e-05 2024-09-11T09:51:27.835 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:27.835 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:27.835 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:27.835 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.835 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.835 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4694.0:56 2.3 e53/15 2:c462f029:ns::rbd_data.1256a0c02879.000000000000000c:head v 53'7, mlcod=53'4)", 2024-09-11T09:51:27.835 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.159549+0000", 2024-09-11T09:51:27.835 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 4.5841374879999996, 2024-09-11T09:51:27.835 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.015591666000000001, 2024-09-11T09:51:27.835 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:27.835 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:27.835 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:27.835 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:27.836 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.836 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:27.836 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.159549+0000", 2024-09-11T09:51:27.836 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:27.836 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.836 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.836 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:27.836 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.159549+0000", 2024-09-11T09:51:27.836 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:27.836 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.836 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.836 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:27.837 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.159550+0000", 2024-09-11T09:51:27.837 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.1500000000000004e-07 2024-09-11T09:51:27.837 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.837 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.837 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:27.837 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.167834+0000", 2024-09-11T09:51:27.837 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0082840220000000003 2024-09-11T09:51:27.837 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.837 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.837 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:27.837 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.167857+0000", 2024-09-11T09:51:27.837 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.3337e-05 2024-09-11T09:51:27.837 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.838 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.838 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:27.838 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.167869+0000", 2024-09-11T09:51:27.838 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1556000000000001e-05 2024-09-11T09:51:27.838 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.838 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.838 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:27.838 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.167939+0000", 2024-09-11T09:51:27.838 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.0674999999999997e-05 2024-09-11T09:51:27.838 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.838 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.838 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:27.838 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.167967+0000", 2024-09-11T09:51:27.839 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.7322999999999999e-05 2024-09-11T09:51:27.839 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.839 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.839 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:27.839 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.175079+0000", 2024-09-11T09:51:27.839 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0071126150000000001 2024-09-11T09:51:27.839 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.839 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.839 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:27.839 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.175141+0000", 2024-09-11T09:51:27.839 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.1523e-05 2024-09-11T09:51:27.839 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:27.839 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:27.839 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:27.840 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.840 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.840 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4694.0:62 2.1 2:839846fa:ns::rbd_header.1256a0c02879:head [call lock.unlock in=10b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:27.840 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.202300+0000", 2024-09-11T09:51:27.840 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 4.5413870200000002, 2024-09-11T09:51:27.840 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.001036546, 2024-09-11T09:51:27.840 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:27.840 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:27.840 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:27.840 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:27.840 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4694", 2024-09-11T09:51:27.840 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:51:27.840 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 62 2024-09-11T09:51:27.840 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.841 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:27.841 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.841 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:27.841 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.202300+0000", 2024-09-11T09:51:27.841 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:27.841 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.841 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.841 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:27.841 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.202300+0000", 2024-09-11T09:51:27.841 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:27.841 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.841 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.841 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:27.842 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.202301+0000", 2024-09-11T09:51:27.842 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.454e-06 2024-09-11T09:51:27.842 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.842 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.842 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:27.842 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.202309+0000", 2024-09-11T09:51:27.842 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.7409999999999992e-06 2024-09-11T09:51:27.842 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.842 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.842 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:27.842 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.202326+0000", 2024-09-11T09:51:27.842 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6657e-05 2024-09-11T09:51:27.842 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.843 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.843 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:27.843 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.202333+0000", 2024-09-11T09:51:27.843 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.1400000000000002e-06 2024-09-11T09:51:27.843 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.843 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.843 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:27.843 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.202386+0000", 2024-09-11T09:51:27.843 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.3638999999999999e-05 2024-09-11T09:51:27.843 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.843 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.843 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:27.843 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.202445+0000", 2024-09-11T09:51:27.844 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.8121000000000001e-05 2024-09-11T09:51:27.844 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.844 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.844 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1", 2024-09-11T09:51:27.844 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.202552+0000", 2024-09-11T09:51:27.844 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000107067 2024-09-11T09:51:27.844 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.844 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.844 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:27.844 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.203012+0000", 2024-09-11T09:51:27.844 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00046002100000000002 2024-09-11T09:51:27.844 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.844 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.845 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:27.845 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.203280+0000", 2024-09-11T09:51:27.845 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00026879300000000003 2024-09-11T09:51:27.845 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.845 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.845 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:27.845 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.203311+0000", 2024-09-11T09:51:27.845 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.1013000000000003e-05 2024-09-11T09:51:27.845 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.845 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.845 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:27.845 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.203336+0000", 2024-09-11T09:51:27.845 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.4899999999999999e-05 2024-09-11T09:51:27.845 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:27.846 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:27.846 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:27.846 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.846 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.846 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4694.0:64 2.1 2:839846fa:ns::rbd_header.1256a0c02879:head [watch unwatch cookie 139806419992736] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:27.846 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.203588+0000", 2024-09-11T09:51:27.846 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 4.5400990480000001, 2024-09-11T09:51:27.846 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0011675959999999999, 2024-09-11T09:51:27.846 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:27.846 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:27.846 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:27.846 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:27.847 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4694", 2024-09-11T09:51:27.847 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:51:27.847 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 64 2024-09-11T09:51:27.847 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.847 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:27.847 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.847 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:27.847 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.203588+0000", 2024-09-11T09:51:27.847 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:27.847 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.847 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.847 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:27.848 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.203588+0000", 2024-09-11T09:51:27.848 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:27.848 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.848 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.848 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:27.848 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.203588+0000", 2024-09-11T09:51:27.848 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.6600000000000002e-07 2024-09-11T09:51:27.848 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.848 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.848 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:27.848 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.203590+0000", 2024-09-11T09:51:27.848 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.773e-06 2024-09-11T09:51:27.848 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.849 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.849 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:27.849 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.203594+0000", 2024-09-11T09:51:27.849 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.6129999999999997e-06 2024-09-11T09:51:27.849 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.849 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.849 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:27.849 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.203598+0000", 2024-09-11T09:51:27.849 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.0809999999999998e-06 2024-09-11T09:51:27.849 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.849 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.849 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:27.850 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.203853+0000", 2024-09-11T09:51:27.850 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00025543300000000001 2024-09-11T09:51:27.850 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.850 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.850 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:27.850 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.203898+0000", 2024-09-11T09:51:27.850 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.4771000000000002e-05 2024-09-11T09:51:27.850 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.850 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.850 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1", 2024-09-11T09:51:27.850 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.203974+0000", 2024-09-11T09:51:27.850 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.6062000000000005e-05 2024-09-11T09:51:27.850 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.851 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.851 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:27.851 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.204452+0000", 2024-09-11T09:51:27.851 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00047848399999999999 2024-09-11T09:51:27.851 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.851 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.851 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:27.851 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.204667+0000", 2024-09-11T09:51:27.851 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00021479600000000001 2024-09-11T09:51:27.851 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.851 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.851 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:27.851 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.204699+0000", 2024-09-11T09:51:27.852 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.1690000000000003e-05 2024-09-11T09:51:27.852 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.852 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.852 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:27.852 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.204755+0000", 2024-09-11T09:51:27.852 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.6527000000000003e-05 2024-09-11T09:51:27.852 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:27.852 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:27.852 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:27.852 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.852 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.852 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4694.0:65 2.1 2:839846fa:ns::rbd_header.1256a0c02879:head [notify-ack] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:27.852 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.204024+0000", 2024-09-11T09:51:27.853 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 4.5396622860000004, 2024-09-11T09:51:27.853 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00087190399999999995, 2024-09-11T09:51:27.853 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:27.853 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:27.853 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "started", 2024-09-11T09:51:27.853 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:27.853 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4694", 2024-09-11T09:51:27.853 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:51:27.853 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 65 2024-09-11T09:51:27.853 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.853 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:27.853 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.853 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:27.853 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.204024+0000", 2024-09-11T09:51:27.854 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:27.854 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.854 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.854 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:27.854 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.204024+0000", 2024-09-11T09:51:27.854 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:27.854 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.854 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.854 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:27.854 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.204025+0000", 2024-09-11T09:51:27.854 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.8400000000000004e-07 2024-09-11T09:51:27.854 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.854 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.855 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:27.855 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.204028+0000", 2024-09-11T09:51:27.855 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.3280000000000002e-06 2024-09-11T09:51:27.855 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.855 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.855 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:27.855 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.204035+0000", 2024-09-11T09:51:27.855 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.5740000000000001e-06 2024-09-11T09:51:27.855 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.855 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.855 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:27.855 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.204042+0000", 2024-09-11T09:51:27.855 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.5100000000000004e-06 2024-09-11T09:51:27.855 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.856 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.856 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:27.856 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.204137+0000", 2024-09-11T09:51:27.856 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.5198999999999994e-05 2024-09-11T09:51:27.856 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.856 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.856 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for rw locks", 2024-09-11T09:51:27.856 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.204212+0000", 2024-09-11T09:51:27.856 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.5407999999999994e-05 2024-09-11T09:51:27.856 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.856 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.856 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:27.856 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.204785+0000", 2024-09-11T09:51:27.857 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00057256500000000003 2024-09-11T09:51:27.857 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.857 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.857 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:27.857 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.204842+0000", 2024-09-11T09:51:27.857 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.7327999999999997e-05 2024-09-11T09:51:27.857 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:27.857 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:27.857 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:27.857 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.204896+0000", 2024-09-11T09:51:27.857 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.4308000000000002e-05 2024-09-11T09:51:27.857 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:27.857 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:27.857 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:27.858 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:27.858 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:27.858 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:51:27.860 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-11T09:51:27.860 INFO:tasks.ceph.osd.0.smithi017.stderr:2024-09-11T09:51:27.860+0000 7fb73c0f1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31665) UID: 0 2024-09-11T09:51:27.861 INFO:tasks.workunit.client.0.smithi107.stderr:rbd-nbd: must specify command 2024-09-11T09:51:27.861 INFO:tasks.workunit.client.0.smithi107.stderr:+ return 0 2024-09-11T09:51:27.861 INFO:tasks.workunit.client.0.smithi107.stderr:+ expect_false rbd-nbd INVALIDCMD 2024-09-11T09:51:27.861 INFO:tasks.workunit.client.0.smithi107.stderr:+ rbd-nbd INVALIDCMD 2024-09-11T09:51:27.861 INFO:tasks.workunit.client.0.smithi107.stderr:rbd-nbd: unknown command: INVALIDCMD 2024-09-11T09:51:27.861 INFO:tasks.workunit.client.0.smithi107.stderr:+ return 0 2024-09-11T09:51:27.861 INFO:tasks.workunit.client.0.smithi107.stderr:++ id -u 2024-09-11T09:51:27.861 INFO:tasks.workunit.client.0.smithi107.stderr:+ '[' 1000 -ne 0 ']' 2024-09-11T09:51:27.861 INFO:tasks.workunit.client.0.smithi107.stderr:+ expect_false rbd device --device-type nbd map testrbdnbd27121 2024-09-11T09:51:27.861 INFO:tasks.workunit.client.0.smithi107.stderr:+ rbd device --device-type nbd map testrbdnbd27121 2024-09-11T09:51:27.881 INFO:tasks.workunit.client.0.smithi107.stderr:modprobe: ERROR: could not insert 'nbd': Operation not permitted 2024-09-11T09:51:27.897 INFO:tasks.workunit.client.0.smithi107.stderr:rbd-nbd: netlink connect failed: Operation not permitted 2024-09-11T09:51:27.915 INFO:tasks.workunit.client.0.smithi107.stderr:rbd: rbd-nbd failed with error: /usr/bin/rbd-nbd: exit status: 1 2024-09-11T09:51:27.921 INFO:tasks.workunit.client.0.smithi107.stderr:+ return 0 2024-09-11T09:51:27.921 INFO:tasks.workunit.client.0.smithi107.stderr:+ expect_false _sudo rbd device --device-type nbd map INVALIDIMAGE 2024-09-11T09:51:27.921 INFO:tasks.workunit.client.0.smithi107.stderr:+ _sudo rbd device --device-type nbd map INVALIDIMAGE 2024-09-11T09:51:27.922 INFO:tasks.workunit.client.0.smithi107.stderr:+ local cmd 2024-09-11T09:51:27.922 INFO:tasks.workunit.client.0.smithi107.stderr:++ id -u 2024-09-11T09:51:27.922 INFO:tasks.workunit.client.0.smithi107.stderr:+ '[' 1000 -eq 0 ']' 2024-09-11T09:51:27.922 INFO:tasks.workunit.client.0.smithi107.stderr:++ which rbd 2024-09-11T09:51:27.930 INFO:tasks.workunit.client.0.smithi107.stderr:+ cmd=/usr/bin/rbd 2024-09-11T09:51:27.930 INFO:tasks.workunit.client.0.smithi107.stderr:+ shift 2024-09-11T09:51:27.930 INFO:tasks.workunit.client.0.smithi107.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd map INVALIDIMAGE 2024-09-11T09:51:27.960 INFO:tasks.ceph.osd.0.smithi017.stderr:2024-09-11T09:51:27.961+0000 7fb73c0f1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31665) UID: 0 2024-09-11T09:51:27.963 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-11T09:51:27.963 INFO:teuthology.orchestra.run.smithi096.stdout: "ops": [], 2024-09-11T09:51:27.963 INFO:teuthology.orchestra.run.smithi096.stdout: "num_ops": 0 2024-09-11T09:51:27.963 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-11T09:51:27.971 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-11T09:51:28.023 INFO:tasks.workunit.client.0.smithi107.stderr:rbd: rbd-nbd failed with error: /usr/bin/rbd-nbd: exit status: 1 2024-09-11T09:51:28.027 INFO:tasks.workunit.client.0.smithi107.stderr:+ return 0 2024-09-11T09:51:28.027 INFO:tasks.workunit.client.0.smithi107.stderr:+ expect_false _sudo rbd-nbd --device INVALIDDEV map testrbdnbd27121 2024-09-11T09:51:28.027 INFO:tasks.workunit.client.0.smithi107.stderr:+ _sudo rbd-nbd --device INVALIDDEV map testrbdnbd27121 2024-09-11T09:51:28.027 INFO:tasks.workunit.client.0.smithi107.stderr:+ local cmd 2024-09-11T09:51:28.027 INFO:tasks.workunit.client.0.smithi107.stderr:++ id -u 2024-09-11T09:51:28.028 INFO:tasks.workunit.client.0.smithi107.stderr:+ '[' 1000 -eq 0 ']' 2024-09-11T09:51:28.028 INFO:tasks.workunit.client.0.smithi107.stderr:++ which rbd-nbd 2024-09-11T09:51:28.030 INFO:tasks.workunit.client.0.smithi107.stderr:+ cmd=/usr/bin/rbd-nbd 2024-09-11T09:51:28.030 INFO:tasks.workunit.client.0.smithi107.stderr:+ shift 2024-09-11T09:51:28.030 INFO:tasks.workunit.client.0.smithi107.stderr:+ sudo -nE /usr/bin/rbd-nbd --device INVALIDDEV map testrbdnbd27121 2024-09-11T09:51:28.060 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:51:28.061+0000 7f3c96ec9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31909) UID: 0 2024-09-11T09:51:28.077 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-11T09:51:28.077 INFO:teuthology.orchestra.run.smithi096.stdout: "ops": [], 2024-09-11T09:51:28.077 INFO:teuthology.orchestra.run.smithi096.stdout: "complaint_time": 30, 2024-09-11T09:51:28.077 INFO:teuthology.orchestra.run.smithi096.stdout: "num_blocked_ops": 0 2024-09-11T09:51:28.077 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-11T09:51:28.086 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-11T09:51:28.092 INFO:tasks.workunit.client.0.smithi107.stderr:rbd-nbd: invalid device path: INVALIDDEV (expected /dev/nbd{num}) 2024-09-11T09:51:28.099 INFO:tasks.workunit.client.0.smithi107.stderr:+ return 0 2024-09-11T09:51:28.099 INFO:tasks.workunit.client.0.smithi107.stderr:+ expect_false rbd device --device-type nbd --format INVALID list 2024-09-11T09:51:28.099 INFO:tasks.workunit.client.0.smithi107.stderr:+ rbd device --device-type nbd --format INVALID list 2024-09-11T09:51:28.125 INFO:tasks.workunit.client.0.smithi107.stderr:rbd: the argument for option '--format' is invalid 2024-09-11T09:51:28.126 INFO:tasks.workunit.client.0.smithi107.stderr:+ return 0 2024-09-11T09:51:28.126 INFO:tasks.workunit.client.0.smithi107.stderr:+ rbd device --device-type nbd --format json --pretty-format list 2024-09-11T09:51:28.161 INFO:tasks.ceph.osd.0.smithi017.stderr:2024-09-11T09:51:28.162+0000 7fb73c0f1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31665) UID: 0 2024-09-11T09:51:28.175 INFO:tasks.workunit.client.0.smithi107.stdout:[] 2024-09-11T09:51:28.178 INFO:tasks.workunit.client.0.smithi107.stderr:+ rbd device --device-type nbd --format xml list 2024-09-11T09:51:28.230 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-11T09:51:28.231 INFO:teuthology.orchestra.run.smithi096.stdout: "size": 20, 2024-09-11T09:51:28.231 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 600, 2024-09-11T09:51:28.231 INFO:teuthology.orchestra.run.smithi096.stdout: "ops": [ 2024-09-11T09:51:28.231 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.231 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:111 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:51:28.231 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.711668+0000", 2024-09-11T09:51:28.231 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 76.518616265999995, 2024-09-11T09:51:28.231 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0077145570000000004, 2024-09-11T09:51:28.231 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:28.231 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:28.231 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:28.231 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:28.232 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:51:28.232 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:51:28.232 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 111 2024-09-11T09:51:28.232 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.232 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:28.232 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.232 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:28.232 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711668+0000", 2024-09-11T09:51:28.232 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:28.232 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.232 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.232 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:28.232 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711668+0000", 2024-09-11T09:51:28.233 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:28.233 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.233 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.233 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:28.233 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711669+0000", 2024-09-11T09:51:28.233 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.99e-07 2024-09-11T09:51:28.233 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.233 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.233 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:28.233 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711673+0000", 2024-09-11T09:51:28.233 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.7440000000000001e-06 2024-09-11T09:51:28.233 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.233 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.234 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:28.234 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711686+0000", 2024-09-11T09:51:28.234 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.3317e-05 2024-09-11T09:51:28.234 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.234 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.234 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:28.234 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711697+0000", 2024-09-11T09:51:28.234 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1022000000000001e-05 2024-09-11T09:51:28.234 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.234 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.234 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:28.234 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.717065+0000", 2024-09-11T09:51:28.234 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.005367531 2024-09-11T09:51:28.235 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.235 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.235 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:28.235 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.717114+0000", 2024-09-11T09:51:28.235 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.9707999999999999e-05 2024-09-11T09:51:28.235 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.235 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.235 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:28.235 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.717190+0000", 2024-09-11T09:51:28.235 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.5394000000000004e-05 2024-09-11T09:51:28.235 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.235 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.235 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:28.236 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718793+0000", 2024-09-11T09:51:28.236 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001603653 2024-09-11T09:51:28.236 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.236 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.236 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:28.236 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719329+0000", 2024-09-11T09:51:28.236 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00053543799999999995 2024-09-11T09:51:28.236 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.236 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.236 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:28.236 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719359+0000", 2024-09-11T09:51:28.236 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.0420999999999999e-05 2024-09-11T09:51:28.236 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.237 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.237 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:28.237 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719383+0000", 2024-09-11T09:51:28.237 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.3730000000000001e-05 2024-09-11T09:51:28.237 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:28.237 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:28.237 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:28.237 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.237 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.237 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:112 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:51:28.237 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.711712+0000", 2024-09-11T09:51:28.237 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 76.518572706, 2024-09-11T09:51:28.237 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.008162084, 2024-09-11T09:51:28.238 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:28.238 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:28.238 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:28.238 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:28.238 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:51:28.238 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:51:28.238 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 112 2024-09-11T09:51:28.238 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.238 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:28.238 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.239 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:28.239 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711712+0000", 2024-09-11T09:51:28.239 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:28.239 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.239 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.239 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:28.239 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711712+0000", 2024-09-11T09:51:28.239 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:28.239 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.239 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.240 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:28.240 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711712+0000", 2024-09-11T09:51:28.240 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.6000000000000004e-07 2024-09-11T09:51:28.240 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.240 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.240 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:28.240 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711808+0000", 2024-09-11T09:51:28.240 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.5977999999999995e-05 2024-09-11T09:51:28.240 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.240 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.241 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:28.241 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711819+0000", 2024-09-11T09:51:28.241 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.0755e-05 2024-09-11T09:51:28.241 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.241 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.241 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:28.241 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711826+0000", 2024-09-11T09:51:28.241 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.7970000000000003e-06 2024-09-11T09:51:28.241 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.241 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.241 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:28.242 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.717435+0000", 2024-09-11T09:51:28.242 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0056090239999999998 2024-09-11T09:51:28.242 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.242 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.242 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:28.242 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.717496+0000", 2024-09-11T09:51:28.242 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.0543e-05 2024-09-11T09:51:28.242 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.242 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.242 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:28.242 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.717591+0000", 2024-09-11T09:51:28.242 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.5667000000000004e-05 2024-09-11T09:51:28.242 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.243 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.243 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:28.243 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718831+0000", 2024-09-11T09:51:28.243 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001240204 2024-09-11T09:51:28.243 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.243 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.243 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:28.243 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719819+0000", 2024-09-11T09:51:28.243 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00098751500000000001 2024-09-11T09:51:28.243 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.243 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.243 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:28.243 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719850+0000", 2024-09-11T09:51:28.244 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.1535999999999998e-05 2024-09-11T09:51:28.244 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.244 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.244 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:28.244 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719874+0000", 2024-09-11T09:51:28.244 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.3504999999999999e-05 2024-09-11T09:51:28.244 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:28.244 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:28.244 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:28.244 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.244 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.244 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:113 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:51:28.244 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.711842+0000", 2024-09-11T09:51:28.245 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 76.518442207000007, 2024-09-11T09:51:28.245 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0081160099999999999, 2024-09-11T09:51:28.245 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:28.245 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:28.245 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:28.245 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:28.245 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:51:28.245 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:51:28.245 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 113 2024-09-11T09:51:28.245 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.245 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:28.245 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.245 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:28.246 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711842+0000", 2024-09-11T09:51:28.246 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:28.246 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.246 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.246 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:28.246 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711842+0000", 2024-09-11T09:51:28.246 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:28.246 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.246 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.246 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:28.246 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711843+0000", 2024-09-11T09:51:28.246 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.4399999999999999e-07 2024-09-11T09:51:28.246 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.247 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.247 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:28.247 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711848+0000", 2024-09-11T09:51:28.247 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.4159999999999997e-06 2024-09-11T09:51:28.247 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.247 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.247 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:28.247 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711856+0000", 2024-09-11T09:51:28.247 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.7870000000000007e-06 2024-09-11T09:51:28.247 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.247 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.247 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:28.247 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711862+0000", 2024-09-11T09:51:28.248 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.3689999999999998e-06 2024-09-11T09:51:28.248 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.248 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.248 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:28.248 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.717965+0000", 2024-09-11T09:51:28.248 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0061028899999999997 2024-09-11T09:51:28.248 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.248 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.248 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:28.248 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718023+0000", 2024-09-11T09:51:28.248 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.8313000000000002e-05 2024-09-11T09:51:28.248 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.248 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.248 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:28.249 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718109+0000", 2024-09-11T09:51:28.249 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.6205000000000006e-05 2024-09-11T09:51:28.249 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.249 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.249 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:28.249 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719808+0000", 2024-09-11T09:51:28.249 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0016990060000000001 2024-09-11T09:51:28.249 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.249 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.249 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:28.249 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719883+0000", 2024-09-11T09:51:28.249 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.4624999999999998e-05 2024-09-11T09:51:28.250 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.250 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.250 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:28.250 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719910+0000", 2024-09-11T09:51:28.250 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.7512e-05 2024-09-11T09:51:28.250 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.250 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.250 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:28.250 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719958+0000", 2024-09-11T09:51:28.250 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.8143000000000003e-05 2024-09-11T09:51:28.250 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:28.250 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:28.250 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:28.250 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.251 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.251 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:51:28.251 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.711887+0000", 2024-09-11T09:51:28.251 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 76.518397593000003, 2024-09-11T09:51:28.251 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0086063589999999992, 2024-09-11T09:51:28.251 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:28.251 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:28.251 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:28.251 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:28.251 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:51:28.251 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:51:28.251 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 114 2024-09-11T09:51:28.251 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.252 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:28.252 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.252 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:28.252 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711887+0000", 2024-09-11T09:51:28.252 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:28.252 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.252 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.252 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:28.252 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711887+0000", 2024-09-11T09:51:28.252 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:28.252 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.252 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.252 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:28.253 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711887+0000", 2024-09-11T09:51:28.253 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.1500000000000005e-07 2024-09-11T09:51:28.253 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.253 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.253 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:28.253 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711897+0000", 2024-09-11T09:51:28.253 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.1900000000000001e-06 2024-09-11T09:51:28.253 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.253 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.253 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:28.253 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711902+0000", 2024-09-11T09:51:28.253 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.2349999999999999e-06 2024-09-11T09:51:28.253 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.254 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.254 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:28.254 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711906+0000", 2024-09-11T09:51:28.254 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.7129999999999999e-06 2024-09-11T09:51:28.254 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.254 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.254 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:28.254 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718429+0000", 2024-09-11T09:51:28.254 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0065231960000000002 2024-09-11T09:51:28.254 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.254 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.254 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:28.254 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718480+0000", 2024-09-11T09:51:28.254 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.1161e-05 2024-09-11T09:51:28.255 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.255 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.255 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:28.255 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718558+0000", 2024-09-11T09:51:28.255 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.7519000000000001e-05 2024-09-11T09:51:28.255 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.255 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.255 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:28.255 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720038+0000", 2024-09-11T09:51:28.255 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001480016 2024-09-11T09:51:28.255 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.255 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.255 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:28.255 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720417+0000", 2024-09-11T09:51:28.256 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00037943699999999999 2024-09-11T09:51:28.256 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.256 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.256 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:28.256 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720452+0000", 2024-09-11T09:51:28.256 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.4595000000000003e-05 2024-09-11T09:51:28.256 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.256 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.256 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:28.256 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720493+0000", 2024-09-11T09:51:28.256 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.1782000000000001e-05 2024-09-11T09:51:28.256 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:28.256 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:28.257 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:28.257 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.257 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.257 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:51:28.257 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.711912+0000", 2024-09-11T09:51:28.257 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 76.518372924000005, 2024-09-11T09:51:28.257 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0086864750000000008, 2024-09-11T09:51:28.257 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:28.257 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:28.257 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:28.257 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:28.257 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:51:28.258 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:51:28.258 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 115 2024-09-11T09:51:28.258 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.258 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:28.258 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.258 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:28.258 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711912+0000", 2024-09-11T09:51:28.258 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:28.258 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.258 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.258 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:28.258 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711912+0000", 2024-09-11T09:51:28.258 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:28.259 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.259 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.259 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:28.259 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711912+0000", 2024-09-11T09:51:28.259 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.3900000000000001e-07 2024-09-11T09:51:28.259 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.259 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.259 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:28.259 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711969+0000", 2024-09-11T09:51:28.259 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.7501999999999999e-05 2024-09-11T09:51:28.259 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.259 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.259 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:28.259 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711986+0000", 2024-09-11T09:51:28.260 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.6402999999999999e-05 2024-09-11T09:51:28.260 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.260 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.260 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:28.260 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711990+0000", 2024-09-11T09:51:28.260 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.929e-06 2024-09-11T09:51:28.260 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.260 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.260 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:28.260 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718861+0000", 2024-09-11T09:51:28.260 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0068710530000000002 2024-09-11T09:51:28.260 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.261 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.261 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:28.261 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718908+0000", 2024-09-11T09:51:28.261 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.7431e-05 2024-09-11T09:51:28.261 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.261 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.261 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:28.261 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718984+0000", 2024-09-11T09:51:28.261 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.5500999999999994e-05 2024-09-11T09:51:28.261 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.261 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.261 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:28.262 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720500+0000", 2024-09-11T09:51:28.262 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0015158820000000001 2024-09-11T09:51:28.262 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.262 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.262 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:28.262 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720536+0000", 2024-09-11T09:51:28.262 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.6118000000000002e-05 2024-09-11T09:51:28.262 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.262 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.262 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:28.262 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720570+0000", 2024-09-11T09:51:28.262 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.4387999999999997e-05 2024-09-11T09:51:28.263 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.263 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.263 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:28.263 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720598+0000", 2024-09-11T09:51:28.263 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.8028999999999999e-05 2024-09-11T09:51:28.263 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:28.263 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:28.263 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:28.263 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.263 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.263 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:51:28.263 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.711998+0000", 2024-09-11T09:51:28.264 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 76.518286744999997, 2024-09-11T09:51:28.264 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.010785813999999999, 2024-09-11T09:51:28.264 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:28.264 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:28.264 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:28.264 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:28.264 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:51:28.264 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:51:28.264 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 116 2024-09-11T09:51:28.264 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.264 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:28.264 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.264 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:28.265 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711998+0000", 2024-09-11T09:51:28.265 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:28.265 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.265 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.265 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:28.265 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711998+0000", 2024-09-11T09:51:28.265 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:28.265 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.265 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.265 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:28.265 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711998+0000", 2024-09-11T09:51:28.265 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.3300000000000003e-07 2024-09-11T09:51:28.266 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.266 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.266 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:28.266 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712001+0000", 2024-09-11T09:51:28.266 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.9349999999999999e-06 2024-09-11T09:51:28.266 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.266 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.266 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:28.266 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712006+0000", 2024-09-11T09:51:28.266 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.1630000000000004e-06 2024-09-11T09:51:28.267 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.267 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.267 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:28.267 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712010+0000", 2024-09-11T09:51:28.267 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.574e-06 2024-09-11T09:51:28.267 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.267 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.267 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:28.267 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719415+0000", 2024-09-11T09:51:28.267 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0074049099999999998 2024-09-11T09:51:28.268 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.268 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.268 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:28.268 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719465+0000", 2024-09-11T09:51:28.268 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.0129000000000001e-05 2024-09-11T09:51:28.268 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.268 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.268 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:28.268 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719545+0000", 2024-09-11T09:51:28.268 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.0253999999999994e-05 2024-09-11T09:51:28.269 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.269 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.269 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:28.269 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720636+0000", 2024-09-11T09:51:28.269 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0010909330000000001 2024-09-11T09:51:28.269 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.269 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.269 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:28.269 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.722707+0000", 2024-09-11T09:51:28.269 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0020713300000000001 2024-09-11T09:51:28.270 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.270 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.270 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:28.270 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.722743+0000", 2024-09-11T09:51:28.270 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.5351000000000001e-05 2024-09-11T09:51:28.270 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.270 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.270 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:28.270 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.722784+0000", 2024-09-11T09:51:28.270 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.0802e-05 2024-09-11T09:51:28.271 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:28.271 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:28.271 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:28.271 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.271 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.271 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:51:28.271 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.712014+0000", 2024-09-11T09:51:28.271 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 76.518270705000006, 2024-09-11T09:51:28.271 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.013420938, 2024-09-11T09:51:28.271 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:28.271 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:28.272 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:28.272 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:28.272 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:51:28.272 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:51:28.272 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 117 2024-09-11T09:51:28.272 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.272 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:28.272 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.272 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:28.272 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712014+0000", 2024-09-11T09:51:28.272 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:28.272 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.272 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.273 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:28.273 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712014+0000", 2024-09-11T09:51:28.273 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:28.273 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.273 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.273 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:28.273 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712014+0000", 2024-09-11T09:51:28.273 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.2000000000000001e-07 2024-09-11T09:51:28.273 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.273 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.273 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:28.273 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712016+0000", 2024-09-11T09:51:28.273 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.2309999999999999e-06 2024-09-11T09:51:28.274 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.274 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.274 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:28.274 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712020+0000", 2024-09-11T09:51:28.274 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.3830000000000001e-06 2024-09-11T09:51:28.274 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.274 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.274 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:28.274 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712023+0000", 2024-09-11T09:51:28.274 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.3340000000000002e-06 2024-09-11T09:51:28.274 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.274 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.274 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:28.275 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720083+0000", 2024-09-11T09:51:28.275 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0080601189999999993 2024-09-11T09:51:28.275 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.275 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.275 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:28.275 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720125+0000", 2024-09-11T09:51:28.275 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.2029e-05 2024-09-11T09:51:28.275 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.275 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.275 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:28.275 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720205+0000", 2024-09-11T09:51:28.275 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.0010000000000001e-05 2024-09-11T09:51:28.276 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.276 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.276 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:28.276 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720849+0000", 2024-09-11T09:51:28.276 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00064369899999999996 2024-09-11T09:51:28.276 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.276 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.276 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:28.276 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.725357+0000", 2024-09-11T09:51:28.276 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0045077379999999998 2024-09-11T09:51:28.276 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.276 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.276 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:28.277 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.725392+0000", 2024-09-11T09:51:28.277 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.5862999999999999e-05 2024-09-11T09:51:28.277 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.277 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.277 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:28.277 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.725435+0000", 2024-09-11T09:51:28.277 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.2311999999999997e-05 2024-09-11T09:51:28.277 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:28.277 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:28.277 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:28.277 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.277 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.277 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:159 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:51:28.278 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:12.452099+0000", 2024-09-11T09:51:28.278 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 75.778185117000007, 2024-09-11T09:51:28.278 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0069885950000000002, 2024-09-11T09:51:28.278 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:28.278 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:28.278 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:28.278 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:28.278 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:51:28.278 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:51:28.278 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 159 2024-09-11T09:51:28.278 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.278 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:28.279 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.279 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:28.279 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452099+0000", 2024-09-11T09:51:28.279 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:28.279 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.279 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.279 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:28.279 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452099+0000", 2024-09-11T09:51:28.279 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:28.279 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.279 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.279 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:28.279 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452100+0000", 2024-09-11T09:51:28.280 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.2800000000000003e-07 2024-09-11T09:51:28.280 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.280 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.280 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:28.280 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452102+0000", 2024-09-11T09:51:28.280 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.824e-06 2024-09-11T09:51:28.280 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.280 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.280 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:28.280 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452105+0000", 2024-09-11T09:51:28.280 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.0460000000000001e-06 2024-09-11T09:51:28.280 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.281 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.281 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:28.281 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452107+0000", 2024-09-11T09:51:28.281 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.543e-06 2024-09-11T09:51:28.281 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.281 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.281 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:28.281 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.455484+0000", 2024-09-11T09:51:28.281 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0033769759999999998 2024-09-11T09:51:28.281 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.281 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.281 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:28.282 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.455527+0000", 2024-09-11T09:51:28.282 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.2589999999999997e-05 2024-09-11T09:51:28.282 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.282 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.282 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:28.282 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.455594+0000", 2024-09-11T09:51:28.282 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.6775999999999999e-05 2024-09-11T09:51:28.282 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.282 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.282 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:28.282 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.456312+0000", 2024-09-11T09:51:28.283 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00071874700000000005 2024-09-11T09:51:28.283 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.283 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.283 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:28.283 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.459015+0000", 2024-09-11T09:51:28.283 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0027028540000000002 2024-09-11T09:51:28.283 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.283 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.283 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:28.283 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.459054+0000", 2024-09-11T09:51:28.283 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.8840000000000001e-05 2024-09-11T09:51:28.284 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.284 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.284 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:28.284 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.459088+0000", 2024-09-11T09:51:28.284 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.4071e-05 2024-09-11T09:51:28.284 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:28.284 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:28.284 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:28.284 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.284 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.285 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4688.0:38 2.5 2:abde2694:::rbd_data.1250f5e25b80.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:28.285 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:22.903926+0000", 2024-09-11T09:51:28.285 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 5.326358441, 2024-09-11T09:51:28.285 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.028303676, 2024-09-11T09:51:28.285 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:28.285 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:28.285 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:28.285 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:28.285 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4688", 2024-09-11T09:51:28.285 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:51:28.285 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 38 2024-09-11T09:51:28.286 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.286 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:28.286 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.286 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:28.286 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.903926+0000", 2024-09-11T09:51:28.286 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:28.286 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.286 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.286 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:28.286 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.903926+0000", 2024-09-11T09:51:28.287 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:28.287 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.287 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.287 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:28.287 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.903928+0000", 2024-09-11T09:51:28.287 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.4249999999999999e-06 2024-09-11T09:51:28.287 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.287 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.287 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:28.287 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.914774+0000", 2024-09-11T09:51:28.287 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.010846613 2024-09-11T09:51:28.288 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.288 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.288 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:28.288 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.914793+0000", 2024-09-11T09:51:28.288 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.8879999999999999e-05 2024-09-11T09:51:28.288 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.288 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.288 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:28.288 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.914801+0000", 2024-09-11T09:51:28.289 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.2230000000000007e-06 2024-09-11T09:51:28.289 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.289 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.289 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:28.289 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.914854+0000", 2024-09-11T09:51:28.289 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.2883999999999997e-05 2024-09-11T09:51:28.289 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.289 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.289 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:28.289 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.914951+0000", 2024-09-11T09:51:28.289 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.6731000000000006e-05 2024-09-11T09:51:28.289 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.290 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.290 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:28.290 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.915080+0000", 2024-09-11T09:51:28.290 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00012926999999999999 2024-09-11T09:51:28.290 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.290 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.290 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:28.290 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.931469+0000", 2024-09-11T09:51:28.290 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.016389107 2024-09-11T09:51:28.290 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.290 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.290 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:28.291 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.932118+0000", 2024-09-11T09:51:28.291 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00064915600000000002 2024-09-11T09:51:28.291 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.291 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.291 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:28.291 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.932177+0000", 2024-09-11T09:51:28.291 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.9032e-05 2024-09-11T09:51:28.291 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.291 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.291 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:28.291 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.932230+0000", 2024-09-11T09:51:28.291 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.2355000000000003e-05 2024-09-11T09:51:28.291 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:28.292 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:28.292 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:28.292 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.292 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.292 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_repop(client.4688.0:36 2.6 e53/15 2:61e1b442:::rbd_data.1250f5e25b80.0000000000000003:head v 53'2, mlcod=0'0)", 2024-09-11T09:51:28.292 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:22.915597+0000", 2024-09-11T09:51:28.292 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 5.3146878910000002, 2024-09-11T09:51:28.292 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.020430798, 2024-09-11T09:51:28.292 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:28.292 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:28.293 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:28.293 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:28.293 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.293 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:28.293 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.915597+0000", 2024-09-11T09:51:28.293 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:28.293 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.293 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.293 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:28.293 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.915597+0000", 2024-09-11T09:51:28.293 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:28.294 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.294 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.294 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:28.294 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.915598+0000", 2024-09-11T09:51:28.294 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.7150000000000001e-06 2024-09-11T09:51:28.294 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.294 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.294 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:28.294 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921741+0000", 2024-09-11T09:51:28.295 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0061424339999999996 2024-09-11T09:51:28.295 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.295 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.295 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:28.295 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921757+0000", 2024-09-11T09:51:28.295 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.6144e-05 2024-09-11T09:51:28.295 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.295 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.295 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:28.295 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921765+0000", 2024-09-11T09:51:28.296 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.0809999999999994e-06 2024-09-11T09:51:28.296 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.296 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.296 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:28.296 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921826+0000", 2024-09-11T09:51:28.296 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.0878999999999998e-05 2024-09-11T09:51:28.296 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.296 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.296 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:28.296 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921849+0000", 2024-09-11T09:51:28.297 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.2966999999999999e-05 2024-09-11T09:51:28.297 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.297 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.297 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:28.297 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.935968+0000", 2024-09-11T09:51:28.297 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.014119191 2024-09-11T09:51:28.297 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.297 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.297 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:28.297 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.936027+0000", 2024-09-11T09:51:28.298 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.9386999999999999e-05 2024-09-11T09:51:28.298 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:28.298 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:28.298 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:28.298 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.298 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.298 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_repop(client.4688.0:39 2.6 e53/15 2:743e7616:::rbd_data.1250f5e25b80.0000000000000005:head v 53'3, mlcod=0'0)", 2024-09-11T09:51:28.298 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:22.921806+0000", 2024-09-11T09:51:28.298 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 5.3084788410000003, 2024-09-11T09:51:28.298 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.015275018, 2024-09-11T09:51:28.299 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:28.299 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:28.299 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:28.299 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:28.299 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.299 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:28.299 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921806+0000", 2024-09-11T09:51:28.299 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:28.299 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.299 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.299 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:28.299 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921806+0000", 2024-09-11T09:51:28.300 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:28.300 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.300 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.300 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:28.300 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921806+0000", 2024-09-11T09:51:28.300 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.2200000000000003e-07 2024-09-11T09:51:28.300 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.300 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.300 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:28.300 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.925590+0000", 2024-09-11T09:51:28.300 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0037838949999999998 2024-09-11T09:51:28.300 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.300 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.301 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:28.301 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.925605+0000", 2024-09-11T09:51:28.301 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.4321999999999999e-05 2024-09-11T09:51:28.301 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.301 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.301 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:28.301 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.925613+0000", 2024-09-11T09:51:28.301 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.1980000000000001e-06 2024-09-11T09:51:28.301 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.302 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.302 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:28.302 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.925857+0000", 2024-09-11T09:51:28.302 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000244064 2024-09-11T09:51:28.302 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.302 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.302 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:28.302 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.925875+0000", 2024-09-11T09:51:28.302 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.838e-05 2024-09-11T09:51:28.302 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.302 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.302 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:28.302 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.937042+0000", 2024-09-11T09:51:28.303 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.011166416 2024-09-11T09:51:28.303 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.303 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.303 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:28.303 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.937081+0000", 2024-09-11T09:51:28.303 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.9020999999999998e-05 2024-09-11T09:51:28.303 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:28.303 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:28.303 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:28.303 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.303 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.303 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4688.0:49 2.5 2:b6dd9fc9:::rbd_data.1250f5e25b80.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:28.304 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:22.938548+0000", 2024-09-11T09:51:28.304 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 5.2917368380000003, 2024-09-11T09:51:28.304 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.022766950000000001, 2024-09-11T09:51:28.304 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:28.304 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:28.304 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:28.304 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:28.304 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4688", 2024-09-11T09:51:28.304 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:51:28.304 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 49 2024-09-11T09:51:28.304 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.304 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:28.304 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.305 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:28.305 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.938548+0000", 2024-09-11T09:51:28.305 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:28.305 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.305 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.305 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:28.305 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.938548+0000", 2024-09-11T09:51:28.305 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:28.305 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.305 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.305 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:28.305 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.938550+0000", 2024-09-11T09:51:28.305 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.7710000000000001e-06 2024-09-11T09:51:28.305 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.306 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.306 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:28.306 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.944237+0000", 2024-09-11T09:51:28.306 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0056879790000000001 2024-09-11T09:51:28.306 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.306 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.306 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:28.306 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.944262+0000", 2024-09-11T09:51:28.306 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.4247e-05 2024-09-11T09:51:28.306 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.306 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.306 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:28.306 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.944274+0000", 2024-09-11T09:51:28.307 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.2639999999999999e-05 2024-09-11T09:51:28.307 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.307 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.307 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:28.307 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.944344+0000", 2024-09-11T09:51:28.307 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.9574999999999997e-05 2024-09-11T09:51:28.307 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.307 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.307 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:28.307 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.944474+0000", 2024-09-11T09:51:28.307 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000130464 2024-09-11T09:51:28.307 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.307 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.308 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:28.308 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.944585+0000", 2024-09-11T09:51:28.308 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000110096 2024-09-11T09:51:28.308 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.308 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.308 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:28.308 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.952724+0000", 2024-09-11T09:51:28.308 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0081398900000000003 2024-09-11T09:51:28.308 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.308 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.308 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:28.308 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.961219+0000", 2024-09-11T09:51:28.308 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0084949789999999997 2024-09-11T09:51:28.309 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.309 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.309 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:28.309 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.961280+0000", 2024-09-11T09:51:28.309 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.0404e-05 2024-09-11T09:51:28.309 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.309 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.309 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:28.309 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.961315+0000", 2024-09-11T09:51:28.309 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.4904999999999999e-05 2024-09-11T09:51:28.309 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:28.309 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:28.309 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:28.310 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.310 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.310 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_repop(client.4688.0:44 2.6 e53/15 2:67246a22:::rbd_data.1250f5e25b80.0000000000000007:head v 53'4, mlcod=53'2)", 2024-09-11T09:51:28.310 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:22.951721+0000", 2024-09-11T09:51:28.310 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 5.2785635659999999, 2024-09-11T09:51:28.310 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.01080981, 2024-09-11T09:51:28.310 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:28.310 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:28.310 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:28.310 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:28.310 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.311 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:28.311 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.951721+0000", 2024-09-11T09:51:28.311 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:28.311 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.311 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.311 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:28.311 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.951721+0000", 2024-09-11T09:51:28.311 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:28.311 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.311 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.311 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:28.311 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.951722+0000", 2024-09-11T09:51:28.312 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.0279999999999999e-06 2024-09-11T09:51:28.312 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.312 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.312 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:28.312 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.955612+0000", 2024-09-11T09:51:28.312 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0038896920000000001 2024-09-11T09:51:28.312 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.312 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.312 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:28.312 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.955628+0000", 2024-09-11T09:51:28.312 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.5945000000000001e-05 2024-09-11T09:51:28.313 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.313 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.313 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:28.313 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.955635+0000", 2024-09-11T09:51:28.313 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.8080000000000005e-06 2024-09-11T09:51:28.313 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.313 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.313 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:28.313 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.955700+0000", 2024-09-11T09:51:28.313 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.4114000000000003e-05 2024-09-11T09:51:28.313 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.314 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.314 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:28.314 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.955724+0000", 2024-09-11T09:51:28.314 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.3989999999999999e-05 2024-09-11T09:51:28.314 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.314 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.314 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:28.314 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.962490+0000", 2024-09-11T09:51:28.314 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0067662599999999996 2024-09-11T09:51:28.314 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.314 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.315 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:28.315 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.962531+0000", 2024-09-11T09:51:28.315 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.0973000000000002e-05 2024-09-11T09:51:28.315 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:28.315 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:28.315 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:28.315 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.315 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.315 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_repop(client.4688.0:51 2.6 e53/15 2:6b5894fc:::rbd_data.1250f5e25b80.000000000000000a:head v 53'5, mlcod=53'2)", 2024-09-11T09:51:28.315 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:22.959030+0000", 2024-09-11T09:51:28.315 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 5.2712543920000003, 2024-09-11T09:51:28.316 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.011503638, 2024-09-11T09:51:28.316 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:28.316 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:28.316 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:28.316 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:28.316 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.316 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:28.316 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.959030+0000", 2024-09-11T09:51:28.316 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:28.316 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.316 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.317 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:28.317 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.959030+0000", 2024-09-11T09:51:28.317 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:28.317 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.317 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.317 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:28.317 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.959032+0000", 2024-09-11T09:51:28.317 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.356e-06 2024-09-11T09:51:28.317 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.317 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.317 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:28.320 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.962899+0000", 2024-09-11T09:51:28.320 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.003867639 2024-09-11T09:51:28.320 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.320 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.320 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:28.321 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.962909+0000", 2024-09-11T09:51:28.321 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.0105999999999999e-05 2024-09-11T09:51:28.321 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.321 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.321 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:28.321 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.962915+0000", 2024-09-11T09:51:28.321 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.8869999999999997e-06 2024-09-11T09:51:28.321 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.321 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.321 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:28.321 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.962962+0000", 2024-09-11T09:51:28.322 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.6511999999999998e-05 2024-09-11T09:51:28.322 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.322 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.322 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:28.322 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.962989+0000", 2024-09-11T09:51:28.322 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.7702999999999999e-05 2024-09-11T09:51:28.322 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.322 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.322 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:28.322 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.970495+0000", 2024-09-11T09:51:28.322 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0075052449999999998 2024-09-11T09:51:28.322 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.323 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.323 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:28.323 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.970534+0000", 2024-09-11T09:51:28.323 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.9190000000000003e-05 2024-09-11T09:51:28.323 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:28.323 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:28.323 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:28.323 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.323 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.323 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_repop(client.4688.0:58 2.6 e53/15 2:6277c644:::rbd_data.1250f5e25b80.000000000000000e:head v 53'6, mlcod=53'4)", 2024-09-11T09:51:28.323 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:22.996199+0000", 2024-09-11T09:51:28.323 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 5.2340854720000003, 2024-09-11T09:51:28.323 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.010145573999999999, 2024-09-11T09:51:28.324 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:28.324 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:28.324 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:28.324 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:28.324 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.324 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:28.324 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.996199+0000", 2024-09-11T09:51:28.324 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:28.324 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.324 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.324 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:28.324 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.996199+0000", 2024-09-11T09:51:28.325 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:28.325 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.325 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.325 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:28.325 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.996200+0000", 2024-09-11T09:51:28.325 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.85e-07 2024-09-11T09:51:28.325 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.325 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.325 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:28.325 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.999914+0000", 2024-09-11T09:51:28.325 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.003714406 2024-09-11T09:51:28.325 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.325 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.326 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:28.326 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.999928+0000", 2024-09-11T09:51:28.326 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.3149000000000001e-05 2024-09-11T09:51:28.326 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.326 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.326 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:28.326 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.999951+0000", 2024-09-11T09:51:28.326 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.3116e-05 2024-09-11T09:51:28.326 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.326 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.326 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:28.326 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.000003+0000", 2024-09-11T09:51:28.326 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.2169000000000001e-05 2024-09-11T09:51:28.327 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.327 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.327 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:28.327 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.000018+0000", 2024-09-11T09:51:28.327 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.5500000000000001e-05 2024-09-11T09:51:28.327 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.327 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.327 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:28.327 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.006303+0000", 2024-09-11T09:51:28.327 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0062846710000000004 2024-09-11T09:51:28.327 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.327 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.328 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:28.328 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.006345+0000", 2024-09-11T09:51:28.328 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.1678000000000003e-05 2024-09-11T09:51:28.328 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:28.328 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:28.328 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:28.328 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.328 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.328 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4694.0:42 2.5 2:a4fc00ec:ns::rbd_data.1256a0c02879.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:28.328 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.111570+0000", 2024-09-11T09:51:28.328 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 5.1187143659999998, 2024-09-11T09:51:28.328 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.038002197000000001, 2024-09-11T09:51:28.329 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:28.329 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:28.329 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:28.329 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:28.329 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4694", 2024-09-11T09:51:28.329 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:51:28.329 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 42 2024-09-11T09:51:28.329 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.329 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:28.329 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.329 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:28.329 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.111570+0000", 2024-09-11T09:51:28.329 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:28.330 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.330 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.330 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:28.330 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.111570+0000", 2024-09-11T09:51:28.330 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:28.330 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.330 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.330 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:28.330 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.111572+0000", 2024-09-11T09:51:28.330 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.3909999999999999e-06 2024-09-11T09:51:28.330 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.330 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.330 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:28.331 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.135929+0000", 2024-09-11T09:51:28.331 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.024357410999999999 2024-09-11T09:51:28.331 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.331 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.331 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:28.331 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.135951+0000", 2024-09-11T09:51:28.331 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.1775000000000001e-05 2024-09-11T09:51:28.331 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.331 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.331 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:28.331 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.135962+0000", 2024-09-11T09:51:28.331 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.0903e-05 2024-09-11T09:51:28.332 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.332 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.332 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:28.332 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.136045+0000", 2024-09-11T09:51:28.332 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.3128999999999996e-05 2024-09-11T09:51:28.332 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.332 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.332 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:28.332 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.136229+0000", 2024-09-11T09:51:28.332 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00018393700000000001 2024-09-11T09:51:28.332 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.333 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.333 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:28.333 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.136394+0000", 2024-09-11T09:51:28.333 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000165527 2024-09-11T09:51:28.333 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.333 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.333 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:28.333 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.149320+0000", 2024-09-11T09:51:28.333 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.01292568 2024-09-11T09:51:28.333 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.333 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.333 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:28.334 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.149493+0000", 2024-09-11T09:51:28.334 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00017285999999999999 2024-09-11T09:51:28.334 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.334 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.334 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:28.334 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.149539+0000", 2024-09-11T09:51:28.334 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.6628000000000001e-05 2024-09-11T09:51:28.334 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.334 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.334 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:28.334 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.149572+0000", 2024-09-11T09:51:28.334 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.2956000000000001e-05 2024-09-11T09:51:28.334 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:28.334 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:28.335 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:28.335 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.335 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.335 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_repop(client.4694.0:45 2.6 e53/15 2:671fb830:ns::rbd_data.1256a0c02879.0000000000000007:head v 53'9, mlcod=53'7)", 2024-09-11T09:51:28.335 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.121833+0000", 2024-09-11T09:51:28.335 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 5.1084520080000004, 2024-09-11T09:51:28.335 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.023876192000000001, 2024-09-11T09:51:28.335 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:28.335 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:28.335 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:28.335 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:28.335 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.335 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:28.335 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.121833+0000", 2024-09-11T09:51:28.336 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:28.336 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.336 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.336 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:28.336 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.121833+0000", 2024-09-11T09:51:28.336 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:28.336 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.336 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.336 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:28.336 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.121834+0000", 2024-09-11T09:51:28.336 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.471e-06 2024-09-11T09:51:28.336 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.336 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.337 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:28.337 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.128990+0000", 2024-09-11T09:51:28.337 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0071557879999999997 2024-09-11T09:51:28.337 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.337 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.337 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:28.337 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.129011+0000", 2024-09-11T09:51:28.337 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.0760000000000001e-05 2024-09-11T09:51:28.337 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.337 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.337 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:28.337 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.129022+0000", 2024-09-11T09:51:28.337 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1056000000000001e-05 2024-09-11T09:51:28.338 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.338 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.338 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:28.338 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.129097+0000", 2024-09-11T09:51:28.338 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.5082999999999999e-05 2024-09-11T09:51:28.338 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.338 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.338 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:28.338 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.129130+0000", 2024-09-11T09:51:28.338 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.3345e-05 2024-09-11T09:51:28.338 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.338 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.338 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:28.338 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.145657+0000", 2024-09-11T09:51:28.339 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.016527224 2024-09-11T09:51:28.339 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.339 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.339 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:28.339 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.145709+0000", 2024-09-11T09:51:28.339 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.1465000000000003e-05 2024-09-11T09:51:28.339 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:28.339 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:28.339 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:28.339 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.339 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.339 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4694.0:43 2.0 2:16c8ae84:ns::rbd_data.1256a0c02879.0000000000000005:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:28.339 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.135999+0000", 2024-09-11T09:51:28.340 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 5.0942860579999998, 2024-09-11T09:51:28.340 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.039468309, 2024-09-11T09:51:28.340 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:28.340 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:28.340 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:28.340 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:28.340 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4694", 2024-09-11T09:51:28.340 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:51:28.340 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 43 2024-09-11T09:51:28.340 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.340 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:28.340 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.340 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:28.340 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.135999+0000", 2024-09-11T09:51:28.341 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:28.341 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.341 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.341 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:28.341 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.135999+0000", 2024-09-11T09:51:28.341 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:28.341 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.341 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.341 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:28.341 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.136000+0000", 2024-09-11T09:51:28.341 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.6330000000000001e-06 2024-09-11T09:51:28.341 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.342 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.342 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:28.342 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.157961+0000", 2024-09-11T09:51:28.342 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0219606 2024-09-11T09:51:28.342 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.342 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.342 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:28.342 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.157995+0000", 2024-09-11T09:51:28.342 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.4391999999999999e-05 2024-09-11T09:51:28.342 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.342 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.342 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:28.342 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.158011+0000", 2024-09-11T09:51:28.343 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.5794000000000002e-05 2024-09-11T09:51:28.343 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.343 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.343 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:28.343 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.158083+0000", 2024-09-11T09:51:28.343 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.2273000000000004e-05 2024-09-11T09:51:28.343 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.343 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.343 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:28.343 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.158239+0000", 2024-09-11T09:51:28.343 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000155372 2024-09-11T09:51:28.343 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.343 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.343 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 1", 2024-09-11T09:51:28.344 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.158403+0000", 2024-09-11T09:51:28.344 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00016408199999999999 2024-09-11T09:51:28.344 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.344 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.344 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:28.344 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.173860+0000", 2024-09-11T09:51:28.344 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.015457347 2024-09-11T09:51:28.344 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.344 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.344 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:28.344 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.175400+0000", 2024-09-11T09:51:28.344 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001540006 2024-09-11T09:51:28.345 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.345 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.345 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:28.345 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.175438+0000", 2024-09-11T09:51:28.345 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.8135e-05 2024-09-11T09:51:28.345 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.345 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.345 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:28.345 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.175467+0000", 2024-09-11T09:51:28.345 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.8674999999999999e-05 2024-09-11T09:51:28.345 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:28.345 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:28.345 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:28.346 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.346 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.346 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4694.0:57 2.0 2:04b5549c:ns::rbd_data.1256a0c02879.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:28.346 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.158037+0000", 2024-09-11T09:51:28.346 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 5.0722473199999998, 2024-09-11T09:51:28.346 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.027748790999999998, 2024-09-11T09:51:28.346 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:28.346 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:28.346 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:28.346 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:28.346 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4694", 2024-09-11T09:51:28.346 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:51:28.346 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 57 2024-09-11T09:51:28.346 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.347 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:28.347 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.347 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:28.347 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.158037+0000", 2024-09-11T09:51:28.347 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:28.347 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.347 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.347 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:28.347 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.158037+0000", 2024-09-11T09:51:28.347 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:28.347 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.347 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.347 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:28.348 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.158039+0000", 2024-09-11T09:51:28.348 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.356e-06 2024-09-11T09:51:28.348 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.348 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.348 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:28.348 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.176367+0000", 2024-09-11T09:51:28.348 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.018328739 2024-09-11T09:51:28.348 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.348 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.348 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:28.348 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.176378+0000", 2024-09-11T09:51:28.348 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.0713999999999999e-05 2024-09-11T09:51:28.348 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.349 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.349 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:28.349 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.176383+0000", 2024-09-11T09:51:28.349 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.0209999999999999e-06 2024-09-11T09:51:28.349 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.349 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.349 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:28.349 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.176426+0000", 2024-09-11T09:51:28.349 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.2623999999999997e-05 2024-09-11T09:51:28.349 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.349 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.349 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:28.349 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.176552+0000", 2024-09-11T09:51:28.349 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000126076 2024-09-11T09:51:28.350 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.350 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.350 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 1", 2024-09-11T09:51:28.350 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.176665+0000", 2024-09-11T09:51:28.350 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000112995 2024-09-11T09:51:28.350 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.350 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.350 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:28.350 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.183639+0000", 2024-09-11T09:51:28.350 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0069740039999999998 2024-09-11T09:51:28.350 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.350 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.350 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:28.351 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.185689+0000", 2024-09-11T09:51:28.351 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.002049852 2024-09-11T09:51:28.351 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.351 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.351 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:28.351 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.185725+0000", 2024-09-11T09:51:28.351 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.6804999999999997e-05 2024-09-11T09:51:28.351 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.351 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.351 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:28.351 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.185786+0000", 2024-09-11T09:51:28.351 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.0605e-05 2024-09-11T09:51:28.351 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:28.351 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:28.352 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:28.352 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.352 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.352 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4694.0:60 2.0 2:054f2af4:ns::rbd_data.1256a0c02879.000000000000000f:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:28.352 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.186855+0000", 2024-09-11T09:51:28.352 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 5.043429304, 2024-09-11T09:51:28.352 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.013682913, 2024-09-11T09:51:28.352 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:28.352 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:28.352 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:28.352 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:28.352 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4694", 2024-09-11T09:51:28.352 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:51:28.353 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 60 2024-09-11T09:51:28.353 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.353 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:28.353 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.353 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:28.353 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.186855+0000", 2024-09-11T09:51:28.353 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:28.353 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.353 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.353 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:28.353 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.186855+0000", 2024-09-11T09:51:28.353 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:28.353 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.354 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.354 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:28.354 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.186856+0000", 2024-09-11T09:51:28.354 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.1200000000000001e-07 2024-09-11T09:51:28.354 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.354 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.354 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:28.354 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.191484+0000", 2024-09-11T09:51:28.354 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0046274159999999996 2024-09-11T09:51:28.354 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.354 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.354 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:28.354 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.191496+0000", 2024-09-11T09:51:28.354 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.2149e-05 2024-09-11T09:51:28.355 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.355 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.355 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:28.355 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.191502+0000", 2024-09-11T09:51:28.355 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.1890000000000004e-06 2024-09-11T09:51:28.355 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.355 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.355 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:28.355 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.191544+0000", 2024-09-11T09:51:28.355 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.1724999999999998e-05 2024-09-11T09:51:28.355 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.355 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.355 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:28.356 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.191641+0000", 2024-09-11T09:51:28.356 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.6939000000000001e-05 2024-09-11T09:51:28.356 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.356 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.356 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 1", 2024-09-11T09:51:28.356 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.191731+0000", 2024-09-11T09:51:28.356 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.0651999999999999e-05 2024-09-11T09:51:28.356 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.356 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.356 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:28.356 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.199852+0000", 2024-09-11T09:51:28.356 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0081204529999999997 2024-09-11T09:51:28.356 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.357 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.357 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:28.357 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.200461+0000", 2024-09-11T09:51:28.357 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00060900899999999998 2024-09-11T09:51:28.357 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.357 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.357 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:28.357 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.200497+0000", 2024-09-11T09:51:28.357 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.5800000000000003e-05 2024-09-11T09:51:28.357 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.357 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.357 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:28.357 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.200538+0000", 2024-09-11T09:51:28.358 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.1668999999999997e-05 2024-09-11T09:51:28.358 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:28.358 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:28.358 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:28.358 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:28.358 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:28.358 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-11T09:51:28.360 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-11T09:51:28.361 INFO:teuthology.orchestra.run.smithi096.stderr:Exporting 3.3 info 3.3( empty local-lis/les=0/0 n=0 ec=54/54 lis/c=0/0 les/c/f=0/0/0 sis=54) 2024-09-11T09:51:28.361 INFO:teuthology.orchestra.run.smithi096.stderr:Export successful 2024-09-11T09:51:28.362 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-09-11T09:51:28.360+0000 7f7b6c7d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31904) UID: 0 2024-09-11T09:51:28.362 INFO:tasks.workunit.client.0.smithi107.stderr:++ _sudo rbd device --device-type nbd map rbd/testrbdnbd27121 2024-09-11T09:51:28.362 INFO:tasks.workunit.client.0.smithi107.stderr:++ local cmd 2024-09-11T09:51:28.362 INFO:tasks.workunit.client.0.smithi107.stderr:+++ id -u 2024-09-11T09:51:28.362 INFO:tasks.workunit.client.0.smithi107.stderr:++ '[' 1000 -eq 0 ']' 2024-09-11T09:51:28.362 INFO:tasks.workunit.client.0.smithi107.stderr:+++ which rbd 2024-09-11T09:51:28.362 INFO:tasks.workunit.client.0.smithi107.stderr:++ cmd=/usr/bin/rbd 2024-09-11T09:51:28.362 INFO:tasks.workunit.client.0.smithi107.stderr:++ shift 2024-09-11T09:51:28.362 INFO:tasks.workunit.client.0.smithi107.stderr:++ sudo -nE /usr/bin/rbd device --device-type nbd map rbd/testrbdnbd27121 2024-09-11T09:51:28.362 INFO:tasks.workunit.client.0.smithi107.stderr:+ DEV=/dev/nbd0 2024-09-11T09:51:28.362 INFO:tasks.workunit.client.0.smithi107.stderr:+ get_pid rbd 2024-09-11T09:51:28.362 INFO:tasks.workunit.client.0.smithi107.stderr:+ local pool=rbd 2024-09-11T09:51:28.363 INFO:tasks.workunit.client.0.smithi107.stderr:+ local ns= 2024-09-11T09:51:28.363 INFO:tasks.workunit.client.0.smithi107.stderr:++ rbd device --device-type nbd --format xml list 2024-09-11T09:51:28.363 INFO:tasks.workunit.client.0.smithi107.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd27121'\''][device='\''/dev/nbd0'\'']/id' 2024-09-11T09:51:28.436 INFO:tasks.workunit.client.0.smithi107.stderr:+ PID=27460 2024-09-11T09:51:28.436 INFO:tasks.workunit.client.0.smithi107.stderr:+ test -n 27460 2024-09-11T09:51:28.436 INFO:tasks.workunit.client.0.smithi107.stderr:+ ps -p 27460 -C rbd-nbd 2024-09-11T09:51:28.451 INFO:tasks.workunit.client.0.smithi107.stdout: PID TTY TIME CMD 2024-09-11T09:51:28.451 INFO:tasks.workunit.client.0.smithi107.stdout: 27460 ? 00:00:00 rbd-nbd 2024-09-11T09:51:28.452 INFO:tasks.workunit.client.0.smithi107.stderr:+ expect_false _sudo rbd-nbd --device /dev/nbd0 map rbd/testrbdnbd27121 2024-09-11T09:51:28.452 INFO:tasks.workunit.client.0.smithi107.stderr:+ _sudo rbd-nbd --device /dev/nbd0 map rbd/testrbdnbd27121 2024-09-11T09:51:28.452 INFO:tasks.workunit.client.0.smithi107.stderr:+ local cmd 2024-09-11T09:51:28.452 INFO:tasks.workunit.client.0.smithi107.stderr:++ id -u 2024-09-11T09:51:28.452 INFO:tasks.workunit.client.0.smithi107.stderr:+ '[' 1000 -eq 0 ']' 2024-09-11T09:51:28.453 INFO:tasks.workunit.client.0.smithi107.stderr:++ which rbd-nbd 2024-09-11T09:51:28.454 INFO:tasks.workunit.client.0.smithi107.stderr:+ cmd=/usr/bin/rbd-nbd 2024-09-11T09:51:28.454 INFO:tasks.workunit.client.0.smithi107.stderr:+ shift 2024-09-11T09:51:28.454 INFO:tasks.workunit.client.0.smithi107.stderr:+ sudo -nE /usr/bin/rbd-nbd --device /dev/nbd0 map rbd/testrbdnbd27121 2024-09-11T09:51:28.461 INFO:tasks.ceph.osd.0.smithi017.stderr:2024-09-11T09:51:28.462+0000 7fb73c0f1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31665) UID: 0 2024-09-11T09:51:28.503 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-11T09:51:28.503 INFO:teuthology.orchestra.run.smithi096.stdout: "ops": [], 2024-09-11T09:51:28.503 INFO:teuthology.orchestra.run.smithi096.stdout: "num_ops": 0 2024-09-11T09:51:28.503 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-11T09:51:28.512 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-11T09:51:28.520 INFO:tasks.workunit.client.0.smithi107.stderr:rbd-nbd: netlink connect failed: Object busy 2024-09-11T09:51:28.529 INFO:tasks.workunit.client.0.smithi107.stderr:+ return 0 2024-09-11T09:51:28.529 INFO:tasks.workunit.client.0.smithi107.stderr:+ dev1=/dev/nbd0 2024-09-11T09:51:28.529 INFO:tasks.workunit.client.0.smithi107.stderr:+ unmap_device /dev/nbd0 27460 2024-09-11T09:51:28.529 INFO:tasks.workunit.client.0.smithi107.stderr:+ local args=/dev/nbd0 2024-09-11T09:51:28.529 INFO:tasks.workunit.client.0.smithi107.stderr:+ local pid=27460 2024-09-11T09:51:28.529 INFO:tasks.workunit.client.0.smithi107.stderr:+ _sudo rbd device --device-type nbd unmap /dev/nbd0 2024-09-11T09:51:28.529 INFO:tasks.workunit.client.0.smithi107.stderr:+ local cmd 2024-09-11T09:51:28.529 INFO:tasks.workunit.client.0.smithi107.stderr:++ id -u 2024-09-11T09:51:28.530 INFO:tasks.workunit.client.0.smithi107.stderr:+ '[' 1000 -eq 0 ']' 2024-09-11T09:51:28.530 INFO:tasks.workunit.client.0.smithi107.stderr:++ which rbd 2024-09-11T09:51:28.531 INFO:tasks.workunit.client.0.smithi107.stderr:+ cmd=/usr/bin/rbd 2024-09-11T09:51:28.532 INFO:tasks.workunit.client.0.smithi107.stderr:+ shift 2024-09-11T09:51:28.532 INFO:tasks.workunit.client.0.smithi107.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd unmap /dev/nbd0 2024-09-11T09:51:28.561 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-09-11T09:51:28.562+0000 7f7b6c7d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31904) UID: 0 2024-09-11T09:51:28.607 INFO:tasks.workunit.client.0.smithi107.stderr:+ rbd device --device-type nbd list 2024-09-11T09:51:28.607 INFO:tasks.workunit.client.0.smithi107.stderr:+ expect_false grep '^27460\b' 2024-09-11T09:51:28.607 INFO:tasks.workunit.client.0.smithi107.stderr:+ grep '^27460\b' 2024-09-11T09:51:28.615 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-11T09:51:28.615 INFO:teuthology.orchestra.run.smithi096.stdout: "ops": [], 2024-09-11T09:51:28.615 INFO:teuthology.orchestra.run.smithi096.stdout: "complaint_time": 30, 2024-09-11T09:51:28.615 INFO:teuthology.orchestra.run.smithi096.stdout: "num_blocked_ops": 0 2024-09-11T09:51:28.616 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-11T09:51:28.625 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-11T09:51:28.653 INFO:tasks.workunit.client.0.smithi107.stderr:+ return 0 2024-09-11T09:51:28.654 INFO:tasks.workunit.client.0.smithi107.stderr:+ ps -C rbd-nbd 2024-09-11T09:51:28.654 INFO:tasks.workunit.client.0.smithi107.stderr:+ expect_false grep '^ *27460\b' 2024-09-11T09:51:28.654 INFO:tasks.workunit.client.0.smithi107.stderr:+ grep '^ *27460\b' 2024-09-11T09:51:28.665 INFO:tasks.workunit.client.0.smithi107.stderr:+ return 0 2024-09-11T09:51:28.665 INFO:tasks.workunit.client.0.smithi107.stderr:+ sleep 0.5 2024-09-11T09:51:28.666 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-09-11T09:51:28.667+0000 7f7b6c7d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31904) UID: 0 2024-09-11T09:51:28.763 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:51:28.764+0000 7f3c96ec9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31909) UID: 0 2024-09-11T09:51:28.770 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-11T09:51:28.770 INFO:teuthology.orchestra.run.smithi096.stdout: "size": 20, 2024-09-11T09:51:28.770 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 600, 2024-09-11T09:51:28.771 INFO:teuthology.orchestra.run.smithi096.stdout: "ops": [ 2024-09-11T09:51:28.771 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.771 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4230.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-11T09:51:28.771 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:18.490736+0000", 2024-09-11T09:51:28.771 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 70.279691350999997, 2024-09-11T09:51:28.771 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0059097569999999999, 2024-09-11T09:51:28.771 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:28.771 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:28.771 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:28.771 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:28.771 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4230", 2024-09-11T09:51:28.771 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.17:0/4101109576", 2024-09-11T09:51:28.771 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 1 2024-09-11T09:51:28.772 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.772 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:28.772 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.772 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:28.772 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.490736+0000", 2024-09-11T09:51:28.772 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:28.772 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.772 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.772 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:28.772 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.490736+0000", 2024-09-11T09:51:28.772 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:28.772 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.772 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.773 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:28.773 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.490742+0000", 2024-09-11T09:51:28.773 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.48e-06 2024-09-11T09:51:28.773 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.773 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.773 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:28.773 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.490749+0000", 2024-09-11T09:51:28.773 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.4569999999999999e-06 2024-09-11T09:51:28.773 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.773 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.773 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:28.773 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.490762+0000", 2024-09-11T09:51:28.773 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.2721000000000001e-05 2024-09-11T09:51:28.774 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.774 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.774 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:28.774 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.490774+0000", 2024-09-11T09:51:28.774 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1823e-05 2024-09-11T09:51:28.774 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.774 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.774 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:28.774 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.490846+0000", 2024-09-11T09:51:28.774 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.1878000000000005e-05 2024-09-11T09:51:28.774 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.774 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.774 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for peered", 2024-09-11T09:51:28.775 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.490857+0000", 2024-09-11T09:51:28.775 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1017e-05 2024-09-11T09:51:28.775 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.775 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.775 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:28.775 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.493856+0000", 2024-09-11T09:51:28.775 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00299885 2024-09-11T09:51:28.775 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.775 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.775 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:28.775 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.494020+0000", 2024-09-11T09:51:28.775 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00016436599999999999 2024-09-11T09:51:28.775 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.776 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.776 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 2", 2024-09-11T09:51:28.776 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.494186+0000", 2024-09-11T09:51:28.776 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00016562799999999999 2024-09-11T09:51:28.776 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.776 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.776 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:28.776 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.495407+0000", 2024-09-11T09:51:28.776 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0012217420000000001 2024-09-11T09:51:28.776 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.776 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.776 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:28.777 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.496579+0000", 2024-09-11T09:51:28.777 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001171844 2024-09-11T09:51:28.777 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.777 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.777 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:28.777 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.496620+0000", 2024-09-11T09:51:28.777 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.0479000000000003e-05 2024-09-11T09:51:28.777 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.777 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.777 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:28.777 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.496646+0000", 2024-09-11T09:51:28.777 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.6472e-05 2024-09-11T09:51:28.777 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:28.778 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:28.778 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:28.778 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.778 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.778 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4230.0:5 2.2 2:5cea7035:::rbd_info:head [create,write 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-11T09:51:28.778 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:19.492662+0000", 2024-09-11T09:51:28.778 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 69.277765729999999, 2024-09-11T09:51:28.778 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0043824759999999997, 2024-09-11T09:51:28.778 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:28.778 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:28.778 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:28.778 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:28.779 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4230", 2024-09-11T09:51:28.779 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.17:0/4101109576", 2024-09-11T09:51:28.779 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 5 2024-09-11T09:51:28.779 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.779 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:28.779 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.780 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:28.780 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.492662+0000", 2024-09-11T09:51:28.780 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:28.780 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.780 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.780 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:28.780 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.492662+0000", 2024-09-11T09:51:28.780 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:28.780 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.780 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.780 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:28.780 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.492664+0000", 2024-09-11T09:51:28.780 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.3599999999999999e-06 2024-09-11T09:51:28.780 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.781 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.781 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:28.781 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.492675+0000", 2024-09-11T09:51:28.781 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1182000000000001e-05 2024-09-11T09:51:28.781 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.781 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.781 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:28.781 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.492693+0000", 2024-09-11T09:51:28.781 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.7031999999999999e-05 2024-09-11T09:51:28.782 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.782 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.782 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:28.782 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.492702+0000", 2024-09-11T09:51:28.782 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.5540000000000003e-06 2024-09-11T09:51:28.782 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.782 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.782 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:28.782 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.492767+0000", 2024-09-11T09:51:28.782 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.5411000000000001e-05 2024-09-11T09:51:28.782 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.782 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.782 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op must wait for map", 2024-09-11T09:51:28.783 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.492773+0000", 2024-09-11T09:51:28.783 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.1039999999999998e-06 2024-09-11T09:51:28.783 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.783 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.783 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:28.783 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.494354+0000", 2024-09-11T09:51:28.783 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0015814049999999999 2024-09-11T09:51:28.783 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.783 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.784 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:28.784 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.494476+0000", 2024-09-11T09:51:28.784 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000122089 2024-09-11T09:51:28.784 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.784 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.784 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 1", 2024-09-11T09:51:28.784 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.494577+0000", 2024-09-11T09:51:28.784 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000100536 2024-09-11T09:51:28.784 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.784 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.784 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:28.784 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.495770+0000", 2024-09-11T09:51:28.785 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0011932469999999999 2024-09-11T09:51:28.785 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.785 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.785 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:28.785 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.496966+0000", 2024-09-11T09:51:28.785 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001196003 2024-09-11T09:51:28.785 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.785 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.785 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:28.785 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.497011+0000", 2024-09-11T09:51:28.785 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.5229999999999999e-05 2024-09-11T09:51:28.785 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.785 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.786 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:28.786 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.497044+0000", 2024-09-11T09:51:28.786 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.3322999999999999e-05 2024-09-11T09:51:28.786 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:28.786 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:28.786 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:28.786 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.786 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.786 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4230.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-09-11T09:51:28.786 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:20.498634+0000", 2024-09-11T09:51:28.786 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 68.271793329999994, 2024-09-11T09:51:28.786 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0050648539999999997, 2024-09-11T09:51:28.787 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:28.787 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:28.787 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:28.787 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:28.787 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4230", 2024-09-11T09:51:28.787 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.17:0/4101109576", 2024-09-11T09:51:28.787 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 7 2024-09-11T09:51:28.787 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.787 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:28.787 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.788 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:28.788 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.498634+0000", 2024-09-11T09:51:28.788 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:28.788 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.788 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.788 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:28.788 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.498634+0000", 2024-09-11T09:51:28.788 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:28.788 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.788 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.788 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:28.788 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.498637+0000", 2024-09-11T09:51:28.788 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.1919999999999999e-06 2024-09-11T09:51:28.789 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.789 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.789 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:28.789 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.498648+0000", 2024-09-11T09:51:28.789 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1116e-05 2024-09-11T09:51:28.789 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.789 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.789 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:28.789 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.498666+0000", 2024-09-11T09:51:28.790 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.8411999999999999e-05 2024-09-11T09:51:28.790 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.790 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.790 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:28.790 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.498675+0000", 2024-09-11T09:51:28.790 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.4059999999999994e-06 2024-09-11T09:51:28.790 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.790 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.790 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:28.791 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.498745+0000", 2024-09-11T09:51:28.791 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.9764000000000005e-05 2024-09-11T09:51:28.791 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.791 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.791 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op must wait for map", 2024-09-11T09:51:28.791 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.498754+0000", 2024-09-11T09:51:28.791 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.5969999999999992e-06 2024-09-11T09:51:28.791 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.791 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.791 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:28.791 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.500385+0000", 2024-09-11T09:51:28.791 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0016308360000000001 2024-09-11T09:51:28.791 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.792 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.792 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:28.792 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.500473+0000", 2024-09-11T09:51:28.792 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.7966000000000004e-05 2024-09-11T09:51:28.792 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.792 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.792 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 1", 2024-09-11T09:51:28.792 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.500591+0000", 2024-09-11T09:51:28.792 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000117895 2024-09-11T09:51:28.792 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.792 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.792 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:28.793 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.501685+0000", 2024-09-11T09:51:28.793 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001094886 2024-09-11T09:51:28.793 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.793 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.793 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:28.793 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.503627+0000", 2024-09-11T09:51:28.793 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001941199 2024-09-11T09:51:28.793 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.793 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.793 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:28.793 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.503667+0000", 2024-09-11T09:51:28.793 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.0645000000000001e-05 2024-09-11T09:51:28.793 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.793 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.794 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:28.794 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.503699+0000", 2024-09-11T09:51:28.794 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.1940000000000003e-05 2024-09-11T09:51:28.794 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:28.794 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:28.794 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:28.794 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.794 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.794 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4688.0:32 2.2 2:45941fd2:::rbd_data.1250f5e25b80.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:28.794 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:22.896607+0000", 2024-09-11T09:51:28.794 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 5.873820727, 2024-09-11T09:51:28.794 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.038655179999999997, 2024-09-11T09:51:28.794 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:28.795 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:28.795 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:28.795 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:28.795 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4688", 2024-09-11T09:51:28.795 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:51:28.795 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 32 2024-09-11T09:51:28.795 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.795 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:28.795 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.795 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:28.795 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.896607+0000", 2024-09-11T09:51:28.795 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:28.795 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.795 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.796 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:28.796 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.896607+0000", 2024-09-11T09:51:28.796 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:28.796 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.796 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.796 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:28.796 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.896608+0000", 2024-09-11T09:51:28.796 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.0690000000000001e-06 2024-09-11T09:51:28.796 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.796 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.797 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:28.797 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.914461+0000", 2024-09-11T09:51:28.797 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.017852903 2024-09-11T09:51:28.797 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.797 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.797 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:28.797 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.914495+0000", 2024-09-11T09:51:28.797 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.4301000000000002e-05 2024-09-11T09:51:28.797 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.797 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.797 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:28.797 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.914522+0000", 2024-09-11T09:51:28.797 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.7127e-05 2024-09-11T09:51:28.798 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.798 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.798 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:28.798 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.914583+0000", 2024-09-11T09:51:28.798 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.0730999999999998e-05 2024-09-11T09:51:28.798 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.798 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.798 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:28.798 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.914709+0000", 2024-09-11T09:51:28.798 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00012590400000000001 2024-09-11T09:51:28.798 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.799 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.799 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 1", 2024-09-11T09:51:28.799 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.914809+0000", 2024-09-11T09:51:28.799 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.9882000000000004e-05 2024-09-11T09:51:28.799 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.799 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.799 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:28.800 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.932551+0000", 2024-09-11T09:51:28.800 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.017742048999999999 2024-09-11T09:51:28.800 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.800 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.800 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:28.800 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.935144+0000", 2024-09-11T09:51:28.800 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0025927630000000001 2024-09-11T09:51:28.800 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.800 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.800 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:28.801 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.935205+0000", 2024-09-11T09:51:28.801 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.1634e-05 2024-09-11T09:51:28.801 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.801 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.801 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:28.801 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.935262+0000", 2024-09-11T09:51:28.801 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.6817000000000002e-05 2024-09-11T09:51:28.801 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:28.801 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:28.801 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:28.801 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.801 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.801 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_repop(client.4688.0:40 2.7 e53/38 2:fe41411c:::rbd_object_map.1250f5e25b80:head v 53'9, mlcod=53'7)", 2024-09-11T09:51:28.801 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:22.920457+0000", 2024-09-11T09:51:28.802 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 5.8499706360000001, 2024-09-11T09:51:28.802 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.011132817, 2024-09-11T09:51:28.802 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:28.802 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:28.802 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:28.802 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:28.802 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.802 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:28.802 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.920457+0000", 2024-09-11T09:51:28.802 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:28.802 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.802 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.802 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:28.802 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.920457+0000", 2024-09-11T09:51:28.803 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:28.803 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.803 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.803 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:28.803 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.920459+0000", 2024-09-11T09:51:28.803 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.6729999999999999e-06 2024-09-11T09:51:28.803 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.803 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.803 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:28.803 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.920469+0000", 2024-09-11T09:51:28.803 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.944e-06 2024-09-11T09:51:28.803 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.803 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.804 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:28.804 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.920495+0000", 2024-09-11T09:51:28.804 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.6401999999999999e-05 2024-09-11T09:51:28.804 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.804 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.804 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:28.804 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.920506+0000", 2024-09-11T09:51:28.804 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.133e-05 2024-09-11T09:51:28.804 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.804 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.805 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:28.805 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.920581+0000", 2024-09-11T09:51:28.805 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.4932999999999996e-05 2024-09-11T09:51:28.805 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.805 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.805 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:28.805 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.920612+0000", 2024-09-11T09:51:28.805 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.0537999999999998e-05 2024-09-11T09:51:28.805 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.805 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.805 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:28.805 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.931537+0000", 2024-09-11T09:51:28.805 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.010925239 2024-09-11T09:51:28.805 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.806 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.806 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:28.806 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.931590+0000", 2024-09-11T09:51:28.806 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.2757999999999999e-05 2024-09-11T09:51:28.806 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:28.806 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:28.806 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:28.806 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.806 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.806 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_repop(client.4688.0:41 2.7 e53/38 2:fe41411c:::rbd_object_map.1250f5e25b80:head v 53'10, mlcod=53'7)", 2024-09-11T09:51:28.806 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:22.921422+0000", 2024-09-11T09:51:28.806 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 5.8490052200000004, 2024-09-11T09:51:28.806 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.013734942, 2024-09-11T09:51:28.806 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:28.807 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:28.807 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:28.807 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:28.807 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.807 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:28.807 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921422+0000", 2024-09-11T09:51:28.807 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:28.807 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.807 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.807 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:28.808 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921422+0000", 2024-09-11T09:51:28.808 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:28.808 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.808 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.808 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:28.808 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921426+0000", 2024-09-11T09:51:28.808 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.6150000000000001e-06 2024-09-11T09:51:28.808 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.808 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.808 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:28.808 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921435+0000", 2024-09-11T09:51:28.808 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.9840000000000007e-06 2024-09-11T09:51:28.808 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.809 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.809 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:28.809 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921444+0000", 2024-09-11T09:51:28.809 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.6000000000000007e-06 2024-09-11T09:51:28.809 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.809 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.809 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:28.809 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921450+0000", 2024-09-11T09:51:28.809 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.4690000000000002e-06 2024-09-11T09:51:28.809 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.809 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.809 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:28.810 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921511+0000", 2024-09-11T09:51:28.810 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.0779999999999997e-05 2024-09-11T09:51:28.810 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.810 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.810 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:28.810 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921531+0000", 2024-09-11T09:51:28.810 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.0302999999999999e-05 2024-09-11T09:51:28.810 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.810 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.810 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:28.810 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.935115+0000", 2024-09-11T09:51:28.810 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.013583471 2024-09-11T09:51:28.810 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.810 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.811 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:28.811 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.935157+0000", 2024-09-11T09:51:28.811 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.2719999999999998e-05 2024-09-11T09:51:28.811 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:28.811 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:28.811 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:28.811 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.811 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.811 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4688.0:42 2.3 2:c54742d8:::rbd_data.1250f5e25b80.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:28.811 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:22.932560+0000", 2024-09-11T09:51:28.811 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 5.8378674769999996, 2024-09-11T09:51:28.811 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.017798198000000001, 2024-09-11T09:51:28.811 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:28.811 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:28.812 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:28.812 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:28.812 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4688", 2024-09-11T09:51:28.812 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:51:28.812 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 42 2024-09-11T09:51:28.812 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.812 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:28.812 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.812 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:28.812 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.932560+0000", 2024-09-11T09:51:28.812 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:28.812 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.812 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.813 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:28.813 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.932560+0000", 2024-09-11T09:51:28.813 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:28.813 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.813 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.813 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:28.813 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.932562+0000", 2024-09-11T09:51:28.813 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.17e-06 2024-09-11T09:51:28.813 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.813 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.813 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:28.814 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.938851+0000", 2024-09-11T09:51:28.814 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.006288971 2024-09-11T09:51:28.814 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.814 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.814 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:28.814 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.938865+0000", 2024-09-11T09:51:28.814 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.365e-05 2024-09-11T09:51:28.814 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.814 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.814 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:28.814 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.938871+0000", 2024-09-11T09:51:28.814 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.8850000000000004e-06 2024-09-11T09:51:28.814 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.815 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.815 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:28.815 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.938926+0000", 2024-09-11T09:51:28.815 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.4691999999999997e-05 2024-09-11T09:51:28.815 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.815 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.815 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:28.815 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.939059+0000", 2024-09-11T09:51:28.815 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00013310899999999999 2024-09-11T09:51:28.815 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.815 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.815 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 2", 2024-09-11T09:51:28.815 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.939171+0000", 2024-09-11T09:51:28.815 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000112718 2024-09-11T09:51:28.815 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.816 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.816 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:28.816 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.950262+0000", 2024-09-11T09:51:28.816 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.011091063 2024-09-11T09:51:28.816 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.816 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.816 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:28.816 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.950274+0000", 2024-09-11T09:51:28.816 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1963e-05 2024-09-11T09:51:28.816 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.817 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.817 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:28.817 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.950318+0000", 2024-09-11T09:51:28.817 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.3519999999999997e-05 2024-09-11T09:51:28.817 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.817 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.817 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:28.817 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.950358+0000", 2024-09-11T09:51:28.817 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.0457000000000002e-05 2024-09-11T09:51:28.817 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:28.817 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:28.818 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:28.818 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.818 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.818 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_repop(client.4694.0:34 2.7 e53/38 2:f80bfc0c:ns::rbd_data.1256a0c02879.0000000000000002:head v 53'25, mlcod=53'23)", 2024-09-11T09:51:28.818 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.094944+0000", 2024-09-11T09:51:28.818 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 5.6754840849999999, 2024-09-11T09:51:28.818 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.010896785000000001, 2024-09-11T09:51:28.818 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:28.818 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:28.818 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:28.818 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:28.818 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.818 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:28.818 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.094944+0000", 2024-09-11T09:51:28.819 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:28.819 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.819 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.819 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:28.819 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.094944+0000", 2024-09-11T09:51:28.819 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:28.819 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.819 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.819 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:28.819 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.094944+0000", 2024-09-11T09:51:28.819 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.0699999999999996e-07 2024-09-11T09:51:28.819 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.820 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.820 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:28.820 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.099060+0000", 2024-09-11T09:51:28.820 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0041155580000000001 2024-09-11T09:51:28.820 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.820 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.820 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:28.820 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.099082+0000", 2024-09-11T09:51:28.820 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.2323999999999998e-05 2024-09-11T09:51:28.820 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.820 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.820 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:28.820 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.099094+0000", 2024-09-11T09:51:28.821 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1535000000000001e-05 2024-09-11T09:51:28.821 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.821 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.821 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:28.821 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.099160+0000", 2024-09-11T09:51:28.821 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.6449e-05 2024-09-11T09:51:28.821 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.821 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.821 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:28.821 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.099183+0000", 2024-09-11T09:51:28.821 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.3108e-05 2024-09-11T09:51:28.821 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.822 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.822 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:28.822 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.105776+0000", 2024-09-11T09:51:28.822 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0065922760000000002 2024-09-11T09:51:28.822 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.822 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.822 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:28.822 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.105840+0000", 2024-09-11T09:51:28.822 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.4727999999999994e-05 2024-09-11T09:51:28.822 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:28.822 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:28.822 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:28.822 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.823 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.823 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_repop(client.4694.0:35 2.7 e53/38 2:e8766325:ns::rbd_object_map.1256a0c02879:head v 53'26, mlcod=53'23)", 2024-09-11T09:51:28.823 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.100964+0000", 2024-09-11T09:51:28.823 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 5.6694635570000003, 2024-09-11T09:51:28.823 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0049649359999999997, 2024-09-11T09:51:28.823 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:28.823 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:28.823 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:28.823 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:28.823 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.823 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:28.823 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.100964+0000", 2024-09-11T09:51:28.823 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:28.823 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.824 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.824 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:28.824 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.100964+0000", 2024-09-11T09:51:28.824 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:28.824 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.824 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.824 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:28.824 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.100966+0000", 2024-09-11T09:51:28.824 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.6330000000000001e-06 2024-09-11T09:51:28.824 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.824 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.824 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:28.825 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.100979+0000", 2024-09-11T09:51:28.825 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.2813000000000001e-05 2024-09-11T09:51:28.825 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.825 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.825 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:28.825 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.100992+0000", 2024-09-11T09:51:28.825 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.3414999999999999e-05 2024-09-11T09:51:28.825 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.825 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.825 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:28.825 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.101002+0000", 2024-09-11T09:51:28.825 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.0159e-05 2024-09-11T09:51:28.825 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.826 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.826 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:28.826 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.102365+0000", 2024-09-11T09:51:28.826 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001362912 2024-09-11T09:51:28.826 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.826 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.826 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:28.826 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.102382+0000", 2024-09-11T09:51:28.826 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.7433000000000001e-05 2024-09-11T09:51:28.826 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.826 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.826 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:28.826 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.105901+0000", 2024-09-11T09:51:28.827 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.003518476 2024-09-11T09:51:28.827 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.827 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.827 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:28.827 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.105929+0000", 2024-09-11T09:51:28.827 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.8095000000000001e-05 2024-09-11T09:51:28.827 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:28.827 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:28.827 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:28.827 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.827 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.827 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_repop(client.4694.0:36 2.7 e53/38 2:e8766325:ns::rbd_object_map.1256a0c02879:head v 53'27, mlcod=53'23)", 2024-09-11T09:51:28.827 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.101688+0000", 2024-09-11T09:51:28.828 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 5.6687398810000005, 2024-09-11T09:51:28.828 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0042633289999999997, 2024-09-11T09:51:28.828 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:28.828 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:28.828 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:28.828 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:28.828 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.828 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:28.828 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.101688+0000", 2024-09-11T09:51:28.828 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:28.828 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.828 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.828 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:28.829 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.101688+0000", 2024-09-11T09:51:28.829 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:28.829 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.829 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.829 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:28.829 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.101689+0000", 2024-09-11T09:51:28.829 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.9299999999999997e-07 2024-09-11T09:51:28.829 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.829 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.829 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:28.829 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.101694+0000", 2024-09-11T09:51:28.829 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.6779999999999997e-06 2024-09-11T09:51:28.829 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.830 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.830 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:28.830 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.101707+0000", 2024-09-11T09:51:28.830 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.2549e-05 2024-09-11T09:51:28.830 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.830 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.830 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:28.830 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.101714+0000", 2024-09-11T09:51:28.830 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.9210000000000003e-06 2024-09-11T09:51:28.830 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.830 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.830 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:28.830 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.102635+0000", 2024-09-11T09:51:28.831 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00092109400000000001 2024-09-11T09:51:28.831 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.831 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.831 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:28.831 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.102649+0000", 2024-09-11T09:51:28.831 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.3913e-05 2024-09-11T09:51:28.831 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.831 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.831 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:28.831 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.105931+0000", 2024-09-11T09:51:28.831 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.003281876 2024-09-11T09:51:28.831 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.831 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.831 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:28.832 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.105951+0000", 2024-09-11T09:51:28.832 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.0505000000000001e-05 2024-09-11T09:51:28.832 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:28.832 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:28.832 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:28.832 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.832 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.832 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_repop(client.4694.0:37 2.7 e53/38 2:e8766325:ns::rbd_object_map.1256a0c02879:head v 53'28, mlcod=53'23)", 2024-09-11T09:51:28.832 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.102274+0000", 2024-09-11T09:51:28.832 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 5.6681538309999997, 2024-09-11T09:51:28.832 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0036897850000000001, 2024-09-11T09:51:28.832 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:28.832 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:28.833 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:28.833 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:28.833 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.833 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:28.833 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.102274+0000", 2024-09-11T09:51:28.833 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:28.833 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.833 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.833 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:28.833 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.102274+0000", 2024-09-11T09:51:28.833 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:28.833 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.834 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.834 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:28.834 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.102274+0000", 2024-09-11T09:51:28.834 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.1699999999999998e-07 2024-09-11T09:51:28.834 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.834 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.834 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:28.834 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.102280+0000", 2024-09-11T09:51:28.834 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.1200000000000001e-06 2024-09-11T09:51:28.834 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.834 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.834 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:28.835 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.102291+0000", 2024-09-11T09:51:28.835 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1032e-05 2024-09-11T09:51:28.835 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.835 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.835 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:28.835 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.102297+0000", 2024-09-11T09:51:28.835 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.4919999999999998e-06 2024-09-11T09:51:28.835 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.835 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.835 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:28.835 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.102902+0000", 2024-09-11T09:51:28.835 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00060473799999999996 2024-09-11T09:51:28.835 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.835 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.836 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:28.836 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.102913+0000", 2024-09-11T09:51:28.836 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.131e-05 2024-09-11T09:51:28.836 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.836 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.836 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:28.836 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.105952+0000", 2024-09-11T09:51:28.836 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0030386200000000001 2024-09-11T09:51:28.836 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.836 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.836 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:28.836 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.105964+0000", 2024-09-11T09:51:28.837 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1856e-05 2024-09-11T09:51:28.837 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:28.837 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:28.837 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:28.837 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.837 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.837 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4694.0:41 2.2 2:5e2505c4:ns::rbd_data.1256a0c02879.0000000000000003:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:28.837 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.107511+0000", 2024-09-11T09:51:28.837 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 5.6629164039999997, 2024-09-11T09:51:28.837 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.018571066000000001, 2024-09-11T09:51:28.837 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:28.837 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:28.837 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:28.837 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:28.838 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4694", 2024-09-11T09:51:28.838 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:51:28.838 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 41 2024-09-11T09:51:28.838 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.838 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:28.838 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.838 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:28.838 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.107511+0000", 2024-09-11T09:51:28.838 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:28.838 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.838 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.838 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:28.838 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.107511+0000", 2024-09-11T09:51:28.838 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:28.839 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.839 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.839 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:28.839 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.107513+0000", 2024-09-11T09:51:28.839 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.3489999999999999e-06 2024-09-11T09:51:28.839 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.839 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.839 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:28.839 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.112704+0000", 2024-09-11T09:51:28.839 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0051918509999999999 2024-09-11T09:51:28.839 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.840 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.840 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:28.840 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.112718+0000", 2024-09-11T09:51:28.840 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.359e-05 2024-09-11T09:51:28.840 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.840 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.840 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:28.840 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.112725+0000", 2024-09-11T09:51:28.840 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.9940000000000003e-06 2024-09-11T09:51:28.840 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.840 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.840 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:28.840 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.112788+0000", 2024-09-11T09:51:28.841 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.2447999999999996e-05 2024-09-11T09:51:28.841 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.841 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.841 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:28.841 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.112937+0000", 2024-09-11T09:51:28.841 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00014921499999999999 2024-09-11T09:51:28.841 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.841 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.841 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 1", 2024-09-11T09:51:28.841 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.113085+0000", 2024-09-11T09:51:28.841 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000148085 2024-09-11T09:51:28.842 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.842 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.842 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:28.842 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.124318+0000", 2024-09-11T09:51:28.842 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.011233158 2024-09-11T09:51:28.842 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.842 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.842 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:28.842 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.125957+0000", 2024-09-11T09:51:28.843 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001638701 2024-09-11T09:51:28.843 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.843 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.843 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:28.843 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.126029+0000", 2024-09-11T09:51:28.843 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.2125000000000005e-05 2024-09-11T09:51:28.843 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.843 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.843 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:28.843 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.126082+0000", 2024-09-11T09:51:28.843 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.3550000000000001e-05 2024-09-11T09:51:28.843 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:28.843 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:28.843 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:28.844 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.844 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.844 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4694.0:44 2.3 2:cd1a7d1c:ns::rbd_data.1256a0c02879.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:28.844 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.112744+0000", 2024-09-11T09:51:28.844 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 5.6576838370000004, 2024-09-11T09:51:28.844 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.021672212999999999, 2024-09-11T09:51:28.844 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:28.844 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:28.844 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:28.844 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:28.844 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4694", 2024-09-11T09:51:28.844 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:51:28.844 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 44 2024-09-11T09:51:28.844 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.845 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:28.845 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.845 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:28.845 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.112744+0000", 2024-09-11T09:51:28.845 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:28.845 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.845 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.845 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:28.845 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.112744+0000", 2024-09-11T09:51:28.845 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:28.845 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.845 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.845 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:28.845 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.112745+0000", 2024-09-11T09:51:28.845 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.2900000000000002e-07 2024-09-11T09:51:28.846 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.846 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.846 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:28.846 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.119929+0000", 2024-09-11T09:51:28.846 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0071846230000000002 2024-09-11T09:51:28.846 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.846 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.846 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:28.846 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.119949+0000", 2024-09-11T09:51:28.846 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.9264000000000001e-05 2024-09-11T09:51:28.846 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.846 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.846 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:28.846 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.119959+0000", 2024-09-11T09:51:28.847 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.0465999999999999e-05 2024-09-11T09:51:28.847 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.847 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.847 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:28.847 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.120028+0000", 2024-09-11T09:51:28.847 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.9318000000000003e-05 2024-09-11T09:51:28.847 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.847 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.847 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:28.847 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.120160+0000", 2024-09-11T09:51:28.847 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000132015 2024-09-11T09:51:28.847 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.847 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.847 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 2", 2024-09-11T09:51:28.847 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.120260+0000", 2024-09-11T09:51:28.848 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.9397999999999999e-05 2024-09-11T09:51:28.848 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.848 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.848 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:28.848 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.134246+0000", 2024-09-11T09:51:28.848 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.013986721000000001 2024-09-11T09:51:28.848 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.848 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.848 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:28.848 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.134295+0000", 2024-09-11T09:51:28.848 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.8649000000000001e-05 2024-09-11T09:51:28.849 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.849 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.849 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:28.849 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.134361+0000", 2024-09-11T09:51:28.849 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.6081999999999995e-05 2024-09-11T09:51:28.849 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.849 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.849 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:28.849 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.134416+0000", 2024-09-11T09:51:28.849 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.4848000000000001e-05 2024-09-11T09:51:28.850 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:28.850 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:28.850 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:28.850 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.850 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.850 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4694.0:46 2.3 2:ce0e1469:ns::rbd_data.1256a0c02879.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:28.850 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.119993+0000", 2024-09-11T09:51:28.850 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 5.6504343610000003, 2024-09-11T09:51:28.850 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.025840610999999999, 2024-09-11T09:51:28.850 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:28.850 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:28.851 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:28.851 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:28.851 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4694", 2024-09-11T09:51:28.851 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:51:28.851 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 46 2024-09-11T09:51:28.851 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.851 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:28.851 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.851 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:28.851 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.119993+0000", 2024-09-11T09:51:28.852 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:28.852 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.852 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.852 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:28.852 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.119993+0000", 2024-09-11T09:51:28.852 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:28.852 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.852 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.852 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:28.852 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.119995+0000", 2024-09-11T09:51:28.852 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1930000000000001e-06 2024-09-11T09:51:28.852 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.852 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.853 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:28.853 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.128458+0000", 2024-09-11T09:51:28.853 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0084632630000000004 2024-09-11T09:51:28.853 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.853 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.853 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:28.853 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.128478+0000", 2024-09-11T09:51:28.853 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.0353e-05 2024-09-11T09:51:28.853 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.853 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.853 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:28.853 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.128491+0000", 2024-09-11T09:51:28.853 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.2954e-05 2024-09-11T09:51:28.853 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.854 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.854 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:28.854 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.128558+0000", 2024-09-11T09:51:28.854 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.7076999999999995e-05 2024-09-11T09:51:28.854 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.854 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.854 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:28.854 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.128708+0000", 2024-09-11T09:51:28.854 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00014939599999999999 2024-09-11T09:51:28.855 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.855 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.855 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 2", 2024-09-11T09:51:28.855 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.128866+0000", 2024-09-11T09:51:28.855 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000158145 2024-09-11T09:51:28.855 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.855 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.855 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:28.855 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.141045+0000", 2024-09-11T09:51:28.856 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.012179331999999999 2024-09-11T09:51:28.856 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.856 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.856 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:28.856 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.145742+0000", 2024-09-11T09:51:28.856 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0046972050000000003 2024-09-11T09:51:28.856 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.856 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.856 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:28.856 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.145779+0000", 2024-09-11T09:51:28.856 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.6544000000000001e-05 2024-09-11T09:51:28.856 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.856 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.857 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:28.857 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.145834+0000", 2024-09-11T09:51:28.857 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.5148999999999997e-05 2024-09-11T09:51:28.857 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:28.857 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:28.857 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:28.857 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.857 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.857 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4694.0:48 2.3 2:d2090b70:ns::rbd_data.1256a0c02879.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:28.857 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.130540+0000", 2024-09-11T09:51:28.857 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 5.6398880949999999, 2024-09-11T09:51:28.857 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.017167789999999999, 2024-09-11T09:51:28.857 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:28.858 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:28.858 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:28.858 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:28.858 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4694", 2024-09-11T09:51:28.858 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:51:28.858 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 48 2024-09-11T09:51:28.858 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.858 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:28.858 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.858 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:28.858 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.130540+0000", 2024-09-11T09:51:28.858 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:28.858 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.859 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.859 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:28.859 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.130540+0000", 2024-09-11T09:51:28.859 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:28.859 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.859 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.859 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:28.859 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.130542+0000", 2024-09-11T09:51:28.859 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.9209999999999999e-06 2024-09-11T09:51:28.859 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.860 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.860 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:28.860 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.134691+0000", 2024-09-11T09:51:28.860 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0041498739999999996 2024-09-11T09:51:28.860 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.860 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.860 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:28.860 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.134705+0000", 2024-09-11T09:51:28.860 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.3669000000000001e-05 2024-09-11T09:51:28.860 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.860 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.860 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:28.860 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.134712+0000", 2024-09-11T09:51:28.861 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.8410000000000002e-06 2024-09-11T09:51:28.861 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.861 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.861 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:28.861 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.134754+0000", 2024-09-11T09:51:28.861 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.2524999999999997e-05 2024-09-11T09:51:28.861 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.861 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.861 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:28.861 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.134875+0000", 2024-09-11T09:51:28.861 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000120332 2024-09-11T09:51:28.862 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.862 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.862 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 2", 2024-09-11T09:51:28.862 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.135005+0000", 2024-09-11T09:51:28.862 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00013024900000000001 2024-09-11T09:51:28.862 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.862 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.862 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:28.862 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.145971+0000", 2024-09-11T09:51:28.862 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.010966203000000001 2024-09-11T09:51:28.862 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.862 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.863 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:28.863 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.147600+0000", 2024-09-11T09:51:28.863 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001628382 2024-09-11T09:51:28.863 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.863 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.863 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:28.863 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.147635+0000", 2024-09-11T09:51:28.863 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.5420000000000003e-05 2024-09-11T09:51:28.863 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.863 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.863 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:28.863 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.147707+0000", 2024-09-11T09:51:28.864 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.2373999999999995e-05 2024-09-11T09:51:28.864 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:28.864 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:28.864 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:28.864 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.864 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.864 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_repop(client.4694.0:49 2.7 e53/38 2:e8766325:ns::rbd_object_map.1256a0c02879:head v 53'33, mlcod=53'31)", 2024-09-11T09:51:28.864 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.135391+0000", 2024-09-11T09:51:28.864 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 5.6350365299999998, 2024-09-11T09:51:28.864 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0051189859999999998, 2024-09-11T09:51:28.864 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:28.864 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:28.865 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:28.865 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:28.865 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.865 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:28.865 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.135391+0000", 2024-09-11T09:51:28.865 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:28.865 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.865 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.865 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:28.865 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.135391+0000", 2024-09-11T09:51:28.865 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:28.865 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.866 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.866 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:28.866 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.135393+0000", 2024-09-11T09:51:28.866 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.4020000000000001e-06 2024-09-11T09:51:28.866 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.866 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.866 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:28.866 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.135400+0000", 2024-09-11T09:51:28.866 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.5129999999999999e-06 2024-09-11T09:51:28.866 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.866 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.866 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:28.867 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.135411+0000", 2024-09-11T09:51:28.867 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1038e-05 2024-09-11T09:51:28.867 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.867 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.867 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:28.867 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.135418+0000", 2024-09-11T09:51:28.867 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.1489999999999997e-06 2024-09-11T09:51:28.867 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.867 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.867 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:28.868 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.135470+0000", 2024-09-11T09:51:28.868 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.2021999999999998e-05 2024-09-11T09:51:28.868 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.868 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.868 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:28.868 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.135501+0000", 2024-09-11T09:51:28.868 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.0349999999999999e-05 2024-09-11T09:51:28.868 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.868 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.868 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:28.868 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.140453+0000", 2024-09-11T09:51:28.869 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0049522409999999996 2024-09-11T09:51:28.869 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.869 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.869 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:28.869 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.140510+0000", 2024-09-11T09:51:28.869 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.7271000000000001e-05 2024-09-11T09:51:28.869 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:28.869 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:28.869 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:28.869 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.869 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.869 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4694.0:54 2.3 2:c44f0253:ns::rbd_data.1256a0c02879.000000000000000b:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:28.870 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.150151+0000", 2024-09-11T09:51:28.870 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 5.6202766789999998, 2024-09-11T09:51:28.870 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.015103484, 2024-09-11T09:51:28.870 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:28.870 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:28.870 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:28.870 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:28.870 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4694", 2024-09-11T09:51:28.870 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:51:28.870 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 54 2024-09-11T09:51:28.870 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.870 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:28.870 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.871 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:28.871 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.150151+0000", 2024-09-11T09:51:28.871 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:28.871 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.871 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.871 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:28.871 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.150151+0000", 2024-09-11T09:51:28.871 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:28.871 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.871 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.871 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:28.871 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.150152+0000", 2024-09-11T09:51:28.871 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.2550000000000001e-06 2024-09-11T09:51:28.872 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.872 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.872 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:28.872 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.154528+0000", 2024-09-11T09:51:28.872 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0043758219999999997 2024-09-11T09:51:28.872 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.872 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.872 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:28.872 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.154541+0000", 2024-09-11T09:51:28.872 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.3414999999999999e-05 2024-09-11T09:51:28.872 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.872 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.872 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:28.873 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.154551+0000", 2024-09-11T09:51:28.873 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.4350000000000003e-06 2024-09-11T09:51:28.873 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.873 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.873 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:28.873 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.154615+0000", 2024-09-11T09:51:28.873 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.3627000000000006e-05 2024-09-11T09:51:28.873 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.873 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.873 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:28.873 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.154717+0000", 2024-09-11T09:51:28.873 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000102729 2024-09-11T09:51:28.873 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.874 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.874 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 2", 2024-09-11T09:51:28.874 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.154826+0000", 2024-09-11T09:51:28.874 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000108409 2024-09-11T09:51:28.874 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.874 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.874 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:28.874 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.165094+0000", 2024-09-11T09:51:28.874 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.010268210999999999 2024-09-11T09:51:28.874 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.874 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.874 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:28.874 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.165138+0000", 2024-09-11T09:51:28.874 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.4078e-05 2024-09-11T09:51:28.875 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.875 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.875 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:28.875 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.165204+0000", 2024-09-11T09:51:28.875 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.5716000000000006e-05 2024-09-11T09:51:28.875 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.875 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.876 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:28.876 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.165254+0000", 2024-09-11T09:51:28.876 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.0787e-05 2024-09-11T09:51:28.876 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:28.876 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:28.876 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:28.877 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.877 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.877 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_repop(client.4694.0:55 2.7 e53/38 2:e8766325:ns::rbd_object_map.1256a0c02879:head v 53'37, mlcod=53'34)", 2024-09-11T09:51:28.877 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.150221+0000", 2024-09-11T09:51:28.877 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 5.6202068140000003, 2024-09-11T09:51:28.877 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00530015, 2024-09-11T09:51:28.878 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:28.878 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:28.878 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:28.878 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:28.878 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.878 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:28.878 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.150221+0000", 2024-09-11T09:51:28.878 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:28.879 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.879 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.879 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:28.879 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.150221+0000", 2024-09-11T09:51:28.879 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:28.879 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.879 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.879 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:28.879 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.150221+0000", 2024-09-11T09:51:28.879 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.9100000000000004e-07 2024-09-11T09:51:28.879 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.879 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.879 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:28.880 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.150226+0000", 2024-09-11T09:51:28.880 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.2250000000000002e-06 2024-09-11T09:51:28.880 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.880 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.880 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:28.880 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.150251+0000", 2024-09-11T09:51:28.880 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.4987e-05 2024-09-11T09:51:28.880 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.880 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.880 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:28.880 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.150258+0000", 2024-09-11T09:51:28.880 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.0790000000000003e-06 2024-09-11T09:51:28.881 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.881 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.881 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:28.881 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.150305+0000", 2024-09-11T09:51:28.881 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.6879000000000003e-05 2024-09-11T09:51:28.881 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.881 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.881 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:28.881 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.150320+0000", 2024-09-11T09:51:28.881 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.5268999999999999e-05 2024-09-11T09:51:28.881 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.881 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.881 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:28.882 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.155493+0000", 2024-09-11T09:51:28.882 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0051731390000000002 2024-09-11T09:51:28.882 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.882 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.882 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:28.882 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.155521+0000", 2024-09-11T09:51:28.882 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.8081000000000001e-05 2024-09-11T09:51:28.882 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:28.882 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:28.882 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:28.882 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.882 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.882 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4694.0:56 2.3 2:c462f029:ns::rbd_data.1256a0c02879.000000000000000c:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:28.883 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.154576+0000", 2024-09-11T09:51:28.883 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 5.6158518380000002, 2024-09-11T09:51:28.883 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.020781621, 2024-09-11T09:51:28.883 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:28.883 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:28.883 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:28.883 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:28.883 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4694", 2024-09-11T09:51:28.883 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:51:28.883 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 56 2024-09-11T09:51:28.883 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.884 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:28.884 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.884 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:28.884 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.154576+0000", 2024-09-11T09:51:28.884 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:28.884 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.884 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.884 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:28.884 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.154576+0000", 2024-09-11T09:51:28.884 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:28.884 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.885 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.885 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:28.885 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.154577+0000", 2024-09-11T09:51:28.885 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.2249999999999999e-06 2024-09-11T09:51:28.885 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.885 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.885 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:28.885 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.158783+0000", 2024-09-11T09:51:28.885 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0042063889999999996 2024-09-11T09:51:28.885 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.885 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.885 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:28.886 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.158803+0000", 2024-09-11T09:51:28.886 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.9576999999999999e-05 2024-09-11T09:51:28.886 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.886 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.886 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:28.886 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.158815+0000", 2024-09-11T09:51:28.886 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.2221999999999999e-05 2024-09-11T09:51:28.886 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.886 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.886 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:28.886 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.158902+0000", 2024-09-11T09:51:28.887 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.6390000000000005e-05 2024-09-11T09:51:28.887 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.887 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.887 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:28.887 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.159141+0000", 2024-09-11T09:51:28.887 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000238886 2024-09-11T09:51:28.887 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.887 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.887 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 2", 2024-09-11T09:51:28.887 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.159279+0000", 2024-09-11T09:51:28.887 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000138073 2024-09-11T09:51:28.888 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.888 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.888 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:28.888 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.173297+0000", 2024-09-11T09:51:28.888 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.014018113 2024-09-11T09:51:28.888 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.888 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.888 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:28.888 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.175278+0000", 2024-09-11T09:51:28.888 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0019817390000000002 2024-09-11T09:51:28.888 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.888 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.889 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:28.889 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.175316+0000", 2024-09-11T09:51:28.889 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.7829e-05 2024-09-11T09:51:28.889 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.889 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.889 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:28.889 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.175357+0000", 2024-09-11T09:51:28.889 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.1177999999999998e-05 2024-09-11T09:51:28.889 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:28.889 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:28.889 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:28.889 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.890 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.890 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4694.0:59 2.2 2:41adefc9:ns::rbd_data.1256a0c02879.000000000000000e:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:28.890 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.158843+0000", 2024-09-11T09:51:28.890 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 5.611584272, 2024-09-11T09:51:28.890 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.017874207, 2024-09-11T09:51:28.890 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:28.890 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:28.890 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:28.890 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:28.890 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4694", 2024-09-11T09:51:28.890 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:51:28.890 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 59 2024-09-11T09:51:28.891 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.891 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:28.891 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.891 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:28.891 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.158843+0000", 2024-09-11T09:51:28.891 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:28.891 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.891 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.891 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:28.891 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.158843+0000", 2024-09-11T09:51:28.891 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:28.891 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.892 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.892 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:28.892 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.158845+0000", 2024-09-11T09:51:28.892 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.7439999999999999e-06 2024-09-11T09:51:28.892 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.892 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.892 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:28.892 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.162713+0000", 2024-09-11T09:51:28.892 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0038678829999999999 2024-09-11T09:51:28.892 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.892 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.892 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:28.892 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.162728+0000", 2024-09-11T09:51:28.893 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.5114e-05 2024-09-11T09:51:28.893 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.893 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.893 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:28.893 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.162737+0000", 2024-09-11T09:51:28.893 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.1099999999999992e-06 2024-09-11T09:51:28.893 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.893 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.893 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:28.893 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.162799+0000", 2024-09-11T09:51:28.893 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.1934999999999996e-05 2024-09-11T09:51:28.893 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.894 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.894 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:28.894 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.162911+0000", 2024-09-11T09:51:28.894 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000111867 2024-09-11T09:51:28.894 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.894 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.894 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 1", 2024-09-11T09:51:28.894 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.163030+0000", 2024-09-11T09:51:28.894 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000119108 2024-09-11T09:51:28.894 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.894 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.894 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:28.895 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.174086+0000", 2024-09-11T09:51:28.895 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.01105543 2024-09-11T09:51:28.895 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.895 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.895 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:28.895 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.176614+0000", 2024-09-11T09:51:28.895 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0025284940000000001 2024-09-11T09:51:28.895 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.895 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.895 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:28.895 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.176672+0000", 2024-09-11T09:51:28.895 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.7892000000000001e-05 2024-09-11T09:51:28.895 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:28.896 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:28.896 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:28.896 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.176718+0000", 2024-09-11T09:51:28.896 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.5630000000000002e-05 2024-09-11T09:51:28.896 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:28.896 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:28.896 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:28.896 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:28.896 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:28.896 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-11T09:51:28.899 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-11T09:51:28.900 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-4 --journal-path /var/lib/ceph/osd/ceph-4/journal --force --op remove --pgid 3.3 2024-09-11T09:51:28.900 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:51:28.898+0000 7f3c96ec9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31909) UID: 0 2024-09-11T09:51:28.998 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-09-11T09:51:28.999+0000 7f7b6c7d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31904) UID: 0 2024-09-11T09:51:29.020 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:51:29.021 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [], 2024-09-11T09:51:29.021 INFO:teuthology.orchestra.run.smithi017.stdout: "num_ops": 0 2024-09-11T09:51:29.021 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:51:29.030 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-11T09:51:29.099 INFO:tasks.ceph.osd.2.smithi017.stderr:2024-09-11T09:51:29.100+0000 7fd20b170640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31672) UID: 0 2024-09-11T09:51:29.138 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:51:29.138 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [], 2024-09-11T09:51:29.138 INFO:teuthology.orchestra.run.smithi017.stdout: "complaint_time": 30, 2024-09-11T09:51:29.138 INFO:teuthology.orchestra.run.smithi017.stdout: "num_blocked_ops": 0 2024-09-11T09:51:29.138 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:51:29.147 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-11T09:51:29.166 INFO:tasks.workunit.client.0.smithi107.stderr:+ DEV= 2024-09-11T09:51:29.166 INFO:tasks.workunit.client.0.smithi107.stderr:++ _sudo rbd-nbd --device /dev/nbd0 map rbd/testrbdnbd27121 2024-09-11T09:51:29.167 INFO:tasks.workunit.client.0.smithi107.stderr:++ local cmd 2024-09-11T09:51:29.167 INFO:tasks.workunit.client.0.smithi107.stderr:+++ id -u 2024-09-11T09:51:29.167 INFO:tasks.workunit.client.0.smithi107.stderr:++ '[' 1000 -eq 0 ']' 2024-09-11T09:51:29.167 INFO:tasks.workunit.client.0.smithi107.stderr:+++ which rbd-nbd 2024-09-11T09:51:29.169 INFO:tasks.workunit.client.0.smithi107.stderr:++ cmd=/usr/bin/rbd-nbd 2024-09-11T09:51:29.169 INFO:tasks.workunit.client.0.smithi107.stderr:++ shift 2024-09-11T09:51:29.169 INFO:tasks.workunit.client.0.smithi107.stderr:++ sudo -nE /usr/bin/rbd-nbd --device /dev/nbd0 map rbd/testrbdnbd27121 2024-09-11T09:51:29.231 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:51:29.232+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:51:29.247 INFO:tasks.workunit.client.0.smithi107.stderr:+ DEV=/dev/nbd0 2024-09-11T09:51:29.247 INFO:tasks.workunit.client.0.smithi107.stderr:+ '[' /dev/nbd0 = /dev/nbd0 ']' 2024-09-11T09:51:29.247 INFO:tasks.workunit.client.0.smithi107.stderr:+ rbd device --device-type nbd list 2024-09-11T09:51:29.247 INFO:tasks.workunit.client.0.smithi107.stderr:+ grep testrbdnbd27121 2024-09-11T09:51:29.301 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:51:29.301 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 20, 2024-09-11T09:51:29.301 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 600, 2024-09-11T09:51:29.301 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [ 2024-09-11T09:51:29.301 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.301 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4688.0:34 2.4 e53/32 2:2367385a:::rbd_data.1250f5e25b80.0000000000000002:head v 53'6, mlcod=53'4)", 2024-09-11T09:51:29.301 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.907415+0000", 2024-09-11T09:51:29.302 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 6.3931654130000002, 2024-09-11T09:51:29.302 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.011285778999999999, 2024-09-11T09:51:29.302 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:29.302 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:29.302 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:29.302 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:29.302 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.302 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:29.302 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.907415+0000", 2024-09-11T09:51:29.302 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:29.302 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.302 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.302 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:29.303 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.907415+0000", 2024-09-11T09:51:29.303 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:29.303 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.303 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.303 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:29.303 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.907417+0000", 2024-09-11T09:51:29.303 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.2910000000000002e-06 2024-09-11T09:51:29.303 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.303 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.303 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:29.303 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.910477+0000", 2024-09-11T09:51:29.303 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0030597829999999999 2024-09-11T09:51:29.303 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.303 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.304 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:29.304 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.910501+0000", 2024-09-11T09:51:29.304 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.3332e-05 2024-09-11T09:51:29.304 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.304 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.304 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:29.304 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.910514+0000", 2024-09-11T09:51:29.304 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3957e-05 2024-09-11T09:51:29.304 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.304 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.304 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:29.304 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.910612+0000", 2024-09-11T09:51:29.304 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.7075000000000001e-05 2024-09-11T09:51:29.305 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.305 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.305 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:29.305 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.910640+0000", 2024-09-11T09:51:29.305 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.8768999999999998e-05 2024-09-11T09:51:29.305 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.305 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.305 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:29.305 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.918640+0000", 2024-09-11T09:51:29.305 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0079991899999999998 2024-09-11T09:51:29.305 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.305 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.306 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:29.306 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.918701+0000", 2024-09-11T09:51:29.306 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.1382000000000003e-05 2024-09-11T09:51:29.306 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:29.306 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:29.306 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:29.306 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.306 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.306 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4688.0:38 2.5 e53/32 2:abde2694:::rbd_data.1250f5e25b80.0000000000000004:head v 53'1, mlcod=0'0)", 2024-09-11T09:51:29.306 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.915432+0000", 2024-09-11T09:51:29.306 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 6.3851490330000003, 2024-09-11T09:51:29.306 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.015849652999999998, 2024-09-11T09:51:29.306 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:29.306 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:29.307 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:29.307 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:29.307 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.307 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:29.307 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.915432+0000", 2024-09-11T09:51:29.307 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:29.307 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.307 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.307 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:29.307 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.915432+0000", 2024-09-11T09:51:29.307 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:29.307 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.308 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.308 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:29.308 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.915432+0000", 2024-09-11T09:51:29.308 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.91e-07 2024-09-11T09:51:29.308 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.308 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.308 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:29.308 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.924943+0000", 2024-09-11T09:51:29.308 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0095110630000000002 2024-09-11T09:51:29.308 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.308 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.308 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:29.308 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.924960+0000", 2024-09-11T09:51:29.308 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6915999999999999e-05 2024-09-11T09:51:29.309 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.309 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.309 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:29.309 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.924968+0000", 2024-09-11T09:51:29.309 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.6999999999999997e-06 2024-09-11T09:51:29.309 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.309 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.309 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:29.309 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.925025+0000", 2024-09-11T09:51:29.309 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.6949e-05 2024-09-11T09:51:29.309 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.309 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.309 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:29.310 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.925043+0000", 2024-09-11T09:51:29.310 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7784000000000001e-05 2024-09-11T09:51:29.310 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.310 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.310 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:29.310 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.931237+0000", 2024-09-11T09:51:29.310 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0061935389999999996 2024-09-11T09:51:29.310 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.310 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.310 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:29.310 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.931281+0000", 2024-09-11T09:51:29.310 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.4511000000000001e-05 2024-09-11T09:51:29.310 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:29.310 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:29.311 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:29.311 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.311 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.311 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:40 2.7 2:fe41411c:::rbd_object_map.1250f5e25b80:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:29.311 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.919652+0000", 2024-09-11T09:51:29.311 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 6.380928366, 2024-09-11T09:51:29.311 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.012187073, 2024-09-11T09:51:29.311 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:29.311 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:29.311 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:29.311 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:29.311 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:51:29.311 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:51:29.311 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 40 2024-09-11T09:51:29.312 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.312 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:29.312 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.312 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:29.312 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919652+0000", 2024-09-11T09:51:29.312 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:29.312 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.312 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.312 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:29.312 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919652+0000", 2024-09-11T09:51:29.312 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:29.312 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.312 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.313 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:29.313 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919654+0000", 2024-09-11T09:51:29.313 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.88e-06 2024-09-11T09:51:29.313 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.313 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.313 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:29.313 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919669+0000", 2024-09-11T09:51:29.313 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.5271e-05 2024-09-11T09:51:29.313 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.313 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.313 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:29.313 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919687+0000", 2024-09-11T09:51:29.313 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7448999999999998e-05 2024-09-11T09:51:29.313 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.314 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.314 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:29.314 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919695+0000", 2024-09-11T09:51:29.314 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.1149999999999994e-06 2024-09-11T09:51:29.314 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.314 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.314 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:29.314 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919757+0000", 2024-09-11T09:51:29.314 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.2348999999999995e-05 2024-09-11T09:51:29.314 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.314 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.314 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:29.314 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919875+0000", 2024-09-11T09:51:29.315 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00011750499999999999 2024-09-11T09:51:29.315 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.315 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.315 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 5", 2024-09-11T09:51:29.315 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.920201+0000", 2024-09-11T09:51:29.315 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00032628099999999999 2024-09-11T09:51:29.315 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.315 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.315 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:29.315 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.922089+0000", 2024-09-11T09:51:29.315 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.001887978 2024-09-11T09:51:29.315 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.315 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.315 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:29.316 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.931752+0000", 2024-09-11T09:51:29.316 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0096626309999999997 2024-09-11T09:51:29.316 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.316 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.316 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:29.316 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.931794+0000", 2024-09-11T09:51:29.316 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.2849999999999998e-05 2024-09-11T09:51:29.316 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.316 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.316 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:29.316 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.931839+0000", 2024-09-11T09:51:29.316 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.4764e-05 2024-09-11T09:51:29.316 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:29.317 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:29.317 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:29.317 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.317 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.317 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:41 2.7 2:fe41411c:::rbd_object_map.1250f5e25b80:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:29.317 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.919716+0000", 2024-09-11T09:51:29.317 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 6.3808645219999995, 2024-09-11T09:51:29.317 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.015641605999999999, 2024-09-11T09:51:29.317 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:29.317 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:29.317 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:29.317 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:29.317 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:51:29.318 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:51:29.318 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 41 2024-09-11T09:51:29.318 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.318 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:29.318 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.318 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:29.318 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919716+0000", 2024-09-11T09:51:29.318 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:29.318 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.318 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.318 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:29.318 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919716+0000", 2024-09-11T09:51:29.318 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:29.318 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.319 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.319 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:29.319 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919716+0000", 2024-09-11T09:51:29.319 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.9999999999999998e-07 2024-09-11T09:51:29.319 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.319 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.319 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:29.319 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919720+0000", 2024-09-11T09:51:29.319 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.6440000000000002e-06 2024-09-11T09:51:29.319 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.319 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.319 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:29.319 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919729+0000", 2024-09-11T09:51:29.319 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.1759999999999999e-06 2024-09-11T09:51:29.320 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.320 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.320 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:29.320 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919736+0000", 2024-09-11T09:51:29.320 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.5390000000000004e-06 2024-09-11T09:51:29.320 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.320 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.320 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:29.320 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.920669+0000", 2024-09-11T09:51:29.320 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00093275599999999997 2024-09-11T09:51:29.320 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.320 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.320 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:29.321 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.920797+0000", 2024-09-11T09:51:29.321 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00012893400000000001 2024-09-11T09:51:29.321 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.321 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.321 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 5", 2024-09-11T09:51:29.321 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.921269+0000", 2024-09-11T09:51:29.321 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00047128600000000001 2024-09-11T09:51:29.321 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.321 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.321 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:29.321 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.923198+0000", 2024-09-11T09:51:29.321 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0019289369999999999 2024-09-11T09:51:29.321 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.322 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.322 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:29.322 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.935300+0000", 2024-09-11T09:51:29.322 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.012102392 2024-09-11T09:51:29.322 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.322 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.322 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:29.322 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.935332+0000", 2024-09-11T09:51:29.322 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.2292000000000002e-05 2024-09-11T09:51:29.322 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.322 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.322 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:29.322 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.935358+0000", 2024-09-11T09:51:29.322 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.525e-05 2024-09-11T09:51:29.323 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:29.323 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:29.323 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:29.323 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.323 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.323 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:48 2.7 2:fe41411c:::rbd_object_map.1250f5e25b80:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:29.323 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.938799+0000", 2024-09-11T09:51:29.323 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 6.3617814130000001, 2024-09-11T09:51:29.323 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0041642340000000002, 2024-09-11T09:51:29.323 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:29.323 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:29.323 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:29.323 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:29.323 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:51:29.324 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:51:29.324 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 48 2024-09-11T09:51:29.324 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.324 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:29.324 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.324 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:29.324 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.938799+0000", 2024-09-11T09:51:29.324 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:29.324 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.324 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.324 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:29.324 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.938799+0000", 2024-09-11T09:51:29.324 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:29.325 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.325 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.325 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:29.325 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.938800+0000", 2024-09-11T09:51:29.325 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.71e-07 2024-09-11T09:51:29.325 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.325 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.325 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:29.325 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.938805+0000", 2024-09-11T09:51:29.325 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.3140000000000003e-06 2024-09-11T09:51:29.325 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.325 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.325 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:29.325 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.938816+0000", 2024-09-11T09:51:29.326 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0231000000000001e-05 2024-09-11T09:51:29.326 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.326 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.326 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:29.326 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.938822+0000", 2024-09-11T09:51:29.326 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.9550000000000003e-06 2024-09-11T09:51:29.326 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.326 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.326 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:29.326 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.938868+0000", 2024-09-11T09:51:29.326 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.5562999999999997e-05 2024-09-11T09:51:29.326 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.326 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.326 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:29.327 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.938918+0000", 2024-09-11T09:51:29.327 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.0317e-05 2024-09-11T09:51:29.327 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.327 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.327 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 5", 2024-09-11T09:51:29.327 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.939114+0000", 2024-09-11T09:51:29.327 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00019538300000000001 2024-09-11T09:51:29.327 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.327 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.327 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:29.327 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.939663+0000", 2024-09-11T09:51:29.327 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00054932199999999998 2024-09-11T09:51:29.327 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.327 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.328 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:29.328 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.942910+0000", 2024-09-11T09:51:29.328 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0032473839999999999 2024-09-11T09:51:29.328 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.328 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.328 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:29.328 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.942940+0000", 2024-09-11T09:51:29.328 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.9306999999999999e-05 2024-09-11T09:51:29.328 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.328 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.328 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:29.328 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.942963+0000", 2024-09-11T09:51:29.328 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.3587e-05 2024-09-11T09:51:29.329 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:29.329 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:29.329 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:29.329 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.329 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.329 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4688.0:49 2.5 e53/32 2:b6dd9fc9:::rbd_data.1250f5e25b80.0000000000000009:head v 53'2, mlcod=0'0)", 2024-09-11T09:51:29.329 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.944712+0000", 2024-09-11T09:51:29.329 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 6.3558685739999996, 2024-09-11T09:51:29.329 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.016185069999999999, 2024-09-11T09:51:29.329 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:29.329 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:29.329 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:29.329 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:29.330 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.330 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:29.330 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.944712+0000", 2024-09-11T09:51:29.330 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:29.330 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.330 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.330 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:29.330 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.944712+0000", 2024-09-11T09:51:29.330 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:29.330 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.330 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.330 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:29.330 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.944713+0000", 2024-09-11T09:51:29.331 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0139999999999999e-06 2024-09-11T09:51:29.331 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.331 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.331 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:29.331 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.955518+0000", 2024-09-11T09:51:29.331 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.01080542 2024-09-11T09:51:29.331 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.331 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.331 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:29.331 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.955536+0000", 2024-09-11T09:51:29.331 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7818000000000001e-05 2024-09-11T09:51:29.331 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.331 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.331 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:29.332 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.955544+0000", 2024-09-11T09:51:29.332 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.1699999999999997e-06 2024-09-11T09:51:29.332 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.332 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.332 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:29.332 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.955607+0000", 2024-09-11T09:51:29.332 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.2451000000000002e-05 2024-09-11T09:51:29.332 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.332 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.332 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:29.332 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.955632+0000", 2024-09-11T09:51:29.332 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.5335000000000001e-05 2024-09-11T09:51:29.332 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.332 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.333 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:29.333 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.960826+0000", 2024-09-11T09:51:29.333 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0051938560000000002 2024-09-11T09:51:29.333 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.333 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.333 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:29.333 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.960897+0000", 2024-09-11T09:51:29.333 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.1006000000000004e-05 2024-09-11T09:51:29.333 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:29.333 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:29.333 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:29.333 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.333 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.333 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:53 2.7 2:fe41411c:::rbd_object_map.1250f5e25b80:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:29.334 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.951291+0000", 2024-09-11T09:51:29.334 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 6.3492895840000001, 2024-09-11T09:51:29.334 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0044964999999999996, 2024-09-11T09:51:29.334 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:29.334 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:29.334 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:29.334 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:29.334 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:51:29.334 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:51:29.334 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 53 2024-09-11T09:51:29.334 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.334 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:29.334 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.335 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:29.335 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951291+0000", 2024-09-11T09:51:29.335 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:29.335 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.335 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.335 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:29.335 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951291+0000", 2024-09-11T09:51:29.335 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:29.335 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.335 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.335 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:29.335 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951293+0000", 2024-09-11T09:51:29.335 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7889999999999999e-06 2024-09-11T09:51:29.335 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.336 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.336 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:29.336 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951302+0000", 2024-09-11T09:51:29.336 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.9819999999999997e-06 2024-09-11T09:51:29.336 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.336 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.336 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:29.336 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951324+0000", 2024-09-11T09:51:29.336 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.2141e-05 2024-09-11T09:51:29.336 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.336 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.336 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:29.336 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951338+0000", 2024-09-11T09:51:29.337 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3750999999999999e-05 2024-09-11T09:51:29.337 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.337 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.337 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:29.337 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951409+0000", 2024-09-11T09:51:29.337 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.1658999999999999e-05 2024-09-11T09:51:29.337 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.337 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.337 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:29.337 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951517+0000", 2024-09-11T09:51:29.337 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00010746799999999999 2024-09-11T09:51:29.337 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.337 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.337 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 5", 2024-09-11T09:51:29.338 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951942+0000", 2024-09-11T09:51:29.338 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00042524300000000001 2024-09-11T09:51:29.338 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.338 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.338 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:29.338 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.952872+0000", 2024-09-11T09:51:29.338 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00092976300000000001 2024-09-11T09:51:29.338 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.338 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.338 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:29.338 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.955682+0000", 2024-09-11T09:51:29.338 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0028099459999999998 2024-09-11T09:51:29.338 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.339 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.339 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:29.339 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.955746+0000", 2024-09-11T09:51:29.339 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.4084999999999994e-05 2024-09-11T09:51:29.339 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.339 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.339 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:29.339 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.955787+0000", 2024-09-11T09:51:29.339 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.1672999999999999e-05 2024-09-11T09:51:29.339 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:29.339 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:29.339 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:29.339 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.339 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.340 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4688.0:52 2.4 e53/32 2:26d5c2f0:::rbd_data.1250f5e25b80.000000000000000b:head v 53'8, mlcod=53'6)", 2024-09-11T09:51:29.340 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.963677+0000", 2024-09-11T09:51:29.340 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 6.3369040290000003, 2024-09-11T09:51:29.340 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0089437969999999999, 2024-09-11T09:51:29.340 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:29.340 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:29.340 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:29.340 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:29.340 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.340 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:29.340 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.963677+0000", 2024-09-11T09:51:29.340 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:29.340 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.341 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.341 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:29.341 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.963677+0000", 2024-09-11T09:51:29.341 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:29.341 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.341 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.341 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:29.341 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.963677+0000", 2024-09-11T09:51:29.341 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.7199999999999998e-07 2024-09-11T09:51:29.341 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.341 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.341 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:29.341 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.965506+0000", 2024-09-11T09:51:29.342 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.001829064 2024-09-11T09:51:29.342 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.342 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.342 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:29.342 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.965538+0000", 2024-09-11T09:51:29.342 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.1306999999999997e-05 2024-09-11T09:51:29.342 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.342 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.342 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:29.342 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.965553+0000", 2024-09-11T09:51:29.342 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.4924e-05 2024-09-11T09:51:29.342 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.342 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.342 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:29.343 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.965663+0000", 2024-09-11T09:51:29.343 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000110315 2024-09-11T09:51:29.343 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.343 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.343 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:29.343 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.965692+0000", 2024-09-11T09:51:29.343 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.934e-05 2024-09-11T09:51:29.343 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.343 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.343 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:29.343 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.972537+0000", 2024-09-11T09:51:29.343 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0068444969999999997 2024-09-11T09:51:29.343 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.344 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.344 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:29.344 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.972620+0000", 2024-09-11T09:51:29.344 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.3578000000000005e-05 2024-09-11T09:51:29.344 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:29.344 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:29.344 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:29.344 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.344 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.344 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4688.0:54 2.4 e53/32 2:239e915b:::rbd_data.1250f5e25b80.000000000000000c:head v 53'9, mlcod=53'7)", 2024-09-11T09:51:29.344 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.987028+0000", 2024-09-11T09:51:29.344 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 6.313552112, 2024-09-11T09:51:29.345 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0094155650000000007, 2024-09-11T09:51:29.345 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:29.345 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:29.345 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:29.345 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:29.345 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.345 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:29.345 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.987028+0000", 2024-09-11T09:51:29.345 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:29.345 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.345 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.345 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:29.345 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.987028+0000", 2024-09-11T09:51:29.346 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:29.346 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.346 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.346 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:29.346 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.987030+0000", 2024-09-11T09:51:29.346 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6130000000000001e-06 2024-09-11T09:51:29.346 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.346 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.346 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:29.346 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.988575+0000", 2024-09-11T09:51:29.346 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.001545193 2024-09-11T09:51:29.346 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.346 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.346 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:29.347 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.988604+0000", 2024-09-11T09:51:29.347 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.9173e-05 2024-09-11T09:51:29.347 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.347 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.347 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:29.347 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.988620+0000", 2024-09-11T09:51:29.347 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.5517999999999999e-05 2024-09-11T09:51:29.347 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.347 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.347 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:29.347 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.988704+0000", 2024-09-11T09:51:29.347 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.3588e-05 2024-09-11T09:51:29.348 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.348 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.348 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:29.348 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.988735+0000", 2024-09-11T09:51:29.348 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.1643000000000002e-05 2024-09-11T09:51:29.348 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.348 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.348 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:29.348 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.996383+0000", 2024-09-11T09:51:29.348 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0076478099999999997 2024-09-11T09:51:29.348 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.348 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.348 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:29.348 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.996444+0000", 2024-09-11T09:51:29.349 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.1026999999999997e-05 2024-09-11T09:51:29.349 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:29.349 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:29.349 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:29.349 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.349 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.349 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4688.0:56 2.4 e53/32 2:2167012f:::rbd_data.1250f5e25b80.000000000000000d:head v 53'10, mlcod=53'7)", 2024-09-11T09:51:29.349 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.993894+0000", 2024-09-11T09:51:29.349 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 6.3066865700000001, 2024-09-11T09:51:29.349 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0099268559999999995, 2024-09-11T09:51:29.349 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:29.349 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:29.349 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:29.350 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:29.350 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.350 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:29.350 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.993894+0000", 2024-09-11T09:51:29.350 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:29.350 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.350 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.350 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:29.350 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.993894+0000", 2024-09-11T09:51:29.350 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:29.350 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.350 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.350 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:29.351 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.993896+0000", 2024-09-11T09:51:29.351 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6249999999999999e-06 2024-09-11T09:51:29.351 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.351 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.351 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:29.351 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.997606+0000", 2024-09-11T09:51:29.351 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.003710737 2024-09-11T09:51:29.351 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.351 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.351 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:29.351 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.997628+0000", 2024-09-11T09:51:29.351 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.1971e-05 2024-09-11T09:51:29.351 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.352 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.352 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:29.352 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.997640+0000", 2024-09-11T09:51:29.352 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1481000000000001e-05 2024-09-11T09:51:29.352 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.352 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.352 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:29.352 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.997712+0000", 2024-09-11T09:51:29.352 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.2255000000000005e-05 2024-09-11T09:51:29.352 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.352 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.352 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:29.352 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.997738+0000", 2024-09-11T09:51:29.353 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.6050999999999998e-05 2024-09-11T09:51:29.353 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.353 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.353 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:29.353 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.003779+0000", 2024-09-11T09:51:29.353 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0060410960000000001 2024-09-11T09:51:29.353 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.353 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.353 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:29.353 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.003821+0000", 2024-09-11T09:51:29.353 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.1640000000000001e-05 2024-09-11T09:51:29.353 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:29.353 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:29.354 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:29.354 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.354 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.354 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "pg_update_log_missing(2.4 epoch 53/32 rep_tid 16 entries 53'11 (0'0) error 2:2906e6a7:::rbd_header.1250f5e25b80:head by client.4688.0:59 0.000000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 53'7)", 2024-09-11T09:51:29.354 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.998956+0000", 2024-09-11T09:51:29.354 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 6.3016244009999998, 2024-09-11T09:51:29.354 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.004934127, 2024-09-11T09:51:29.354 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:29.354 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:29.354 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "reached pg", 2024-09-11T09:51:29.354 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:29.354 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.354 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:29.354 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.998956+0000", 2024-09-11T09:51:29.355 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:29.355 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.355 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.355 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:29.355 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.998956+0000", 2024-09-11T09:51:29.355 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:29.355 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.355 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.355 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:29.355 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.998957+0000", 2024-09-11T09:51:29.355 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.139e-06 2024-09-11T09:51:29.355 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.355 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.356 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:29.356 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.998985+0000", 2024-09-11T09:51:29.356 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.7912e-05 2024-09-11T09:51:29.356 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.356 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.356 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:29.356 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.999011+0000", 2024-09-11T09:51:29.356 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.6061e-05 2024-09-11T09:51:29.356 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.356 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.356 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:29.356 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.999028+0000", 2024-09-11T09:51:29.356 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7065999999999999e-05 2024-09-11T09:51:29.357 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.357 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.357 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:29.357 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.001919+0000", 2024-09-11T09:51:29.357 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.002891167 2024-09-11T09:51:29.357 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.357 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.357 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:29.357 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.003890+0000", 2024-09-11T09:51:29.357 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0019707819999999999 2024-09-11T09:51:29.357 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:29.357 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:29.357 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:29.358 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.358 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.358 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4694.0:34 2.7 2:f80bfc0c:ns::rbd_data.1256a0c02879.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:29.358 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.085183+0000", 2024-09-11T09:51:29.358 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 6.215397888, 2024-09-11T09:51:29.358 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.020908599, 2024-09-11T09:51:29.358 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:29.358 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:29.358 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:29.358 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:29.358 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4694", 2024-09-11T09:51:29.358 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:51:29.359 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 34 2024-09-11T09:51:29.359 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.359 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:29.359 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.359 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:29.359 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.085183+0000", 2024-09-11T09:51:29.359 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:29.359 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.359 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.359 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:29.359 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.085183+0000", 2024-09-11T09:51:29.359 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:29.359 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.360 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.360 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:29.360 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.085183+0000", 2024-09-11T09:51:29.360 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.4300000000000002e-07 2024-09-11T09:51:29.360 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.360 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.360 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:29.360 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094337+0000", 2024-09-11T09:51:29.360 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0091532769999999996 2024-09-11T09:51:29.360 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.360 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.360 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:29.361 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094362+0000", 2024-09-11T09:51:29.361 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.5287999999999999e-05 2024-09-11T09:51:29.361 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.361 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.361 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:29.361 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094375+0000", 2024-09-11T09:51:29.361 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2446000000000001e-05 2024-09-11T09:51:29.361 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.361 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.361 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:29.361 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094455+0000", 2024-09-11T09:51:29.361 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.0092999999999994e-05 2024-09-11T09:51:29.361 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.362 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.362 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:29.362 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094618+0000", 2024-09-11T09:51:29.362 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000163747 2024-09-11T09:51:29.362 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.362 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.362 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 5", 2024-09-11T09:51:29.362 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094783+0000", 2024-09-11T09:51:29.362 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000165038 2024-09-11T09:51:29.362 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.362 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.362 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:29.362 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.105853+0000", 2024-09-11T09:51:29.363 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.011069294 2024-09-11T09:51:29.363 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.363 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.363 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:29.363 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.106004+0000", 2024-09-11T09:51:29.363 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00015123900000000001 2024-09-11T09:51:29.363 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.363 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.363 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:29.363 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.106062+0000", 2024-09-11T09:51:29.363 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.8205000000000003e-05 2024-09-11T09:51:29.363 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.363 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.363 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:29.364 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.106091+0000", 2024-09-11T09:51:29.364 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.9128999999999998e-05 2024-09-11T09:51:29.364 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:29.364 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:29.364 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:29.364 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.364 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.364 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4694.0:35 2.7 2:e8766325:ns::rbd_object_map.1256a0c02879:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:29.364 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.094405+0000", 2024-09-11T09:51:29.364 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 6.2061752520000004, 2024-09-11T09:51:29.364 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.012623564, 2024-09-11T09:51:29.364 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:29.364 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:29.365 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:29.365 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:29.365 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4694", 2024-09-11T09:51:29.365 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:51:29.365 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 35 2024-09-11T09:51:29.365 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.365 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:29.365 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.365 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:29.365 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094405+0000", 2024-09-11T09:51:29.365 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:29.365 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.365 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.365 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:29.366 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094405+0000", 2024-09-11T09:51:29.366 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:29.366 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.366 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.366 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:29.366 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094407+0000", 2024-09-11T09:51:29.366 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.739e-06 2024-09-11T09:51:29.366 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.366 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.366 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:29.366 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094411+0000", 2024-09-11T09:51:29.366 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.3340000000000003e-06 2024-09-11T09:51:29.366 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.366 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.367 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:29.367 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094419+0000", 2024-09-11T09:51:29.367 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.8909999999999995e-06 2024-09-11T09:51:29.367 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.367 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.367 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:29.367 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094424+0000", 2024-09-11T09:51:29.367 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.7639999999999996e-06 2024-09-11T09:51:29.367 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.367 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.367 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:29.367 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.100209+0000", 2024-09-11T09:51:29.367 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0057850489999999996 2024-09-11T09:51:29.367 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.368 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.368 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:29.368 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.100341+0000", 2024-09-11T09:51:29.368 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000131438 2024-09-11T09:51:29.368 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.368 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.368 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 5", 2024-09-11T09:51:29.368 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.100805+0000", 2024-09-11T09:51:29.368 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00046471499999999998 2024-09-11T09:51:29.368 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.368 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.368 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:29.368 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.106148+0000", 2024-09-11T09:51:29.369 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0053426460000000004 2024-09-11T09:51:29.369 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.369 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.369 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:29.369 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.106929+0000", 2024-09-11T09:51:29.369 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00078161300000000001 2024-09-11T09:51:29.369 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.369 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.369 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:29.369 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.106985+0000", 2024-09-11T09:51:29.369 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.5223999999999998e-05 2024-09-11T09:51:29.369 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.369 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.369 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:29.370 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.107029+0000", 2024-09-11T09:51:29.370 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.4150999999999998e-05 2024-09-11T09:51:29.370 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:29.370 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:29.370 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:29.370 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.370 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.370 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4694.0:36 2.7 2:e8766325:ns::rbd_object_map.1256a0c02879:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:29.370 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.094452+0000", 2024-09-11T09:51:29.370 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 6.2061282420000001, 2024-09-11T09:51:29.370 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.012680383, 2024-09-11T09:51:29.370 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:29.370 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:29.370 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:29.371 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:29.371 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4694", 2024-09-11T09:51:29.371 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:51:29.371 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 36 2024-09-11T09:51:29.371 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.371 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:29.371 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.371 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:29.371 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094452+0000", 2024-09-11T09:51:29.371 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:29.371 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.371 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.371 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:29.372 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094452+0000", 2024-09-11T09:51:29.372 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:29.372 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.372 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.372 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:29.372 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094453+0000", 2024-09-11T09:51:29.372 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.1800000000000002e-07 2024-09-11T09:51:29.372 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.372 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.372 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:29.372 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094455+0000", 2024-09-11T09:51:29.372 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.7020000000000001e-06 2024-09-11T09:51:29.372 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.372 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.373 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:29.373 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094463+0000", 2024-09-11T09:51:29.373 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.1999999999999997e-06 2024-09-11T09:51:29.373 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.373 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.373 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:29.373 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094467+0000", 2024-09-11T09:51:29.373 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.729e-06 2024-09-11T09:51:29.373 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.373 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.373 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:29.373 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.101269+0000", 2024-09-11T09:51:29.374 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0068018799999999997 2024-09-11T09:51:29.374 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.374 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.374 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:29.374 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.101353+0000", 2024-09-11T09:51:29.374 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.3827999999999998e-05 2024-09-11T09:51:29.374 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.374 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.374 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 5", 2024-09-11T09:51:29.374 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.101588+0000", 2024-09-11T09:51:29.374 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00023467099999999999 2024-09-11T09:51:29.374 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.374 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.374 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:29.375 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.106182+0000", 2024-09-11T09:51:29.375 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0045939489999999999 2024-09-11T09:51:29.375 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.375 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.375 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:29.375 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.107041+0000", 2024-09-11T09:51:29.375 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00085965800000000002 2024-09-11T09:51:29.375 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.375 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.375 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:29.375 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.107084+0000", 2024-09-11T09:51:29.375 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.261e-05 2024-09-11T09:51:29.375 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.376 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.376 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:29.376 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.107133+0000", 2024-09-11T09:51:29.376 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.8838000000000002e-05 2024-09-11T09:51:29.376 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:29.376 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:29.376 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:29.376 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.376 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.376 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4694.0:37 2.7 2:e8766325:ns::rbd_object_map.1256a0c02879:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:29.376 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.094473+0000", 2024-09-11T09:51:29.376 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 6.2061070909999998, 2024-09-11T09:51:29.376 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.012778484999999999, 2024-09-11T09:51:29.376 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:29.377 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:29.377 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:29.377 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:29.377 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4694", 2024-09-11T09:51:29.377 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:51:29.377 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 37 2024-09-11T09:51:29.377 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.377 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:29.377 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.377 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:29.377 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094473+0000", 2024-09-11T09:51:29.377 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:29.377 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.378 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.378 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:29.378 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094473+0000", 2024-09-11T09:51:29.378 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:29.378 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.378 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.378 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:29.378 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094474+0000", 2024-09-11T09:51:29.378 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.9699999999999996e-07 2024-09-11T09:51:29.378 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.378 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.378 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:29.378 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094477+0000", 2024-09-11T09:51:29.378 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.151e-06 2024-09-11T09:51:29.379 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.379 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.379 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:29.379 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094483+0000", 2024-09-11T09:51:29.379 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.6330000000000001e-06 2024-09-11T09:51:29.379 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.379 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.379 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:29.379 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094487+0000", 2024-09-11T09:51:29.379 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.5399999999999997e-06 2024-09-11T09:51:29.379 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.379 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.379 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:29.379 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.101866+0000", 2024-09-11T09:51:29.380 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.007378768 2024-09-11T09:51:29.380 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.380 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.380 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:29.380 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.101960+0000", 2024-09-11T09:51:29.380 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.3861000000000001e-05 2024-09-11T09:51:29.380 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.380 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.380 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 5", 2024-09-11T09:51:29.380 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.102197+0000", 2024-09-11T09:51:29.380 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00023723200000000001 2024-09-11T09:51:29.380 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.380 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.381 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:29.381 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.106222+0000", 2024-09-11T09:51:29.381 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0040249689999999998 2024-09-11T09:51:29.381 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.381 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.381 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:29.381 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.107141+0000", 2024-09-11T09:51:29.381 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00091913900000000002 2024-09-11T09:51:29.381 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.381 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.381 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:29.381 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.107198+0000", 2024-09-11T09:51:29.382 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.6564999999999998e-05 2024-09-11T09:51:29.382 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.382 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.382 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:29.382 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.107252+0000", 2024-09-11T09:51:29.382 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.4030000000000003e-05 2024-09-11T09:51:29.382 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:29.382 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:29.382 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:29.382 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.382 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.382 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4694.0:33 2.4 e53/32 2:3bfc818c:ns::rbd_data.1256a0c02879.0000000000000001:head v 53'14, mlcod=53'12)", 2024-09-11T09:51:29.382 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.096496+0000", 2024-09-11T09:51:29.383 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 6.2040845180000002, 2024-09-11T09:51:29.383 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0099139310000000008, 2024-09-11T09:51:29.383 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:29.383 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:29.383 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:29.383 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:29.383 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.383 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:29.383 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.096496+0000", 2024-09-11T09:51:29.383 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:29.383 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.383 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.383 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:29.384 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.096496+0000", 2024-09-11T09:51:29.384 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:29.384 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.384 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.384 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:29.384 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.096498+0000", 2024-09-11T09:51:29.384 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6309999999999999e-06 2024-09-11T09:51:29.384 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.384 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.384 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:29.384 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.098087+0000", 2024-09-11T09:51:29.384 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0015889019999999999 2024-09-11T09:51:29.384 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.385 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.385 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:29.385 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.098112+0000", 2024-09-11T09:51:29.385 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.5369000000000001e-05 2024-09-11T09:51:29.385 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.385 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.385 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:29.385 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.098127+0000", 2024-09-11T09:51:29.385 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.5218999999999999e-05 2024-09-11T09:51:29.385 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.385 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.385 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:29.385 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.098209+0000", 2024-09-11T09:51:29.386 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.2120999999999995e-05 2024-09-11T09:51:29.386 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.386 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.386 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:29.386 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.098237+0000", 2024-09-11T09:51:29.386 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.7657999999999998e-05 2024-09-11T09:51:29.386 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.386 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.386 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:29.386 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.106376+0000", 2024-09-11T09:51:29.386 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0081386749999999997 2024-09-11T09:51:29.386 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.386 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.387 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:29.387 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.106410+0000", 2024-09-11T09:51:29.387 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.4356000000000001e-05 2024-09-11T09:51:29.387 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:29.387 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:29.387 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:29.387 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.387 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.387 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4694.0:49 2.7 2:e8766325:ns::rbd_object_map.1256a0c02879:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:29.387 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.134806+0000", 2024-09-11T09:51:29.387 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 6.1657744489999997, 2024-09-11T09:51:29.387 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0061629049999999998, 2024-09-11T09:51:29.388 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:29.388 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:29.388 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:29.388 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:29.388 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4694", 2024-09-11T09:51:29.388 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:51:29.388 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 49 2024-09-11T09:51:29.388 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.388 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:29.388 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.388 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:29.388 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.134806+0000", 2024-09-11T09:51:29.388 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:29.389 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.389 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.389 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:29.389 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.134806+0000", 2024-09-11T09:51:29.389 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:29.389 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.389 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.389 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:29.389 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.134808+0000", 2024-09-11T09:51:29.389 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.9870000000000002e-06 2024-09-11T09:51:29.389 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.389 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.389 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:29.390 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.134818+0000", 2024-09-11T09:51:29.390 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.5599999999999999e-06 2024-09-11T09:51:29.390 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.390 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.390 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:29.390 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.134835+0000", 2024-09-11T09:51:29.390 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7399000000000001e-05 2024-09-11T09:51:29.390 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.390 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.390 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:29.390 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.134845+0000", 2024-09-11T09:51:29.390 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0285000000000001e-05 2024-09-11T09:51:29.390 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.390 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.391 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:29.391 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.134911+0000", 2024-09-11T09:51:29.391 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.5654999999999994e-05 2024-09-11T09:51:29.391 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.391 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.391 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:29.391 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.135023+0000", 2024-09-11T09:51:29.391 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00011186799999999999 2024-09-11T09:51:29.391 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.391 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.391 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 5", 2024-09-11T09:51:29.391 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.135290+0000", 2024-09-11T09:51:29.391 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00026725300000000001 2024-09-11T09:51:29.392 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.392 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.392 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:29.392 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.136506+0000", 2024-09-11T09:51:29.392 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0012161139999999999 2024-09-11T09:51:29.392 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.392 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.392 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:29.392 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.140884+0000", 2024-09-11T09:51:29.392 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0043781990000000002 2024-09-11T09:51:29.392 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.392 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.392 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:29.393 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.140923+0000", 2024-09-11T09:51:29.393 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.8813999999999998e-05 2024-09-11T09:51:29.393 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.393 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.393 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:29.393 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.140969+0000", 2024-09-11T09:51:29.393 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.5770999999999999e-05 2024-09-11T09:51:29.393 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:29.393 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:29.393 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:29.393 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.393 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.393 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4694.0:42 2.5 e53/32 2:a4fc00ec:ns::rbd_data.1256a0c02879.0000000000000004:head v 53'3, mlcod=53'1)", 2024-09-11T09:51:29.394 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.136646+0000", 2024-09-11T09:51:29.394 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 6.1639341429999996, 2024-09-11T09:51:29.394 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.012545501000000001, 2024-09-11T09:51:29.394 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:29.394 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:29.394 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:29.394 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:29.394 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.394 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:29.394 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.136646+0000", 2024-09-11T09:51:29.394 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:29.394 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.394 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.394 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:29.395 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.136646+0000", 2024-09-11T09:51:29.395 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:29.395 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.395 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.395 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:29.395 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.136647+0000", 2024-09-11T09:51:29.395 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.2900000000000002e-07 2024-09-11T09:51:29.395 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.395 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.395 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:29.395 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.143620+0000", 2024-09-11T09:51:29.395 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0069727440000000003 2024-09-11T09:51:29.395 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.395 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.396 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:29.396 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.143642+0000", 2024-09-11T09:51:29.396 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.1846000000000001e-05 2024-09-11T09:51:29.396 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.396 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.396 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:29.396 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.143654+0000", 2024-09-11T09:51:29.396 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2081e-05 2024-09-11T09:51:29.396 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.396 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.396 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:29.396 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.143717+0000", 2024-09-11T09:51:29.396 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.3443999999999998e-05 2024-09-11T09:51:29.397 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.397 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.397 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:29.397 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.143740+0000", 2024-09-11T09:51:29.397 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.2626999999999999e-05 2024-09-11T09:51:29.397 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.397 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.397 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:29.397 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.149142+0000", 2024-09-11T09:51:29.397 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0054017630000000004 2024-09-11T09:51:29.397 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.397 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.397 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:29.397 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.149192+0000", 2024-09-11T09:51:29.398 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.0067e-05 2024-09-11T09:51:29.398 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:29.398 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:29.398 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:29.398 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.398 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.398 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4694.0:53 2.7 2:e8766325:ns::rbd_object_map.1256a0c02879:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:29.398 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.147978+0000", 2024-09-11T09:51:29.398 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 6.1526026590000003, 2024-09-11T09:51:29.398 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0044549769999999997, 2024-09-11T09:51:29.398 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:29.398 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:29.398 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:29.399 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:29.399 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4694", 2024-09-11T09:51:29.399 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:51:29.399 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 53 2024-09-11T09:51:29.399 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.399 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:29.399 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.399 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:29.399 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.147978+0000", 2024-09-11T09:51:29.399 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:29.399 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.399 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.399 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:29.399 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.147978+0000", 2024-09-11T09:51:29.400 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:29.400 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.400 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.400 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:29.400 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.147980+0000", 2024-09-11T09:51:29.400 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.8309999999999999e-06 2024-09-11T09:51:29.400 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.400 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.400 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:29.400 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.147989+0000", 2024-09-11T09:51:29.400 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.9639999999999992e-06 2024-09-11T09:51:29.400 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.400 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.401 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:29.401 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.148007+0000", 2024-09-11T09:51:29.401 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7884999999999999e-05 2024-09-11T09:51:29.401 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.401 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.401 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:29.401 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.148017+0000", 2024-09-11T09:51:29.401 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0895e-05 2024-09-11T09:51:29.401 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.401 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.401 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:29.401 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.148091+0000", 2024-09-11T09:51:29.401 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.3709999999999997e-05 2024-09-11T09:51:29.402 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.402 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.402 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:29.402 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.148189+0000", 2024-09-11T09:51:29.402 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.7403000000000002e-05 2024-09-11T09:51:29.402 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.402 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.402 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 5", 2024-09-11T09:51:29.402 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.148504+0000", 2024-09-11T09:51:29.402 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00031589900000000002 2024-09-11T09:51:29.402 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.402 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.402 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:29.402 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.149514+0000", 2024-09-11T09:51:29.403 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0010096 2024-09-11T09:51:29.403 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.403 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.403 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:29.403 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.152367+0000", 2024-09-11T09:51:29.403 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0028532309999999999 2024-09-11T09:51:29.403 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.403 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.403 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:29.403 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.152402+0000", 2024-09-11T09:51:29.403 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.4232e-05 2024-09-11T09:51:29.403 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.403 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.403 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:29.404 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.152433+0000", 2024-09-11T09:51:29.404 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.1327e-05 2024-09-11T09:51:29.404 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:29.404 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:29.404 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:29.404 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.404 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.404 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4694.0:55 2.7 2:e8766325:ns::rbd_object_map.1256a0c02879:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:29.404 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.149838+0000", 2024-09-11T09:51:29.404 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 6.1507429260000004, 2024-09-11T09:51:29.404 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0060695159999999996, 2024-09-11T09:51:29.404 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:29.405 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:29.405 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:29.405 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:29.405 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4694", 2024-09-11T09:51:29.405 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:51:29.405 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 55 2024-09-11T09:51:29.405 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.405 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:29.405 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.405 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:29.405 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.149838+0000", 2024-09-11T09:51:29.405 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:29.405 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.406 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.406 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:29.406 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.149838+0000", 2024-09-11T09:51:29.406 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:29.406 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.406 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.406 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:29.406 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.149839+0000", 2024-09-11T09:51:29.406 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.119e-06 2024-09-11T09:51:29.406 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.406 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.406 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:29.406 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.149845+0000", 2024-09-11T09:51:29.407 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.7810000000000002e-06 2024-09-11T09:51:29.407 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.407 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.407 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:29.407 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.149856+0000", 2024-09-11T09:51:29.407 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1199999999999999e-05 2024-09-11T09:51:29.407 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.407 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.407 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:29.407 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.149863+0000", 2024-09-11T09:51:29.407 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.4109999999999999e-06 2024-09-11T09:51:29.407 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.407 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.408 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:29.408 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.149908+0000", 2024-09-11T09:51:29.408 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.5184e-05 2024-09-11T09:51:29.408 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.408 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.408 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:29.408 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.149964+0000", 2024-09-11T09:51:29.408 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.5395000000000001e-05 2024-09-11T09:51:29.408 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.408 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.408 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 5", 2024-09-11T09:51:29.408 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.150157+0000", 2024-09-11T09:51:29.408 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000193489 2024-09-11T09:51:29.409 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.409 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.409 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:29.409 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.152238+0000", 2024-09-11T09:51:29.409 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0020803480000000001 2024-09-11T09:51:29.409 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.409 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.409 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:29.409 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.155828+0000", 2024-09-11T09:51:29.409 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0035902210000000002 2024-09-11T09:51:29.409 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.409 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.409 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:29.409 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.155873+0000", 2024-09-11T09:51:29.410 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.5145999999999998e-05 2024-09-11T09:51:29.410 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:29.410 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:29.410 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:29.410 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.155907+0000", 2024-09-11T09:51:29.410 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.4221999999999999e-05 2024-09-11T09:51:29.410 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:29.410 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:29.410 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:29.410 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:29.410 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:29.410 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:51:29.411 INFO:tasks.ceph.osd.0.smithi017.stderr:2024-09-11T09:51:29.301+0000 7fb73c0f1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31665) UID: 0 2024-09-11T09:51:29.412 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-11T09:51:29.412 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T09:51:29.413 INFO:tasks.workunit.client.0.smithi107.stdout:27563 rbd testrbdnbd27121 - /dev/nbd0 67a007f9-7c31-4d40-897d-4124fe986499 2024-09-11T09:51:29.414 INFO:tasks.workunit.client.0.smithi107.stderr:+ get_pid rbd 2024-09-11T09:51:29.414 INFO:tasks.workunit.client.0.smithi107.stderr:+ local pool=rbd 2024-09-11T09:51:29.414 INFO:tasks.workunit.client.0.smithi107.stderr:+ local ns= 2024-09-11T09:51:29.414 INFO:tasks.workunit.client.0.smithi107.stderr:++ rbd device --device-type nbd --format xml list 2024-09-11T09:51:29.414 INFO:tasks.workunit.client.0.smithi107.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd27121'\''][device='\''/dev/nbd0'\'']/id' 2024-09-11T09:51:29.414 INFO:tasks.workunit.client.0.smithi107.stderr:+ PID=27563 2024-09-11T09:51:29.414 INFO:tasks.workunit.client.0.smithi107.stderr:+ test -n 27563 2024-09-11T09:51:29.414 INFO:tasks.workunit.client.0.smithi107.stderr:+ ps -p 27563 -C rbd-nbd 2024-09-11T09:51:29.414 INFO:tasks.workunit.client.0.smithi107.stdout: PID TTY TIME CMD 2024-09-11T09:51:29.414 INFO:tasks.workunit.client.0.smithi107.stdout: 27563 ? 00:00:00 rbd-nbd 2024-09-11T09:51:29.415 INFO:tasks.workunit.client.0.smithi107.stderr:++ dd if=/tmp/tmp.rvrcDirNue/data bs=1M 2024-09-11T09:51:29.415 INFO:tasks.workunit.client.0.smithi107.stderr:++ md5sum 2024-09-11T09:51:29.453 INFO:tasks.ceph.osd.0.smithi017.stderr:2024-09-11T09:51:29.454+0000 7fb73c0f1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31665) UID: 0 2024-09-11T09:51:29.489 INFO:tasks.workunit.client.0.smithi107.stderr:64+0 records in 2024-09-11T09:51:29.489 INFO:tasks.workunit.client.0.smithi107.stderr:64+0 records out 2024-09-11T09:51:29.489 INFO:tasks.workunit.client.0.smithi107.stderr:67108864 bytes (67 MB, 64 MiB) copied, 0.108903 s, 616 MB/s 2024-09-11T09:51:29.490 INFO:tasks.workunit.client.0.smithi107.stderr:++ _sudo dd if=/dev/nbd0 bs=1M 2024-09-11T09:51:29.490 INFO:tasks.workunit.client.0.smithi107.stderr:++ local cmd 2024-09-11T09:51:29.490 INFO:tasks.workunit.client.0.smithi107.stderr:++ md5sum 2024-09-11T09:51:29.490 INFO:tasks.workunit.client.0.smithi107.stderr:+++ id -u 2024-09-11T09:51:29.491 INFO:tasks.workunit.client.0.smithi107.stderr:++ '[' 1000 -eq 0 ']' 2024-09-11T09:51:29.491 INFO:tasks.workunit.client.0.smithi107.stderr:+++ which dd 2024-09-11T09:51:29.493 INFO:tasks.workunit.client.0.smithi107.stderr:++ cmd=/usr/bin/dd 2024-09-11T09:51:29.493 INFO:tasks.workunit.client.0.smithi107.stderr:++ shift 2024-09-11T09:51:29.493 INFO:tasks.workunit.client.0.smithi107.stderr:++ sudo -nE /usr/bin/dd if=/dev/nbd0 bs=1M 2024-09-11T09:51:29.514 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:51:29.515+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:51:29.569 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:51:29.569 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [], 2024-09-11T09:51:29.569 INFO:teuthology.orchestra.run.smithi017.stdout: "num_ops": 0 2024-09-11T09:51:29.569 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:51:29.579 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-11T09:51:29.584 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-11T09:51:29.585 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T09:51:29.602 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-11T09:51:29.602 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T09:51:29.615 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-09-11T09:51:29.616+0000 7f7b6c7d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31904) UID: 0 2024-09-11T09:51:29.620 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-11T09:51:29.620 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T09:51:29.639 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-11T09:51:29.639 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T09:51:29.639 INFO:teuthology.orchestra.run.smithi017.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-09-11T09:51:29.656 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-11T09:51:29.657 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T09:51:29.660 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-11T09:51:29.661 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-11T09:51:29.699 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:51:29.699 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:51:29.708 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-11T09:51:29.732 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-11T09:51:29.749 INFO:tasks.ceph.osd.0.smithi017.stderr:2024-09-11T09:51:29.750+0000 7fb73c0f1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31665) UID: 0 2024-09-11T09:51:29.817 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:51:29.818+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:51:29.833 INFO:tasks.workunit.client.0.smithi107.stderr:64+0 records in 2024-09-11T09:51:29.833 INFO:tasks.workunit.client.0.smithi107.stderr:64+0 records out 2024-09-11T09:51:29.833 INFO:tasks.workunit.client.0.smithi107.stderr:67108864 bytes (67 MB, 64 MiB) copied, 0.330944 s, 203 MB/s 2024-09-11T09:51:29.835 INFO:tasks.workunit.client.0.smithi107.stderr:+ '[' '7c7289afcdcad439c008685755da6751 -' = '7c7289afcdcad439c008685755da6751 -' ']' 2024-09-11T09:51:29.835 INFO:tasks.workunit.client.0.smithi107.stderr:+ dd if=/dev/urandom of=/tmp/tmp.rvrcDirNue/data bs=1M count=64 2024-09-11T09:51:29.867 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:51:29.867 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:51:29.877 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-11T09:51:29.918 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:51:29.919+0000 7f3c96ec9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31909) UID: 0 2024-09-11T09:51:30.019 INFO:tasks.ceph.osd.0.smithi017.stderr:2024-09-11T09:51:30.020+0000 7fb73c0f1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31665) UID: 0 2024-09-11T09:51:30.026 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:51:30.026 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:51:30.035 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-11T09:51:30.120 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:51:30.121+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:51:30.182 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:51:30.183 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:51:30.192 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-11T09:51:30.221 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-09-11T09:51:30.221+0000 7f7b6c7d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31904) UID: 0 2024-09-11T09:51:30.321 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:51:30.322+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:51:30.328 INFO:tasks.workunit.client.0.smithi107.stderr:64+0 records in 2024-09-11T09:51:30.328 INFO:tasks.workunit.client.0.smithi107.stderr:64+0 records out 2024-09-11T09:51:30.328 INFO:tasks.workunit.client.0.smithi107.stderr:67108864 bytes (67 MB, 64 MiB) copied, 0.484895 s, 138 MB/s 2024-09-11T09:51:30.328 INFO:tasks.workunit.client.0.smithi107.stderr:+ _sudo dd if=/tmp/tmp.rvrcDirNue/data of=/dev/nbd0 bs=1M oflag=direct 2024-09-11T09:51:30.328 INFO:tasks.workunit.client.0.smithi107.stderr:+ local cmd 2024-09-11T09:51:30.329 INFO:tasks.workunit.client.0.smithi107.stderr:++ id -u 2024-09-11T09:51:30.329 INFO:tasks.workunit.client.0.smithi107.stderr:+ '[' 1000 -eq 0 ']' 2024-09-11T09:51:30.330 INFO:tasks.workunit.client.0.smithi107.stderr:++ which dd 2024-09-11T09:51:30.332 INFO:tasks.workunit.client.0.smithi107.stderr:+ cmd=/usr/bin/dd 2024-09-11T09:51:30.332 INFO:tasks.workunit.client.0.smithi107.stderr:+ shift 2024-09-11T09:51:30.332 INFO:tasks.workunit.client.0.smithi107.stderr:+ sudo -nE /usr/bin/dd if=/tmp/tmp.rvrcDirNue/data of=/dev/nbd0 bs=1M oflag=direct 2024-09-11T09:51:30.339 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:51:30.340 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:51:30.349 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-11T09:51:30.422 INFO:tasks.ceph.osd.0.smithi017.stderr:2024-09-11T09:51:30.423+0000 7fb73c0f1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31665) UID: 0 2024-09-11T09:51:30.457 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:51:30.457 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:51:30.466 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-11T09:51:30.523 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:51:30.524+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:51:30.580 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:51:30.581 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:51:30.584 INFO:teuthology.orchestra.run.smithi096.stdout: marking collection for removal 2024-09-11T09:51:30.584 INFO:teuthology.orchestra.run.smithi096.stdout:setting '_remove' omap key 2024-09-11T09:51:30.584 INFO:teuthology.orchestra.run.smithi096.stdout:finish_remove_pgs 3.3_head removing 3.3 2024-09-11T09:51:30.590 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-11T09:51:30.590 INFO:teuthology.orchestra.run.smithi096.stdout:Remove successful 2024-09-11T09:51:30.633 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:51:30.634+0000 7f3c96ec9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31909) UID: 0 2024-09-11T09:51:30.695 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:51:30.695 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:51:30.704 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-11T09:51:30.724 INFO:tasks.ceph.osd.2.smithi017.stderr:2024-09-11T09:51:30.725+0000 7fd20b170640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31672) UID: 0 2024-09-11T09:51:30.732 INFO:teuthology.orchestra.run.smithi017.stderr:nodeep-scrub is set 2024-09-11T09:51:30.778 INFO:tasks.workunit.client.0.smithi107.stderr:64+0 records in 2024-09-11T09:51:30.778 INFO:tasks.workunit.client.0.smithi107.stderr:64+0 records out 2024-09-11T09:51:30.778 INFO:tasks.workunit.client.0.smithi107.stderr:67108864 bytes (67 MB, 64 MiB) copied, 0.434544 s, 154 MB/s 2024-09-11T09:51:30.780 INFO:tasks.workunit.client.0.smithi107.stderr:++ dd if=/tmp/tmp.rvrcDirNue/data bs=1M 2024-09-11T09:51:30.780 INFO:tasks.workunit.client.0.smithi107.stderr:++ md5sum 2024-09-11T09:51:30.825 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:51:30.826+0000 7f3c96ec9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31909) UID: 0 2024-09-11T09:51:30.848 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:51:30.848 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:51:30.857 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-11T09:51:30.886 INFO:tasks.workunit.client.0.smithi107.stderr:64+0 records in 2024-09-11T09:51:30.886 INFO:tasks.workunit.client.0.smithi107.stderr:64+0 records out 2024-09-11T09:51:30.886 INFO:tasks.workunit.client.0.smithi107.stderr:67108864 bytes (67 MB, 64 MiB) copied, 0.105422 s, 637 MB/s 2024-09-11T09:51:30.887 INFO:tasks.workunit.client.0.smithi107.stderr:++ rbd -p rbd --no-progress export testrbdnbd27121 - 2024-09-11T09:51:30.887 INFO:tasks.workunit.client.0.smithi107.stderr:++ md5sum 2024-09-11T09:51:30.926 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:51:30.927+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:51:30.944 INFO:tasks.workunit.client.0.smithi107.stderr:2024-09-11T09:51:30.944+0000 7eff4a946640 0 -- 172.21.15.107:0/2768581708 >> [v2:172.21.15.96:6816/332854807,v1:172.21.15.96:6817/332854807] conn(0x7eff28008f20 msgr2=0x7eff2800b390 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-11T09:51:30.951 INFO:tasks.workunit.client.0.smithi107.stderr:2024-09-11T09:51:30.951+0000 7eff43fff640 0 -- 172.21.15.107:0/2768581708 >> [v2:172.21.15.96:6808/509935370,v1:172.21.15.96:6809/509935370] conn(0x7eff28004a50 msgr2=0x7eff28004ee0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-11T09:51:31.004 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:51:31.005 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:51:31.013 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-11T09:51:31.027 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:51:31.028+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:51:31.118 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-objectstore-tool --err-to-stderr --data-path /var/lib/ceph/osd/ceph-4 --journal-path /var/lib/ceph/osd/ceph-4/journal '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --op import --file /var/log/ceph/exp.3.3.4 2024-09-11T09:51:31.119 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:51:31.119 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:51:31.127 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-11T09:51:31.128 INFO:tasks.ceph.osd.2.smithi017.stderr:2024-09-11T09:51:31.129+0000 7fd20b170640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31672) UID: 0 2024-09-11T09:51:31.229 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:51:31.230+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:51:31.235 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:51:31.235 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:51:31.246 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-11T09:51:31.265 INFO:tasks.workunit.client.0.smithi107.stderr:+ '[' '22cf9921093f850f30ba7695226848ca -' = '22cf9921093f850f30ba7695226848ca -' ']' 2024-09-11T09:51:31.265 INFO:tasks.workunit.client.0.smithi107.stderr:+ unmap_device /dev/nbd0 27563 2024-09-11T09:51:31.266 INFO:tasks.workunit.client.0.smithi107.stderr:+ local args=/dev/nbd0 2024-09-11T09:51:31.266 INFO:tasks.workunit.client.0.smithi107.stderr:+ local pid=27563 2024-09-11T09:51:31.266 INFO:tasks.workunit.client.0.smithi107.stderr:+ _sudo rbd device --device-type nbd unmap /dev/nbd0 2024-09-11T09:51:31.266 INFO:tasks.workunit.client.0.smithi107.stderr:+ local cmd 2024-09-11T09:51:31.266 INFO:tasks.workunit.client.0.smithi107.stderr:++ id -u 2024-09-11T09:51:31.266 INFO:tasks.workunit.client.0.smithi107.stderr:+ '[' 1000 -eq 0 ']' 2024-09-11T09:51:31.266 INFO:tasks.workunit.client.0.smithi107.stderr:++ which rbd 2024-09-11T09:51:31.268 INFO:tasks.workunit.client.0.smithi107.stderr:+ cmd=/usr/bin/rbd 2024-09-11T09:51:31.268 INFO:tasks.workunit.client.0.smithi107.stderr:+ shift 2024-09-11T09:51:31.268 INFO:tasks.workunit.client.0.smithi107.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd unmap /dev/nbd0 2024-09-11T09:51:31.330 INFO:tasks.ceph.osd.2.smithi017.stderr:2024-09-11T09:51:31.331+0000 7fd20b170640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31672) UID: 0 2024-09-11T09:51:31.349 INFO:tasks.workunit.client.0.smithi107.stderr:+ rbd device --device-type nbd list 2024-09-11T09:51:31.349 INFO:tasks.workunit.client.0.smithi107.stderr:+ expect_false grep '^27563\b' 2024-09-11T09:51:31.349 INFO:tasks.workunit.client.0.smithi107.stderr:+ grep '^27563\b' 2024-09-11T09:51:31.394 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:51:31.394 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:51:31.395 INFO:tasks.workunit.client.0.smithi107.stderr:+ return 0 2024-09-11T09:51:31.396 INFO:tasks.workunit.client.0.smithi107.stderr:+ ps -C rbd-nbd 2024-09-11T09:51:31.396 INFO:tasks.workunit.client.0.smithi107.stderr:+ expect_false grep '^ *27563\b' 2024-09-11T09:51:31.396 INFO:tasks.workunit.client.0.smithi107.stderr:+ grep '^ *27563\b' 2024-09-11T09:51:31.403 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-11T09:51:31.407 INFO:tasks.workunit.client.0.smithi107.stderr:+ return 0 2024-09-11T09:51:31.407 INFO:tasks.workunit.client.0.smithi107.stderr:+ sleep 0.5 2024-09-11T09:51:31.430 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-09-11T09:51:31.431+0000 7f7b6c7d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31904) UID: 0 2024-09-11T09:51:31.531 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-09-11T09:51:31.532+0000 7f7b6c7d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31904) UID: 0 2024-09-11T09:51:31.550 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:51:31.550 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:51:31.559 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-11T09:51:31.633 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-09-11T09:51:31.634+0000 7f7b6c7d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31904) UID: 0 2024-09-11T09:51:31.661 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T09:51:31.666 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:51:31.666 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:51:31.676 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-11T09:51:31.761 INFO:tasks.ceph.osd.2.smithi017.stderr:2024-09-11T09:51:31.762+0000 7fd20b170640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31672) UID: 0 2024-09-11T09:51:31.785 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-11T09:51:31.785 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T09:51:31.802 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-11T09:51:31.803 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T09:51:31.820 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-11T09:51:31.820 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T09:51:31.827 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:51:31.827 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [], 2024-09-11T09:51:31.827 INFO:teuthology.orchestra.run.smithi017.stdout: "complaint_time": 30, 2024-09-11T09:51:31.827 INFO:teuthology.orchestra.run.smithi017.stdout: "num_blocked_ops": 0 2024-09-11T09:51:31.827 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:51:31.834 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:51:31.835+0000 7f3c96ec9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31909) UID: 0 2024-09-11T09:51:31.837 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-11T09:51:31.838 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-11T09:51:31.838 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T09:51:31.839 INFO:teuthology.orchestra.run.smithi017.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-09-11T09:51:31.855 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-11T09:51:31.856 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T09:51:31.859 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-11T09:51:31.859 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-11T09:51:31.909 INFO:tasks.workunit.client.0.smithi107.stderr:++ _sudo rbd device --device-type nbd --options notrim map rbd/testrbdnbd27121 2024-09-11T09:51:31.909 INFO:tasks.workunit.client.0.smithi107.stderr:++ local cmd 2024-09-11T09:51:31.909 INFO:tasks.workunit.client.0.smithi107.stderr:+++ id -u 2024-09-11T09:51:31.910 INFO:tasks.workunit.client.0.smithi107.stderr:++ '[' 1000 -eq 0 ']' 2024-09-11T09:51:31.910 INFO:tasks.workunit.client.0.smithi107.stderr:+++ which rbd 2024-09-11T09:51:31.912 INFO:tasks.workunit.client.0.smithi107.stderr:++ cmd=/usr/bin/rbd 2024-09-11T09:51:31.912 INFO:tasks.workunit.client.0.smithi107.stderr:++ shift 2024-09-11T09:51:31.912 INFO:tasks.workunit.client.0.smithi107.stderr:++ sudo -nE /usr/bin/rbd device --device-type nbd --options notrim map rbd/testrbdnbd27121 2024-09-11T09:51:31.935 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:51:31.937+0000 7f3c96ec9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31909) UID: 0 2024-09-11T09:51:31.947 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:51:31.947 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 20, 2024-09-11T09:51:31.947 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 600, 2024-09-11T09:51:31.947 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [ 2024-09-11T09:51:31.947 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:31.947 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:34 2.4 2:2367385a:::rbd_data.1250f5e25b80.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:31.947 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.897717+0000", 2024-09-11T09:51:31.947 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 9.0489649839999995, 2024-09-11T09:51:31.947 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.021323661000000001, 2024-09-11T09:51:31.947 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:31.947 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:31.948 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:31.948 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:31.948 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:51:31.948 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:51:31.948 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 34 2024-09-11T09:51:31.948 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:31.948 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:31.948 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:31.948 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:31.948 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.897717+0000", 2024-09-11T09:51:31.948 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:31.948 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:31.949 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:31.949 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:31.949 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.897717+0000", 2024-09-11T09:51:31.949 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:31.949 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:31.949 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:31.949 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:31.949 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.897718+0000", 2024-09-11T09:51:31.949 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.1800000000000004e-07 2024-09-11T09:51:31.949 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:31.949 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:31.949 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:31.950 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.905339+0000", 2024-09-11T09:51:31.950 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0076210239999999997 2024-09-11T09:51:31.950 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:31.950 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:31.950 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:31.950 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.905366+0000", 2024-09-11T09:51:31.950 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.6959000000000001e-05 2024-09-11T09:51:31.950 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:31.950 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:31.950 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:31.950 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.905375+0000", 2024-09-11T09:51:31.950 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.516e-06 2024-09-11T09:51:31.951 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:31.951 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:31.951 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:31.951 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.905475+0000", 2024-09-11T09:51:31.951 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.9738e-05 2024-09-11T09:51:31.951 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:31.951 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:31.951 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:31.951 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.905660+0000", 2024-09-11T09:51:31.951 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00018554199999999999 2024-09-11T09:51:31.951 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:31.952 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:31.952 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:31.952 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.905808+0000", 2024-09-11T09:51:31.952 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000147386 2024-09-11T09:51:31.952 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:31.952 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:31.952 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:31.952 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.918230+0000", 2024-09-11T09:51:31.952 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.012421690000000001 2024-09-11T09:51:31.952 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:31.952 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:31.952 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:31.952 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.918932+0000", 2024-09-11T09:51:31.953 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00070267300000000001 2024-09-11T09:51:31.953 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:31.953 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:31.953 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:31.953 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.918989+0000", 2024-09-11T09:51:31.953 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.7268999999999997e-05 2024-09-11T09:51:31.953 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:31.953 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:31.953 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:31.953 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919040+0000", 2024-09-11T09:51:31.953 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.0946000000000003e-05 2024-09-11T09:51:31.953 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:31.953 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:31.954 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:31.954 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:31.954 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:31.954 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4688.0:31 2.1 e53/15 2:8654721b:::rbd_data.1250f5e25b80.0000000000000000:head v 53'2, mlcod=0'0)", 2024-09-11T09:51:31.954 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.904661+0000", 2024-09-11T09:51:31.954 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 9.0420204949999992, 2024-09-11T09:51:31.954 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.014152804999999999, 2024-09-11T09:51:31.954 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:31.954 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:31.954 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:31.954 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:31.954 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:31.954 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:31.955 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.904661+0000", 2024-09-11T09:51:31.955 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:31.955 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:31.955 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:31.955 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:31.955 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.904661+0000", 2024-09-11T09:51:31.955 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:31.955 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:31.955 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:31.955 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:31.955 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.904662+0000", 2024-09-11T09:51:31.955 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0759999999999999e-06 2024-09-11T09:51:31.955 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:31.956 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:31.956 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:31.956 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.907298+0000", 2024-09-11T09:51:31.956 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0026361240000000001 2024-09-11T09:51:31.956 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:31.956 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:31.956 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:31.956 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.907320+0000", 2024-09-11T09:51:31.956 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.1268999999999999e-05 2024-09-11T09:51:31.956 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:31.956 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:31.956 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:31.957 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.907331+0000", 2024-09-11T09:51:31.957 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1031e-05 2024-09-11T09:51:31.957 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:31.957 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:31.957 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:31.957 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.907403+0000", 2024-09-11T09:51:31.957 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.2571999999999996e-05 2024-09-11T09:51:31.957 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:31.957 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:31.957 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:31.957 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.907428+0000", 2024-09-11T09:51:31.957 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.4380999999999999e-05 2024-09-11T09:51:31.957 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:31.958 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:31.958 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:31.958 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.918750+0000", 2024-09-11T09:51:31.958 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.011322211 2024-09-11T09:51:31.958 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:31.958 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:31.958 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:31.958 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.918814+0000", 2024-09-11T09:51:31.958 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.4140999999999995e-05 2024-09-11T09:51:31.958 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:31.958 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:31.959 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:31.959 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:31.959 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:31.959 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:36 2.6 2:61e1b442:::rbd_data.1250f5e25b80.0000000000000003:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:31.959 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.905404+0000", 2024-09-11T09:51:31.959 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 9.0412776239999992, 2024-09-11T09:51:31.959 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.030912479999999999, 2024-09-11T09:51:31.959 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:31.959 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:31.959 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:31.959 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:31.959 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:51:31.959 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:51:31.960 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 36 2024-09-11T09:51:31.960 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:31.960 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:31.960 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:31.960 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:31.960 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.905404+0000", 2024-09-11T09:51:31.960 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:31.960 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:31.960 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:31.960 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:31.960 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.905404+0000", 2024-09-11T09:51:31.961 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:31.961 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:31.961 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:31.961 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:31.961 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.905405+0000", 2024-09-11T09:51:31.961 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2270000000000001e-06 2024-09-11T09:51:31.961 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:31.961 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:31.961 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:31.961 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.914859+0000", 2024-09-11T09:51:31.961 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0094541190000000004 2024-09-11T09:51:31.961 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:31.962 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:31.962 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:31.962 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.914879+0000", 2024-09-11T09:51:31.962 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.0052999999999999e-05 2024-09-11T09:51:31.962 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:31.962 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:31.962 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:31.962 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.914897+0000", 2024-09-11T09:51:31.962 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7331000000000001e-05 2024-09-11T09:51:31.962 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:31.962 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:31.962 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:31.962 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.915019+0000", 2024-09-11T09:51:31.963 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00012168 2024-09-11T09:51:31.963 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:31.963 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:31.963 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:31.963 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.915269+0000", 2024-09-11T09:51:31.963 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00025054899999999998 2024-09-11T09:51:31.963 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:31.963 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:31.963 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 3", 2024-09-11T09:51:31.963 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.915479+0000", 2024-09-11T09:51:31.963 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00021017799999999999 2024-09-11T09:51:31.963 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:31.963 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:31.964 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:31.964 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.926070+0000", 2024-09-11T09:51:31.964 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.010590624999999999 2024-09-11T09:51:31.964 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:31.964 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:31.964 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:31.964 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.936198+0000", 2024-09-11T09:51:31.964 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.010128194 2024-09-11T09:51:31.964 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:31.964 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:31.964 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:31.964 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.936263+0000", 2024-09-11T09:51:31.965 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.5366999999999999e-05 2024-09-11T09:51:31.965 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:31.965 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:31.965 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:31.965 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.936317+0000", 2024-09-11T09:51:31.965 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.3156999999999999e-05 2024-09-11T09:51:31.965 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:31.965 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:31.965 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:31.966 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:31.966 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:31.966 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:39 2.6 2:743e7616:::rbd_data.1250f5e25b80.0000000000000005:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:31.966 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.914960+0000", 2024-09-11T09:51:31.966 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 9.0317213219999992, 2024-09-11T09:51:31.966 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.022371518, 2024-09-11T09:51:31.966 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:31.966 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:31.966 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:31.966 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:31.966 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:51:31.967 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:51:31.967 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 39 2024-09-11T09:51:31.967 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:31.967 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:31.967 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:31.967 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:31.967 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.914960+0000", 2024-09-11T09:51:31.967 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:31.967 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:31.968 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:31.968 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:31.968 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.914960+0000", 2024-09-11T09:51:31.968 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:31.968 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:31.968 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:31.968 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:31.968 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.914963+0000", 2024-09-11T09:51:31.968 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.1119999999999999e-06 2024-09-11T09:51:31.968 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:31.968 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:31.969 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:31.969 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.920417+0000", 2024-09-11T09:51:31.969 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0054543969999999997 2024-09-11T09:51:31.969 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:31.969 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:31.969 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:31.969 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.920430+0000", 2024-09-11T09:51:31.969 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3414000000000001e-05 2024-09-11T09:51:31.969 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:31.969 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:31.969 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:31.969 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.920437+0000", 2024-09-11T09:51:31.970 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.6159999999999998e-06 2024-09-11T09:51:31.970 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:31.970 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:31.970 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:31.970 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.921325+0000", 2024-09-11T09:51:31.970 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000887731 2024-09-11T09:51:31.970 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:31.970 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:31.970 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:31.970 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.921422+0000", 2024-09-11T09:51:31.970 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.7201999999999995e-05 2024-09-11T09:51:31.970 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:31.970 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:31.971 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 3", 2024-09-11T09:51:31.971 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.921511+0000", 2024-09-11T09:51:31.971 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.8880000000000003e-05 2024-09-11T09:51:31.971 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:31.971 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:31.971 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:31.971 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.928092+0000", 2024-09-11T09:51:31.971 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0065816290000000003 2024-09-11T09:51:31.971 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:31.971 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:31.971 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:31.971 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.937240+0000", 2024-09-11T09:51:31.971 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0091481070000000008 2024-09-11T09:51:31.971 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:31.971 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:31.972 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:31.972 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.937294+0000", 2024-09-11T09:51:31.972 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.3100000000000003e-05 2024-09-11T09:51:31.972 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:31.972 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:31.972 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:31.972 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.937332+0000", 2024-09-11T09:51:31.972 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.8330000000000001e-05 2024-09-11T09:51:31.972 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:31.972 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:31.972 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:31.972 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:31.972 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:31.973 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4688.0:32 2.2 e53/15 2:45941fd2:::rbd_data.1250f5e25b80.0000000000000001:head v 53'3, mlcod=17'1)", 2024-09-11T09:51:31.973 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.916509+0000", 2024-09-11T09:51:31.973 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 9.0301722400000006, 2024-09-11T09:51:31.973 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.015885318999999998, 2024-09-11T09:51:31.973 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:31.973 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:31.973 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:31.973 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:31.973 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:31.973 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:31.973 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.916509+0000", 2024-09-11T09:51:31.973 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:31.973 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:31.974 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:31.974 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:31.974 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.916509+0000", 2024-09-11T09:51:31.974 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:31.974 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:31.974 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:31.974 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:31.974 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.916511+0000", 2024-09-11T09:51:31.974 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.052e-06 2024-09-11T09:51:31.974 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:31.974 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:31.974 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:31.974 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.927019+0000", 2024-09-11T09:51:31.974 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.010508224 2024-09-11T09:51:31.975 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:31.975 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:31.975 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:31.975 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.927038+0000", 2024-09-11T09:51:31.975 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.9590999999999999e-05 2024-09-11T09:51:31.975 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:31.975 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:31.975 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:31.975 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.927050+0000", 2024-09-11T09:51:31.975 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1439e-05 2024-09-11T09:51:31.975 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:31.975 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:31.975 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:31.975 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.927115+0000", 2024-09-11T09:51:31.976 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.5301000000000004e-05 2024-09-11T09:51:31.976 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:31.976 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:31.976 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:31.976 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.927132+0000", 2024-09-11T09:51:31.976 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7297000000000001e-05 2024-09-11T09:51:31.976 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:31.976 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:31.976 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:31.976 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.932354+0000", 2024-09-11T09:51:31.976 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0052211870000000004 2024-09-11T09:51:31.976 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:31.976 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:31.976 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:31.976 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.932395+0000", 2024-09-11T09:51:31.977 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.1227999999999999e-05 2024-09-11T09:51:31.977 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:31.977 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:31.977 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:31.977 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:31.977 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:31.977 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:44 2.6 2:67246a22:::rbd_data.1250f5e25b80.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:31.977 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.943181+0000", 2024-09-11T09:51:31.977 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 9.0035010290000006, 2024-09-11T09:51:31.977 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.022629371999999998, 2024-09-11T09:51:31.977 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:31.977 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:31.978 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:31.978 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:31.978 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:51:31.978 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:51:31.978 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 44 2024-09-11T09:51:31.978 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:31.978 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:31.978 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:31.978 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:31.978 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.943181+0000", 2024-09-11T09:51:31.978 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:31.978 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:31.979 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:31.979 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:31.979 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.943181+0000", 2024-09-11T09:51:31.979 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:31.979 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:31.979 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:31.979 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:31.979 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.943182+0000", 2024-09-11T09:51:31.979 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.0500000000000002e-07 2024-09-11T09:51:31.979 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:31.979 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:31.980 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:31.980 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951240+0000", 2024-09-11T09:51:31.980 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.008058529 2024-09-11T09:51:31.980 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:31.980 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:31.980 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:31.980 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951260+0000", 2024-09-11T09:51:31.980 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.9570000000000001e-05 2024-09-11T09:51:31.980 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:31.980 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:31.980 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:31.981 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951270+0000", 2024-09-11T09:51:31.981 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0018e-05 2024-09-11T09:51:31.981 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:31.981 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:31.981 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:31.981 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951349+0000", 2024-09-11T09:51:31.981 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.8801000000000007e-05 2024-09-11T09:51:31.981 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:31.981 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:31.981 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:31.982 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951483+0000", 2024-09-11T09:51:31.982 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00013427 2024-09-11T09:51:31.982 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:31.982 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:31.982 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 3", 2024-09-11T09:51:31.982 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951593+0000", 2024-09-11T09:51:31.982 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00011031000000000001 2024-09-11T09:51:31.983 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:31.983 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:31.983 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:31.983 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.957303+0000", 2024-09-11T09:51:31.983 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0057097959999999996 2024-09-11T09:51:31.983 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:31.983 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:31.983 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:31.983 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.965690+0000", 2024-09-11T09:51:31.983 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0083867420000000008 2024-09-11T09:51:31.983 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:31.983 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:31.984 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:31.984 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.965749+0000", 2024-09-11T09:51:31.984 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.8981999999999999e-05 2024-09-11T09:51:31.984 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:31.984 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:31.984 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:31.984 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.965810+0000", 2024-09-11T09:51:31.984 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.1449000000000001e-05 2024-09-11T09:51:31.984 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:31.984 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:31.984 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:31.984 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:31.984 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:31.985 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:51 2.6 2:6b5894fc:::rbd_data.1250f5e25b80.000000000000000a:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:31.985 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.951324+0000", 2024-09-11T09:51:31.985 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 8.9953579559999994, 2024-09-11T09:51:31.985 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.021375273, 2024-09-11T09:51:31.985 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:31.985 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:31.985 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:31.985 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:31.985 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:51:31.985 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:51:31.985 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 51 2024-09-11T09:51:31.985 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:31.985 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:31.986 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:31.986 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:31.986 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951324+0000", 2024-09-11T09:51:31.986 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:31.986 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:31.986 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:31.986 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:31.986 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951324+0000", 2024-09-11T09:51:31.986 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:31.986 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:31.986 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:31.986 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:31.987 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951324+0000", 2024-09-11T09:51:31.987 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.39e-07 2024-09-11T09:51:31.987 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:31.987 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:31.987 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:31.987 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.958537+0000", 2024-09-11T09:51:31.987 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0072126769999999998 2024-09-11T09:51:31.987 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:31.987 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:31.987 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:31.987 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.958554+0000", 2024-09-11T09:51:31.987 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7232e-05 2024-09-11T09:51:31.988 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:31.988 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:31.988 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:31.988 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.958570+0000", 2024-09-11T09:51:31.988 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.5747999999999999e-05 2024-09-11T09:51:31.988 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:31.988 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:31.988 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:31.988 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.958638+0000", 2024-09-11T09:51:31.988 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.7810000000000003e-05 2024-09-11T09:51:31.988 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:31.988 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:31.989 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:31.989 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.958758+0000", 2024-09-11T09:51:31.989 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000120689 2024-09-11T09:51:31.989 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:31.989 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:31.989 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 3", 2024-09-11T09:51:31.989 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.958865+0000", 2024-09-11T09:51:31.989 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00010692 2024-09-11T09:51:31.989 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:31.989 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:31.989 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:31.989 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.970712+0000", 2024-09-11T09:51:31.990 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.011846327 2024-09-11T09:51:31.990 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:31.990 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:31.990 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:31.990 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.972603+0000", 2024-09-11T09:51:31.990 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0018918610000000001 2024-09-11T09:51:31.990 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:31.990 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:31.990 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:31.990 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.972653+0000", 2024-09-11T09:51:31.990 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.9762999999999998e-05 2024-09-11T09:51:31.990 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:31.991 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:31.991 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:31.991 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.972699+0000", 2024-09-11T09:51:31.991 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.5806999999999997e-05 2024-09-11T09:51:31.991 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:31.991 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:31.991 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:31.991 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:31.991 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:31.991 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:52 2.4 2:26d5c2f0:::rbd_data.1250f5e25b80.000000000000000b:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:31.991 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.958596+0000", 2024-09-11T09:51:31.991 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 8.9880857610000007, 2024-09-11T09:51:31.991 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.016477849999999999, 2024-09-11T09:51:31.991 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:31.992 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:31.992 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:31.992 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:31.992 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:51:31.992 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:51:31.992 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 52 2024-09-11T09:51:31.992 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:31.992 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:31.992 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:31.992 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:31.992 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.958596+0000", 2024-09-11T09:51:31.992 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:31.993 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:31.993 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:31.993 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:31.993 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.958596+0000", 2024-09-11T09:51:31.993 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:31.993 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:31.993 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:31.993 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:31.993 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.958597+0000", 2024-09-11T09:51:31.993 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1960000000000001e-06 2024-09-11T09:51:31.993 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:31.993 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:31.994 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:31.994 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.963157+0000", 2024-09-11T09:51:31.994 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0045603029999999999 2024-09-11T09:51:31.994 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:31.994 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:31.994 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:31.996 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.963171+0000", 2024-09-11T09:51:31.996 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3777999999999999e-05 2024-09-11T09:51:31.996 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:31.996 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:31.996 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:31.996 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.963181+0000", 2024-09-11T09:51:31.997 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.9010000000000006e-06 2024-09-11T09:51:31.997 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:31.997 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:31.997 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:31.997 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.963246+0000", 2024-09-11T09:51:31.997 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.5351000000000005e-05 2024-09-11T09:51:31.997 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:31.997 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:31.997 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:31.997 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.963419+0000", 2024-09-11T09:51:31.997 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00017299700000000001 2024-09-11T09:51:31.997 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:31.998 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:31.998 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:31.998 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.963570+0000", 2024-09-11T09:51:31.998 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00015066999999999999 2024-09-11T09:51:31.998 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:31.998 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:31.998 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:31.998 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.973881+0000", 2024-09-11T09:51:31.998 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.010311268 2024-09-11T09:51:31.998 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:31.998 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:31.998 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:31.998 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.974963+0000", 2024-09-11T09:51:31.999 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.001081493 2024-09-11T09:51:31.999 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:31.999 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:31.999 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:31.999 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.975012+0000", 2024-09-11T09:51:31.999 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.9471000000000001e-05 2024-09-11T09:51:31.999 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:31.999 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:31.999 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:31.999 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.975074+0000", 2024-09-11T09:51:31.999 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.1421999999999996e-05 2024-09-11T09:51:31.999 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:31.999 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:32.000 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:32.000 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.000 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.000 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4688.0:47 2.1 e53/15 2:95f53b3e:::rbd_data.1250f5e25b80.0000000000000008:head v 53'3, mlcod=53'1)", 2024-09-11T09:51:32.000 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.960861+0000", 2024-09-11T09:51:32.000 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 8.9858203430000003, 2024-09-11T09:51:32.000 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.012346107, 2024-09-11T09:51:32.000 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:32.000 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:32.000 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:32.000 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:32.000 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.000 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:32.001 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.960861+0000", 2024-09-11T09:51:32.001 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:32.001 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.001 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.001 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:32.001 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.960861+0000", 2024-09-11T09:51:32.001 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:32.001 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.001 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.001 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:32.001 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.960863+0000", 2024-09-11T09:51:32.001 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7150000000000001e-06 2024-09-11T09:51:32.002 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.002 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.002 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:32.002 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.962428+0000", 2024-09-11T09:51:32.002 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.001564668 2024-09-11T09:51:32.002 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.002 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.002 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:32.002 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.962446+0000", 2024-09-11T09:51:32.002 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.8476999999999999e-05 2024-09-11T09:51:32.002 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.002 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.002 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:32.002 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.962458+0000", 2024-09-11T09:51:32.003 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1263999999999999e-05 2024-09-11T09:51:32.003 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.003 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.003 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:32.003 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.962575+0000", 2024-09-11T09:51:32.003 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000117025 2024-09-11T09:51:32.003 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.003 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.003 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:32.003 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.962601+0000", 2024-09-11T09:51:32.003 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.5984999999999999e-05 2024-09-11T09:51:32.003 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.003 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.004 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:32.004 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.973065+0000", 2024-09-11T09:51:32.004 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.010464753 2024-09-11T09:51:32.004 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.004 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.004 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:32.004 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.973207+0000", 2024-09-11T09:51:32.004 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00014222 2024-09-11T09:51:32.004 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:32.004 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:32.004 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:32.004 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.004 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.005 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:54 2.4 2.da8979c4 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:32.005 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.963207+0000", 2024-09-11T09:51:32.005 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 8.9834749049999996, 2024-09-11T09:51:32.005 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.010641575, 2024-09-11T09:51:32.005 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:32.005 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:32.005 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "reached pg", 2024-09-11T09:51:32.005 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:32.005 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:51:32.005 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:51:32.005 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 54 2024-09-11T09:51:32.005 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.005 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:32.006 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.006 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:32.006 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.963207+0000", 2024-09-11T09:51:32.006 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:32.006 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.006 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.006 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:32.006 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.963207+0000", 2024-09-11T09:51:32.006 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:32.006 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.006 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.006 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:32.006 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.963208+0000", 2024-09-11T09:51:32.007 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2979999999999999e-06 2024-09-11T09:51:32.007 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.007 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.007 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:32.007 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.967419+0000", 2024-09-11T09:51:32.007 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0042111750000000002 2024-09-11T09:51:32.007 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.007 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.007 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:32.007 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.967439+0000", 2024-09-11T09:51:32.007 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.0129999999999999e-05 2024-09-11T09:51:32.007 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.008 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.008 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:32.008 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.967453+0000", 2024-09-11T09:51:32.008 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3781000000000001e-05 2024-09-11T09:51:32.008 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.008 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.008 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:32.008 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.973840+0000", 2024-09-11T09:51:32.008 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0063872269999999997 2024-09-11T09:51:32.008 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.008 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.008 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:32.009 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.973848+0000", 2024-09-11T09:51:32.009 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.9640000000000003e-06 2024-09-11T09:51:32.009 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:32.009 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:32.009 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:32.009 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.009 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.009 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:54 2.4 2:239e915b:::rbd_data.1250f5e25b80.000000000000000c:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:32.009 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.982401+0000", 2024-09-11T09:51:32.009 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 8.9642809779999997, 2024-09-11T09:51:32.009 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.016817347, 2024-09-11T09:51:32.009 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:32.010 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:32.010 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:32.010 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:32.010 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:51:32.010 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:51:32.010 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 54 2024-09-11T09:51:32.010 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.010 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:32.010 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.010 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:32.010 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.982401+0000", 2024-09-11T09:51:32.011 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:32.011 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.011 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.011 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:32.011 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.982401+0000", 2024-09-11T09:51:32.011 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:32.011 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.011 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.011 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:32.011 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.982402+0000", 2024-09-11T09:51:32.011 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.8999999999999995e-07 2024-09-11T09:51:32.012 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.012 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.012 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:32.012 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.986535+0000", 2024-09-11T09:51:32.012 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0041333689999999996 2024-09-11T09:51:32.012 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.012 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.012 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:32.012 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.986555+0000", 2024-09-11T09:51:32.012 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.9851e-05 2024-09-11T09:51:32.012 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.012 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.013 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:32.013 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.986569+0000", 2024-09-11T09:51:32.013 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3784000000000001e-05 2024-09-11T09:51:32.013 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.013 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.013 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:32.013 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.986683+0000", 2024-09-11T09:51:32.013 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000114474 2024-09-11T09:51:32.013 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.013 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.013 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:32.013 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.986815+0000", 2024-09-11T09:51:32.013 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00013217799999999999 2024-09-11T09:51:32.014 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.014 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.014 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:32.014 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.986918+0000", 2024-09-11T09:51:32.014 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00010235599999999999 2024-09-11T09:51:32.014 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.014 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.014 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:32.014 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.998983+0000", 2024-09-11T09:51:32.014 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.012065352 2024-09-11T09:51:32.014 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.014 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.015 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:32.015 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.999081+0000", 2024-09-11T09:51:32.015 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.8091000000000007e-05 2024-09-11T09:51:32.015 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.015 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.015 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:32.015 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.999159+0000", 2024-09-11T09:51:32.015 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.7823000000000004e-05 2024-09-11T09:51:32.015 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.015 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.015 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:32.015 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.999218+0000", 2024-09-11T09:51:32.015 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.9178999999999997e-05 2024-09-11T09:51:32.015 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:32.016 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:32.016 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:32.016 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.016 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.016 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:56 2.4 2:2167012f:::rbd_data.1250f5e25b80.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:32.016 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.986661+0000", 2024-09-11T09:51:32.016 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 8.9600205250000009, 2024-09-11T09:51:32.016 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.017353539000000001, 2024-09-11T09:51:32.016 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:32.016 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:32.016 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:32.016 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:32.016 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:51:32.017 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:51:32.017 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 56 2024-09-11T09:51:32.017 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.017 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:32.017 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.017 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:32.017 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.986661+0000", 2024-09-11T09:51:32.017 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:32.017 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.017 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.017 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:32.017 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.986661+0000", 2024-09-11T09:51:32.017 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:32.018 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.018 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.018 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:32.018 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.986663+0000", 2024-09-11T09:51:32.018 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.8989999999999999e-06 2024-09-11T09:51:32.018 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.018 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.018 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:32.018 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.990444+0000", 2024-09-11T09:51:32.018 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0037808569999999999 2024-09-11T09:51:32.018 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.018 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.018 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:32.018 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.990462+0000", 2024-09-11T09:51:32.019 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.8450000000000001e-05 2024-09-11T09:51:32.019 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.019 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.019 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:32.019 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.990476+0000", 2024-09-11T09:51:32.019 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3565e-05 2024-09-11T09:51:32.019 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.019 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.019 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:32.019 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.993582+0000", 2024-09-11T09:51:32.019 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0031059109999999998 2024-09-11T09:51:32.019 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.020 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.020 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:32.020 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.993686+0000", 2024-09-11T09:51:32.020 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000104424 2024-09-11T09:51:32.020 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.020 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.020 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:32.020 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.993799+0000", 2024-09-11T09:51:32.020 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000112985 2024-09-11T09:51:32.020 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.020 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.020 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:32.020 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.002449+0000", 2024-09-11T09:51:32.021 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.008650151 2024-09-11T09:51:32.021 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.021 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.021 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:32.021 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.003948+0000", 2024-09-11T09:51:32.021 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.001498636 2024-09-11T09:51:32.021 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.021 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.021 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:32.021 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.003985+0000", 2024-09-11T09:51:32.021 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.7054999999999997e-05 2024-09-11T09:51:32.021 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.022 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.022 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:32.022 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.004015+0000", 2024-09-11T09:51:32.022 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.9606000000000001e-05 2024-09-11T09:51:32.022 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:32.022 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:32.022 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:32.022 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.022 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.022 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:58 2.6 2:6277c644:::rbd_data.1250f5e25b80.000000000000000e:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:32.022 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.990540+0000", 2024-09-11T09:51:32.022 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 8.9561417300000006, 2024-09-11T09:51:32.023 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.016232068999999998, 2024-09-11T09:51:32.023 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:32.023 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:32.023 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:32.023 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:32.023 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:51:32.023 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:51:32.023 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 58 2024-09-11T09:51:32.023 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.023 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:32.023 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.023 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:32.024 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.990540+0000", 2024-09-11T09:51:32.024 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:32.024 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.024 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.024 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:32.024 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.990540+0000", 2024-09-11T09:51:32.024 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:32.024 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.024 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.024 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:32.025 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.990541+0000", 2024-09-11T09:51:32.025 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2359999999999999e-06 2024-09-11T09:51:32.025 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.025 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.025 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:32.025 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.994626+0000", 2024-09-11T09:51:32.025 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.004084865 2024-09-11T09:51:32.025 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.025 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.025 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:32.025 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.994644+0000", 2024-09-11T09:51:32.025 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7866000000000001e-05 2024-09-11T09:51:32.025 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.025 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.026 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:32.026 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.994653+0000", 2024-09-11T09:51:32.026 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.2159999999999995e-06 2024-09-11T09:51:32.026 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.026 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.026 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:32.026 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.994732+0000", 2024-09-11T09:51:32.026 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.8954000000000003e-05 2024-09-11T09:51:32.026 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.026 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.026 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:32.027 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.994906+0000", 2024-09-11T09:51:32.027 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0001741 2024-09-11T09:51:32.027 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.027 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.027 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 3", 2024-09-11T09:51:32.027 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.995083+0000", 2024-09-11T09:51:32.027 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00017637100000000001 2024-09-11T09:51:32.027 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.027 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.027 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:32.027 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.003066+0000", 2024-09-11T09:51:32.027 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0079833360000000006 2024-09-11T09:51:32.028 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.028 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.028 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:32.028 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.006634+0000", 2024-09-11T09:51:32.028 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.003568309 2024-09-11T09:51:32.028 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.028 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.028 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:32.028 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.006716+0000", 2024-09-11T09:51:32.028 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.1377000000000004e-05 2024-09-11T09:51:32.028 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.028 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.029 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:32.029 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.006772+0000", 2024-09-11T09:51:32.029 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.6439e-05 2024-09-11T09:51:32.029 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:32.029 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:32.029 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:32.029 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.029 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.029 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:59 2.4 2:2906e6a7:::rbd_header.1250f5e25b80:head [watch reconnect cookie 140424358411408 gen 2] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:32.029 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.994684+0000", 2024-09-11T09:51:32.029 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 8.9519980990000008, 2024-09-11T09:51:32.029 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0094134349999999995, 2024-09-11T09:51:32.029 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:32.030 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:32.030 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "started", 2024-09-11T09:51:32.030 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:32.030 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:51:32.030 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:51:32.030 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 59 2024-09-11T09:51:32.030 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.030 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:32.030 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.030 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:32.030 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.994684+0000", 2024-09-11T09:51:32.030 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:32.030 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.031 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.031 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:32.031 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.994684+0000", 2024-09-11T09:51:32.031 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:32.031 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.031 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.031 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:32.031 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.994686+0000", 2024-09-11T09:51:32.031 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.8159999999999998e-06 2024-09-11T09:51:32.031 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.031 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.031 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:32.031 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.994691+0000", 2024-09-11T09:51:32.032 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.5009999999999998e-06 2024-09-11T09:51:32.032 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.032 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.032 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:32.032 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.994698+0000", 2024-09-11T09:51:32.032 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.3039999999999999e-06 2024-09-11T09:51:32.032 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.032 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.032 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:32.032 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.994703+0000", 2024-09-11T09:51:32.032 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.5190000000000003e-06 2024-09-11T09:51:32.033 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.033 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.033 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:32.033 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.998678+0000", 2024-09-11T09:51:32.033 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0039752340000000002 2024-09-11T09:51:32.033 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.033 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.033 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:32.033 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.998748+0000", 2024-09-11T09:51:32.033 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.0114e-05 2024-09-11T09:51:32.033 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.033 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.034 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:32.034 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.004097+0000", 2024-09-11T09:51:32.034 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0053489469999999997 2024-09-11T09:51:32.034 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:32.034 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:32.034 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:32.034 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.034 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.034 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4694.0:33 2.4 2:3bfc818c:ns::rbd_data.1256a0c02879.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:32.034 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.084537+0000", 2024-09-11T09:51:32.034 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 8.8621446709999994, 2024-09-11T09:51:32.035 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.022075773999999999, 2024-09-11T09:51:32.035 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:32.035 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:32.035 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:32.035 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:32.035 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4694", 2024-09-11T09:51:32.035 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:51:32.035 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 33 2024-09-11T09:51:32.035 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.035 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:32.036 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.036 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:32.036 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.084537+0000", 2024-09-11T09:51:32.036 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:32.036 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.036 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.036 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:32.036 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.084537+0000", 2024-09-11T09:51:32.036 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:32.036 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.036 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.037 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:32.037 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.084538+0000", 2024-09-11T09:51:32.037 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.5000000000000001e-07 2024-09-11T09:51:32.037 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.037 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.037 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:32.037 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.095984+0000", 2024-09-11T09:51:32.037 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.011446543 2024-09-11T09:51:32.037 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.037 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.037 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:32.037 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.096014+0000", 2024-09-11T09:51:32.037 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.9745000000000001e-05 2024-09-11T09:51:32.038 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.038 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.038 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:32.038 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.096030+0000", 2024-09-11T09:51:32.038 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6152999999999999e-05 2024-09-11T09:51:32.038 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.038 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.038 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:32.038 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.096113+0000", 2024-09-11T09:51:32.038 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.2731999999999993e-05 2024-09-11T09:51:32.038 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.038 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.038 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:32.039 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.096273+0000", 2024-09-11T09:51:32.039 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00015973699999999999 2024-09-11T09:51:32.039 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.039 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.039 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:32.039 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.096394+0000", 2024-09-11T09:51:32.039 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000120883 2024-09-11T09:51:32.039 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.039 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.039 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:32.039 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.106048+0000", 2024-09-11T09:51:32.039 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0096548020000000005 2024-09-11T09:51:32.040 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.040 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.040 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:32.040 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.106531+0000", 2024-09-11T09:51:32.040 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00048264299999999999 2024-09-11T09:51:32.040 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.040 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.040 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:32.040 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.106565+0000", 2024-09-11T09:51:32.040 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.4343e-05 2024-09-11T09:51:32.040 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.041 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.041 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:32.041 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.106613+0000", 2024-09-11T09:51:32.041 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.7342999999999997e-05 2024-09-11T09:51:32.041 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:32.041 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:32.041 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:32.041 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.041 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.041 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4694.0:32 2.1 e53/15 2:8a862b48:ns::rbd_data.1256a0c02879.0000000000000000:head v 53'8, mlcod=53'6)", 2024-09-11T09:51:32.041 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.091418+0000", 2024-09-11T09:51:32.042 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 8.8552635019999997, 2024-09-11T09:51:32.042 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.013027372000000001, 2024-09-11T09:51:32.042 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:32.042 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:32.042 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:32.042 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:32.042 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.042 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:32.042 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.091418+0000", 2024-09-11T09:51:32.042 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:32.042 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.042 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.043 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:32.043 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.091418+0000", 2024-09-11T09:51:32.043 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:32.043 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.043 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.043 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:32.043 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.091420+0000", 2024-09-11T09:51:32.043 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6899999999999999e-06 2024-09-11T09:51:32.043 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.043 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.043 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:32.043 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.093086+0000", 2024-09-11T09:51:32.043 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0016662510000000001 2024-09-11T09:51:32.044 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.044 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.044 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:32.044 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.093107+0000", 2024-09-11T09:51:32.044 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.0999e-05 2024-09-11T09:51:32.044 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.044 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.044 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:32.044 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.093120+0000", 2024-09-11T09:51:32.044 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2544999999999999e-05 2024-09-11T09:51:32.044 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.044 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.045 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:32.045 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.093186+0000", 2024-09-11T09:51:32.045 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.6595999999999997e-05 2024-09-11T09:51:32.045 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.045 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.045 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:32.045 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.093208+0000", 2024-09-11T09:51:32.045 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.1624000000000002e-05 2024-09-11T09:51:32.045 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.045 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.045 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:32.045 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.104392+0000", 2024-09-11T09:51:32.046 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.011184265000000001 2024-09-11T09:51:32.046 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.046 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.046 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:32.046 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.104446+0000", 2024-09-11T09:51:32.046 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.3402000000000001e-05 2024-09-11T09:51:32.046 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:32.046 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:32.046 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:32.046 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.046 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.046 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4694.0:45 2.6 2:671fb830:ns::rbd_data.1256a0c02879.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:32.046 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.109629+0000", 2024-09-11T09:51:32.047 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 8.8370526609999995, 2024-09-11T09:51:32.047 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.036403143999999998, 2024-09-11T09:51:32.047 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:32.047 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:32.047 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:32.047 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:32.047 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4694", 2024-09-11T09:51:32.047 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:51:32.047 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 45 2024-09-11T09:51:32.047 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.048 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:32.048 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.048 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:32.048 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.109629+0000", 2024-09-11T09:51:32.048 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:32.048 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.048 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.048 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:32.048 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.109629+0000", 2024-09-11T09:51:32.048 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:32.048 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.049 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.049 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:32.049 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.109630+0000", 2024-09-11T09:51:32.049 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1200000000000001e-06 2024-09-11T09:51:32.049 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.049 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.049 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:32.049 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.121370+0000", 2024-09-11T09:51:32.049 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.011739571000000001 2024-09-11T09:51:32.049 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.049 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.049 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:32.050 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.121391+0000", 2024-09-11T09:51:32.050 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.1175e-05 2024-09-11T09:51:32.050 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.050 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.050 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:32.050 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.121404+0000", 2024-09-11T09:51:32.050 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2807e-05 2024-09-11T09:51:32.050 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.050 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.050 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:32.050 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.121498+0000", 2024-09-11T09:51:32.050 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.3770999999999994e-05 2024-09-11T09:51:32.051 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.051 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.051 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:32.051 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.121612+0000", 2024-09-11T09:51:32.051 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000114666 2024-09-11T09:51:32.051 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.051 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.051 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 3", 2024-09-11T09:51:32.051 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.121711+0000", 2024-09-11T09:51:32.051 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.8647999999999995e-05 2024-09-11T09:51:32.051 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.051 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.052 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:32.052 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.126558+0000", 2024-09-11T09:51:32.052 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0048473769999999999 2024-09-11T09:51:32.052 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.052 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.052 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:32.052 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.145912+0000", 2024-09-11T09:51:32.052 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.019354281000000001 2024-09-11T09:51:32.052 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.052 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.052 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:32.053 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.145977+0000", 2024-09-11T09:51:32.053 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.4135999999999997e-05 2024-09-11T09:51:32.053 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.053 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.053 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:32.053 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.146032+0000", 2024-09-11T09:51:32.053 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.5591999999999999e-05 2024-09-11T09:51:32.053 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:32.053 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:32.053 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:32.053 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.054 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.054 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4694.0:41 2.2 e53/15 2:5e2505c4:ns::rbd_data.1256a0c02879.0000000000000003:head v 53'4, mlcod=18'2)", 2024-09-11T09:51:32.054 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.113199+0000", 2024-09-11T09:51:32.054 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 8.8334822509999995, 2024-09-11T09:51:32.054 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.011404512, 2024-09-11T09:51:32.054 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:32.054 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:32.054 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:32.054 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:32.054 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.054 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:32.055 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.113199+0000", 2024-09-11T09:51:32.055 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:32.055 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.055 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.055 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:32.055 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.113199+0000", 2024-09-11T09:51:32.055 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:32.055 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.055 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.055 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:32.055 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.113200+0000", 2024-09-11T09:51:32.055 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.2799999999999995e-07 2024-09-11T09:51:32.055 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.056 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.056 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:32.056 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.118872+0000", 2024-09-11T09:51:32.056 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0056718089999999999 2024-09-11T09:51:32.056 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.056 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.056 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:32.056 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.118889+0000", 2024-09-11T09:51:32.056 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7428999999999999e-05 2024-09-11T09:51:32.056 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.057 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.057 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:32.057 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.118900+0000", 2024-09-11T09:51:32.057 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0243e-05 2024-09-11T09:51:32.057 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.057 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.057 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:32.057 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.118959+0000", 2024-09-11T09:51:32.057 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.9364000000000003e-05 2024-09-11T09:51:32.057 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.057 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.058 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:32.058 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.118976+0000", 2024-09-11T09:51:32.058 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7124000000000001e-05 2024-09-11T09:51:32.058 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.058 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.058 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:32.058 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.124557+0000", 2024-09-11T09:51:32.058 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0055812659999999997 2024-09-11T09:51:32.058 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.058 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.058 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:32.059 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.124604+0000", 2024-09-11T09:51:32.059 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.6548999999999998e-05 2024-09-11T09:51:32.059 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:32.059 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:32.059 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:32.059 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.059 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.059 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4694.0:43 2.0 e53/15 2:16c8ae84:ns::rbd_data.1256a0c02879.0000000000000005:head v 53'2, mlcod=0'0)", 2024-09-11T09:51:32.059 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.158559+0000", 2024-09-11T09:51:32.059 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 8.7881229249999997, 2024-09-11T09:51:32.059 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.016713677999999999, 2024-09-11T09:51:32.060 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:32.060 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:32.060 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:32.060 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:32.060 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.060 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:32.060 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.158559+0000", 2024-09-11T09:51:32.060 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:32.060 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.060 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.060 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:32.060 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.158559+0000", 2024-09-11T09:51:32.061 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:32.061 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.061 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.061 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:32.061 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.158560+0000", 2024-09-11T09:51:32.061 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1459999999999999e-06 2024-09-11T09:51:32.061 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.061 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.061 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:32.061 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.166465+0000", 2024-09-11T09:51:32.061 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.007904767 2024-09-11T09:51:32.061 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.061 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.062 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:32.062 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.166491+0000", 2024-09-11T09:51:32.062 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.6308e-05 2024-09-11T09:51:32.062 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.062 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.062 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:32.062 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.166502+0000", 2024-09-11T09:51:32.062 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0736e-05 2024-09-11T09:51:32.062 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.062 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.062 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:32.062 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.166564+0000", 2024-09-11T09:51:32.063 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.2578999999999998e-05 2024-09-11T09:51:32.063 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.063 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.063 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:32.063 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.166585+0000", 2024-09-11T09:51:32.063 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.0395999999999999e-05 2024-09-11T09:51:32.063 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.063 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.063 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:32.063 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.175227+0000", 2024-09-11T09:51:32.064 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0086426420000000007 2024-09-11T09:51:32.064 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.064 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.064 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:32.064 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.175272+0000", 2024-09-11T09:51:32.064 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.5104e-05 2024-09-11T09:51:32.064 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:32.064 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:32.064 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:32.064 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.064 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.064 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4694.0:59 2.2 e53/15 2:41adefc9:ns::rbd_data.1256a0c02879.000000000000000e:head v 53'5, mlcod=53'3)", 2024-09-11T09:51:32.065 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.163389+0000", 2024-09-11T09:51:32.065 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 8.7832929489999998, 2024-09-11T09:51:32.065 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.013095719, 2024-09-11T09:51:32.065 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:32.065 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:32.065 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:32.065 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:32.065 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.065 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:32.065 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.163389+0000", 2024-09-11T09:51:32.066 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:32.066 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.066 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.066 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:32.066 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.163389+0000", 2024-09-11T09:51:32.066 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:32.066 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.066 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.066 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:32.067 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.163390+0000", 2024-09-11T09:51:32.067 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.336e-06 2024-09-11T09:51:32.067 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.067 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.067 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:32.067 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.170335+0000", 2024-09-11T09:51:32.067 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0069451080000000002 2024-09-11T09:51:32.067 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.067 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.067 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:32.067 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.170354+0000", 2024-09-11T09:51:32.068 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.8423e-05 2024-09-11T09:51:32.068 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.068 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.068 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:32.068 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.170366+0000", 2024-09-11T09:51:32.068 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2235999999999999e-05 2024-09-11T09:51:32.068 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.068 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.068 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:32.068 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.170439+0000", 2024-09-11T09:51:32.068 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.2989000000000002e-05 2024-09-11T09:51:32.069 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.069 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.069 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:32.069 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.170460+0000", 2024-09-11T09:51:32.069 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.1211000000000001e-05 2024-09-11T09:51:32.069 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.069 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.069 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:32.069 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.176442+0000", 2024-09-11T09:51:32.069 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0059821969999999999 2024-09-11T09:51:32.069 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.069 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.070 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:32.070 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.176484+0000", 2024-09-11T09:51:32.070 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.2218999999999997e-05 2024-09-11T09:51:32.070 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:32.070 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:32.070 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:32.070 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:32.070 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:32.070 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:51:32.072 INFO:tasks.workunit.client.0.smithi107.stderr:+ DEV=/dev/nbd0 2024-09-11T09:51:32.072 INFO:tasks.workunit.client.0.smithi107.stderr:+ get_pid rbd 2024-09-11T09:51:32.072 INFO:tasks.workunit.client.0.smithi107.stderr:+ local pool=rbd 2024-09-11T09:51:32.073 INFO:tasks.workunit.client.0.smithi107.stderr:+ local ns= 2024-09-11T09:51:32.073 INFO:tasks.workunit.client.0.smithi107.stderr:++ rbd device --device-type nbd --format xml list 2024-09-11T09:51:32.073 INFO:tasks.workunit.client.0.smithi107.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd27121'\''][device='\''/dev/nbd0'\'']/id' 2024-09-11T09:51:32.073 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-11T09:51:32.074 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:51:32.073+0000 7f3c96ec9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31909) UID: 0 2024-09-11T09:51:32.079 INFO:tasks.workunit.client.0.smithi107.stderr:+ PID=27682 2024-09-11T09:51:32.079 INFO:tasks.workunit.client.0.smithi107.stderr:+ test -n 27682 2024-09-11T09:51:32.079 INFO:tasks.workunit.client.0.smithi107.stderr:+ ps -p 27682 -C rbd-nbd 2024-09-11T09:51:32.090 INFO:tasks.workunit.client.0.smithi107.stdout: PID TTY TIME CMD 2024-09-11T09:51:32.090 INFO:tasks.workunit.client.0.smithi107.stdout: 27682 ? 00:00:00 rbd-nbd 2024-09-11T09:51:32.091 INFO:tasks.workunit.client.0.smithi107.stderr:++ rbd -p rbd --format xml du testrbdnbd27121 2024-09-11T09:51:32.091 INFO:tasks.workunit.client.0.smithi107.stderr:++ xmlstarlet sel -t -m //stats/images/image/provisioned_size -v . 2024-09-11T09:51:32.136 INFO:tasks.workunit.client.0.smithi107.stderr:+ provisioned=67108864 2024-09-11T09:51:32.137 INFO:tasks.workunit.client.0.smithi107.stderr:++ rbd -p rbd --format xml du testrbdnbd27121 2024-09-11T09:51:32.137 INFO:tasks.workunit.client.0.smithi107.stderr:++ xmlstarlet sel -t -m //stats/images/image/used_size -v . 2024-09-11T09:51:32.172 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:51:32.173+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:51:32.184 INFO:tasks.workunit.client.0.smithi107.stderr:+ used=67108864 2024-09-11T09:51:32.184 INFO:tasks.workunit.client.0.smithi107.stderr:+ '[' 67108864 -eq 67108864 ']' 2024-09-11T09:51:32.184 INFO:tasks.workunit.client.0.smithi107.stderr:+ expect_false _sudo blkdiscard /dev/nbd0 2024-09-11T09:51:32.184 INFO:tasks.workunit.client.0.smithi107.stderr:+ _sudo blkdiscard /dev/nbd0 2024-09-11T09:51:32.184 INFO:tasks.workunit.client.0.smithi107.stderr:+ local cmd 2024-09-11T09:51:32.184 INFO:tasks.workunit.client.0.smithi107.stderr:++ id -u 2024-09-11T09:51:32.185 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:51:32.185 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [], 2024-09-11T09:51:32.185 INFO:teuthology.orchestra.run.smithi017.stdout: "num_ops": 0 2024-09-11T09:51:32.185 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:51:32.185 INFO:tasks.workunit.client.0.smithi107.stderr:+ '[' 1000 -eq 0 ']' 2024-09-11T09:51:32.185 INFO:tasks.workunit.client.0.smithi107.stderr:++ which blkdiscard 2024-09-11T09:51:32.187 INFO:tasks.workunit.client.0.smithi107.stderr:+ cmd=/usr/sbin/blkdiscard 2024-09-11T09:51:32.187 INFO:tasks.workunit.client.0.smithi107.stderr:+ shift 2024-09-11T09:51:32.187 INFO:tasks.workunit.client.0.smithi107.stderr:+ sudo -nE /usr/sbin/blkdiscard /dev/nbd0 2024-09-11T09:51:32.193 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-11T09:51:32.209 INFO:tasks.workunit.client.0.smithi107.stderr:blkdiscard: /dev/nbd0: BLKDISCARD ioctl failed: Operation not supported 2024-09-11T09:51:32.211 INFO:tasks.workunit.client.0.smithi107.stderr:+ return 0 2024-09-11T09:51:32.211 INFO:tasks.workunit.client.0.smithi107.stderr:+ sync 2024-09-11T09:51:32.278 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:51:32.279+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:51:32.341 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:51:32.341 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [], 2024-09-11T09:51:32.341 INFO:teuthology.orchestra.run.smithi017.stdout: "complaint_time": 30, 2024-09-11T09:51:32.341 INFO:teuthology.orchestra.run.smithi017.stdout: "num_blocked_ops": 0 2024-09-11T09:51:32.341 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:51:32.349 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-11T09:51:32.374 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:51:32.374+0000 7f3c96ec9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31909) UID: 0 2024-09-11T09:51:32.475 INFO:tasks.ceph.osd.2.smithi017.stderr:2024-09-11T09:51:32.476+0000 7fd20b170640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31672) UID: 0 2024-09-11T09:51:32.498 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:51:32.498 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 20, 2024-09-11T09:51:32.498 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 600, 2024-09-11T09:51:32.498 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [ 2024-09-11T09:51:32.499 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.499 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:6 2.1 2:92878cd2:::rbd_id.testrbdnbd27121:head [create,call rbd.set_id in=9b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:32.499 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.873158+0000", 2024-09-11T09:51:32.499 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 9.6251348520000004, 2024-09-11T09:51:32.499 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.001212411, 2024-09-11T09:51:32.499 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:32.499 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:32.499 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:32.499 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:32.499 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:51:32.499 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:51:32.499 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 6 2024-09-11T09:51:32.500 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.500 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:32.500 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.500 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:32.500 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.873158+0000", 2024-09-11T09:51:32.500 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:32.500 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.500 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.500 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:32.500 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.873158+0000", 2024-09-11T09:51:32.500 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:32.500 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.500 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.501 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:32.501 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.873159+0000", 2024-09-11T09:51:32.501 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.6400000000000001e-07 2024-09-11T09:51:32.501 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.501 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.501 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:32.501 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.873164+0000", 2024-09-11T09:51:32.501 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.7419999999999997e-06 2024-09-11T09:51:32.501 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.501 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.501 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:32.501 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.873175+0000", 2024-09-11T09:51:32.501 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0774999999999999e-05 2024-09-11T09:51:32.502 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.502 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.502 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:32.502 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.873182+0000", 2024-09-11T09:51:32.502 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.5240000000000003e-06 2024-09-11T09:51:32.502 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.502 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.502 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:32.502 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.873223+0000", 2024-09-11T09:51:32.502 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.0479000000000003e-05 2024-09-11T09:51:32.502 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.502 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.502 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:32.503 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.873299+0000", 2024-09-11T09:51:32.503 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.6774999999999996e-05 2024-09-11T09:51:32.503 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.503 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.503 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1", 2024-09-11T09:51:32.503 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.873412+0000", 2024-09-11T09:51:32.503 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000112489 2024-09-11T09:51:32.503 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.503 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.503 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:32.503 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.874213+0000", 2024-09-11T09:51:32.503 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00080068299999999995 2024-09-11T09:51:32.503 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.504 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.504 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:32.504 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.874311+0000", 2024-09-11T09:51:32.504 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.8301999999999995e-05 2024-09-11T09:51:32.504 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.504 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.504 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:32.504 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.874344+0000", 2024-09-11T09:51:32.504 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.3441000000000001e-05 2024-09-11T09:51:32.504 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.504 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.504 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:32.504 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.874371+0000", 2024-09-11T09:51:32.505 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.6437000000000001e-05 2024-09-11T09:51:32.505 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:32.505 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:32.505 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:32.505 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.505 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.505 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:31 2.1 2:8654721b:::rbd_data.1250f5e25b80.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:32.505 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.896024+0000", 2024-09-11T09:51:32.505 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 9.6022692440000004, 2024-09-11T09:51:32.505 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.023148401999999998, 2024-09-11T09:51:32.505 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:32.505 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:32.505 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:32.506 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:32.506 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:51:32.506 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:51:32.506 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 31 2024-09-11T09:51:32.506 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.506 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:32.506 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.506 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:32.506 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.896024+0000", 2024-09-11T09:51:32.506 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:32.506 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.506 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.506 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:32.506 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.896024+0000", 2024-09-11T09:51:32.507 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:32.507 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.507 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.507 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:32.507 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.896025+0000", 2024-09-11T09:51:32.507 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.54e-07 2024-09-11T09:51:32.507 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.507 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.507 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:32.507 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.904287+0000", 2024-09-11T09:51:32.507 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0082624729999999993 2024-09-11T09:51:32.507 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.507 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.508 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:32.508 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.904305+0000", 2024-09-11T09:51:32.508 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7912999999999999e-05 2024-09-11T09:51:32.508 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.508 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.508 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:32.508 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.904312+0000", 2024-09-11T09:51:32.508 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.0230000000000004e-06 2024-09-11T09:51:32.508 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.508 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.508 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:32.508 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.904395+0000", 2024-09-11T09:51:32.508 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.2355999999999996e-05 2024-09-11T09:51:32.509 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.509 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.509 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:32.509 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.904499+0000", 2024-09-11T09:51:32.509 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000104672 2024-09-11T09:51:32.509 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.509 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.509 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1", 2024-09-11T09:51:32.509 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.904594+0000", 2024-09-11T09:51:32.509 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.5129000000000003e-05 2024-09-11T09:51:32.509 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.509 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.509 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:32.509 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.918183+0000", 2024-09-11T09:51:32.510 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.013588355999999999 2024-09-11T09:51:32.510 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.512 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.512 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:32.512 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919053+0000", 2024-09-11T09:51:32.513 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00087066800000000001 2024-09-11T09:51:32.513 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.513 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.513 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:32.513 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919125+0000", 2024-09-11T09:51:32.513 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.1333000000000003e-05 2024-09-11T09:51:32.513 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.513 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.513 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:32.513 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919172+0000", 2024-09-11T09:51:32.513 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.7725000000000001e-05 2024-09-11T09:51:32.513 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:32.513 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:32.514 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:32.514 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.514 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.514 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4688.0:42 2.3 e53/15 2:c54742d8:::rbd_data.1250f5e25b80.0000000000000006:head v 53'2, mlcod=0'0)", 2024-09-11T09:51:32.514 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.939272+0000", 2024-09-11T09:51:32.514 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 9.5590212599999997, 2024-09-11T09:51:32.514 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.010821958, 2024-09-11T09:51:32.514 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:32.514 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:32.514 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:32.514 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:32.514 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.514 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:32.514 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.939272+0000", 2024-09-11T09:51:32.515 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:32.515 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.515 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.515 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:32.515 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.939272+0000", 2024-09-11T09:51:32.515 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:32.515 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.515 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.515 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:32.515 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.939273+0000", 2024-09-11T09:51:32.515 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.7899999999999997e-07 2024-09-11T09:51:32.515 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.515 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.516 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:32.516 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.944403+0000", 2024-09-11T09:51:32.516 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0051302129999999998 2024-09-11T09:51:32.516 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.516 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.516 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:32.516 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.944413+0000", 2024-09-11T09:51:32.516 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0182e-05 2024-09-11T09:51:32.516 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.516 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.516 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:32.516 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.944420+0000", 2024-09-11T09:51:32.516 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.7000000000000002e-06 2024-09-11T09:51:32.517 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.517 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.517 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:32.517 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.944482+0000", 2024-09-11T09:51:32.517 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.1911999999999999e-05 2024-09-11T09:51:32.517 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.517 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.517 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:32.517 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.944508+0000", 2024-09-11T09:51:32.517 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.6582000000000001e-05 2024-09-11T09:51:32.517 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.517 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.518 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:32.518 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.950050+0000", 2024-09-11T09:51:32.518 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0055414080000000003 2024-09-11T09:51:32.518 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.518 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.518 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:32.518 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.950094+0000", 2024-09-11T09:51:32.518 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.4082000000000002e-05 2024-09-11T09:51:32.518 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:32.518 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:32.518 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:32.518 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.518 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.519 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:47 2.1 2:95f53b3e:::rbd_data.1250f5e25b80.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:32.519 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.948330+0000", 2024-09-11T09:51:32.519 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 9.5499629160000001, 2024-09-11T09:51:32.519 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.025156603999999999, 2024-09-11T09:51:32.519 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:32.519 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:32.519 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:32.519 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:32.519 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:51:32.519 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:51:32.519 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 47 2024-09-11T09:51:32.519 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.519 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:32.519 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.520 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:32.520 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.948330+0000", 2024-09-11T09:51:32.520 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:32.520 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.520 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.520 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:32.520 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.948330+0000", 2024-09-11T09:51:32.520 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:32.520 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.520 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.520 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:32.520 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.948333+0000", 2024-09-11T09:51:32.520 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.481e-06 2024-09-11T09:51:32.520 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.521 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.521 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:32.521 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.960263+0000", 2024-09-11T09:51:32.521 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.011930671 2024-09-11T09:51:32.521 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.521 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.521 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:32.521 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.960289+0000", 2024-09-11T09:51:32.521 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.5573000000000001e-05 2024-09-11T09:51:32.521 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.521 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.521 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:32.521 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.960304+0000", 2024-09-11T09:51:32.522 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.4647000000000001e-05 2024-09-11T09:51:32.522 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.522 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.522 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:32.522 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.960390+0000", 2024-09-11T09:51:32.522 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.6088999999999995e-05 2024-09-11T09:51:32.522 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.522 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.522 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:32.522 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.960562+0000", 2024-09-11T09:51:32.522 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00017237400000000001 2024-09-11T09:51:32.522 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.522 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.522 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1", 2024-09-11T09:51:32.523 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.960725+0000", 2024-09-11T09:51:32.523 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00016321599999999999 2024-09-11T09:51:32.523 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.523 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.523 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:32.523 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.969465+0000", 2024-09-11T09:51:32.523 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0087396899999999996 2024-09-11T09:51:32.523 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.523 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.523 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:32.523 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.973393+0000", 2024-09-11T09:51:32.523 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0039274669999999996 2024-09-11T09:51:32.523 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.523 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.524 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:32.524 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.973443+0000", 2024-09-11T09:51:32.524 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.0340000000000003e-05 2024-09-11T09:51:32.524 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.524 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.524 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:32.524 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.973487+0000", 2024-09-11T09:51:32.524 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.4056e-05 2024-09-11T09:51:32.524 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:32.524 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:32.524 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:32.524 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.524 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.525 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:61 2.1 2:9af038b1:::rbd_data.1250f5e25b80.000000000000000f:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:32.525 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.975764+0000", 2024-09-11T09:51:32.525 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 9.5225293919999991, 2024-09-11T09:51:32.525 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.016158891000000002, 2024-09-11T09:51:32.525 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:32.525 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:32.525 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:32.525 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:32.525 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:51:32.525 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:51:32.525 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 61 2024-09-11T09:51:32.525 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.525 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:32.525 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.526 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:32.526 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.975764+0000", 2024-09-11T09:51:32.526 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:32.526 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.526 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.526 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:32.526 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.975764+0000", 2024-09-11T09:51:32.526 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:32.526 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.526 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.526 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:32.526 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.975766+0000", 2024-09-11T09:51:32.526 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.767e-06 2024-09-11T09:51:32.526 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.527 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.527 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:32.527 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.983204+0000", 2024-09-11T09:51:32.527 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0074388400000000004 2024-09-11T09:51:32.527 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.527 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.527 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:32.527 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.983226+0000", 2024-09-11T09:51:32.527 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.1279000000000001e-05 2024-09-11T09:51:32.527 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.527 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.527 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:32.527 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.983238+0000", 2024-09-11T09:51:32.527 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2563e-05 2024-09-11T09:51:32.528 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.528 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.528 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:32.528 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.983318+0000", 2024-09-11T09:51:32.528 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.0075999999999997e-05 2024-09-11T09:51:32.528 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.528 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.528 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:32.528 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.983429+0000", 2024-09-11T09:51:32.528 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000110618 2024-09-11T09:51:32.528 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.528 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.528 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1", 2024-09-11T09:51:32.529 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.983525+0000", 2024-09-11T09:51:32.529 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.603e-05 2024-09-11T09:51:32.529 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.529 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.529 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:32.529 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.989004+0000", 2024-09-11T09:51:32.529 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0054785429999999998 2024-09-11T09:51:32.529 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.529 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.529 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:32.529 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.991788+0000", 2024-09-11T09:51:32.529 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0027845729999999998 2024-09-11T09:51:32.529 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.529 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.530 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:32.530 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.991870+0000", 2024-09-11T09:51:32.530 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.1567e-05 2024-09-11T09:51:32.530 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.530 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.530 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:32.530 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.991923+0000", 2024-09-11T09:51:32.530 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.3035000000000003e-05 2024-09-11T09:51:32.530 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:32.530 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:32.530 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:32.530 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.530 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.530 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4694.0:9 2.1 2:839846fa:ns::rbd_header.1256a0c02879:head [create,call rbd.create in=9b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:32.531 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.062019+0000", 2024-09-11T09:51:32.531 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 9.4362746319999999, 2024-09-11T09:51:32.531 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.001293296, 2024-09-11T09:51:32.531 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:32.531 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:32.531 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:32.531 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:32.531 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4694", 2024-09-11T09:51:32.531 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:51:32.531 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 9 2024-09-11T09:51:32.531 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.531 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:32.531 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.531 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:32.532 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.062019+0000", 2024-09-11T09:51:32.532 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:32.532 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.532 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.532 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:32.532 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.062019+0000", 2024-09-11T09:51:32.532 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:32.532 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.532 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.532 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:32.532 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.062020+0000", 2024-09-11T09:51:32.532 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2610000000000001e-06 2024-09-11T09:51:32.532 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.533 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.533 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:32.533 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.062025+0000", 2024-09-11T09:51:32.533 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.7879999999999997e-06 2024-09-11T09:51:32.533 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.533 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.533 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:32.533 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.062033+0000", 2024-09-11T09:51:32.533 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.9489999999999996e-06 2024-09-11T09:51:32.533 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.533 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.533 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:32.533 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.062039+0000", 2024-09-11T09:51:32.533 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.781e-06 2024-09-11T09:51:32.534 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.534 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.534 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:32.534 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.062099+0000", 2024-09-11T09:51:32.534 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.9905999999999999e-05 2024-09-11T09:51:32.534 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.534 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.534 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:32.534 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.062247+0000", 2024-09-11T09:51:32.534 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000147472 2024-09-11T09:51:32.534 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.534 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.534 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1", 2024-09-11T09:51:32.534 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.062468+0000", 2024-09-11T09:51:32.535 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00022112999999999999 2024-09-11T09:51:32.535 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.535 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.535 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:32.535 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.063076+0000", 2024-09-11T09:51:32.535 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00060857100000000003 2024-09-11T09:51:32.535 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.535 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.535 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:32.535 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.063251+0000", 2024-09-11T09:51:32.535 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00017496499999999999 2024-09-11T09:51:32.535 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.535 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.535 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:32.536 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.063285+0000", 2024-09-11T09:51:32.536 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.3633999999999997e-05 2024-09-11T09:51:32.536 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.536 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.536 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:32.536 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.063312+0000", 2024-09-11T09:51:32.536 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.6838999999999998e-05 2024-09-11T09:51:32.536 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:32.536 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:32.536 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:32.536 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.536 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.536 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4694.0:32 2.1 2:8a862b48:ns::rbd_data.1256a0c02879.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:32.537 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.084089+0000", 2024-09-11T09:51:32.537 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 9.4142039180000001, 2024-09-11T09:51:32.537 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.020630553999999999, 2024-09-11T09:51:32.537 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:32.537 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:32.537 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:32.537 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:32.537 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4694", 2024-09-11T09:51:32.537 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:51:32.537 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 32 2024-09-11T09:51:32.537 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.537 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:32.537 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.538 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:32.538 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.084089+0000", 2024-09-11T09:51:32.538 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:32.538 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.538 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.538 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:32.538 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.084089+0000", 2024-09-11T09:51:32.538 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:32.538 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.538 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.538 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:32.538 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.084090+0000", 2024-09-11T09:51:32.538 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.2900000000000003e-07 2024-09-11T09:51:32.539 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.539 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.539 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:32.539 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.090866+0000", 2024-09-11T09:51:32.539 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0067756609999999997 2024-09-11T09:51:32.539 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.539 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.539 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:32.539 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.090891+0000", 2024-09-11T09:51:32.539 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.5119000000000001e-05 2024-09-11T09:51:32.539 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.539 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.539 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:32.539 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.090900+0000", 2024-09-11T09:51:32.540 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.4110000000000002e-06 2024-09-11T09:51:32.540 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.540 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.540 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:32.540 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.090968+0000", 2024-09-11T09:51:32.540 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.8041999999999997e-05 2024-09-11T09:51:32.540 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.540 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.540 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:32.540 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.091115+0000", 2024-09-11T09:51:32.540 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00014633499999999999 2024-09-11T09:51:32.540 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.541 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.541 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1", 2024-09-11T09:51:32.541 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.091284+0000", 2024-09-11T09:51:32.541 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000169836 2024-09-11T09:51:32.541 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.541 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.541 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:32.541 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.100896+0000", 2024-09-11T09:51:32.541 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0096118639999999995 2024-09-11T09:51:32.541 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.542 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.542 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:32.542 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.104604+0000", 2024-09-11T09:51:32.542 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0037074619999999999 2024-09-11T09:51:32.542 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.542 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.542 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:32.542 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.104669+0000", 2024-09-11T09:51:32.542 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.4836000000000001e-05 2024-09-11T09:51:32.542 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.542 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.542 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:32.542 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.104720+0000", 2024-09-11T09:51:32.543 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.1258999999999998e-05 2024-09-11T09:51:32.543 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:32.543 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:32.543 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:32.543 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.543 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.543 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4694.0:44 2.3 e53/15 2:cd1a7d1c:ns::rbd_data.1256a0c02879.0000000000000006:head v 53'3, mlcod=16'1)", 2024-09-11T09:51:32.543 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.120387+0000", 2024-09-11T09:51:32.543 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 9.377906265, 2024-09-11T09:51:32.543 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.011988468, 2024-09-11T09:51:32.543 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:32.543 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:32.543 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:32.544 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:32.544 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.544 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:32.544 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.120387+0000", 2024-09-11T09:51:32.544 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:32.544 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.544 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.544 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:32.544 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.120387+0000", 2024-09-11T09:51:32.544 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:32.544 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.544 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.544 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:32.545 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.120389+0000", 2024-09-11T09:51:32.545 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.018e-06 2024-09-11T09:51:32.545 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.545 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.545 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:32.545 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.127403+0000", 2024-09-11T09:51:32.545 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0070144769999999999 2024-09-11T09:51:32.545 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.545 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.545 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:32.545 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.127414+0000", 2024-09-11T09:51:32.545 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0991e-05 2024-09-11T09:51:32.545 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.546 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.546 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:32.546 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.127423+0000", 2024-09-11T09:51:32.546 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.8580000000000006e-06 2024-09-11T09:51:32.546 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.546 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.546 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:32.546 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.127468+0000", 2024-09-11T09:51:32.546 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.4830999999999998e-05 2024-09-11T09:51:32.546 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.546 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.546 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:32.546 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.127485+0000", 2024-09-11T09:51:32.547 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6838999999999999e-05 2024-09-11T09:51:32.547 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.547 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.547 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:32.547 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.132341+0000", 2024-09-11T09:51:32.547 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0048557959999999999 2024-09-11T09:51:32.547 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.547 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.547 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:32.547 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.132375+0000", 2024-09-11T09:51:32.547 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.4657999999999999e-05 2024-09-11T09:51:32.547 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:32.547 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:32.548 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:32.548 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.548 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.548 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4694.0:46 2.3 e53/15 2:ce0e1469:ns::rbd_data.1256a0c02879.0000000000000008:head v 53'4, mlcod=16'1)", 2024-09-11T09:51:32.548 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.129008+0000", 2024-09-11T09:51:32.548 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 9.3692856790000008, 2024-09-11T09:51:32.548 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0087151220000000005, 2024-09-11T09:51:32.548 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:32.548 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:32.548 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:32.548 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:32.548 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.548 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:32.549 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.129008+0000", 2024-09-11T09:51:32.549 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:32.549 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.549 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.549 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:32.549 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.129008+0000", 2024-09-11T09:51:32.549 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:32.549 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.549 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.549 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:32.549 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.129009+0000", 2024-09-11T09:51:32.549 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.367e-06 2024-09-11T09:51:32.549 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.549 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.550 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:32.550 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.132839+0000", 2024-09-11T09:51:32.550 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0038297610000000001 2024-09-11T09:51:32.550 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.550 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.550 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:32.550 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.132849+0000", 2024-09-11T09:51:32.550 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0672e-05 2024-09-11T09:51:32.550 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.550 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.550 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:32.550 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.132856+0000", 2024-09-11T09:51:32.550 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.1380000000000004e-06 2024-09-11T09:51:32.550 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.551 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.551 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:32.551 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.132891+0000", 2024-09-11T09:51:32.551 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.5712e-05 2024-09-11T09:51:32.551 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.551 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.551 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:32.551 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.132907+0000", 2024-09-11T09:51:32.551 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6166000000000001e-05 2024-09-11T09:51:32.551 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.551 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.551 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:32.551 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.137681+0000", 2024-09-11T09:51:32.552 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0047735299999999998 2024-09-11T09:51:32.552 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.552 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.552 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:32.552 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.137723+0000", 2024-09-11T09:51:32.552 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.1776000000000001e-05 2024-09-11T09:51:32.552 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:32.552 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:32.552 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:32.552 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.552 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.552 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4694.0:48 2.3 e53/15 2:d2090b70:ns::rbd_data.1256a0c02879.0000000000000009:head v 53'5, mlcod=53'2)", 2024-09-11T09:51:32.552 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.135108+0000", 2024-09-11T09:51:32.552 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 9.3631847819999994, 2024-09-11T09:51:32.553 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.010710679000000001, 2024-09-11T09:51:32.553 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:32.553 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:32.553 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:32.553 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:32.553 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.553 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:32.553 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.135108+0000", 2024-09-11T09:51:32.553 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:32.553 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.553 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.553 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:32.554 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.135108+0000", 2024-09-11T09:51:32.554 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:32.554 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.554 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.554 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:32.554 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.135110+0000", 2024-09-11T09:51:32.554 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.122e-06 2024-09-11T09:51:32.554 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.554 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.555 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:32.555 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.140876+0000", 2024-09-11T09:51:32.555 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.005766162 2024-09-11T09:51:32.555 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.555 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.555 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:32.555 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.140888+0000", 2024-09-11T09:51:32.555 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.201e-05 2024-09-11T09:51:32.555 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.555 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.555 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:32.555 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.140894+0000", 2024-09-11T09:51:32.555 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.1739999999999997e-06 2024-09-11T09:51:32.556 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.556 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.556 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:32.556 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.140971+0000", 2024-09-11T09:51:32.556 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.7495000000000003e-05 2024-09-11T09:51:32.556 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.556 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.556 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:32.556 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.140995+0000", 2024-09-11T09:51:32.556 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.3204999999999999e-05 2024-09-11T09:51:32.556 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.556 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.556 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:32.556 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.145757+0000", 2024-09-11T09:51:32.557 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0047627190000000003 2024-09-11T09:51:32.557 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.557 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.557 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:32.557 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.145819+0000", 2024-09-11T09:51:32.557 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.1791999999999994e-05 2024-09-11T09:51:32.557 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:32.557 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:32.557 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:32.557 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.557 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.557 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4694.0:50 2.1 2:83078786:ns::rbd_data.1256a0c02879.000000000000000a:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:32.557 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.141467+0000", 2024-09-11T09:51:32.558 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 9.3568260579999993, 2024-09-11T09:51:32.558 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.012785889, 2024-09-11T09:51:32.558 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:32.558 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:32.558 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:32.558 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:32.558 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4694", 2024-09-11T09:51:32.558 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:51:32.558 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 50 2024-09-11T09:51:32.558 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.558 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:32.558 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.558 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:32.559 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.141467+0000", 2024-09-11T09:51:32.559 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:32.559 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.559 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.559 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:32.559 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.141467+0000", 2024-09-11T09:51:32.559 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:32.559 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.559 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.559 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:32.559 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.141468+0000", 2024-09-11T09:51:32.559 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1009999999999999e-06 2024-09-11T09:51:32.559 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.560 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.560 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:32.560 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.146432+0000", 2024-09-11T09:51:32.560 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0049640539999999999 2024-09-11T09:51:32.560 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.560 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.560 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:32.560 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.146446+0000", 2024-09-11T09:51:32.560 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3254e-05 2024-09-11T09:51:32.560 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.560 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.560 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:32.560 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.146452+0000", 2024-09-11T09:51:32.561 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.207e-06 2024-09-11T09:51:32.561 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.561 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.561 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:32.561 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.146510+0000", 2024-09-11T09:51:32.561 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.8322999999999997e-05 2024-09-11T09:51:32.561 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.561 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.561 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:32.561 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.146650+0000", 2024-09-11T09:51:32.561 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00013947100000000001 2024-09-11T09:51:32.561 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.561 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.561 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1", 2024-09-11T09:51:32.562 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.146779+0000", 2024-09-11T09:51:32.562 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000129059 2024-09-11T09:51:32.562 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.562 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.562 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:32.562 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.153778+0000", 2024-09-11T09:51:32.562 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0069997130000000003 2024-09-11T09:51:32.562 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.562 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.562 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:32.562 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.154180+0000", 2024-09-11T09:51:32.562 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00040159800000000003 2024-09-11T09:51:32.562 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.563 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.563 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:32.563 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.154218+0000", 2024-09-11T09:51:32.563 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.7795e-05 2024-09-11T09:51:32.563 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.563 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.563 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:32.563 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.154253+0000", 2024-09-11T09:51:32.563 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.5314000000000001e-05 2024-09-11T09:51:32.563 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:32.563 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:32.563 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:32.563 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.563 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.564 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4694.0:54 2.3 e53/15 2:c44f0253:ns::rbd_data.1256a0c02879.000000000000000b:head v 53'6, mlcod=53'4)", 2024-09-11T09:51:32.564 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.154926+0000", 2024-09-11T09:51:32.564 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 9.3433676830000003, 2024-09-11T09:51:32.564 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0089497099999999996, 2024-09-11T09:51:32.564 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:32.564 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:32.564 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:32.564 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:32.564 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.564 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:32.564 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.154926+0000", 2024-09-11T09:51:32.564 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:32.564 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.565 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.565 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:32.565 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.154926+0000", 2024-09-11T09:51:32.565 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:32.565 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.565 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.565 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:32.565 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.154927+0000", 2024-09-11T09:51:32.565 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.7000000000000003e-07 2024-09-11T09:51:32.565 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.565 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.565 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:32.565 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.158712+0000", 2024-09-11T09:51:32.566 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0037855469999999998 2024-09-11T09:51:32.566 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.566 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.566 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:32.566 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.158726+0000", 2024-09-11T09:51:32.566 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3621000000000001e-05 2024-09-11T09:51:32.566 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.566 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.566 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:32.566 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.158732+0000", 2024-09-11T09:51:32.566 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.9089999999999997e-06 2024-09-11T09:51:32.566 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.566 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.566 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:32.567 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.158781+0000", 2024-09-11T09:51:32.567 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.8900000000000003e-05 2024-09-11T09:51:32.567 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.567 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.567 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:32.567 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.158805+0000", 2024-09-11T09:51:32.567 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.44e-05 2024-09-11T09:51:32.567 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.567 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.567 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:32.567 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.163826+0000", 2024-09-11T09:51:32.567 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0050206399999999998 2024-09-11T09:51:32.567 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.568 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.568 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:32.568 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.163875+0000", 2024-09-11T09:51:32.568 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.9722999999999998e-05 2024-09-11T09:51:32.568 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:32.568 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:32.568 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:32.568 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.568 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.568 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4694.0:56 2.3 e53/15 2:c462f029:ns::rbd_data.1256a0c02879.000000000000000c:head v 53'7, mlcod=53'4)", 2024-09-11T09:51:32.568 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.159549+0000", 2024-09-11T09:51:32.568 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 9.338743977, 2024-09-11T09:51:32.568 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.015591666000000001, 2024-09-11T09:51:32.569 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:32.569 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:32.569 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:32.569 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:32.569 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.569 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:32.569 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.159549+0000", 2024-09-11T09:51:32.569 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:32.569 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.569 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.569 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:32.569 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.159549+0000", 2024-09-11T09:51:32.569 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:32.569 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.570 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.570 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:32.570 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.159550+0000", 2024-09-11T09:51:32.570 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.1500000000000004e-07 2024-09-11T09:51:32.570 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.570 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.570 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:32.570 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.167834+0000", 2024-09-11T09:51:32.570 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0082840220000000003 2024-09-11T09:51:32.570 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.570 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.570 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:32.570 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.167857+0000", 2024-09-11T09:51:32.571 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.3337e-05 2024-09-11T09:51:32.571 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.571 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.571 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:32.571 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.167869+0000", 2024-09-11T09:51:32.571 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1556000000000001e-05 2024-09-11T09:51:32.571 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.571 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.571 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:32.571 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.167939+0000", 2024-09-11T09:51:32.571 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.0674999999999997e-05 2024-09-11T09:51:32.571 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.571 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.572 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:32.572 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.167967+0000", 2024-09-11T09:51:32.572 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.7322999999999999e-05 2024-09-11T09:51:32.572 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.572 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.572 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:32.572 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.175079+0000", 2024-09-11T09:51:32.572 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0071126150000000001 2024-09-11T09:51:32.572 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.572 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.572 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:32.572 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.175141+0000", 2024-09-11T09:51:32.572 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.1523e-05 2024-09-11T09:51:32.573 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:32.573 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:32.573 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:32.573 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.573 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.573 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4694.0:62 2.1 2:839846fa:ns::rbd_header.1256a0c02879:head [call lock.unlock in=10b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:32.573 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.202300+0000", 2024-09-11T09:51:32.573 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 9.2959935090000005, 2024-09-11T09:51:32.573 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.001036546, 2024-09-11T09:51:32.573 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:32.573 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:32.573 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:32.573 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:32.574 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4694", 2024-09-11T09:51:32.574 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:51:32.574 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 62 2024-09-11T09:51:32.574 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.574 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:32.574 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.574 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:32.574 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.202300+0000", 2024-09-11T09:51:32.574 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:32.574 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.574 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.574 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:32.574 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.202300+0000", 2024-09-11T09:51:32.574 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:32.575 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.575 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.575 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:32.575 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.202301+0000", 2024-09-11T09:51:32.575 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.454e-06 2024-09-11T09:51:32.575 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.575 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.575 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:32.575 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.202309+0000", 2024-09-11T09:51:32.575 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.7409999999999992e-06 2024-09-11T09:51:32.575 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.575 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.575 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:32.576 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.202326+0000", 2024-09-11T09:51:32.576 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6657e-05 2024-09-11T09:51:32.576 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.576 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.576 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:32.576 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.202333+0000", 2024-09-11T09:51:32.576 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.1400000000000002e-06 2024-09-11T09:51:32.576 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.576 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.576 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:32.576 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.202386+0000", 2024-09-11T09:51:32.576 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.3638999999999999e-05 2024-09-11T09:51:32.576 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.577 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.577 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:32.577 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.202445+0000", 2024-09-11T09:51:32.577 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.8121000000000001e-05 2024-09-11T09:51:32.577 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.577 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.577 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1", 2024-09-11T09:51:32.577 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.202552+0000", 2024-09-11T09:51:32.577 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000107067 2024-09-11T09:51:32.577 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.577 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.577 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:32.577 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.203012+0000", 2024-09-11T09:51:32.577 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00046002100000000002 2024-09-11T09:51:32.578 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.578 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.578 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:32.578 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.203280+0000", 2024-09-11T09:51:32.578 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00026879300000000003 2024-09-11T09:51:32.578 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.578 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.578 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:32.578 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.203311+0000", 2024-09-11T09:51:32.578 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.1013000000000003e-05 2024-09-11T09:51:32.578 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.578 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.578 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:32.579 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.203336+0000", 2024-09-11T09:51:32.579 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.4899999999999999e-05 2024-09-11T09:51:32.579 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:32.579 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:32.579 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:32.579 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.579 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.579 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4694.0:64 2.1 2:839846fa:ns::rbd_header.1256a0c02879:head [watch unwatch cookie 139806419992736] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:32.579 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.203588+0000", 2024-09-11T09:51:32.579 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 9.2947055370000005, 2024-09-11T09:51:32.579 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0011675959999999999, 2024-09-11T09:51:32.579 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:32.579 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:32.580 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:32.580 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:32.580 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4694", 2024-09-11T09:51:32.580 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:51:32.580 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 64 2024-09-11T09:51:32.580 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.580 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:32.580 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.580 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:32.580 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.203588+0000", 2024-09-11T09:51:32.580 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:32.580 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.580 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.580 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:32.581 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.203588+0000", 2024-09-11T09:51:32.581 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:32.581 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.581 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.581 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:32.581 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.203588+0000", 2024-09-11T09:51:32.581 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.6600000000000002e-07 2024-09-11T09:51:32.581 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.581 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.581 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:32.581 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.203590+0000", 2024-09-11T09:51:32.581 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.773e-06 2024-09-11T09:51:32.581 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.582 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.582 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:32.582 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.203594+0000", 2024-09-11T09:51:32.582 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.6129999999999997e-06 2024-09-11T09:51:32.582 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.582 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.582 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:32.582 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.203598+0000", 2024-09-11T09:51:32.582 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.0809999999999998e-06 2024-09-11T09:51:32.582 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.582 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.582 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:32.582 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.203853+0000", 2024-09-11T09:51:32.583 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00025543300000000001 2024-09-11T09:51:32.583 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.583 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.583 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:32.583 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.203898+0000", 2024-09-11T09:51:32.583 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.4771000000000002e-05 2024-09-11T09:51:32.583 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.583 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.583 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1", 2024-09-11T09:51:32.583 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.203974+0000", 2024-09-11T09:51:32.583 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.6062000000000005e-05 2024-09-11T09:51:32.583 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.583 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.583 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:32.584 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.204452+0000", 2024-09-11T09:51:32.584 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00047848399999999999 2024-09-11T09:51:32.584 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.584 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.584 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:32.584 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.204667+0000", 2024-09-11T09:51:32.584 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00021479600000000001 2024-09-11T09:51:32.584 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.584 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.584 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:32.584 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.204699+0000", 2024-09-11T09:51:32.584 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.1690000000000003e-05 2024-09-11T09:51:32.584 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.585 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.585 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:32.585 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.204755+0000", 2024-09-11T09:51:32.585 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.6527000000000003e-05 2024-09-11T09:51:32.585 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:32.585 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:32.585 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:32.585 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.585 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.585 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4953.0:1 2.1 2:92878cd2:::rbd_id.testrbdnbd27121:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e58)", 2024-09-11T09:51:32.585 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:27.868133+0000", 2024-09-11T09:51:32.585 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 4.6301605080000003, 2024-09-11T09:51:32.585 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.001745866, 2024-09-11T09:51:32.585 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:32.586 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:32.586 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "started", 2024-09-11T09:51:32.586 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:32.586 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4953", 2024-09-11T09:51:32.586 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/2878448434", 2024-09-11T09:51:32.586 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 1 2024-09-11T09:51:32.586 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.586 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:32.586 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.586 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:32.586 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:27.868133+0000", 2024-09-11T09:51:32.586 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:32.586 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.586 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.587 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:32.587 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:27.868133+0000", 2024-09-11T09:51:32.587 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:32.587 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.587 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.587 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:32.587 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:27.868134+0000", 2024-09-11T09:51:32.587 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6249999999999999e-06 2024-09-11T09:51:32.587 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.587 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.587 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:32.587 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:27.868140+0000", 2024-09-11T09:51:32.587 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.7459999999999997e-06 2024-09-11T09:51:32.587 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.588 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.588 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:32.588 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:27.868150+0000", 2024-09-11T09:51:32.588 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0011e-05 2024-09-11T09:51:32.588 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.588 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.588 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:32.588 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:27.868160+0000", 2024-09-11T09:51:32.588 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.3999999999999998e-06 2024-09-11T09:51:32.588 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.588 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.588 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:32.588 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:27.868217+0000", 2024-09-11T09:51:32.589 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.7034999999999999e-05 2024-09-11T09:51:32.589 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.589 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.589 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op must wait for map", 2024-09-11T09:51:32.589 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:27.868222+0000", 2024-09-11T09:51:32.589 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.8939999999999998e-06 2024-09-11T09:51:32.589 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.589 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.589 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:32.589 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:27.869732+0000", 2024-09-11T09:51:32.589 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0015099 2024-09-11T09:51:32.589 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.589 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.590 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:32.590 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:27.869820+0000", 2024-09-11T09:51:32.590 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.7793000000000005e-05 2024-09-11T09:51:32.590 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.590 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.590 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:32.590 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:27.869879+0000", 2024-09-11T09:51:32.590 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.8461999999999997e-05 2024-09-11T09:51:32.590 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:32.590 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:32.590 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:32.590 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.590 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.590 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4807.0:48 2.1 2:8654721b:::rbd_data.1250f5e25b80.0000000000000000:head [read 69632~61440] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e58)", 2024-09-11T09:51:32.591 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:28.353535+0000", 2024-09-11T09:51:32.591 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 4.144758715, 2024-09-11T09:51:32.591 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.001338119, 2024-09-11T09:51:32.591 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:32.591 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:32.591 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "started", 2024-09-11T09:51:32.591 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:32.591 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4807", 2024-09-11T09:51:32.591 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/2963330868", 2024-09-11T09:51:32.591 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 48 2024-09-11T09:51:32.591 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.591 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:32.591 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.592 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:32.592 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:28.353535+0000", 2024-09-11T09:51:32.592 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:32.592 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.592 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.592 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:32.592 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:28.353535+0000", 2024-09-11T09:51:32.592 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:32.592 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.592 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.592 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:32.592 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:28.353535+0000", 2024-09-11T09:51:32.592 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.8799999999999998e-07 2024-09-11T09:51:32.593 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.593 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.593 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:32.593 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:28.353536+0000", 2024-09-11T09:51:32.593 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3239999999999999e-06 2024-09-11T09:51:32.593 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.593 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.593 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:32.593 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:28.353540+0000", 2024-09-11T09:51:32.593 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.4979999999999998e-06 2024-09-11T09:51:32.593 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.593 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.593 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:32.594 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:28.353542+0000", 2024-09-11T09:51:32.594 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.6079999999999998e-06 2024-09-11T09:51:32.594 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.594 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.594 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:32.594 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:28.354553+0000", 2024-09-11T09:51:32.594 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0010111180000000001 2024-09-11T09:51:32.594 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.594 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.594 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:32.594 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:28.354629+0000", 2024-09-11T09:51:32.594 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.5456000000000004e-05 2024-09-11T09:51:32.594 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.595 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.595 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:32.595 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:28.354873+0000", 2024-09-11T09:51:32.595 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00024382700000000001 2024-09-11T09:51:32.595 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:32.595 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:32.595 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:32.595 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.595 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.595 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4807.0:56 2.1 2:9af038b1:::rbd_data.1250f5e25b80.000000000000000f:head [sparse-read 3731456~69632] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e58)", 2024-09-11T09:51:32.595 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:28.357995+0000", 2024-09-11T09:51:32.595 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 4.1402982020000003, 2024-09-11T09:51:32.596 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0013349659999999999, 2024-09-11T09:51:32.596 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:32.596 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:32.596 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "started", 2024-09-11T09:51:32.596 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:32.596 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4807", 2024-09-11T09:51:32.596 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/2963330868", 2024-09-11T09:51:32.596 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 56 2024-09-11T09:51:32.596 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.596 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:32.596 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.596 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:32.596 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:28.357995+0000", 2024-09-11T09:51:32.597 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:32.597 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.597 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.597 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:32.597 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:28.357995+0000", 2024-09-11T09:51:32.597 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:32.597 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.597 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.597 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:32.597 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:28.357995+0000", 2024-09-11T09:51:32.597 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.77e-07 2024-09-11T09:51:32.597 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.597 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.597 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:32.598 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:28.357997+0000", 2024-09-11T09:51:32.598 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2219999999999999e-06 2024-09-11T09:51:32.598 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.598 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.598 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:32.598 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:28.358000+0000", 2024-09-11T09:51:32.598 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.2049999999999998e-06 2024-09-11T09:51:32.598 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.598 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.598 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:32.598 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:28.358002+0000", 2024-09-11T09:51:32.598 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.6000000000000001e-06 2024-09-11T09:51:32.598 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.599 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.599 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:32.599 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:28.359004+0000", 2024-09-11T09:51:32.599 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.001001273 2024-09-11T09:51:32.599 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.599 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.599 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:32.599 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:28.359087+0000", 2024-09-11T09:51:32.599 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.3672999999999997e-05 2024-09-11T09:51:32.599 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.599 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.599 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:32.599 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:28.359330+0000", 2024-09-11T09:51:32.600 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000242616 2024-09-11T09:51:32.600 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:32.600 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:32.600 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:32.600 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.600 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.600 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4807.0:60 2.1 2:9af038b1:::rbd_data.1250f5e25b80.000000000000000f:head [read 3895296~36864] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e58)", 2024-09-11T09:51:32.600 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:28.359787+0000", 2024-09-11T09:51:32.600 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 4.1385065110000001, 2024-09-11T09:51:32.600 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0010384280000000001, 2024-09-11T09:51:32.600 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:32.600 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:32.600 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "started", 2024-09-11T09:51:32.601 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:32.601 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4807", 2024-09-11T09:51:32.601 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/2963330868", 2024-09-11T09:51:32.601 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 60 2024-09-11T09:51:32.601 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.601 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:32.601 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.601 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:32.601 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:28.359787+0000", 2024-09-11T09:51:32.601 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:32.601 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.601 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.601 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:32.602 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:28.359787+0000", 2024-09-11T09:51:32.602 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:32.602 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.602 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.602 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:32.602 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:28.359787+0000", 2024-09-11T09:51:32.602 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.6100000000000002e-07 2024-09-11T09:51:32.602 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.602 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.602 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:32.602 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:28.359793+0000", 2024-09-11T09:51:32.602 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.207e-06 2024-09-11T09:51:32.602 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.602 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.603 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:32.603 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:28.359799+0000", 2024-09-11T09:51:32.603 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.229e-06 2024-09-11T09:51:32.603 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.603 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.603 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:32.603 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:28.359802+0000", 2024-09-11T09:51:32.603 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.9409999999999999e-06 2024-09-11T09:51:32.603 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.603 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.603 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:32.603 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:28.360625+0000", 2024-09-11T09:51:32.603 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00082282500000000001 2024-09-11T09:51:32.604 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.604 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.604 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:32.604 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:28.360665+0000", 2024-09-11T09:51:32.604 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.9601000000000003e-05 2024-09-11T09:51:32.604 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.604 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.604 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:32.604 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:28.360825+0000", 2024-09-11T09:51:32.604 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000160364 2024-09-11T09:51:32.604 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:32.604 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:32.604 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:32.604 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.605 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.605 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.5001.0:56 2.1 2:9af038b1:::rbd_data.1250f5e25b80.000000000000000f:head [read 3710976~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e58)", 2024-09-11T09:51:32.605 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:29.260660+0000", 2024-09-11T09:51:32.605 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 3.2376334189999998, 2024-09-11T09:51:32.605 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.001058458, 2024-09-11T09:51:32.605 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:32.605 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:32.605 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "started", 2024-09-11T09:51:32.605 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:32.605 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.5001", 2024-09-11T09:51:32.605 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/2374572508", 2024-09-11T09:51:32.605 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 56 2024-09-11T09:51:32.605 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.606 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:32.606 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.606 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:32.606 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:29.260660+0000", 2024-09-11T09:51:32.606 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:32.606 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.606 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.606 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:32.606 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:29.260660+0000", 2024-09-11T09:51:32.606 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:32.606 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.606 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.606 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:32.606 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:29.260660+0000", 2024-09-11T09:51:32.607 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.8900000000000001e-07 2024-09-11T09:51:32.607 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.607 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.607 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:32.607 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:29.260661+0000", 2024-09-11T09:51:32.607 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.049e-06 2024-09-11T09:51:32.607 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.607 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.607 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:32.607 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:29.260664+0000", 2024-09-11T09:51:32.607 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.1920000000000001e-06 2024-09-11T09:51:32.607 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.607 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.607 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:32.608 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:29.260667+0000", 2024-09-11T09:51:32.608 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.5009999999999999e-06 2024-09-11T09:51:32.608 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.608 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.608 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:32.608 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:29.261548+0000", 2024-09-11T09:51:32.608 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00088111199999999998 2024-09-11T09:51:32.608 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.608 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.608 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:32.608 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:29.261618+0000", 2024-09-11T09:51:32.608 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.9894999999999994e-05 2024-09-11T09:51:32.608 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:32.608 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:32.609 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:32.609 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:29.261718+0000", 2024-09-11T09:51:32.609 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00010042 2024-09-11T09:51:32.609 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:32.609 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:32.609 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:32.609 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:32.609 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:32.609 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:51:32.613 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-11T09:51:32.614 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:51:32.614+0000 7f3c96ec9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31909) UID: 0 2024-09-11T09:51:32.714 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-09-11T09:51:32.715+0000 7f7b6c7d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31904) UID: 0 2024-09-11T09:51:32.722 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-11T09:51:32.722 INFO:teuthology.orchestra.run.smithi096.stdout: "ops": [], 2024-09-11T09:51:32.722 INFO:teuthology.orchestra.run.smithi096.stdout: "num_ops": 0 2024-09-11T09:51:32.722 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-11T09:51:32.731 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-11T09:51:32.747 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-11T09:51:32.757 INFO:teuthology.orchestra.run.smithi096.stdout:get_pg_num_history pg_num_history pg_num_history(e54 pg_nums {1={12=1},2={15=8},3={54=8}} deleted_pools ) 2024-09-11T09:51:32.757 INFO:teuthology.orchestra.run.smithi096.stdout:Importing pgid 3.3 2024-09-11T09:51:32.758 INFO:teuthology.orchestra.run.smithi096.stdout:pool 3 pg_num 8 -> 8 2024-09-11T09:51:32.758 INFO:teuthology.orchestra.run.smithi096.stdout:write_pg epoch 54 info 3.3( empty local-lis/les=0/0 n=0 ec=54/54 lis/c=0/0 les/c/f=0/0/0 sis=54) 2024-09-11T09:51:32.762 INFO:teuthology.orchestra.run.smithi096.stdout:Import successful 2024-09-11T09:51:32.816 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-09-11T09:51:32.815+0000 7f7b6c7d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31904) UID: 0 2024-09-11T09:51:32.860 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-11T09:51:32.860 INFO:teuthology.orchestra.run.smithi096.stdout: "ops": [], 2024-09-11T09:51:32.860 INFO:teuthology.orchestra.run.smithi096.stdout: "complaint_time": 30, 2024-09-11T09:51:32.860 INFO:teuthology.orchestra.run.smithi096.stdout: "num_blocked_ops": 0 2024-09-11T09:51:32.860 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-11T09:51:32.869 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-11T09:51:32.915 INFO:tasks.ceph.osd.2.smithi017.stderr:2024-09-11T09:51:32.916+0000 7fd20b170640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31672) UID: 0 2024-09-11T09:51:33.013 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-11T09:51:33.013 INFO:teuthology.orchestra.run.smithi096.stdout: "size": 20, 2024-09-11T09:51:33.013 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 600, 2024-09-11T09:51:33.014 INFO:teuthology.orchestra.run.smithi096.stdout: "ops": [ 2024-09-11T09:51:33.014 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.014 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:111 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:51:33.014 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.711668+0000", 2024-09-11T09:51:33.014 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 81.301325199000004, 2024-09-11T09:51:33.014 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0077145570000000004, 2024-09-11T09:51:33.014 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:33.014 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:33.014 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:33.015 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:33.015 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:51:33.015 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:51:33.015 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 111 2024-09-11T09:51:33.015 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.015 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:33.015 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.015 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:33.015 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711668+0000", 2024-09-11T09:51:33.015 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:33.015 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.015 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.015 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:33.015 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711668+0000", 2024-09-11T09:51:33.016 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:33.016 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.016 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.016 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:33.016 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711669+0000", 2024-09-11T09:51:33.016 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.99e-07 2024-09-11T09:51:33.016 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.016 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.016 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:33.016 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711673+0000", 2024-09-11T09:51:33.016 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.7440000000000001e-06 2024-09-11T09:51:33.016 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.016 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.017 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:33.017 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711686+0000", 2024-09-11T09:51:33.017 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.3317e-05 2024-09-11T09:51:33.017 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.017 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.017 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:33.017 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711697+0000", 2024-09-11T09:51:33.018 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1022000000000001e-05 2024-09-11T09:51:33.018 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.018 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.018 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:33.018 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.717065+0000", 2024-09-11T09:51:33.018 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.005367531 2024-09-11T09:51:33.018 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.018 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.018 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:33.018 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.717114+0000", 2024-09-11T09:51:33.018 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.9707999999999999e-05 2024-09-11T09:51:33.018 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.018 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.019 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:33.019 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.717190+0000", 2024-09-11T09:51:33.019 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.5394000000000004e-05 2024-09-11T09:51:33.019 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.019 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.019 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:33.019 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718793+0000", 2024-09-11T09:51:33.019 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001603653 2024-09-11T09:51:33.019 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.019 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.019 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:33.019 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719329+0000", 2024-09-11T09:51:33.019 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00053543799999999995 2024-09-11T09:51:33.019 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.020 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.020 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:33.020 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719359+0000", 2024-09-11T09:51:33.020 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.0420999999999999e-05 2024-09-11T09:51:33.020 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.020 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.020 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:33.020 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719383+0000", 2024-09-11T09:51:33.020 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.3730000000000001e-05 2024-09-11T09:51:33.020 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:33.021 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:33.021 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:33.021 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.021 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.021 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:112 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:51:33.021 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.711712+0000", 2024-09-11T09:51:33.021 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 81.301281638999995, 2024-09-11T09:51:33.021 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.008162084, 2024-09-11T09:51:33.021 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:33.021 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:33.022 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:33.022 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:33.022 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:51:33.022 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:51:33.022 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 112 2024-09-11T09:51:33.022 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.022 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:33.022 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.022 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:33.022 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711712+0000", 2024-09-11T09:51:33.022 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:33.022 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.022 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.023 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:33.023 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711712+0000", 2024-09-11T09:51:33.023 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:33.023 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.023 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.023 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:33.023 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711712+0000", 2024-09-11T09:51:33.023 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.6000000000000004e-07 2024-09-11T09:51:33.023 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.023 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.023 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:33.023 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711808+0000", 2024-09-11T09:51:33.023 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.5977999999999995e-05 2024-09-11T09:51:33.024 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.024 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.024 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:33.024 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711819+0000", 2024-09-11T09:51:33.024 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.0755e-05 2024-09-11T09:51:33.024 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.024 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.024 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:33.024 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711826+0000", 2024-09-11T09:51:33.025 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.7970000000000003e-06 2024-09-11T09:51:33.025 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.025 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.025 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:33.025 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.717435+0000", 2024-09-11T09:51:33.025 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0056090239999999998 2024-09-11T09:51:33.025 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.025 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.025 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:33.025 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.717496+0000", 2024-09-11T09:51:33.025 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.0543e-05 2024-09-11T09:51:33.025 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.025 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.026 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:33.026 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.717591+0000", 2024-09-11T09:51:33.026 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.5667000000000004e-05 2024-09-11T09:51:33.026 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.026 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.026 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:33.026 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718831+0000", 2024-09-11T09:51:33.026 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001240204 2024-09-11T09:51:33.027 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.027 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.027 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:33.027 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719819+0000", 2024-09-11T09:51:33.027 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00098751500000000001 2024-09-11T09:51:33.027 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.027 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.027 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:33.027 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719850+0000", 2024-09-11T09:51:33.027 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.1535999999999998e-05 2024-09-11T09:51:33.027 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.027 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.027 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:33.027 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719874+0000", 2024-09-11T09:51:33.028 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.3504999999999999e-05 2024-09-11T09:51:33.028 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:33.028 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:33.028 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:33.028 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.028 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.028 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:113 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:51:33.028 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.711842+0000", 2024-09-11T09:51:33.028 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 81.301151140000002, 2024-09-11T09:51:33.028 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0081160099999999999, 2024-09-11T09:51:33.028 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:33.028 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:33.028 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:33.029 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:33.029 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:51:33.029 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:51:33.029 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 113 2024-09-11T09:51:33.029 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.029 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:33.029 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.029 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:33.029 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711842+0000", 2024-09-11T09:51:33.029 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:33.030 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.030 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.030 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:33.030 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711842+0000", 2024-09-11T09:51:33.030 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:33.030 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.030 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.030 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:33.030 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711843+0000", 2024-09-11T09:51:33.030 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.4399999999999999e-07 2024-09-11T09:51:33.030 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.031 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.031 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:33.031 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711848+0000", 2024-09-11T09:51:33.031 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.4159999999999997e-06 2024-09-11T09:51:33.031 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.031 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.031 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:33.031 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711856+0000", 2024-09-11T09:51:33.031 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.7870000000000007e-06 2024-09-11T09:51:33.031 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.031 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.031 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:33.031 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711862+0000", 2024-09-11T09:51:33.032 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.3689999999999998e-06 2024-09-11T09:51:33.032 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.032 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.032 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:33.032 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.717965+0000", 2024-09-11T09:51:33.032 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0061028899999999997 2024-09-11T09:51:33.032 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.032 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.032 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:33.032 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718023+0000", 2024-09-11T09:51:33.032 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.8313000000000002e-05 2024-09-11T09:51:33.032 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.033 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.033 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:33.033 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718109+0000", 2024-09-11T09:51:33.033 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.6205000000000006e-05 2024-09-11T09:51:33.033 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.033 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.033 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:33.033 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719808+0000", 2024-09-11T09:51:33.033 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0016990060000000001 2024-09-11T09:51:33.033 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.033 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.033 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:33.033 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719883+0000", 2024-09-11T09:51:33.033 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.4624999999999998e-05 2024-09-11T09:51:33.034 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.034 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.034 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:33.034 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719910+0000", 2024-09-11T09:51:33.034 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.7512e-05 2024-09-11T09:51:33.034 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.034 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.034 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:33.034 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719958+0000", 2024-09-11T09:51:33.034 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.8143000000000003e-05 2024-09-11T09:51:33.034 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:33.034 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:33.034 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:33.035 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.035 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.035 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:51:33.035 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.711887+0000", 2024-09-11T09:51:33.035 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 81.301106525999998, 2024-09-11T09:51:33.035 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0086063589999999992, 2024-09-11T09:51:33.035 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:33.035 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:33.035 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:33.035 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:33.035 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:51:33.035 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:51:33.035 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 114 2024-09-11T09:51:33.035 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.036 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:33.036 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.036 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:33.036 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711887+0000", 2024-09-11T09:51:33.036 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:33.036 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.036 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.036 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:33.036 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711887+0000", 2024-09-11T09:51:33.036 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:33.036 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.036 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.036 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:33.036 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711887+0000", 2024-09-11T09:51:33.037 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.1500000000000005e-07 2024-09-11T09:51:33.037 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.037 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.037 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:33.037 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711897+0000", 2024-09-11T09:51:33.037 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.1900000000000001e-06 2024-09-11T09:51:33.037 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.037 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.037 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:33.037 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711902+0000", 2024-09-11T09:51:33.037 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.2349999999999999e-06 2024-09-11T09:51:33.037 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.037 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.038 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:33.038 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711906+0000", 2024-09-11T09:51:33.038 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.7129999999999999e-06 2024-09-11T09:51:33.038 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.038 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.038 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:33.038 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718429+0000", 2024-09-11T09:51:33.038 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0065231960000000002 2024-09-11T09:51:33.038 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.038 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.038 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:33.038 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718480+0000", 2024-09-11T09:51:33.038 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.1161e-05 2024-09-11T09:51:33.039 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.039 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.039 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:33.039 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718558+0000", 2024-09-11T09:51:33.039 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.7519000000000001e-05 2024-09-11T09:51:33.039 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.039 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.039 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:33.039 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720038+0000", 2024-09-11T09:51:33.039 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001480016 2024-09-11T09:51:33.039 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.039 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.039 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:33.040 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720417+0000", 2024-09-11T09:51:33.040 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00037943699999999999 2024-09-11T09:51:33.040 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.040 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.040 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:33.040 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720452+0000", 2024-09-11T09:51:33.040 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.4595000000000003e-05 2024-09-11T09:51:33.040 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.040 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.040 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:33.040 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720493+0000", 2024-09-11T09:51:33.040 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.1782000000000001e-05 2024-09-11T09:51:33.041 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:33.041 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:33.041 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:33.041 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.041 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.041 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:51:33.041 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.711912+0000", 2024-09-11T09:51:33.041 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 81.301081857, 2024-09-11T09:51:33.041 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0086864750000000008, 2024-09-11T09:51:33.041 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:33.041 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:33.041 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:33.042 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:33.042 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:51:33.042 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:51:33.042 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 115 2024-09-11T09:51:33.042 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.042 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:33.042 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.042 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:33.042 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711912+0000", 2024-09-11T09:51:33.042 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:33.042 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.042 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.042 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:33.043 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711912+0000", 2024-09-11T09:51:33.043 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:33.043 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.043 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.043 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:33.043 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711912+0000", 2024-09-11T09:51:33.043 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.3900000000000001e-07 2024-09-11T09:51:33.043 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.043 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.043 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:33.043 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711969+0000", 2024-09-11T09:51:33.043 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.7501999999999999e-05 2024-09-11T09:51:33.043 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.043 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.044 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:33.044 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711986+0000", 2024-09-11T09:51:33.044 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.6402999999999999e-05 2024-09-11T09:51:33.044 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.044 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.044 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:33.044 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711990+0000", 2024-09-11T09:51:33.044 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.929e-06 2024-09-11T09:51:33.044 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.044 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.044 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:33.044 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718861+0000", 2024-09-11T09:51:33.044 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0068710530000000002 2024-09-11T09:51:33.045 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.045 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.045 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:33.045 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718908+0000", 2024-09-11T09:51:33.045 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.7431e-05 2024-09-11T09:51:33.045 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.045 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.045 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:33.045 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718984+0000", 2024-09-11T09:51:33.045 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.5500999999999994e-05 2024-09-11T09:51:33.045 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.046 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.046 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:33.046 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720500+0000", 2024-09-11T09:51:33.046 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0015158820000000001 2024-09-11T09:51:33.046 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.046 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.046 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:33.046 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720536+0000", 2024-09-11T09:51:33.046 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.6118000000000002e-05 2024-09-11T09:51:33.046 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.046 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.046 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:33.046 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720570+0000", 2024-09-11T09:51:33.047 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.4387999999999997e-05 2024-09-11T09:51:33.047 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.047 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.047 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:33.047 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720598+0000", 2024-09-11T09:51:33.047 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.8028999999999999e-05 2024-09-11T09:51:33.047 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:33.047 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:33.047 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:33.047 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.047 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.047 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:51:33.047 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.711998+0000", 2024-09-11T09:51:33.047 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 81.300995678000007, 2024-09-11T09:51:33.048 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.010785813999999999, 2024-09-11T09:51:33.048 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:33.048 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:33.048 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:33.048 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:33.048 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:51:33.048 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:51:33.048 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 116 2024-09-11T09:51:33.048 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.048 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:33.048 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.048 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:33.049 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711998+0000", 2024-09-11T09:51:33.049 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:33.049 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.049 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.049 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:33.049 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711998+0000", 2024-09-11T09:51:33.049 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:33.049 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.049 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.049 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:33.049 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711998+0000", 2024-09-11T09:51:33.049 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.3300000000000003e-07 2024-09-11T09:51:33.049 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.049 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.050 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:33.050 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712001+0000", 2024-09-11T09:51:33.050 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.9349999999999999e-06 2024-09-11T09:51:33.050 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.050 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.050 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:33.050 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712006+0000", 2024-09-11T09:51:33.050 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.1630000000000004e-06 2024-09-11T09:51:33.050 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.050 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.050 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:33.050 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712010+0000", 2024-09-11T09:51:33.050 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.574e-06 2024-09-11T09:51:33.050 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.051 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.051 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:33.051 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719415+0000", 2024-09-11T09:51:33.051 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0074049099999999998 2024-09-11T09:51:33.051 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.051 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.051 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:33.051 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719465+0000", 2024-09-11T09:51:33.051 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.0129000000000001e-05 2024-09-11T09:51:33.051 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.051 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.051 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:33.051 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719545+0000", 2024-09-11T09:51:33.051 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.0253999999999994e-05 2024-09-11T09:51:33.052 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.052 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.052 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:33.052 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720636+0000", 2024-09-11T09:51:33.052 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0010909330000000001 2024-09-11T09:51:33.052 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.052 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.052 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:33.052 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.722707+0000", 2024-09-11T09:51:33.052 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0020713300000000001 2024-09-11T09:51:33.052 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.052 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.053 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:33.053 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.722743+0000", 2024-09-11T09:51:33.053 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.5351000000000001e-05 2024-09-11T09:51:33.053 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.053 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.053 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:33.053 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.722784+0000", 2024-09-11T09:51:33.053 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.0802e-05 2024-09-11T09:51:33.053 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:33.053 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:33.053 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:33.053 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.053 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.053 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:51:33.054 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.712014+0000", 2024-09-11T09:51:33.054 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 81.300979638000001, 2024-09-11T09:51:33.054 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.013420938, 2024-09-11T09:51:33.054 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:33.054 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:33.054 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:33.054 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:33.054 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:51:33.054 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:51:33.054 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 117 2024-09-11T09:51:33.054 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.054 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:33.054 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.054 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:33.055 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712014+0000", 2024-09-11T09:51:33.055 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:33.055 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.055 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.055 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:33.055 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712014+0000", 2024-09-11T09:51:33.055 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:33.055 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.055 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.055 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:33.055 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712014+0000", 2024-09-11T09:51:33.055 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.2000000000000001e-07 2024-09-11T09:51:33.055 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.056 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.056 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:33.056 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712016+0000", 2024-09-11T09:51:33.056 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.2309999999999999e-06 2024-09-11T09:51:33.056 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.056 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.056 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:33.056 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712020+0000", 2024-09-11T09:51:33.056 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.3830000000000001e-06 2024-09-11T09:51:33.056 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.056 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.056 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:33.057 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712023+0000", 2024-09-11T09:51:33.057 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.3340000000000002e-06 2024-09-11T09:51:33.057 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.057 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.057 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:33.057 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720083+0000", 2024-09-11T09:51:33.057 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0080601189999999993 2024-09-11T09:51:33.057 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.057 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.057 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:33.057 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720125+0000", 2024-09-11T09:51:33.057 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.2029e-05 2024-09-11T09:51:33.058 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.058 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.058 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:33.058 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720205+0000", 2024-09-11T09:51:33.058 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.0010000000000001e-05 2024-09-11T09:51:33.058 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.058 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.058 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:33.058 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720849+0000", 2024-09-11T09:51:33.058 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00064369899999999996 2024-09-11T09:51:33.058 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.058 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.058 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:33.059 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.725357+0000", 2024-09-11T09:51:33.059 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0045077379999999998 2024-09-11T09:51:33.059 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.059 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.059 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:33.059 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.725392+0000", 2024-09-11T09:51:33.059 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.5862999999999999e-05 2024-09-11T09:51:33.059 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.059 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.059 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:33.059 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.725435+0000", 2024-09-11T09:51:33.059 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.2311999999999997e-05 2024-09-11T09:51:33.059 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:33.060 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:33.060 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:33.060 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.060 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.060 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:159 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:51:33.060 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:12.452099+0000", 2024-09-11T09:51:33.060 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 80.560894050000002, 2024-09-11T09:51:33.060 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0069885950000000002, 2024-09-11T09:51:33.060 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:33.060 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:33.060 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:33.060 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:33.061 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:51:33.061 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:51:33.061 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 159 2024-09-11T09:51:33.061 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.061 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:33.061 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.061 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:33.061 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452099+0000", 2024-09-11T09:51:33.061 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:33.061 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.061 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.061 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:33.061 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452099+0000", 2024-09-11T09:51:33.062 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:33.062 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.062 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.062 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:33.062 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452100+0000", 2024-09-11T09:51:33.062 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.2800000000000003e-07 2024-09-11T09:51:33.062 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.062 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.062 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:33.062 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452102+0000", 2024-09-11T09:51:33.062 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.824e-06 2024-09-11T09:51:33.062 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.062 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.062 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:33.063 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452105+0000", 2024-09-11T09:51:33.063 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.0460000000000001e-06 2024-09-11T09:51:33.063 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.063 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.063 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:33.063 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452107+0000", 2024-09-11T09:51:33.063 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.543e-06 2024-09-11T09:51:33.063 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.063 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.064 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:33.064 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.455484+0000", 2024-09-11T09:51:33.064 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0033769759999999998 2024-09-11T09:51:33.064 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.064 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.064 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:33.064 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.455527+0000", 2024-09-11T09:51:33.064 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.2589999999999997e-05 2024-09-11T09:51:33.064 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.064 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.064 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:33.064 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.455594+0000", 2024-09-11T09:51:33.065 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.6775999999999999e-05 2024-09-11T09:51:33.065 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.065 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.065 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:33.065 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.456312+0000", 2024-09-11T09:51:33.065 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00071874700000000005 2024-09-11T09:51:33.065 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.065 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.065 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:33.065 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.459015+0000", 2024-09-11T09:51:33.065 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0027028540000000002 2024-09-11T09:51:33.065 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.065 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.066 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:33.066 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.459054+0000", 2024-09-11T09:51:33.066 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.8840000000000001e-05 2024-09-11T09:51:33.066 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.066 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.066 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:33.066 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.459088+0000", 2024-09-11T09:51:33.066 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.4071e-05 2024-09-11T09:51:33.066 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:33.066 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:33.066 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:33.066 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.067 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.067 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4688.0:38 2.5 2:abde2694:::rbd_data.1250f5e25b80.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:33.067 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:22.903926+0000", 2024-09-11T09:51:33.067 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 10.109067374, 2024-09-11T09:51:33.067 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.028303676, 2024-09-11T09:51:33.067 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:33.067 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:33.067 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:33.067 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:33.067 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4688", 2024-09-11T09:51:33.067 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:51:33.067 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 38 2024-09-11T09:51:33.067 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.068 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:33.068 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.068 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:33.068 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.903926+0000", 2024-09-11T09:51:33.068 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:33.068 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.068 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.068 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:33.068 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.903926+0000", 2024-09-11T09:51:33.068 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:33.068 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.068 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.069 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:33.069 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.903928+0000", 2024-09-11T09:51:33.069 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.4249999999999999e-06 2024-09-11T09:51:33.069 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.069 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.069 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:33.069 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.914774+0000", 2024-09-11T09:51:33.069 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.010846613 2024-09-11T09:51:33.069 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.069 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.069 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:33.069 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.914793+0000", 2024-09-11T09:51:33.069 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.8879999999999999e-05 2024-09-11T09:51:33.070 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.070 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.070 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:33.070 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.914801+0000", 2024-09-11T09:51:33.070 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.2230000000000007e-06 2024-09-11T09:51:33.070 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.070 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.070 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:33.070 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.914854+0000", 2024-09-11T09:51:33.070 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.2883999999999997e-05 2024-09-11T09:51:33.070 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.070 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.071 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:33.071 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.914951+0000", 2024-09-11T09:51:33.071 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.6731000000000006e-05 2024-09-11T09:51:33.071 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.071 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.071 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:33.071 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.915080+0000", 2024-09-11T09:51:33.071 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00012926999999999999 2024-09-11T09:51:33.071 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.071 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.071 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:33.071 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.931469+0000", 2024-09-11T09:51:33.071 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.016389107 2024-09-11T09:51:33.071 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.072 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.072 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:33.072 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.932118+0000", 2024-09-11T09:51:33.072 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00064915600000000002 2024-09-11T09:51:33.072 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.072 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.072 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:33.072 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.932177+0000", 2024-09-11T09:51:33.072 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.9032e-05 2024-09-11T09:51:33.072 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.072 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.072 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:33.073 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.932230+0000", 2024-09-11T09:51:33.073 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.2355000000000003e-05 2024-09-11T09:51:33.073 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:33.073 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:33.073 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:33.073 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.073 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.073 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_repop(client.4688.0:36 2.6 e53/15 2:61e1b442:::rbd_data.1250f5e25b80.0000000000000003:head v 53'2, mlcod=0'0)", 2024-09-11T09:51:33.073 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:22.915597+0000", 2024-09-11T09:51:33.073 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 10.097396824, 2024-09-11T09:51:33.073 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.020430798, 2024-09-11T09:51:33.073 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:33.073 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:33.074 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:33.074 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:33.074 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.074 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:33.074 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.915597+0000", 2024-09-11T09:51:33.074 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:33.074 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.074 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.074 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:33.074 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.915597+0000", 2024-09-11T09:51:33.074 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:33.074 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.074 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.075 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:33.075 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.915598+0000", 2024-09-11T09:51:33.075 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.7150000000000001e-06 2024-09-11T09:51:33.075 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.075 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.075 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:33.075 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921741+0000", 2024-09-11T09:51:33.075 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0061424339999999996 2024-09-11T09:51:33.075 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.075 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.075 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:33.075 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921757+0000", 2024-09-11T09:51:33.075 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.6144e-05 2024-09-11T09:51:33.076 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.076 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.076 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:33.076 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921765+0000", 2024-09-11T09:51:33.076 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.0809999999999994e-06 2024-09-11T09:51:33.076 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.076 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.076 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:33.076 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921826+0000", 2024-09-11T09:51:33.076 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.0878999999999998e-05 2024-09-11T09:51:33.076 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.076 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.076 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:33.077 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921849+0000", 2024-09-11T09:51:33.077 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.2966999999999999e-05 2024-09-11T09:51:33.077 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.077 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.077 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:33.077 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.935968+0000", 2024-09-11T09:51:33.077 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.014119191 2024-09-11T09:51:33.077 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.077 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.077 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:33.077 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.936027+0000", 2024-09-11T09:51:33.077 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.9386999999999999e-05 2024-09-11T09:51:33.077 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:33.077 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:33.078 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:33.078 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.078 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.078 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_repop(client.4688.0:39 2.6 e53/15 2:743e7616:::rbd_data.1250f5e25b80.0000000000000005:head v 53'3, mlcod=0'0)", 2024-09-11T09:51:33.078 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:22.921806+0000", 2024-09-11T09:51:33.078 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 10.091187774, 2024-09-11T09:51:33.078 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.015275018, 2024-09-11T09:51:33.078 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:33.078 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:33.078 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:33.078 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:33.078 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.078 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:33.078 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921806+0000", 2024-09-11T09:51:33.079 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:33.079 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.079 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.079 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:33.079 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921806+0000", 2024-09-11T09:51:33.079 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:33.079 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.079 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.079 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:33.079 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921806+0000", 2024-09-11T09:51:33.079 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.2200000000000003e-07 2024-09-11T09:51:33.079 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.079 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.080 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:33.080 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.925590+0000", 2024-09-11T09:51:33.080 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0037838949999999998 2024-09-11T09:51:33.080 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.080 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.080 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:33.080 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.925605+0000", 2024-09-11T09:51:33.080 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.4321999999999999e-05 2024-09-11T09:51:33.080 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.080 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.080 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:33.080 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.925613+0000", 2024-09-11T09:51:33.080 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.1980000000000001e-06 2024-09-11T09:51:33.081 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.081 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.081 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:33.081 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.925857+0000", 2024-09-11T09:51:33.081 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000244064 2024-09-11T09:51:33.081 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.081 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.081 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:33.081 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.925875+0000", 2024-09-11T09:51:33.081 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.838e-05 2024-09-11T09:51:33.081 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.081 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.081 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:33.082 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.937042+0000", 2024-09-11T09:51:33.082 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.011166416 2024-09-11T09:51:33.082 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.082 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.082 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:33.082 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.937081+0000", 2024-09-11T09:51:33.082 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.9020999999999998e-05 2024-09-11T09:51:33.082 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:33.082 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:33.082 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:33.082 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.082 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.082 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4688.0:49 2.5 2:b6dd9fc9:::rbd_data.1250f5e25b80.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:33.083 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:22.938548+0000", 2024-09-11T09:51:33.083 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 10.074445771000001, 2024-09-11T09:51:33.083 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.022766950000000001, 2024-09-11T09:51:33.083 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:33.083 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:33.083 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:33.083 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:33.083 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4688", 2024-09-11T09:51:33.083 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:51:33.083 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 49 2024-09-11T09:51:33.083 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.083 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:33.083 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.083 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:33.084 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.938548+0000", 2024-09-11T09:51:33.084 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:33.084 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.084 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.084 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:33.084 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.938548+0000", 2024-09-11T09:51:33.084 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:33.084 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.084 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.084 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:33.084 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.938550+0000", 2024-09-11T09:51:33.084 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.7710000000000001e-06 2024-09-11T09:51:33.085 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.085 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.085 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:33.085 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.944237+0000", 2024-09-11T09:51:33.085 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0056879790000000001 2024-09-11T09:51:33.085 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.085 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.085 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:33.085 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.944262+0000", 2024-09-11T09:51:33.085 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.4247e-05 2024-09-11T09:51:33.085 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.085 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.085 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:33.086 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.944274+0000", 2024-09-11T09:51:33.086 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.2639999999999999e-05 2024-09-11T09:51:33.086 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.086 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.086 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:33.086 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.944344+0000", 2024-09-11T09:51:33.086 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.9574999999999997e-05 2024-09-11T09:51:33.086 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.086 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.086 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:33.086 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.944474+0000", 2024-09-11T09:51:33.086 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000130464 2024-09-11T09:51:33.086 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.086 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.087 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:33.087 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.944585+0000", 2024-09-11T09:51:33.087 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000110096 2024-09-11T09:51:33.087 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.087 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.087 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:33.087 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.952724+0000", 2024-09-11T09:51:33.087 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0081398900000000003 2024-09-11T09:51:33.087 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.087 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.087 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:33.087 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.961219+0000", 2024-09-11T09:51:33.088 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0084949789999999997 2024-09-11T09:51:33.088 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.088 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.088 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:33.088 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.961280+0000", 2024-09-11T09:51:33.088 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.0404e-05 2024-09-11T09:51:33.088 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.088 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.088 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:33.088 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.961315+0000", 2024-09-11T09:51:33.088 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.4904999999999999e-05 2024-09-11T09:51:33.088 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:33.088 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:33.089 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:33.089 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.089 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.089 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_repop(client.4688.0:44 2.6 e53/15 2:67246a22:::rbd_data.1250f5e25b80.0000000000000007:head v 53'4, mlcod=53'2)", 2024-09-11T09:51:33.089 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:22.951721+0000", 2024-09-11T09:51:33.089 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 10.061272498999999, 2024-09-11T09:51:33.089 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.01080981, 2024-09-11T09:51:33.089 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:33.089 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:33.089 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:33.089 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:33.089 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.089 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:33.090 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.951721+0000", 2024-09-11T09:51:33.090 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:33.090 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.090 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.090 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:33.090 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.951721+0000", 2024-09-11T09:51:33.090 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:33.090 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.090 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.090 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:33.090 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.951722+0000", 2024-09-11T09:51:33.090 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.0279999999999999e-06 2024-09-11T09:51:33.090 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.091 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.091 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:33.091 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.955612+0000", 2024-09-11T09:51:33.091 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0038896920000000001 2024-09-11T09:51:33.091 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.091 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.091 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:33.091 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.955628+0000", 2024-09-11T09:51:33.091 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.5945000000000001e-05 2024-09-11T09:51:33.091 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.091 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.091 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:33.091 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.955635+0000", 2024-09-11T09:51:33.092 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.8080000000000005e-06 2024-09-11T09:51:33.092 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.092 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.092 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:33.092 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.955700+0000", 2024-09-11T09:51:33.092 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.4114000000000003e-05 2024-09-11T09:51:33.092 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.092 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.092 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:33.092 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.955724+0000", 2024-09-11T09:51:33.092 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.3989999999999999e-05 2024-09-11T09:51:33.092 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.092 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.093 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:33.093 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.962490+0000", 2024-09-11T09:51:33.093 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0067662599999999996 2024-09-11T09:51:33.093 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.093 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.093 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:33.093 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.962531+0000", 2024-09-11T09:51:33.093 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.0973000000000002e-05 2024-09-11T09:51:33.093 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:33.093 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:33.093 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:33.093 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.093 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.094 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_repop(client.4688.0:51 2.6 e53/15 2:6b5894fc:::rbd_data.1250f5e25b80.000000000000000a:head v 53'5, mlcod=53'2)", 2024-09-11T09:51:33.094 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:22.959030+0000", 2024-09-11T09:51:33.094 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 10.053963325, 2024-09-11T09:51:33.094 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.011503638, 2024-09-11T09:51:33.094 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:33.094 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:33.094 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:33.094 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:33.094 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.094 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:33.094 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.959030+0000", 2024-09-11T09:51:33.094 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:33.094 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.095 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.095 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:33.095 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.959030+0000", 2024-09-11T09:51:33.095 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:33.095 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.095 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.095 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:33.095 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.959032+0000", 2024-09-11T09:51:33.095 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.356e-06 2024-09-11T09:51:33.095 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.095 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.095 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:33.095 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.962899+0000", 2024-09-11T09:51:33.096 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.003867639 2024-09-11T09:51:33.096 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.096 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.096 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:33.096 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.962909+0000", 2024-09-11T09:51:33.096 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.0105999999999999e-05 2024-09-11T09:51:33.096 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.096 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.096 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:33.096 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.962915+0000", 2024-09-11T09:51:33.096 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.8869999999999997e-06 2024-09-11T09:51:33.096 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.097 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.097 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:33.097 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.962962+0000", 2024-09-11T09:51:33.097 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.6511999999999998e-05 2024-09-11T09:51:33.097 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.097 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.097 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:33.097 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.962989+0000", 2024-09-11T09:51:33.097 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.7702999999999999e-05 2024-09-11T09:51:33.097 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.097 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.097 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:33.097 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.970495+0000", 2024-09-11T09:51:33.097 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0075052449999999998 2024-09-11T09:51:33.098 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.098 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.098 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:33.098 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.970534+0000", 2024-09-11T09:51:33.098 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.9190000000000003e-05 2024-09-11T09:51:33.098 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:33.098 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:33.098 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:33.098 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.098 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.098 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_repop(client.4688.0:58 2.6 e53/15 2:6277c644:::rbd_data.1250f5e25b80.000000000000000e:head v 53'6, mlcod=53'4)", 2024-09-11T09:51:33.098 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:22.996199+0000", 2024-09-11T09:51:33.098 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 10.016794405000001, 2024-09-11T09:51:33.099 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.010145573999999999, 2024-09-11T09:51:33.099 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:33.099 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:33.099 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:33.099 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:33.099 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.099 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:33.099 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.996199+0000", 2024-09-11T09:51:33.099 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:33.099 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.099 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.099 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:33.099 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.996199+0000", 2024-09-11T09:51:33.099 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:33.100 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.100 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.100 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:33.100 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.996200+0000", 2024-09-11T09:51:33.100 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.85e-07 2024-09-11T09:51:33.100 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.100 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.100 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:33.100 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.999914+0000", 2024-09-11T09:51:33.100 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.003714406 2024-09-11T09:51:33.100 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.100 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.100 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:33.101 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.999928+0000", 2024-09-11T09:51:33.101 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.3149000000000001e-05 2024-09-11T09:51:33.101 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.101 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.101 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:33.101 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.999951+0000", 2024-09-11T09:51:33.101 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.3116e-05 2024-09-11T09:51:33.101 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.101 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.101 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:33.101 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.000003+0000", 2024-09-11T09:51:33.101 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.2169000000000001e-05 2024-09-11T09:51:33.101 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.102 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.102 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:33.102 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.000018+0000", 2024-09-11T09:51:33.102 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.5500000000000001e-05 2024-09-11T09:51:33.102 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.102 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.102 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:33.102 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.006303+0000", 2024-09-11T09:51:33.102 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0062846710000000004 2024-09-11T09:51:33.102 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.102 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.102 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:33.102 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.006345+0000", 2024-09-11T09:51:33.102 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.1678000000000003e-05 2024-09-11T09:51:33.103 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:33.103 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:33.103 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:33.103 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.103 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.103 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4694.0:42 2.5 2:a4fc00ec:ns::rbd_data.1256a0c02879.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:33.103 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.111570+0000", 2024-09-11T09:51:33.103 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 9.9014232989999993, 2024-09-11T09:51:33.103 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.038002197000000001, 2024-09-11T09:51:33.103 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:33.103 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:33.103 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:33.103 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:33.103 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4694", 2024-09-11T09:51:33.104 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:51:33.104 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 42 2024-09-11T09:51:33.104 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.104 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:33.104 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.104 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:33.104 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.111570+0000", 2024-09-11T09:51:33.104 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:33.104 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.104 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.104 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:33.104 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.111570+0000", 2024-09-11T09:51:33.104 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:33.105 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.105 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.105 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:33.105 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.111572+0000", 2024-09-11T09:51:33.105 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.3909999999999999e-06 2024-09-11T09:51:33.105 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.105 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.105 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:33.105 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.135929+0000", 2024-09-11T09:51:33.105 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.024357410999999999 2024-09-11T09:51:33.105 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.105 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.105 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:33.106 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.135951+0000", 2024-09-11T09:51:33.106 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.1775000000000001e-05 2024-09-11T09:51:33.106 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.106 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.106 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:33.106 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.135962+0000", 2024-09-11T09:51:33.106 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.0903e-05 2024-09-11T09:51:33.106 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.106 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.106 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:33.106 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.136045+0000", 2024-09-11T09:51:33.106 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.3128999999999996e-05 2024-09-11T09:51:33.106 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.107 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.107 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:33.107 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.136229+0000", 2024-09-11T09:51:33.107 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00018393700000000001 2024-09-11T09:51:33.107 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.107 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.107 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:33.107 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.136394+0000", 2024-09-11T09:51:33.107 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000165527 2024-09-11T09:51:33.107 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.107 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.107 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:33.107 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.149320+0000", 2024-09-11T09:51:33.107 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.01292568 2024-09-11T09:51:33.108 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.108 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.108 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:33.108 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.149493+0000", 2024-09-11T09:51:33.108 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00017285999999999999 2024-09-11T09:51:33.108 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.108 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.108 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:33.108 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.149539+0000", 2024-09-11T09:51:33.108 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.6628000000000001e-05 2024-09-11T09:51:33.108 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.108 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.108 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:33.109 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.149572+0000", 2024-09-11T09:51:33.109 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.2956000000000001e-05 2024-09-11T09:51:33.109 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:33.109 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:33.109 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:33.109 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.109 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.109 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_repop(client.4694.0:45 2.6 e53/15 2:671fb830:ns::rbd_data.1256a0c02879.0000000000000007:head v 53'9, mlcod=53'7)", 2024-09-11T09:51:33.109 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.121833+0000", 2024-09-11T09:51:33.109 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 9.8911609410000008, 2024-09-11T09:51:33.109 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.023876192000000001, 2024-09-11T09:51:33.109 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:33.109 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:33.110 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:33.110 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:33.110 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.110 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:33.110 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.121833+0000", 2024-09-11T09:51:33.110 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:33.110 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.110 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.110 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:33.110 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.121833+0000", 2024-09-11T09:51:33.110 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:33.110 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.110 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.110 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:33.111 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.121834+0000", 2024-09-11T09:51:33.111 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.471e-06 2024-09-11T09:51:33.111 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.111 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.111 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:33.111 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.128990+0000", 2024-09-11T09:51:33.111 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0071557879999999997 2024-09-11T09:51:33.111 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.111 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.111 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:33.111 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.129011+0000", 2024-09-11T09:51:33.111 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.0760000000000001e-05 2024-09-11T09:51:33.111 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.112 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.112 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:33.112 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.129022+0000", 2024-09-11T09:51:33.112 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1056000000000001e-05 2024-09-11T09:51:33.112 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.112 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.112 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:33.112 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.129097+0000", 2024-09-11T09:51:33.112 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.5082999999999999e-05 2024-09-11T09:51:33.112 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.112 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.112 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:33.112 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.129130+0000", 2024-09-11T09:51:33.113 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.3345e-05 2024-09-11T09:51:33.113 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.113 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.113 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:33.113 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.145657+0000", 2024-09-11T09:51:33.113 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.016527224 2024-09-11T09:51:33.113 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.113 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.113 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:33.113 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.145709+0000", 2024-09-11T09:51:33.113 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.1465000000000003e-05 2024-09-11T09:51:33.113 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:33.113 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:33.114 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:33.114 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.114 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.114 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4694.0:43 2.0 2:16c8ae84:ns::rbd_data.1256a0c02879.0000000000000005:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:33.114 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.135999+0000", 2024-09-11T09:51:33.114 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 9.8769949910000001, 2024-09-11T09:51:33.114 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.039468309, 2024-09-11T09:51:33.114 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:33.114 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:33.114 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:33.114 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:33.114 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4694", 2024-09-11T09:51:33.114 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:51:33.115 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 43 2024-09-11T09:51:33.115 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.115 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:33.115 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.115 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:33.115 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.135999+0000", 2024-09-11T09:51:33.115 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:33.115 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.115 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.115 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:33.115 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.135999+0000", 2024-09-11T09:51:33.115 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:33.115 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.115 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.116 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:33.116 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.136000+0000", 2024-09-11T09:51:33.116 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.6330000000000001e-06 2024-09-11T09:51:33.116 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.116 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.116 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:33.116 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.157961+0000", 2024-09-11T09:51:33.116 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0219606 2024-09-11T09:51:33.116 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.116 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.116 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:33.116 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.157995+0000", 2024-09-11T09:51:33.116 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.4391999999999999e-05 2024-09-11T09:51:33.117 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.117 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.117 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:33.117 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.158011+0000", 2024-09-11T09:51:33.117 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.5794000000000002e-05 2024-09-11T09:51:33.117 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.117 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.117 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:33.117 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.158083+0000", 2024-09-11T09:51:33.117 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.2273000000000004e-05 2024-09-11T09:51:33.117 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.117 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.117 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:33.118 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.158239+0000", 2024-09-11T09:51:33.118 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000155372 2024-09-11T09:51:33.118 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.118 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.118 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 1", 2024-09-11T09:51:33.118 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.158403+0000", 2024-09-11T09:51:33.118 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00016408199999999999 2024-09-11T09:51:33.118 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.118 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.118 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:33.118 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.173860+0000", 2024-09-11T09:51:33.118 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.015457347 2024-09-11T09:51:33.118 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.119 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.119 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:33.119 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.175400+0000", 2024-09-11T09:51:33.119 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001540006 2024-09-11T09:51:33.119 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.119 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.119 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:33.119 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.175438+0000", 2024-09-11T09:51:33.119 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.8135e-05 2024-09-11T09:51:33.119 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.119 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.119 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:33.119 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.175467+0000", 2024-09-11T09:51:33.119 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.8674999999999999e-05 2024-09-11T09:51:33.120 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:33.120 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:33.120 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:33.120 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.120 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.120 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4694.0:57 2.0 2:04b5549c:ns::rbd_data.1256a0c02879.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:33.120 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.158037+0000", 2024-09-11T09:51:33.120 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 9.8549562529999992, 2024-09-11T09:51:33.120 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.027748790999999998, 2024-09-11T09:51:33.120 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:33.120 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:33.120 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:33.120 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:33.121 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4694", 2024-09-11T09:51:33.121 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:51:33.121 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 57 2024-09-11T09:51:33.121 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.121 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:33.121 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.121 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:33.121 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.158037+0000", 2024-09-11T09:51:33.121 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:33.121 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.121 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.121 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:33.121 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.158037+0000", 2024-09-11T09:51:33.122 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:33.122 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.122 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.122 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:33.122 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.158039+0000", 2024-09-11T09:51:33.122 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.356e-06 2024-09-11T09:51:33.122 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.122 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.122 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:33.122 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.176367+0000", 2024-09-11T09:51:33.122 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.018328739 2024-09-11T09:51:33.122 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.122 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.122 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:33.123 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.176378+0000", 2024-09-11T09:51:33.123 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.0713999999999999e-05 2024-09-11T09:51:33.123 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.123 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.123 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:33.123 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.176383+0000", 2024-09-11T09:51:33.123 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.0209999999999999e-06 2024-09-11T09:51:33.123 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.123 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.123 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:33.123 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.176426+0000", 2024-09-11T09:51:33.123 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.2623999999999997e-05 2024-09-11T09:51:33.123 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.124 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.124 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:33.124 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.176552+0000", 2024-09-11T09:51:33.124 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000126076 2024-09-11T09:51:33.124 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.124 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.124 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 1", 2024-09-11T09:51:33.124 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.176665+0000", 2024-09-11T09:51:33.124 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000112995 2024-09-11T09:51:33.124 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.124 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.124 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:33.124 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.183639+0000", 2024-09-11T09:51:33.125 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0069740039999999998 2024-09-11T09:51:33.125 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.125 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.125 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:33.125 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.185689+0000", 2024-09-11T09:51:33.125 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.002049852 2024-09-11T09:51:33.125 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.125 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.125 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:33.125 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.185725+0000", 2024-09-11T09:51:33.125 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.6804999999999997e-05 2024-09-11T09:51:33.125 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.125 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.126 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:33.126 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.185786+0000", 2024-09-11T09:51:33.126 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.0605e-05 2024-09-11T09:51:33.126 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:33.126 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:33.126 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:33.126 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.126 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.126 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4694.0:60 2.0 2:054f2af4:ns::rbd_data.1256a0c02879.000000000000000f:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:33.126 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.186855+0000", 2024-09-11T09:51:33.126 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 9.8261382370000003, 2024-09-11T09:51:33.126 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.013682913, 2024-09-11T09:51:33.126 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:33.127 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:33.127 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:33.127 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:33.127 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4694", 2024-09-11T09:51:33.127 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:51:33.127 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 60 2024-09-11T09:51:33.127 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.127 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:33.127 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.127 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:33.127 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.186855+0000", 2024-09-11T09:51:33.127 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:33.127 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.127 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.128 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:33.128 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.186855+0000", 2024-09-11T09:51:33.128 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:33.128 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.128 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.128 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:33.128 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.186856+0000", 2024-09-11T09:51:33.128 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.1200000000000001e-07 2024-09-11T09:51:33.128 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.128 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.128 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:33.128 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.191484+0000", 2024-09-11T09:51:33.128 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0046274159999999996 2024-09-11T09:51:33.129 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.129 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.129 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:33.129 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.191496+0000", 2024-09-11T09:51:33.129 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.2149e-05 2024-09-11T09:51:33.129 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.129 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.129 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:33.129 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.191502+0000", 2024-09-11T09:51:33.129 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.1890000000000004e-06 2024-09-11T09:51:33.129 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.129 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.130 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:33.130 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.191544+0000", 2024-09-11T09:51:33.130 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.1724999999999998e-05 2024-09-11T09:51:33.130 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.130 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.130 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:33.130 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.191641+0000", 2024-09-11T09:51:33.130 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.6939000000000001e-05 2024-09-11T09:51:33.130 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.130 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.130 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 1", 2024-09-11T09:51:33.130 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.191731+0000", 2024-09-11T09:51:33.130 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.0651999999999999e-05 2024-09-11T09:51:33.130 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.131 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.131 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:33.131 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.199852+0000", 2024-09-11T09:51:33.131 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0081204529999999997 2024-09-11T09:51:33.131 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.131 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.131 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:33.131 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.200461+0000", 2024-09-11T09:51:33.131 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00060900899999999998 2024-09-11T09:51:33.131 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.131 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.131 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:33.131 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.200497+0000", 2024-09-11T09:51:33.132 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.5800000000000003e-05 2024-09-11T09:51:33.132 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.132 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.132 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:33.132 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.200538+0000", 2024-09-11T09:51:33.132 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.1668999999999997e-05 2024-09-11T09:51:33.132 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:33.132 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:33.132 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:33.132 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:33.132 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:33.132 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-11T09:51:33.134 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-11T09:51:33.134 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:51:33.135+0000 7f3c96ec9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31909) UID: 0 2024-09-11T09:51:33.235 INFO:tasks.ceph.osd.2.smithi017.stderr:2024-09-11T09:51:33.236+0000 7fd20b170640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31672) UID: 0 2024-09-11T09:51:33.277 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-11T09:51:33.277 INFO:teuthology.orchestra.run.smithi096.stdout: "ops": [], 2024-09-11T09:51:33.277 INFO:teuthology.orchestra.run.smithi096.stdout: "num_ops": 0 2024-09-11T09:51:33.278 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-11T09:51:33.286 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-11T09:51:33.291 DEBUG:teuthology.orchestra.run.smithi096:> sudo rm -f /var/log/ceph/exp.3.3.4 2024-09-11T09:51:33.335 INFO:tasks.ceph.osd.2.smithi017.stderr:2024-09-11T09:51:33.336+0000 7fd20b170640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31672) UID: 0 2024-09-11T09:51:33.390 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-11T09:51:33.390 INFO:teuthology.orchestra.run.smithi096.stdout: "ops": [], 2024-09-11T09:51:33.390 INFO:teuthology.orchestra.run.smithi096.stdout: "complaint_time": 30, 2024-09-11T09:51:33.390 INFO:teuthology.orchestra.run.smithi096.stdout: "num_blocked_ops": 0 2024-09-11T09:51:33.390 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-11T09:51:33.399 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-11T09:51:33.437 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:51:33.437+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:51:33.537 INFO:tasks.ceph.osd.2.smithi017.stderr:2024-09-11T09:51:33.538+0000 7fd20b170640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31672) UID: 0 2024-09-11T09:51:33.544 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-11T09:51:33.544 INFO:teuthology.orchestra.run.smithi096.stdout: "size": 20, 2024-09-11T09:51:33.544 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 600, 2024-09-11T09:51:33.544 INFO:teuthology.orchestra.run.smithi096.stdout: "ops": [ 2024-09-11T09:51:33.544 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.544 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4230.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-11T09:51:33.544 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:18.490736+0000", 2024-09-11T09:51:33.544 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 75.053234794999995, 2024-09-11T09:51:33.544 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0059097569999999999, 2024-09-11T09:51:33.545 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:33.545 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:33.545 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:33.545 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:33.545 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4230", 2024-09-11T09:51:33.545 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.17:0/4101109576", 2024-09-11T09:51:33.545 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 1 2024-09-11T09:51:33.545 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.545 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:33.545 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.545 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:33.545 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.490736+0000", 2024-09-11T09:51:33.545 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:33.545 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.546 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.546 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:33.546 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.490736+0000", 2024-09-11T09:51:33.546 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:33.546 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.546 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.546 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:33.546 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.490742+0000", 2024-09-11T09:51:33.546 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.48e-06 2024-09-11T09:51:33.546 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.546 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.546 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:33.546 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.490749+0000", 2024-09-11T09:51:33.547 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.4569999999999999e-06 2024-09-11T09:51:33.547 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.547 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.547 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:33.547 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.490762+0000", 2024-09-11T09:51:33.547 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.2721000000000001e-05 2024-09-11T09:51:33.547 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.547 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.547 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:33.547 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.490774+0000", 2024-09-11T09:51:33.547 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1823e-05 2024-09-11T09:51:33.547 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.547 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.547 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:33.548 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.490846+0000", 2024-09-11T09:51:33.548 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.1878000000000005e-05 2024-09-11T09:51:33.548 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.548 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.548 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for peered", 2024-09-11T09:51:33.548 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.490857+0000", 2024-09-11T09:51:33.548 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1017e-05 2024-09-11T09:51:33.548 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.548 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.548 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:33.548 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.493856+0000", 2024-09-11T09:51:33.548 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00299885 2024-09-11T09:51:33.548 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.549 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.549 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:33.549 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.494020+0000", 2024-09-11T09:51:33.549 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00016436599999999999 2024-09-11T09:51:33.549 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.549 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.549 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 2", 2024-09-11T09:51:33.549 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.494186+0000", 2024-09-11T09:51:33.549 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00016562799999999999 2024-09-11T09:51:33.549 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.549 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.549 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:33.549 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.495407+0000", 2024-09-11T09:51:33.550 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0012217420000000001 2024-09-11T09:51:33.550 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.550 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.550 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:33.550 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.496579+0000", 2024-09-11T09:51:33.550 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001171844 2024-09-11T09:51:33.550 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.550 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.550 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:33.550 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.496620+0000", 2024-09-11T09:51:33.550 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.0479000000000003e-05 2024-09-11T09:51:33.550 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.550 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.551 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:33.551 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.496646+0000", 2024-09-11T09:51:33.551 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.6472e-05 2024-09-11T09:51:33.551 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:33.551 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:33.551 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:33.551 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.551 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.551 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4230.0:5 2.2 2:5cea7035:::rbd_info:head [create,write 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-11T09:51:33.551 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:19.492662+0000", 2024-09-11T09:51:33.551 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 74.051309173999996, 2024-09-11T09:51:33.551 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0043824759999999997, 2024-09-11T09:51:33.551 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:33.551 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:33.552 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:33.552 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:33.552 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4230", 2024-09-11T09:51:33.552 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.17:0/4101109576", 2024-09-11T09:51:33.552 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 5 2024-09-11T09:51:33.552 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.552 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:33.552 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.552 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:33.552 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.492662+0000", 2024-09-11T09:51:33.552 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:33.552 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.552 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.553 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:33.553 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.492662+0000", 2024-09-11T09:51:33.553 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:33.553 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.553 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.553 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:33.553 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.492664+0000", 2024-09-11T09:51:33.553 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.3599999999999999e-06 2024-09-11T09:51:33.553 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.553 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.553 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:33.553 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.492675+0000", 2024-09-11T09:51:33.553 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1182000000000001e-05 2024-09-11T09:51:33.554 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.554 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.554 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:33.554 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.492693+0000", 2024-09-11T09:51:33.554 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.7031999999999999e-05 2024-09-11T09:51:33.554 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.554 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.554 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:33.554 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.492702+0000", 2024-09-11T09:51:33.554 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.5540000000000003e-06 2024-09-11T09:51:33.554 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.554 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.554 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:33.554 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.492767+0000", 2024-09-11T09:51:33.555 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.5411000000000001e-05 2024-09-11T09:51:33.555 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.555 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.555 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op must wait for map", 2024-09-11T09:51:33.555 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.492773+0000", 2024-09-11T09:51:33.555 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.1039999999999998e-06 2024-09-11T09:51:33.555 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.555 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.555 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:33.555 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.494354+0000", 2024-09-11T09:51:33.555 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0015814049999999999 2024-09-11T09:51:33.555 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.555 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.555 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:33.556 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.494476+0000", 2024-09-11T09:51:33.556 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000122089 2024-09-11T09:51:33.556 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.556 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.556 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 1", 2024-09-11T09:51:33.556 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.494577+0000", 2024-09-11T09:51:33.556 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000100536 2024-09-11T09:51:33.556 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.556 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.556 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:33.556 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.495770+0000", 2024-09-11T09:51:33.556 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0011932469999999999 2024-09-11T09:51:33.556 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.557 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.557 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:33.557 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.496966+0000", 2024-09-11T09:51:33.557 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001196003 2024-09-11T09:51:33.557 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.557 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.557 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:33.557 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.497011+0000", 2024-09-11T09:51:33.557 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.5229999999999999e-05 2024-09-11T09:51:33.557 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.557 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.557 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:33.557 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.497044+0000", 2024-09-11T09:51:33.558 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.3322999999999999e-05 2024-09-11T09:51:33.558 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:33.558 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:33.558 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:33.558 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.558 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.558 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4230.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-09-11T09:51:33.558 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:20.498634+0000", 2024-09-11T09:51:33.558 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 73.045336774000006, 2024-09-11T09:51:33.559 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0050648539999999997, 2024-09-11T09:51:33.559 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:33.559 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:33.559 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:33.559 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:33.559 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4230", 2024-09-11T09:51:33.559 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.17:0/4101109576", 2024-09-11T09:51:33.559 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 7 2024-09-11T09:51:33.559 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.559 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:33.559 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.559 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:33.559 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.498634+0000", 2024-09-11T09:51:33.560 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:33.560 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.560 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.560 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:33.560 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.498634+0000", 2024-09-11T09:51:33.560 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:33.560 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.560 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.560 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:33.560 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.498637+0000", 2024-09-11T09:51:33.560 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.1919999999999999e-06 2024-09-11T09:51:33.560 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.560 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.560 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:33.561 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.498648+0000", 2024-09-11T09:51:33.561 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1116e-05 2024-09-11T09:51:33.561 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.561 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.561 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:33.561 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.498666+0000", 2024-09-11T09:51:33.561 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.8411999999999999e-05 2024-09-11T09:51:33.561 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.561 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.561 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:33.561 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.498675+0000", 2024-09-11T09:51:33.561 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.4059999999999994e-06 2024-09-11T09:51:33.561 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.562 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.562 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:33.562 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.498745+0000", 2024-09-11T09:51:33.562 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.9764000000000005e-05 2024-09-11T09:51:33.562 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.562 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.562 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op must wait for map", 2024-09-11T09:51:33.562 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.498754+0000", 2024-09-11T09:51:33.562 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.5969999999999992e-06 2024-09-11T09:51:33.562 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.562 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.562 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:33.562 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.500385+0000", 2024-09-11T09:51:33.562 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0016308360000000001 2024-09-11T09:51:33.563 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.563 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.563 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:33.563 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.500473+0000", 2024-09-11T09:51:33.563 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.7966000000000004e-05 2024-09-11T09:51:33.563 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.563 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.563 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 1", 2024-09-11T09:51:33.563 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.500591+0000", 2024-09-11T09:51:33.563 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000117895 2024-09-11T09:51:33.563 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.563 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.563 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:33.563 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.501685+0000", 2024-09-11T09:51:33.564 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001094886 2024-09-11T09:51:33.564 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.564 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.564 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:33.564 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.503627+0000", 2024-09-11T09:51:33.564 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001941199 2024-09-11T09:51:33.564 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.564 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.564 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:33.564 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.503667+0000", 2024-09-11T09:51:33.564 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.0645000000000001e-05 2024-09-11T09:51:33.564 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.564 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.565 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:33.565 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.503699+0000", 2024-09-11T09:51:33.565 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.1940000000000003e-05 2024-09-11T09:51:33.565 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:33.565 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:33.565 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:33.565 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.565 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.565 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4688.0:32 2.2 2:45941fd2:::rbd_data.1250f5e25b80.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:33.565 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:22.896607+0000", 2024-09-11T09:51:33.566 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 10.647364171, 2024-09-11T09:51:33.566 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.038655179999999997, 2024-09-11T09:51:33.566 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:33.566 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:33.566 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:33.566 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:33.566 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4688", 2024-09-11T09:51:33.566 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:51:33.566 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 32 2024-09-11T09:51:33.566 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.566 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:33.566 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.566 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:33.566 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.896607+0000", 2024-09-11T09:51:33.567 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:33.567 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.567 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.567 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:33.567 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.896607+0000", 2024-09-11T09:51:33.567 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:33.567 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.567 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.567 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:33.567 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.896608+0000", 2024-09-11T09:51:33.567 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.0690000000000001e-06 2024-09-11T09:51:33.567 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.567 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.567 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:33.568 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.914461+0000", 2024-09-11T09:51:33.568 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.017852903 2024-09-11T09:51:33.568 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.568 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.568 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:33.568 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.914495+0000", 2024-09-11T09:51:33.568 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.4301000000000002e-05 2024-09-11T09:51:33.568 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.568 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.568 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:33.568 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.914522+0000", 2024-09-11T09:51:33.568 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.7127e-05 2024-09-11T09:51:33.568 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.569 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.569 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:33.569 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.914583+0000", 2024-09-11T09:51:33.569 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.0730999999999998e-05 2024-09-11T09:51:33.569 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.569 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.569 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:33.569 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.914709+0000", 2024-09-11T09:51:33.569 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00012590400000000001 2024-09-11T09:51:33.569 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.569 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.569 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 1", 2024-09-11T09:51:33.569 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.914809+0000", 2024-09-11T09:51:33.570 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.9882000000000004e-05 2024-09-11T09:51:33.570 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.570 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.570 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:33.570 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.932551+0000", 2024-09-11T09:51:33.570 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.017742048999999999 2024-09-11T09:51:33.570 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.570 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.570 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:33.570 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.935144+0000", 2024-09-11T09:51:33.570 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0025927630000000001 2024-09-11T09:51:33.570 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.570 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.570 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:33.571 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.935205+0000", 2024-09-11T09:51:33.571 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.1634e-05 2024-09-11T09:51:33.571 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.571 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.571 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:33.571 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.935262+0000", 2024-09-11T09:51:33.571 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.6817000000000002e-05 2024-09-11T09:51:33.571 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:33.571 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:33.571 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:33.571 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.571 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.571 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_repop(client.4688.0:40 2.7 e53/38 2:fe41411c:::rbd_object_map.1250f5e25b80:head v 53'9, mlcod=53'7)", 2024-09-11T09:51:33.571 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:22.920457+0000", 2024-09-11T09:51:33.572 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 10.62351408, 2024-09-11T09:51:33.572 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.011132817, 2024-09-11T09:51:33.572 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:33.572 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:33.572 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:33.572 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:33.572 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.572 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:33.572 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.920457+0000", 2024-09-11T09:51:33.572 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:33.572 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.572 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.572 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:33.572 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.920457+0000", 2024-09-11T09:51:33.573 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:33.573 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.573 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.573 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:33.573 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.920459+0000", 2024-09-11T09:51:33.573 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.6729999999999999e-06 2024-09-11T09:51:33.573 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.573 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.573 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:33.573 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.920469+0000", 2024-09-11T09:51:33.573 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.944e-06 2024-09-11T09:51:33.573 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.573 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.574 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:33.574 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.920495+0000", 2024-09-11T09:51:33.574 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.6401999999999999e-05 2024-09-11T09:51:33.574 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.574 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.574 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:33.574 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.920506+0000", 2024-09-11T09:51:33.574 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.133e-05 2024-09-11T09:51:33.574 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.574 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.574 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:33.574 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.920581+0000", 2024-09-11T09:51:33.574 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.4932999999999996e-05 2024-09-11T09:51:33.574 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.575 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.575 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:33.575 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.920612+0000", 2024-09-11T09:51:33.575 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.0537999999999998e-05 2024-09-11T09:51:33.575 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.575 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.575 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:33.575 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.931537+0000", 2024-09-11T09:51:33.575 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.010925239 2024-09-11T09:51:33.575 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.575 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.575 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:33.575 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.931590+0000", 2024-09-11T09:51:33.576 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.2757999999999999e-05 2024-09-11T09:51:33.576 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:33.576 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:33.576 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:33.576 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.576 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.576 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_repop(client.4688.0:41 2.7 e53/38 2:fe41411c:::rbd_object_map.1250f5e25b80:head v 53'10, mlcod=53'7)", 2024-09-11T09:51:33.576 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:22.921422+0000", 2024-09-11T09:51:33.576 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 10.622548664, 2024-09-11T09:51:33.576 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.013734942, 2024-09-11T09:51:33.576 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:33.576 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:33.576 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:33.577 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:33.577 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.577 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:33.577 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921422+0000", 2024-09-11T09:51:33.577 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:33.577 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.577 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.577 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:33.577 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921422+0000", 2024-09-11T09:51:33.577 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:33.577 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.577 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.577 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:33.577 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921426+0000", 2024-09-11T09:51:33.578 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.6150000000000001e-06 2024-09-11T09:51:33.578 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.578 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.578 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:33.578 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921435+0000", 2024-09-11T09:51:33.578 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.9840000000000007e-06 2024-09-11T09:51:33.578 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.578 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.578 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:33.578 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921444+0000", 2024-09-11T09:51:33.578 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.6000000000000007e-06 2024-09-11T09:51:33.578 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.578 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.578 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:33.579 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921450+0000", 2024-09-11T09:51:33.579 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.4690000000000002e-06 2024-09-11T09:51:33.579 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.579 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.579 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:33.579 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921511+0000", 2024-09-11T09:51:33.579 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.0779999999999997e-05 2024-09-11T09:51:33.579 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.579 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.579 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:33.579 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921531+0000", 2024-09-11T09:51:33.579 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.0302999999999999e-05 2024-09-11T09:51:33.579 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.579 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.580 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:33.580 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.935115+0000", 2024-09-11T09:51:33.580 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.013583471 2024-09-11T09:51:33.580 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.580 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.580 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:33.580 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.935157+0000", 2024-09-11T09:51:33.580 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.2719999999999998e-05 2024-09-11T09:51:33.580 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:33.580 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:33.580 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:33.580 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.581 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.581 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4688.0:42 2.3 2:c54742d8:::rbd_data.1250f5e25b80.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:33.581 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:22.932560+0000", 2024-09-11T09:51:33.581 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 10.611410921000001, 2024-09-11T09:51:33.581 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.017798198000000001, 2024-09-11T09:51:33.581 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:33.581 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:33.581 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:33.581 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:33.581 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4688", 2024-09-11T09:51:33.582 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:51:33.582 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 42 2024-09-11T09:51:33.582 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.582 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:33.582 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.582 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:33.582 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.932560+0000", 2024-09-11T09:51:33.582 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:33.582 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.582 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.582 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:33.582 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.932560+0000", 2024-09-11T09:51:33.583 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:33.583 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.583 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.583 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:33.583 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.932562+0000", 2024-09-11T09:51:33.583 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.17e-06 2024-09-11T09:51:33.583 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.583 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.583 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:33.583 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.938851+0000", 2024-09-11T09:51:33.584 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.006288971 2024-09-11T09:51:33.584 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.584 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.584 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:33.584 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.938865+0000", 2024-09-11T09:51:33.584 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.365e-05 2024-09-11T09:51:33.584 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.584 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.584 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:33.584 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.938871+0000", 2024-09-11T09:51:33.584 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.8850000000000004e-06 2024-09-11T09:51:33.584 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.584 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.585 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:33.585 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.938926+0000", 2024-09-11T09:51:33.585 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.4691999999999997e-05 2024-09-11T09:51:33.585 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.585 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.585 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:33.585 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.939059+0000", 2024-09-11T09:51:33.585 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00013310899999999999 2024-09-11T09:51:33.585 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.585 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.585 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 2", 2024-09-11T09:51:33.585 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.939171+0000", 2024-09-11T09:51:33.585 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000112718 2024-09-11T09:51:33.586 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.586 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.586 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:33.586 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.950262+0000", 2024-09-11T09:51:33.586 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.011091063 2024-09-11T09:51:33.586 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.586 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.586 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:33.586 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.950274+0000", 2024-09-11T09:51:33.586 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1963e-05 2024-09-11T09:51:33.586 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.586 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.586 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:33.586 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.950318+0000", 2024-09-11T09:51:33.587 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.3519999999999997e-05 2024-09-11T09:51:33.587 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.587 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.587 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:33.587 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.950358+0000", 2024-09-11T09:51:33.587 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.0457000000000002e-05 2024-09-11T09:51:33.587 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:33.587 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:33.587 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:33.587 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.587 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.587 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_repop(client.4694.0:34 2.7 e53/38 2:f80bfc0c:ns::rbd_data.1256a0c02879.0000000000000002:head v 53'25, mlcod=53'23)", 2024-09-11T09:51:33.588 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.094944+0000", 2024-09-11T09:51:33.588 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 10.449027529, 2024-09-11T09:51:33.588 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.010896785000000001, 2024-09-11T09:51:33.588 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:33.588 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:33.588 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:33.588 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:33.588 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.588 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:33.588 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.094944+0000", 2024-09-11T09:51:33.588 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:33.588 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.589 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.589 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:33.589 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.094944+0000", 2024-09-11T09:51:33.589 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:33.589 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.589 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.589 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:33.589 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.094944+0000", 2024-09-11T09:51:33.589 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.0699999999999996e-07 2024-09-11T09:51:33.589 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.589 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.589 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:33.589 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.099060+0000", 2024-09-11T09:51:33.590 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0041155580000000001 2024-09-11T09:51:33.590 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.590 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.590 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:33.590 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.099082+0000", 2024-09-11T09:51:33.590 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.2323999999999998e-05 2024-09-11T09:51:33.590 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.590 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.590 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:33.590 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.099094+0000", 2024-09-11T09:51:33.590 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1535000000000001e-05 2024-09-11T09:51:33.590 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.590 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.591 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:33.591 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.099160+0000", 2024-09-11T09:51:33.591 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.6449e-05 2024-09-11T09:51:33.591 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.591 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.591 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:33.591 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.099183+0000", 2024-09-11T09:51:33.591 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.3108e-05 2024-09-11T09:51:33.591 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.591 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.591 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:33.591 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.105776+0000", 2024-09-11T09:51:33.591 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0065922760000000002 2024-09-11T09:51:33.592 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.592 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.592 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:33.592 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.105840+0000", 2024-09-11T09:51:33.592 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.4727999999999994e-05 2024-09-11T09:51:33.592 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:33.592 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:33.592 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:33.592 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.592 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.592 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_repop(client.4694.0:35 2.7 e53/38 2:e8766325:ns::rbd_object_map.1256a0c02879:head v 53'26, mlcod=53'23)", 2024-09-11T09:51:33.592 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.100964+0000", 2024-09-11T09:51:33.593 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 10.443007001, 2024-09-11T09:51:33.593 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0049649359999999997, 2024-09-11T09:51:33.593 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:33.593 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:33.593 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:33.593 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:33.593 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.593 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:33.593 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.100964+0000", 2024-09-11T09:51:33.593 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:33.593 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.593 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.593 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:33.594 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.100964+0000", 2024-09-11T09:51:33.594 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:33.594 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.594 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.594 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:33.594 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.100966+0000", 2024-09-11T09:51:33.594 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.6330000000000001e-06 2024-09-11T09:51:33.594 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.594 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.594 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:33.594 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.100979+0000", 2024-09-11T09:51:33.594 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.2813000000000001e-05 2024-09-11T09:51:33.594 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.594 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.595 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:33.595 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.100992+0000", 2024-09-11T09:51:33.595 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.3414999999999999e-05 2024-09-11T09:51:33.595 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.595 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.595 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:33.595 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.101002+0000", 2024-09-11T09:51:33.595 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.0159e-05 2024-09-11T09:51:33.595 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.595 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.595 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:33.595 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.102365+0000", 2024-09-11T09:51:33.596 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001362912 2024-09-11T09:51:33.596 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.596 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.596 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:33.596 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.102382+0000", 2024-09-11T09:51:33.596 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.7433000000000001e-05 2024-09-11T09:51:33.596 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.596 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.596 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:33.596 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.105901+0000", 2024-09-11T09:51:33.596 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.003518476 2024-09-11T09:51:33.596 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.596 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.597 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:33.597 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.105929+0000", 2024-09-11T09:51:33.597 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.8095000000000001e-05 2024-09-11T09:51:33.597 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:33.597 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:33.597 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:33.597 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.597 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.597 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_repop(client.4694.0:36 2.7 e53/38 2:e8766325:ns::rbd_object_map.1256a0c02879:head v 53'27, mlcod=53'23)", 2024-09-11T09:51:33.597 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.101688+0000", 2024-09-11T09:51:33.597 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 10.442283325, 2024-09-11T09:51:33.597 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0042633289999999997, 2024-09-11T09:51:33.597 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:33.598 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:33.598 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:33.598 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:33.598 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.598 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:33.598 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.101688+0000", 2024-09-11T09:51:33.598 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:33.598 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.598 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.598 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:33.598 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.101688+0000", 2024-09-11T09:51:33.598 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:33.598 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.599 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.599 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:33.599 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.101689+0000", 2024-09-11T09:51:33.599 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.9299999999999997e-07 2024-09-11T09:51:33.599 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.599 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.599 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:33.599 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.101694+0000", 2024-09-11T09:51:33.599 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.6779999999999997e-06 2024-09-11T09:51:33.599 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.599 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.599 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:33.599 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.101707+0000", 2024-09-11T09:51:33.600 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.2549e-05 2024-09-11T09:51:33.600 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.600 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.600 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:33.600 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.101714+0000", 2024-09-11T09:51:33.600 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.9210000000000003e-06 2024-09-11T09:51:33.600 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.600 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.600 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:33.600 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.102635+0000", 2024-09-11T09:51:33.600 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00092109400000000001 2024-09-11T09:51:33.600 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.600 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.600 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:33.601 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.102649+0000", 2024-09-11T09:51:33.601 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.3913e-05 2024-09-11T09:51:33.601 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.601 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.601 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:33.601 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.105931+0000", 2024-09-11T09:51:33.601 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.003281876 2024-09-11T09:51:33.601 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.601 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.601 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:33.601 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.105951+0000", 2024-09-11T09:51:33.601 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.0505000000000001e-05 2024-09-11T09:51:33.601 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:33.601 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:33.602 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:33.602 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.602 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.602 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_repop(client.4694.0:37 2.7 e53/38 2:e8766325:ns::rbd_object_map.1256a0c02879:head v 53'28, mlcod=53'23)", 2024-09-11T09:51:33.602 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.102274+0000", 2024-09-11T09:51:33.602 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 10.441697274999999, 2024-09-11T09:51:33.602 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0036897850000000001, 2024-09-11T09:51:33.602 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:33.602 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:33.602 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:33.602 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:33.602 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.602 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:33.603 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.102274+0000", 2024-09-11T09:51:33.603 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:33.603 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.603 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.603 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:33.603 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.102274+0000", 2024-09-11T09:51:33.603 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:33.603 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.603 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.603 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:33.603 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.102274+0000", 2024-09-11T09:51:33.603 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.1699999999999998e-07 2024-09-11T09:51:33.603 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.603 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.604 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:33.604 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.102280+0000", 2024-09-11T09:51:33.604 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.1200000000000001e-06 2024-09-11T09:51:33.604 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.604 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.604 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:33.604 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.102291+0000", 2024-09-11T09:51:33.604 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1032e-05 2024-09-11T09:51:33.604 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.604 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.604 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:33.604 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.102297+0000", 2024-09-11T09:51:33.604 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.4919999999999998e-06 2024-09-11T09:51:33.604 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.605 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.605 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:33.605 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.102902+0000", 2024-09-11T09:51:33.605 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00060473799999999996 2024-09-11T09:51:33.605 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.605 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.605 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:33.605 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.102913+0000", 2024-09-11T09:51:33.605 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.131e-05 2024-09-11T09:51:33.605 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.605 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.605 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:33.605 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.105952+0000", 2024-09-11T09:51:33.606 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0030386200000000001 2024-09-11T09:51:33.606 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.606 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.606 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:33.606 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.105964+0000", 2024-09-11T09:51:33.606 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1856e-05 2024-09-11T09:51:33.606 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:33.606 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:33.606 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:33.606 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.606 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.606 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4694.0:41 2.2 2:5e2505c4:ns::rbd_data.1256a0c02879.0000000000000003:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:33.606 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.107511+0000", 2024-09-11T09:51:33.606 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 10.436459848, 2024-09-11T09:51:33.607 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.018571066000000001, 2024-09-11T09:51:33.607 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:33.607 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:33.607 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:33.607 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:33.607 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4694", 2024-09-11T09:51:33.607 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:51:33.607 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 41 2024-09-11T09:51:33.607 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.607 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:33.607 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.607 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:33.607 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.107511+0000", 2024-09-11T09:51:33.608 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:33.608 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.608 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.608 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:33.608 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.107511+0000", 2024-09-11T09:51:33.608 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:33.608 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.608 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.608 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:33.608 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.107513+0000", 2024-09-11T09:51:33.608 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.3489999999999999e-06 2024-09-11T09:51:33.608 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.608 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.609 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:33.609 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.112704+0000", 2024-09-11T09:51:33.609 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0051918509999999999 2024-09-11T09:51:33.609 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.609 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.609 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:33.609 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.112718+0000", 2024-09-11T09:51:33.609 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.359e-05 2024-09-11T09:51:33.609 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.609 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.609 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:33.609 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.112725+0000", 2024-09-11T09:51:33.609 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.9940000000000003e-06 2024-09-11T09:51:33.609 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.610 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.610 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:33.610 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.112788+0000", 2024-09-11T09:51:33.610 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.2447999999999996e-05 2024-09-11T09:51:33.610 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.610 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.610 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:33.610 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.112937+0000", 2024-09-11T09:51:33.610 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00014921499999999999 2024-09-11T09:51:33.610 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.610 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.610 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 1", 2024-09-11T09:51:33.610 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.113085+0000", 2024-09-11T09:51:33.610 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000148085 2024-09-11T09:51:33.611 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.611 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.611 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:33.611 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.124318+0000", 2024-09-11T09:51:33.611 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.011233158 2024-09-11T09:51:33.611 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.611 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.611 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:33.611 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.125957+0000", 2024-09-11T09:51:33.611 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001638701 2024-09-11T09:51:33.611 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.611 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.611 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:33.612 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.126029+0000", 2024-09-11T09:51:33.612 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.2125000000000005e-05 2024-09-11T09:51:33.612 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.612 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.612 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:33.612 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.126082+0000", 2024-09-11T09:51:33.612 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.3550000000000001e-05 2024-09-11T09:51:33.612 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:33.612 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:33.612 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:33.612 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.612 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.612 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4694.0:44 2.3 2:cd1a7d1c:ns::rbd_data.1256a0c02879.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:33.613 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.112744+0000", 2024-09-11T09:51:33.613 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 10.431227281, 2024-09-11T09:51:33.613 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.021672212999999999, 2024-09-11T09:51:33.613 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:33.613 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:33.613 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:33.613 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:33.613 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4694", 2024-09-11T09:51:33.613 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:51:33.613 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 44 2024-09-11T09:51:33.613 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.613 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:33.613 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.614 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:33.614 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.112744+0000", 2024-09-11T09:51:33.614 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:33.614 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.614 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.614 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:33.614 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.112744+0000", 2024-09-11T09:51:33.614 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:33.614 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.614 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.614 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:33.614 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.112745+0000", 2024-09-11T09:51:33.614 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.2900000000000002e-07 2024-09-11T09:51:33.614 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.615 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.615 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:33.615 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.119929+0000", 2024-09-11T09:51:33.615 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0071846230000000002 2024-09-11T09:51:33.615 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.615 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.615 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:33.615 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.119949+0000", 2024-09-11T09:51:33.615 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.9264000000000001e-05 2024-09-11T09:51:33.615 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.615 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.615 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:33.616 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.119959+0000", 2024-09-11T09:51:33.616 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.0465999999999999e-05 2024-09-11T09:51:33.616 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.616 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.616 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:33.616 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.120028+0000", 2024-09-11T09:51:33.616 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.9318000000000003e-05 2024-09-11T09:51:33.616 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.616 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.616 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:33.616 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.120160+0000", 2024-09-11T09:51:33.616 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000132015 2024-09-11T09:51:33.616 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.616 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.617 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 2", 2024-09-11T09:51:33.617 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.120260+0000", 2024-09-11T09:51:33.617 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.9397999999999999e-05 2024-09-11T09:51:33.617 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.617 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.617 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:33.617 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.134246+0000", 2024-09-11T09:51:33.617 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.013986721000000001 2024-09-11T09:51:33.617 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.617 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.617 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:33.617 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.134295+0000", 2024-09-11T09:51:33.617 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.8649000000000001e-05 2024-09-11T09:51:33.618 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.618 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.618 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:33.618 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.134361+0000", 2024-09-11T09:51:33.618 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.6081999999999995e-05 2024-09-11T09:51:33.618 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.618 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.618 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:33.618 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.134416+0000", 2024-09-11T09:51:33.618 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.4848000000000001e-05 2024-09-11T09:51:33.618 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:33.618 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:33.618 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:33.618 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.619 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.619 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4694.0:46 2.3 2:ce0e1469:ns::rbd_data.1256a0c02879.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:33.619 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.119993+0000", 2024-09-11T09:51:33.619 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 10.423977805, 2024-09-11T09:51:33.619 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.025840610999999999, 2024-09-11T09:51:33.619 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:33.619 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:33.619 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:33.619 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:33.619 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4694", 2024-09-11T09:51:33.619 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:51:33.619 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 46 2024-09-11T09:51:33.620 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.620 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:33.620 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.620 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:33.620 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.119993+0000", 2024-09-11T09:51:33.620 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:33.620 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.620 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.620 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:33.620 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.119993+0000", 2024-09-11T09:51:33.620 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:33.620 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.620 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.621 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:33.621 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.119995+0000", 2024-09-11T09:51:33.621 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1930000000000001e-06 2024-09-11T09:51:33.621 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.621 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.621 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:33.621 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.128458+0000", 2024-09-11T09:51:33.621 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0084632630000000004 2024-09-11T09:51:33.621 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.621 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.621 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:33.621 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.128478+0000", 2024-09-11T09:51:33.621 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.0353e-05 2024-09-11T09:51:33.621 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.622 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.622 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:33.622 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.128491+0000", 2024-09-11T09:51:33.622 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.2954e-05 2024-09-11T09:51:33.622 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.622 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.622 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:33.622 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.128558+0000", 2024-09-11T09:51:33.622 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.7076999999999995e-05 2024-09-11T09:51:33.622 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.622 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.622 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:33.622 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.128708+0000", 2024-09-11T09:51:33.622 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00014939599999999999 2024-09-11T09:51:33.623 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.623 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.623 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 2", 2024-09-11T09:51:33.623 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.128866+0000", 2024-09-11T09:51:33.623 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000158145 2024-09-11T09:51:33.623 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.623 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.623 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:33.623 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.141045+0000", 2024-09-11T09:51:33.623 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.012179331999999999 2024-09-11T09:51:33.623 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.623 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.623 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:33.624 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.145742+0000", 2024-09-11T09:51:33.624 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0046972050000000003 2024-09-11T09:51:33.624 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.624 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.624 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:33.624 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.145779+0000", 2024-09-11T09:51:33.624 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.6544000000000001e-05 2024-09-11T09:51:33.624 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.624 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.624 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:33.624 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.145834+0000", 2024-09-11T09:51:33.624 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.5148999999999997e-05 2024-09-11T09:51:33.624 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:33.624 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:33.625 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:33.625 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.625 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.625 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4694.0:48 2.3 2:d2090b70:ns::rbd_data.1256a0c02879.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:33.625 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.130540+0000", 2024-09-11T09:51:33.625 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 10.413431538999999, 2024-09-11T09:51:33.625 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.017167789999999999, 2024-09-11T09:51:33.625 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:33.625 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:33.625 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:33.625 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:33.625 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4694", 2024-09-11T09:51:33.625 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:51:33.626 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 48 2024-09-11T09:51:33.626 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.626 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:33.626 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.626 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:33.626 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.130540+0000", 2024-09-11T09:51:33.626 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:33.626 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.626 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.626 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:33.626 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.130540+0000", 2024-09-11T09:51:33.626 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:33.626 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.626 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.627 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:33.627 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.130542+0000", 2024-09-11T09:51:33.627 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.9209999999999999e-06 2024-09-11T09:51:33.627 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.627 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.627 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:33.627 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.134691+0000", 2024-09-11T09:51:33.627 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0041498739999999996 2024-09-11T09:51:33.627 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.627 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.627 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:33.627 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.134705+0000", 2024-09-11T09:51:33.627 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.3669000000000001e-05 2024-09-11T09:51:33.627 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.628 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.628 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:33.628 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.134712+0000", 2024-09-11T09:51:33.628 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.8410000000000002e-06 2024-09-11T09:51:33.628 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.628 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.628 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:33.628 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.134754+0000", 2024-09-11T09:51:33.628 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.2524999999999997e-05 2024-09-11T09:51:33.628 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.628 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.628 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:33.628 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.134875+0000", 2024-09-11T09:51:33.628 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000120332 2024-09-11T09:51:33.629 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.629 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.629 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 2", 2024-09-11T09:51:33.629 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.135005+0000", 2024-09-11T09:51:33.629 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00013024900000000001 2024-09-11T09:51:33.629 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.629 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.629 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:33.629 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.145971+0000", 2024-09-11T09:51:33.629 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.010966203000000001 2024-09-11T09:51:33.629 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.629 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.629 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:33.629 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.147600+0000", 2024-09-11T09:51:33.630 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001628382 2024-09-11T09:51:33.630 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.630 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.630 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:33.630 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.147635+0000", 2024-09-11T09:51:33.630 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.5420000000000003e-05 2024-09-11T09:51:33.630 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.630 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.630 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:33.630 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.147707+0000", 2024-09-11T09:51:33.630 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.2373999999999995e-05 2024-09-11T09:51:33.630 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:33.630 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:33.630 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:33.631 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.631 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.631 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_repop(client.4694.0:49 2.7 e53/38 2:e8766325:ns::rbd_object_map.1256a0c02879:head v 53'33, mlcod=53'31)", 2024-09-11T09:51:33.631 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.135391+0000", 2024-09-11T09:51:33.631 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 10.408579974, 2024-09-11T09:51:33.631 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0051189859999999998, 2024-09-11T09:51:33.631 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:33.631 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:33.631 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:33.631 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:33.631 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.631 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:33.632 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.135391+0000", 2024-09-11T09:51:33.632 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:33.632 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.632 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.632 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:33.632 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.135391+0000", 2024-09-11T09:51:33.632 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:33.632 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.632 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.632 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:33.632 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.135393+0000", 2024-09-11T09:51:33.632 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.4020000000000001e-06 2024-09-11T09:51:33.632 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.633 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.633 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:33.633 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.135400+0000", 2024-09-11T09:51:33.633 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.5129999999999999e-06 2024-09-11T09:51:33.633 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.633 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.633 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:33.633 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.135411+0000", 2024-09-11T09:51:33.633 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1038e-05 2024-09-11T09:51:33.633 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.633 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.633 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:33.633 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.135418+0000", 2024-09-11T09:51:33.634 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.1489999999999997e-06 2024-09-11T09:51:33.634 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.634 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.634 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:33.634 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.135470+0000", 2024-09-11T09:51:33.634 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.2021999999999998e-05 2024-09-11T09:51:33.634 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.634 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.634 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:33.634 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.135501+0000", 2024-09-11T09:51:33.634 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.0349999999999999e-05 2024-09-11T09:51:33.634 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.634 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.634 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:33.635 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.140453+0000", 2024-09-11T09:51:33.635 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0049522409999999996 2024-09-11T09:51:33.635 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.635 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.635 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:33.635 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.140510+0000", 2024-09-11T09:51:33.635 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.7271000000000001e-05 2024-09-11T09:51:33.635 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:33.635 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:33.635 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:33.635 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.635 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.635 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4694.0:54 2.3 2:c44f0253:ns::rbd_data.1256a0c02879.000000000000000b:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:33.635 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.150151+0000", 2024-09-11T09:51:33.636 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 10.393820122999999, 2024-09-11T09:51:33.636 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.015103484, 2024-09-11T09:51:33.636 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:33.636 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:33.636 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:33.636 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:33.636 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4694", 2024-09-11T09:51:33.636 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:51:33.636 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 54 2024-09-11T09:51:33.636 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.636 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:33.636 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.637 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:33.637 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.150151+0000", 2024-09-11T09:51:33.637 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:33.637 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.637 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.637 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:33.637 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.150151+0000", 2024-09-11T09:51:33.637 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:33.637 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.637 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.637 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:33.637 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.150152+0000", 2024-09-11T09:51:33.637 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.2550000000000001e-06 2024-09-11T09:51:33.638 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.638 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.638 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:33.638 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.154528+0000", 2024-09-11T09:51:33.638 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0043758219999999997 2024-09-11T09:51:33.638 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.638 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.638 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:33.638 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.154541+0000", 2024-09-11T09:51:33.638 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.3414999999999999e-05 2024-09-11T09:51:33.638 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.638 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.638 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:33.638 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.154551+0000", 2024-09-11T09:51:33.639 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.4350000000000003e-06 2024-09-11T09:51:33.639 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.639 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.639 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:33.639 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.154615+0000", 2024-09-11T09:51:33.639 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.3627000000000006e-05 2024-09-11T09:51:33.639 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.639 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.639 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:33.639 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.154717+0000", 2024-09-11T09:51:33.639 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000102729 2024-09-11T09:51:33.639 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.639 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.639 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 2", 2024-09-11T09:51:33.640 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.154826+0000", 2024-09-11T09:51:33.640 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000108409 2024-09-11T09:51:33.640 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.640 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.640 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:33.640 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.165094+0000", 2024-09-11T09:51:33.640 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.010268210999999999 2024-09-11T09:51:33.640 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.640 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.640 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:33.640 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.165138+0000", 2024-09-11T09:51:33.640 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.4078e-05 2024-09-11T09:51:33.641 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.641 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.641 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:33.641 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.165204+0000", 2024-09-11T09:51:33.641 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.5716000000000006e-05 2024-09-11T09:51:33.641 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.641 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.641 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:33.641 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.165254+0000", 2024-09-11T09:51:33.641 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.0787e-05 2024-09-11T09:51:33.641 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:33.641 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:33.642 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:33.642 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.642 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.642 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_repop(client.4694.0:55 2.7 e53/38 2:e8766325:ns::rbd_object_map.1256a0c02879:head v 53'37, mlcod=53'34)", 2024-09-11T09:51:33.642 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.150221+0000", 2024-09-11T09:51:33.642 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 10.393750258000001, 2024-09-11T09:51:33.642 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00530015, 2024-09-11T09:51:33.642 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:33.642 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:33.642 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:33.642 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:33.642 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.642 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:33.643 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.150221+0000", 2024-09-11T09:51:33.643 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:33.643 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.643 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.643 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:33.643 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.150221+0000", 2024-09-11T09:51:33.643 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:33.643 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.643 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.643 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:33.643 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.150221+0000", 2024-09-11T09:51:33.643 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.9100000000000004e-07 2024-09-11T09:51:33.643 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.644 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.644 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:33.644 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.150226+0000", 2024-09-11T09:51:33.644 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.2250000000000002e-06 2024-09-11T09:51:33.644 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.644 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.644 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:33.644 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.150251+0000", 2024-09-11T09:51:33.644 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.4987e-05 2024-09-11T09:51:33.644 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.644 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.644 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:33.644 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.150258+0000", 2024-09-11T09:51:33.645 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.0790000000000003e-06 2024-09-11T09:51:33.645 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.645 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.645 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:33.645 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.150305+0000", 2024-09-11T09:51:33.645 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.6879000000000003e-05 2024-09-11T09:51:33.645 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.645 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.645 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:33.645 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.150320+0000", 2024-09-11T09:51:33.645 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.5268999999999999e-05 2024-09-11T09:51:33.645 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.645 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.646 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:33.646 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.155493+0000", 2024-09-11T09:51:33.646 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0051731390000000002 2024-09-11T09:51:33.646 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.646 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.646 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:33.646 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.155521+0000", 2024-09-11T09:51:33.646 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.8081000000000001e-05 2024-09-11T09:51:33.646 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:33.646 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:33.646 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:33.646 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.646 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.647 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4694.0:56 2.3 2:c462f029:ns::rbd_data.1256a0c02879.000000000000000c:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:33.647 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.154576+0000", 2024-09-11T09:51:33.647 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 10.389395282000001, 2024-09-11T09:51:33.647 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.020781621, 2024-09-11T09:51:33.647 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:33.647 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:33.647 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:33.647 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:33.647 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4694", 2024-09-11T09:51:33.647 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:51:33.647 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 56 2024-09-11T09:51:33.647 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.647 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:33.647 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.648 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:33.648 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.154576+0000", 2024-09-11T09:51:33.648 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:33.648 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.648 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.648 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:33.648 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.154576+0000", 2024-09-11T09:51:33.648 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:33.648 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.648 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.648 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:33.648 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.154577+0000", 2024-09-11T09:51:33.648 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.2249999999999999e-06 2024-09-11T09:51:33.648 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.649 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.649 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:33.649 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.158783+0000", 2024-09-11T09:51:33.649 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0042063889999999996 2024-09-11T09:51:33.649 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.649 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.649 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:33.649 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.158803+0000", 2024-09-11T09:51:33.649 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.9576999999999999e-05 2024-09-11T09:51:33.649 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.649 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.649 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:33.649 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.158815+0000", 2024-09-11T09:51:33.650 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.2221999999999999e-05 2024-09-11T09:51:33.650 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.650 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.650 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:33.650 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.158902+0000", 2024-09-11T09:51:33.650 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.6390000000000005e-05 2024-09-11T09:51:33.650 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.650 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.650 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:33.650 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.159141+0000", 2024-09-11T09:51:33.650 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000238886 2024-09-11T09:51:33.650 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.650 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.650 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 2", 2024-09-11T09:51:33.651 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.159279+0000", 2024-09-11T09:51:33.651 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000138073 2024-09-11T09:51:33.651 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.651 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.651 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:33.651 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.173297+0000", 2024-09-11T09:51:33.651 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.014018113 2024-09-11T09:51:33.651 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.651 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.651 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:33.651 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.175278+0000", 2024-09-11T09:51:33.651 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0019817390000000002 2024-09-11T09:51:33.651 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.651 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.652 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:33.652 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.175316+0000", 2024-09-11T09:51:33.652 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.7829e-05 2024-09-11T09:51:33.652 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.652 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.652 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:33.652 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.175357+0000", 2024-09-11T09:51:33.652 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.1177999999999998e-05 2024-09-11T09:51:33.652 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:33.652 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:33.652 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:33.652 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.653 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.653 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4694.0:59 2.2 2:41adefc9:ns::rbd_data.1256a0c02879.000000000000000e:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:33.653 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.158843+0000", 2024-09-11T09:51:33.653 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 10.385127716, 2024-09-11T09:51:33.653 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.017874207, 2024-09-11T09:51:33.653 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:33.653 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:33.653 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:33.653 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:33.653 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4694", 2024-09-11T09:51:33.653 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:51:33.653 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 59 2024-09-11T09:51:33.653 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.653 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:33.654 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.654 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:33.654 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.158843+0000", 2024-09-11T09:51:33.654 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:33.654 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.654 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.654 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:33.654 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.158843+0000", 2024-09-11T09:51:33.654 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:33.654 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.654 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.654 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:33.654 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.158845+0000", 2024-09-11T09:51:33.655 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.7439999999999999e-06 2024-09-11T09:51:33.655 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.655 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.655 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:33.655 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.162713+0000", 2024-09-11T09:51:33.655 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0038678829999999999 2024-09-11T09:51:33.655 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.655 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.655 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:33.655 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.162728+0000", 2024-09-11T09:51:33.655 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.5114e-05 2024-09-11T09:51:33.655 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.655 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.656 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:33.656 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.162737+0000", 2024-09-11T09:51:33.656 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.1099999999999992e-06 2024-09-11T09:51:33.656 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.656 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.656 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:33.656 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.162799+0000", 2024-09-11T09:51:33.656 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.1934999999999996e-05 2024-09-11T09:51:33.656 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.656 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.656 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:33.656 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.162911+0000", 2024-09-11T09:51:33.657 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000111867 2024-09-11T09:51:33.657 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.657 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.657 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 1", 2024-09-11T09:51:33.657 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.163030+0000", 2024-09-11T09:51:33.657 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000119108 2024-09-11T09:51:33.657 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.657 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.657 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:33.657 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.174086+0000", 2024-09-11T09:51:33.657 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.01105543 2024-09-11T09:51:33.657 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.657 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.657 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:33.658 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.176614+0000", 2024-09-11T09:51:33.658 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0025284940000000001 2024-09-11T09:51:33.658 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.658 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.658 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:33.658 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.176672+0000", 2024-09-11T09:51:33.658 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.7892000000000001e-05 2024-09-11T09:51:33.658 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:33.658 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:33.658 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:33.658 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.176718+0000", 2024-09-11T09:51:33.658 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.5630000000000002e-05 2024-09-11T09:51:33.658 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:33.659 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:33.659 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:33.659 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:33.659 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:33.659 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-11T09:51:33.661 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-11T09:51:33.661 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:51:33.662+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:51:33.740 INFO:teuthology.orchestra.run.smithi017.stderr:noscrub is unset 2024-09-11T09:51:33.761 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:51:33.762+0000 7f3c96ec9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31909) UID: 0 2024-09-11T09:51:33.777 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:51:33.777 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [], 2024-09-11T09:51:33.777 INFO:teuthology.orchestra.run.smithi017.stdout: "num_ops": 0 2024-09-11T09:51:33.777 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:51:33.786 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-11T09:51:33.814 INFO:tasks.workunit.client.0.smithi107.stderr:++ rbd -p rbd --format xml du testrbdnbd27121 2024-09-11T09:51:33.814 INFO:tasks.workunit.client.0.smithi107.stderr:++ xmlstarlet sel -t -m //stats/images/image/provisioned_size -v . 2024-09-11T09:51:33.860 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T09:51:33.868 INFO:tasks.workunit.client.0.smithi107.stderr:+ provisioned=67108864 2024-09-11T09:51:33.869 INFO:tasks.workunit.client.0.smithi107.stderr:++ rbd -p rbd --format xml du testrbdnbd27121 2024-09-11T09:51:33.869 INFO:tasks.workunit.client.0.smithi107.stderr:++ xmlstarlet sel -t -m //stats/images/image/used_size -v . 2024-09-11T09:51:33.871 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:51:33.872+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:51:33.919 INFO:tasks.workunit.client.0.smithi107.stderr:+ used=67108864 2024-09-11T09:51:33.919 INFO:tasks.workunit.client.0.smithi107.stderr:+ '[' 67108864 -eq 67108864 ']' 2024-09-11T09:51:33.919 INFO:tasks.workunit.client.0.smithi107.stderr:+ unmap_device /dev/nbd0 27682 2024-09-11T09:51:33.919 INFO:tasks.workunit.client.0.smithi107.stderr:+ local args=/dev/nbd0 2024-09-11T09:51:33.919 INFO:tasks.workunit.client.0.smithi107.stderr:+ local pid=27682 2024-09-11T09:51:33.919 INFO:tasks.workunit.client.0.smithi107.stderr:+ _sudo rbd device --device-type nbd unmap /dev/nbd0 2024-09-11T09:51:33.919 INFO:tasks.workunit.client.0.smithi107.stderr:+ local cmd 2024-09-11T09:51:33.919 INFO:tasks.workunit.client.0.smithi107.stderr:++ id -u 2024-09-11T09:51:33.920 INFO:tasks.workunit.client.0.smithi107.stderr:+ '[' 1000 -eq 0 ']' 2024-09-11T09:51:33.920 INFO:tasks.workunit.client.0.smithi107.stderr:++ which rbd 2024-09-11T09:51:33.921 INFO:tasks.workunit.client.0.smithi107.stderr:+ cmd=/usr/bin/rbd 2024-09-11T09:51:33.921 INFO:tasks.workunit.client.0.smithi107.stderr:+ shift 2024-09-11T09:51:33.921 INFO:tasks.workunit.client.0.smithi107.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd unmap /dev/nbd0 2024-09-11T09:51:33.937 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:51:33.937 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [], 2024-09-11T09:51:33.937 INFO:teuthology.orchestra.run.smithi017.stdout: "complaint_time": 30, 2024-09-11T09:51:33.937 INFO:teuthology.orchestra.run.smithi017.stdout: "num_blocked_ops": 0 2024-09-11T09:51:33.937 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:51:33.946 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-11T09:51:33.962 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-09-11T09:51:33.964+0000 7f7b6c7d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31904) UID: 0 2024-09-11T09:51:33.998 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-11T09:51:33.998 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T09:51:34.000 INFO:tasks.workunit.client.0.smithi107.stderr:+ rbd device --device-type nbd list 2024-09-11T09:51:34.000 INFO:tasks.workunit.client.0.smithi107.stderr:+ expect_false grep '^27682\b' 2024-09-11T09:51:34.000 INFO:tasks.workunit.client.0.smithi107.stderr:+ grep '^27682\b' 2024-09-11T09:51:34.017 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-11T09:51:34.017 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T09:51:34.033 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-11T09:51:34.034 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T09:51:34.048 INFO:tasks.workunit.client.0.smithi107.stderr:+ return 0 2024-09-11T09:51:34.048 INFO:tasks.workunit.client.0.smithi107.stderr:+ ps -C rbd-nbd 2024-09-11T09:51:34.049 INFO:tasks.workunit.client.0.smithi107.stderr:+ expect_false grep '^ *27682\b' 2024-09-11T09:51:34.049 INFO:tasks.workunit.client.0.smithi107.stderr:+ grep '^ *27682\b' 2024-09-11T09:51:34.051 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-11T09:51:34.051 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T09:51:34.051 INFO:teuthology.orchestra.run.smithi017.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-09-11T09:51:34.057 INFO:tasks.workunit.client.0.smithi107.stderr:+ return 0 2024-09-11T09:51:34.057 INFO:tasks.workunit.client.0.smithi107.stderr:+ sleep 0.5 2024-09-11T09:51:34.059 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:51:34.059 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:51:34.063 INFO:tasks.ceph.osd.0.smithi017.stderr:2024-09-11T09:51:34.064+0000 7fb73c0f1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31665) UID: 0 2024-09-11T09:51:34.068 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-11T09:51:34.068 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-11T09:51:34.069 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T09:51:34.071 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-11T09:51:34.072 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-11T09:51:34.164 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-09-11T09:51:34.164+0000 7f7b6c7d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31904) UID: 0 2024-09-11T09:51:34.219 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:51:34.220 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:51:34.228 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-11T09:51:34.272 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:51:34.273+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:51:34.335 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:51:34.335 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:51:34.344 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-11T09:51:34.385 INFO:tasks.ceph.osd.2.smithi017.stderr:2024-09-11T09:51:34.386+0000 7fd20b170640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31672) UID: 0 2024-09-11T09:51:34.467 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:51:34.468+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:51:34.492 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:51:34.492 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:51:34.501 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-11T09:51:34.559 INFO:tasks.workunit.client.0.smithi107.stderr:++ _sudo rbd device --device-type nbd map rbd/testrbdnbd27121 2024-09-11T09:51:34.559 INFO:tasks.workunit.client.0.smithi107.stderr:++ local cmd 2024-09-11T09:51:34.559 INFO:tasks.workunit.client.0.smithi107.stderr:+++ id -u 2024-09-11T09:51:34.560 INFO:tasks.workunit.client.0.smithi107.stderr:++ '[' 1000 -eq 0 ']' 2024-09-11T09:51:34.560 INFO:tasks.workunit.client.0.smithi107.stderr:+++ which rbd 2024-09-11T09:51:34.562 INFO:tasks.workunit.client.0.smithi107.stderr:++ cmd=/usr/bin/rbd 2024-09-11T09:51:34.562 INFO:tasks.workunit.client.0.smithi107.stderr:++ shift 2024-09-11T09:51:34.562 INFO:tasks.workunit.client.0.smithi107.stderr:++ sudo -nE /usr/bin/rbd device --device-type nbd map rbd/testrbdnbd27121 2024-09-11T09:51:34.585 INFO:tasks.ceph.osd.0.smithi017.stderr:2024-09-11T09:51:34.586+0000 7fb73c0f1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31665) UID: 0 2024-09-11T09:51:34.651 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:51:34.651 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:51:34.661 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-11T09:51:34.667 INFO:tasks.workunit.client.0.smithi107.stderr:+ DEV=/dev/nbd0 2024-09-11T09:51:34.667 INFO:tasks.workunit.client.0.smithi107.stderr:+ get_pid rbd 2024-09-11T09:51:34.667 INFO:tasks.workunit.client.0.smithi107.stderr:+ local pool=rbd 2024-09-11T09:51:34.667 INFO:tasks.workunit.client.0.smithi107.stderr:+ local ns= 2024-09-11T09:51:34.668 INFO:tasks.workunit.client.0.smithi107.stderr:++ rbd device --device-type nbd --format xml list 2024-09-11T09:51:34.668 INFO:tasks.workunit.client.0.smithi107.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd27121'\''][device='\''/dev/nbd0'\'']/id' 2024-09-11T09:51:34.702 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:51:34.703+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:51:34.729 INFO:tasks.workunit.client.0.smithi107.stderr:+ PID=27844 2024-09-11T09:51:34.730 INFO:tasks.workunit.client.0.smithi107.stderr:+ test -n 27844 2024-09-11T09:51:34.730 INFO:tasks.workunit.client.0.smithi107.stderr:+ ps -p 27844 -C rbd-nbd 2024-09-11T09:51:34.740 INFO:tasks.workunit.client.0.smithi107.stdout: PID TTY TIME CMD 2024-09-11T09:51:34.741 INFO:tasks.workunit.client.0.smithi107.stdout: 27844 ? 00:00:00 rbd-nbd 2024-09-11T09:51:34.741 INFO:tasks.workunit.client.0.smithi107.stderr:++ rbd -p rbd --format xml du testrbdnbd27121 2024-09-11T09:51:34.741 INFO:tasks.workunit.client.0.smithi107.stderr:++ xmlstarlet sel -t -m //stats/images/image/provisioned_size -v . 2024-09-11T09:51:34.769 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:51:34.770+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:51:34.786 INFO:tasks.workunit.client.0.smithi107.stderr:+ provisioned=67108864 2024-09-11T09:51:34.787 INFO:tasks.workunit.client.0.smithi107.stderr:++ rbd -p rbd --format xml du testrbdnbd27121 2024-09-11T09:51:34.787 INFO:tasks.workunit.client.0.smithi107.stderr:++ xmlstarlet sel -t -m //stats/images/image/used_size -v . 2024-09-11T09:51:34.811 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:51:34.811 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:51:34.820 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-11T09:51:34.833 INFO:tasks.workunit.client.0.smithi107.stderr:+ used=67108864 2024-09-11T09:51:34.833 INFO:tasks.workunit.client.0.smithi107.stderr:+ '[' 67108864 -eq 67108864 ']' 2024-09-11T09:51:34.833 INFO:tasks.workunit.client.0.smithi107.stderr:+ _sudo blkdiscard /dev/nbd0 2024-09-11T09:51:34.833 INFO:tasks.workunit.client.0.smithi107.stderr:+ local cmd 2024-09-11T09:51:34.834 INFO:tasks.workunit.client.0.smithi107.stderr:++ id -u 2024-09-11T09:51:34.834 INFO:tasks.workunit.client.0.smithi107.stderr:+ '[' 1000 -eq 0 ']' 2024-09-11T09:51:34.834 INFO:tasks.workunit.client.0.smithi107.stderr:++ which blkdiscard 2024-09-11T09:51:34.836 INFO:tasks.workunit.client.0.smithi107.stderr:+ cmd=/usr/sbin/blkdiscard 2024-09-11T09:51:34.836 INFO:tasks.workunit.client.0.smithi107.stderr:+ shift 2024-09-11T09:51:34.836 INFO:tasks.workunit.client.0.smithi107.stderr:+ sudo -nE /usr/sbin/blkdiscard /dev/nbd0 2024-09-11T09:51:34.908 INFO:tasks.ceph.osd.0.smithi017.stderr:2024-09-11T09:51:34.909+0000 7fb73c0f1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31665) UID: 0 2024-09-11T09:51:34.971 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:51:34.971 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:51:34.971 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:51:34.972+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:51:34.980 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-11T09:51:35.072 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-09-11T09:51:35.073+0000 7f7b6c7d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31904) UID: 0 2024-09-11T09:51:35.095 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:51:35.095 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:51:35.104 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-11T09:51:35.188 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:51:35.189+0000 7f3c96ec9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31909) UID: 0 2024-09-11T09:51:35.247 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:51:35.247 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:51:35.256 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-11T09:51:35.265 INFO:tasks.workunit.client.0.smithi107.stderr:+ sync 2024-09-11T09:51:35.297 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-09-11T09:51:35.298+0000 7f7b6c7d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31904) UID: 0 2024-09-11T09:51:35.310 INFO:tasks.workunit.client.0.smithi107.stderr:++ rbd -p rbd --format xml du testrbdnbd27121 2024-09-11T09:51:35.310 INFO:tasks.workunit.client.0.smithi107.stderr:++ xmlstarlet sel -t -m //stats/images/image/provisioned_size -v . 2024-09-11T09:51:35.359 INFO:tasks.workunit.client.0.smithi107.stderr:+ provisioned=67108864 2024-09-11T09:51:35.360 INFO:tasks.workunit.client.0.smithi107.stderr:++ rbd -p rbd --format xml du testrbdnbd27121 2024-09-11T09:51:35.360 INFO:tasks.workunit.client.0.smithi107.stderr:++ xmlstarlet sel -t -m //stats/images/image/used_size -v . 2024-09-11T09:51:35.374 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-09-11T09:51:35.375+0000 7f7b6c7d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31904) UID: 0 2024-09-11T09:51:35.402 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:51:35.402 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:51:35.408 INFO:tasks.workunit.client.0.smithi107.stderr:+ used=0 2024-09-11T09:51:35.408 INFO:tasks.workunit.client.0.smithi107.stderr:+ '[' 0 -lt 67108864 ']' 2024-09-11T09:51:35.408 INFO:tasks.workunit.client.0.smithi107.stderr:+ unmap_device /dev/nbd0 27844 2024-09-11T09:51:35.408 INFO:tasks.workunit.client.0.smithi107.stderr:+ local args=/dev/nbd0 2024-09-11T09:51:35.408 INFO:tasks.workunit.client.0.smithi107.stderr:+ local pid=27844 2024-09-11T09:51:35.408 INFO:tasks.workunit.client.0.smithi107.stderr:+ _sudo rbd device --device-type nbd unmap /dev/nbd0 2024-09-11T09:51:35.408 INFO:tasks.workunit.client.0.smithi107.stderr:+ local cmd 2024-09-11T09:51:35.408 INFO:tasks.workunit.client.0.smithi107.stderr:++ id -u 2024-09-11T09:51:35.409 INFO:tasks.workunit.client.0.smithi107.stderr:+ '[' 1000 -eq 0 ']' 2024-09-11T09:51:35.409 INFO:tasks.workunit.client.0.smithi107.stderr:++ which rbd 2024-09-11T09:51:35.410 INFO:tasks.workunit.client.0.smithi107.stderr:+ cmd=/usr/bin/rbd 2024-09-11T09:51:35.410 INFO:tasks.workunit.client.0.smithi107.stderr:+ shift 2024-09-11T09:51:35.410 INFO:tasks.workunit.client.0.smithi107.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd unmap /dev/nbd0 2024-09-11T09:51:35.411 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-11T09:51:35.489 INFO:tasks.workunit.client.0.smithi107.stderr:+ rbd device --device-type nbd list 2024-09-11T09:51:35.490 INFO:tasks.workunit.client.0.smithi107.stderr:+ expect_false grep '^27844\b' 2024-09-11T09:51:35.490 INFO:tasks.workunit.client.0.smithi107.stderr:+ grep '^27844\b' 2024-09-11T09:51:35.495 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:51:35.496+0000 7f3c96ec9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31909) UID: 0 2024-09-11T09:51:35.536 INFO:tasks.workunit.client.0.smithi107.stderr:+ return 0 2024-09-11T09:51:35.537 INFO:tasks.workunit.client.0.smithi107.stderr:+ ps -C rbd-nbd 2024-09-11T09:51:35.537 INFO:tasks.workunit.client.0.smithi107.stderr:+ expect_false grep '^ *27844\b' 2024-09-11T09:51:35.537 INFO:tasks.workunit.client.0.smithi107.stderr:+ grep '^ *27844\b' 2024-09-11T09:51:35.548 INFO:tasks.workunit.client.0.smithi107.stderr:+ return 0 2024-09-11T09:51:35.548 INFO:tasks.workunit.client.0.smithi107.stderr:+ sleep 0.5 2024-09-11T09:51:35.553 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:51:35.554 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:51:35.563 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-11T09:51:35.604 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-09-11T09:51:35.605+0000 7f7b6c7d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31904) UID: 0 2024-09-11T09:51:35.676 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-09-11T09:51:35.677+0000 7f7b6c7d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31904) UID: 0 2024-09-11T09:51:35.706 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:51:35.706 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:51:35.715 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-11T09:51:35.755 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-11T09:51:35.799 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-09-11T09:51:35.800+0000 7f7b6c7d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31904) UID: 0 2024-09-11T09:51:35.858 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:51:35.858 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:51:35.867 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-11T09:51:35.910 INFO:tasks.ceph.osd.2.smithi017.stderr:2024-09-11T09:51:35.911+0000 7fd20b170640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31672) UID: 0 2024-09-11T09:51:35.977 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:51:35.977 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:51:35.978 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-09-11T09:51:35.979+0000 7f7b6c7d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31904) UID: 0 2024-09-11T09:51:35.985 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-11T09:51:36.050 INFO:tasks.workunit.client.0.smithi107.stderr:++ _sudo rbd device -t nbd -o try-netlink map rbd/testrbdnbd27121 2024-09-11T09:51:36.050 INFO:tasks.workunit.client.0.smithi107.stderr:++ local cmd 2024-09-11T09:51:36.050 INFO:tasks.workunit.client.0.smithi107.stderr:+++ id -u 2024-09-11T09:51:36.050 INFO:tasks.workunit.client.0.smithi107.stderr:++ '[' 1000 -eq 0 ']' 2024-09-11T09:51:36.051 INFO:tasks.workunit.client.0.smithi107.stderr:+++ which rbd 2024-09-11T09:51:36.053 INFO:tasks.workunit.client.0.smithi107.stderr:++ cmd=/usr/bin/rbd 2024-09-11T09:51:36.053 INFO:tasks.workunit.client.0.smithi107.stderr:++ shift 2024-09-11T09:51:36.053 INFO:tasks.workunit.client.0.smithi107.stderr:++ sudo -nE /usr/bin/rbd device -t nbd -o try-netlink map rbd/testrbdnbd27121 2024-09-11T09:51:36.072 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T09:51:36.116 INFO:tasks.ceph.osd.0.smithi017.stderr:2024-09-11T09:51:36.117+0000 7fb73c0f1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31665) UID: 0 2024-09-11T09:51:36.135 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:51:36.135 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:51:36.145 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-11T09:51:36.164 INFO:tasks.workunit.client.0.smithi107.stderr:+ DEV=/dev/nbd0 2024-09-11T09:51:36.164 INFO:tasks.workunit.client.0.smithi107.stderr:+ get_pid rbd 2024-09-11T09:51:36.164 INFO:tasks.workunit.client.0.smithi107.stderr:+ local pool=rbd 2024-09-11T09:51:36.164 INFO:tasks.workunit.client.0.smithi107.stderr:+ local ns= 2024-09-11T09:51:36.164 INFO:tasks.workunit.client.0.smithi107.stderr:++ rbd device --device-type nbd --format xml list 2024-09-11T09:51:36.164 INFO:tasks.workunit.client.0.smithi107.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd27121'\''][device='\''/dev/nbd0'\'']/id' 2024-09-11T09:51:36.180 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:51:36.181+0000 7f3c96ec9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31909) UID: 0 2024-09-11T09:51:36.198 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-11T09:51:36.198 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T09:51:36.216 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-11T09:51:36.216 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T09:51:36.224 INFO:tasks.workunit.client.0.smithi107.stderr:+ PID=28006 2024-09-11T09:51:36.225 INFO:tasks.workunit.client.0.smithi107.stderr:+ test -n 28006 2024-09-11T09:51:36.225 INFO:tasks.workunit.client.0.smithi107.stderr:+ ps -p 28006 -C rbd-nbd 2024-09-11T09:51:36.236 INFO:tasks.workunit.client.0.smithi107.stdout: PID TTY TIME CMD 2024-09-11T09:51:36.236 INFO:tasks.workunit.client.0.smithi107.stdout: 28006 ? 00:00:00 rbd-nbd 2024-09-11T09:51:36.236 INFO:tasks.workunit.client.0.smithi107.stderr:++ basename /dev/nbd0 2024-09-11T09:51:36.237 INFO:tasks.workunit.client.0.smithi107.stderr:+ devname=nbd0 2024-09-11T09:51:36.237 INFO:tasks.workunit.client.0.smithi107.stderr:++ awk -v dev=nbd0 '$4 == dev {print $3}' /proc/partitions 2024-09-11T09:51:36.239 INFO:tasks.workunit.client.0.smithi107.stderr:+ blocks=65536 2024-09-11T09:51:36.239 INFO:tasks.workunit.client.0.smithi107.stderr:+ test -n 65536 2024-09-11T09:51:36.239 INFO:tasks.workunit.client.0.smithi107.stderr:+ rbd resize rbd/testrbdnbd27121 --size 128M 2024-09-11T09:51:36.250 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-11T09:51:36.250 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T09:51:36.281 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-09-11T09:51:36.282+0000 7f7b6c7d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31904) UID: 0 2024-09-11T09:51:36.283 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-11T09:51:36.283 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T09:51:36.283 INFO:teuthology.orchestra.run.smithi017.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-09-11T09:51:36.302 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-11T09:51:36.302 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T09:51:36.305 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-11T09:51:36.305 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-11T09:51:36.320 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:51:36.320 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 20, 2024-09-11T09:51:36.320 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 600, 2024-09-11T09:51:36.321 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [ 2024-09-11T09:51:36.321 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.321 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4688.0:34 2.4 e53/32 2:2367385a:::rbd_data.1250f5e25b80.0000000000000002:head v 53'6, mlcod=53'4)", 2024-09-11T09:51:36.321 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.907415+0000", 2024-09-11T09:51:36.321 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 13.412797011, 2024-09-11T09:51:36.321 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.011285778999999999, 2024-09-11T09:51:36.321 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:36.321 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:36.321 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:36.321 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:36.321 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.321 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:36.321 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.907415+0000", 2024-09-11T09:51:36.321 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:36.322 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.322 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.322 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:36.322 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.907415+0000", 2024-09-11T09:51:36.322 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:36.322 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.322 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.322 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:36.322 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.907417+0000", 2024-09-11T09:51:36.322 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.2910000000000002e-06 2024-09-11T09:51:36.322 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.322 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.322 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:36.323 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.910477+0000", 2024-09-11T09:51:36.323 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0030597829999999999 2024-09-11T09:51:36.323 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.323 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.323 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:36.323 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.910501+0000", 2024-09-11T09:51:36.323 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.3332e-05 2024-09-11T09:51:36.323 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.323 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.323 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:36.323 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.910514+0000", 2024-09-11T09:51:36.323 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3957e-05 2024-09-11T09:51:36.323 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.324 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.324 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:36.324 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.910612+0000", 2024-09-11T09:51:36.324 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.7075000000000001e-05 2024-09-11T09:51:36.324 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.324 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.324 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:36.324 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.910640+0000", 2024-09-11T09:51:36.324 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.8768999999999998e-05 2024-09-11T09:51:36.324 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.324 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.324 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:36.324 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.918640+0000", 2024-09-11T09:51:36.324 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0079991899999999998 2024-09-11T09:51:36.325 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.325 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.325 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:36.325 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.918701+0000", 2024-09-11T09:51:36.325 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.1382000000000003e-05 2024-09-11T09:51:36.325 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:36.325 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:36.325 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:36.325 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.325 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.325 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4688.0:38 2.5 e53/32 2:abde2694:::rbd_data.1250f5e25b80.0000000000000004:head v 53'1, mlcod=0'0)", 2024-09-11T09:51:36.325 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.915432+0000", 2024-09-11T09:51:36.326 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 13.404780631, 2024-09-11T09:51:36.326 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.015849652999999998, 2024-09-11T09:51:36.326 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:36.326 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:36.326 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:36.326 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:36.326 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.326 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:36.326 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.915432+0000", 2024-09-11T09:51:36.326 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:36.326 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.326 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.326 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:36.327 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.915432+0000", 2024-09-11T09:51:36.327 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:36.327 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.327 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.327 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:36.327 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.915432+0000", 2024-09-11T09:51:36.327 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.91e-07 2024-09-11T09:51:36.327 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.327 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.327 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:36.327 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.924943+0000", 2024-09-11T09:51:36.327 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0095110630000000002 2024-09-11T09:51:36.327 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.328 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.328 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:36.328 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.924960+0000", 2024-09-11T09:51:36.328 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6915999999999999e-05 2024-09-11T09:51:36.328 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.328 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.328 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:36.328 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.924968+0000", 2024-09-11T09:51:36.328 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.6999999999999997e-06 2024-09-11T09:51:36.328 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.328 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.328 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:36.328 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.925025+0000", 2024-09-11T09:51:36.329 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.6949e-05 2024-09-11T09:51:36.329 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.329 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.329 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:36.329 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.925043+0000", 2024-09-11T09:51:36.329 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7784000000000001e-05 2024-09-11T09:51:36.329 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.329 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.329 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:36.329 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.931237+0000", 2024-09-11T09:51:36.329 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0061935389999999996 2024-09-11T09:51:36.329 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.329 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.330 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:36.330 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.931281+0000", 2024-09-11T09:51:36.330 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.4511000000000001e-05 2024-09-11T09:51:36.330 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:36.330 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:36.330 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:36.330 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.330 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.330 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:40 2.7 2:fe41411c:::rbd_object_map.1250f5e25b80:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:36.330 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.919652+0000", 2024-09-11T09:51:36.330 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 13.400559963999999, 2024-09-11T09:51:36.330 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.012187073, 2024-09-11T09:51:36.330 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:36.331 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:36.331 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:36.331 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:36.331 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:51:36.331 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:51:36.331 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 40 2024-09-11T09:51:36.331 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.331 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:36.331 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.331 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:36.331 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919652+0000", 2024-09-11T09:51:36.332 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:36.332 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.332 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.332 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:36.332 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919652+0000", 2024-09-11T09:51:36.332 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:36.332 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.332 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.332 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:36.332 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919654+0000", 2024-09-11T09:51:36.332 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.88e-06 2024-09-11T09:51:36.332 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.333 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.333 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:36.333 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919669+0000", 2024-09-11T09:51:36.333 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.5271e-05 2024-09-11T09:51:36.333 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.333 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.333 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:36.333 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919687+0000", 2024-09-11T09:51:36.333 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7448999999999998e-05 2024-09-11T09:51:36.333 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.333 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.333 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:36.334 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919695+0000", 2024-09-11T09:51:36.334 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.1149999999999994e-06 2024-09-11T09:51:36.334 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.334 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.334 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:36.334 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919757+0000", 2024-09-11T09:51:36.334 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.2348999999999995e-05 2024-09-11T09:51:36.334 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.334 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.334 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:36.334 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919875+0000", 2024-09-11T09:51:36.334 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00011750499999999999 2024-09-11T09:51:36.334 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.334 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.335 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 5", 2024-09-11T09:51:36.335 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.920201+0000", 2024-09-11T09:51:36.335 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00032628099999999999 2024-09-11T09:51:36.335 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.335 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.335 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:36.335 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.922089+0000", 2024-09-11T09:51:36.335 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.001887978 2024-09-11T09:51:36.335 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.335 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.335 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:36.335 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.931752+0000", 2024-09-11T09:51:36.335 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0096626309999999997 2024-09-11T09:51:36.336 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.336 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.336 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:36.336 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.931794+0000", 2024-09-11T09:51:36.336 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.2849999999999998e-05 2024-09-11T09:51:36.336 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.336 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.336 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:36.336 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.931839+0000", 2024-09-11T09:51:36.336 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.4764e-05 2024-09-11T09:51:36.336 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:36.336 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:36.336 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:36.336 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.337 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.337 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:41 2.7 2:fe41411c:::rbd_object_map.1250f5e25b80:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:36.337 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.919716+0000", 2024-09-11T09:51:36.337 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 13.40049612, 2024-09-11T09:51:36.337 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.015641605999999999, 2024-09-11T09:51:36.337 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:36.337 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:36.337 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:36.337 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:36.337 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:51:36.337 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:51:36.337 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 41 2024-09-11T09:51:36.337 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.337 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:36.338 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.338 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:36.338 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919716+0000", 2024-09-11T09:51:36.338 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:36.338 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.338 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.338 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:36.338 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919716+0000", 2024-09-11T09:51:36.338 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:36.338 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.338 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.338 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:36.338 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919716+0000", 2024-09-11T09:51:36.338 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.9999999999999998e-07 2024-09-11T09:51:36.339 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.339 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.339 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:36.339 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919720+0000", 2024-09-11T09:51:36.339 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.6440000000000002e-06 2024-09-11T09:51:36.339 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.339 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.339 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:36.339 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919729+0000", 2024-09-11T09:51:36.339 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.1759999999999999e-06 2024-09-11T09:51:36.339 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.339 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.339 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:36.339 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919736+0000", 2024-09-11T09:51:36.340 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.5390000000000004e-06 2024-09-11T09:51:36.340 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.340 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.340 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:36.340 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.920669+0000", 2024-09-11T09:51:36.340 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00093275599999999997 2024-09-11T09:51:36.340 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.340 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.340 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:36.340 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.920797+0000", 2024-09-11T09:51:36.340 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00012893400000000001 2024-09-11T09:51:36.340 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.340 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.340 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 5", 2024-09-11T09:51:36.341 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.921269+0000", 2024-09-11T09:51:36.341 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00047128600000000001 2024-09-11T09:51:36.341 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.341 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.341 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:36.341 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.923198+0000", 2024-09-11T09:51:36.341 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0019289369999999999 2024-09-11T09:51:36.341 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.341 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.341 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:36.341 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.935300+0000", 2024-09-11T09:51:36.341 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.012102392 2024-09-11T09:51:36.341 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.341 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.342 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:36.342 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.935332+0000", 2024-09-11T09:51:36.342 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.2292000000000002e-05 2024-09-11T09:51:36.342 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.342 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.342 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:36.342 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.935358+0000", 2024-09-11T09:51:36.342 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.525e-05 2024-09-11T09:51:36.342 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:36.342 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:36.342 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:36.342 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.343 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.343 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:48 2.7 2:fe41411c:::rbd_object_map.1250f5e25b80:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:36.343 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.938799+0000", 2024-09-11T09:51:36.343 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 13.381413010999999, 2024-09-11T09:51:36.343 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0041642340000000002, 2024-09-11T09:51:36.343 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:36.343 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:36.343 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:36.343 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:36.343 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:51:36.343 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:51:36.343 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 48 2024-09-11T09:51:36.343 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.344 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:36.344 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.344 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:36.344 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.938799+0000", 2024-09-11T09:51:36.344 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:36.344 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.344 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.344 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:36.344 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.938799+0000", 2024-09-11T09:51:36.344 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:36.344 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.344 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.344 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:36.345 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.938800+0000", 2024-09-11T09:51:36.345 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.71e-07 2024-09-11T09:51:36.345 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.345 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.345 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:36.345 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.938805+0000", 2024-09-11T09:51:36.345 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.3140000000000003e-06 2024-09-11T09:51:36.345 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.345 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.345 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:36.345 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.938816+0000", 2024-09-11T09:51:36.345 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0231000000000001e-05 2024-09-11T09:51:36.345 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.345 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.346 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:36.346 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.938822+0000", 2024-09-11T09:51:36.346 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.9550000000000003e-06 2024-09-11T09:51:36.346 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.346 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.346 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:36.346 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.938868+0000", 2024-09-11T09:51:36.346 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.5562999999999997e-05 2024-09-11T09:51:36.346 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.346 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.346 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:36.346 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.938918+0000", 2024-09-11T09:51:36.346 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.0317e-05 2024-09-11T09:51:36.346 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.347 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.347 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 5", 2024-09-11T09:51:36.347 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.939114+0000", 2024-09-11T09:51:36.347 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00019538300000000001 2024-09-11T09:51:36.347 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.347 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.347 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:36.347 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.939663+0000", 2024-09-11T09:51:36.347 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00054932199999999998 2024-09-11T09:51:36.347 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.347 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.347 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:36.347 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.942910+0000", 2024-09-11T09:51:36.348 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0032473839999999999 2024-09-11T09:51:36.348 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.348 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.348 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:36.348 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.942940+0000", 2024-09-11T09:51:36.348 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.9306999999999999e-05 2024-09-11T09:51:36.348 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.348 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.348 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:36.348 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.942963+0000", 2024-09-11T09:51:36.348 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.3587e-05 2024-09-11T09:51:36.348 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:36.348 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:36.348 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:36.349 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.349 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.349 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4688.0:49 2.5 e53/32 2:b6dd9fc9:::rbd_data.1250f5e25b80.0000000000000009:head v 53'2, mlcod=0'0)", 2024-09-11T09:51:36.349 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.944712+0000", 2024-09-11T09:51:36.349 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 13.375500172000001, 2024-09-11T09:51:36.349 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.016185069999999999, 2024-09-11T09:51:36.349 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:36.349 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:36.349 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:36.349 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:36.349 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.349 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:36.349 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.944712+0000", 2024-09-11T09:51:36.349 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:36.350 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.350 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.350 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:36.350 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.944712+0000", 2024-09-11T09:51:36.350 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:36.350 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.350 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.350 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:36.350 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.944713+0000", 2024-09-11T09:51:36.350 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0139999999999999e-06 2024-09-11T09:51:36.350 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.350 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.350 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:36.351 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.955518+0000", 2024-09-11T09:51:36.351 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.01080542 2024-09-11T09:51:36.351 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.351 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.351 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:36.351 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.955536+0000", 2024-09-11T09:51:36.351 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7818000000000001e-05 2024-09-11T09:51:36.351 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.351 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.351 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:36.351 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.955544+0000", 2024-09-11T09:51:36.351 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.1699999999999997e-06 2024-09-11T09:51:36.351 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.351 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.351 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:36.352 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.955607+0000", 2024-09-11T09:51:36.352 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.2451000000000002e-05 2024-09-11T09:51:36.352 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.352 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.352 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:36.352 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.955632+0000", 2024-09-11T09:51:36.352 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.5335000000000001e-05 2024-09-11T09:51:36.352 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.352 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.352 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:36.352 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.960826+0000", 2024-09-11T09:51:36.352 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0051938560000000002 2024-09-11T09:51:36.352 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.352 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.353 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:36.353 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.960897+0000", 2024-09-11T09:51:36.353 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.1006000000000004e-05 2024-09-11T09:51:36.353 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:36.353 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:36.353 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:36.353 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.353 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.353 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:53 2.7 2:fe41411c:::rbd_object_map.1250f5e25b80:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:36.353 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.951291+0000", 2024-09-11T09:51:36.353 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 13.368921181999999, 2024-09-11T09:51:36.353 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0044964999999999996, 2024-09-11T09:51:36.353 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:36.354 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:36.354 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:36.354 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:36.354 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:51:36.354 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:51:36.354 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 53 2024-09-11T09:51:36.354 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.354 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:36.354 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.354 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:36.354 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951291+0000", 2024-09-11T09:51:36.354 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:36.355 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.355 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.355 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:36.355 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951291+0000", 2024-09-11T09:51:36.355 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:36.355 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.355 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.355 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:36.355 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951293+0000", 2024-09-11T09:51:36.355 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7889999999999999e-06 2024-09-11T09:51:36.355 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.355 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.355 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:36.355 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951302+0000", 2024-09-11T09:51:36.356 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.9819999999999997e-06 2024-09-11T09:51:36.356 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.356 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.356 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:36.356 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951324+0000", 2024-09-11T09:51:36.356 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.2141e-05 2024-09-11T09:51:36.356 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.356 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.356 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:36.356 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951338+0000", 2024-09-11T09:51:36.356 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3750999999999999e-05 2024-09-11T09:51:36.356 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.356 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.356 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:36.357 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951409+0000", 2024-09-11T09:51:36.357 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.1658999999999999e-05 2024-09-11T09:51:36.357 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.357 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.357 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:36.357 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951517+0000", 2024-09-11T09:51:36.357 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00010746799999999999 2024-09-11T09:51:36.357 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.357 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.357 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 5", 2024-09-11T09:51:36.357 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951942+0000", 2024-09-11T09:51:36.357 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00042524300000000001 2024-09-11T09:51:36.357 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.358 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.358 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:36.358 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.952872+0000", 2024-09-11T09:51:36.358 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00092976300000000001 2024-09-11T09:51:36.358 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.358 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.358 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:36.358 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.955682+0000", 2024-09-11T09:51:36.358 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0028099459999999998 2024-09-11T09:51:36.358 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.358 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.358 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:36.358 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.955746+0000", 2024-09-11T09:51:36.359 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.4084999999999994e-05 2024-09-11T09:51:36.359 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.359 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.359 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:36.359 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.955787+0000", 2024-09-11T09:51:36.359 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.1672999999999999e-05 2024-09-11T09:51:36.359 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:36.359 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:36.359 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:36.359 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.359 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.359 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4688.0:52 2.4 e53/32 2:26d5c2f0:::rbd_data.1250f5e25b80.000000000000000b:head v 53'8, mlcod=53'6)", 2024-09-11T09:51:36.359 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.963677+0000", 2024-09-11T09:51:36.359 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 13.356535627, 2024-09-11T09:51:36.360 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0089437969999999999, 2024-09-11T09:51:36.360 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:36.360 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:36.360 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:36.360 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:36.360 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.360 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:36.360 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.963677+0000", 2024-09-11T09:51:36.360 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:36.360 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.360 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.360 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:36.360 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.963677+0000", 2024-09-11T09:51:36.361 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:36.361 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.361 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.361 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:36.361 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.963677+0000", 2024-09-11T09:51:36.361 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.7199999999999998e-07 2024-09-11T09:51:36.361 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.361 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.361 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:36.361 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.965506+0000", 2024-09-11T09:51:36.361 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.001829064 2024-09-11T09:51:36.361 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.362 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.362 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:36.362 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.965538+0000", 2024-09-11T09:51:36.362 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.1306999999999997e-05 2024-09-11T09:51:36.362 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.362 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.362 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:36.362 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.965553+0000", 2024-09-11T09:51:36.362 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.4924e-05 2024-09-11T09:51:36.362 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.362 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.362 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:36.363 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.965663+0000", 2024-09-11T09:51:36.363 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000110315 2024-09-11T09:51:36.363 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.363 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.363 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:36.363 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.965692+0000", 2024-09-11T09:51:36.363 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.934e-05 2024-09-11T09:51:36.363 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.363 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.363 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:36.363 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.972537+0000", 2024-09-11T09:51:36.363 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0068444969999999997 2024-09-11T09:51:36.363 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.363 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.364 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:36.364 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.972620+0000", 2024-09-11T09:51:36.364 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.3578000000000005e-05 2024-09-11T09:51:36.364 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:36.364 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:36.364 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:36.364 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.364 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.364 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4688.0:54 2.4 e53/32 2:239e915b:::rbd_data.1250f5e25b80.000000000000000c:head v 53'9, mlcod=53'7)", 2024-09-11T09:51:36.364 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.987028+0000", 2024-09-11T09:51:36.364 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 13.33318371, 2024-09-11T09:51:36.364 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0094155650000000007, 2024-09-11T09:51:36.364 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:36.365 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:36.365 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:36.365 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:36.365 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.365 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:36.365 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.987028+0000", 2024-09-11T09:51:36.365 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:36.365 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.365 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.365 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:36.365 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.987028+0000", 2024-09-11T09:51:36.365 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:36.365 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.366 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.366 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:36.366 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.987030+0000", 2024-09-11T09:51:36.366 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6130000000000001e-06 2024-09-11T09:51:36.366 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.366 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.366 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:36.366 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.988575+0000", 2024-09-11T09:51:36.366 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.001545193 2024-09-11T09:51:36.366 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.366 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.366 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:36.366 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.988604+0000", 2024-09-11T09:51:36.366 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.9173e-05 2024-09-11T09:51:36.366 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.367 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.367 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:36.367 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.988620+0000", 2024-09-11T09:51:36.367 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.5517999999999999e-05 2024-09-11T09:51:36.367 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.367 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.367 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:36.367 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.988704+0000", 2024-09-11T09:51:36.367 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.3588e-05 2024-09-11T09:51:36.367 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.367 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.367 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:36.367 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.988735+0000", 2024-09-11T09:51:36.367 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.1643000000000002e-05 2024-09-11T09:51:36.368 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.368 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.368 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:36.368 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.996383+0000", 2024-09-11T09:51:36.368 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0076478099999999997 2024-09-11T09:51:36.368 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.368 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.368 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:36.368 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.996444+0000", 2024-09-11T09:51:36.368 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.1026999999999997e-05 2024-09-11T09:51:36.368 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:36.368 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:36.368 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:36.368 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.369 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.369 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4688.0:56 2.4 e53/32 2:2167012f:::rbd_data.1250f5e25b80.000000000000000d:head v 53'10, mlcod=53'7)", 2024-09-11T09:51:36.369 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.993894+0000", 2024-09-11T09:51:36.369 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 13.326318168, 2024-09-11T09:51:36.369 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0099268559999999995, 2024-09-11T09:51:36.369 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:36.369 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:36.369 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:36.369 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:36.369 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.370 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:36.370 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.993894+0000", 2024-09-11T09:51:36.370 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:36.370 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.370 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.370 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:36.370 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.993894+0000", 2024-09-11T09:51:36.370 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:36.370 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.370 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.370 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:36.370 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.993896+0000", 2024-09-11T09:51:36.371 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6249999999999999e-06 2024-09-11T09:51:36.371 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.371 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.371 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:36.371 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.997606+0000", 2024-09-11T09:51:36.371 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.003710737 2024-09-11T09:51:36.371 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.371 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.371 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:36.371 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.997628+0000", 2024-09-11T09:51:36.371 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.1971e-05 2024-09-11T09:51:36.371 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.371 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.371 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:36.372 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.997640+0000", 2024-09-11T09:51:36.372 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1481000000000001e-05 2024-09-11T09:51:36.372 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.372 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.372 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:36.372 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.997712+0000", 2024-09-11T09:51:36.372 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.2255000000000005e-05 2024-09-11T09:51:36.372 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.372 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.372 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:36.372 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.997738+0000", 2024-09-11T09:51:36.372 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.6050999999999998e-05 2024-09-11T09:51:36.372 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.373 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.373 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:36.373 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.003779+0000", 2024-09-11T09:51:36.373 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0060410960000000001 2024-09-11T09:51:36.373 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.373 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.373 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:36.373 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.003821+0000", 2024-09-11T09:51:36.373 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.1640000000000001e-05 2024-09-11T09:51:36.373 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:36.373 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:36.373 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:36.373 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.374 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.374 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "pg_update_log_missing(2.4 epoch 53/32 rep_tid 16 entries 53'11 (0'0) error 2:2906e6a7:::rbd_header.1250f5e25b80:head by client.4688.0:59 0.000000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 53'7)", 2024-09-11T09:51:36.374 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.998956+0000", 2024-09-11T09:51:36.374 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 13.321255999, 2024-09-11T09:51:36.374 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.004934127, 2024-09-11T09:51:36.374 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:36.374 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:36.374 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "reached pg", 2024-09-11T09:51:36.374 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:36.374 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.374 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:36.374 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.998956+0000", 2024-09-11T09:51:36.375 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:36.375 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.375 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.375 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:36.375 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.998956+0000", 2024-09-11T09:51:36.375 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:36.375 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.375 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.375 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:36.375 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.998957+0000", 2024-09-11T09:51:36.375 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.139e-06 2024-09-11T09:51:36.375 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.375 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.376 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:36.376 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.998985+0000", 2024-09-11T09:51:36.376 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.7912e-05 2024-09-11T09:51:36.376 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.376 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.376 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:36.376 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.999011+0000", 2024-09-11T09:51:36.376 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.6061e-05 2024-09-11T09:51:36.376 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.376 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.376 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:36.376 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.999028+0000", 2024-09-11T09:51:36.376 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7065999999999999e-05 2024-09-11T09:51:36.376 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.377 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.377 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:36.377 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.001919+0000", 2024-09-11T09:51:36.377 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.002891167 2024-09-11T09:51:36.377 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.377 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.377 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:36.377 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.003890+0000", 2024-09-11T09:51:36.377 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0019707819999999999 2024-09-11T09:51:36.377 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:36.377 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:36.377 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:36.377 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.377 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.378 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4694.0:34 2.7 2:f80bfc0c:ns::rbd_data.1256a0c02879.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:36.378 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.085183+0000", 2024-09-11T09:51:36.378 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 13.235029486, 2024-09-11T09:51:36.378 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.020908599, 2024-09-11T09:51:36.378 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:36.378 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:36.378 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:36.378 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:36.378 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4694", 2024-09-11T09:51:36.378 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:51:36.378 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 34 2024-09-11T09:51:36.378 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.378 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:36.379 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.379 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:36.379 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.085183+0000", 2024-09-11T09:51:36.379 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:36.379 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.379 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.379 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:36.379 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.085183+0000", 2024-09-11T09:51:36.379 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:36.379 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.379 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.379 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:36.379 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.085183+0000", 2024-09-11T09:51:36.379 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.4300000000000002e-07 2024-09-11T09:51:36.380 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.380 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.380 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:36.380 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094337+0000", 2024-09-11T09:51:36.380 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0091532769999999996 2024-09-11T09:51:36.380 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.380 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.380 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:36.380 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094362+0000", 2024-09-11T09:51:36.380 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.5287999999999999e-05 2024-09-11T09:51:36.380 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.380 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.380 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:36.381 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094375+0000", 2024-09-11T09:51:36.381 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2446000000000001e-05 2024-09-11T09:51:36.381 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.381 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.381 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:36.381 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094455+0000", 2024-09-11T09:51:36.381 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.0092999999999994e-05 2024-09-11T09:51:36.381 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.381 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.381 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:36.381 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094618+0000", 2024-09-11T09:51:36.381 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000163747 2024-09-11T09:51:36.381 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.382 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.382 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 5", 2024-09-11T09:51:36.382 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094783+0000", 2024-09-11T09:51:36.382 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000165038 2024-09-11T09:51:36.382 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.382 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.382 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:36.382 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.105853+0000", 2024-09-11T09:51:36.382 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.011069294 2024-09-11T09:51:36.382 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.382 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.382 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:36.382 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.106004+0000", 2024-09-11T09:51:36.382 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00015123900000000001 2024-09-11T09:51:36.383 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.383 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.383 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:36.383 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.106062+0000", 2024-09-11T09:51:36.383 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.8205000000000003e-05 2024-09-11T09:51:36.383 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.383 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.383 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:36.383 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.106091+0000", 2024-09-11T09:51:36.383 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.9128999999999998e-05 2024-09-11T09:51:36.383 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:36.383 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:36.383 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:36.384 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.384 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.384 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4694.0:35 2.7 2:e8766325:ns::rbd_object_map.1256a0c02879:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:36.384 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.094405+0000", 2024-09-11T09:51:36.384 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 13.22580685, 2024-09-11T09:51:36.384 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.012623564, 2024-09-11T09:51:36.384 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:36.384 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:36.384 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:36.384 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:36.384 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4694", 2024-09-11T09:51:36.384 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:51:36.384 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 35 2024-09-11T09:51:36.384 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.385 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:36.385 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.385 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:36.385 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094405+0000", 2024-09-11T09:51:36.385 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:36.385 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.385 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.385 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:36.385 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094405+0000", 2024-09-11T09:51:36.385 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:36.385 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.385 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.385 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:36.386 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094407+0000", 2024-09-11T09:51:36.386 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.739e-06 2024-09-11T09:51:36.386 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.386 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.386 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:36.386 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094411+0000", 2024-09-11T09:51:36.386 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.3340000000000003e-06 2024-09-11T09:51:36.386 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.386 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.386 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:36.386 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094419+0000", 2024-09-11T09:51:36.386 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.8909999999999995e-06 2024-09-11T09:51:36.386 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.387 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.387 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:36.387 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094424+0000", 2024-09-11T09:51:36.387 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.7639999999999996e-06 2024-09-11T09:51:36.387 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.387 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.387 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:36.387 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.100209+0000", 2024-09-11T09:51:36.387 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0057850489999999996 2024-09-11T09:51:36.387 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.387 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.387 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:36.387 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.100341+0000", 2024-09-11T09:51:36.387 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000131438 2024-09-11T09:51:36.388 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.388 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.388 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 5", 2024-09-11T09:51:36.388 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.100805+0000", 2024-09-11T09:51:36.388 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00046471499999999998 2024-09-11T09:51:36.388 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.388 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.388 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:36.388 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.106148+0000", 2024-09-11T09:51:36.388 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0053426460000000004 2024-09-11T09:51:36.388 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.388 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.388 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:36.389 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.106929+0000", 2024-09-11T09:51:36.389 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00078161300000000001 2024-09-11T09:51:36.389 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.389 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.389 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:36.389 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.106985+0000", 2024-09-11T09:51:36.389 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.5223999999999998e-05 2024-09-11T09:51:36.389 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.389 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.389 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:36.389 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.107029+0000", 2024-09-11T09:51:36.389 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.4150999999999998e-05 2024-09-11T09:51:36.389 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:36.390 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:36.390 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:36.390 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.390 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.390 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4694.0:36 2.7 2:e8766325:ns::rbd_object_map.1256a0c02879:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:36.390 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.094452+0000", 2024-09-11T09:51:36.390 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 13.22575984, 2024-09-11T09:51:36.390 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.012680383, 2024-09-11T09:51:36.390 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:36.390 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:36.390 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:36.390 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:36.391 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4694", 2024-09-11T09:51:36.391 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:51:36.391 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 36 2024-09-11T09:51:36.391 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.391 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:36.391 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.391 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:36.391 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094452+0000", 2024-09-11T09:51:36.391 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:36.391 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.391 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.391 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:36.391 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094452+0000", 2024-09-11T09:51:36.392 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:36.392 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.392 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.392 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:36.392 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094453+0000", 2024-09-11T09:51:36.392 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.1800000000000002e-07 2024-09-11T09:51:36.392 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.392 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.392 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:36.392 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094455+0000", 2024-09-11T09:51:36.392 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.7020000000000001e-06 2024-09-11T09:51:36.392 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.392 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.393 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:36.393 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094463+0000", 2024-09-11T09:51:36.393 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.1999999999999997e-06 2024-09-11T09:51:36.393 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.393 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.393 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:36.393 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094467+0000", 2024-09-11T09:51:36.393 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.729e-06 2024-09-11T09:51:36.393 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.393 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.393 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:36.393 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.101269+0000", 2024-09-11T09:51:36.393 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0068018799999999997 2024-09-11T09:51:36.393 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.394 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.394 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:36.394 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.101353+0000", 2024-09-11T09:51:36.394 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.3827999999999998e-05 2024-09-11T09:51:36.394 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.394 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.394 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 5", 2024-09-11T09:51:36.394 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.101588+0000", 2024-09-11T09:51:36.394 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00023467099999999999 2024-09-11T09:51:36.394 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.394 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.394 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:36.394 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.106182+0000", 2024-09-11T09:51:36.394 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0045939489999999999 2024-09-11T09:51:36.395 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.395 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.395 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:36.395 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.107041+0000", 2024-09-11T09:51:36.395 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00085965800000000002 2024-09-11T09:51:36.395 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.395 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.395 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:36.395 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.107084+0000", 2024-09-11T09:51:36.395 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.261e-05 2024-09-11T09:51:36.395 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.395 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.395 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:36.395 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.107133+0000", 2024-09-11T09:51:36.396 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.8838000000000002e-05 2024-09-11T09:51:36.396 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:36.396 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:36.396 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:36.396 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.396 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.396 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4694.0:37 2.7 2:e8766325:ns::rbd_object_map.1256a0c02879:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:36.396 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.094473+0000", 2024-09-11T09:51:36.396 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 13.225738689, 2024-09-11T09:51:36.396 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.012778484999999999, 2024-09-11T09:51:36.396 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:36.396 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:36.396 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:36.397 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:36.397 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4694", 2024-09-11T09:51:36.397 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:51:36.397 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 37 2024-09-11T09:51:36.397 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.397 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:36.397 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.397 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:36.397 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094473+0000", 2024-09-11T09:51:36.397 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:36.397 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.397 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.397 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:36.397 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094473+0000", 2024-09-11T09:51:36.397 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:36.398 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.398 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.398 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:36.398 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094474+0000", 2024-09-11T09:51:36.398 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.9699999999999996e-07 2024-09-11T09:51:36.398 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.398 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.398 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:36.398 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094477+0000", 2024-09-11T09:51:36.398 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.151e-06 2024-09-11T09:51:36.398 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.398 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.398 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:36.399 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094483+0000", 2024-09-11T09:51:36.399 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.6330000000000001e-06 2024-09-11T09:51:36.399 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.399 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.399 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:36.399 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094487+0000", 2024-09-11T09:51:36.399 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.5399999999999997e-06 2024-09-11T09:51:36.399 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.399 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.399 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:36.399 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.101866+0000", 2024-09-11T09:51:36.399 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.007378768 2024-09-11T09:51:36.399 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.399 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.399 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:36.399 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.101960+0000", 2024-09-11T09:51:36.400 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.3861000000000001e-05 2024-09-11T09:51:36.400 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.400 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.400 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 5", 2024-09-11T09:51:36.400 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.102197+0000", 2024-09-11T09:51:36.400 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00023723200000000001 2024-09-11T09:51:36.400 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.400 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.400 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:36.400 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.106222+0000", 2024-09-11T09:51:36.400 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0040249689999999998 2024-09-11T09:51:36.400 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.400 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.400 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:36.401 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.107141+0000", 2024-09-11T09:51:36.401 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00091913900000000002 2024-09-11T09:51:36.401 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.401 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.401 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:36.401 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.107198+0000", 2024-09-11T09:51:36.401 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.6564999999999998e-05 2024-09-11T09:51:36.401 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.401 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.401 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:36.401 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.107252+0000", 2024-09-11T09:51:36.401 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.4030000000000003e-05 2024-09-11T09:51:36.401 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:36.401 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:36.401 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:36.402 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.402 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.402 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4694.0:33 2.4 e53/32 2:3bfc818c:ns::rbd_data.1256a0c02879.0000000000000001:head v 53'14, mlcod=53'12)", 2024-09-11T09:51:36.402 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.096496+0000", 2024-09-11T09:51:36.402 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 13.223716116, 2024-09-11T09:51:36.402 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0099139310000000008, 2024-09-11T09:51:36.402 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:36.402 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:36.402 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:36.402 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:36.402 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.402 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:36.402 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.096496+0000", 2024-09-11T09:51:36.403 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:36.403 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.403 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.403 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:36.403 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.096496+0000", 2024-09-11T09:51:36.403 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:36.403 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.403 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.403 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:36.403 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.096498+0000", 2024-09-11T09:51:36.403 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6309999999999999e-06 2024-09-11T09:51:36.403 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.403 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.404 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:36.404 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.098087+0000", 2024-09-11T09:51:36.404 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0015889019999999999 2024-09-11T09:51:36.404 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.404 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.404 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:36.404 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.098112+0000", 2024-09-11T09:51:36.404 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.5369000000000001e-05 2024-09-11T09:51:36.404 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.404 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.404 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:36.404 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.098127+0000", 2024-09-11T09:51:36.404 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.5218999999999999e-05 2024-09-11T09:51:36.404 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.405 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.405 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:36.405 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.098209+0000", 2024-09-11T09:51:36.405 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.2120999999999995e-05 2024-09-11T09:51:36.405 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.405 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.405 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:36.405 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.098237+0000", 2024-09-11T09:51:36.405 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.7657999999999998e-05 2024-09-11T09:51:36.405 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.405 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.405 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:36.405 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.106376+0000", 2024-09-11T09:51:36.406 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0081386749999999997 2024-09-11T09:51:36.406 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.406 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.406 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:36.406 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.106410+0000", 2024-09-11T09:51:36.406 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.4356000000000001e-05 2024-09-11T09:51:36.406 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:36.406 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:36.406 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:36.406 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.406 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.406 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4694.0:49 2.7 2:e8766325:ns::rbd_object_map.1256a0c02879:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:36.406 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.134806+0000", 2024-09-11T09:51:36.406 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 13.185406047000001, 2024-09-11T09:51:36.407 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0061629049999999998, 2024-09-11T09:51:36.407 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:36.407 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:36.407 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:36.407 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:36.407 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4694", 2024-09-11T09:51:36.407 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:51:36.407 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 49 2024-09-11T09:51:36.407 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.407 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:36.407 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.407 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:36.407 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.134806+0000", 2024-09-11T09:51:36.407 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:36.408 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.408 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.408 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:36.408 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.134806+0000", 2024-09-11T09:51:36.408 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:36.408 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.408 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.408 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:36.408 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.134808+0000", 2024-09-11T09:51:36.408 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.9870000000000002e-06 2024-09-11T09:51:36.408 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.408 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.408 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:36.409 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.134818+0000", 2024-09-11T09:51:36.409 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.5599999999999999e-06 2024-09-11T09:51:36.409 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.409 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.409 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:36.409 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.134835+0000", 2024-09-11T09:51:36.409 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7399000000000001e-05 2024-09-11T09:51:36.409 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.409 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.409 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:36.409 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.134845+0000", 2024-09-11T09:51:36.409 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0285000000000001e-05 2024-09-11T09:51:36.409 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.409 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.410 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:36.410 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.134911+0000", 2024-09-11T09:51:36.410 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.5654999999999994e-05 2024-09-11T09:51:36.410 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.410 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.410 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:36.410 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.135023+0000", 2024-09-11T09:51:36.410 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00011186799999999999 2024-09-11T09:51:36.410 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.410 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.410 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 5", 2024-09-11T09:51:36.410 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.135290+0000", 2024-09-11T09:51:36.410 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00026725300000000001 2024-09-11T09:51:36.411 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.411 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.411 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:36.411 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.136506+0000", 2024-09-11T09:51:36.411 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0012161139999999999 2024-09-11T09:51:36.411 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.411 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.411 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:36.411 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.140884+0000", 2024-09-11T09:51:36.411 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0043781990000000002 2024-09-11T09:51:36.411 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.411 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.411 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:36.411 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.140923+0000", 2024-09-11T09:51:36.412 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.8813999999999998e-05 2024-09-11T09:51:36.412 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.412 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.412 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:36.412 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.140969+0000", 2024-09-11T09:51:36.412 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.5770999999999999e-05 2024-09-11T09:51:36.412 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:36.412 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:36.412 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:36.412 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.412 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.412 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4694.0:42 2.5 e53/32 2:a4fc00ec:ns::rbd_data.1256a0c02879.0000000000000004:head v 53'3, mlcod=53'1)", 2024-09-11T09:51:36.413 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.136646+0000", 2024-09-11T09:51:36.413 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 13.183565741000001, 2024-09-11T09:51:36.413 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.012545501000000001, 2024-09-11T09:51:36.413 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:36.413 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:36.413 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:36.413 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:36.413 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.413 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:36.413 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.136646+0000", 2024-09-11T09:51:36.413 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:36.413 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.413 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.414 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:36.414 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.136646+0000", 2024-09-11T09:51:36.414 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:36.414 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.414 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.414 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:36.414 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.136647+0000", 2024-09-11T09:51:36.414 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.2900000000000002e-07 2024-09-11T09:51:36.414 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.414 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.414 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:36.414 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.143620+0000", 2024-09-11T09:51:36.414 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0069727440000000003 2024-09-11T09:51:36.414 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.415 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.415 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:36.415 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.143642+0000", 2024-09-11T09:51:36.415 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.1846000000000001e-05 2024-09-11T09:51:36.415 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.415 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.415 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:36.415 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.143654+0000", 2024-09-11T09:51:36.415 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2081e-05 2024-09-11T09:51:36.415 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.415 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.415 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:36.415 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.143717+0000", 2024-09-11T09:51:36.416 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.3443999999999998e-05 2024-09-11T09:51:36.416 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.416 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.416 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:36.416 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.143740+0000", 2024-09-11T09:51:36.416 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.2626999999999999e-05 2024-09-11T09:51:36.416 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.416 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.416 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:36.416 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.149142+0000", 2024-09-11T09:51:36.416 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0054017630000000004 2024-09-11T09:51:36.416 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.416 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.417 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:36.417 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.149192+0000", 2024-09-11T09:51:36.417 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.0067e-05 2024-09-11T09:51:36.417 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:36.417 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:36.417 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:36.417 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.417 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.417 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4694.0:53 2.7 2:e8766325:ns::rbd_object_map.1256a0c02879:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:36.417 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.147978+0000", 2024-09-11T09:51:36.417 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 13.172234257, 2024-09-11T09:51:36.417 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0044549769999999997, 2024-09-11T09:51:36.417 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:36.417 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:36.418 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:36.418 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:36.418 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4694", 2024-09-11T09:51:36.418 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:51:36.418 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 53 2024-09-11T09:51:36.418 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.418 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:36.418 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.418 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:36.418 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.147978+0000", 2024-09-11T09:51:36.418 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:36.418 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.418 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.419 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:36.419 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.147978+0000", 2024-09-11T09:51:36.419 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:36.419 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.419 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.419 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:36.419 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.147980+0000", 2024-09-11T09:51:36.419 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.8309999999999999e-06 2024-09-11T09:51:36.419 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.419 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.419 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:36.419 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.147989+0000", 2024-09-11T09:51:36.419 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.9639999999999992e-06 2024-09-11T09:51:36.419 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.420 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.420 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:36.420 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.148007+0000", 2024-09-11T09:51:36.420 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7884999999999999e-05 2024-09-11T09:51:36.420 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.420 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.420 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:36.420 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.148017+0000", 2024-09-11T09:51:36.420 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0895e-05 2024-09-11T09:51:36.420 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.420 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.420 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:36.420 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.148091+0000", 2024-09-11T09:51:36.421 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.3709999999999997e-05 2024-09-11T09:51:36.421 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.421 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.421 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:36.421 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.148189+0000", 2024-09-11T09:51:36.421 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.7403000000000002e-05 2024-09-11T09:51:36.421 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.421 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.421 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 5", 2024-09-11T09:51:36.421 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.148504+0000", 2024-09-11T09:51:36.421 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00031589900000000002 2024-09-11T09:51:36.421 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.422 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.422 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:36.422 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.149514+0000", 2024-09-11T09:51:36.422 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0010096 2024-09-11T09:51:36.422 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.422 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.422 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:36.422 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.152367+0000", 2024-09-11T09:51:36.422 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0028532309999999999 2024-09-11T09:51:36.422 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.422 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.422 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:36.422 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.152402+0000", 2024-09-11T09:51:36.423 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.4232e-05 2024-09-11T09:51:36.423 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.423 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.423 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:36.423 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.152433+0000", 2024-09-11T09:51:36.423 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.1327e-05 2024-09-11T09:51:36.423 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:36.423 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:36.423 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:36.423 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.423 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.423 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4694.0:55 2.7 2:e8766325:ns::rbd_object_map.1256a0c02879:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:36.423 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.149838+0000", 2024-09-11T09:51:36.423 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 13.170374524, 2024-09-11T09:51:36.423 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0060695159999999996, 2024-09-11T09:51:36.424 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:36.424 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:36.424 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:36.424 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:36.424 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4694", 2024-09-11T09:51:36.424 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:51:36.424 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 55 2024-09-11T09:51:36.424 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.424 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:36.424 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.424 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:36.424 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.149838+0000", 2024-09-11T09:51:36.424 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:36.424 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.425 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.425 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:36.425 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.149838+0000", 2024-09-11T09:51:36.425 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:36.425 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.425 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.425 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:36.425 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.149839+0000", 2024-09-11T09:51:36.425 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.119e-06 2024-09-11T09:51:36.425 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.425 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.425 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:36.425 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.149845+0000", 2024-09-11T09:51:36.426 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.7810000000000002e-06 2024-09-11T09:51:36.426 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.426 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.426 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:36.426 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.149856+0000", 2024-09-11T09:51:36.426 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1199999999999999e-05 2024-09-11T09:51:36.426 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.426 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.426 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:36.426 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.149863+0000", 2024-09-11T09:51:36.426 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.4109999999999999e-06 2024-09-11T09:51:36.426 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.426 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.426 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:36.427 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.149908+0000", 2024-09-11T09:51:36.427 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.5184e-05 2024-09-11T09:51:36.427 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.427 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.427 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:36.427 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.149964+0000", 2024-09-11T09:51:36.427 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.5395000000000001e-05 2024-09-11T09:51:36.427 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.427 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.427 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 5", 2024-09-11T09:51:36.427 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.150157+0000", 2024-09-11T09:51:36.427 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000193489 2024-09-11T09:51:36.427 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.427 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.428 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:36.428 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.152238+0000", 2024-09-11T09:51:36.428 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0020803480000000001 2024-09-11T09:51:36.428 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.428 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.428 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:36.428 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.155828+0000", 2024-09-11T09:51:36.428 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0035902210000000002 2024-09-11T09:51:36.428 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.428 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.428 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:36.428 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.155873+0000", 2024-09-11T09:51:36.428 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.5145999999999998e-05 2024-09-11T09:51:36.428 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.428 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.429 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:36.429 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.155907+0000", 2024-09-11T09:51:36.429 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.4221999999999999e-05 2024-09-11T09:51:36.429 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:36.429 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:36.429 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:36.429 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:36.429 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:36.429 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:51:36.431 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-11T09:51:36.431 INFO:tasks.ceph.osd.0.smithi017.stderr:2024-09-11T09:51:36.432+0000 7fb73c0f1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31665) UID: 0 2024-09-11T09:51:36.489 INFO:tasks.workunit.client.0.smithi107.stderr: Resizing image: 100% complete...done. 2024-09-11T09:51:36.496 INFO:tasks.workunit.client.0.smithi107.stderr:+ rbd info rbd/testrbdnbd27121 2024-09-11T09:51:36.531 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-09-11T09:51:36.532+0000 7f7b6c7d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31904) UID: 0 2024-09-11T09:51:36.534 INFO:tasks.workunit.client.0.smithi107.stdout:rbd image 'testrbdnbd27121': 2024-09-11T09:51:36.535 INFO:tasks.workunit.client.0.smithi107.stdout: size 128 MiB in 32 objects 2024-09-11T09:51:36.535 INFO:tasks.workunit.client.0.smithi107.stdout: order 22 (4 MiB objects) 2024-09-11T09:51:36.535 INFO:tasks.workunit.client.0.smithi107.stdout: snapshot_count: 0 2024-09-11T09:51:36.535 INFO:tasks.workunit.client.0.smithi107.stdout: id: 1250f5e25b80 2024-09-11T09:51:36.535 INFO:tasks.workunit.client.0.smithi107.stdout: block_name_prefix: rbd_data.1250f5e25b80 2024-09-11T09:51:36.535 INFO:tasks.workunit.client.0.smithi107.stdout: format: 2 2024-09-11T09:51:36.535 INFO:tasks.workunit.client.0.smithi107.stdout: features: layering, exclusive-lock, object-map, fast-diff, deep-flatten 2024-09-11T09:51:36.535 INFO:tasks.workunit.client.0.smithi107.stdout: op_features: 2024-09-11T09:51:36.535 INFO:tasks.workunit.client.0.smithi107.stdout: flags: 2024-09-11T09:51:36.535 INFO:tasks.workunit.client.0.smithi107.stdout: create_timestamp: Wed Sep 11 09:51:22 2024 2024-09-11T09:51:36.535 INFO:tasks.workunit.client.0.smithi107.stdout: access_timestamp: Wed Sep 11 09:51:22 2024 2024-09-11T09:51:36.535 INFO:tasks.workunit.client.0.smithi107.stdout: modify_timestamp: Wed Sep 11 09:51:22 2024 2024-09-11T09:51:36.538 INFO:tasks.workunit.client.0.smithi107.stderr:++ awk -v dev=nbd0 '$4 == dev {print $3}' /proc/partitions 2024-09-11T09:51:36.539 INFO:tasks.workunit.client.0.smithi107.stderr:+ blocks2=131072 2024-09-11T09:51:36.540 INFO:tasks.workunit.client.0.smithi107.stderr:+ test -n 131072 2024-09-11T09:51:36.540 INFO:tasks.workunit.client.0.smithi107.stderr:+ test 131072 -eq 131072 2024-09-11T09:51:36.540 INFO:tasks.workunit.client.0.smithi107.stderr:+ rbd resize rbd/testrbdnbd27121 --allow-shrink --size 64M 2024-09-11T09:51:36.584 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:51:36.584 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [ 2024-09-11T09:51:36.584 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.584 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4984.0:17 2.4 2:2906e6a7:::rbd_header.1250f5e25b80:head [notify cookie 139933188720896 in=38b out=8b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e60)", 2024-09-11T09:51:36.585 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:36.585711+0000", 2024-09-11T09:51:36.585 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 0.00019970200000000001, 2024-09-11T09:51:36.585 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00022427100000000001, 2024-09-11T09:51:36.585 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:36.585 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:36.585 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "started", 2024-09-11T09:51:36.585 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:36.585 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4984", 2024-09-11T09:51:36.585 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/2767063524", 2024-09-11T09:51:36.585 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 17 2024-09-11T09:51:36.585 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.585 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:36.586 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.586 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:36.586 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:36.585711+0000", 2024-09-11T09:51:36.586 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:36.586 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.586 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.586 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:36.586 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:36.585711+0000", 2024-09-11T09:51:36.586 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:36.586 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.586 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.586 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:36.586 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:36.585712+0000", 2024-09-11T09:51:36.586 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.8899999999999998e-07 2024-09-11T09:51:36.587 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.587 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.587 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:36.587 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:36.585716+0000", 2024-09-11T09:51:36.587 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.0180000000000003e-06 2024-09-11T09:51:36.587 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.587 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.587 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:36.587 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:36.585726+0000", 2024-09-11T09:51:36.587 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0267e-05 2024-09-11T09:51:36.587 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.587 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.587 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:36.588 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:36.585731+0000", 2024-09-11T09:51:36.588 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.1209999999999998e-06 2024-09-11T09:51:36.588 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.588 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.588 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:36.588 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:36.585775+0000", 2024-09-11T09:51:36.588 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.3509e-05 2024-09-11T09:51:36.588 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.588 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.588 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:36.588 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:36.585822+0000", 2024-09-11T09:51:36.588 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.6990999999999998e-05 2024-09-11T09:51:36.588 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:36.589 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:36.589 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:36.589 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.589 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.589 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4984.0:18 2.4 2.e5676094 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e60)", 2024-09-11T09:51:36.589 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:36.585824+0000", 2024-09-11T09:51:36.589 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 8.6995000000000003e-05, 2024-09-11T09:51:36.589 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000173795, 2024-09-11T09:51:36.589 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:36.589 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:36.589 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "reached pg", 2024-09-11T09:51:36.589 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:36.589 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4984", 2024-09-11T09:51:36.589 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/2767063524", 2024-09-11T09:51:36.590 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 18 2024-09-11T09:51:36.590 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.590 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:36.590 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.590 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:36.590 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:36.585824+0000", 2024-09-11T09:51:36.590 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:36.590 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.590 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.590 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:36.590 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:36.585824+0000", 2024-09-11T09:51:36.590 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:36.590 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.591 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.591 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:36.591 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:36.585824+0000", 2024-09-11T09:51:36.591 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.9999999999999997e-07 2024-09-11T09:51:36.591 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.591 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.591 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:36.591 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:36.585828+0000", 2024-09-11T09:51:36.591 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.7170000000000002e-06 2024-09-11T09:51:36.591 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.591 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.591 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:36.591 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:36.585834+0000", 2024-09-11T09:51:36.591 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.0039999999999996e-06 2024-09-11T09:51:36.592 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.592 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.592 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:36.592 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:36.585846+0000", 2024-09-11T09:51:36.592 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1539000000000001e-05 2024-09-11T09:51:36.592 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.592 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.592 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:36.592 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:36.585985+0000", 2024-09-11T09:51:36.592 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000139276 2024-09-11T09:51:36.592 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:36.592 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:36.592 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:36.593 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.593 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.593 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4984.0:19 2.4 2.e5676094 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e60)", 2024-09-11T09:51:36.593 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:36.585995+0000", 2024-09-11T09:51:36.593 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 4294967295.9999156, 2024-09-11T09:51:36.593 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.7791999999999999e-05, 2024-09-11T09:51:36.593 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:36.593 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:36.593 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "queued for pg", 2024-09-11T09:51:36.593 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:36.593 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4984", 2024-09-11T09:51:36.593 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/2767063524", 2024-09-11T09:51:36.593 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 19 2024-09-11T09:51:36.593 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.594 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:36.594 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.594 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:36.594 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:36.585995+0000", 2024-09-11T09:51:36.594 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:36.594 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.594 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.594 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:36.594 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:36.585995+0000", 2024-09-11T09:51:36.594 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:36.594 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.594 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.594 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:36.595 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:36.585996+0000", 2024-09-11T09:51:36.595 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.8999999999999996e-07 2024-09-11T09:51:36.595 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.595 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.595 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:36.595 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:36.585999+0000", 2024-09-11T09:51:36.595 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.05e-06 2024-09-11T09:51:36.595 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.595 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.595 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:36.595 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:36.586007+0000", 2024-09-11T09:51:36.595 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.5099999999999998e-06 2024-09-11T09:51:36.595 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.595 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.596 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:36.596 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:36.586014+0000", 2024-09-11T09:51:36.596 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.6069999999999996e-06 2024-09-11T09:51:36.596 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:36.596 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:36.596 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:36.596 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:36.596 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-11T09:51:36.596 INFO:teuthology.orchestra.run.smithi017.stdout: "num_ops": 3 2024-09-11T09:51:36.596 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:51:36.598 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-11T09:51:36.599 INFO:tasks.workunit.client.0.smithi107.stderr: Resizing image: 50% complete... Resizing image: 53% complete... Resizing image: 56% complete... Resizing image: 59% complete... Resizing image: 62% complete... Resizing image: 65% complete... Resizing image: 68% complete... Resizing image: 71% complete... Resizing image: 75% complete... Resizing image: 78% complete... Resizing image: 81% complete... Resizing image: 84% complete... Resizing image: 87% complete... Resizing image: 90% complete... Resizing image: 93% complete... Resizing image: 96% complete... Resizing image: 100% complete...done. 2024-09-11T09:51:36.599 INFO:tasks.workunit.client.0.smithi107.stderr:++ awk -v dev=nbd0 '$4 == dev {print $3}' /proc/partitions 2024-09-11T09:51:36.599 INFO:tasks.workunit.client.0.smithi107.stderr:+ blocks2=65536 2024-09-11T09:51:36.599 INFO:tasks.workunit.client.0.smithi107.stderr:+ test -n 65536 2024-09-11T09:51:36.599 INFO:tasks.workunit.client.0.smithi107.stderr:+ test 65536 -eq 65536 2024-09-11T09:51:36.599 INFO:tasks.workunit.client.0.smithi107.stderr:+ unmap_device /dev/nbd0 28006 2024-09-11T09:51:36.600 INFO:tasks.workunit.client.0.smithi107.stderr:+ local args=/dev/nbd0 2024-09-11T09:51:36.600 INFO:tasks.workunit.client.0.smithi107.stderr:+ local pid=28006 2024-09-11T09:51:36.600 INFO:tasks.workunit.client.0.smithi107.stderr:+ _sudo rbd device --device-type nbd unmap /dev/nbd0 2024-09-11T09:51:36.600 INFO:tasks.workunit.client.0.smithi107.stderr:+ local cmd 2024-09-11T09:51:36.600 INFO:tasks.workunit.client.0.smithi107.stderr:++ id -u 2024-09-11T09:51:36.600 INFO:tasks.workunit.client.0.smithi107.stderr:+ '[' 1000 -eq 0 ']' 2024-09-11T09:51:36.600 INFO:tasks.workunit.client.0.smithi107.stderr:++ which rbd 2024-09-11T09:51:36.601 INFO:tasks.workunit.client.0.smithi107.stderr:+ cmd=/usr/bin/rbd 2024-09-11T09:51:36.601 INFO:tasks.workunit.client.0.smithi107.stderr:+ shift 2024-09-11T09:51:36.601 INFO:tasks.workunit.client.0.smithi107.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd unmap /dev/nbd0 2024-09-11T09:51:36.644 INFO:tasks.ceph.osd.2.smithi017.stderr:2024-09-11T09:51:36.645+0000 7fd20b170640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31672) UID: 0 2024-09-11T09:51:36.681 INFO:tasks.workunit.client.0.smithi107.stderr:+ rbd device --device-type nbd list 2024-09-11T09:51:36.681 INFO:tasks.workunit.client.0.smithi107.stderr:+ expect_false grep '^28006\b' 2024-09-11T09:51:36.681 INFO:tasks.workunit.client.0.smithi107.stderr:+ grep '^28006\b' 2024-09-11T09:51:36.708 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:51:36.708 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [], 2024-09-11T09:51:36.708 INFO:teuthology.orchestra.run.smithi017.stdout: "complaint_time": 30, 2024-09-11T09:51:36.708 INFO:teuthology.orchestra.run.smithi017.stdout: "num_blocked_ops": 0 2024-09-11T09:51:36.708 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:51:36.716 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-11T09:51:36.728 INFO:tasks.workunit.client.0.smithi107.stderr:+ return 0 2024-09-11T09:51:36.729 INFO:tasks.workunit.client.0.smithi107.stderr:+ ps -C rbd-nbd 2024-09-11T09:51:36.729 INFO:tasks.workunit.client.0.smithi107.stderr:+ expect_false grep '^ *28006\b' 2024-09-11T09:51:36.729 INFO:tasks.workunit.client.0.smithi107.stderr:+ grep '^ *28006\b' 2024-09-11T09:51:36.733 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:51:36.734+0000 7f3c96ec9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31909) UID: 0 2024-09-11T09:51:36.740 INFO:tasks.workunit.client.0.smithi107.stderr:+ return 0 2024-09-11T09:51:36.740 INFO:tasks.workunit.client.0.smithi107.stderr:+ sleep 0.5 2024-09-11T09:51:36.743 INFO:teuthology.orchestra.run.smithi017.stderr:nodeep-scrub is unset 2024-09-11T09:51:36.835 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-09-11T09:51:36.836+0000 7f7b6c7d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31904) UID: 0 2024-09-11T09:51:36.852 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:51:36.852 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 20, 2024-09-11T09:51:36.853 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 600, 2024-09-11T09:51:36.853 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [ 2024-09-11T09:51:36.853 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.853 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:34 2.4 2:2367385a:::rbd_data.1250f5e25b80.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:36.853 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.897717+0000", 2024-09-11T09:51:36.853 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 13.954504503999999, 2024-09-11T09:51:36.853 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.021323661000000001, 2024-09-11T09:51:36.853 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:36.853 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:36.853 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:36.853 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:36.853 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:51:36.853 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:51:36.854 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 34 2024-09-11T09:51:36.854 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.854 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:36.854 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.854 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:36.854 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.897717+0000", 2024-09-11T09:51:36.854 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:36.854 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.854 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.854 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:36.854 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.897717+0000", 2024-09-11T09:51:36.854 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:36.854 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.855 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.855 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:36.855 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.897718+0000", 2024-09-11T09:51:36.855 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.1800000000000004e-07 2024-09-11T09:51:36.855 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.855 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.855 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:36.855 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.905339+0000", 2024-09-11T09:51:36.855 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0076210239999999997 2024-09-11T09:51:36.855 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.855 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.855 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:36.855 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.905366+0000", 2024-09-11T09:51:36.855 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.6959000000000001e-05 2024-09-11T09:51:36.856 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.856 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.856 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:36.856 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.905375+0000", 2024-09-11T09:51:36.856 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.516e-06 2024-09-11T09:51:36.856 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.856 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.856 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:36.856 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.905475+0000", 2024-09-11T09:51:36.856 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.9738e-05 2024-09-11T09:51:36.856 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.856 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.856 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:36.857 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.905660+0000", 2024-09-11T09:51:36.857 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00018554199999999999 2024-09-11T09:51:36.857 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.857 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.857 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:36.857 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.905808+0000", 2024-09-11T09:51:36.857 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000147386 2024-09-11T09:51:36.857 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.857 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.857 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:36.857 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.918230+0000", 2024-09-11T09:51:36.857 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.012421690000000001 2024-09-11T09:51:36.857 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.857 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.858 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:36.858 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.918932+0000", 2024-09-11T09:51:36.858 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00070267300000000001 2024-09-11T09:51:36.858 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.858 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.858 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:36.858 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.918989+0000", 2024-09-11T09:51:36.858 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.7268999999999997e-05 2024-09-11T09:51:36.858 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.858 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.858 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:36.858 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919040+0000", 2024-09-11T09:51:36.858 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.0946000000000003e-05 2024-09-11T09:51:36.859 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:36.859 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:36.859 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:36.859 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.859 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.859 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4688.0:31 2.1 e53/15 2:8654721b:::rbd_data.1250f5e25b80.0000000000000000:head v 53'2, mlcod=0'0)", 2024-09-11T09:51:36.859 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.904661+0000", 2024-09-11T09:51:36.859 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 13.947560015000001, 2024-09-11T09:51:36.859 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.014152804999999999, 2024-09-11T09:51:36.859 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:36.859 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:36.859 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:36.859 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:36.860 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.860 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:36.860 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.904661+0000", 2024-09-11T09:51:36.860 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:36.860 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.860 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.860 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:36.860 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.904661+0000", 2024-09-11T09:51:36.860 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:36.860 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.860 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.860 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:36.860 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.904662+0000", 2024-09-11T09:51:36.860 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0759999999999999e-06 2024-09-11T09:51:36.861 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.861 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.861 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:36.861 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.907298+0000", 2024-09-11T09:51:36.861 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0026361240000000001 2024-09-11T09:51:36.861 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.861 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.861 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:36.861 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.907320+0000", 2024-09-11T09:51:36.861 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.1268999999999999e-05 2024-09-11T09:51:36.861 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.861 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.861 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:36.861 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.907331+0000", 2024-09-11T09:51:36.862 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1031e-05 2024-09-11T09:51:36.862 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.862 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.862 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:36.862 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.907403+0000", 2024-09-11T09:51:36.862 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.2571999999999996e-05 2024-09-11T09:51:36.862 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.862 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.862 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:36.862 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.907428+0000", 2024-09-11T09:51:36.862 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.4380999999999999e-05 2024-09-11T09:51:36.862 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.862 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.862 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:36.863 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.918750+0000", 2024-09-11T09:51:36.863 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.011322211 2024-09-11T09:51:36.863 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.863 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.863 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:36.863 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.918814+0000", 2024-09-11T09:51:36.863 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.4140999999999995e-05 2024-09-11T09:51:36.863 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:36.863 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:36.863 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:36.863 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.863 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.863 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:36 2.6 2:61e1b442:::rbd_data.1250f5e25b80.0000000000000003:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:36.863 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.905404+0000", 2024-09-11T09:51:36.864 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 13.946817144000001, 2024-09-11T09:51:36.864 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.030912479999999999, 2024-09-11T09:51:36.864 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:36.864 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:36.864 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:36.864 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:36.864 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:51:36.864 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:51:36.864 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 36 2024-09-11T09:51:36.864 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.864 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:36.864 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.864 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:36.865 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.905404+0000", 2024-09-11T09:51:36.865 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:36.865 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.865 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.865 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:36.865 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.905404+0000", 2024-09-11T09:51:36.865 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:36.865 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.865 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.865 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:36.865 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.905405+0000", 2024-09-11T09:51:36.865 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2270000000000001e-06 2024-09-11T09:51:36.865 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.865 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.866 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:36.866 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.914859+0000", 2024-09-11T09:51:36.866 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0094541190000000004 2024-09-11T09:51:36.866 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.866 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.866 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:36.866 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.914879+0000", 2024-09-11T09:51:36.866 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.0052999999999999e-05 2024-09-11T09:51:36.866 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.866 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.866 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:36.866 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.914897+0000", 2024-09-11T09:51:36.866 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7331000000000001e-05 2024-09-11T09:51:36.866 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.867 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.867 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:36.867 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.915019+0000", 2024-09-11T09:51:36.867 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00012168 2024-09-11T09:51:36.867 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.867 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.867 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:36.867 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.915269+0000", 2024-09-11T09:51:36.867 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00025054899999999998 2024-09-11T09:51:36.867 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.867 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.867 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 3", 2024-09-11T09:51:36.867 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.915479+0000", 2024-09-11T09:51:36.867 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00021017799999999999 2024-09-11T09:51:36.868 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.868 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.868 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:36.868 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.926070+0000", 2024-09-11T09:51:36.868 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.010590624999999999 2024-09-11T09:51:36.868 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.868 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.868 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:36.868 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.936198+0000", 2024-09-11T09:51:36.868 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.010128194 2024-09-11T09:51:36.868 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.868 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.868 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:36.869 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.936263+0000", 2024-09-11T09:51:36.869 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.5366999999999999e-05 2024-09-11T09:51:36.869 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.869 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.869 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:36.869 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.936317+0000", 2024-09-11T09:51:36.869 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.3156999999999999e-05 2024-09-11T09:51:36.869 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:36.869 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:36.869 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:36.869 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.869 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.869 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:39 2.6 2:743e7616:::rbd_data.1250f5e25b80.0000000000000005:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:36.869 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.914960+0000", 2024-09-11T09:51:36.870 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 13.937260842000001, 2024-09-11T09:51:36.870 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.022371518, 2024-09-11T09:51:36.870 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:36.870 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:36.870 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:36.870 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:36.870 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:51:36.870 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:51:36.870 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 39 2024-09-11T09:51:36.870 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.870 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:36.870 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.870 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:36.870 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.914960+0000", 2024-09-11T09:51:36.871 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:36.871 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.871 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.871 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:36.871 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.914960+0000", 2024-09-11T09:51:36.871 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:36.871 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.871 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.871 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:36.871 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.914963+0000", 2024-09-11T09:51:36.871 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.1119999999999999e-06 2024-09-11T09:51:36.871 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.871 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.872 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:36.872 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.920417+0000", 2024-09-11T09:51:36.872 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0054543969999999997 2024-09-11T09:51:36.872 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.872 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.872 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:36.872 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.920430+0000", 2024-09-11T09:51:36.872 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3414000000000001e-05 2024-09-11T09:51:36.872 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.872 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.872 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:36.872 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.920437+0000", 2024-09-11T09:51:36.872 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.6159999999999998e-06 2024-09-11T09:51:36.872 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.873 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.873 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:36.873 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.921325+0000", 2024-09-11T09:51:36.873 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000887731 2024-09-11T09:51:36.873 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.873 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.873 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:36.873 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.921422+0000", 2024-09-11T09:51:36.873 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.7201999999999995e-05 2024-09-11T09:51:36.873 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.873 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.873 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 3", 2024-09-11T09:51:36.873 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.921511+0000", 2024-09-11T09:51:36.873 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.8880000000000003e-05 2024-09-11T09:51:36.874 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.874 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.874 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:36.874 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.928092+0000", 2024-09-11T09:51:36.874 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0065816290000000003 2024-09-11T09:51:36.874 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.874 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.874 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:36.874 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.937240+0000", 2024-09-11T09:51:36.874 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0091481070000000008 2024-09-11T09:51:36.874 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.874 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.874 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:36.874 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.937294+0000", 2024-09-11T09:51:36.875 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.3100000000000003e-05 2024-09-11T09:51:36.875 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.875 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.875 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:36.875 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.937332+0000", 2024-09-11T09:51:36.875 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.8330000000000001e-05 2024-09-11T09:51:36.875 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:36.875 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:36.875 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:36.875 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.875 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.875 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4688.0:32 2.2 e53/15 2:45941fd2:::rbd_data.1250f5e25b80.0000000000000001:head v 53'3, mlcod=17'1)", 2024-09-11T09:51:36.875 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.916509+0000", 2024-09-11T09:51:36.875 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 13.93571176, 2024-09-11T09:51:36.876 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.015885318999999998, 2024-09-11T09:51:36.876 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:36.876 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:36.876 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:36.876 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:36.876 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.876 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:36.876 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.916509+0000", 2024-09-11T09:51:36.876 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:36.876 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.876 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.876 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:36.876 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.916509+0000", 2024-09-11T09:51:36.876 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:36.877 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.877 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.877 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:36.877 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.916511+0000", 2024-09-11T09:51:36.877 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.052e-06 2024-09-11T09:51:36.877 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.877 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.877 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:36.877 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.927019+0000", 2024-09-11T09:51:36.877 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.010508224 2024-09-11T09:51:36.877 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.877 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.877 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:36.877 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.927038+0000", 2024-09-11T09:51:36.878 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.9590999999999999e-05 2024-09-11T09:51:36.878 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.878 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.878 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:36.878 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.927050+0000", 2024-09-11T09:51:36.878 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1439e-05 2024-09-11T09:51:36.878 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.878 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.878 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:36.878 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.927115+0000", 2024-09-11T09:51:36.878 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.5301000000000004e-05 2024-09-11T09:51:36.878 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.878 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.878 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:36.879 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.927132+0000", 2024-09-11T09:51:36.879 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7297000000000001e-05 2024-09-11T09:51:36.879 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.879 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.879 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:36.879 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.932354+0000", 2024-09-11T09:51:36.879 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0052211870000000004 2024-09-11T09:51:36.879 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.879 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.879 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:36.879 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.932395+0000", 2024-09-11T09:51:36.879 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.1227999999999999e-05 2024-09-11T09:51:36.879 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:36.879 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:36.880 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:36.880 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.880 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.880 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:44 2.6 2:67246a22:::rbd_data.1250f5e25b80.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:36.880 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.943181+0000", 2024-09-11T09:51:36.880 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 13.909040549, 2024-09-11T09:51:36.880 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.022629371999999998, 2024-09-11T09:51:36.880 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:36.880 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:36.880 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:36.880 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:36.880 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:51:36.881 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:51:36.881 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 44 2024-09-11T09:51:36.881 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.881 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:36.881 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.881 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:36.881 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.943181+0000", 2024-09-11T09:51:36.881 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:36.881 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.881 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.881 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:36.881 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.943181+0000", 2024-09-11T09:51:36.881 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:36.881 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.882 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.882 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:36.882 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.943182+0000", 2024-09-11T09:51:36.882 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.0500000000000002e-07 2024-09-11T09:51:36.882 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.882 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.882 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:36.882 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951240+0000", 2024-09-11T09:51:36.882 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.008058529 2024-09-11T09:51:36.882 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.882 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.882 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:36.882 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951260+0000", 2024-09-11T09:51:36.882 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.9570000000000001e-05 2024-09-11T09:51:36.883 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.883 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.883 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:36.883 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951270+0000", 2024-09-11T09:51:36.883 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0018e-05 2024-09-11T09:51:36.883 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.883 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.883 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:36.883 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951349+0000", 2024-09-11T09:51:36.883 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.8801000000000007e-05 2024-09-11T09:51:36.883 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.883 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.883 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:36.884 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951483+0000", 2024-09-11T09:51:36.884 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00013427 2024-09-11T09:51:36.884 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.884 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.884 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 3", 2024-09-11T09:51:36.884 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951593+0000", 2024-09-11T09:51:36.884 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00011031000000000001 2024-09-11T09:51:36.884 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.884 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.884 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:36.884 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.957303+0000", 2024-09-11T09:51:36.884 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0057097959999999996 2024-09-11T09:51:36.884 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.884 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.885 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:36.885 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.965690+0000", 2024-09-11T09:51:36.885 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0083867420000000008 2024-09-11T09:51:36.885 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.885 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.885 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:36.885 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.965749+0000", 2024-09-11T09:51:36.885 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.8981999999999999e-05 2024-09-11T09:51:36.885 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.885 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.885 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:36.885 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.965810+0000", 2024-09-11T09:51:36.885 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.1449000000000001e-05 2024-09-11T09:51:36.885 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:36.886 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:36.886 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:36.886 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.886 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.886 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:51 2.6 2:6b5894fc:::rbd_data.1250f5e25b80.000000000000000a:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:36.886 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.951324+0000", 2024-09-11T09:51:36.886 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 13.900897476000001, 2024-09-11T09:51:36.886 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.021375273, 2024-09-11T09:51:36.886 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:36.886 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:36.886 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:36.886 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:36.886 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:51:36.886 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:51:36.887 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 51 2024-09-11T09:51:36.887 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.887 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:36.887 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.887 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:36.887 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951324+0000", 2024-09-11T09:51:36.887 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:36.887 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.887 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.887 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:36.887 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951324+0000", 2024-09-11T09:51:36.887 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:36.887 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.888 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.888 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:36.888 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951324+0000", 2024-09-11T09:51:36.888 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.39e-07 2024-09-11T09:51:36.888 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.888 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.888 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:36.888 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.958537+0000", 2024-09-11T09:51:36.888 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0072126769999999998 2024-09-11T09:51:36.888 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.888 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.888 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:36.888 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.958554+0000", 2024-09-11T09:51:36.888 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7232e-05 2024-09-11T09:51:36.889 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.889 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.889 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:36.889 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.958570+0000", 2024-09-11T09:51:36.889 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.5747999999999999e-05 2024-09-11T09:51:36.889 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.889 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.889 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:36.889 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.958638+0000", 2024-09-11T09:51:36.889 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.7810000000000003e-05 2024-09-11T09:51:36.889 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.889 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.889 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:36.889 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.958758+0000", 2024-09-11T09:51:36.890 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000120689 2024-09-11T09:51:36.890 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.890 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.890 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 3", 2024-09-11T09:51:36.890 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.958865+0000", 2024-09-11T09:51:36.890 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00010692 2024-09-11T09:51:36.890 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.890 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.890 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:36.890 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.970712+0000", 2024-09-11T09:51:36.890 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.011846327 2024-09-11T09:51:36.890 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.890 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.891 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:36.891 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.972603+0000", 2024-09-11T09:51:36.891 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0018918610000000001 2024-09-11T09:51:36.891 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.891 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.891 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:36.891 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.972653+0000", 2024-09-11T09:51:36.891 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.9762999999999998e-05 2024-09-11T09:51:36.891 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.891 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.891 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:36.891 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.972699+0000", 2024-09-11T09:51:36.891 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.5806999999999997e-05 2024-09-11T09:51:36.891 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:36.892 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:36.892 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:36.892 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.892 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.892 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:52 2.4 2:26d5c2f0:::rbd_data.1250f5e25b80.000000000000000b:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:36.892 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.958596+0000", 2024-09-11T09:51:36.892 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 13.893625281, 2024-09-11T09:51:36.892 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.016477849999999999, 2024-09-11T09:51:36.892 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:36.892 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:36.892 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:36.892 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:36.892 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:51:36.892 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:51:36.893 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 52 2024-09-11T09:51:36.893 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.893 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:36.893 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.893 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:36.893 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.958596+0000", 2024-09-11T09:51:36.893 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:36.893 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.893 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.893 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:36.893 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.958596+0000", 2024-09-11T09:51:36.893 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:36.893 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.894 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.894 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:36.894 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.958597+0000", 2024-09-11T09:51:36.894 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1960000000000001e-06 2024-09-11T09:51:36.894 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.894 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.894 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:36.894 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.963157+0000", 2024-09-11T09:51:36.894 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0045603029999999999 2024-09-11T09:51:36.894 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.894 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.894 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:36.894 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.963171+0000", 2024-09-11T09:51:36.894 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3777999999999999e-05 2024-09-11T09:51:36.895 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.895 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.895 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:36.895 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.963181+0000", 2024-09-11T09:51:36.895 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.9010000000000006e-06 2024-09-11T09:51:36.895 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.895 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.895 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:36.895 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.963246+0000", 2024-09-11T09:51:36.895 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.5351000000000005e-05 2024-09-11T09:51:36.895 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.895 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.895 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:36.895 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.963419+0000", 2024-09-11T09:51:36.896 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00017299700000000001 2024-09-11T09:51:36.896 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.896 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.896 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:36.896 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.963570+0000", 2024-09-11T09:51:36.896 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00015066999999999999 2024-09-11T09:51:36.896 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.896 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.896 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:36.896 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.973881+0000", 2024-09-11T09:51:36.896 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.010311268 2024-09-11T09:51:36.896 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.896 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.897 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:36.897 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.974963+0000", 2024-09-11T09:51:36.897 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.001081493 2024-09-11T09:51:36.897 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.897 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.897 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:36.897 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.975012+0000", 2024-09-11T09:51:36.897 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.9471000000000001e-05 2024-09-11T09:51:36.897 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.897 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.897 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:36.897 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.975074+0000", 2024-09-11T09:51:36.897 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.1421999999999996e-05 2024-09-11T09:51:36.897 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:36.898 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:36.898 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:36.898 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.898 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.898 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4688.0:47 2.1 e53/15 2:95f53b3e:::rbd_data.1250f5e25b80.0000000000000008:head v 53'3, mlcod=53'1)", 2024-09-11T09:51:36.898 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.960861+0000", 2024-09-11T09:51:36.898 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 13.891359863, 2024-09-11T09:51:36.898 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.012346107, 2024-09-11T09:51:36.898 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:36.898 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:36.898 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:36.898 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:36.898 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.898 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:36.899 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.960861+0000", 2024-09-11T09:51:36.899 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:36.899 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.899 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.899 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:36.899 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.960861+0000", 2024-09-11T09:51:36.899 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:36.899 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.899 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.899 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:36.899 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.960863+0000", 2024-09-11T09:51:36.899 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7150000000000001e-06 2024-09-11T09:51:36.899 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.900 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.900 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:36.900 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.962428+0000", 2024-09-11T09:51:36.900 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.001564668 2024-09-11T09:51:36.900 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.900 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.900 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:36.900 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.962446+0000", 2024-09-11T09:51:36.900 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.8476999999999999e-05 2024-09-11T09:51:36.900 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.900 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.900 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:36.900 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.962458+0000", 2024-09-11T09:51:36.901 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1263999999999999e-05 2024-09-11T09:51:36.901 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.901 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.901 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:36.901 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.962575+0000", 2024-09-11T09:51:36.901 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000117025 2024-09-11T09:51:36.901 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.901 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.901 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:36.901 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.962601+0000", 2024-09-11T09:51:36.901 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.5984999999999999e-05 2024-09-11T09:51:36.901 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.901 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.901 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:36.902 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.973065+0000", 2024-09-11T09:51:36.902 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.010464753 2024-09-11T09:51:36.902 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.902 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.902 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:36.902 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.973207+0000", 2024-09-11T09:51:36.902 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00014222 2024-09-11T09:51:36.902 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:36.902 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:36.902 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:36.902 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.902 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.902 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:54 2.4 2.da8979c4 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:36.902 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.963207+0000", 2024-09-11T09:51:36.903 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 13.889014424999999, 2024-09-11T09:51:36.903 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.010641575, 2024-09-11T09:51:36.903 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:36.903 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:36.903 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "reached pg", 2024-09-11T09:51:36.903 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:36.903 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:51:36.903 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:51:36.903 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 54 2024-09-11T09:51:36.903 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.903 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:36.903 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.903 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:36.903 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.963207+0000", 2024-09-11T09:51:36.904 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:36.904 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.904 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.904 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:36.904 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.963207+0000", 2024-09-11T09:51:36.904 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:36.904 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.904 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.904 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:36.904 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.963208+0000", 2024-09-11T09:51:36.904 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2979999999999999e-06 2024-09-11T09:51:36.904 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.904 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.904 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:36.905 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.967419+0000", 2024-09-11T09:51:36.905 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0042111750000000002 2024-09-11T09:51:36.905 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.905 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.905 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:36.905 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.967439+0000", 2024-09-11T09:51:36.905 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.0129999999999999e-05 2024-09-11T09:51:36.905 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.905 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.905 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:36.905 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.967453+0000", 2024-09-11T09:51:36.905 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3781000000000001e-05 2024-09-11T09:51:36.905 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.905 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.906 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:36.906 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.973840+0000", 2024-09-11T09:51:36.906 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0063872269999999997 2024-09-11T09:51:36.906 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.906 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.906 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:36.906 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.973848+0000", 2024-09-11T09:51:36.906 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.9640000000000003e-06 2024-09-11T09:51:36.906 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:36.906 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:36.906 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:36.906 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.906 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.906 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:54 2.4 2:239e915b:::rbd_data.1250f5e25b80.000000000000000c:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:36.907 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.982401+0000", 2024-09-11T09:51:36.907 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 13.869820497999999, 2024-09-11T09:51:36.907 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.016817347, 2024-09-11T09:51:36.907 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:36.907 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:36.907 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:36.907 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:36.907 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:51:36.907 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:51:36.907 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 54 2024-09-11T09:51:36.907 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.907 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:36.907 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.907 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:36.908 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.982401+0000", 2024-09-11T09:51:36.908 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:36.908 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.908 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.908 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:36.908 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.982401+0000", 2024-09-11T09:51:36.908 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:36.908 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.909 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.909 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:36.909 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.982402+0000", 2024-09-11T09:51:36.909 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.8999999999999995e-07 2024-09-11T09:51:36.909 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.909 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.909 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:36.909 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.986535+0000", 2024-09-11T09:51:36.909 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0041333689999999996 2024-09-11T09:51:36.909 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.909 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.909 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:36.909 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.986555+0000", 2024-09-11T09:51:36.910 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.9851e-05 2024-09-11T09:51:36.910 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.910 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.910 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:36.910 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.986569+0000", 2024-09-11T09:51:36.910 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3784000000000001e-05 2024-09-11T09:51:36.910 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.910 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.910 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:36.910 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.986683+0000", 2024-09-11T09:51:36.910 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000114474 2024-09-11T09:51:36.910 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.910 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.911 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:36.911 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.986815+0000", 2024-09-11T09:51:36.911 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00013217799999999999 2024-09-11T09:51:36.911 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.911 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.911 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:36.911 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.986918+0000", 2024-09-11T09:51:36.911 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00010235599999999999 2024-09-11T09:51:36.911 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.911 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.911 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:36.911 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.998983+0000", 2024-09-11T09:51:36.911 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.012065352 2024-09-11T09:51:36.912 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.912 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.912 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:36.912 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.999081+0000", 2024-09-11T09:51:36.912 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.8091000000000007e-05 2024-09-11T09:51:36.912 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.912 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.912 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:36.912 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.999159+0000", 2024-09-11T09:51:36.912 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.7823000000000004e-05 2024-09-11T09:51:36.912 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.912 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.912 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:36.913 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.999218+0000", 2024-09-11T09:51:36.913 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.9178999999999997e-05 2024-09-11T09:51:36.913 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:36.913 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:36.913 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:36.913 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.913 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.913 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:56 2.4 2:2167012f:::rbd_data.1250f5e25b80.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:36.913 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.986661+0000", 2024-09-11T09:51:36.913 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 13.865560045, 2024-09-11T09:51:36.913 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.017353539000000001, 2024-09-11T09:51:36.913 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:36.913 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:36.914 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:36.914 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:36.914 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:51:36.914 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:51:36.914 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 56 2024-09-11T09:51:36.914 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.914 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:36.914 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.914 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:36.914 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.986661+0000", 2024-09-11T09:51:36.914 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:36.914 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.914 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.915 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:36.915 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.986661+0000", 2024-09-11T09:51:36.915 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:36.915 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.915 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.915 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:36.915 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.986663+0000", 2024-09-11T09:51:36.915 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.8989999999999999e-06 2024-09-11T09:51:36.915 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.915 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.915 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:36.915 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.990444+0000", 2024-09-11T09:51:36.915 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0037808569999999999 2024-09-11T09:51:36.915 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.916 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.916 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:36.916 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.990462+0000", 2024-09-11T09:51:36.916 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.8450000000000001e-05 2024-09-11T09:51:36.916 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.916 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.916 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:36.916 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.990476+0000", 2024-09-11T09:51:36.916 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3565e-05 2024-09-11T09:51:36.916 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.916 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.916 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:36.917 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.993582+0000", 2024-09-11T09:51:36.917 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0031059109999999998 2024-09-11T09:51:36.917 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.917 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.917 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:36.917 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.993686+0000", 2024-09-11T09:51:36.917 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000104424 2024-09-11T09:51:36.917 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.917 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.917 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:36.917 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.993799+0000", 2024-09-11T09:51:36.917 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000112985 2024-09-11T09:51:36.917 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.917 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.918 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:36.918 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.002449+0000", 2024-09-11T09:51:36.918 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.008650151 2024-09-11T09:51:36.918 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.918 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.918 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:36.918 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.003948+0000", 2024-09-11T09:51:36.918 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.001498636 2024-09-11T09:51:36.918 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.918 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.918 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:36.918 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.003985+0000", 2024-09-11T09:51:36.918 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.7054999999999997e-05 2024-09-11T09:51:36.919 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.919 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.919 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:36.919 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.004015+0000", 2024-09-11T09:51:36.919 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.9606000000000001e-05 2024-09-11T09:51:36.919 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:36.919 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:36.919 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:36.919 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.919 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.919 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:58 2.6 2:6277c644:::rbd_data.1250f5e25b80.000000000000000e:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:36.919 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.990540+0000", 2024-09-11T09:51:36.919 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 13.86168125, 2024-09-11T09:51:36.919 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.016232068999999998, 2024-09-11T09:51:36.920 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:36.920 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:36.920 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:36.920 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:36.920 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:51:36.920 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:51:36.920 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 58 2024-09-11T09:51:36.920 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.920 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:36.920 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.920 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:36.920 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.990540+0000", 2024-09-11T09:51:36.921 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:36.921 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.921 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.921 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:36.921 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.990540+0000", 2024-09-11T09:51:36.921 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:36.921 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.921 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.921 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:36.921 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.990541+0000", 2024-09-11T09:51:36.921 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2359999999999999e-06 2024-09-11T09:51:36.921 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.921 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.922 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:36.922 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.994626+0000", 2024-09-11T09:51:36.922 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.004084865 2024-09-11T09:51:36.922 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.922 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.922 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:36.922 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.994644+0000", 2024-09-11T09:51:36.922 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7866000000000001e-05 2024-09-11T09:51:36.922 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.922 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.922 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:36.922 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.994653+0000", 2024-09-11T09:51:36.922 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.2159999999999995e-06 2024-09-11T09:51:36.922 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.923 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.923 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:36.923 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.994732+0000", 2024-09-11T09:51:36.923 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.8954000000000003e-05 2024-09-11T09:51:36.923 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.923 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.923 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:36.923 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.994906+0000", 2024-09-11T09:51:36.923 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0001741 2024-09-11T09:51:36.923 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.923 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.923 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 3", 2024-09-11T09:51:36.923 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.995083+0000", 2024-09-11T09:51:36.924 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00017637100000000001 2024-09-11T09:51:36.924 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.924 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.924 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:36.924 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.003066+0000", 2024-09-11T09:51:36.924 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0079833360000000006 2024-09-11T09:51:36.924 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.924 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.924 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:36.924 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.006634+0000", 2024-09-11T09:51:36.924 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.003568309 2024-09-11T09:51:36.924 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.924 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.925 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:36.925 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.006716+0000", 2024-09-11T09:51:36.925 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.1377000000000004e-05 2024-09-11T09:51:36.925 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.925 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.925 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:36.925 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.006772+0000", 2024-09-11T09:51:36.925 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.6439e-05 2024-09-11T09:51:36.925 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:36.925 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:36.925 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:36.925 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.925 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.926 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:59 2.4 2:2906e6a7:::rbd_header.1250f5e25b80:head [watch reconnect cookie 140424358411408 gen 2] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:36.926 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.994684+0000", 2024-09-11T09:51:36.926 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 13.857537619, 2024-09-11T09:51:36.926 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0094134349999999995, 2024-09-11T09:51:36.926 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:36.926 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:36.926 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "started", 2024-09-11T09:51:36.926 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:36.926 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:51:36.926 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:51:36.926 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 59 2024-09-11T09:51:36.926 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.926 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:36.926 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.927 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:36.927 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.994684+0000", 2024-09-11T09:51:36.927 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:36.927 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.927 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.927 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:36.927 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.994684+0000", 2024-09-11T09:51:36.927 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:36.927 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.927 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.927 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:36.927 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.994686+0000", 2024-09-11T09:51:36.927 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.8159999999999998e-06 2024-09-11T09:51:36.928 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.928 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.928 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:36.928 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.994691+0000", 2024-09-11T09:51:36.928 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.5009999999999998e-06 2024-09-11T09:51:36.928 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.928 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.928 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:36.928 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.994698+0000", 2024-09-11T09:51:36.928 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.3039999999999999e-06 2024-09-11T09:51:36.928 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.928 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.929 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:36.929 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.994703+0000", 2024-09-11T09:51:36.929 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.5190000000000003e-06 2024-09-11T09:51:36.929 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.929 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.929 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:36.929 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.998678+0000", 2024-09-11T09:51:36.929 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0039752340000000002 2024-09-11T09:51:36.929 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.929 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.929 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:36.929 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.998748+0000", 2024-09-11T09:51:36.929 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.0114e-05 2024-09-11T09:51:36.930 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.930 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.930 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:36.930 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.004097+0000", 2024-09-11T09:51:36.930 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0053489469999999997 2024-09-11T09:51:36.930 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:36.930 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:36.930 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:36.930 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.930 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.931 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4694.0:33 2.4 2:3bfc818c:ns::rbd_data.1256a0c02879.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:36.931 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.084537+0000", 2024-09-11T09:51:36.931 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 13.767684191000001, 2024-09-11T09:51:36.931 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.022075773999999999, 2024-09-11T09:51:36.931 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:36.931 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:36.931 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:36.931 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:36.931 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4694", 2024-09-11T09:51:36.931 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:51:36.931 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 33 2024-09-11T09:51:36.931 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.931 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:36.932 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.932 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:36.932 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.084537+0000", 2024-09-11T09:51:36.932 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:36.932 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.932 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.932 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:36.932 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.084537+0000", 2024-09-11T09:51:36.932 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:36.932 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.932 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.932 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:36.933 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.084538+0000", 2024-09-11T09:51:36.933 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.5000000000000001e-07 2024-09-11T09:51:36.933 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.933 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.933 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:36.933 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.095984+0000", 2024-09-11T09:51:36.933 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.011446543 2024-09-11T09:51:36.933 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.933 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.933 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:36.933 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.096014+0000", 2024-09-11T09:51:36.933 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.9745000000000001e-05 2024-09-11T09:51:36.933 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.934 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.934 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:36.934 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.096030+0000", 2024-09-11T09:51:36.934 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6152999999999999e-05 2024-09-11T09:51:36.934 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.934 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.934 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:36.934 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.096113+0000", 2024-09-11T09:51:36.934 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.2731999999999993e-05 2024-09-11T09:51:36.934 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.934 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.934 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:36.934 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.096273+0000", 2024-09-11T09:51:36.935 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00015973699999999999 2024-09-11T09:51:36.935 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.935 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.935 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:36.935 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.096394+0000", 2024-09-11T09:51:36.935 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000120883 2024-09-11T09:51:36.935 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.935 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.935 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:36.935 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.106048+0000", 2024-09-11T09:51:36.935 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0096548020000000005 2024-09-11T09:51:36.935 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.935 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.935 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:36.936 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.106531+0000", 2024-09-11T09:51:36.936 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00048264299999999999 2024-09-11T09:51:36.936 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.936 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.936 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:36.936 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.106565+0000", 2024-09-11T09:51:36.936 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.4343e-05 2024-09-11T09:51:36.936 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.936 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.936 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:36.936 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.106613+0000", 2024-09-11T09:51:36.936 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.7342999999999997e-05 2024-09-11T09:51:36.936 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:36.937 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:36.937 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:36.937 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.937 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.937 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4694.0:32 2.1 e53/15 2:8a862b48:ns::rbd_data.1256a0c02879.0000000000000000:head v 53'8, mlcod=53'6)", 2024-09-11T09:51:36.937 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.091418+0000", 2024-09-11T09:51:36.937 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 13.760803021999999, 2024-09-11T09:51:36.937 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.013027372000000001, 2024-09-11T09:51:36.937 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:36.937 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:36.937 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:36.937 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:36.937 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.938 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:36.938 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.091418+0000", 2024-09-11T09:51:36.938 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:36.938 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.938 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.938 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:36.938 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.091418+0000", 2024-09-11T09:51:36.938 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:36.938 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.938 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.938 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:36.938 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.091420+0000", 2024-09-11T09:51:36.938 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6899999999999999e-06 2024-09-11T09:51:36.939 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.939 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.939 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:36.939 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.093086+0000", 2024-09-11T09:51:36.939 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0016662510000000001 2024-09-11T09:51:36.939 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.939 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.939 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:36.939 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.093107+0000", 2024-09-11T09:51:36.939 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.0999e-05 2024-09-11T09:51:36.939 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.939 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.939 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:36.940 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.093120+0000", 2024-09-11T09:51:36.940 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2544999999999999e-05 2024-09-11T09:51:36.940 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.940 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.940 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:36.940 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.093186+0000", 2024-09-11T09:51:36.940 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.6595999999999997e-05 2024-09-11T09:51:36.940 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.940 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.940 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:36.940 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.093208+0000", 2024-09-11T09:51:36.940 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.1624000000000002e-05 2024-09-11T09:51:36.940 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.941 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.941 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:36.941 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.104392+0000", 2024-09-11T09:51:36.941 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.011184265000000001 2024-09-11T09:51:36.941 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.941 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.941 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:36.941 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.104446+0000", 2024-09-11T09:51:36.941 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.3402000000000001e-05 2024-09-11T09:51:36.941 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:36.941 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:36.941 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:36.941 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.942 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.942 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4694.0:45 2.6 2:671fb830:ns::rbd_data.1256a0c02879.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:36.942 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.109629+0000", 2024-09-11T09:51:36.942 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 13.742592180999999, 2024-09-11T09:51:36.942 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.036403143999999998, 2024-09-11T09:51:36.942 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:36.942 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:36.942 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:36.942 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:36.942 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4694", 2024-09-11T09:51:36.942 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:51:36.942 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 45 2024-09-11T09:51:36.942 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.942 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:36.943 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.943 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:36.943 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.109629+0000", 2024-09-11T09:51:36.943 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:36.943 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.943 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.943 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:36.943 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.109629+0000", 2024-09-11T09:51:36.943 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:36.943 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.943 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.943 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:36.943 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.109630+0000", 2024-09-11T09:51:36.944 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1200000000000001e-06 2024-09-11T09:51:36.944 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.944 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.944 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:36.944 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.121370+0000", 2024-09-11T09:51:36.944 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.011739571000000001 2024-09-11T09:51:36.944 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.944 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.944 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:36.944 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.121391+0000", 2024-09-11T09:51:36.944 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.1175e-05 2024-09-11T09:51:36.944 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.944 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.945 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:36.945 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.121404+0000", 2024-09-11T09:51:36.945 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2807e-05 2024-09-11T09:51:36.945 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.945 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.945 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:36.945 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.121498+0000", 2024-09-11T09:51:36.945 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.3770999999999994e-05 2024-09-11T09:51:36.945 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.945 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.945 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:36.945 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.121612+0000", 2024-09-11T09:51:36.945 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000114666 2024-09-11T09:51:36.946 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.946 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.946 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 3", 2024-09-11T09:51:36.946 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.121711+0000", 2024-09-11T09:51:36.946 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.8647999999999995e-05 2024-09-11T09:51:36.946 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.946 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.946 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:36.946 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.126558+0000", 2024-09-11T09:51:36.946 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0048473769999999999 2024-09-11T09:51:36.946 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.946 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.946 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:36.947 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.145912+0000", 2024-09-11T09:51:36.947 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.019354281000000001 2024-09-11T09:51:36.947 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.947 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.947 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:36.947 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.145977+0000", 2024-09-11T09:51:36.947 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.4135999999999997e-05 2024-09-11T09:51:36.947 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.947 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.947 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:36.947 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.146032+0000", 2024-09-11T09:51:36.947 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.5591999999999999e-05 2024-09-11T09:51:36.947 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:36.947 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:36.948 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:36.948 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.948 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.948 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4694.0:41 2.2 e53/15 2:5e2505c4:ns::rbd_data.1256a0c02879.0000000000000003:head v 53'4, mlcod=18'2)", 2024-09-11T09:51:36.948 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.113199+0000", 2024-09-11T09:51:36.948 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 13.739021771000001, 2024-09-11T09:51:36.948 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.011404512, 2024-09-11T09:51:36.948 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:36.948 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:36.948 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:36.948 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:36.948 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.949 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:36.949 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.113199+0000", 2024-09-11T09:51:36.949 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:36.949 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.949 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.949 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:36.949 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.113199+0000", 2024-09-11T09:51:36.949 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:36.949 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.949 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.949 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:36.949 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.113200+0000", 2024-09-11T09:51:36.949 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.2799999999999995e-07 2024-09-11T09:51:36.949 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.950 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.950 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:36.950 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.118872+0000", 2024-09-11T09:51:36.950 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0056718089999999999 2024-09-11T09:51:36.950 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.950 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.950 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:36.950 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.118889+0000", 2024-09-11T09:51:36.950 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7428999999999999e-05 2024-09-11T09:51:36.950 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.950 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.950 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:36.950 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.118900+0000", 2024-09-11T09:51:36.951 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0243e-05 2024-09-11T09:51:36.951 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.951 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.951 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:36.951 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.118959+0000", 2024-09-11T09:51:36.951 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.9364000000000003e-05 2024-09-11T09:51:36.951 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.951 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.951 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:36.951 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.118976+0000", 2024-09-11T09:51:36.951 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7124000000000001e-05 2024-09-11T09:51:36.951 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.951 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.952 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:36.952 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.124557+0000", 2024-09-11T09:51:36.952 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0055812659999999997 2024-09-11T09:51:36.952 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.952 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.952 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:36.952 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.124604+0000", 2024-09-11T09:51:36.952 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.6548999999999998e-05 2024-09-11T09:51:36.952 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:36.952 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:36.952 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:36.952 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.952 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.953 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4694.0:43 2.0 e53/15 2:16c8ae84:ns::rbd_data.1256a0c02879.0000000000000005:head v 53'2, mlcod=0'0)", 2024-09-11T09:51:36.953 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.158559+0000", 2024-09-11T09:51:36.953 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 13.693662444999999, 2024-09-11T09:51:36.953 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.016713677999999999, 2024-09-11T09:51:36.953 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:36.953 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:36.953 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:36.953 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:36.953 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.953 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:36.953 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.158559+0000", 2024-09-11T09:51:36.953 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:36.953 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.954 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.954 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:36.954 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.158559+0000", 2024-09-11T09:51:36.954 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:36.954 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.954 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.954 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:36.954 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.158560+0000", 2024-09-11T09:51:36.954 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1459999999999999e-06 2024-09-11T09:51:36.954 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.954 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.954 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:36.954 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.166465+0000", 2024-09-11T09:51:36.955 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.007904767 2024-09-11T09:51:36.955 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.955 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.955 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:36.955 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.166491+0000", 2024-09-11T09:51:36.955 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.6308e-05 2024-09-11T09:51:36.955 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.955 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.955 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:36.955 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.166502+0000", 2024-09-11T09:51:36.955 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0736e-05 2024-09-11T09:51:36.955 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.955 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.955 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:36.956 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.166564+0000", 2024-09-11T09:51:36.956 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.2578999999999998e-05 2024-09-11T09:51:36.956 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.956 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.956 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:36.956 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.166585+0000", 2024-09-11T09:51:36.956 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.0395999999999999e-05 2024-09-11T09:51:36.956 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.956 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.956 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:36.956 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.175227+0000", 2024-09-11T09:51:36.956 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0086426420000000007 2024-09-11T09:51:36.956 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.957 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.957 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:36.957 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.175272+0000", 2024-09-11T09:51:36.957 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.5104e-05 2024-09-11T09:51:36.957 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:36.957 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:36.957 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:36.957 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.957 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.957 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4694.0:59 2.2 e53/15 2:41adefc9:ns::rbd_data.1256a0c02879.000000000000000e:head v 53'5, mlcod=53'3)", 2024-09-11T09:51:36.957 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.163389+0000", 2024-09-11T09:51:36.957 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 13.688832468999999, 2024-09-11T09:51:36.957 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.013095719, 2024-09-11T09:51:36.957 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:36.958 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:36.958 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:36.958 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:36.958 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.958 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:36.958 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.163389+0000", 2024-09-11T09:51:36.958 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:36.958 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.958 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.958 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:36.958 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.163389+0000", 2024-09-11T09:51:36.958 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:36.958 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.959 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.959 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:36.959 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.163390+0000", 2024-09-11T09:51:36.959 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.336e-06 2024-09-11T09:51:36.959 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.959 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.959 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:36.959 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.170335+0000", 2024-09-11T09:51:36.959 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0069451080000000002 2024-09-11T09:51:36.959 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.959 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.959 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:36.959 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.170354+0000", 2024-09-11T09:51:36.960 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.8423e-05 2024-09-11T09:51:36.960 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.960 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.960 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:36.960 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.170366+0000", 2024-09-11T09:51:36.960 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2235999999999999e-05 2024-09-11T09:51:36.960 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.960 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.960 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:36.960 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.170439+0000", 2024-09-11T09:51:36.960 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.2989000000000002e-05 2024-09-11T09:51:36.960 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.960 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.960 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:36.961 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.170460+0000", 2024-09-11T09:51:36.961 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.1211000000000001e-05 2024-09-11T09:51:36.961 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.961 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.961 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:36.961 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.176442+0000", 2024-09-11T09:51:36.961 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0059821969999999999 2024-09-11T09:51:36.961 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:36.961 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:36.961 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:36.961 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.176484+0000", 2024-09-11T09:51:36.961 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.2218999999999997e-05 2024-09-11T09:51:36.961 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:36.962 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:36.962 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:36.962 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:36.962 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:36.962 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:51:36.963 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-11T09:51:36.964 INFO:tasks.ceph.osd.0.smithi017.stderr:2024-09-11T09:51:36.963+0000 7fb73c0f1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31665) UID: 0 2024-09-11T09:51:37.064 INFO:tasks.ceph.osd.2.smithi017.stderr:2024-09-11T09:51:37.064+0000 7fd20b170640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31672) UID: 0 2024-09-11T09:51:37.112 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:51:37.113 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [], 2024-09-11T09:51:37.113 INFO:teuthology.orchestra.run.smithi017.stdout: "num_ops": 0 2024-09-11T09:51:37.113 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:51:37.121 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-11T09:51:37.165 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-09-11T09:51:37.166+0000 7f7b6c7d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31904) UID: 0 2024-09-11T09:51:37.242 INFO:tasks.workunit.client.0.smithi107.stderr:++ _sudo rbd --device-type nbd map --read-only rbd/testrbdnbd27121 2024-09-11T09:51:37.242 INFO:tasks.workunit.client.0.smithi107.stderr:++ local cmd 2024-09-11T09:51:37.242 INFO:tasks.workunit.client.0.smithi107.stderr:+++ id -u 2024-09-11T09:51:37.242 INFO:tasks.workunit.client.0.smithi107.stderr:++ '[' 1000 -eq 0 ']' 2024-09-11T09:51:37.243 INFO:tasks.workunit.client.0.smithi107.stderr:+++ which rbd 2024-09-11T09:51:37.245 INFO:tasks.workunit.client.0.smithi107.stderr:++ cmd=/usr/bin/rbd 2024-09-11T09:51:37.245 INFO:tasks.workunit.client.0.smithi107.stderr:++ shift 2024-09-11T09:51:37.245 INFO:tasks.workunit.client.0.smithi107.stderr:++ sudo -nE /usr/bin/rbd --device-type nbd map --read-only rbd/testrbdnbd27121 2024-09-11T09:51:37.265 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-09-11T09:51:37.267+0000 7f7b6c7d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31904) UID: 0 2024-09-11T09:51:37.269 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:51:37.269 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [], 2024-09-11T09:51:37.269 INFO:teuthology.orchestra.run.smithi017.stdout: "complaint_time": 30, 2024-09-11T09:51:37.269 INFO:teuthology.orchestra.run.smithi017.stdout: "num_blocked_ops": 0 2024-09-11T09:51:37.269 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:51:37.278 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-11T09:51:37.353 INFO:tasks.workunit.client.0.smithi107.stderr:+ DEV=/dev/nbd0 2024-09-11T09:51:37.353 INFO:tasks.workunit.client.0.smithi107.stderr:++ rbd device --device-type nbd list 2024-09-11T09:51:37.354 INFO:tasks.workunit.client.0.smithi107.stderr:++ awk -v pool=rbd -v img=testrbdnbd27121 -v dev=/dev/nbd0 '$2 == pool && $3 == img && $5 == dev {print $1}' 2024-09-11T09:51:37.366 INFO:tasks.ceph.osd.0.smithi017.stderr:2024-09-11T09:51:37.366+0000 7fb73c0f1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31665) UID: 0 2024-09-11T09:51:37.389 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:51:37.389 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 20, 2024-09-11T09:51:37.389 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 600, 2024-09-11T09:51:37.389 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [ 2024-09-11T09:51:37.389 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.390 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:6 2.1 2:92878cd2:::rbd_id.testrbdnbd27121:head [create,call rbd.set_id in=9b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:37.390 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.873158+0000", 2024-09-11T09:51:37.390 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 14.51620864, 2024-09-11T09:51:37.390 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.001212411, 2024-09-11T09:51:37.390 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:37.390 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:37.390 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:37.390 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:37.390 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:51:37.390 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:51:37.390 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 6 2024-09-11T09:51:37.390 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.390 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:37.391 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.391 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:37.391 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.873158+0000", 2024-09-11T09:51:37.391 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:37.391 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.391 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.391 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:37.391 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.873158+0000", 2024-09-11T09:51:37.391 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:37.391 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.391 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.391 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:37.391 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.873159+0000", 2024-09-11T09:51:37.392 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.6400000000000001e-07 2024-09-11T09:51:37.392 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.392 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.392 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:37.392 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.873164+0000", 2024-09-11T09:51:37.392 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.7419999999999997e-06 2024-09-11T09:51:37.392 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.392 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.392 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:37.392 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.873175+0000", 2024-09-11T09:51:37.392 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0774999999999999e-05 2024-09-11T09:51:37.392 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.392 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.393 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:37.393 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.873182+0000", 2024-09-11T09:51:37.393 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.5240000000000003e-06 2024-09-11T09:51:37.393 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.393 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.393 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:37.393 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.873223+0000", 2024-09-11T09:51:37.393 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.0479000000000003e-05 2024-09-11T09:51:37.393 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.393 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.393 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:37.393 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.873299+0000", 2024-09-11T09:51:37.393 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.6774999999999996e-05 2024-09-11T09:51:37.393 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.394 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.394 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1", 2024-09-11T09:51:37.394 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.873412+0000", 2024-09-11T09:51:37.394 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000112489 2024-09-11T09:51:37.394 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.394 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.394 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:37.394 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.874213+0000", 2024-09-11T09:51:37.394 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00080068299999999995 2024-09-11T09:51:37.394 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.394 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.394 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:37.394 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.874311+0000", 2024-09-11T09:51:37.395 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.8301999999999995e-05 2024-09-11T09:51:37.395 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.395 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.395 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:37.395 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.874344+0000", 2024-09-11T09:51:37.395 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.3441000000000001e-05 2024-09-11T09:51:37.395 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.395 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.395 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:37.395 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.874371+0000", 2024-09-11T09:51:37.395 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.6437000000000001e-05 2024-09-11T09:51:37.395 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:37.395 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:37.395 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:37.396 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.396 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.396 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:31 2.1 2:8654721b:::rbd_data.1250f5e25b80.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:37.396 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.896024+0000", 2024-09-11T09:51:37.396 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 14.493343032, 2024-09-11T09:51:37.396 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.023148401999999998, 2024-09-11T09:51:37.396 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:37.396 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:37.396 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:37.396 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:37.396 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:51:37.396 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:51:37.397 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 31 2024-09-11T09:51:37.397 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.397 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:37.397 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.397 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:37.397 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.896024+0000", 2024-09-11T09:51:37.397 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:37.397 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.397 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.397 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:37.397 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.896024+0000", 2024-09-11T09:51:37.397 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:37.397 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.397 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.398 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:37.398 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.896025+0000", 2024-09-11T09:51:37.398 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.54e-07 2024-09-11T09:51:37.398 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.398 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.398 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:37.398 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.904287+0000", 2024-09-11T09:51:37.398 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0082624729999999993 2024-09-11T09:51:37.398 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.398 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.398 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:37.398 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.904305+0000", 2024-09-11T09:51:37.398 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7912999999999999e-05 2024-09-11T09:51:37.398 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.399 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.399 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:37.399 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.904312+0000", 2024-09-11T09:51:37.399 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.0230000000000004e-06 2024-09-11T09:51:37.399 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.399 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.399 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:37.399 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.904395+0000", 2024-09-11T09:51:37.399 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.2355999999999996e-05 2024-09-11T09:51:37.399 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.399 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.399 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:37.399 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.904499+0000", 2024-09-11T09:51:37.400 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000104672 2024-09-11T09:51:37.400 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.400 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.400 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1", 2024-09-11T09:51:37.400 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.904594+0000", 2024-09-11T09:51:37.400 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.5129000000000003e-05 2024-09-11T09:51:37.400 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.400 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.400 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:37.400 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.918183+0000", 2024-09-11T09:51:37.400 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.013588355999999999 2024-09-11T09:51:37.400 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.400 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.401 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:37.401 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919053+0000", 2024-09-11T09:51:37.401 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00087066800000000001 2024-09-11T09:51:37.401 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.401 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.401 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:37.401 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919125+0000", 2024-09-11T09:51:37.401 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.1333000000000003e-05 2024-09-11T09:51:37.401 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.401 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.402 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:37.402 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919172+0000", 2024-09-11T09:51:37.402 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.7725000000000001e-05 2024-09-11T09:51:37.402 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:37.402 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:37.402 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:37.402 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.402 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.402 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4688.0:42 2.3 e53/15 2:c54742d8:::rbd_data.1250f5e25b80.0000000000000006:head v 53'2, mlcod=0'0)", 2024-09-11T09:51:37.402 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.939272+0000", 2024-09-11T09:51:37.402 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 14.450095048, 2024-09-11T09:51:37.402 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.010821958, 2024-09-11T09:51:37.402 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:37.402 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:37.403 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:37.403 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:37.403 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.403 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:37.403 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.939272+0000", 2024-09-11T09:51:37.403 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:37.403 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.403 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.403 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:37.403 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.939272+0000", 2024-09-11T09:51:37.403 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:37.403 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.403 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.403 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:37.404 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.939273+0000", 2024-09-11T09:51:37.404 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.7899999999999997e-07 2024-09-11T09:51:37.404 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.404 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.404 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:37.404 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.944403+0000", 2024-09-11T09:51:37.404 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0051302129999999998 2024-09-11T09:51:37.404 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.404 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.404 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:37.404 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.944413+0000", 2024-09-11T09:51:37.404 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0182e-05 2024-09-11T09:51:37.405 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.405 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.405 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:37.405 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.944420+0000", 2024-09-11T09:51:37.405 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.7000000000000002e-06 2024-09-11T09:51:37.405 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.405 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.405 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:37.405 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.944482+0000", 2024-09-11T09:51:37.405 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.1911999999999999e-05 2024-09-11T09:51:37.405 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.405 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.405 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:37.405 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.944508+0000", 2024-09-11T09:51:37.406 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.6582000000000001e-05 2024-09-11T09:51:37.406 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.406 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.406 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:37.406 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.950050+0000", 2024-09-11T09:51:37.406 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0055414080000000003 2024-09-11T09:51:37.406 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.406 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.406 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:37.406 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.950094+0000", 2024-09-11T09:51:37.406 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.4082000000000002e-05 2024-09-11T09:51:37.406 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:37.406 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:37.406 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:37.407 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.407 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.407 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:47 2.1 2:95f53b3e:::rbd_data.1250f5e25b80.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:37.407 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.948330+0000", 2024-09-11T09:51:37.407 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 14.441036704, 2024-09-11T09:51:37.407 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.025156603999999999, 2024-09-11T09:51:37.407 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:37.407 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:37.407 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:37.407 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:37.407 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:51:37.407 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:51:37.407 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 47 2024-09-11T09:51:37.408 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.408 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:37.408 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.408 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:37.408 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.948330+0000", 2024-09-11T09:51:37.408 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:37.408 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.408 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.408 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:37.408 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.948330+0000", 2024-09-11T09:51:37.408 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:37.408 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.408 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.409 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:37.409 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.948333+0000", 2024-09-11T09:51:37.409 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.481e-06 2024-09-11T09:51:37.409 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.409 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.409 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:37.409 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.960263+0000", 2024-09-11T09:51:37.409 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.011930671 2024-09-11T09:51:37.409 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.409 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.409 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:37.409 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.960289+0000", 2024-09-11T09:51:37.409 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.5573000000000001e-05 2024-09-11T09:51:37.409 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.410 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.410 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:37.410 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.960304+0000", 2024-09-11T09:51:37.410 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.4647000000000001e-05 2024-09-11T09:51:37.410 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.410 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.410 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:37.410 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.960390+0000", 2024-09-11T09:51:37.410 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.6088999999999995e-05 2024-09-11T09:51:37.410 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.410 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.410 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:37.410 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.960562+0000", 2024-09-11T09:51:37.411 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00017237400000000001 2024-09-11T09:51:37.411 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.411 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.411 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1", 2024-09-11T09:51:37.411 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.960725+0000", 2024-09-11T09:51:37.411 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00016321599999999999 2024-09-11T09:51:37.411 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.411 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.411 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:37.411 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.969465+0000", 2024-09-11T09:51:37.411 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0087396899999999996 2024-09-11T09:51:37.411 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.411 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.411 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:37.412 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.973393+0000", 2024-09-11T09:51:37.412 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0039274669999999996 2024-09-11T09:51:37.412 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.412 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.412 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:37.412 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.973443+0000", 2024-09-11T09:51:37.412 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.0340000000000003e-05 2024-09-11T09:51:37.412 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.412 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.412 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:37.412 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.973487+0000", 2024-09-11T09:51:37.412 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.4056e-05 2024-09-11T09:51:37.412 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:37.413 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:37.413 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:37.413 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.413 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.413 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:61 2.1 2:9af038b1:::rbd_data.1250f5e25b80.000000000000000f:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:37.413 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.975764+0000", 2024-09-11T09:51:37.413 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 14.413603180000001, 2024-09-11T09:51:37.413 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.016158891000000002, 2024-09-11T09:51:37.413 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:37.413 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:37.413 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:37.413 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:37.413 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:51:37.414 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:51:37.414 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 61 2024-09-11T09:51:37.414 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.414 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:37.414 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.414 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:37.414 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.975764+0000", 2024-09-11T09:51:37.414 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:37.414 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.414 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.414 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:37.414 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.975764+0000", 2024-09-11T09:51:37.415 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:37.415 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.415 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.415 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:37.415 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.975766+0000", 2024-09-11T09:51:37.415 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.767e-06 2024-09-11T09:51:37.415 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.415 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.415 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:37.415 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.983204+0000", 2024-09-11T09:51:37.415 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0074388400000000004 2024-09-11T09:51:37.415 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.415 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.416 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:37.416 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.983226+0000", 2024-09-11T09:51:37.416 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.1279000000000001e-05 2024-09-11T09:51:37.416 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.416 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.416 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:37.416 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.983238+0000", 2024-09-11T09:51:37.416 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2563e-05 2024-09-11T09:51:37.416 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.416 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.416 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:37.416 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.983318+0000", 2024-09-11T09:51:37.416 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.0075999999999997e-05 2024-09-11T09:51:37.417 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.417 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.417 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:37.417 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.983429+0000", 2024-09-11T09:51:37.417 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000110618 2024-09-11T09:51:37.417 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.417 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.417 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1", 2024-09-11T09:51:37.417 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.983525+0000", 2024-09-11T09:51:37.417 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.603e-05 2024-09-11T09:51:37.417 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.417 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.417 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:37.417 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.989004+0000", 2024-09-11T09:51:37.418 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0054785429999999998 2024-09-11T09:51:37.418 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.418 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.418 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:37.418 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.991788+0000", 2024-09-11T09:51:37.418 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0027845729999999998 2024-09-11T09:51:37.418 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.418 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.418 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:37.418 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.991870+0000", 2024-09-11T09:51:37.418 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.1567e-05 2024-09-11T09:51:37.418 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.418 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.419 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:37.419 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.991923+0000", 2024-09-11T09:51:37.419 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.3035000000000003e-05 2024-09-11T09:51:37.419 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:37.419 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:37.419 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:37.419 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.419 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.419 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4694.0:9 2.1 2:839846fa:ns::rbd_header.1256a0c02879:head [create,call rbd.create in=9b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:37.419 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.062019+0000", 2024-09-11T09:51:37.419 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 14.32734842, 2024-09-11T09:51:37.419 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.001293296, 2024-09-11T09:51:37.419 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:37.419 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:37.420 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:37.420 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:37.420 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4694", 2024-09-11T09:51:37.420 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:51:37.420 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 9 2024-09-11T09:51:37.420 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.420 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:37.420 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.420 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:37.420 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.062019+0000", 2024-09-11T09:51:37.420 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:37.420 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.420 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.421 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:37.421 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.062019+0000", 2024-09-11T09:51:37.421 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:37.421 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.421 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.421 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:37.421 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.062020+0000", 2024-09-11T09:51:37.421 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2610000000000001e-06 2024-09-11T09:51:37.421 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.421 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.421 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:37.421 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.062025+0000", 2024-09-11T09:51:37.421 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.7879999999999997e-06 2024-09-11T09:51:37.422 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.422 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.422 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:37.422 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.062033+0000", 2024-09-11T09:51:37.422 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.9489999999999996e-06 2024-09-11T09:51:37.422 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.422 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.422 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:37.422 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.062039+0000", 2024-09-11T09:51:37.422 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.781e-06 2024-09-11T09:51:37.422 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.422 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.422 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:37.422 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.062099+0000", 2024-09-11T09:51:37.423 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.9905999999999999e-05 2024-09-11T09:51:37.423 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.423 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.423 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:37.423 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.062247+0000", 2024-09-11T09:51:37.423 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000147472 2024-09-11T09:51:37.423 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.423 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.423 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1", 2024-09-11T09:51:37.423 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.062468+0000", 2024-09-11T09:51:37.423 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00022112999999999999 2024-09-11T09:51:37.423 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.423 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.423 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:37.424 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.063076+0000", 2024-09-11T09:51:37.424 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00060857100000000003 2024-09-11T09:51:37.424 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.424 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.424 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:37.424 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.063251+0000", 2024-09-11T09:51:37.424 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00017496499999999999 2024-09-11T09:51:37.424 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.424 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.424 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:37.424 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.063285+0000", 2024-09-11T09:51:37.424 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.3633999999999997e-05 2024-09-11T09:51:37.424 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.425 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.425 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:37.425 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.063312+0000", 2024-09-11T09:51:37.425 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.6838999999999998e-05 2024-09-11T09:51:37.425 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:37.425 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:37.425 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:37.425 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.425 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.425 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4694.0:32 2.1 2:8a862b48:ns::rbd_data.1256a0c02879.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:37.425 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.084089+0000", 2024-09-11T09:51:37.425 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 14.305277706, 2024-09-11T09:51:37.425 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.020630553999999999, 2024-09-11T09:51:37.425 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:37.426 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:37.426 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:37.426 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:37.426 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4694", 2024-09-11T09:51:37.426 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:51:37.426 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 32 2024-09-11T09:51:37.426 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.426 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:37.426 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.426 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:37.426 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.084089+0000", 2024-09-11T09:51:37.426 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:37.426 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.427 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.427 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:37.427 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.084089+0000", 2024-09-11T09:51:37.427 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:37.427 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.427 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.427 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:37.427 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.084090+0000", 2024-09-11T09:51:37.427 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.2900000000000003e-07 2024-09-11T09:51:37.427 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.427 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.427 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:37.427 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.090866+0000", 2024-09-11T09:51:37.427 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0067756609999999997 2024-09-11T09:51:37.428 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.428 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.428 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:37.428 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.090891+0000", 2024-09-11T09:51:37.428 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.5119000000000001e-05 2024-09-11T09:51:37.428 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.428 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.428 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:37.428 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.090900+0000", 2024-09-11T09:51:37.428 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.4110000000000002e-06 2024-09-11T09:51:37.428 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.428 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.428 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:37.429 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.090968+0000", 2024-09-11T09:51:37.429 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.8041999999999997e-05 2024-09-11T09:51:37.429 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.429 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.429 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:37.429 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.091115+0000", 2024-09-11T09:51:37.429 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00014633499999999999 2024-09-11T09:51:37.429 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.429 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.429 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1", 2024-09-11T09:51:37.429 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.091284+0000", 2024-09-11T09:51:37.429 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000169836 2024-09-11T09:51:37.429 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.430 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.430 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:37.430 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.100896+0000", 2024-09-11T09:51:37.430 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0096118639999999995 2024-09-11T09:51:37.430 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.430 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.430 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:37.430 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.104604+0000", 2024-09-11T09:51:37.430 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0037074619999999999 2024-09-11T09:51:37.430 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.430 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.430 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:37.430 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.104669+0000", 2024-09-11T09:51:37.430 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.4836000000000001e-05 2024-09-11T09:51:37.431 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.431 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.431 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:37.431 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.104720+0000", 2024-09-11T09:51:37.431 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.1258999999999998e-05 2024-09-11T09:51:37.431 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:37.431 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:37.431 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:37.431 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.431 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.431 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4694.0:44 2.3 e53/15 2:cd1a7d1c:ns::rbd_data.1256a0c02879.0000000000000006:head v 53'3, mlcod=16'1)", 2024-09-11T09:51:37.431 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.120387+0000", 2024-09-11T09:51:37.431 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 14.268980053, 2024-09-11T09:51:37.431 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.011988468, 2024-09-11T09:51:37.432 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:37.432 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:37.432 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:37.432 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:37.432 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.432 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:37.432 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.120387+0000", 2024-09-11T09:51:37.432 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:37.432 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.432 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.432 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:37.432 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.120387+0000", 2024-09-11T09:51:37.433 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:37.433 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.433 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.433 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:37.433 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.120389+0000", 2024-09-11T09:51:37.433 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.018e-06 2024-09-11T09:51:37.433 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.433 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.433 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:37.433 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.127403+0000", 2024-09-11T09:51:37.433 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0070144769999999999 2024-09-11T09:51:37.433 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.433 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.433 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:37.434 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.127414+0000", 2024-09-11T09:51:37.434 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0991e-05 2024-09-11T09:51:37.434 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.434 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.434 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:37.434 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.127423+0000", 2024-09-11T09:51:37.434 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.8580000000000006e-06 2024-09-11T09:51:37.434 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.434 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.434 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:37.434 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.127468+0000", 2024-09-11T09:51:37.434 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.4830999999999998e-05 2024-09-11T09:51:37.434 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.435 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.435 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:37.435 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.127485+0000", 2024-09-11T09:51:37.435 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6838999999999999e-05 2024-09-11T09:51:37.435 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.435 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.435 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:37.435 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.132341+0000", 2024-09-11T09:51:37.435 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0048557959999999999 2024-09-11T09:51:37.435 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.435 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.435 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:37.435 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.132375+0000", 2024-09-11T09:51:37.435 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.4657999999999999e-05 2024-09-11T09:51:37.436 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:37.436 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:37.436 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:37.436 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.436 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.436 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4694.0:46 2.3 e53/15 2:ce0e1469:ns::rbd_data.1256a0c02879.0000000000000008:head v 53'4, mlcod=16'1)", 2024-09-11T09:51:37.436 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.129008+0000", 2024-09-11T09:51:37.436 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 14.260359467000001, 2024-09-11T09:51:37.436 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0087151220000000005, 2024-09-11T09:51:37.436 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:37.436 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:37.436 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:37.436 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:37.436 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.437 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:37.437 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.129008+0000", 2024-09-11T09:51:37.437 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:37.437 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.437 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.437 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:37.437 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.129008+0000", 2024-09-11T09:51:37.437 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:37.437 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.437 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.437 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:37.437 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.129009+0000", 2024-09-11T09:51:37.437 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.367e-06 2024-09-11T09:51:37.437 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.438 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.438 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:37.438 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.132839+0000", 2024-09-11T09:51:37.438 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0038297610000000001 2024-09-11T09:51:37.438 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.438 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.438 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:37.438 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.132849+0000", 2024-09-11T09:51:37.438 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0672e-05 2024-09-11T09:51:37.438 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.438 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.438 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:37.438 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.132856+0000", 2024-09-11T09:51:37.438 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.1380000000000004e-06 2024-09-11T09:51:37.439 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.439 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.439 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:37.439 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.132891+0000", 2024-09-11T09:51:37.439 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.5712e-05 2024-09-11T09:51:37.439 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.439 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.439 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:37.439 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.132907+0000", 2024-09-11T09:51:37.439 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6166000000000001e-05 2024-09-11T09:51:37.439 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.439 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.439 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:37.439 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.137681+0000", 2024-09-11T09:51:37.440 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0047735299999999998 2024-09-11T09:51:37.440 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.440 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.440 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:37.440 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.137723+0000", 2024-09-11T09:51:37.440 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.1776000000000001e-05 2024-09-11T09:51:37.440 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:37.440 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:37.440 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:37.440 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.440 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.440 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4694.0:48 2.3 e53/15 2:d2090b70:ns::rbd_data.1256a0c02879.0000000000000009:head v 53'5, mlcod=53'2)", 2024-09-11T09:51:37.440 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.135108+0000", 2024-09-11T09:51:37.441 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 14.254258569999999, 2024-09-11T09:51:37.441 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.010710679000000001, 2024-09-11T09:51:37.441 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:37.441 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:37.441 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:37.441 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:37.441 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.441 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:37.441 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.135108+0000", 2024-09-11T09:51:37.441 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:37.441 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.441 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.441 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:37.441 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.135108+0000", 2024-09-11T09:51:37.442 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:37.442 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.442 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.442 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:37.442 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.135110+0000", 2024-09-11T09:51:37.442 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.122e-06 2024-09-11T09:51:37.442 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.442 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.442 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:37.442 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.140876+0000", 2024-09-11T09:51:37.442 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.005766162 2024-09-11T09:51:37.442 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.442 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.443 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:37.443 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.140888+0000", 2024-09-11T09:51:37.443 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.201e-05 2024-09-11T09:51:37.443 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.443 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.443 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:37.443 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.140894+0000", 2024-09-11T09:51:37.443 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.1739999999999997e-06 2024-09-11T09:51:37.443 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.443 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.443 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:37.443 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.140971+0000", 2024-09-11T09:51:37.443 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.7495000000000003e-05 2024-09-11T09:51:37.443 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.444 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.444 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:37.444 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.140995+0000", 2024-09-11T09:51:37.444 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.3204999999999999e-05 2024-09-11T09:51:37.444 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.444 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.444 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:37.444 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.145757+0000", 2024-09-11T09:51:37.444 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0047627190000000003 2024-09-11T09:51:37.444 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.444 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.444 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:37.444 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.145819+0000", 2024-09-11T09:51:37.445 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.1791999999999994e-05 2024-09-11T09:51:37.445 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:37.445 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:37.445 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:37.445 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.445 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.445 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4694.0:50 2.1 2:83078786:ns::rbd_data.1256a0c02879.000000000000000a:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:37.445 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.141467+0000", 2024-09-11T09:51:37.445 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 14.247899845999999, 2024-09-11T09:51:37.445 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.012785889, 2024-09-11T09:51:37.445 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:37.445 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:37.445 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:37.445 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:37.446 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4694", 2024-09-11T09:51:37.446 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:51:37.446 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 50 2024-09-11T09:51:37.446 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.446 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:37.446 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.446 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:37.446 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.141467+0000", 2024-09-11T09:51:37.446 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:37.446 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.446 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.446 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:37.446 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.141467+0000", 2024-09-11T09:51:37.447 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:37.447 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.447 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.447 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:37.447 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.141468+0000", 2024-09-11T09:51:37.447 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1009999999999999e-06 2024-09-11T09:51:37.447 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.447 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.447 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:37.447 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.146432+0000", 2024-09-11T09:51:37.447 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0049640539999999999 2024-09-11T09:51:37.447 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.447 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.447 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:37.448 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.146446+0000", 2024-09-11T09:51:37.448 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3254e-05 2024-09-11T09:51:37.448 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.448 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.448 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:37.448 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.146452+0000", 2024-09-11T09:51:37.448 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.207e-06 2024-09-11T09:51:37.448 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.448 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.448 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:37.448 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.146510+0000", 2024-09-11T09:51:37.448 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.8322999999999997e-05 2024-09-11T09:51:37.448 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.448 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.449 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:37.449 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.146650+0000", 2024-09-11T09:51:37.449 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00013947100000000001 2024-09-11T09:51:37.449 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.449 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.449 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1", 2024-09-11T09:51:37.449 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.146779+0000", 2024-09-11T09:51:37.449 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000129059 2024-09-11T09:51:37.449 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.449 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.449 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:37.449 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.153778+0000", 2024-09-11T09:51:37.449 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0069997130000000003 2024-09-11T09:51:37.449 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.450 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.450 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:37.450 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.154180+0000", 2024-09-11T09:51:37.450 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00040159800000000003 2024-09-11T09:51:37.450 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.450 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.450 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:37.450 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.154218+0000", 2024-09-11T09:51:37.450 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.7795e-05 2024-09-11T09:51:37.450 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.450 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.450 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:37.450 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.154253+0000", 2024-09-11T09:51:37.451 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.5314000000000001e-05 2024-09-11T09:51:37.451 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:37.451 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:37.451 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:37.451 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.451 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.451 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4694.0:54 2.3 e53/15 2:c44f0253:ns::rbd_data.1256a0c02879.000000000000000b:head v 53'6, mlcod=53'4)", 2024-09-11T09:51:37.451 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.154926+0000", 2024-09-11T09:51:37.451 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 14.234441471, 2024-09-11T09:51:37.451 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0089497099999999996, 2024-09-11T09:51:37.451 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:37.451 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:37.451 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:37.451 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:37.452 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.452 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:37.452 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.154926+0000", 2024-09-11T09:51:37.452 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:37.452 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.452 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.452 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:37.452 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.154926+0000", 2024-09-11T09:51:37.452 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:37.452 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.452 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.452 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:37.453 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.154927+0000", 2024-09-11T09:51:37.453 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.7000000000000003e-07 2024-09-11T09:51:37.453 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.453 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.453 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:37.453 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.158712+0000", 2024-09-11T09:51:37.453 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0037855469999999998 2024-09-11T09:51:37.453 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.453 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.453 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:37.453 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.158726+0000", 2024-09-11T09:51:37.453 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3621000000000001e-05 2024-09-11T09:51:37.453 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.454 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.454 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:37.454 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.158732+0000", 2024-09-11T09:51:37.454 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.9089999999999997e-06 2024-09-11T09:51:37.454 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.454 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.454 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:37.454 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.158781+0000", 2024-09-11T09:51:37.454 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.8900000000000003e-05 2024-09-11T09:51:37.454 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.454 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.454 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:37.454 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.158805+0000", 2024-09-11T09:51:37.454 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.44e-05 2024-09-11T09:51:37.455 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.455 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.455 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:37.455 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.163826+0000", 2024-09-11T09:51:37.455 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0050206399999999998 2024-09-11T09:51:37.455 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.455 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.455 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:37.455 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.163875+0000", 2024-09-11T09:51:37.455 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.9722999999999998e-05 2024-09-11T09:51:37.455 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:37.455 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:37.455 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:37.455 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.456 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.456 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4694.0:56 2.3 e53/15 2:c462f029:ns::rbd_data.1256a0c02879.000000000000000c:head v 53'7, mlcod=53'4)", 2024-09-11T09:51:37.456 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.159549+0000", 2024-09-11T09:51:37.456 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 14.229817765, 2024-09-11T09:51:37.456 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.015591666000000001, 2024-09-11T09:51:37.456 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:37.456 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:37.456 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:37.456 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:37.456 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.456 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:37.456 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.159549+0000", 2024-09-11T09:51:37.456 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:37.457 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.457 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.457 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:37.457 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.159549+0000", 2024-09-11T09:51:37.457 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:37.457 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.457 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.457 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:37.457 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.159550+0000", 2024-09-11T09:51:37.457 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.1500000000000004e-07 2024-09-11T09:51:37.457 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.457 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.457 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:37.457 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.167834+0000", 2024-09-11T09:51:37.458 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0082840220000000003 2024-09-11T09:51:37.458 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.458 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.458 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:37.458 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.167857+0000", 2024-09-11T09:51:37.458 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.3337e-05 2024-09-11T09:51:37.458 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.458 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.458 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:37.458 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.167869+0000", 2024-09-11T09:51:37.458 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1556000000000001e-05 2024-09-11T09:51:37.458 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.458 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.458 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:37.459 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.167939+0000", 2024-09-11T09:51:37.459 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.0674999999999997e-05 2024-09-11T09:51:37.459 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.459 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.459 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:37.459 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.167967+0000", 2024-09-11T09:51:37.459 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.7322999999999999e-05 2024-09-11T09:51:37.459 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.459 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.459 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:37.460 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.175079+0000", 2024-09-11T09:51:37.460 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0071126150000000001 2024-09-11T09:51:37.460 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.460 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.460 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:37.460 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.175141+0000", 2024-09-11T09:51:37.460 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.1523e-05 2024-09-11T09:51:37.460 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:37.460 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:37.460 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:37.460 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.460 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.460 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4694.0:62 2.1 2:839846fa:ns::rbd_header.1256a0c02879:head [call lock.unlock in=10b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:37.461 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.202300+0000", 2024-09-11T09:51:37.461 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 14.187067297, 2024-09-11T09:51:37.461 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.001036546, 2024-09-11T09:51:37.461 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:37.461 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:37.461 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:37.461 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:37.461 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4694", 2024-09-11T09:51:37.461 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:51:37.461 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 62 2024-09-11T09:51:37.461 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.461 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:37.461 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.462 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:37.462 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.202300+0000", 2024-09-11T09:51:37.462 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:37.462 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.462 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.462 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:37.462 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.202300+0000", 2024-09-11T09:51:37.462 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:37.462 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.462 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.462 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:37.462 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.202301+0000", 2024-09-11T09:51:37.462 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.454e-06 2024-09-11T09:51:37.463 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.463 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.463 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:37.463 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.202309+0000", 2024-09-11T09:51:37.463 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.7409999999999992e-06 2024-09-11T09:51:37.463 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.463 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.463 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:37.463 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.202326+0000", 2024-09-11T09:51:37.463 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6657e-05 2024-09-11T09:51:37.463 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.463 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.463 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:37.464 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.202333+0000", 2024-09-11T09:51:37.464 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.1400000000000002e-06 2024-09-11T09:51:37.464 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.464 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.464 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:37.464 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.202386+0000", 2024-09-11T09:51:37.464 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.3638999999999999e-05 2024-09-11T09:51:37.464 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.464 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.464 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:37.464 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.202445+0000", 2024-09-11T09:51:37.464 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.8121000000000001e-05 2024-09-11T09:51:37.464 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.465 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.465 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1", 2024-09-11T09:51:37.465 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.202552+0000", 2024-09-11T09:51:37.465 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000107067 2024-09-11T09:51:37.465 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.465 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.465 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:37.465 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.203012+0000", 2024-09-11T09:51:37.465 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00046002100000000002 2024-09-11T09:51:37.465 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.465 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.465 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:37.465 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.203280+0000", 2024-09-11T09:51:37.465 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00026879300000000003 2024-09-11T09:51:37.466 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.466 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.466 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:37.466 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.203311+0000", 2024-09-11T09:51:37.466 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.1013000000000003e-05 2024-09-11T09:51:37.466 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.466 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.466 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:37.466 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.203336+0000", 2024-09-11T09:51:37.466 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.4899999999999999e-05 2024-09-11T09:51:37.466 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:37.466 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:37.466 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:37.466 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.467 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.467 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4694.0:64 2.1 2:839846fa:ns::rbd_header.1256a0c02879:head [watch unwatch cookie 139806419992736] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:37.467 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.203588+0000", 2024-09-11T09:51:37.467 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 14.185779325, 2024-09-11T09:51:37.467 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0011675959999999999, 2024-09-11T09:51:37.467 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:37.467 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:37.467 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:37.467 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:37.467 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4694", 2024-09-11T09:51:37.467 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:51:37.467 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 64 2024-09-11T09:51:37.467 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.468 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:37.468 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.468 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:37.468 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.203588+0000", 2024-09-11T09:51:37.468 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:37.468 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.468 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.468 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:37.468 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.203588+0000", 2024-09-11T09:51:37.468 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:37.468 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.468 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.468 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:37.468 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.203588+0000", 2024-09-11T09:51:37.469 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.6600000000000002e-07 2024-09-11T09:51:37.469 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.469 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.469 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:37.469 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.203590+0000", 2024-09-11T09:51:37.469 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.773e-06 2024-09-11T09:51:37.469 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.469 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.469 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:37.469 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.203594+0000", 2024-09-11T09:51:37.469 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.6129999999999997e-06 2024-09-11T09:51:37.469 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.469 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.470 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:37.470 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.203598+0000", 2024-09-11T09:51:37.470 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.0809999999999998e-06 2024-09-11T09:51:37.470 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.470 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.470 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:37.470 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.203853+0000", 2024-09-11T09:51:37.470 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00025543300000000001 2024-09-11T09:51:37.470 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.470 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.470 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:37.470 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.203898+0000", 2024-09-11T09:51:37.470 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.4771000000000002e-05 2024-09-11T09:51:37.470 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.471 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.471 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1", 2024-09-11T09:51:37.471 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.203974+0000", 2024-09-11T09:51:37.471 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.6062000000000005e-05 2024-09-11T09:51:37.471 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.471 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.471 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:37.471 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.204452+0000", 2024-09-11T09:51:37.471 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00047848399999999999 2024-09-11T09:51:37.471 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.471 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.471 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:37.471 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.204667+0000", 2024-09-11T09:51:37.471 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00021479600000000001 2024-09-11T09:51:37.472 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.472 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.472 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:37.472 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.204699+0000", 2024-09-11T09:51:37.472 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.1690000000000003e-05 2024-09-11T09:51:37.472 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.472 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.472 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:37.472 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.204755+0000", 2024-09-11T09:51:37.472 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.6527000000000003e-05 2024-09-11T09:51:37.472 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:37.472 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:37.472 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:37.473 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.473 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.473 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4953.0:1 2.1 2:92878cd2:::rbd_id.testrbdnbd27121:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e58)", 2024-09-11T09:51:37.473 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:27.868133+0000", 2024-09-11T09:51:37.473 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 9.5212342959999994, 2024-09-11T09:51:37.473 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.001745866, 2024-09-11T09:51:37.473 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:37.473 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:37.473 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "started", 2024-09-11T09:51:37.473 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:37.474 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4953", 2024-09-11T09:51:37.474 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/2878448434", 2024-09-11T09:51:37.474 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 1 2024-09-11T09:51:37.474 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.474 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:37.474 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.474 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:37.474 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:27.868133+0000", 2024-09-11T09:51:37.474 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:37.474 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.474 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.474 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:37.474 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:27.868133+0000", 2024-09-11T09:51:37.475 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:37.475 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.475 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.475 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:37.475 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:27.868134+0000", 2024-09-11T09:51:37.475 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6249999999999999e-06 2024-09-11T09:51:37.475 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.475 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.475 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:37.475 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:27.868140+0000", 2024-09-11T09:51:37.475 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.7459999999999997e-06 2024-09-11T09:51:37.475 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.475 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.475 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:37.476 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:27.868150+0000", 2024-09-11T09:51:37.476 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0011e-05 2024-09-11T09:51:37.476 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.476 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.476 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:37.476 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:27.868160+0000", 2024-09-11T09:51:37.476 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.3999999999999998e-06 2024-09-11T09:51:37.476 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.476 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.476 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:37.476 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:27.868217+0000", 2024-09-11T09:51:37.477 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.7034999999999999e-05 2024-09-11T09:51:37.477 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.477 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.477 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op must wait for map", 2024-09-11T09:51:37.477 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:27.868222+0000", 2024-09-11T09:51:37.477 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.8939999999999998e-06 2024-09-11T09:51:37.477 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.477 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.477 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:37.477 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:27.869732+0000", 2024-09-11T09:51:37.477 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0015099 2024-09-11T09:51:37.477 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.477 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.477 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:37.478 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:27.869820+0000", 2024-09-11T09:51:37.478 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.7793000000000005e-05 2024-09-11T09:51:37.478 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.478 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.478 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:37.478 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:27.869879+0000", 2024-09-11T09:51:37.478 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.8461999999999997e-05 2024-09-11T09:51:37.478 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:37.478 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:37.478 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:37.478 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.478 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.478 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4807.0:48 2.1 2:8654721b:::rbd_data.1250f5e25b80.0000000000000000:head [read 69632~61440] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e58)", 2024-09-11T09:51:37.479 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:28.353535+0000", 2024-09-11T09:51:37.479 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 9.035832503, 2024-09-11T09:51:37.479 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.001338119, 2024-09-11T09:51:37.479 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:37.479 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:37.479 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "started", 2024-09-11T09:51:37.479 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:37.479 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4807", 2024-09-11T09:51:37.479 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/2963330868", 2024-09-11T09:51:37.479 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 48 2024-09-11T09:51:37.479 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.479 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:37.479 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.479 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:37.480 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:28.353535+0000", 2024-09-11T09:51:37.480 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:37.480 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.480 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.480 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:37.480 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:28.353535+0000", 2024-09-11T09:51:37.480 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:37.480 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.480 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.480 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:37.480 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:28.353535+0000", 2024-09-11T09:51:37.480 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.8799999999999998e-07 2024-09-11T09:51:37.480 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.481 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.481 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:37.481 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:28.353536+0000", 2024-09-11T09:51:37.481 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3239999999999999e-06 2024-09-11T09:51:37.481 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.481 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.481 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:37.481 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:28.353540+0000", 2024-09-11T09:51:37.481 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.4979999999999998e-06 2024-09-11T09:51:37.481 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.481 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.481 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:37.481 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:28.353542+0000", 2024-09-11T09:51:37.482 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.6079999999999998e-06 2024-09-11T09:51:37.482 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.482 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.482 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:37.482 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:28.354553+0000", 2024-09-11T09:51:37.482 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0010111180000000001 2024-09-11T09:51:37.482 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.482 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.482 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:37.482 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:28.354629+0000", 2024-09-11T09:51:37.482 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.5456000000000004e-05 2024-09-11T09:51:37.482 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.482 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.483 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:37.483 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:28.354873+0000", 2024-09-11T09:51:37.483 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00024382700000000001 2024-09-11T09:51:37.483 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:37.483 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:37.483 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:37.483 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.483 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.483 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4807.0:56 2.1 2:9af038b1:::rbd_data.1250f5e25b80.000000000000000f:head [sparse-read 3731456~69632] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e58)", 2024-09-11T09:51:37.483 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:28.357995+0000", 2024-09-11T09:51:37.483 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 9.0313719900000002, 2024-09-11T09:51:37.483 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0013349659999999999, 2024-09-11T09:51:37.483 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:37.483 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:37.484 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "started", 2024-09-11T09:51:37.484 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:37.484 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4807", 2024-09-11T09:51:37.484 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/2963330868", 2024-09-11T09:51:37.484 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 56 2024-09-11T09:51:37.484 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.484 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:37.484 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.484 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:37.484 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:28.357995+0000", 2024-09-11T09:51:37.484 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:37.484 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.484 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.485 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:37.485 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:28.357995+0000", 2024-09-11T09:51:37.485 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:37.485 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.485 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.485 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:37.485 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:28.357995+0000", 2024-09-11T09:51:37.485 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.77e-07 2024-09-11T09:51:37.485 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.485 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.485 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:37.485 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:28.357997+0000", 2024-09-11T09:51:37.485 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2219999999999999e-06 2024-09-11T09:51:37.485 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.486 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.486 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:37.486 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:28.358000+0000", 2024-09-11T09:51:37.486 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.2049999999999998e-06 2024-09-11T09:51:37.486 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.486 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.486 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:37.486 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:28.358002+0000", 2024-09-11T09:51:37.486 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.6000000000000001e-06 2024-09-11T09:51:37.486 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.486 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.486 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:37.486 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:28.359004+0000", 2024-09-11T09:51:37.486 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.001001273 2024-09-11T09:51:37.487 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.487 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.487 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:37.487 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:28.359087+0000", 2024-09-11T09:51:37.487 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.3672999999999997e-05 2024-09-11T09:51:37.487 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.487 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.487 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:37.487 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:28.359330+0000", 2024-09-11T09:51:37.487 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000242616 2024-09-11T09:51:37.487 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:37.487 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:37.487 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:37.488 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.488 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.488 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4807.0:60 2.1 2:9af038b1:::rbd_data.1250f5e25b80.000000000000000f:head [read 3895296~36864] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e58)", 2024-09-11T09:51:37.488 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:28.359787+0000", 2024-09-11T09:51:37.488 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 9.0295802989999991, 2024-09-11T09:51:37.488 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0010384280000000001, 2024-09-11T09:51:37.488 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:37.488 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:37.488 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "started", 2024-09-11T09:51:37.488 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:37.488 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4807", 2024-09-11T09:51:37.488 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/2963330868", 2024-09-11T09:51:37.488 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 60 2024-09-11T09:51:37.488 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.489 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:37.489 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.489 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:37.489 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:28.359787+0000", 2024-09-11T09:51:37.489 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:37.489 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.489 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.489 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:37.489 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:28.359787+0000", 2024-09-11T09:51:37.489 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:37.489 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.489 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.489 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:37.490 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:28.359787+0000", 2024-09-11T09:51:37.490 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.6100000000000002e-07 2024-09-11T09:51:37.490 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.490 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.490 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:37.490 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:28.359793+0000", 2024-09-11T09:51:37.490 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.207e-06 2024-09-11T09:51:37.490 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.490 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.490 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:37.490 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:28.359799+0000", 2024-09-11T09:51:37.490 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.229e-06 2024-09-11T09:51:37.490 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.491 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.491 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:37.491 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:28.359802+0000", 2024-09-11T09:51:37.491 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.9409999999999999e-06 2024-09-11T09:51:37.491 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.491 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.491 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:37.491 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:28.360625+0000", 2024-09-11T09:51:37.491 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00082282500000000001 2024-09-11T09:51:37.491 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.491 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.491 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:37.491 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:28.360665+0000", 2024-09-11T09:51:37.491 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.9601000000000003e-05 2024-09-11T09:51:37.492 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.492 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.492 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:37.492 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:28.360825+0000", 2024-09-11T09:51:37.492 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000160364 2024-09-11T09:51:37.492 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:37.492 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:37.492 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:37.492 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.492 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.492 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.5001.0:56 2.1 2:9af038b1:::rbd_data.1250f5e25b80.000000000000000f:head [read 3710976~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e58)", 2024-09-11T09:51:37.492 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:29.260660+0000", 2024-09-11T09:51:37.492 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 8.1287072069999997, 2024-09-11T09:51:37.493 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.001058458, 2024-09-11T09:51:37.493 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:37.493 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:37.493 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "started", 2024-09-11T09:51:37.493 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:37.493 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.5001", 2024-09-11T09:51:37.493 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/2374572508", 2024-09-11T09:51:37.493 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 56 2024-09-11T09:51:37.493 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.493 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:37.493 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.493 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:37.493 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:29.260660+0000", 2024-09-11T09:51:37.493 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:37.494 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.494 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.494 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:37.494 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:29.260660+0000", 2024-09-11T09:51:37.494 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:37.494 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.494 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.494 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:37.494 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:29.260660+0000", 2024-09-11T09:51:37.494 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.8900000000000001e-07 2024-09-11T09:51:37.494 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.494 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.494 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:37.495 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:29.260661+0000", 2024-09-11T09:51:37.495 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.049e-06 2024-09-11T09:51:37.495 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.495 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.495 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:37.495 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:29.260664+0000", 2024-09-11T09:51:37.495 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.1920000000000001e-06 2024-09-11T09:51:37.495 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.495 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.495 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:37.495 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:29.260667+0000", 2024-09-11T09:51:37.495 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.5009999999999999e-06 2024-09-11T09:51:37.495 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.496 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.496 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:37.496 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:29.261548+0000", 2024-09-11T09:51:37.496 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00088111199999999998 2024-09-11T09:51:37.496 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.496 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.496 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:37.496 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:29.261618+0000", 2024-09-11T09:51:37.496 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.9894999999999994e-05 2024-09-11T09:51:37.496 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:37.496 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:37.496 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:37.496 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:29.261718+0000", 2024-09-11T09:51:37.496 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00010042 2024-09-11T09:51:37.497 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:37.497 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:37.497 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:37.497 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:37.497 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:37.497 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:51:37.498 INFO:tasks.workunit.client.0.smithi107.stderr:+ PID=28136 2024-09-11T09:51:37.498 INFO:tasks.workunit.client.0.smithi107.stderr:+ test -n 28136 2024-09-11T09:51:37.498 INFO:tasks.workunit.client.0.smithi107.stderr:+ ps -p 28136 -C rbd-nbd 2024-09-11T09:51:37.499 INFO:tasks.workunit.client.0.smithi107.stdout: PID TTY TIME CMD 2024-09-11T09:51:37.499 INFO:tasks.workunit.client.0.smithi107.stdout: 28136 ? 00:00:00 rbd-nbd 2024-09-11T09:51:37.499 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-11T09:51:37.500 INFO:tasks.workunit.client.0.smithi107.stderr:+ _sudo dd if=/dev/nbd0 of=/dev/null bs=1M 2024-09-11T09:51:37.500 INFO:tasks.workunit.client.0.smithi107.stderr:+ local cmd 2024-09-11T09:51:37.501 INFO:tasks.workunit.client.0.smithi107.stderr:++ id -u 2024-09-11T09:51:37.501 INFO:tasks.workunit.client.0.smithi107.stderr:+ '[' 1000 -eq 0 ']' 2024-09-11T09:51:37.501 INFO:tasks.workunit.client.0.smithi107.stderr:++ which dd 2024-09-11T09:51:37.501 INFO:tasks.workunit.client.0.smithi107.stderr:+ cmd=/usr/bin/dd 2024-09-11T09:51:37.501 INFO:tasks.workunit.client.0.smithi107.stderr:+ shift 2024-09-11T09:51:37.501 INFO:tasks.workunit.client.0.smithi107.stderr:+ sudo -nE /usr/bin/dd if=/dev/nbd0 of=/dev/null bs=1M 2024-09-11T09:51:37.501 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:51:37.499+0000 7f3c96ec9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31909) UID: 0 2024-09-11T09:51:37.513 INFO:tasks.workunit.client.0.smithi107.stderr:64+0 records in 2024-09-11T09:51:37.513 INFO:tasks.workunit.client.0.smithi107.stderr:64+0 records out 2024-09-11T09:51:37.513 INFO:tasks.workunit.client.0.smithi107.stderr:67108864 bytes (67 MB, 64 MiB) copied, 0.0792864 s, 846 MB/s 2024-09-11T09:51:37.514 INFO:tasks.workunit.client.0.smithi107.stderr:+ expect_false _sudo dd if=/tmp/tmp.rvrcDirNue/data of=/dev/nbd0 bs=1M oflag=direct 2024-09-11T09:51:37.515 INFO:tasks.workunit.client.0.smithi107.stderr:+ _sudo dd if=/tmp/tmp.rvrcDirNue/data of=/dev/nbd0 bs=1M oflag=direct 2024-09-11T09:51:37.515 INFO:tasks.workunit.client.0.smithi107.stderr:+ local cmd 2024-09-11T09:51:37.515 INFO:tasks.workunit.client.0.smithi107.stderr:++ id -u 2024-09-11T09:51:37.516 INFO:tasks.workunit.client.0.smithi107.stderr:+ '[' 1000 -eq 0 ']' 2024-09-11T09:51:37.516 INFO:tasks.workunit.client.0.smithi107.stderr:++ which dd 2024-09-11T09:51:37.517 INFO:tasks.workunit.client.0.smithi107.stderr:+ cmd=/usr/bin/dd 2024-09-11T09:51:37.518 INFO:tasks.workunit.client.0.smithi107.stderr:+ shift 2024-09-11T09:51:37.518 INFO:tasks.workunit.client.0.smithi107.stderr:+ sudo -nE /usr/bin/dd if=/tmp/tmp.rvrcDirNue/data of=/dev/nbd0 bs=1M oflag=direct 2024-09-11T09:51:37.529 INFO:tasks.workunit.client.0.smithi107.stderr:/usr/bin/dd: error writing '/dev/nbd0': Operation not permitted 2024-09-11T09:51:37.529 INFO:tasks.workunit.client.0.smithi107.stderr:1+0 records in 2024-09-11T09:51:37.529 INFO:tasks.workunit.client.0.smithi107.stderr:0+0 records out 2024-09-11T09:51:37.529 INFO:tasks.workunit.client.0.smithi107.stderr:0 bytes copied, 0.000434347 s, 0.0 kB/s 2024-09-11T09:51:37.529 INFO:tasks.workunit.client.0.smithi107.stderr:+ return 0 2024-09-11T09:51:37.529 INFO:tasks.workunit.client.0.smithi107.stderr:+ unmap_device /dev/nbd0 28136 2024-09-11T09:51:37.529 INFO:tasks.workunit.client.0.smithi107.stderr:+ local args=/dev/nbd0 2024-09-11T09:51:37.529 INFO:tasks.workunit.client.0.smithi107.stderr:+ local pid=28136 2024-09-11T09:51:37.530 INFO:tasks.workunit.client.0.smithi107.stderr:+ _sudo rbd device --device-type nbd unmap /dev/nbd0 2024-09-11T09:51:37.530 INFO:tasks.workunit.client.0.smithi107.stderr:+ local cmd 2024-09-11T09:51:37.530 INFO:tasks.workunit.client.0.smithi107.stderr:++ id -u 2024-09-11T09:51:37.530 INFO:tasks.workunit.client.0.smithi107.stderr:+ '[' 1000 -eq 0 ']' 2024-09-11T09:51:37.530 INFO:tasks.workunit.client.0.smithi107.stderr:++ which rbd 2024-09-11T09:51:37.531 INFO:tasks.workunit.client.0.smithi107.stderr:+ cmd=/usr/bin/rbd 2024-09-11T09:51:37.531 INFO:tasks.workunit.client.0.smithi107.stderr:+ shift 2024-09-11T09:51:37.531 INFO:tasks.workunit.client.0.smithi107.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd unmap /dev/nbd0 2024-09-11T09:51:37.599 INFO:tasks.ceph.osd.0.smithi017.stderr:2024-09-11T09:51:37.599+0000 7fb73c0f1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31665) UID: 0 2024-09-11T09:51:37.610 INFO:tasks.workunit.client.0.smithi107.stderr:+ rbd device --device-type nbd list 2024-09-11T09:51:37.610 INFO:tasks.workunit.client.0.smithi107.stderr:+ expect_false grep '^28136\b' 2024-09-11T09:51:37.610 INFO:tasks.workunit.client.0.smithi107.stderr:+ grep '^28136\b' 2024-09-11T09:51:37.613 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-11T09:51:37.613 INFO:teuthology.orchestra.run.smithi096.stdout: "ops": [], 2024-09-11T09:51:37.613 INFO:teuthology.orchestra.run.smithi096.stdout: "num_ops": 0 2024-09-11T09:51:37.613 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-11T09:51:37.622 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-11T09:51:37.656 INFO:tasks.workunit.client.0.smithi107.stderr:+ return 0 2024-09-11T09:51:37.657 INFO:tasks.workunit.client.0.smithi107.stderr:+ ps -C rbd-nbd 2024-09-11T09:51:37.657 INFO:tasks.workunit.client.0.smithi107.stderr:+ expect_false grep '^ *28136\b' 2024-09-11T09:51:37.657 INFO:tasks.workunit.client.0.smithi107.stderr:+ grep '^ *28136\b' 2024-09-11T09:51:37.668 INFO:tasks.workunit.client.0.smithi107.stderr:+ return 0 2024-09-11T09:51:37.668 INFO:tasks.workunit.client.0.smithi107.stderr:+ sleep 0.5 2024-09-11T09:51:37.699 INFO:tasks.ceph.osd.2.smithi017.stderr:2024-09-11T09:51:37.700+0000 7fd20b170640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31672) UID: 0 2024-09-11T09:51:37.724 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-11T09:51:37.725 INFO:teuthology.orchestra.run.smithi096.stdout: "ops": [], 2024-09-11T09:51:37.725 INFO:teuthology.orchestra.run.smithi096.stdout: "complaint_time": 30, 2024-09-11T09:51:37.725 INFO:teuthology.orchestra.run.smithi096.stdout: "num_blocked_ops": 0 2024-09-11T09:51:37.725 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-11T09:51:37.733 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-11T09:51:37.800 INFO:tasks.ceph.osd.2.smithi017.stderr:2024-09-11T09:51:37.800+0000 7fd20b170640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31672) UID: 0 2024-09-11T09:51:37.837 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-11T09:51:37.837 INFO:teuthology.orchestra.run.smithi096.stdout: "size": 20, 2024-09-11T09:51:37.837 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 600, 2024-09-11T09:51:37.838 INFO:teuthology.orchestra.run.smithi096.stdout: "ops": [ 2024-09-11T09:51:37.838 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.838 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:111 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:51:37.838 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.711668+0000", 2024-09-11T09:51:37.838 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 86.125392251999997, 2024-09-11T09:51:37.838 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0077145570000000004, 2024-09-11T09:51:37.838 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:37.838 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:37.838 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:37.838 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:37.838 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:51:37.838 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:51:37.838 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 111 2024-09-11T09:51:37.839 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.839 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:37.839 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.839 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:37.839 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711668+0000", 2024-09-11T09:51:37.839 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:37.839 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.839 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.839 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:37.839 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711668+0000", 2024-09-11T09:51:37.839 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:37.839 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.839 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.840 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:37.840 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711669+0000", 2024-09-11T09:51:37.840 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.99e-07 2024-09-11T09:51:37.840 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.840 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.840 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:37.840 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711673+0000", 2024-09-11T09:51:37.840 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.7440000000000001e-06 2024-09-11T09:51:37.840 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.840 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.840 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:37.840 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711686+0000", 2024-09-11T09:51:37.840 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.3317e-05 2024-09-11T09:51:37.841 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.841 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.841 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:37.841 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711697+0000", 2024-09-11T09:51:37.841 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1022000000000001e-05 2024-09-11T09:51:37.841 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.841 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.841 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:37.841 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.717065+0000", 2024-09-11T09:51:37.841 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.005367531 2024-09-11T09:51:37.841 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.841 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.841 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:37.841 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.717114+0000", 2024-09-11T09:51:37.842 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.9707999999999999e-05 2024-09-11T09:51:37.842 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.842 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.842 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:37.842 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.717190+0000", 2024-09-11T09:51:37.842 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.5394000000000004e-05 2024-09-11T09:51:37.842 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.842 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.842 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:37.842 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718793+0000", 2024-09-11T09:51:37.842 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001603653 2024-09-11T09:51:37.842 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.842 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.842 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:37.843 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719329+0000", 2024-09-11T09:51:37.843 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00053543799999999995 2024-09-11T09:51:37.843 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.843 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.843 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:37.843 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719359+0000", 2024-09-11T09:51:37.843 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.0420999999999999e-05 2024-09-11T09:51:37.843 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.843 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.843 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:37.843 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719383+0000", 2024-09-11T09:51:37.843 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.3730000000000001e-05 2024-09-11T09:51:37.843 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:37.844 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:37.844 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:37.844 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.844 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.844 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:112 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:51:37.844 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.711712+0000", 2024-09-11T09:51:37.844 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 86.125348692000003, 2024-09-11T09:51:37.844 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.008162084, 2024-09-11T09:51:37.844 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:37.844 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:37.844 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:37.844 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:37.844 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:51:37.845 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:51:37.845 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 112 2024-09-11T09:51:37.845 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.845 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:37.845 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.845 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:37.845 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711712+0000", 2024-09-11T09:51:37.845 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:37.845 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.845 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.845 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:37.845 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711712+0000", 2024-09-11T09:51:37.845 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:37.845 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.846 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.846 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:37.846 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711712+0000", 2024-09-11T09:51:37.846 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.6000000000000004e-07 2024-09-11T09:51:37.846 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.846 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.846 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:37.846 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711808+0000", 2024-09-11T09:51:37.846 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.5977999999999995e-05 2024-09-11T09:51:37.846 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.846 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.846 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:37.846 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711819+0000", 2024-09-11T09:51:37.847 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.0755e-05 2024-09-11T09:51:37.847 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.847 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.847 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:37.847 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711826+0000", 2024-09-11T09:51:37.847 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.7970000000000003e-06 2024-09-11T09:51:37.847 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.847 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.847 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:37.847 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.717435+0000", 2024-09-11T09:51:37.847 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0056090239999999998 2024-09-11T09:51:37.847 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.847 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.847 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:37.848 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.717496+0000", 2024-09-11T09:51:37.848 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.0543e-05 2024-09-11T09:51:37.848 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.848 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.848 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:37.848 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.717591+0000", 2024-09-11T09:51:37.848 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.5667000000000004e-05 2024-09-11T09:51:37.848 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.848 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.848 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:37.848 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718831+0000", 2024-09-11T09:51:37.848 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001240204 2024-09-11T09:51:37.848 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.848 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.849 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:37.849 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719819+0000", 2024-09-11T09:51:37.849 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00098751500000000001 2024-09-11T09:51:37.849 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.849 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.849 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:37.849 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719850+0000", 2024-09-11T09:51:37.849 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.1535999999999998e-05 2024-09-11T09:51:37.849 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.849 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.849 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:37.849 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719874+0000", 2024-09-11T09:51:37.850 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.3504999999999999e-05 2024-09-11T09:51:37.850 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:37.850 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:37.850 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:37.850 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.850 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.850 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:113 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:51:37.850 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.711842+0000", 2024-09-11T09:51:37.850 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 86.125218192999995, 2024-09-11T09:51:37.850 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0081160099999999999, 2024-09-11T09:51:37.850 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:37.850 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:37.850 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:37.851 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:37.851 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:51:37.851 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:51:37.851 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 113 2024-09-11T09:51:37.851 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.851 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:37.851 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.851 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:37.851 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711842+0000", 2024-09-11T09:51:37.851 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:37.851 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.851 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.851 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:37.851 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711842+0000", 2024-09-11T09:51:37.852 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:37.852 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.852 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.852 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:37.852 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711843+0000", 2024-09-11T09:51:37.852 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.4399999999999999e-07 2024-09-11T09:51:37.852 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.852 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.852 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:37.852 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711848+0000", 2024-09-11T09:51:37.852 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.4159999999999997e-06 2024-09-11T09:51:37.852 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.853 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.853 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:37.853 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711856+0000", 2024-09-11T09:51:37.853 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.7870000000000007e-06 2024-09-11T09:51:37.853 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.853 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.853 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:37.853 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711862+0000", 2024-09-11T09:51:37.853 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.3689999999999998e-06 2024-09-11T09:51:37.853 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.853 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.853 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:37.854 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.717965+0000", 2024-09-11T09:51:37.854 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0061028899999999997 2024-09-11T09:51:37.854 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.854 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.854 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:37.854 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718023+0000", 2024-09-11T09:51:37.854 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.8313000000000002e-05 2024-09-11T09:51:37.854 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.854 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.854 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:37.854 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718109+0000", 2024-09-11T09:51:37.854 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.6205000000000006e-05 2024-09-11T09:51:37.854 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.855 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.855 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:37.855 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719808+0000", 2024-09-11T09:51:37.855 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0016990060000000001 2024-09-11T09:51:37.855 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.855 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.855 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:37.855 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719883+0000", 2024-09-11T09:51:37.855 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.4624999999999998e-05 2024-09-11T09:51:37.855 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.855 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.855 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:37.855 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719910+0000", 2024-09-11T09:51:37.855 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.7512e-05 2024-09-11T09:51:37.856 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.856 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.856 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:37.856 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719958+0000", 2024-09-11T09:51:37.856 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.8143000000000003e-05 2024-09-11T09:51:37.856 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:37.856 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:37.856 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:37.856 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.856 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.856 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:51:37.856 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.711887+0000", 2024-09-11T09:51:37.856 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 86.125173579000005, 2024-09-11T09:51:37.857 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0086063589999999992, 2024-09-11T09:51:37.857 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:37.857 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:37.857 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:37.857 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:37.857 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:51:37.857 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:51:37.857 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 114 2024-09-11T09:51:37.857 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.857 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:37.857 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.857 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:37.857 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711887+0000", 2024-09-11T09:51:37.858 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:37.858 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.858 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.858 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:37.858 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711887+0000", 2024-09-11T09:51:37.858 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:37.858 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.858 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.858 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:37.858 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711887+0000", 2024-09-11T09:51:37.858 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.1500000000000005e-07 2024-09-11T09:51:37.858 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.858 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.858 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:37.859 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711897+0000", 2024-09-11T09:51:37.859 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.1900000000000001e-06 2024-09-11T09:51:37.859 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.859 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.859 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:37.859 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711902+0000", 2024-09-11T09:51:37.859 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.2349999999999999e-06 2024-09-11T09:51:37.859 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.859 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.859 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:37.859 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711906+0000", 2024-09-11T09:51:37.859 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.7129999999999999e-06 2024-09-11T09:51:37.859 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.860 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.860 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:37.860 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718429+0000", 2024-09-11T09:51:37.860 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0065231960000000002 2024-09-11T09:51:37.860 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.860 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.860 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:37.860 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718480+0000", 2024-09-11T09:51:37.860 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.1161e-05 2024-09-11T09:51:37.860 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.860 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.860 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:37.860 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718558+0000", 2024-09-11T09:51:37.861 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.7519000000000001e-05 2024-09-11T09:51:37.861 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.861 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.861 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:37.861 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720038+0000", 2024-09-11T09:51:37.861 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001480016 2024-09-11T09:51:37.861 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.861 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.861 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:37.861 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720417+0000", 2024-09-11T09:51:37.861 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00037943699999999999 2024-09-11T09:51:37.861 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.861 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.862 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:37.862 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720452+0000", 2024-09-11T09:51:37.862 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.4595000000000003e-05 2024-09-11T09:51:37.862 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.862 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.862 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:37.862 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720493+0000", 2024-09-11T09:51:37.862 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.1782000000000001e-05 2024-09-11T09:51:37.862 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:37.862 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:37.862 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:37.862 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.862 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.862 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:51:37.863 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.711912+0000", 2024-09-11T09:51:37.863 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 86.125148909999993, 2024-09-11T09:51:37.863 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0086864750000000008, 2024-09-11T09:51:37.863 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:37.865 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:37.865 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:37.865 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:37.865 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:51:37.865 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:51:37.865 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 115 2024-09-11T09:51:37.865 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.865 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:37.865 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.866 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:37.866 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711912+0000", 2024-09-11T09:51:37.866 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:37.866 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.866 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.866 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:37.866 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711912+0000", 2024-09-11T09:51:37.866 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:37.867 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.867 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.867 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:37.867 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711912+0000", 2024-09-11T09:51:37.867 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.3900000000000001e-07 2024-09-11T09:51:37.868 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.868 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.868 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:37.868 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711969+0000", 2024-09-11T09:51:37.868 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.7501999999999999e-05 2024-09-11T09:51:37.868 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.869 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.869 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:37.869 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711986+0000", 2024-09-11T09:51:37.869 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.6402999999999999e-05 2024-09-11T09:51:37.869 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.869 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.870 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:37.870 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711990+0000", 2024-09-11T09:51:37.870 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.929e-06 2024-09-11T09:51:37.870 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.870 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.870 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:37.871 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718861+0000", 2024-09-11T09:51:37.871 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0068710530000000002 2024-09-11T09:51:37.871 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.871 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.872 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:37.872 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718908+0000", 2024-09-11T09:51:37.872 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.7431e-05 2024-09-11T09:51:37.872 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.872 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.872 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:37.873 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718984+0000", 2024-09-11T09:51:37.873 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.5500999999999994e-05 2024-09-11T09:51:37.873 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.873 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.873 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:37.874 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720500+0000", 2024-09-11T09:51:37.874 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0015158820000000001 2024-09-11T09:51:37.874 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.874 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.874 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:37.874 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720536+0000", 2024-09-11T09:51:37.875 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.6118000000000002e-05 2024-09-11T09:51:37.875 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.875 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.875 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:37.875 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720570+0000", 2024-09-11T09:51:37.875 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.4387999999999997e-05 2024-09-11T09:51:37.875 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.875 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.875 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:37.875 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720598+0000", 2024-09-11T09:51:37.875 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.8028999999999999e-05 2024-09-11T09:51:37.876 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:37.876 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:37.876 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:37.876 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.876 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.876 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:51:37.876 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.711998+0000", 2024-09-11T09:51:37.876 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 86.125062731, 2024-09-11T09:51:37.876 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.010785813999999999, 2024-09-11T09:51:37.876 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:37.876 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:37.876 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:37.876 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:37.877 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:51:37.877 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:51:37.877 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 116 2024-09-11T09:51:37.877 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.877 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:37.877 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.877 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:37.877 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711998+0000", 2024-09-11T09:51:37.877 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:37.877 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.878 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.878 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:37.878 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711998+0000", 2024-09-11T09:51:37.878 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:37.878 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.878 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.878 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:37.878 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711998+0000", 2024-09-11T09:51:37.878 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.3300000000000003e-07 2024-09-11T09:51:37.878 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.878 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.878 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:37.878 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712001+0000", 2024-09-11T09:51:37.878 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.9349999999999999e-06 2024-09-11T09:51:37.879 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.879 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.879 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:37.879 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712006+0000", 2024-09-11T09:51:37.879 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.1630000000000004e-06 2024-09-11T09:51:37.879 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.879 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.879 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:37.879 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712010+0000", 2024-09-11T09:51:37.879 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.574e-06 2024-09-11T09:51:37.879 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.879 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.879 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:37.880 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719415+0000", 2024-09-11T09:51:37.880 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0074049099999999998 2024-09-11T09:51:37.880 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.880 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.880 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:37.880 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719465+0000", 2024-09-11T09:51:37.880 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.0129000000000001e-05 2024-09-11T09:51:37.880 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.880 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.880 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:37.880 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719545+0000", 2024-09-11T09:51:37.880 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.0253999999999994e-05 2024-09-11T09:51:37.880 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.881 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.881 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:37.881 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720636+0000", 2024-09-11T09:51:37.881 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0010909330000000001 2024-09-11T09:51:37.881 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.881 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.881 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:37.881 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.722707+0000", 2024-09-11T09:51:37.881 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0020713300000000001 2024-09-11T09:51:37.881 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.881 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.881 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:37.881 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.722743+0000", 2024-09-11T09:51:37.882 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.5351000000000001e-05 2024-09-11T09:51:37.882 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.882 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.882 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:37.882 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.722784+0000", 2024-09-11T09:51:37.882 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.0802e-05 2024-09-11T09:51:37.882 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:37.882 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:37.882 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:37.882 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.882 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.882 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:51:37.882 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.712014+0000", 2024-09-11T09:51:37.883 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 86.125046690999994, 2024-09-11T09:51:37.883 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.013420938, 2024-09-11T09:51:37.883 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:37.883 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:37.883 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:37.883 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:37.883 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:51:37.883 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:51:37.883 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 117 2024-09-11T09:51:37.883 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.883 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:37.884 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.884 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:37.884 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712014+0000", 2024-09-11T09:51:37.884 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:37.884 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.884 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.884 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:37.884 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712014+0000", 2024-09-11T09:51:37.884 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:37.884 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.884 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.884 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:37.885 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712014+0000", 2024-09-11T09:51:37.885 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.2000000000000001e-07 2024-09-11T09:51:37.885 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.885 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.885 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:37.885 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712016+0000", 2024-09-11T09:51:37.885 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.2309999999999999e-06 2024-09-11T09:51:37.885 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.885 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.885 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:37.885 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712020+0000", 2024-09-11T09:51:37.885 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.3830000000000001e-06 2024-09-11T09:51:37.885 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.886 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.886 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:37.886 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712023+0000", 2024-09-11T09:51:37.886 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.3340000000000002e-06 2024-09-11T09:51:37.886 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.886 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.886 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:37.887 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720083+0000", 2024-09-11T09:51:37.887 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0080601189999999993 2024-09-11T09:51:37.887 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.887 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.887 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:37.887 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720125+0000", 2024-09-11T09:51:37.888 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.2029e-05 2024-09-11T09:51:37.888 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.888 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.888 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:37.888 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720205+0000", 2024-09-11T09:51:37.888 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.0010000000000001e-05 2024-09-11T09:51:37.888 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.888 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.888 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:37.888 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720849+0000", 2024-09-11T09:51:37.888 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00064369899999999996 2024-09-11T09:51:37.889 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.889 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.889 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:37.889 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.725357+0000", 2024-09-11T09:51:37.889 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0045077379999999998 2024-09-11T09:51:37.889 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.889 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.889 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:37.889 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.725392+0000", 2024-09-11T09:51:37.889 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.5862999999999999e-05 2024-09-11T09:51:37.889 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.889 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.890 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:37.890 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.725435+0000", 2024-09-11T09:51:37.890 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.2311999999999997e-05 2024-09-11T09:51:37.890 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:37.890 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:37.890 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:37.890 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.890 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.890 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:159 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:51:37.891 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:12.452099+0000", 2024-09-11T09:51:37.891 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 85.384961102999995, 2024-09-11T09:51:37.891 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0069885950000000002, 2024-09-11T09:51:37.891 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:37.891 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:37.891 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:37.891 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:37.891 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:51:37.891 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:51:37.891 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 159 2024-09-11T09:51:37.892 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.892 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:37.892 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.892 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:37.892 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452099+0000", 2024-09-11T09:51:37.892 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:37.892 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.892 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.892 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:37.892 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452099+0000", 2024-09-11T09:51:37.893 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:37.893 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.893 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.893 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:37.893 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452100+0000", 2024-09-11T09:51:37.893 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.2800000000000003e-07 2024-09-11T09:51:37.893 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.893 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.893 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:37.893 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452102+0000", 2024-09-11T09:51:37.893 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.824e-06 2024-09-11T09:51:37.893 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.893 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.893 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:37.894 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452105+0000", 2024-09-11T09:51:37.894 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.0460000000000001e-06 2024-09-11T09:51:37.894 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.894 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.894 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:37.894 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452107+0000", 2024-09-11T09:51:37.894 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.543e-06 2024-09-11T09:51:37.895 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.895 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.895 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:37.895 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.455484+0000", 2024-09-11T09:51:37.895 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0033769759999999998 2024-09-11T09:51:37.895 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.895 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.896 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:37.896 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.455527+0000", 2024-09-11T09:51:37.896 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.2589999999999997e-05 2024-09-11T09:51:37.896 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.896 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.896 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:37.897 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.455594+0000", 2024-09-11T09:51:37.897 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.6775999999999999e-05 2024-09-11T09:51:37.897 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.897 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.897 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:37.897 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.456312+0000", 2024-09-11T09:51:37.897 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00071874700000000005 2024-09-11T09:51:37.897 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.897 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.897 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:37.897 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.459015+0000", 2024-09-11T09:51:37.897 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0027028540000000002 2024-09-11T09:51:37.897 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.898 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.898 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:37.898 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.459054+0000", 2024-09-11T09:51:37.898 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.8840000000000001e-05 2024-09-11T09:51:37.898 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.898 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.898 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:37.898 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.459088+0000", 2024-09-11T09:51:37.898 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.4071e-05 2024-09-11T09:51:37.899 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:37.899 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:37.899 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:37.899 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.899 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.900 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4688.0:38 2.5 2:abde2694:::rbd_data.1250f5e25b80.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:37.900 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:22.903926+0000", 2024-09-11T09:51:37.900 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 14.933134427000001, 2024-09-11T09:51:37.901 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.028303676, 2024-09-11T09:51:37.901 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:37.901 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:37.901 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:37.901 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:37.901 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4688", 2024-09-11T09:51:37.902 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:51:37.902 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 38 2024-09-11T09:51:37.902 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.902 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:37.902 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.902 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:37.902 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.903926+0000", 2024-09-11T09:51:37.902 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:37.902 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.903 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.903 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:37.903 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.903926+0000", 2024-09-11T09:51:37.903 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:37.903 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.903 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.903 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:37.903 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.903928+0000", 2024-09-11T09:51:37.903 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.4249999999999999e-06 2024-09-11T09:51:37.903 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.904 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.904 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:37.904 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.914774+0000", 2024-09-11T09:51:37.904 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.010846613 2024-09-11T09:51:37.904 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.904 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.904 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:37.904 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.914793+0000", 2024-09-11T09:51:37.904 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.8879999999999999e-05 2024-09-11T09:51:37.904 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.904 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.904 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:37.904 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.914801+0000", 2024-09-11T09:51:37.905 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.2230000000000007e-06 2024-09-11T09:51:37.905 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.905 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.905 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:37.905 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.914854+0000", 2024-09-11T09:51:37.905 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.2883999999999997e-05 2024-09-11T09:51:37.905 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.905 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.905 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:37.905 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.914951+0000", 2024-09-11T09:51:37.905 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.6731000000000006e-05 2024-09-11T09:51:37.905 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.905 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.906 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:37.906 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.915080+0000", 2024-09-11T09:51:37.906 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00012926999999999999 2024-09-11T09:51:37.906 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.906 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.906 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:37.906 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.931469+0000", 2024-09-11T09:51:37.906 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.016389107 2024-09-11T09:51:37.906 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.907 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.907 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:37.907 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.932118+0000", 2024-09-11T09:51:37.907 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00064915600000000002 2024-09-11T09:51:37.907 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.907 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.907 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:37.907 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.932177+0000", 2024-09-11T09:51:37.907 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.9032e-05 2024-09-11T09:51:37.907 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.907 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.907 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:37.908 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.932230+0000", 2024-09-11T09:51:37.908 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.2355000000000003e-05 2024-09-11T09:51:37.908 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:37.908 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:37.908 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:37.908 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.908 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.908 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_repop(client.4688.0:36 2.6 e53/15 2:61e1b442:::rbd_data.1250f5e25b80.0000000000000003:head v 53'2, mlcod=0'0)", 2024-09-11T09:51:37.908 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:22.915597+0000", 2024-09-11T09:51:37.908 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 14.921463877000001, 2024-09-11T09:51:37.908 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.020430798, 2024-09-11T09:51:37.908 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:37.908 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:37.909 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:37.909 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:37.909 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.909 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:37.909 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.915597+0000", 2024-09-11T09:51:37.909 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:37.909 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.909 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.909 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:37.909 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.915597+0000", 2024-09-11T09:51:37.909 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:37.909 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.909 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.910 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:37.910 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.915598+0000", 2024-09-11T09:51:37.910 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.7150000000000001e-06 2024-09-11T09:51:37.910 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.910 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.910 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:37.910 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921741+0000", 2024-09-11T09:51:37.910 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0061424339999999996 2024-09-11T09:51:37.910 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.910 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.910 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:37.911 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921757+0000", 2024-09-11T09:51:37.911 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.6144e-05 2024-09-11T09:51:37.911 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.911 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.911 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:37.911 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921765+0000", 2024-09-11T09:51:37.911 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.0809999999999994e-06 2024-09-11T09:51:37.911 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.911 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.911 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:37.911 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921826+0000", 2024-09-11T09:51:37.911 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.0878999999999998e-05 2024-09-11T09:51:37.911 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.912 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.912 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:37.912 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921849+0000", 2024-09-11T09:51:37.912 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.2966999999999999e-05 2024-09-11T09:51:37.912 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.912 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.912 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:37.912 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.935968+0000", 2024-09-11T09:51:37.912 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.014119191 2024-09-11T09:51:37.912 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.912 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.912 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:37.913 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.936027+0000", 2024-09-11T09:51:37.913 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.9386999999999999e-05 2024-09-11T09:51:37.913 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:37.913 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:37.913 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:37.913 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.913 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.913 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_repop(client.4688.0:39 2.6 e53/15 2:743e7616:::rbd_data.1250f5e25b80.0000000000000005:head v 53'3, mlcod=0'0)", 2024-09-11T09:51:37.913 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:22.921806+0000", 2024-09-11T09:51:37.913 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 14.915254827, 2024-09-11T09:51:37.913 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.015275018, 2024-09-11T09:51:37.913 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:37.914 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:37.914 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:37.914 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:37.914 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.914 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:37.914 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921806+0000", 2024-09-11T09:51:37.914 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:37.914 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.914 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.914 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:37.914 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921806+0000", 2024-09-11T09:51:37.914 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:37.914 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.915 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.915 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:37.915 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921806+0000", 2024-09-11T09:51:37.915 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.2200000000000003e-07 2024-09-11T09:51:37.915 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.915 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.915 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:37.915 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.925590+0000", 2024-09-11T09:51:37.915 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0037838949999999998 2024-09-11T09:51:37.915 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.916 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.916 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:37.916 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.925605+0000", 2024-09-11T09:51:37.916 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.4321999999999999e-05 2024-09-11T09:51:37.916 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.916 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.916 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:37.916 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.925613+0000", 2024-09-11T09:51:37.916 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.1980000000000001e-06 2024-09-11T09:51:37.917 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.917 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.917 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:37.917 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.925857+0000", 2024-09-11T09:51:37.917 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000244064 2024-09-11T09:51:37.917 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.917 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.917 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:37.917 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.925875+0000", 2024-09-11T09:51:37.917 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.838e-05 2024-09-11T09:51:37.917 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.918 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.918 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:37.918 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.937042+0000", 2024-09-11T09:51:37.918 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.011166416 2024-09-11T09:51:37.918 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.918 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.918 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:37.918 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.937081+0000", 2024-09-11T09:51:37.918 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.9020999999999998e-05 2024-09-11T09:51:37.918 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:37.918 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:37.918 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:37.919 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.919 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.919 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4688.0:49 2.5 2:b6dd9fc9:::rbd_data.1250f5e25b80.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:37.919 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:22.938548+0000", 2024-09-11T09:51:37.919 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 14.898512824000001, 2024-09-11T09:51:37.919 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.022766950000000001, 2024-09-11T09:51:37.919 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:37.919 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:37.919 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:37.919 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:37.919 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4688", 2024-09-11T09:51:37.919 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:51:37.920 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 49 2024-09-11T09:51:37.920 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.920 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:37.920 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.920 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:37.920 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.938548+0000", 2024-09-11T09:51:37.920 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:37.920 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.920 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.920 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:37.920 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.938548+0000", 2024-09-11T09:51:37.920 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:37.920 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.921 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.921 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:37.921 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.938550+0000", 2024-09-11T09:51:37.921 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.7710000000000001e-06 2024-09-11T09:51:37.921 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.921 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.921 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:37.921 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.944237+0000", 2024-09-11T09:51:37.921 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0056879790000000001 2024-09-11T09:51:37.921 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.921 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.921 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:37.922 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.944262+0000", 2024-09-11T09:51:37.922 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.4247e-05 2024-09-11T09:51:37.922 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.922 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.922 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:37.922 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.944274+0000", 2024-09-11T09:51:37.922 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.2639999999999999e-05 2024-09-11T09:51:37.922 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.922 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.922 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:37.922 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.944344+0000", 2024-09-11T09:51:37.922 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.9574999999999997e-05 2024-09-11T09:51:37.923 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.923 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.923 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:37.923 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.944474+0000", 2024-09-11T09:51:37.923 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000130464 2024-09-11T09:51:37.923 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.923 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.923 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:37.923 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.944585+0000", 2024-09-11T09:51:37.923 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000110096 2024-09-11T09:51:37.924 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.924 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.924 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:37.924 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.952724+0000", 2024-09-11T09:51:37.924 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0081398900000000003 2024-09-11T09:51:37.924 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.924 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.924 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:37.924 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.961219+0000", 2024-09-11T09:51:37.924 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0084949789999999997 2024-09-11T09:51:37.924 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.924 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.925 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:37.925 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.961280+0000", 2024-09-11T09:51:37.925 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.0404e-05 2024-09-11T09:51:37.925 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.925 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.925 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:37.925 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.961315+0000", 2024-09-11T09:51:37.925 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.4904999999999999e-05 2024-09-11T09:51:37.925 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:37.925 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:37.925 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:37.925 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.925 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.926 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_repop(client.4688.0:44 2.6 e53/15 2:67246a22:::rbd_data.1250f5e25b80.0000000000000007:head v 53'4, mlcod=53'2)", 2024-09-11T09:51:37.926 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:22.951721+0000", 2024-09-11T09:51:37.926 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 14.885339552, 2024-09-11T09:51:37.926 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.01080981, 2024-09-11T09:51:37.926 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:37.926 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:37.926 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:37.926 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:37.926 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.926 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:37.926 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.951721+0000", 2024-09-11T09:51:37.926 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:37.926 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.927 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.927 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:37.927 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.951721+0000", 2024-09-11T09:51:37.927 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:37.927 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.927 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.927 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:37.927 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.951722+0000", 2024-09-11T09:51:37.927 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.0279999999999999e-06 2024-09-11T09:51:37.927 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.927 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.927 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:37.927 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.955612+0000", 2024-09-11T09:51:37.928 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0038896920000000001 2024-09-11T09:51:37.928 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.928 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.928 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:37.928 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.955628+0000", 2024-09-11T09:51:37.928 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.5945000000000001e-05 2024-09-11T09:51:37.928 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.928 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.928 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:37.928 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.955635+0000", 2024-09-11T09:51:37.928 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.8080000000000005e-06 2024-09-11T09:51:37.928 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.928 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.929 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:37.929 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.955700+0000", 2024-09-11T09:51:37.929 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.4114000000000003e-05 2024-09-11T09:51:37.929 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.929 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.929 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:37.929 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.955724+0000", 2024-09-11T09:51:37.929 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.3989999999999999e-05 2024-09-11T09:51:37.929 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.929 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.929 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:37.929 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.962490+0000", 2024-09-11T09:51:37.929 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0067662599999999996 2024-09-11T09:51:37.929 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.930 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.930 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:37.930 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.962531+0000", 2024-09-11T09:51:37.930 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.0973000000000002e-05 2024-09-11T09:51:37.930 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:37.930 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:37.930 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:37.930 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.930 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.930 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_repop(client.4688.0:51 2.6 e53/15 2:6b5894fc:::rbd_data.1250f5e25b80.000000000000000a:head v 53'5, mlcod=53'2)", 2024-09-11T09:51:37.930 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:22.959030+0000", 2024-09-11T09:51:37.930 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 14.878030378, 2024-09-11T09:51:37.930 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.011503638, 2024-09-11T09:51:37.931 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:37.931 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:37.931 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:37.931 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:37.931 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.931 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:37.931 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.959030+0000", 2024-09-11T09:51:37.931 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:37.931 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.931 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.931 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:37.931 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.959030+0000", 2024-09-11T09:51:37.932 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:37.932 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.932 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.932 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:37.932 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.959032+0000", 2024-09-11T09:51:37.932 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.356e-06 2024-09-11T09:51:37.932 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.932 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.932 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:37.932 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.962899+0000", 2024-09-11T09:51:37.932 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.003867639 2024-09-11T09:51:37.932 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.932 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.933 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:37.933 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.962909+0000", 2024-09-11T09:51:37.936 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.0105999999999999e-05 2024-09-11T09:51:37.936 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.936 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.937 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:37.937 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.962915+0000", 2024-09-11T09:51:37.937 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.8869999999999997e-06 2024-09-11T09:51:37.937 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.937 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.937 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:37.937 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.962962+0000", 2024-09-11T09:51:37.937 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.6511999999999998e-05 2024-09-11T09:51:37.937 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.937 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.937 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:37.937 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.962989+0000", 2024-09-11T09:51:37.938 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.7702999999999999e-05 2024-09-11T09:51:37.938 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.938 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.938 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:37.938 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.970495+0000", 2024-09-11T09:51:37.938 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0075052449999999998 2024-09-11T09:51:37.938 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.938 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.938 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:37.938 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.970534+0000", 2024-09-11T09:51:37.938 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.9190000000000003e-05 2024-09-11T09:51:37.938 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:37.938 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:37.939 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:37.939 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.939 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.939 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_repop(client.4688.0:58 2.6 e53/15 2:6277c644:::rbd_data.1250f5e25b80.000000000000000e:head v 53'6, mlcod=53'4)", 2024-09-11T09:51:37.939 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:22.996199+0000", 2024-09-11T09:51:37.939 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 14.840861457999999, 2024-09-11T09:51:37.939 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.010145573999999999, 2024-09-11T09:51:37.939 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:37.939 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:37.939 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:37.939 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:37.939 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.939 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:37.940 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.996199+0000", 2024-09-11T09:51:37.940 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:37.940 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.940 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.940 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:37.940 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.996199+0000", 2024-09-11T09:51:37.940 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:37.940 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.940 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.940 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:37.941 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.996200+0000", 2024-09-11T09:51:37.941 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.85e-07 2024-09-11T09:51:37.941 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.941 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.941 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:37.941 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.999914+0000", 2024-09-11T09:51:37.941 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.003714406 2024-09-11T09:51:37.941 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.941 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.941 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:37.941 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.999928+0000", 2024-09-11T09:51:37.942 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.3149000000000001e-05 2024-09-11T09:51:37.942 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.942 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.942 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:37.942 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.999951+0000", 2024-09-11T09:51:37.942 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.3116e-05 2024-09-11T09:51:37.942 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.942 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.942 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:37.942 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.000003+0000", 2024-09-11T09:51:37.943 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.2169000000000001e-05 2024-09-11T09:51:37.943 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.943 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.943 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:37.943 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.000018+0000", 2024-09-11T09:51:37.943 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.5500000000000001e-05 2024-09-11T09:51:37.943 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.943 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.943 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:37.943 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.006303+0000", 2024-09-11T09:51:37.943 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0062846710000000004 2024-09-11T09:51:37.943 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.944 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.944 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:37.944 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.006345+0000", 2024-09-11T09:51:37.944 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.1678000000000003e-05 2024-09-11T09:51:37.944 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:37.944 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:37.944 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:37.944 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.944 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.944 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4694.0:42 2.5 2:a4fc00ec:ns::rbd_data.1256a0c02879.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:37.944 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.111570+0000", 2024-09-11T09:51:37.944 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 14.725490352, 2024-09-11T09:51:37.945 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.038002197000000001, 2024-09-11T09:51:37.945 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:37.945 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:37.945 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:37.945 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:37.945 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4694", 2024-09-11T09:51:37.945 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:51:37.945 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 42 2024-09-11T09:51:37.945 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.945 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:37.945 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.945 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:37.945 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.111570+0000", 2024-09-11T09:51:37.946 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:37.946 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.946 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.946 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:37.946 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.111570+0000", 2024-09-11T09:51:37.946 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:37.946 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.946 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.946 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:37.946 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.111572+0000", 2024-09-11T09:51:37.946 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.3909999999999999e-06 2024-09-11T09:51:37.946 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.946 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.947 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:37.947 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.135929+0000", 2024-09-11T09:51:37.947 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.024357410999999999 2024-09-11T09:51:37.947 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.947 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.947 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:37.947 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.135951+0000", 2024-09-11T09:51:37.947 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.1775000000000001e-05 2024-09-11T09:51:37.947 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.947 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.947 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:37.947 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.135962+0000", 2024-09-11T09:51:37.947 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.0903e-05 2024-09-11T09:51:37.947 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.948 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.948 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:37.948 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.136045+0000", 2024-09-11T09:51:37.948 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.3128999999999996e-05 2024-09-11T09:51:37.948 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.948 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.948 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:37.948 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.136229+0000", 2024-09-11T09:51:37.948 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00018393700000000001 2024-09-11T09:51:37.948 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.948 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.948 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:37.948 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.136394+0000", 2024-09-11T09:51:37.949 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000165527 2024-09-11T09:51:37.949 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.949 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.949 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:37.949 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.149320+0000", 2024-09-11T09:51:37.949 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.01292568 2024-09-11T09:51:37.949 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.949 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.949 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:37.949 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.149493+0000", 2024-09-11T09:51:37.949 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00017285999999999999 2024-09-11T09:51:37.949 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.950 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.950 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:37.950 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.149539+0000", 2024-09-11T09:51:37.950 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.6628000000000001e-05 2024-09-11T09:51:37.950 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.950 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.950 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:37.950 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.149572+0000", 2024-09-11T09:51:37.950 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.2956000000000001e-05 2024-09-11T09:51:37.950 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:37.950 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:37.950 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:37.950 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.950 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.951 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_repop(client.4694.0:45 2.6 e53/15 2:671fb830:ns::rbd_data.1256a0c02879.0000000000000007:head v 53'9, mlcod=53'7)", 2024-09-11T09:51:37.951 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.121833+0000", 2024-09-11T09:51:37.951 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 14.715227993999999, 2024-09-11T09:51:37.951 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.023876192000000001, 2024-09-11T09:51:37.951 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:37.951 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:37.951 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:37.951 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:37.951 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.951 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:37.951 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.121833+0000", 2024-09-11T09:51:37.951 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:37.951 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.951 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.952 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:37.952 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.121833+0000", 2024-09-11T09:51:37.952 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:37.952 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.952 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.952 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:37.952 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.121834+0000", 2024-09-11T09:51:37.952 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.471e-06 2024-09-11T09:51:37.952 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.952 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.952 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:37.952 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.128990+0000", 2024-09-11T09:51:37.952 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0071557879999999997 2024-09-11T09:51:37.953 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.953 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.953 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:37.953 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.129011+0000", 2024-09-11T09:51:37.953 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.0760000000000001e-05 2024-09-11T09:51:37.953 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.953 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.953 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:37.953 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.129022+0000", 2024-09-11T09:51:37.953 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1056000000000001e-05 2024-09-11T09:51:37.954 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.954 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.954 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:37.954 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.129097+0000", 2024-09-11T09:51:37.954 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.5082999999999999e-05 2024-09-11T09:51:37.954 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.954 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.954 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:37.954 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.129130+0000", 2024-09-11T09:51:37.954 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.3345e-05 2024-09-11T09:51:37.954 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.954 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.954 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:37.955 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.145657+0000", 2024-09-11T09:51:37.955 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.016527224 2024-09-11T09:51:37.955 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.955 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.955 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:37.955 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.145709+0000", 2024-09-11T09:51:37.955 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.1465000000000003e-05 2024-09-11T09:51:37.955 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:37.955 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:37.955 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:37.955 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.955 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.955 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4694.0:43 2.0 2:16c8ae84:ns::rbd_data.1256a0c02879.0000000000000005:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:37.955 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.135999+0000", 2024-09-11T09:51:37.956 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 14.701062044, 2024-09-11T09:51:37.956 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.039468309, 2024-09-11T09:51:37.956 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:37.956 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:37.956 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:37.956 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:37.956 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4694", 2024-09-11T09:51:37.956 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:51:37.956 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 43 2024-09-11T09:51:37.956 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.956 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:37.956 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.957 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:37.957 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.135999+0000", 2024-09-11T09:51:37.957 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:37.957 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.957 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.957 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:37.957 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.135999+0000", 2024-09-11T09:51:37.957 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:37.958 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.958 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.958 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:37.958 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.136000+0000", 2024-09-11T09:51:37.958 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.6330000000000001e-06 2024-09-11T09:51:37.958 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.958 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.958 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:37.958 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.157961+0000", 2024-09-11T09:51:37.958 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0219606 2024-09-11T09:51:37.958 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.959 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.959 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:37.959 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.157995+0000", 2024-09-11T09:51:37.959 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.4391999999999999e-05 2024-09-11T09:51:37.959 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.959 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.959 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:37.959 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.158011+0000", 2024-09-11T09:51:37.959 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.5794000000000002e-05 2024-09-11T09:51:37.959 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.959 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.959 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:37.959 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.158083+0000", 2024-09-11T09:51:37.959 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.2273000000000004e-05 2024-09-11T09:51:37.960 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.960 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.960 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:37.960 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.158239+0000", 2024-09-11T09:51:37.960 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000155372 2024-09-11T09:51:37.960 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.960 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.960 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 1", 2024-09-11T09:51:37.960 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.158403+0000", 2024-09-11T09:51:37.960 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00016408199999999999 2024-09-11T09:51:37.960 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.960 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.960 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:37.961 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.173860+0000", 2024-09-11T09:51:37.961 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.015457347 2024-09-11T09:51:37.961 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.961 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.961 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:37.961 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.175400+0000", 2024-09-11T09:51:37.961 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001540006 2024-09-11T09:51:37.961 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.961 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.961 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:37.961 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.175438+0000", 2024-09-11T09:51:37.961 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.8135e-05 2024-09-11T09:51:37.962 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.962 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.962 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:37.962 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.175467+0000", 2024-09-11T09:51:37.962 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.8674999999999999e-05 2024-09-11T09:51:37.962 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:37.962 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:37.962 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:37.962 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.962 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.962 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4694.0:57 2.0 2:04b5549c:ns::rbd_data.1256a0c02879.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:37.962 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.158037+0000", 2024-09-11T09:51:37.962 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 14.679023305999999, 2024-09-11T09:51:37.963 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.027748790999999998, 2024-09-11T09:51:37.963 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:37.963 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:37.963 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:37.963 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:37.963 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4694", 2024-09-11T09:51:37.963 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:51:37.963 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 57 2024-09-11T09:51:37.963 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.964 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:37.964 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.964 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:37.964 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.158037+0000", 2024-09-11T09:51:37.964 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:37.964 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.964 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.964 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:37.964 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.158037+0000", 2024-09-11T09:51:37.964 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:37.964 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.964 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.965 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:37.965 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.158039+0000", 2024-09-11T09:51:37.965 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.356e-06 2024-09-11T09:51:37.965 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.965 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.965 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:37.965 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.176367+0000", 2024-09-11T09:51:37.965 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.018328739 2024-09-11T09:51:37.965 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.965 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.965 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:37.965 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.176378+0000", 2024-09-11T09:51:37.965 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.0713999999999999e-05 2024-09-11T09:51:37.966 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.966 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.966 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:37.966 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.176383+0000", 2024-09-11T09:51:37.966 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.0209999999999999e-06 2024-09-11T09:51:37.966 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.966 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.966 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:37.966 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.176426+0000", 2024-09-11T09:51:37.966 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.2623999999999997e-05 2024-09-11T09:51:37.967 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.967 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.967 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:37.967 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.176552+0000", 2024-09-11T09:51:37.967 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000126076 2024-09-11T09:51:37.967 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.967 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.967 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 1", 2024-09-11T09:51:37.967 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.176665+0000", 2024-09-11T09:51:37.967 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000112995 2024-09-11T09:51:37.967 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.968 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.968 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:37.968 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.183639+0000", 2024-09-11T09:51:37.968 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0069740039999999998 2024-09-11T09:51:37.968 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.968 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.968 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:37.968 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.185689+0000", 2024-09-11T09:51:37.968 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.002049852 2024-09-11T09:51:37.968 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.968 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.968 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:37.968 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.185725+0000", 2024-09-11T09:51:37.969 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.6804999999999997e-05 2024-09-11T09:51:37.969 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.969 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.969 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:37.969 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.185786+0000", 2024-09-11T09:51:37.969 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.0605e-05 2024-09-11T09:51:37.969 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:37.969 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:37.969 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:37.969 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.969 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.969 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4694.0:60 2.0 2:054f2af4:ns::rbd_data.1256a0c02879.000000000000000f:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:37.970 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.186855+0000", 2024-09-11T09:51:37.970 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 14.650205290000001, 2024-09-11T09:51:37.970 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.013682913, 2024-09-11T09:51:37.970 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:37.970 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:37.970 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:37.970 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:37.970 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4694", 2024-09-11T09:51:37.970 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:51:37.970 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 60 2024-09-11T09:51:37.970 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.970 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:37.971 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.971 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:37.971 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.186855+0000", 2024-09-11T09:51:37.971 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:37.971 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.971 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.971 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:37.971 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.186855+0000", 2024-09-11T09:51:37.971 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:37.971 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.971 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.971 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:37.972 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.186856+0000", 2024-09-11T09:51:37.972 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.1200000000000001e-07 2024-09-11T09:51:37.972 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.972 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.972 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:37.972 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.191484+0000", 2024-09-11T09:51:37.972 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0046274159999999996 2024-09-11T09:51:37.972 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.972 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.972 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:37.972 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.191496+0000", 2024-09-11T09:51:37.973 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.2149e-05 2024-09-11T09:51:37.973 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.973 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.973 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:37.973 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.191502+0000", 2024-09-11T09:51:37.973 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.1890000000000004e-06 2024-09-11T09:51:37.973 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.973 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.973 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:37.973 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.191544+0000", 2024-09-11T09:51:37.973 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.1724999999999998e-05 2024-09-11T09:51:37.974 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.974 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.974 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:37.974 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.191641+0000", 2024-09-11T09:51:37.974 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.6939000000000001e-05 2024-09-11T09:51:37.974 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.974 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.974 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 1", 2024-09-11T09:51:37.974 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.191731+0000", 2024-09-11T09:51:37.974 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.0651999999999999e-05 2024-09-11T09:51:37.974 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.974 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.974 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:37.974 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.199852+0000", 2024-09-11T09:51:37.974 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0081204529999999997 2024-09-11T09:51:37.975 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.975 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.975 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:37.975 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.200461+0000", 2024-09-11T09:51:37.975 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00060900899999999998 2024-09-11T09:51:37.975 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.975 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.975 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:37.975 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.200497+0000", 2024-09-11T09:51:37.975 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.5800000000000003e-05 2024-09-11T09:51:37.976 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:37.976 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:37.976 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:37.976 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.200538+0000", 2024-09-11T09:51:37.976 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.1668999999999997e-05 2024-09-11T09:51:37.976 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:37.976 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:37.976 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:37.976 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:37.976 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:37.976 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-11T09:51:37.978 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-11T09:51:37.979 INFO:tasks.ceph.osd.2.smithi017.stderr:2024-09-11T09:51:37.978+0000 7fd20b170640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31672) UID: 0 2024-09-11T09:51:38.078 INFO:tasks.ceph.osd.0.smithi017.stderr:2024-09-11T09:51:38.078+0000 7fb73c0f1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31665) UID: 0 2024-09-11T09:51:38.081 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-11T09:51:38.081 INFO:teuthology.orchestra.run.smithi096.stdout: "ops": [], 2024-09-11T09:51:38.081 INFO:teuthology.orchestra.run.smithi096.stdout: "num_ops": 0 2024-09-11T09:51:38.081 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-11T09:51:38.091 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-11T09:51:38.169 INFO:tasks.workunit.client.0.smithi107.stderr:++ _sudo rbd --device-type nbd map --exclusive rbd/testrbdnbd27121 2024-09-11T09:51:38.170 INFO:tasks.workunit.client.0.smithi107.stderr:++ local cmd 2024-09-11T09:51:38.170 INFO:tasks.workunit.client.0.smithi107.stderr:+++ id -u 2024-09-11T09:51:38.170 INFO:tasks.workunit.client.0.smithi107.stderr:++ '[' 1000 -eq 0 ']' 2024-09-11T09:51:38.170 INFO:tasks.workunit.client.0.smithi107.stderr:+++ which rbd 2024-09-11T09:51:38.172 INFO:tasks.workunit.client.0.smithi107.stderr:++ cmd=/usr/bin/rbd 2024-09-11T09:51:38.172 INFO:tasks.workunit.client.0.smithi107.stderr:++ shift 2024-09-11T09:51:38.173 INFO:tasks.workunit.client.0.smithi107.stderr:++ sudo -nE /usr/bin/rbd --device-type nbd map --exclusive rbd/testrbdnbd27121 2024-09-11T09:51:38.179 INFO:tasks.ceph.osd.0.smithi017.stderr:2024-09-11T09:51:38.179+0000 7fb73c0f1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31665) UID: 0 2024-09-11T09:51:38.193 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-11T09:51:38.193 INFO:teuthology.orchestra.run.smithi096.stdout: "ops": [], 2024-09-11T09:51:38.193 INFO:teuthology.orchestra.run.smithi096.stdout: "complaint_time": 30, 2024-09-11T09:51:38.193 INFO:teuthology.orchestra.run.smithi096.stdout: "num_blocked_ops": 0 2024-09-11T09:51:38.193 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-11T09:51:38.202 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-11T09:51:38.277 INFO:tasks.workunit.client.0.smithi107.stderr:+ DEV=/dev/nbd0 2024-09-11T09:51:38.277 INFO:tasks.workunit.client.0.smithi107.stderr:+ get_pid rbd 2024-09-11T09:51:38.277 INFO:tasks.workunit.client.0.smithi107.stderr:+ local pool=rbd 2024-09-11T09:51:38.278 INFO:tasks.workunit.client.0.smithi107.stderr:+ local ns= 2024-09-11T09:51:38.278 INFO:tasks.workunit.client.0.smithi107.stderr:++ rbd device --device-type nbd --format xml list 2024-09-11T09:51:38.278 INFO:tasks.workunit.client.0.smithi107.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd27121'\''][device='\''/dev/nbd0'\'']/id' 2024-09-11T09:51:38.280 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:51:38.280+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:51:38.307 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T09:51:38.308 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-11T09:51:38.308 INFO:teuthology.orchestra.run.smithi096.stdout: "size": 20, 2024-09-11T09:51:38.308 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 600, 2024-09-11T09:51:38.308 INFO:teuthology.orchestra.run.smithi096.stdout: "ops": [ 2024-09-11T09:51:38.308 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.308 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4230.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-11T09:51:38.308 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:18.490736+0000", 2024-09-11T09:51:38.308 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 79.817098467999998, 2024-09-11T09:51:38.309 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0059097569999999999, 2024-09-11T09:51:38.309 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:38.309 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:38.309 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:38.309 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:38.309 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4230", 2024-09-11T09:51:38.309 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.17:0/4101109576", 2024-09-11T09:51:38.309 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 1 2024-09-11T09:51:38.309 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.309 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:38.309 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.309 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:38.309 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.490736+0000", 2024-09-11T09:51:38.310 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:38.310 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.310 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.310 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:38.310 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.490736+0000", 2024-09-11T09:51:38.310 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:38.310 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.310 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.310 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:38.310 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.490742+0000", 2024-09-11T09:51:38.310 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.48e-06 2024-09-11T09:51:38.310 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.310 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.311 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:38.311 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.490749+0000", 2024-09-11T09:51:38.311 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.4569999999999999e-06 2024-09-11T09:51:38.311 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.311 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.311 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:38.311 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.490762+0000", 2024-09-11T09:51:38.311 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.2721000000000001e-05 2024-09-11T09:51:38.311 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.311 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.311 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:38.311 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.490774+0000", 2024-09-11T09:51:38.311 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1823e-05 2024-09-11T09:51:38.312 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.312 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.312 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:38.312 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.490846+0000", 2024-09-11T09:51:38.312 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.1878000000000005e-05 2024-09-11T09:51:38.312 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.312 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.312 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for peered", 2024-09-11T09:51:38.312 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.490857+0000", 2024-09-11T09:51:38.312 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1017e-05 2024-09-11T09:51:38.312 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.312 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.312 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:38.313 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.493856+0000", 2024-09-11T09:51:38.313 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00299885 2024-09-11T09:51:38.313 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.313 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.313 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:38.313 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.494020+0000", 2024-09-11T09:51:38.313 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00016436599999999999 2024-09-11T09:51:38.313 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.313 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.313 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 2", 2024-09-11T09:51:38.313 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.494186+0000", 2024-09-11T09:51:38.313 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00016562799999999999 2024-09-11T09:51:38.313 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.314 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.314 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:38.314 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.495407+0000", 2024-09-11T09:51:38.314 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0012217420000000001 2024-09-11T09:51:38.314 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.314 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.314 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:38.314 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.496579+0000", 2024-09-11T09:51:38.314 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001171844 2024-09-11T09:51:38.314 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.314 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.314 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:38.314 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.496620+0000", 2024-09-11T09:51:38.315 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.0479000000000003e-05 2024-09-11T09:51:38.315 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.315 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.315 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:38.315 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.496646+0000", 2024-09-11T09:51:38.315 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.6472e-05 2024-09-11T09:51:38.315 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:38.315 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:38.315 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:38.315 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.315 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.315 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4230.0:5 2.2 2:5cea7035:::rbd_info:head [create,write 0~8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17)", 2024-09-11T09:51:38.315 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:19.492662+0000", 2024-09-11T09:51:38.315 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 78.815172846999999, 2024-09-11T09:51:38.315 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0043824759999999997, 2024-09-11T09:51:38.316 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:38.316 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:38.316 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:38.316 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:38.316 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4230", 2024-09-11T09:51:38.316 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.17:0/4101109576", 2024-09-11T09:51:38.316 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 5 2024-09-11T09:51:38.316 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.316 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:38.316 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.316 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:38.317 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.492662+0000", 2024-09-11T09:51:38.317 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:38.317 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.317 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.317 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:38.317 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.492662+0000", 2024-09-11T09:51:38.317 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:38.317 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.317 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.317 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:38.317 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.492664+0000", 2024-09-11T09:51:38.317 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.3599999999999999e-06 2024-09-11T09:51:38.318 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.318 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.318 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:38.318 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.492675+0000", 2024-09-11T09:51:38.318 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1182000000000001e-05 2024-09-11T09:51:38.318 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.318 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.318 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:38.318 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.492693+0000", 2024-09-11T09:51:38.318 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.7031999999999999e-05 2024-09-11T09:51:38.318 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.318 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.318 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:38.318 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.492702+0000", 2024-09-11T09:51:38.319 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.5540000000000003e-06 2024-09-11T09:51:38.319 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.319 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.319 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:38.319 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.492767+0000", 2024-09-11T09:51:38.319 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.5411000000000001e-05 2024-09-11T09:51:38.319 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.319 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.319 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op must wait for map", 2024-09-11T09:51:38.319 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.492773+0000", 2024-09-11T09:51:38.319 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.1039999999999998e-06 2024-09-11T09:51:38.319 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.319 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.319 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:38.320 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.494354+0000", 2024-09-11T09:51:38.320 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0015814049999999999 2024-09-11T09:51:38.320 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.320 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.320 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:38.320 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.494476+0000", 2024-09-11T09:51:38.320 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000122089 2024-09-11T09:51:38.320 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.320 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.320 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 1", 2024-09-11T09:51:38.320 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.494577+0000", 2024-09-11T09:51:38.320 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000100536 2024-09-11T09:51:38.320 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.320 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.321 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:38.321 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.495770+0000", 2024-09-11T09:51:38.321 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0011932469999999999 2024-09-11T09:51:38.321 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.321 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.321 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:38.321 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.496966+0000", 2024-09-11T09:51:38.321 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001196003 2024-09-11T09:51:38.321 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.321 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.321 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:38.321 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.497011+0000", 2024-09-11T09:51:38.322 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.5229999999999999e-05 2024-09-11T09:51:38.322 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.322 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.322 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:38.322 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:19.497044+0000", 2024-09-11T09:51:38.322 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.3322999999999999e-05 2024-09-11T09:51:38.322 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:38.322 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:38.322 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:38.322 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.322 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.322 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4230.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-09-11T09:51:38.322 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:20.498634+0000", 2024-09-11T09:51:38.322 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 77.809200446999995, 2024-09-11T09:51:38.323 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0050648539999999997, 2024-09-11T09:51:38.323 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:38.323 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:38.323 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:38.323 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:38.323 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4230", 2024-09-11T09:51:38.323 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.17:0/4101109576", 2024-09-11T09:51:38.323 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 7 2024-09-11T09:51:38.323 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.323 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:38.323 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.323 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:38.324 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.498634+0000", 2024-09-11T09:51:38.324 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:38.324 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.324 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.324 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:38.324 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.498634+0000", 2024-09-11T09:51:38.324 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:38.324 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.324 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.324 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:38.324 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.498637+0000", 2024-09-11T09:51:38.324 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.1919999999999999e-06 2024-09-11T09:51:38.325 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.325 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.325 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:38.325 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.498648+0000", 2024-09-11T09:51:38.325 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1116e-05 2024-09-11T09:51:38.325 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.325 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.325 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:38.325 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.498666+0000", 2024-09-11T09:51:38.325 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.8411999999999999e-05 2024-09-11T09:51:38.325 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.325 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.325 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:38.326 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.498675+0000", 2024-09-11T09:51:38.326 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.4059999999999994e-06 2024-09-11T09:51:38.326 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.326 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.326 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:38.326 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.498745+0000", 2024-09-11T09:51:38.326 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.9764000000000005e-05 2024-09-11T09:51:38.326 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.326 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.326 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op must wait for map", 2024-09-11T09:51:38.326 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.498754+0000", 2024-09-11T09:51:38.326 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.5969999999999992e-06 2024-09-11T09:51:38.327 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.327 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.327 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:38.327 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.500385+0000", 2024-09-11T09:51:38.327 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0016308360000000001 2024-09-11T09:51:38.327 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.327 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.327 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:38.327 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.500473+0000", 2024-09-11T09:51:38.327 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.7966000000000004e-05 2024-09-11T09:51:38.327 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.327 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.328 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 1", 2024-09-11T09:51:38.328 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.500591+0000", 2024-09-11T09:51:38.328 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000117895 2024-09-11T09:51:38.328 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.328 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.328 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:38.328 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.501685+0000", 2024-09-11T09:51:38.328 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001094886 2024-09-11T09:51:38.328 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.328 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.328 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:38.328 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.503627+0000", 2024-09-11T09:51:38.329 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001941199 2024-09-11T09:51:38.329 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.329 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.329 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:38.329 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.503667+0000", 2024-09-11T09:51:38.329 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.0645000000000001e-05 2024-09-11T09:51:38.329 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.329 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.329 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:38.329 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.503699+0000", 2024-09-11T09:51:38.329 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.1940000000000003e-05 2024-09-11T09:51:38.330 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:38.330 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:38.330 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:38.330 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.330 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.330 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4688.0:32 2.2 2:45941fd2:::rbd_data.1250f5e25b80.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:38.330 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:22.896607+0000", 2024-09-11T09:51:38.330 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 15.411227844000001, 2024-09-11T09:51:38.330 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.038655179999999997, 2024-09-11T09:51:38.330 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:38.330 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:38.330 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:38.330 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:38.331 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4688", 2024-09-11T09:51:38.331 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:51:38.331 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 32 2024-09-11T09:51:38.331 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.331 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:38.331 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.331 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:38.331 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.896607+0000", 2024-09-11T09:51:38.331 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:38.331 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.331 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.331 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:38.331 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.896607+0000", 2024-09-11T09:51:38.332 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:38.332 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.332 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.332 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:38.332 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.896608+0000", 2024-09-11T09:51:38.332 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.0690000000000001e-06 2024-09-11T09:51:38.332 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.332 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.332 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:38.332 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.914461+0000", 2024-09-11T09:51:38.332 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.017852903 2024-09-11T09:51:38.332 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.332 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.333 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:38.333 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.914495+0000", 2024-09-11T09:51:38.333 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.4301000000000002e-05 2024-09-11T09:51:38.333 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.333 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.333 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:38.333 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.914522+0000", 2024-09-11T09:51:38.333 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.7127e-05 2024-09-11T09:51:38.333 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.333 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.333 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:38.333 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.914583+0000", 2024-09-11T09:51:38.333 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.0730999999999998e-05 2024-09-11T09:51:38.334 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.334 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.334 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:38.334 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.914709+0000", 2024-09-11T09:51:38.334 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00012590400000000001 2024-09-11T09:51:38.334 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.334 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.334 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 1", 2024-09-11T09:51:38.334 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.914809+0000", 2024-09-11T09:51:38.334 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.9882000000000004e-05 2024-09-11T09:51:38.334 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.334 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.334 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:38.334 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.932551+0000", 2024-09-11T09:51:38.335 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.017742048999999999 2024-09-11T09:51:38.335 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.335 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.335 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:38.335 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.935144+0000", 2024-09-11T09:51:38.335 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0025927630000000001 2024-09-11T09:51:38.335 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.335 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.335 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:38.335 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.935205+0000", 2024-09-11T09:51:38.335 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.1634e-05 2024-09-11T09:51:38.335 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.335 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.335 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:38.336 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.935262+0000", 2024-09-11T09:51:38.336 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.6817000000000002e-05 2024-09-11T09:51:38.336 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:38.336 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:38.336 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:38.336 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.336 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.336 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_repop(client.4688.0:40 2.7 e53/38 2:fe41411c:::rbd_object_map.1250f5e25b80:head v 53'9, mlcod=53'7)", 2024-09-11T09:51:38.336 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:22.920457+0000", 2024-09-11T09:51:38.336 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 15.387377752999999, 2024-09-11T09:51:38.336 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.011132817, 2024-09-11T09:51:38.336 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:38.337 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:38.337 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:38.337 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:38.337 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.337 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:38.337 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.920457+0000", 2024-09-11T09:51:38.337 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:38.337 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.337 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.338 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:38.338 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.920457+0000", 2024-09-11T09:51:38.338 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:38.338 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.338 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.338 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:38.338 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.920459+0000", 2024-09-11T09:51:38.338 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.6729999999999999e-06 2024-09-11T09:51:38.338 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.338 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.338 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:38.339 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.920469+0000", 2024-09-11T09:51:38.339 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.944e-06 2024-09-11T09:51:38.339 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.339 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.339 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:38.339 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.920495+0000", 2024-09-11T09:51:38.339 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.6401999999999999e-05 2024-09-11T09:51:38.339 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.339 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.339 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:38.339 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.920506+0000", 2024-09-11T09:51:38.339 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.133e-05 2024-09-11T09:51:38.339 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.340 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.340 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:38.340 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.920581+0000", 2024-09-11T09:51:38.340 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.4932999999999996e-05 2024-09-11T09:51:38.340 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.340 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.340 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:38.340 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.920612+0000", 2024-09-11T09:51:38.340 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.0537999999999998e-05 2024-09-11T09:51:38.340 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.340 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.340 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:38.340 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.931537+0000", 2024-09-11T09:51:38.341 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.010925239 2024-09-11T09:51:38.341 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.341 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.341 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:38.341 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.931590+0000", 2024-09-11T09:51:38.341 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.2757999999999999e-05 2024-09-11T09:51:38.341 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:38.341 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:38.342 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:38.342 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.342 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.342 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_repop(client.4688.0:41 2.7 e53/38 2:fe41411c:::rbd_object_map.1250f5e25b80:head v 53'10, mlcod=53'7)", 2024-09-11T09:51:38.342 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:22.921422+0000", 2024-09-11T09:51:38.342 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 15.386412336999999, 2024-09-11T09:51:38.342 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.013734942, 2024-09-11T09:51:38.342 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:38.342 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:38.342 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:38.342 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:38.343 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.343 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:38.343 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921422+0000", 2024-09-11T09:51:38.343 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:38.343 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.343 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.343 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:38.343 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921422+0000", 2024-09-11T09:51:38.343 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:38.343 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.343 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.343 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:38.343 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921426+0000", 2024-09-11T09:51:38.343 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.6150000000000001e-06 2024-09-11T09:51:38.344 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.344 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.344 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:38.344 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921435+0000", 2024-09-11T09:51:38.344 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.9840000000000007e-06 2024-09-11T09:51:38.344 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.344 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.344 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:38.345 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921444+0000", 2024-09-11T09:51:38.345 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.6000000000000007e-06 2024-09-11T09:51:38.345 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.345 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.345 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:38.345 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921450+0000", 2024-09-11T09:51:38.345 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.4690000000000002e-06 2024-09-11T09:51:38.345 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.345 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.345 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:38.345 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921511+0000", 2024-09-11T09:51:38.345 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.0779999999999997e-05 2024-09-11T09:51:38.346 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.346 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.346 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:38.346 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921531+0000", 2024-09-11T09:51:38.346 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.0302999999999999e-05 2024-09-11T09:51:38.346 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.346 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.346 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:38.347 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.935115+0000", 2024-09-11T09:51:38.347 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.013583471 2024-09-11T09:51:38.347 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.347 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.347 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:38.347 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.935157+0000", 2024-09-11T09:51:38.347 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.2719999999999998e-05 2024-09-11T09:51:38.347 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:38.347 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:38.347 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:38.347 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.347 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.347 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4688.0:42 2.3 2:c54742d8:::rbd_data.1250f5e25b80.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:38.348 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:22.932560+0000", 2024-09-11T09:51:38.348 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 15.375274594, 2024-09-11T09:51:38.348 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.017798198000000001, 2024-09-11T09:51:38.348 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:38.348 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:38.348 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:38.348 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:38.348 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4688", 2024-09-11T09:51:38.348 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:51:38.348 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 42 2024-09-11T09:51:38.348 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.348 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:38.348 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.349 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:38.349 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.932560+0000", 2024-09-11T09:51:38.349 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:38.349 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.349 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.349 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:38.349 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.932560+0000", 2024-09-11T09:51:38.349 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:38.349 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.349 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.349 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:38.350 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.932562+0000", 2024-09-11T09:51:38.350 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.17e-06 2024-09-11T09:51:38.350 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.350 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.350 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:38.350 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.938851+0000", 2024-09-11T09:51:38.350 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.006288971 2024-09-11T09:51:38.350 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.350 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.350 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:38.350 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.938865+0000", 2024-09-11T09:51:38.350 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.365e-05 2024-09-11T09:51:38.351 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.351 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.351 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:38.351 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.938871+0000", 2024-09-11T09:51:38.351 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.8850000000000004e-06 2024-09-11T09:51:38.351 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.351 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.351 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:38.351 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.938926+0000", 2024-09-11T09:51:38.351 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.4691999999999997e-05 2024-09-11T09:51:38.351 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.351 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.352 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:38.352 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.939059+0000", 2024-09-11T09:51:38.352 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00013310899999999999 2024-09-11T09:51:38.352 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.352 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.352 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 2", 2024-09-11T09:51:38.352 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.939171+0000", 2024-09-11T09:51:38.352 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000112718 2024-09-11T09:51:38.352 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.352 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.352 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:38.352 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.950262+0000", 2024-09-11T09:51:38.353 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.011091063 2024-09-11T09:51:38.353 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.353 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.353 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:38.353 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.950274+0000", 2024-09-11T09:51:38.353 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1963e-05 2024-09-11T09:51:38.353 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.353 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.353 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:38.353 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.950318+0000", 2024-09-11T09:51:38.353 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.3519999999999997e-05 2024-09-11T09:51:38.353 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.353 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.354 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:38.354 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.950358+0000", 2024-09-11T09:51:38.354 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.0457000000000002e-05 2024-09-11T09:51:38.354 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:38.354 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:38.354 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:38.354 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.354 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.354 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_repop(client.4694.0:34 2.7 e53/38 2:f80bfc0c:ns::rbd_data.1256a0c02879.0000000000000002:head v 53'25, mlcod=53'23)", 2024-09-11T09:51:38.354 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.094944+0000", 2024-09-11T09:51:38.354 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 15.212891202, 2024-09-11T09:51:38.354 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.010896785000000001, 2024-09-11T09:51:38.354 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:38.355 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:38.355 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:38.355 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:38.355 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.355 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:38.355 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.094944+0000", 2024-09-11T09:51:38.355 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:38.355 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.355 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.355 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:38.355 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.094944+0000", 2024-09-11T09:51:38.355 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:38.356 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.356 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.356 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:38.356 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.094944+0000", 2024-09-11T09:51:38.356 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.0699999999999996e-07 2024-09-11T09:51:38.356 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.356 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.356 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:38.356 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.099060+0000", 2024-09-11T09:51:38.356 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0041155580000000001 2024-09-11T09:51:38.356 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.356 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.356 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:38.357 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.099082+0000", 2024-09-11T09:51:38.357 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.2323999999999998e-05 2024-09-11T09:51:38.357 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.357 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.357 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:38.357 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.099094+0000", 2024-09-11T09:51:38.357 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1535000000000001e-05 2024-09-11T09:51:38.357 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.357 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.357 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:38.357 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.099160+0000", 2024-09-11T09:51:38.357 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.6449e-05 2024-09-11T09:51:38.357 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.358 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.358 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:38.358 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.099183+0000", 2024-09-11T09:51:38.358 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.3108e-05 2024-09-11T09:51:38.358 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.358 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.358 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:38.358 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.105776+0000", 2024-09-11T09:51:38.358 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0065922760000000002 2024-09-11T09:51:38.358 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.358 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.358 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:38.359 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.105840+0000", 2024-09-11T09:51:38.359 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.4727999999999994e-05 2024-09-11T09:51:38.359 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:38.359 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:38.359 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:38.359 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.359 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.359 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_repop(client.4694.0:35 2.7 e53/38 2:e8766325:ns::rbd_object_map.1256a0c02879:head v 53'26, mlcod=53'23)", 2024-09-11T09:51:38.359 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.100964+0000", 2024-09-11T09:51:38.359 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 15.206870673999999, 2024-09-11T09:51:38.359 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0049649359999999997, 2024-09-11T09:51:38.359 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:38.359 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:38.359 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:38.360 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:38.360 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.360 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:38.360 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.100964+0000", 2024-09-11T09:51:38.360 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:38.360 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.360 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.360 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:38.360 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.100964+0000", 2024-09-11T09:51:38.360 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:38.360 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.360 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.360 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:38.361 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.100966+0000", 2024-09-11T09:51:38.361 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.6330000000000001e-06 2024-09-11T09:51:38.361 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.361 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.361 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:38.361 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.100979+0000", 2024-09-11T09:51:38.361 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.2813000000000001e-05 2024-09-11T09:51:38.361 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.361 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.361 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:38.361 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.100992+0000", 2024-09-11T09:51:38.361 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.3414999999999999e-05 2024-09-11T09:51:38.361 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.362 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.362 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:38.362 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.101002+0000", 2024-09-11T09:51:38.362 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.0159e-05 2024-09-11T09:51:38.362 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.362 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.362 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:38.362 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.102365+0000", 2024-09-11T09:51:38.362 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001362912 2024-09-11T09:51:38.362 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.362 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.362 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:38.362 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.102382+0000", 2024-09-11T09:51:38.363 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.7433000000000001e-05 2024-09-11T09:51:38.363 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.363 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.363 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:38.363 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.105901+0000", 2024-09-11T09:51:38.363 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.003518476 2024-09-11T09:51:38.363 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.363 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.363 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:38.363 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.105929+0000", 2024-09-11T09:51:38.363 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.8095000000000001e-05 2024-09-11T09:51:38.363 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:38.363 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:38.363 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:38.364 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.364 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.364 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_repop(client.4694.0:36 2.7 e53/38 2:e8766325:ns::rbd_object_map.1256a0c02879:head v 53'27, mlcod=53'23)", 2024-09-11T09:51:38.364 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.101688+0000", 2024-09-11T09:51:38.364 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 15.206146997999999, 2024-09-11T09:51:38.364 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0042633289999999997, 2024-09-11T09:51:38.364 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:38.364 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:38.364 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:38.364 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:38.364 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.364 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:38.365 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.101688+0000", 2024-09-11T09:51:38.365 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:38.365 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.365 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.365 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:38.365 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.101688+0000", 2024-09-11T09:51:38.365 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:38.365 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.365 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.365 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:38.365 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.101689+0000", 2024-09-11T09:51:38.365 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.9299999999999997e-07 2024-09-11T09:51:38.365 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.366 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.366 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:38.366 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.101694+0000", 2024-09-11T09:51:38.366 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.6779999999999997e-06 2024-09-11T09:51:38.366 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.366 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.366 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:38.366 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.101707+0000", 2024-09-11T09:51:38.366 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.2549e-05 2024-09-11T09:51:38.366 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.366 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.366 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:38.367 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.101714+0000", 2024-09-11T09:51:38.367 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.9210000000000003e-06 2024-09-11T09:51:38.367 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.367 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.367 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:38.367 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.102635+0000", 2024-09-11T09:51:38.367 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00092109400000000001 2024-09-11T09:51:38.367 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.367 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.367 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:38.367 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.102649+0000", 2024-09-11T09:51:38.368 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.3913e-05 2024-09-11T09:51:38.368 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.368 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.368 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:38.368 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.105931+0000", 2024-09-11T09:51:38.368 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.003281876 2024-09-11T09:51:38.368 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.368 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.368 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:38.368 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.105951+0000", 2024-09-11T09:51:38.368 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.0505000000000001e-05 2024-09-11T09:51:38.368 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:38.368 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:38.369 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:38.369 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.369 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.369 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_repop(client.4694.0:37 2.7 e53/38 2:e8766325:ns::rbd_object_map.1256a0c02879:head v 53'28, mlcod=53'23)", 2024-09-11T09:51:38.369 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.102274+0000", 2024-09-11T09:51:38.369 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 15.205560948, 2024-09-11T09:51:38.369 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0036897850000000001, 2024-09-11T09:51:38.369 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:38.369 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:38.369 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:38.369 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:38.369 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.369 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:38.370 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.102274+0000", 2024-09-11T09:51:38.370 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:38.370 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.370 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.370 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:38.370 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.102274+0000", 2024-09-11T09:51:38.370 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:38.370 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.370 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.370 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:38.370 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.102274+0000", 2024-09-11T09:51:38.370 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.1699999999999998e-07 2024-09-11T09:51:38.371 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.371 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.371 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:38.371 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.102280+0000", 2024-09-11T09:51:38.371 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.1200000000000001e-06 2024-09-11T09:51:38.371 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.371 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.371 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:38.371 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.102291+0000", 2024-09-11T09:51:38.371 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1032e-05 2024-09-11T09:51:38.371 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.371 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.372 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:38.372 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.102297+0000", 2024-09-11T09:51:38.372 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.4919999999999998e-06 2024-09-11T09:51:38.372 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.372 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.372 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:38.372 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.102902+0000", 2024-09-11T09:51:38.372 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00060473799999999996 2024-09-11T09:51:38.372 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.372 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.373 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:38.373 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.102913+0000", 2024-09-11T09:51:38.373 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.131e-05 2024-09-11T09:51:38.373 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.373 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.373 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:38.373 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.105952+0000", 2024-09-11T09:51:38.373 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0030386200000000001 2024-09-11T09:51:38.373 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.373 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.373 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:38.373 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.105964+0000", 2024-09-11T09:51:38.374 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1856e-05 2024-09-11T09:51:38.374 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:38.374 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:38.374 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:38.374 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.374 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.374 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4694.0:41 2.2 2:5e2505c4:ns::rbd_data.1256a0c02879.0000000000000003:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:38.374 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.107511+0000", 2024-09-11T09:51:38.374 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 15.200323521, 2024-09-11T09:51:38.374 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.018571066000000001, 2024-09-11T09:51:38.374 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:38.374 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:38.374 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:38.375 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:38.375 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4694", 2024-09-11T09:51:38.375 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:51:38.375 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 41 2024-09-11T09:51:38.375 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.375 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:38.375 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.375 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:38.375 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.107511+0000", 2024-09-11T09:51:38.375 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:38.375 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.375 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.375 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:38.376 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.107511+0000", 2024-09-11T09:51:38.376 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:38.376 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.376 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.376 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:38.376 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.107513+0000", 2024-09-11T09:51:38.376 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.3489999999999999e-06 2024-09-11T09:51:38.376 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.376 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.376 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:38.376 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.112704+0000", 2024-09-11T09:51:38.377 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0051918509999999999 2024-09-11T09:51:38.377 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.377 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.377 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:38.377 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.112718+0000", 2024-09-11T09:51:38.377 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.359e-05 2024-09-11T09:51:38.377 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.377 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.377 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:38.377 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.112725+0000", 2024-09-11T09:51:38.377 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.9940000000000003e-06 2024-09-11T09:51:38.378 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.378 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.378 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:38.378 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.112788+0000", 2024-09-11T09:51:38.378 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.2447999999999996e-05 2024-09-11T09:51:38.378 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.378 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.378 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:38.378 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.112937+0000", 2024-09-11T09:51:38.379 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00014921499999999999 2024-09-11T09:51:38.379 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.379 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.379 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 1", 2024-09-11T09:51:38.379 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.113085+0000", 2024-09-11T09:51:38.379 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000148085 2024-09-11T09:51:38.379 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.379 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.379 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:38.379 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.124318+0000", 2024-09-11T09:51:38.379 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.011233158 2024-09-11T09:51:38.379 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.379 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.379 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:38.380 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.125957+0000", 2024-09-11T09:51:38.380 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001638701 2024-09-11T09:51:38.380 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.380 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.380 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:38.380 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.126029+0000", 2024-09-11T09:51:38.380 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.2125000000000005e-05 2024-09-11T09:51:38.380 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.380 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.380 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:38.380 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.126082+0000", 2024-09-11T09:51:38.380 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.3550000000000001e-05 2024-09-11T09:51:38.380 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:38.380 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:38.381 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:38.381 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.381 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.381 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4694.0:44 2.3 2:cd1a7d1c:ns::rbd_data.1256a0c02879.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:38.381 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.112744+0000", 2024-09-11T09:51:38.381 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 15.195090953999999, 2024-09-11T09:51:38.381 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.021672212999999999, 2024-09-11T09:51:38.381 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:38.381 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:38.381 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:38.381 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:38.381 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4694", 2024-09-11T09:51:38.381 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:51:38.382 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 44 2024-09-11T09:51:38.382 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.382 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:38.382 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.382 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:38.382 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.112744+0000", 2024-09-11T09:51:38.382 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:38.382 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.382 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.382 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:38.382 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.112744+0000", 2024-09-11T09:51:38.382 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:38.382 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.382 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.382 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:38.383 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.112745+0000", 2024-09-11T09:51:38.383 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.2900000000000002e-07 2024-09-11T09:51:38.383 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.383 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.383 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:38.383 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.119929+0000", 2024-09-11T09:51:38.383 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0071846230000000002 2024-09-11T09:51:38.383 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.383 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.383 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:38.383 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.119949+0000", 2024-09-11T09:51:38.383 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.9264000000000001e-05 2024-09-11T09:51:38.383 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.384 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.384 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:38.384 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.119959+0000", 2024-09-11T09:51:38.384 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.0465999999999999e-05 2024-09-11T09:51:38.384 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.384 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.384 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:38.384 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.120028+0000", 2024-09-11T09:51:38.384 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.9318000000000003e-05 2024-09-11T09:51:38.385 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.385 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.385 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:38.385 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.120160+0000", 2024-09-11T09:51:38.385 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000132015 2024-09-11T09:51:38.385 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.385 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.385 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 2", 2024-09-11T09:51:38.385 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.120260+0000", 2024-09-11T09:51:38.385 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.9397999999999999e-05 2024-09-11T09:51:38.386 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.386 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.386 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:38.386 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.134246+0000", 2024-09-11T09:51:38.386 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.013986721000000001 2024-09-11T09:51:38.386 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.386 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.386 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:38.386 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.134295+0000", 2024-09-11T09:51:38.386 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.8649000000000001e-05 2024-09-11T09:51:38.386 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.386 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.386 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:38.387 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.134361+0000", 2024-09-11T09:51:38.387 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.6081999999999995e-05 2024-09-11T09:51:38.387 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.387 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.387 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:38.387 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.134416+0000", 2024-09-11T09:51:38.387 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.4848000000000001e-05 2024-09-11T09:51:38.387 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:38.387 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:38.387 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:38.387 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.387 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.387 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4694.0:46 2.3 2:ce0e1469:ns::rbd_data.1256a0c02879.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:38.388 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.119993+0000", 2024-09-11T09:51:38.388 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 15.187841477999999, 2024-09-11T09:51:38.388 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.025840610999999999, 2024-09-11T09:51:38.388 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:38.388 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:38.388 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:38.389 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:38.389 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4694", 2024-09-11T09:51:38.389 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:51:38.389 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 46 2024-09-11T09:51:38.389 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.389 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:38.389 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.389 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:38.389 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.119993+0000", 2024-09-11T09:51:38.389 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:38.389 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.389 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.389 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:38.390 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.119993+0000", 2024-09-11T09:51:38.390 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:38.390 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.390 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.390 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:38.390 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.119995+0000", 2024-09-11T09:51:38.390 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1930000000000001e-06 2024-09-11T09:51:38.390 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.390 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.390 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:38.390 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.128458+0000", 2024-09-11T09:51:38.390 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0084632630000000004 2024-09-11T09:51:38.390 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.391 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.391 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:38.391 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.128478+0000", 2024-09-11T09:51:38.391 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.0353e-05 2024-09-11T09:51:38.391 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.391 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.391 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:38.391 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.128491+0000", 2024-09-11T09:51:38.391 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.2954e-05 2024-09-11T09:51:38.391 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.391 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.391 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:38.391 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.128558+0000", 2024-09-11T09:51:38.392 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.7076999999999995e-05 2024-09-11T09:51:38.392 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.392 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.392 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:38.392 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.128708+0000", 2024-09-11T09:51:38.392 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00014939599999999999 2024-09-11T09:51:38.392 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.392 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.392 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 2", 2024-09-11T09:51:38.392 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.128866+0000", 2024-09-11T09:51:38.392 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000158145 2024-09-11T09:51:38.392 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.393 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.393 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:38.393 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.141045+0000", 2024-09-11T09:51:38.393 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.012179331999999999 2024-09-11T09:51:38.393 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.393 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.393 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:38.393 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.145742+0000", 2024-09-11T09:51:38.393 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0046972050000000003 2024-09-11T09:51:38.393 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.393 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.394 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:38.394 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.145779+0000", 2024-09-11T09:51:38.394 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.6544000000000001e-05 2024-09-11T09:51:38.394 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.394 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.394 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:38.394 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.145834+0000", 2024-09-11T09:51:38.394 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.5148999999999997e-05 2024-09-11T09:51:38.394 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:38.394 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:38.394 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:38.394 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.395 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.395 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4694.0:48 2.3 2:d2090b70:ns::rbd_data.1256a0c02879.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:38.395 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.130540+0000", 2024-09-11T09:51:38.395 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 15.177295212000001, 2024-09-11T09:51:38.395 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.017167789999999999, 2024-09-11T09:51:38.395 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:38.395 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:38.395 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:38.395 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:38.395 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4694", 2024-09-11T09:51:38.395 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:51:38.395 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 48 2024-09-11T09:51:38.395 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.396 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:38.396 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.396 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:38.396 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.130540+0000", 2024-09-11T09:51:38.396 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:38.396 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.396 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.396 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:38.396 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.130540+0000", 2024-09-11T09:51:38.396 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:38.396 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.396 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.397 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:38.397 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.130542+0000", 2024-09-11T09:51:38.397 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.9209999999999999e-06 2024-09-11T09:51:38.397 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.397 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.397 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:38.397 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.134691+0000", 2024-09-11T09:51:38.397 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0041498739999999996 2024-09-11T09:51:38.397 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.397 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.397 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:38.397 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.134705+0000", 2024-09-11T09:51:38.397 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.3669000000000001e-05 2024-09-11T09:51:38.398 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.398 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.398 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:38.398 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.134712+0000", 2024-09-11T09:51:38.398 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.8410000000000002e-06 2024-09-11T09:51:38.398 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.398 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.398 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:38.398 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.134754+0000", 2024-09-11T09:51:38.398 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.2524999999999997e-05 2024-09-11T09:51:38.398 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.399 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.399 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:38.399 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.134875+0000", 2024-09-11T09:51:38.399 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000120332 2024-09-11T09:51:38.399 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.399 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.399 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 2", 2024-09-11T09:51:38.399 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.135005+0000", 2024-09-11T09:51:38.399 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00013024900000000001 2024-09-11T09:51:38.399 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.400 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.400 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:38.400 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.145971+0000", 2024-09-11T09:51:38.400 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.010966203000000001 2024-09-11T09:51:38.400 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.400 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.400 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:38.400 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.147600+0000", 2024-09-11T09:51:38.400 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001628382 2024-09-11T09:51:38.400 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.400 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.400 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:38.400 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.147635+0000", 2024-09-11T09:51:38.400 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.5420000000000003e-05 2024-09-11T09:51:38.401 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.401 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.401 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:38.401 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.147707+0000", 2024-09-11T09:51:38.401 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.2373999999999995e-05 2024-09-11T09:51:38.401 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:38.401 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:38.401 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:38.401 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.401 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.401 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_repop(client.4694.0:49 2.7 e53/38 2:e8766325:ns::rbd_object_map.1256a0c02879:head v 53'33, mlcod=53'31)", 2024-09-11T09:51:38.401 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.135391+0000", 2024-09-11T09:51:38.402 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 15.172443647, 2024-09-11T09:51:38.402 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0051189859999999998, 2024-09-11T09:51:38.402 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:38.402 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:38.402 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:38.402 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:38.402 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.402 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:38.402 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.135391+0000", 2024-09-11T09:51:38.402 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:38.403 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.403 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.403 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:38.403 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.135391+0000", 2024-09-11T09:51:38.403 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:38.403 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.403 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.403 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:38.403 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.135393+0000", 2024-09-11T09:51:38.403 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.4020000000000001e-06 2024-09-11T09:51:38.403 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.403 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.404 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:38.404 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.135400+0000", 2024-09-11T09:51:38.404 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.5129999999999999e-06 2024-09-11T09:51:38.404 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.404 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.404 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:38.404 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.135411+0000", 2024-09-11T09:51:38.404 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1038e-05 2024-09-11T09:51:38.404 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.404 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.404 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:38.404 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.135418+0000", 2024-09-11T09:51:38.404 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.1489999999999997e-06 2024-09-11T09:51:38.405 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.405 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.405 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:38.405 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.135470+0000", 2024-09-11T09:51:38.405 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.2021999999999998e-05 2024-09-11T09:51:38.405 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.405 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.405 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:38.405 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.135501+0000", 2024-09-11T09:51:38.405 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.0349999999999999e-05 2024-09-11T09:51:38.406 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.406 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.406 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:38.406 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.140453+0000", 2024-09-11T09:51:38.406 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0049522409999999996 2024-09-11T09:51:38.406 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.406 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.406 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:38.406 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.140510+0000", 2024-09-11T09:51:38.406 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.7271000000000001e-05 2024-09-11T09:51:38.406 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:38.406 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:38.406 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:38.406 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.407 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.407 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4694.0:54 2.3 2:c44f0253:ns::rbd_data.1256a0c02879.000000000000000b:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:38.407 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.150151+0000", 2024-09-11T09:51:38.407 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 15.157683796000001, 2024-09-11T09:51:38.407 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.015103484, 2024-09-11T09:51:38.407 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:38.407 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:38.407 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:38.407 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:38.407 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4694", 2024-09-11T09:51:38.407 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:51:38.407 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 54 2024-09-11T09:51:38.407 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.407 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:38.408 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.408 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:38.408 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.150151+0000", 2024-09-11T09:51:38.408 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:38.408 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.408 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.408 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:38.408 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.150151+0000", 2024-09-11T09:51:38.408 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:38.408 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.408 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.408 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:38.408 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.150152+0000", 2024-09-11T09:51:38.409 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.2550000000000001e-06 2024-09-11T09:51:38.409 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.409 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.409 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:38.409 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.154528+0000", 2024-09-11T09:51:38.409 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0043758219999999997 2024-09-11T09:51:38.409 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.409 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.409 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:38.409 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.154541+0000", 2024-09-11T09:51:38.410 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.3414999999999999e-05 2024-09-11T09:51:38.410 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.410 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.410 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:38.410 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.154551+0000", 2024-09-11T09:51:38.410 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.4350000000000003e-06 2024-09-11T09:51:38.410 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.410 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.410 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:38.410 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.154615+0000", 2024-09-11T09:51:38.410 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.3627000000000006e-05 2024-09-11T09:51:38.410 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.411 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.411 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:38.411 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.154717+0000", 2024-09-11T09:51:38.411 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000102729 2024-09-11T09:51:38.411 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.411 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.411 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 2", 2024-09-11T09:51:38.411 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.154826+0000", 2024-09-11T09:51:38.411 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000108409 2024-09-11T09:51:38.411 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.411 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.411 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:38.411 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.165094+0000", 2024-09-11T09:51:38.411 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.010268210999999999 2024-09-11T09:51:38.412 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.412 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.412 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:38.412 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.165138+0000", 2024-09-11T09:51:38.412 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.4078e-05 2024-09-11T09:51:38.412 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.412 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.412 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:38.412 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.165204+0000", 2024-09-11T09:51:38.412 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.5716000000000006e-05 2024-09-11T09:51:38.412 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.412 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.412 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:38.413 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.165254+0000", 2024-09-11T09:51:38.413 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.0787e-05 2024-09-11T09:51:38.413 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:38.413 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:38.413 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:38.413 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.413 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.413 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_repop(client.4694.0:55 2.7 e53/38 2:e8766325:ns::rbd_object_map.1256a0c02879:head v 53'37, mlcod=53'34)", 2024-09-11T09:51:38.413 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.150221+0000", 2024-09-11T09:51:38.413 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 15.157613931, 2024-09-11T09:51:38.413 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00530015, 2024-09-11T09:51:38.413 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:38.414 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:38.414 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:38.414 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:38.414 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.414 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:38.414 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.150221+0000", 2024-09-11T09:51:38.414 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:38.414 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.414 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.414 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:38.415 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.150221+0000", 2024-09-11T09:51:38.415 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:38.415 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.415 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.415 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:38.415 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.150221+0000", 2024-09-11T09:51:38.415 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.9100000000000004e-07 2024-09-11T09:51:38.415 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.415 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.415 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:38.415 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.150226+0000", 2024-09-11T09:51:38.415 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.2250000000000002e-06 2024-09-11T09:51:38.415 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.415 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.416 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:38.416 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.150251+0000", 2024-09-11T09:51:38.416 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.4987e-05 2024-09-11T09:51:38.416 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.416 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.416 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:38.416 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.150258+0000", 2024-09-11T09:51:38.416 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.0790000000000003e-06 2024-09-11T09:51:38.416 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.416 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.416 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:38.416 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.150305+0000", 2024-09-11T09:51:38.416 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.6879000000000003e-05 2024-09-11T09:51:38.417 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.417 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.417 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:38.417 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.150320+0000", 2024-09-11T09:51:38.417 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.5268999999999999e-05 2024-09-11T09:51:38.417 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.417 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.417 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:38.417 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.155493+0000", 2024-09-11T09:51:38.417 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0051731390000000002 2024-09-11T09:51:38.418 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.418 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.418 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:38.418 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.155521+0000", 2024-09-11T09:51:38.418 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.8081000000000001e-05 2024-09-11T09:51:38.418 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:38.418 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:38.418 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:38.418 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.418 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.418 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4694.0:56 2.3 2:c462f029:ns::rbd_data.1256a0c02879.000000000000000c:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:38.418 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.154576+0000", 2024-09-11T09:51:38.418 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 15.153258955, 2024-09-11T09:51:38.418 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.020781621, 2024-09-11T09:51:38.419 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:38.419 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:38.419 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:38.419 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:38.419 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4694", 2024-09-11T09:51:38.419 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:51:38.419 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 56 2024-09-11T09:51:38.419 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.419 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:38.419 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.419 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:38.420 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.154576+0000", 2024-09-11T09:51:38.420 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:38.420 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.420 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.420 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:38.420 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.154576+0000", 2024-09-11T09:51:38.420 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:38.420 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.420 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.420 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:38.420 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.154577+0000", 2024-09-11T09:51:38.420 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.2249999999999999e-06 2024-09-11T09:51:38.420 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.421 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.421 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:38.421 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.158783+0000", 2024-09-11T09:51:38.421 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0042063889999999996 2024-09-11T09:51:38.421 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.421 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.421 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:38.421 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.158803+0000", 2024-09-11T09:51:38.421 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.9576999999999999e-05 2024-09-11T09:51:38.421 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.421 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.421 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:38.421 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.158815+0000", 2024-09-11T09:51:38.422 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.2221999999999999e-05 2024-09-11T09:51:38.422 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.422 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.422 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:38.422 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.158902+0000", 2024-09-11T09:51:38.422 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.6390000000000005e-05 2024-09-11T09:51:38.422 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.422 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.422 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:38.422 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.159141+0000", 2024-09-11T09:51:38.422 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000238886 2024-09-11T09:51:38.422 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.422 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.422 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 2", 2024-09-11T09:51:38.423 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.159279+0000", 2024-09-11T09:51:38.423 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000138073 2024-09-11T09:51:38.423 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.423 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.423 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:38.423 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.173297+0000", 2024-09-11T09:51:38.423 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.014018113 2024-09-11T09:51:38.423 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.423 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.423 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:38.423 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.175278+0000", 2024-09-11T09:51:38.423 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0019817390000000002 2024-09-11T09:51:38.423 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.423 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.424 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:38.424 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.175316+0000", 2024-09-11T09:51:38.424 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.7829e-05 2024-09-11T09:51:38.424 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.424 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.424 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:38.424 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.175357+0000", 2024-09-11T09:51:38.424 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.1177999999999998e-05 2024-09-11T09:51:38.424 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:38.424 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:38.424 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:38.424 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.425 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.425 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4694.0:59 2.2 2:41adefc9:ns::rbd_data.1256a0c02879.000000000000000e:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:38.425 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.158843+0000", 2024-09-11T09:51:38.425 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 15.148991389000001, 2024-09-11T09:51:38.425 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.017874207, 2024-09-11T09:51:38.425 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:38.425 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:38.425 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:38.425 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:38.425 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4694", 2024-09-11T09:51:38.425 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:51:38.425 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 59 2024-09-11T09:51:38.425 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.426 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:38.426 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.426 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:38.426 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.158843+0000", 2024-09-11T09:51:38.426 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:38.426 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.426 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.426 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:38.426 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.158843+0000", 2024-09-11T09:51:38.426 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:38.426 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.426 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.427 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:38.427 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.158845+0000", 2024-09-11T09:51:38.427 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.7439999999999999e-06 2024-09-11T09:51:38.427 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.427 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.427 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:38.427 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.162713+0000", 2024-09-11T09:51:38.427 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0038678829999999999 2024-09-11T09:51:38.427 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.427 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.427 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:38.427 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.162728+0000", 2024-09-11T09:51:38.427 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.5114e-05 2024-09-11T09:51:38.428 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.428 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.428 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:38.428 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.162737+0000", 2024-09-11T09:51:38.428 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.1099999999999992e-06 2024-09-11T09:51:38.428 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.428 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.428 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:38.428 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.162799+0000", 2024-09-11T09:51:38.428 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.1934999999999996e-05 2024-09-11T09:51:38.428 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.428 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.428 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:38.429 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.162911+0000", 2024-09-11T09:51:38.429 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000111867 2024-09-11T09:51:38.429 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.429 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.429 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 1", 2024-09-11T09:51:38.429 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.163030+0000", 2024-09-11T09:51:38.429 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000119108 2024-09-11T09:51:38.429 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.429 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.429 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:38.429 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.174086+0000", 2024-09-11T09:51:38.429 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.01105543 2024-09-11T09:51:38.430 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.430 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.430 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:38.430 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.176614+0000", 2024-09-11T09:51:38.430 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0025284940000000001 2024-09-11T09:51:38.430 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.430 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.430 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:38.430 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.176672+0000", 2024-09-11T09:51:38.430 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.7892000000000001e-05 2024-09-11T09:51:38.430 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:38.431 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:38.431 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:38.431 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.176718+0000", 2024-09-11T09:51:38.431 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.5630000000000002e-05 2024-09-11T09:51:38.431 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:38.431 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:38.431 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:38.431 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:38.431 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:38.431 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-11T09:51:38.431 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 5, 0] out_osds: [] dead_osds: [4] live_osds: [0, 1, 2, 3, 5] 2024-09-11T09:51:38.432 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-11T09:51:38.432 INFO:tasks.thrashosds.thrasher:Reviving osd 4 2024-09-11T09:51:38.432 INFO:tasks.ceph.osd.4:Restarting daemon 2024-09-11T09:51:38.432 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-09-11T09:51:38.434 INFO:tasks.workunit.client.0.smithi107.stderr:+ PID=28216 2024-09-11T09:51:38.434 INFO:tasks.workunit.client.0.smithi107.stderr:+ test -n 28216 2024-09-11T09:51:38.434 INFO:tasks.workunit.client.0.smithi107.stderr:+ ps -p 28216 -C rbd-nbd 2024-09-11T09:51:38.434 INFO:tasks.workunit.client.0.smithi107.stdout: PID TTY TIME CMD 2024-09-11T09:51:38.434 INFO:tasks.workunit.client.0.smithi107.stdout: 28216 ? 00:00:00 rbd-nbd 2024-09-11T09:51:38.435 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-11T09:51:38.436 INFO:tasks.workunit.client.0.smithi107.stderr:+ _sudo dd if=/tmp/tmp.rvrcDirNue/data of=/dev/nbd0 bs=1M oflag=direct 2024-09-11T09:51:38.436 INFO:tasks.workunit.client.0.smithi107.stderr:+ local cmd 2024-09-11T09:51:38.436 INFO:tasks.workunit.client.0.smithi107.stderr:++ id -u 2024-09-11T09:51:38.436 INFO:tasks.workunit.client.0.smithi107.stderr:+ '[' 1000 -eq 0 ']' 2024-09-11T09:51:38.436 INFO:tasks.workunit.client.0.smithi107.stderr:++ which dd 2024-09-11T09:51:38.436 INFO:tasks.workunit.client.0.smithi107.stderr:+ cmd=/usr/bin/dd 2024-09-11T09:51:38.437 INFO:tasks.workunit.client.0.smithi107.stderr:+ shift 2024-09-11T09:51:38.437 INFO:tasks.workunit.client.0.smithi107.stderr:+ sudo -nE /usr/bin/dd if=/tmp/tmp.rvrcDirNue/data of=/dev/nbd0 bs=1M oflag=direct 2024-09-11T09:51:38.437 INFO:tasks.ceph.osd.0.smithi017.stderr:2024-09-11T09:51:38.433+0000 7fb73c0f1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31665) UID: 0 2024-09-11T09:51:38.478 INFO:tasks.ceph.osd.4:Started 2024-09-11T09:51:38.478 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-11T09:51:38.534 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:51:38.534+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:51:38.561 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:51:38.561 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [], 2024-09-11T09:51:38.561 INFO:teuthology.orchestra.run.smithi017.stdout: "num_ops": 0 2024-09-11T09:51:38.562 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:51:38.572 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-11T09:51:38.573 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-11T09:51:38.573 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T09:51:38.587 INFO:teuthology.orchestra.run.smithi096.stderr:no valid command found; 10 closest matches: 2024-09-11T09:51:38.588 INFO:teuthology.orchestra.run.smithi096.stderr:0 2024-09-11T09:51:38.588 INFO:teuthology.orchestra.run.smithi096.stderr:1 2024-09-11T09:51:38.588 INFO:teuthology.orchestra.run.smithi096.stderr:2 2024-09-11T09:51:38.588 INFO:teuthology.orchestra.run.smithi096.stderr:abort 2024-09-11T09:51:38.588 INFO:teuthology.orchestra.run.smithi096.stderr:assert 2024-09-11T09:51:38.589 INFO:teuthology.orchestra.run.smithi096.stderr:config diff 2024-09-11T09:51:38.589 INFO:teuthology.orchestra.run.smithi096.stderr:config diff get 2024-09-11T09:51:38.589 INFO:teuthology.orchestra.run.smithi096.stderr:config get 2024-09-11T09:51:38.589 INFO:teuthology.orchestra.run.smithi096.stderr:config help [] 2024-09-11T09:51:38.589 INFO:teuthology.orchestra.run.smithi096.stderr:config set ... 2024-09-11T09:51:38.589 INFO:teuthology.orchestra.run.smithi096.stderr:admin_socket: invalid command 2024-09-11T09:51:38.590 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-11T09:51:38.591 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T09:51:38.592 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-11T09:51:38.593 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-4, ['dump_ops_in_flight'] 2024-09-11T09:51:38.606 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-11T09:51:38.607 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T09:51:38.628 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-11T09:51:38.628 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T09:51:38.628 INFO:teuthology.orchestra.run.smithi017.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-09-11T09:51:38.635 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:51:38.635+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:51:38.648 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-11T09:51:38.648 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T09:51:38.651 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-11T09:51:38.652 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-11T09:51:38.718 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:51:38.718 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:51:38.729 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-11T09:51:38.758 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-11T09:51:38.774 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:51:38.774+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:51:38.814 INFO:tasks.workunit.client.0.smithi107.stderr:64+0 records in 2024-09-11T09:51:38.814 INFO:tasks.workunit.client.0.smithi107.stderr:64+0 records out 2024-09-11T09:51:38.814 INFO:tasks.workunit.client.0.smithi107.stderr:67108864 bytes (67 MB, 64 MiB) copied, 0.452564 s, 148 MB/s 2024-09-11T09:51:38.816 INFO:tasks.workunit.client.0.smithi107.stderr:+ expect_false timeout 10 rbd bench testrbdnbd27121 --io-type write --io-size=1024 --io-total=1024 2024-09-11T09:51:38.816 INFO:tasks.workunit.client.0.smithi107.stderr:+ timeout 10 rbd bench testrbdnbd27121 --io-type write --io-size=1024 --io-total=1024 2024-09-11T09:51:38.836 INFO:tasks.ceph.osd.2.smithi017.stderr:2024-09-11T09:51:38.836+0000 7fd20b170640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31672) UID: 0 2024-09-11T09:51:38.858 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:51:38.858 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:51:38.867 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-11T09:51:38.937 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-09-11T09:51:38.939+0000 7f7b6c7d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31904) UID: 0 2024-09-11T09:51:39.015 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:51:39.015 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:51:39.024 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-11T09:51:39.064 INFO:tasks.workunit.client.0.smithi107.stderr:2024-09-11T09:51:39.065+0000 7ff65affd640 -1 librbd::exclusive_lock::ImageDispatch: 0x7ff640000f60 handle_acquire_lock: failed to acquire exclusive lock: (30) Read-only file system 2024-09-11T09:51:39.064 INFO:tasks.workunit.client.0.smithi107.stderr:rbd: failed to flush: (30) Read-only file system 2024-09-11T09:51:39.064 INFO:tasks.workunit.client.0.smithi107.stderr:2024-09-11T09:51:39.065+0000 7ff65affd640 -1 librbd::io::AioCompletion: 0x55c982893d80 fail: (30) Read-only file system 2024-09-11T09:51:39.064 INFO:tasks.workunit.client.0.smithi107.stderr:bench failed: (30) Read-only file system 2024-09-11T09:51:39.067 INFO:tasks.ceph.osd.2.smithi017.stderr:2024-09-11T09:51:39.067+0000 7fd20b170640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31672) UID: 0 2024-09-11T09:51:39.069 INFO:tasks.workunit.client.0.smithi107.stderr:+ return 0 2024-09-11T09:51:39.069 INFO:tasks.workunit.client.0.smithi107.stderr:+ unmap_device /dev/nbd0 28216 2024-09-11T09:51:39.069 INFO:tasks.workunit.client.0.smithi107.stderr:+ local args=/dev/nbd0 2024-09-11T09:51:39.069 INFO:tasks.workunit.client.0.smithi107.stderr:+ local pid=28216 2024-09-11T09:51:39.069 INFO:tasks.workunit.client.0.smithi107.stderr:+ _sudo rbd device --device-type nbd unmap /dev/nbd0 2024-09-11T09:51:39.069 INFO:tasks.workunit.client.0.smithi107.stderr:+ local cmd 2024-09-11T09:51:39.070 INFO:tasks.workunit.client.0.smithi107.stderr:++ id -u 2024-09-11T09:51:39.070 INFO:tasks.workunit.client.0.smithi107.stderr:+ '[' 1000 -eq 0 ']' 2024-09-11T09:51:39.070 INFO:tasks.workunit.client.0.smithi107.stderr:++ which rbd 2024-09-11T09:51:39.072 INFO:tasks.workunit.client.0.smithi107.stderr:+ cmd=/usr/bin/rbd 2024-09-11T09:51:39.072 INFO:tasks.workunit.client.0.smithi107.stderr:+ shift 2024-09-11T09:51:39.072 INFO:tasks.workunit.client.0.smithi107.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd unmap /dev/nbd0 2024-09-11T09:51:39.132 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:51:39.133 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:51:39.139 INFO:tasks.ceph.osd.2.smithi017.stderr:2024-09-11T09:51:39.139+0000 7fd20b170640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31672) UID: 0 2024-09-11T09:51:39.141 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-11T09:51:39.151 INFO:tasks.workunit.client.0.smithi107.stderr:+ rbd device --device-type nbd list 2024-09-11T09:51:39.151 INFO:tasks.workunit.client.0.smithi107.stderr:+ expect_false grep '^28216\b' 2024-09-11T09:51:39.151 INFO:tasks.workunit.client.0.smithi107.stderr:+ grep '^28216\b' 2024-09-11T09:51:39.197 INFO:tasks.workunit.client.0.smithi107.stderr:+ return 0 2024-09-11T09:51:39.198 INFO:tasks.workunit.client.0.smithi107.stderr:+ ps -C rbd-nbd 2024-09-11T09:51:39.198 INFO:tasks.workunit.client.0.smithi107.stderr:+ expect_false grep '^ *28216\b' 2024-09-11T09:51:39.198 INFO:tasks.workunit.client.0.smithi107.stderr:+ grep '^ *28216\b' 2024-09-11T09:51:39.209 INFO:tasks.workunit.client.0.smithi107.stderr:+ return 0 2024-09-11T09:51:39.209 INFO:tasks.workunit.client.0.smithi107.stderr:+ sleep 0.5 2024-09-11T09:51:39.240 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:51:39.241+0000 7f3c96ec9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31909) UID: 0 2024-09-11T09:51:39.288 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:51:39.289 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:51:39.297 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-11T09:51:39.340 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:51:39.341+0000 7f3c96ec9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31909) UID: 0 2024-09-11T09:51:39.403 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:51:39.404 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:51:39.409 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:51:39.409+0000 7f3a7f013640 -1 Falling back to public interface 2024-09-11T09:51:39.412 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-11T09:51:39.456 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:51:39.456+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:51:39.519 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:51:39.519 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:51:39.528 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-11T09:51:39.569 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:51:39.569+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:51:39.642 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:51:39.642+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:51:39.677 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:51:39.677 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:51:39.686 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-11T09:51:39.710 INFO:tasks.workunit.client.0.smithi107.stderr:+ DEV= 2024-09-11T09:51:39.711 INFO:tasks.workunit.client.0.smithi107.stderr:+ rbd bench testrbdnbd27121 --io-type write --io-size=1024 --io-total=1024 2024-09-11T09:51:39.743 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:51:39.743+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:51:39.749 INFO:teuthology.orchestra.run.smithi017.stderr:noscrub is set 2024-09-11T09:51:39.762 INFO:tasks.workunit.client.0.smithi107.stdout:bench type write io_size 1024 io_threads 16 bytes 1024 pattern sequential 2024-09-11T09:51:39.764 INFO:tasks.workunit.client.0.smithi107.stdout: SEC OPS OPS/SEC BYTES/SEC 2024-09-11T09:51:39.764 INFO:tasks.workunit.client.0.smithi107.stdout:elapsed: 0 ops: 1 ops/sec: 1000.02 bytes/sec: 1000 KiB/s 2024-09-11T09:51:39.773 INFO:tasks.workunit.client.0.smithi107.stderr:++ _sudo rbd device --device-type nbd map rbd/testrbdnbd27121 2024-09-11T09:51:39.773 INFO:tasks.workunit.client.0.smithi107.stderr:++ local cmd 2024-09-11T09:51:39.773 INFO:tasks.workunit.client.0.smithi107.stderr:+++ id -u 2024-09-11T09:51:39.774 INFO:tasks.workunit.client.0.smithi107.stderr:++ '[' 1000 -eq 0 ']' 2024-09-11T09:51:39.774 INFO:tasks.workunit.client.0.smithi107.stderr:+++ which rbd 2024-09-11T09:51:39.776 INFO:tasks.workunit.client.0.smithi107.stderr:++ cmd=/usr/bin/rbd 2024-09-11T09:51:39.776 INFO:tasks.workunit.client.0.smithi107.stderr:++ shift 2024-09-11T09:51:39.776 INFO:tasks.workunit.client.0.smithi107.stderr:++ sudo -nE /usr/bin/rbd device --device-type nbd map rbd/testrbdnbd27121 2024-09-11T09:51:39.800 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:51:39.800 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:51:39.809 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-11T09:51:39.844 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:51:39.844+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:51:39.877 INFO:tasks.workunit.client.0.smithi107.stderr:+ DEV=/dev/nbd0 2024-09-11T09:51:39.878 INFO:tasks.workunit.client.0.smithi107.stderr:+ get_pid rbd 2024-09-11T09:51:39.878 INFO:tasks.workunit.client.0.smithi107.stderr:+ local pool=rbd 2024-09-11T09:51:39.878 INFO:tasks.workunit.client.0.smithi107.stderr:+ local ns= 2024-09-11T09:51:39.878 INFO:tasks.workunit.client.0.smithi107.stderr:++ rbd device --device-type nbd --format xml list 2024-09-11T09:51:39.878 INFO:tasks.workunit.client.0.smithi107.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd27121'\''][device='\''/dev/nbd0'\'']/id' 2024-09-11T09:51:39.912 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:51:39.912 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:51:39.922 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-11T09:51:39.940 INFO:tasks.workunit.client.0.smithi107.stderr:+ PID=28334 2024-09-11T09:51:39.940 INFO:tasks.workunit.client.0.smithi107.stderr:+ test -n 28334 2024-09-11T09:51:39.940 INFO:tasks.workunit.client.0.smithi107.stderr:+ ps -p 28334 -C rbd-nbd 2024-09-11T09:51:39.945 INFO:tasks.ceph.osd.0.smithi017.stderr:2024-09-11T09:51:39.945+0000 7fb73c0f1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31665) UID: 0 2024-09-11T09:51:39.951 INFO:tasks.workunit.client.0.smithi107.stdout: PID TTY TIME CMD 2024-09-11T09:51:39.951 INFO:tasks.workunit.client.0.smithi107.stdout: 28334 ? 00:00:00 rbd-nbd 2024-09-11T09:51:39.952 INFO:tasks.workunit.client.0.smithi107.stderr:+ unmap_device testrbdnbd27121 28334 2024-09-11T09:51:39.952 INFO:tasks.workunit.client.0.smithi107.stderr:+ local args=testrbdnbd27121 2024-09-11T09:51:39.952 INFO:tasks.workunit.client.0.smithi107.stderr:+ local pid=28334 2024-09-11T09:51:39.952 INFO:tasks.workunit.client.0.smithi107.stderr:+ _sudo rbd device --device-type nbd unmap testrbdnbd27121 2024-09-11T09:51:39.952 INFO:tasks.workunit.client.0.smithi107.stderr:+ local cmd 2024-09-11T09:51:39.953 INFO:tasks.workunit.client.0.smithi107.stderr:++ id -u 2024-09-11T09:51:39.953 INFO:tasks.workunit.client.0.smithi107.stderr:+ '[' 1000 -eq 0 ']' 2024-09-11T09:51:39.953 INFO:tasks.workunit.client.0.smithi107.stderr:++ which rbd 2024-09-11T09:51:39.955 INFO:tasks.workunit.client.0.smithi107.stderr:+ cmd=/usr/bin/rbd 2024-09-11T09:51:39.955 INFO:tasks.workunit.client.0.smithi107.stderr:+ shift 2024-09-11T09:51:39.955 INFO:tasks.workunit.client.0.smithi107.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd unmap testrbdnbd27121 2024-09-11T09:51:40.025 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:51:40.025 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:51:40.033 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-11T09:51:40.042 INFO:tasks.workunit.client.0.smithi107.stderr:+ rbd device --device-type nbd list 2024-09-11T09:51:40.042 INFO:tasks.workunit.client.0.smithi107.stderr:+ expect_false grep '^28334\b' 2024-09-11T09:51:40.042 INFO:tasks.workunit.client.0.smithi107.stderr:+ grep '^28334\b' 2024-09-11T09:51:40.077 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-09-11T09:51:40.078+0000 7f7b6c7d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31904) UID: 0 2024-09-11T09:51:40.088 INFO:tasks.workunit.client.0.smithi107.stderr:+ return 0 2024-09-11T09:51:40.089 INFO:tasks.workunit.client.0.smithi107.stderr:+ ps -C rbd-nbd 2024-09-11T09:51:40.089 INFO:tasks.workunit.client.0.smithi107.stderr:+ expect_false grep '^ *28334\b' 2024-09-11T09:51:40.089 INFO:tasks.workunit.client.0.smithi107.stderr:+ grep '^ *28334\b' 2024-09-11T09:51:40.100 INFO:tasks.workunit.client.0.smithi107.stderr:+ return 0 2024-09-11T09:51:40.100 INFO:tasks.workunit.client.0.smithi107.stderr:+ sleep 0.5 2024-09-11T09:51:40.136 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:51:40.136 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:51:40.145 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-11T09:51:40.185 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:51:40.186+0000 7f3c96ec9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31909) UID: 0 2024-09-11T09:51:40.247 INFO:tasks.ceph.osd.0.smithi017.stderr:2024-09-11T09:51:40.248+0000 7fb73c0f1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31665) UID: 0 2024-09-11T09:51:40.287 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:51:40.287 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:51:40.296 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-11T09:51:40.348 INFO:tasks.ceph.osd.2.smithi017.stderr:2024-09-11T09:51:40.348+0000 7fd20b170640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31672) UID: 0 2024-09-11T09:51:40.397 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:51:40.398 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:51:40.406 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-11T09:51:40.450 INFO:tasks.ceph.osd.2.smithi017.stderr:2024-09-11T09:51:40.450+0000 7fd20b170640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31672) UID: 0 2024-09-11T09:51:40.519 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:51:40.520 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:51:40.528 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-11T09:51:40.569 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:51:40.569+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:51:40.601 INFO:tasks.workunit.client.0.smithi107.stderr:+ DEV= 2024-09-11T09:51:40.601 INFO:tasks.workunit.client.0.smithi107.stderr:+ rbd snap create rbd/testrbdnbd27121@snap 2024-09-11T09:51:40.650 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-09-11T09:51:40.652+0000 7f7b6c7d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31904) UID: 0 2024-09-11T09:51:40.652 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T09:51:40.677 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:51:40.678 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:51:40.687 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-11T09:51:40.751 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:51:40.751+0000 7f3c96ec9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31909) UID: 0 2024-09-11T09:51:40.765 INFO:tasks.workunit.client.0.smithi107.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2024-09-11T09:51:40.773 INFO:tasks.workunit.client.0.smithi107.stderr:++ _sudo rbd device --device-type nbd map rbd/testrbdnbd27121@snap 2024-09-11T09:51:40.773 INFO:tasks.workunit.client.0.smithi107.stderr:++ local cmd 2024-09-11T09:51:40.773 INFO:tasks.workunit.client.0.smithi107.stderr:+++ id -u 2024-09-11T09:51:40.774 INFO:tasks.workunit.client.0.smithi107.stderr:++ '[' 1000 -eq 0 ']' 2024-09-11T09:51:40.774 INFO:tasks.workunit.client.0.smithi107.stderr:+++ which rbd 2024-09-11T09:51:40.776 INFO:tasks.workunit.client.0.smithi107.stderr:++ cmd=/usr/bin/rbd 2024-09-11T09:51:40.776 INFO:tasks.workunit.client.0.smithi107.stderr:++ shift 2024-09-11T09:51:40.776 INFO:tasks.workunit.client.0.smithi107.stderr:++ sudo -nE /usr/bin/rbd device --device-type nbd map rbd/testrbdnbd27121@snap 2024-09-11T09:51:40.782 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-11T09:51:40.782 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T09:51:40.801 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-11T09:51:40.801 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T09:51:40.818 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-11T09:51:40.818 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T09:51:40.835 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-11T09:51:40.835 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T09:51:40.835 INFO:teuthology.orchestra.run.smithi017.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-09-11T09:51:40.843 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:51:40.843 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 20, 2024-09-11T09:51:40.843 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 600, 2024-09-11T09:51:40.843 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [ 2024-09-11T09:51:40.843 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.843 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4688.0:34 2.4 e53/32 2:2367385a:::rbd_data.1250f5e25b80.0000000000000002:head v 53'6, mlcod=53'4)", 2024-09-11T09:51:40.844 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.907415+0000", 2024-09-11T09:51:40.844 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 17.935584429999999, 2024-09-11T09:51:40.844 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.011285778999999999, 2024-09-11T09:51:40.844 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:40.844 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:40.844 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:40.844 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:40.844 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.844 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:40.844 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.907415+0000", 2024-09-11T09:51:40.844 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:40.844 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.845 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.845 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:40.845 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.907415+0000", 2024-09-11T09:51:40.845 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:40.845 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.845 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.845 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:40.845 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.907417+0000", 2024-09-11T09:51:40.845 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.2910000000000002e-06 2024-09-11T09:51:40.845 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.845 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.845 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:40.845 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.910477+0000", 2024-09-11T09:51:40.846 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0030597829999999999 2024-09-11T09:51:40.846 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.846 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.846 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:40.846 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.910501+0000", 2024-09-11T09:51:40.846 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.3332e-05 2024-09-11T09:51:40.846 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.846 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.846 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:40.846 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.910514+0000", 2024-09-11T09:51:40.846 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3957e-05 2024-09-11T09:51:40.846 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.846 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.847 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:40.847 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.910612+0000", 2024-09-11T09:51:40.847 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.7075000000000001e-05 2024-09-11T09:51:40.847 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.847 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.847 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:40.847 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.910640+0000", 2024-09-11T09:51:40.847 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.8768999999999998e-05 2024-09-11T09:51:40.847 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.847 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.848 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:40.848 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.918640+0000", 2024-09-11T09:51:40.848 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0079991899999999998 2024-09-11T09:51:40.848 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.848 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.848 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:40.848 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.918701+0000", 2024-09-11T09:51:40.848 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.1382000000000003e-05 2024-09-11T09:51:40.848 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:40.848 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:40.848 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:40.848 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.848 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.849 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4688.0:38 2.5 e53/32 2:abde2694:::rbd_data.1250f5e25b80.0000000000000004:head v 53'1, mlcod=0'0)", 2024-09-11T09:51:40.849 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.915432+0000", 2024-09-11T09:51:40.849 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 17.927568050000001, 2024-09-11T09:51:40.849 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.015849652999999998, 2024-09-11T09:51:40.849 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:40.849 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:40.849 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:40.849 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:40.849 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.849 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:40.849 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.915432+0000", 2024-09-11T09:51:40.849 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:40.850 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.850 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.850 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:40.850 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.915432+0000", 2024-09-11T09:51:40.850 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:40.850 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.850 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.850 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:40.850 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.915432+0000", 2024-09-11T09:51:40.850 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.91e-07 2024-09-11T09:51:40.850 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.850 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.850 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:40.851 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.924943+0000", 2024-09-11T09:51:40.851 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0095110630000000002 2024-09-11T09:51:40.851 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.851 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.851 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:40.851 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.924960+0000", 2024-09-11T09:51:40.851 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6915999999999999e-05 2024-09-11T09:51:40.851 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.851 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.851 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:40.851 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.924968+0000", 2024-09-11T09:51:40.851 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.6999999999999997e-06 2024-09-11T09:51:40.851 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.852 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.852 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:40.852 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.925025+0000", 2024-09-11T09:51:40.852 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.6949e-05 2024-09-11T09:51:40.852 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.852 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.852 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:40.852 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.925043+0000", 2024-09-11T09:51:40.852 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7784000000000001e-05 2024-09-11T09:51:40.852 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.852 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.852 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:40.852 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.931237+0000", 2024-09-11T09:51:40.853 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0061935389999999996 2024-09-11T09:51:40.853 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.853 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.853 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:40.853 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.931281+0000", 2024-09-11T09:51:40.853 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.4511000000000001e-05 2024-09-11T09:51:40.853 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:40.853 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:40.853 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:40.853 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.853 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.853 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:40 2.7 2:fe41411c:::rbd_object_map.1250f5e25b80:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:40.853 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.919652+0000", 2024-09-11T09:51:40.854 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 17.923347382999999, 2024-09-11T09:51:40.854 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.012187073, 2024-09-11T09:51:40.854 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:40.854 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:40.854 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:40.854 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:40.854 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:51:40.854 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:51:40.854 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 40 2024-09-11T09:51:40.854 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.854 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:40.854 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.854 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:40.854 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919652+0000", 2024-09-11T09:51:40.855 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:40.855 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.855 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.855 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:40.855 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919652+0000", 2024-09-11T09:51:40.855 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:40.855 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.855 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.855 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:40.855 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919654+0000", 2024-09-11T09:51:40.855 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.88e-06 2024-09-11T09:51:40.855 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.855 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.856 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:40.856 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919669+0000", 2024-09-11T09:51:40.856 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.5271e-05 2024-09-11T09:51:40.856 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.856 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.856 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:40.856 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919687+0000", 2024-09-11T09:51:40.856 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7448999999999998e-05 2024-09-11T09:51:40.856 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.856 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.856 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:40.856 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919695+0000", 2024-09-11T09:51:40.856 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.1149999999999994e-06 2024-09-11T09:51:40.857 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.857 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.857 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:40.857 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919757+0000", 2024-09-11T09:51:40.857 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.2348999999999995e-05 2024-09-11T09:51:40.857 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.857 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.857 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:40.857 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919875+0000", 2024-09-11T09:51:40.857 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00011750499999999999 2024-09-11T09:51:40.857 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.857 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.857 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 5", 2024-09-11T09:51:40.858 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.920201+0000", 2024-09-11T09:51:40.858 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00032628099999999999 2024-09-11T09:51:40.858 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.858 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.858 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:40.858 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.922089+0000", 2024-09-11T09:51:40.858 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.001887978 2024-09-11T09:51:40.858 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.858 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.858 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:40.859 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.931752+0000", 2024-09-11T09:51:40.859 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0096626309999999997 2024-09-11T09:51:40.859 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.859 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.859 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:40.859 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.931794+0000", 2024-09-11T09:51:40.859 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.2849999999999998e-05 2024-09-11T09:51:40.859 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.859 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.860 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:40.860 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.931839+0000", 2024-09-11T09:51:40.860 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.4764e-05 2024-09-11T09:51:40.860 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:40.860 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:40.860 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:40.860 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.860 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.861 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:41 2.7 2:fe41411c:::rbd_object_map.1250f5e25b80:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:40.861 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.919716+0000", 2024-09-11T09:51:40.861 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 17.923283539, 2024-09-11T09:51:40.861 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.015641605999999999, 2024-09-11T09:51:40.861 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:40.861 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:40.861 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:40.861 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:40.862 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:51:40.862 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:51:40.862 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 41 2024-09-11T09:51:40.862 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.862 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:40.862 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.862 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:40.862 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919716+0000", 2024-09-11T09:51:40.863 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:40.863 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.863 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.863 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:40.863 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919716+0000", 2024-09-11T09:51:40.863 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:40.863 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.863 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.864 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:40.864 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919716+0000", 2024-09-11T09:51:40.864 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.9999999999999998e-07 2024-09-11T09:51:40.864 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.864 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.864 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:40.864 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919720+0000", 2024-09-11T09:51:40.864 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.6440000000000002e-06 2024-09-11T09:51:40.865 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.865 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.865 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:40.865 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919729+0000", 2024-09-11T09:51:40.865 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.1759999999999999e-06 2024-09-11T09:51:40.865 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.865 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.865 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:40.866 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919736+0000", 2024-09-11T09:51:40.866 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.5390000000000004e-06 2024-09-11T09:51:40.866 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.866 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.866 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:40.866 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.920669+0000", 2024-09-11T09:51:40.866 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00093275599999999997 2024-09-11T09:51:40.867 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.867 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.867 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:40.867 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.920797+0000", 2024-09-11T09:51:40.867 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00012893400000000001 2024-09-11T09:51:40.867 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.867 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.867 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 5", 2024-09-11T09:51:40.868 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.921269+0000", 2024-09-11T09:51:40.868 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00047128600000000001 2024-09-11T09:51:40.868 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.868 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.868 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:40.868 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.923198+0000", 2024-09-11T09:51:40.868 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0019289369999999999 2024-09-11T09:51:40.868 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.869 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.869 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:40.869 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.935300+0000", 2024-09-11T09:51:40.869 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.012102392 2024-09-11T09:51:40.869 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.869 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.869 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:40.869 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.935332+0000", 2024-09-11T09:51:40.870 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.2292000000000002e-05 2024-09-11T09:51:40.870 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.870 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.870 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:40.870 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.935358+0000", 2024-09-11T09:51:40.870 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.525e-05 2024-09-11T09:51:40.870 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:40.870 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:40.871 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:40.871 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.871 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.871 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:48 2.7 2:fe41411c:::rbd_object_map.1250f5e25b80:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:40.871 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.938799+0000", 2024-09-11T09:51:40.871 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 17.904200429999999, 2024-09-11T09:51:40.871 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0041642340000000002, 2024-09-11T09:51:40.871 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:40.871 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:40.872 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:40.872 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:40.872 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:51:40.872 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:51:40.872 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 48 2024-09-11T09:51:40.872 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.872 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:40.872 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.873 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:40.873 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.938799+0000", 2024-09-11T09:51:40.873 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:40.873 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.873 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.873 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:40.873 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.938799+0000", 2024-09-11T09:51:40.873 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:40.874 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.874 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.874 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:40.874 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.938800+0000", 2024-09-11T09:51:40.874 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.71e-07 2024-09-11T09:51:40.874 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.874 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.874 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:40.874 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.938805+0000", 2024-09-11T09:51:40.874 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.3140000000000003e-06 2024-09-11T09:51:40.874 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.874 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.874 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:40.874 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.938816+0000", 2024-09-11T09:51:40.875 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0231000000000001e-05 2024-09-11T09:51:40.875 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.875 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.875 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:40.875 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.938822+0000", 2024-09-11T09:51:40.875 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.9550000000000003e-06 2024-09-11T09:51:40.875 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.875 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.875 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:40.875 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.938868+0000", 2024-09-11T09:51:40.875 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.5562999999999997e-05 2024-09-11T09:51:40.875 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.875 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.876 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:40.876 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.938918+0000", 2024-09-11T09:51:40.876 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.0317e-05 2024-09-11T09:51:40.876 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.876 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.876 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 5", 2024-09-11T09:51:40.876 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.939114+0000", 2024-09-11T09:51:40.876 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00019538300000000001 2024-09-11T09:51:40.876 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.876 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.876 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:40.876 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.939663+0000", 2024-09-11T09:51:40.876 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00054932199999999998 2024-09-11T09:51:40.876 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.877 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.877 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:40.877 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.942910+0000", 2024-09-11T09:51:40.877 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0032473839999999999 2024-09-11T09:51:40.877 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.877 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.877 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:40.877 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.942940+0000", 2024-09-11T09:51:40.877 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.9306999999999999e-05 2024-09-11T09:51:40.877 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.877 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.877 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:40.877 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.942963+0000", 2024-09-11T09:51:40.877 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.3587e-05 2024-09-11T09:51:40.878 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:40.878 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:40.878 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:40.878 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.878 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.878 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4688.0:49 2.5 e53/32 2:b6dd9fc9:::rbd_data.1250f5e25b80.0000000000000009:head v 53'2, mlcod=0'0)", 2024-09-11T09:51:40.878 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.944712+0000", 2024-09-11T09:51:40.878 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 17.898287590999999, 2024-09-11T09:51:40.878 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.016185069999999999, 2024-09-11T09:51:40.878 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:40.878 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:40.878 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:40.878 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:40.879 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.879 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:40.879 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.944712+0000", 2024-09-11T09:51:40.879 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:40.879 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.879 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.879 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:40.879 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.944712+0000", 2024-09-11T09:51:40.879 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:40.880 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.880 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.880 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:40.880 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.944713+0000", 2024-09-11T09:51:40.880 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0139999999999999e-06 2024-09-11T09:51:40.880 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.880 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.880 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:40.880 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.955518+0000", 2024-09-11T09:51:40.880 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.01080542 2024-09-11T09:51:40.880 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.880 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.880 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:40.881 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.955536+0000", 2024-09-11T09:51:40.881 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7818000000000001e-05 2024-09-11T09:51:40.881 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.881 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.881 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:40.881 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.955544+0000", 2024-09-11T09:51:40.881 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.1699999999999997e-06 2024-09-11T09:51:40.881 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.881 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.881 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:40.881 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.955607+0000", 2024-09-11T09:51:40.881 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.2451000000000002e-05 2024-09-11T09:51:40.882 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.882 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.882 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:40.882 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.955632+0000", 2024-09-11T09:51:40.882 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.5335000000000001e-05 2024-09-11T09:51:40.882 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.882 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.882 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:40.882 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.960826+0000", 2024-09-11T09:51:40.882 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0051938560000000002 2024-09-11T09:51:40.882 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.882 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.882 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:40.882 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.960897+0000", 2024-09-11T09:51:40.883 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.1006000000000004e-05 2024-09-11T09:51:40.883 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:40.883 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:40.883 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:40.883 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.883 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.883 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:53 2.7 2:fe41411c:::rbd_object_map.1250f5e25b80:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:40.883 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.951291+0000", 2024-09-11T09:51:40.883 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 17.891708601000001, 2024-09-11T09:51:40.883 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0044964999999999996, 2024-09-11T09:51:40.883 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:40.883 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:40.883 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:40.883 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:40.884 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:51:40.884 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:51:40.884 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 53 2024-09-11T09:51:40.884 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.884 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:40.884 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.884 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:40.884 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951291+0000", 2024-09-11T09:51:40.884 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:40.884 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.884 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.884 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:40.884 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951291+0000", 2024-09-11T09:51:40.885 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:40.885 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.885 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.885 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:40.885 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951293+0000", 2024-09-11T09:51:40.885 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7889999999999999e-06 2024-09-11T09:51:40.885 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.885 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.885 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:40.885 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951302+0000", 2024-09-11T09:51:40.885 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.9819999999999997e-06 2024-09-11T09:51:40.885 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.885 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.886 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:40.886 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951324+0000", 2024-09-11T09:51:40.886 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.2141e-05 2024-09-11T09:51:40.886 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.886 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.886 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:40.886 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951338+0000", 2024-09-11T09:51:40.886 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3750999999999999e-05 2024-09-11T09:51:40.886 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.886 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.886 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:40.886 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951409+0000", 2024-09-11T09:51:40.886 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.1658999999999999e-05 2024-09-11T09:51:40.887 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.887 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.887 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:40.887 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951517+0000", 2024-09-11T09:51:40.887 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00010746799999999999 2024-09-11T09:51:40.887 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.887 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.887 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 5", 2024-09-11T09:51:40.887 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951942+0000", 2024-09-11T09:51:40.887 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00042524300000000001 2024-09-11T09:51:40.888 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.888 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.888 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:40.888 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.952872+0000", 2024-09-11T09:51:40.888 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00092976300000000001 2024-09-11T09:51:40.888 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.888 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.888 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:40.888 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.955682+0000", 2024-09-11T09:51:40.888 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0028099459999999998 2024-09-11T09:51:40.889 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.889 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.889 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:40.889 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.955746+0000", 2024-09-11T09:51:40.889 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.4084999999999994e-05 2024-09-11T09:51:40.889 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.889 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.889 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:40.889 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.955787+0000", 2024-09-11T09:51:40.889 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.1672999999999999e-05 2024-09-11T09:51:40.889 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:40.889 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:40.889 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:40.890 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.890 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.890 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4688.0:52 2.4 e53/32 2:26d5c2f0:::rbd_data.1250f5e25b80.000000000000000b:head v 53'8, mlcod=53'6)", 2024-09-11T09:51:40.890 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.963677+0000", 2024-09-11T09:51:40.890 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 17.879323046, 2024-09-11T09:51:40.890 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0089437969999999999, 2024-09-11T09:51:40.890 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:40.890 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:40.890 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:40.890 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:40.890 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.890 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:40.890 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.963677+0000", 2024-09-11T09:51:40.891 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:40.891 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.891 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.891 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:40.891 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.963677+0000", 2024-09-11T09:51:40.891 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:40.891 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.891 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.891 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:40.891 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.963677+0000", 2024-09-11T09:51:40.891 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.7199999999999998e-07 2024-09-11T09:51:40.891 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.891 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.892 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:40.892 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.965506+0000", 2024-09-11T09:51:40.892 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.001829064 2024-09-11T09:51:40.892 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.892 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.892 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:40.892 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.965538+0000", 2024-09-11T09:51:40.892 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.1306999999999997e-05 2024-09-11T09:51:40.892 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.892 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.892 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:40.892 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.965553+0000", 2024-09-11T09:51:40.893 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.4924e-05 2024-09-11T09:51:40.893 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.893 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.893 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:40.893 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.965663+0000", 2024-09-11T09:51:40.893 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000110315 2024-09-11T09:51:40.893 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.893 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.893 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:40.893 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.965692+0000", 2024-09-11T09:51:40.893 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.934e-05 2024-09-11T09:51:40.893 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.893 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.894 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:40.894 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.972537+0000", 2024-09-11T09:51:40.894 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0068444969999999997 2024-09-11T09:51:40.894 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.894 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.894 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:40.894 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.972620+0000", 2024-09-11T09:51:40.894 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.3578000000000005e-05 2024-09-11T09:51:40.894 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:40.894 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:40.895 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:40.895 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.895 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.895 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4688.0:54 2.4 e53/32 2:239e915b:::rbd_data.1250f5e25b80.000000000000000c:head v 53'9, mlcod=53'7)", 2024-09-11T09:51:40.895 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.987028+0000", 2024-09-11T09:51:40.895 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 17.855971129, 2024-09-11T09:51:40.895 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0094155650000000007, 2024-09-11T09:51:40.895 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:40.895 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:40.895 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:40.895 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:40.895 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.895 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:40.896 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.987028+0000", 2024-09-11T09:51:40.896 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:40.896 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.896 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.896 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:40.896 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.987028+0000", 2024-09-11T09:51:40.896 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:40.896 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.896 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.896 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:40.896 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.987030+0000", 2024-09-11T09:51:40.896 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6130000000000001e-06 2024-09-11T09:51:40.896 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.897 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.897 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:40.897 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.988575+0000", 2024-09-11T09:51:40.897 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.001545193 2024-09-11T09:51:40.897 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.897 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.897 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:40.897 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.988604+0000", 2024-09-11T09:51:40.897 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.9173e-05 2024-09-11T09:51:40.897 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.897 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.897 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:40.897 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.988620+0000", 2024-09-11T09:51:40.898 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.5517999999999999e-05 2024-09-11T09:51:40.898 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.898 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.898 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:40.898 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.988704+0000", 2024-09-11T09:51:40.898 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.3588e-05 2024-09-11T09:51:40.898 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.898 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.898 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:40.898 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.988735+0000", 2024-09-11T09:51:40.898 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.1643000000000002e-05 2024-09-11T09:51:40.898 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.898 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.899 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:40.899 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.996383+0000", 2024-09-11T09:51:40.899 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0076478099999999997 2024-09-11T09:51:40.899 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.899 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.899 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:40.899 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.996444+0000", 2024-09-11T09:51:40.899 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.1026999999999997e-05 2024-09-11T09:51:40.899 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:40.899 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:40.899 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:40.899 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.899 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.899 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4688.0:56 2.4 e53/32 2:2167012f:::rbd_data.1250f5e25b80.000000000000000d:head v 53'10, mlcod=53'7)", 2024-09-11T09:51:40.900 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.993894+0000", 2024-09-11T09:51:40.900 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 17.849105587, 2024-09-11T09:51:40.900 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0099268559999999995, 2024-09-11T09:51:40.900 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:40.900 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:40.900 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:40.900 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:40.900 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.900 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:40.900 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.993894+0000", 2024-09-11T09:51:40.900 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:40.900 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.900 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.901 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:40.901 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.993894+0000", 2024-09-11T09:51:40.901 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:40.901 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.901 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.901 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:40.901 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.993896+0000", 2024-09-11T09:51:40.901 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6249999999999999e-06 2024-09-11T09:51:40.902 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.902 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.902 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:40.902 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.997606+0000", 2024-09-11T09:51:40.902 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.003710737 2024-09-11T09:51:40.902 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.903 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.903 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:40.903 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.997628+0000", 2024-09-11T09:51:40.903 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.1971e-05 2024-09-11T09:51:40.903 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.904 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.904 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:40.904 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.997640+0000", 2024-09-11T09:51:40.904 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1481000000000001e-05 2024-09-11T09:51:40.904 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.904 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.905 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:40.905 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.997712+0000", 2024-09-11T09:51:40.905 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.2255000000000005e-05 2024-09-11T09:51:40.905 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.905 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.905 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:40.905 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.997738+0000", 2024-09-11T09:51:40.906 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.6050999999999998e-05 2024-09-11T09:51:40.906 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.906 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.906 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:40.906 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.003779+0000", 2024-09-11T09:51:40.906 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0060410960000000001 2024-09-11T09:51:40.906 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.907 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.907 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:40.907 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.003821+0000", 2024-09-11T09:51:40.907 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.1640000000000001e-05 2024-09-11T09:51:40.907 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:40.907 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:40.907 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:40.908 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.908 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.908 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "pg_update_log_missing(2.4 epoch 53/32 rep_tid 16 entries 53'11 (0'0) error 2:2906e6a7:::rbd_header.1250f5e25b80:head by client.4688.0:59 0.000000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 53'7)", 2024-09-11T09:51:40.908 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.998956+0000", 2024-09-11T09:51:40.908 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 17.844043417999998, 2024-09-11T09:51:40.908 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.004934127, 2024-09-11T09:51:40.909 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:40.909 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:40.909 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "reached pg", 2024-09-11T09:51:40.909 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:40.909 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.909 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:40.910 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.998956+0000", 2024-09-11T09:51:40.910 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:40.910 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.910 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.910 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:40.910 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.998956+0000", 2024-09-11T09:51:40.910 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:40.910 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.911 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.911 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:40.911 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.998957+0000", 2024-09-11T09:51:40.911 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.139e-06 2024-09-11T09:51:40.911 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.911 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.911 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:40.912 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.998985+0000", 2024-09-11T09:51:40.912 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.7912e-05 2024-09-11T09:51:40.912 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.912 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.912 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:40.912 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.999011+0000", 2024-09-11T09:51:40.912 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.6061e-05 2024-09-11T09:51:40.912 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.913 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.913 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:40.913 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.999028+0000", 2024-09-11T09:51:40.913 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7065999999999999e-05 2024-09-11T09:51:40.913 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.913 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.913 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:40.914 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.001919+0000", 2024-09-11T09:51:40.914 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.002891167 2024-09-11T09:51:40.914 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.914 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.914 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:40.914 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.003890+0000", 2024-09-11T09:51:40.915 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0019707819999999999 2024-09-11T09:51:40.915 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:40.915 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:40.915 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:40.915 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.915 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.916 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4694.0:34 2.7 2:f80bfc0c:ns::rbd_data.1256a0c02879.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:40.916 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.085183+0000", 2024-09-11T09:51:40.916 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 17.757816904999999, 2024-09-11T09:51:40.916 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.020908599, 2024-09-11T09:51:40.916 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:40.917 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:40.917 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:40.917 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:40.917 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4694", 2024-09-11T09:51:40.917 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:51:40.918 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 34 2024-09-11T09:51:40.918 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.918 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:40.918 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.918 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:40.918 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.085183+0000", 2024-09-11T09:51:40.918 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:40.918 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.919 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.919 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:40.919 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.085183+0000", 2024-09-11T09:51:40.919 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:40.919 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.919 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.919 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:40.919 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.085183+0000", 2024-09-11T09:51:40.919 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.4300000000000002e-07 2024-09-11T09:51:40.920 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.920 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.920 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:40.920 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094337+0000", 2024-09-11T09:51:40.920 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0091532769999999996 2024-09-11T09:51:40.920 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.920 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.921 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:40.921 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094362+0000", 2024-09-11T09:51:40.921 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.5287999999999999e-05 2024-09-11T09:51:40.921 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.921 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.921 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:40.921 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094375+0000", 2024-09-11T09:51:40.921 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2446000000000001e-05 2024-09-11T09:51:40.922 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.922 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.922 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:40.922 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094455+0000", 2024-09-11T09:51:40.922 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.0092999999999994e-05 2024-09-11T09:51:40.922 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.923 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.923 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:40.923 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094618+0000", 2024-09-11T09:51:40.923 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000163747 2024-09-11T09:51:40.923 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.923 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.924 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 5", 2024-09-11T09:51:40.924 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094783+0000", 2024-09-11T09:51:40.924 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000165038 2024-09-11T09:51:40.924 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.924 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.924 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:40.925 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.105853+0000", 2024-09-11T09:51:40.925 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.011069294 2024-09-11T09:51:40.925 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.925 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.925 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:40.926 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.106004+0000", 2024-09-11T09:51:40.926 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00015123900000000001 2024-09-11T09:51:40.926 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.926 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.926 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:40.926 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.106062+0000", 2024-09-11T09:51:40.926 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.8205000000000003e-05 2024-09-11T09:51:40.926 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.927 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.927 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:40.927 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.106091+0000", 2024-09-11T09:51:40.927 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.9128999999999998e-05 2024-09-11T09:51:40.927 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:40.927 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:40.927 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:40.927 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.927 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.927 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4694.0:35 2.7 2:e8766325:ns::rbd_object_map.1256a0c02879:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:40.927 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.094405+0000", 2024-09-11T09:51:40.927 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 17.748594269000002, 2024-09-11T09:51:40.927 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.012623564, 2024-09-11T09:51:40.928 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:40.928 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:40.928 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:40.928 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:40.928 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4694", 2024-09-11T09:51:40.928 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:51:40.928 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 35 2024-09-11T09:51:40.928 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.928 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:40.928 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.928 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:40.928 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094405+0000", 2024-09-11T09:51:40.928 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:40.929 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.929 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.929 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:40.929 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094405+0000", 2024-09-11T09:51:40.929 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:40.929 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.929 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.929 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:40.929 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094407+0000", 2024-09-11T09:51:40.929 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.739e-06 2024-09-11T09:51:40.929 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.929 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.929 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:40.929 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094411+0000", 2024-09-11T09:51:40.930 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.3340000000000003e-06 2024-09-11T09:51:40.930 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.930 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.930 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:40.930 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094419+0000", 2024-09-11T09:51:40.930 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.8909999999999995e-06 2024-09-11T09:51:40.930 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.930 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.930 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:40.930 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094424+0000", 2024-09-11T09:51:40.930 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.7639999999999996e-06 2024-09-11T09:51:40.930 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.930 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.931 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:40.931 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.100209+0000", 2024-09-11T09:51:40.931 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0057850489999999996 2024-09-11T09:51:40.931 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.931 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.931 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:40.931 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.100341+0000", 2024-09-11T09:51:40.931 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000131438 2024-09-11T09:51:40.931 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.931 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.931 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 5", 2024-09-11T09:51:40.931 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.100805+0000", 2024-09-11T09:51:40.931 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00046471499999999998 2024-09-11T09:51:40.932 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.932 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.932 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:40.932 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.106148+0000", 2024-09-11T09:51:40.932 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0053426460000000004 2024-09-11T09:51:40.932 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.932 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.932 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:40.932 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.106929+0000", 2024-09-11T09:51:40.932 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00078161300000000001 2024-09-11T09:51:40.932 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.932 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.932 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:40.933 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.106985+0000", 2024-09-11T09:51:40.933 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.5223999999999998e-05 2024-09-11T09:51:40.933 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.933 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.933 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:40.933 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.107029+0000", 2024-09-11T09:51:40.933 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.4150999999999998e-05 2024-09-11T09:51:40.933 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:40.933 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:40.933 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:40.933 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.933 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.933 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4694.0:36 2.7 2:e8766325:ns::rbd_object_map.1256a0c02879:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:40.934 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.094452+0000", 2024-09-11T09:51:40.934 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 17.748547258999999, 2024-09-11T09:51:40.934 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.012680383, 2024-09-11T09:51:40.934 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:40.934 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:40.934 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:40.934 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:40.934 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4694", 2024-09-11T09:51:40.934 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:51:40.934 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 36 2024-09-11T09:51:40.934 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.934 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:40.934 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.934 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:40.935 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094452+0000", 2024-09-11T09:51:40.935 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:40.935 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.935 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.935 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:40.935 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094452+0000", 2024-09-11T09:51:40.935 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:40.935 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.935 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.935 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:40.935 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094453+0000", 2024-09-11T09:51:40.935 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.1800000000000002e-07 2024-09-11T09:51:40.935 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.936 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.936 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:40.936 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094455+0000", 2024-09-11T09:51:40.936 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.7020000000000001e-06 2024-09-11T09:51:40.936 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.936 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.936 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:40.936 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094463+0000", 2024-09-11T09:51:40.937 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.1999999999999997e-06 2024-09-11T09:51:40.937 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.937 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.937 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:40.937 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094467+0000", 2024-09-11T09:51:40.937 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.729e-06 2024-09-11T09:51:40.937 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.937 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.937 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:40.937 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.101269+0000", 2024-09-11T09:51:40.937 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0068018799999999997 2024-09-11T09:51:40.937 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.938 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.938 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:40.938 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.101353+0000", 2024-09-11T09:51:40.938 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.3827999999999998e-05 2024-09-11T09:51:40.938 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.938 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.938 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 5", 2024-09-11T09:51:40.938 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.101588+0000", 2024-09-11T09:51:40.938 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00023467099999999999 2024-09-11T09:51:40.938 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.938 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.938 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:40.938 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.106182+0000", 2024-09-11T09:51:40.939 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0045939489999999999 2024-09-11T09:51:40.939 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.939 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.939 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:40.939 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.107041+0000", 2024-09-11T09:51:40.939 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00085965800000000002 2024-09-11T09:51:40.939 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.939 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.939 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:40.939 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.107084+0000", 2024-09-11T09:51:40.939 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.261e-05 2024-09-11T09:51:40.939 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.939 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.939 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:40.940 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.107133+0000", 2024-09-11T09:51:40.940 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.8838000000000002e-05 2024-09-11T09:51:40.940 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:40.940 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:40.940 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:40.940 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.940 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.940 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4694.0:37 2.7 2:e8766325:ns::rbd_object_map.1256a0c02879:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:40.940 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.094473+0000", 2024-09-11T09:51:40.940 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 17.748526108, 2024-09-11T09:51:40.940 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.012778484999999999, 2024-09-11T09:51:40.940 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:40.941 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:40.941 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:40.941 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:40.941 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4694", 2024-09-11T09:51:40.941 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:51:40.941 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 37 2024-09-11T09:51:40.941 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.941 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:40.941 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.941 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:40.941 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094473+0000", 2024-09-11T09:51:40.941 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:40.941 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.942 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.942 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:40.942 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094473+0000", 2024-09-11T09:51:40.942 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:40.942 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.942 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.942 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:40.942 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094474+0000", 2024-09-11T09:51:40.942 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.9699999999999996e-07 2024-09-11T09:51:40.942 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.942 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.942 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:40.942 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094477+0000", 2024-09-11T09:51:40.942 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.151e-06 2024-09-11T09:51:40.943 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.943 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.943 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:40.943 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094483+0000", 2024-09-11T09:51:40.943 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.6330000000000001e-06 2024-09-11T09:51:40.943 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.943 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.943 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:40.943 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094487+0000", 2024-09-11T09:51:40.943 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.5399999999999997e-06 2024-09-11T09:51:40.943 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.943 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.943 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:40.944 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.101866+0000", 2024-09-11T09:51:40.944 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.007378768 2024-09-11T09:51:40.944 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.944 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.944 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:40.944 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.101960+0000", 2024-09-11T09:51:40.944 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.3861000000000001e-05 2024-09-11T09:51:40.944 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.944 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.944 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 5", 2024-09-11T09:51:40.944 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.102197+0000", 2024-09-11T09:51:40.944 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00023723200000000001 2024-09-11T09:51:40.944 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.945 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.945 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:40.945 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.106222+0000", 2024-09-11T09:51:40.945 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0040249689999999998 2024-09-11T09:51:40.945 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.945 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.945 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:40.945 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.107141+0000", 2024-09-11T09:51:40.945 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00091913900000000002 2024-09-11T09:51:40.945 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.945 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.945 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:40.945 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.107198+0000", 2024-09-11T09:51:40.946 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.6564999999999998e-05 2024-09-11T09:51:40.946 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.946 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.946 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:40.946 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.107252+0000", 2024-09-11T09:51:40.946 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.4030000000000003e-05 2024-09-11T09:51:40.946 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:40.946 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:40.946 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:40.946 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.946 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.946 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4694.0:33 2.4 e53/32 2:3bfc818c:ns::rbd_data.1256a0c02879.0000000000000001:head v 53'14, mlcod=53'12)", 2024-09-11T09:51:40.946 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.096496+0000", 2024-09-11T09:51:40.946 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 17.746503534999999, 2024-09-11T09:51:40.947 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0099139310000000008, 2024-09-11T09:51:40.947 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:40.947 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:40.947 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:40.947 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:40.947 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.947 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:40.947 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.096496+0000", 2024-09-11T09:51:40.948 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:40.948 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.948 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.948 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:40.948 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.096496+0000", 2024-09-11T09:51:40.948 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:40.948 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.948 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.948 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:40.948 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.096498+0000", 2024-09-11T09:51:40.948 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6309999999999999e-06 2024-09-11T09:51:40.949 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.949 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.949 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:40.949 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.098087+0000", 2024-09-11T09:51:40.949 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0015889019999999999 2024-09-11T09:51:40.949 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.949 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.949 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:40.949 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.098112+0000", 2024-09-11T09:51:40.949 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.5369000000000001e-05 2024-09-11T09:51:40.949 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.949 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.950 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:40.950 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.098127+0000", 2024-09-11T09:51:40.950 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.5218999999999999e-05 2024-09-11T09:51:40.950 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.950 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.950 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:40.950 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.098209+0000", 2024-09-11T09:51:40.950 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.2120999999999995e-05 2024-09-11T09:51:40.950 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.950 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.950 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:40.950 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.098237+0000", 2024-09-11T09:51:40.950 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.7657999999999998e-05 2024-09-11T09:51:40.951 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.951 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.951 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:40.951 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.106376+0000", 2024-09-11T09:51:40.951 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0081386749999999997 2024-09-11T09:51:40.951 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.951 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.951 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:40.951 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.106410+0000", 2024-09-11T09:51:40.951 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.4356000000000001e-05 2024-09-11T09:51:40.951 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:40.951 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:40.951 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:40.952 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.952 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.952 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4694.0:49 2.7 2:e8766325:ns::rbd_object_map.1256a0c02879:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:40.952 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.134806+0000", 2024-09-11T09:51:40.952 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 17.708193466000001, 2024-09-11T09:51:40.952 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0061629049999999998, 2024-09-11T09:51:40.952 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:40.952 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:40.952 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:40.952 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:40.952 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4694", 2024-09-11T09:51:40.952 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:51:40.953 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 49 2024-09-11T09:51:40.953 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.953 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:40.953 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.953 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:40.953 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.134806+0000", 2024-09-11T09:51:40.953 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:40.953 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.953 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.953 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:40.953 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.134806+0000", 2024-09-11T09:51:40.953 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:40.953 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.954 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.954 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:40.954 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.134808+0000", 2024-09-11T09:51:40.954 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.9870000000000002e-06 2024-09-11T09:51:40.954 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.954 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.954 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:40.954 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.134818+0000", 2024-09-11T09:51:40.954 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.5599999999999999e-06 2024-09-11T09:51:40.954 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.954 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.954 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:40.954 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.134835+0000", 2024-09-11T09:51:40.955 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7399000000000001e-05 2024-09-11T09:51:40.955 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.955 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.955 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:40.955 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.134845+0000", 2024-09-11T09:51:40.955 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0285000000000001e-05 2024-09-11T09:51:40.955 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.955 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.955 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:40.955 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.134911+0000", 2024-09-11T09:51:40.955 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.5654999999999994e-05 2024-09-11T09:51:40.955 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.955 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.956 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:40.956 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.135023+0000", 2024-09-11T09:51:40.956 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00011186799999999999 2024-09-11T09:51:40.956 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.956 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.956 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 5", 2024-09-11T09:51:40.956 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.135290+0000", 2024-09-11T09:51:40.956 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00026725300000000001 2024-09-11T09:51:40.956 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.956 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.956 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:40.956 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.136506+0000", 2024-09-11T09:51:40.956 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0012161139999999999 2024-09-11T09:51:40.957 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.957 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.957 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:40.957 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.140884+0000", 2024-09-11T09:51:40.957 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0043781990000000002 2024-09-11T09:51:40.957 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.957 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.957 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:40.957 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.140923+0000", 2024-09-11T09:51:40.957 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.8813999999999998e-05 2024-09-11T09:51:40.957 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.958 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.958 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:40.958 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.140969+0000", 2024-09-11T09:51:40.958 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.5770999999999999e-05 2024-09-11T09:51:40.958 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:40.958 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:40.958 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:40.958 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.958 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.958 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4694.0:42 2.5 e53/32 2:a4fc00ec:ns::rbd_data.1256a0c02879.0000000000000004:head v 53'3, mlcod=53'1)", 2024-09-11T09:51:40.958 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.136646+0000", 2024-09-11T09:51:40.958 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 17.706353159999999, 2024-09-11T09:51:40.959 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.012545501000000001, 2024-09-11T09:51:40.959 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:40.959 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:40.959 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:40.959 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:40.959 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.959 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:40.959 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.136646+0000", 2024-09-11T09:51:40.959 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:40.959 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.959 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.959 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:40.959 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.136646+0000", 2024-09-11T09:51:40.960 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:40.960 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.960 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.960 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:40.960 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.136647+0000", 2024-09-11T09:51:40.960 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.2900000000000002e-07 2024-09-11T09:51:40.960 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.960 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.960 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:40.960 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.143620+0000", 2024-09-11T09:51:40.960 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0069727440000000003 2024-09-11T09:51:40.960 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.960 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.960 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:40.961 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.143642+0000", 2024-09-11T09:51:40.961 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.1846000000000001e-05 2024-09-11T09:51:40.961 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.961 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.961 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:40.961 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.143654+0000", 2024-09-11T09:51:40.961 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2081e-05 2024-09-11T09:51:40.961 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.961 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.961 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:40.961 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.143717+0000", 2024-09-11T09:51:40.961 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.3443999999999998e-05 2024-09-11T09:51:40.961 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.962 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.962 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:40.962 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.143740+0000", 2024-09-11T09:51:40.962 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.2626999999999999e-05 2024-09-11T09:51:40.962 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.962 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.962 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:40.962 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.149142+0000", 2024-09-11T09:51:40.962 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0054017630000000004 2024-09-11T09:51:40.962 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.962 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.962 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:40.962 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.149192+0000", 2024-09-11T09:51:40.962 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.0067e-05 2024-09-11T09:51:40.963 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:40.963 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:40.963 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:40.963 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.963 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.963 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4694.0:53 2.7 2:e8766325:ns::rbd_object_map.1256a0c02879:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:40.963 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.147978+0000", 2024-09-11T09:51:40.963 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 17.695021676, 2024-09-11T09:51:40.963 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0044549769999999997, 2024-09-11T09:51:40.963 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:40.963 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:40.963 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:40.963 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:40.964 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4694", 2024-09-11T09:51:40.964 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:51:40.964 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 53 2024-09-11T09:51:40.964 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.964 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:40.964 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.964 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:40.964 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.147978+0000", 2024-09-11T09:51:40.964 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:40.964 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.964 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.964 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:40.964 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.147978+0000", 2024-09-11T09:51:40.964 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:40.965 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.965 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.965 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:40.965 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.147980+0000", 2024-09-11T09:51:40.965 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.8309999999999999e-06 2024-09-11T09:51:40.965 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.965 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.965 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:40.965 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.147989+0000", 2024-09-11T09:51:40.965 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.9639999999999992e-06 2024-09-11T09:51:40.965 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.965 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.965 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:40.966 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.148007+0000", 2024-09-11T09:51:40.966 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7884999999999999e-05 2024-09-11T09:51:40.966 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.966 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.966 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:40.966 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.148017+0000", 2024-09-11T09:51:40.966 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0895e-05 2024-09-11T09:51:40.966 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.966 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.966 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:40.966 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.148091+0000", 2024-09-11T09:51:40.966 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.3709999999999997e-05 2024-09-11T09:51:40.966 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.966 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.967 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:40.967 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.148189+0000", 2024-09-11T09:51:40.967 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.7403000000000002e-05 2024-09-11T09:51:40.967 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.967 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.967 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 5", 2024-09-11T09:51:40.967 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.148504+0000", 2024-09-11T09:51:40.967 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00031589900000000002 2024-09-11T09:51:40.967 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.967 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.967 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:40.967 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.149514+0000", 2024-09-11T09:51:40.967 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0010096 2024-09-11T09:51:40.967 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.968 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.968 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:40.968 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.152367+0000", 2024-09-11T09:51:40.968 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0028532309999999999 2024-09-11T09:51:40.968 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.968 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.968 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:40.968 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.152402+0000", 2024-09-11T09:51:40.969 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.4232e-05 2024-09-11T09:51:40.969 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.969 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.969 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:40.969 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.152433+0000", 2024-09-11T09:51:40.969 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.1327e-05 2024-09-11T09:51:40.969 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:40.969 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:40.969 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:40.969 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.969 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.969 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4694.0:55 2.7 2:e8766325:ns::rbd_object_map.1256a0c02879:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:40.969 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.149838+0000", 2024-09-11T09:51:40.970 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 17.693161943, 2024-09-11T09:51:40.970 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0060695159999999996, 2024-09-11T09:51:40.970 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:40.970 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:40.970 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:40.970 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:40.970 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4694", 2024-09-11T09:51:40.970 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:51:40.970 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 55 2024-09-11T09:51:40.970 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.970 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:40.970 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.970 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:40.970 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.149838+0000", 2024-09-11T09:51:40.971 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:40.971 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.971 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.971 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:40.971 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.149838+0000", 2024-09-11T09:51:40.971 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:40.971 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.971 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.971 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:40.971 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.149839+0000", 2024-09-11T09:51:40.971 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.119e-06 2024-09-11T09:51:40.971 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.971 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.971 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:40.972 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.149845+0000", 2024-09-11T09:51:40.972 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.7810000000000002e-06 2024-09-11T09:51:40.972 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.972 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.972 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:40.972 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.149856+0000", 2024-09-11T09:51:40.972 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1199999999999999e-05 2024-09-11T09:51:40.972 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.972 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.972 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:40.972 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.149863+0000", 2024-09-11T09:51:40.972 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.4109999999999999e-06 2024-09-11T09:51:40.973 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.973 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.973 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:40.973 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.149908+0000", 2024-09-11T09:51:40.973 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.5184e-05 2024-09-11T09:51:40.973 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.973 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.973 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:40.973 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.149964+0000", 2024-09-11T09:51:40.973 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.5395000000000001e-05 2024-09-11T09:51:40.973 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.973 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.973 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 5", 2024-09-11T09:51:40.974 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.150157+0000", 2024-09-11T09:51:40.974 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000193489 2024-09-11T09:51:40.974 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.974 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.974 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:40.974 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.152238+0000", 2024-09-11T09:51:40.974 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0020803480000000001 2024-09-11T09:51:40.974 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.974 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.974 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:40.974 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.155828+0000", 2024-09-11T09:51:40.974 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0035902210000000002 2024-09-11T09:51:40.974 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.974 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.975 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:40.975 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.155873+0000", 2024-09-11T09:51:40.975 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.5145999999999998e-05 2024-09-11T09:51:40.975 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:40.975 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:40.975 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:40.975 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.155907+0000", 2024-09-11T09:51:40.975 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.4221999999999999e-05 2024-09-11T09:51:40.976 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:40.976 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:40.976 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:40.976 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:40.976 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:40.976 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:51:40.979 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-11T09:51:40.979 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-11T09:51:40.980 INFO:tasks.workunit.client.0.smithi107.stderr:+ DEV=/dev/nbd0 2024-09-11T09:51:40.980 INFO:tasks.workunit.client.0.smithi107.stderr:+ get_pid rbd 2024-09-11T09:51:40.980 INFO:tasks.workunit.client.0.smithi107.stderr:+ local pool=rbd 2024-09-11T09:51:40.980 INFO:tasks.workunit.client.0.smithi107.stderr:+ local ns= 2024-09-11T09:51:40.980 INFO:tasks.workunit.client.0.smithi107.stderr:++ rbd device --device-type nbd --format xml list 2024-09-11T09:51:40.980 INFO:tasks.workunit.client.0.smithi107.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd27121'\''][device='\''/dev/nbd0'\'']/id' 2024-09-11T09:51:40.980 INFO:tasks.workunit.client.0.smithi107.stderr:+ PID=28435 2024-09-11T09:51:40.980 INFO:tasks.workunit.client.0.smithi107.stderr:+ test -n 28435 2024-09-11T09:51:40.980 INFO:tasks.workunit.client.0.smithi107.stderr:+ ps -p 28435 -C rbd-nbd 2024-09-11T09:51:40.981 INFO:tasks.workunit.client.0.smithi107.stdout: PID TTY TIME CMD 2024-09-11T09:51:40.981 INFO:tasks.workunit.client.0.smithi107.stdout: 28435 ? 00:00:00 rbd-nbd 2024-09-11T09:51:40.981 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-11T09:51:40.982 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T09:51:40.982 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-11T09:51:40.982 INFO:tasks.ceph.osd.0.smithi017.stderr:2024-09-11T09:51:40.978+0000 7fb73c0f1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31665) UID: 0 2024-09-11T09:51:40.983 INFO:tasks.workunit.client.0.smithi107.stderr:+ unmap_device testrbdnbd27121@snap 28435 2024-09-11T09:51:40.983 INFO:tasks.workunit.client.0.smithi107.stderr:+ local args=testrbdnbd27121@snap 2024-09-11T09:51:40.983 INFO:tasks.workunit.client.0.smithi107.stderr:+ local pid=28435 2024-09-11T09:51:40.983 INFO:tasks.workunit.client.0.smithi107.stderr:+ _sudo rbd device --device-type nbd unmap testrbdnbd27121@snap 2024-09-11T09:51:40.983 INFO:tasks.workunit.client.0.smithi107.stderr:+ local cmd 2024-09-11T09:51:40.983 INFO:tasks.workunit.client.0.smithi107.stderr:++ id -u 2024-09-11T09:51:40.983 INFO:tasks.workunit.client.0.smithi107.stderr:+ '[' 1000 -eq 0 ']' 2024-09-11T09:51:40.983 INFO:tasks.workunit.client.0.smithi107.stderr:++ which rbd 2024-09-11T09:51:40.984 INFO:tasks.workunit.client.0.smithi107.stderr:+ cmd=/usr/bin/rbd 2024-09-11T09:51:40.984 INFO:tasks.workunit.client.0.smithi107.stderr:+ shift 2024-09-11T09:51:40.984 INFO:tasks.workunit.client.0.smithi107.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd unmap testrbdnbd27121@snap 2024-09-11T09:51:41.040 INFO:tasks.workunit.client.0.smithi107.stderr:+ rbd device --device-type nbd list 2024-09-11T09:51:41.040 INFO:tasks.workunit.client.0.smithi107.stderr:+ expect_false grep '^28435\b' 2024-09-11T09:51:41.040 INFO:tasks.workunit.client.0.smithi107.stderr:+ grep '^28435\b' 2024-09-11T09:51:41.078 INFO:tasks.ceph.osd.0.smithi017.stderr:2024-09-11T09:51:41.078+0000 7fb73c0f1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31665) UID: 0 2024-09-11T09:51:41.087 INFO:tasks.workunit.client.0.smithi107.stderr:+ return 0 2024-09-11T09:51:41.088 INFO:tasks.workunit.client.0.smithi107.stderr:+ ps -C rbd-nbd 2024-09-11T09:51:41.088 INFO:tasks.workunit.client.0.smithi107.stderr:+ expect_false grep '^ *28435\b' 2024-09-11T09:51:41.089 INFO:tasks.workunit.client.0.smithi107.stderr:+ grep '^ *28435\b' 2024-09-11T09:51:41.099 INFO:tasks.workunit.client.0.smithi107.stderr:+ return 0 2024-09-11T09:51:41.099 INFO:tasks.workunit.client.0.smithi107.stderr:+ sleep 0.5 2024-09-11T09:51:41.126 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:51:41.127 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [], 2024-09-11T09:51:41.127 INFO:teuthology.orchestra.run.smithi017.stdout: "num_ops": 0 2024-09-11T09:51:41.127 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:51:41.135 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-11T09:51:41.219 INFO:tasks.ceph.osd.2.smithi017.stderr:2024-09-11T09:51:41.219+0000 7fd20b170640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31672) UID: 0 2024-09-11T09:51:41.279 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-09-11T09:51:41.281+0000 7f7b6c7d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31904) UID: 0 2024-09-11T09:51:41.282 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:51:41.282 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [], 2024-09-11T09:51:41.282 INFO:teuthology.orchestra.run.smithi017.stdout: "complaint_time": 30, 2024-09-11T09:51:41.282 INFO:teuthology.orchestra.run.smithi017.stdout: "num_blocked_ops": 0 2024-09-11T09:51:41.283 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:51:41.292 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-11T09:51:41.380 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:51:41.380+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:51:41.441 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:51:41.442 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 20, 2024-09-11T09:51:41.442 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 600, 2024-09-11T09:51:41.442 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [ 2024-09-11T09:51:41.442 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.442 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:34 2.4 2:2367385a:::rbd_data.1250f5e25b80.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:41.442 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.897717+0000", 2024-09-11T09:51:41.442 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 18.543682442000001, 2024-09-11T09:51:41.442 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.021323661000000001, 2024-09-11T09:51:41.442 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:41.443 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:41.443 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:41.443 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:41.443 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:51:41.443 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:51:41.443 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 34 2024-09-11T09:51:41.443 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.443 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:41.443 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.443 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:41.443 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.897717+0000", 2024-09-11T09:51:41.443 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:41.443 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.443 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.444 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:41.444 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.897717+0000", 2024-09-11T09:51:41.444 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:41.444 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.444 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.444 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:41.444 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.897718+0000", 2024-09-11T09:51:41.444 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.1800000000000004e-07 2024-09-11T09:51:41.444 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.444 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.444 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:41.444 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.905339+0000", 2024-09-11T09:51:41.445 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0076210239999999997 2024-09-11T09:51:41.445 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.445 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.445 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:41.445 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.905366+0000", 2024-09-11T09:51:41.445 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.6959000000000001e-05 2024-09-11T09:51:41.445 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.445 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.445 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:41.446 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.905375+0000", 2024-09-11T09:51:41.446 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.516e-06 2024-09-11T09:51:41.446 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.446 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.446 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:41.446 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.905475+0000", 2024-09-11T09:51:41.446 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.9738e-05 2024-09-11T09:51:41.446 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.446 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.446 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:41.446 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.905660+0000", 2024-09-11T09:51:41.446 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00018554199999999999 2024-09-11T09:51:41.446 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.447 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.447 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:41.447 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.905808+0000", 2024-09-11T09:51:41.447 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000147386 2024-09-11T09:51:41.447 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.447 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.447 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:41.447 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.918230+0000", 2024-09-11T09:51:41.447 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.012421690000000001 2024-09-11T09:51:41.447 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.447 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.447 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:41.447 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.918932+0000", 2024-09-11T09:51:41.448 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00070267300000000001 2024-09-11T09:51:41.448 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.448 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.448 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:41.448 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.918989+0000", 2024-09-11T09:51:41.448 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.7268999999999997e-05 2024-09-11T09:51:41.448 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.448 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.448 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:41.448 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919040+0000", 2024-09-11T09:51:41.448 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.0946000000000003e-05 2024-09-11T09:51:41.448 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:41.448 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:41.449 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:41.449 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.449 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.449 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4688.0:31 2.1 e53/15 2:8654721b:::rbd_data.1250f5e25b80.0000000000000000:head v 53'2, mlcod=0'0)", 2024-09-11T09:51:41.449 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.904661+0000", 2024-09-11T09:51:41.449 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 18.536737952999999, 2024-09-11T09:51:41.449 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.014152804999999999, 2024-09-11T09:51:41.449 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:41.449 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:41.449 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:41.449 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:41.450 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.450 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:41.450 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.904661+0000", 2024-09-11T09:51:41.450 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:41.450 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.450 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.450 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:41.450 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.904661+0000", 2024-09-11T09:51:41.450 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:41.450 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.450 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.450 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:41.450 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.904662+0000", 2024-09-11T09:51:41.450 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0759999999999999e-06 2024-09-11T09:51:41.451 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.451 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.451 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:41.451 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.907298+0000", 2024-09-11T09:51:41.451 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0026361240000000001 2024-09-11T09:51:41.451 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.451 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.451 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:41.451 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.907320+0000", 2024-09-11T09:51:41.451 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.1268999999999999e-05 2024-09-11T09:51:41.451 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.451 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.451 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:41.452 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.907331+0000", 2024-09-11T09:51:41.452 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1031e-05 2024-09-11T09:51:41.452 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.452 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.452 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:41.452 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.907403+0000", 2024-09-11T09:51:41.452 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.2571999999999996e-05 2024-09-11T09:51:41.452 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.452 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.452 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:41.452 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.907428+0000", 2024-09-11T09:51:41.452 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.4380999999999999e-05 2024-09-11T09:51:41.452 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.453 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.453 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:41.453 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.918750+0000", 2024-09-11T09:51:41.453 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.011322211 2024-09-11T09:51:41.453 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.453 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.453 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:41.453 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.918814+0000", 2024-09-11T09:51:41.453 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.4140999999999995e-05 2024-09-11T09:51:41.454 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:41.454 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:41.454 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:41.454 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.454 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.454 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:36 2.6 2:61e1b442:::rbd_data.1250f5e25b80.0000000000000003:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:41.454 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.905404+0000", 2024-09-11T09:51:41.455 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 18.535995081999999, 2024-09-11T09:51:41.455 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.030912479999999999, 2024-09-11T09:51:41.455 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:41.455 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:41.455 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:41.455 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:41.456 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:51:41.456 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:51:41.456 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 36 2024-09-11T09:51:41.456 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.456 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:41.456 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.456 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:41.456 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.905404+0000", 2024-09-11T09:51:41.456 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:41.456 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.456 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.456 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:41.457 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.905404+0000", 2024-09-11T09:51:41.457 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:41.457 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.457 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.457 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:41.457 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.905405+0000", 2024-09-11T09:51:41.457 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2270000000000001e-06 2024-09-11T09:51:41.457 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.457 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.457 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:41.457 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.914859+0000", 2024-09-11T09:51:41.457 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0094541190000000004 2024-09-11T09:51:41.457 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.458 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.458 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:41.458 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.914879+0000", 2024-09-11T09:51:41.458 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.0052999999999999e-05 2024-09-11T09:51:41.458 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.458 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.458 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:41.458 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.914897+0000", 2024-09-11T09:51:41.458 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7331000000000001e-05 2024-09-11T09:51:41.458 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.458 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.458 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:41.459 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.915019+0000", 2024-09-11T09:51:41.459 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00012168 2024-09-11T09:51:41.459 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.459 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.459 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:41.459 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.915269+0000", 2024-09-11T09:51:41.459 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00025054899999999998 2024-09-11T09:51:41.459 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.459 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.459 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 3", 2024-09-11T09:51:41.459 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.915479+0000", 2024-09-11T09:51:41.459 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00021017799999999999 2024-09-11T09:51:41.459 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.460 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.460 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:41.460 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.926070+0000", 2024-09-11T09:51:41.460 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.010590624999999999 2024-09-11T09:51:41.460 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.460 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.460 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:41.460 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.936198+0000", 2024-09-11T09:51:41.460 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.010128194 2024-09-11T09:51:41.461 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.461 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.461 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:41.461 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.936263+0000", 2024-09-11T09:51:41.461 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.5366999999999999e-05 2024-09-11T09:51:41.461 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.461 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.461 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:41.461 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.936317+0000", 2024-09-11T09:51:41.461 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.3156999999999999e-05 2024-09-11T09:51:41.461 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:41.462 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:41.462 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:41.462 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.462 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.462 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:39 2.6 2:743e7616:::rbd_data.1250f5e25b80.0000000000000005:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:41.462 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.914960+0000", 2024-09-11T09:51:41.462 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 18.526438779999999, 2024-09-11T09:51:41.462 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.022371518, 2024-09-11T09:51:41.462 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:41.462 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:41.463 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:41.463 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:41.463 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:51:41.463 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:51:41.463 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 39 2024-09-11T09:51:41.463 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.463 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:41.463 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.463 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:41.463 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.914960+0000", 2024-09-11T09:51:41.463 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:41.463 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.463 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.463 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:41.463 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.914960+0000", 2024-09-11T09:51:41.464 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:41.464 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.464 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.464 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:41.464 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.914963+0000", 2024-09-11T09:51:41.464 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.1119999999999999e-06 2024-09-11T09:51:41.464 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.464 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.464 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:41.464 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.920417+0000", 2024-09-11T09:51:41.464 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0054543969999999997 2024-09-11T09:51:41.464 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.464 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.465 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:41.465 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.920430+0000", 2024-09-11T09:51:41.465 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3414000000000001e-05 2024-09-11T09:51:41.465 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.465 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.465 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:41.465 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.920437+0000", 2024-09-11T09:51:41.465 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.6159999999999998e-06 2024-09-11T09:51:41.466 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.466 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.466 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:41.466 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.921325+0000", 2024-09-11T09:51:41.466 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000887731 2024-09-11T09:51:41.466 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.466 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.466 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:41.466 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.921422+0000", 2024-09-11T09:51:41.466 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.7201999999999995e-05 2024-09-11T09:51:41.466 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.467 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.467 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 3", 2024-09-11T09:51:41.467 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.921511+0000", 2024-09-11T09:51:41.467 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.8880000000000003e-05 2024-09-11T09:51:41.467 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.467 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.467 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:41.467 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.928092+0000", 2024-09-11T09:51:41.467 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0065816290000000003 2024-09-11T09:51:41.467 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.467 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.467 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:41.467 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.937240+0000", 2024-09-11T09:51:41.468 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0091481070000000008 2024-09-11T09:51:41.468 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.468 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.468 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:41.468 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.937294+0000", 2024-09-11T09:51:41.468 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.3100000000000003e-05 2024-09-11T09:51:41.468 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.468 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.468 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:41.468 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.937332+0000", 2024-09-11T09:51:41.468 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.8330000000000001e-05 2024-09-11T09:51:41.468 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:41.468 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:41.469 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:41.469 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.469 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.469 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4688.0:32 2.2 e53/15 2:45941fd2:::rbd_data.1250f5e25b80.0000000000000001:head v 53'3, mlcod=17'1)", 2024-09-11T09:51:41.469 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.916509+0000", 2024-09-11T09:51:41.469 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 18.524889697999999, 2024-09-11T09:51:41.469 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.015885318999999998, 2024-09-11T09:51:41.469 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:41.469 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:41.469 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:41.469 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:41.469 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.470 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:41.470 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.916509+0000", 2024-09-11T09:51:41.470 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:41.470 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.470 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.470 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:41.470 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.916509+0000", 2024-09-11T09:51:41.470 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:41.470 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.470 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.470 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:41.471 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.916511+0000", 2024-09-11T09:51:41.471 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.052e-06 2024-09-11T09:51:41.471 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.471 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.471 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:41.472 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.927019+0000", 2024-09-11T09:51:41.472 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.010508224 2024-09-11T09:51:41.472 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.472 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.472 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:41.473 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.927038+0000", 2024-09-11T09:51:41.473 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.9590999999999999e-05 2024-09-11T09:51:41.473 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.473 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.473 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:41.473 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.927050+0000", 2024-09-11T09:51:41.474 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1439e-05 2024-09-11T09:51:41.474 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.474 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.474 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:41.474 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.927115+0000", 2024-09-11T09:51:41.474 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.5301000000000004e-05 2024-09-11T09:51:41.474 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.474 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.475 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:41.475 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.927132+0000", 2024-09-11T09:51:41.475 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7297000000000001e-05 2024-09-11T09:51:41.475 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.475 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.475 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:41.475 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.932354+0000", 2024-09-11T09:51:41.475 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0052211870000000004 2024-09-11T09:51:41.476 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.476 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.476 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:41.476 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.932395+0000", 2024-09-11T09:51:41.476 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.1227999999999999e-05 2024-09-11T09:51:41.476 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:41.476 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:41.476 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:41.477 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.477 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.477 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:44 2.6 2:67246a22:::rbd_data.1250f5e25b80.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:41.477 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.943181+0000", 2024-09-11T09:51:41.477 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 18.498218486999999, 2024-09-11T09:51:41.477 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.022629371999999998, 2024-09-11T09:51:41.477 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:41.477 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:41.478 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:41.478 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:41.478 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:51:41.478 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:51:41.478 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 44 2024-09-11T09:51:41.478 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.478 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:41.478 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.479 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:41.479 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.943181+0000", 2024-09-11T09:51:41.479 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:41.479 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.479 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.479 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:41.479 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.943181+0000", 2024-09-11T09:51:41.479 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:41.480 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.480 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.480 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:41.480 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.943182+0000", 2024-09-11T09:51:41.480 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.0500000000000002e-07 2024-09-11T09:51:41.480 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.480 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.480 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:41.481 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951240+0000", 2024-09-11T09:51:41.481 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.008058529 2024-09-11T09:51:41.481 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.481 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.481 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:41.481 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951260+0000", 2024-09-11T09:51:41.481 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.9570000000000001e-05 2024-09-11T09:51:41.482 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.482 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.482 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:41.482 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951270+0000", 2024-09-11T09:51:41.482 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0018e-05 2024-09-11T09:51:41.482 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.482 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.482 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:41.483 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951349+0000", 2024-09-11T09:51:41.483 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.8801000000000007e-05 2024-09-11T09:51:41.483 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.483 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.483 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:41.483 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951483+0000", 2024-09-11T09:51:41.483 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00013427 2024-09-11T09:51:41.483 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.484 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.484 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 3", 2024-09-11T09:51:41.484 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951593+0000", 2024-09-11T09:51:41.484 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00011031000000000001 2024-09-11T09:51:41.484 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.484 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.484 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:41.484 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.957303+0000", 2024-09-11T09:51:41.485 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0057097959999999996 2024-09-11T09:51:41.485 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.485 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.485 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:41.485 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.965690+0000", 2024-09-11T09:51:41.485 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0083867420000000008 2024-09-11T09:51:41.485 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.486 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.486 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:41.486 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.965749+0000", 2024-09-11T09:51:41.486 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.8981999999999999e-05 2024-09-11T09:51:41.486 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.486 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.486 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:41.486 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.965810+0000", 2024-09-11T09:51:41.487 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.1449000000000001e-05 2024-09-11T09:51:41.487 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:41.487 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:41.487 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:41.487 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.487 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.487 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:51 2.6 2:6b5894fc:::rbd_data.1250f5e25b80.000000000000000a:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:41.487 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.951324+0000", 2024-09-11T09:51:41.488 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 18.490075414, 2024-09-11T09:51:41.488 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.021375273, 2024-09-11T09:51:41.488 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:41.488 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:41.488 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:41.488 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:41.488 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:51:41.488 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:51:41.488 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 51 2024-09-11T09:51:41.488 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.489 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:41.489 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.489 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:41.489 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951324+0000", 2024-09-11T09:51:41.489 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:41.489 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.489 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.489 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:41.489 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951324+0000", 2024-09-11T09:51:41.490 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:41.490 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.490 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.490 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:41.490 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951324+0000", 2024-09-11T09:51:41.490 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.39e-07 2024-09-11T09:51:41.490 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.490 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.490 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:41.490 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.958537+0000", 2024-09-11T09:51:41.490 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0072126769999999998 2024-09-11T09:51:41.490 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.490 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.491 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:41.491 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.958554+0000", 2024-09-11T09:51:41.491 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7232e-05 2024-09-11T09:51:41.491 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.491 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.491 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:41.491 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.958570+0000", 2024-09-11T09:51:41.491 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.5747999999999999e-05 2024-09-11T09:51:41.491 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.491 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.491 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:41.491 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.958638+0000", 2024-09-11T09:51:41.492 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.7810000000000003e-05 2024-09-11T09:51:41.492 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.492 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.492 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:41.492 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.958758+0000", 2024-09-11T09:51:41.492 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000120689 2024-09-11T09:51:41.492 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.492 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.492 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 3", 2024-09-11T09:51:41.492 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.958865+0000", 2024-09-11T09:51:41.492 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00010692 2024-09-11T09:51:41.492 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.492 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.493 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:41.493 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.970712+0000", 2024-09-11T09:51:41.493 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.011846327 2024-09-11T09:51:41.493 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.493 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.493 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:41.493 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.972603+0000", 2024-09-11T09:51:41.493 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0018918610000000001 2024-09-11T09:51:41.493 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.493 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.493 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:41.493 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.972653+0000", 2024-09-11T09:51:41.494 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.9762999999999998e-05 2024-09-11T09:51:41.494 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.494 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.494 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:41.494 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.972699+0000", 2024-09-11T09:51:41.494 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.5806999999999997e-05 2024-09-11T09:51:41.494 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:41.494 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:41.494 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:41.495 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.495 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.495 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:52 2.4 2:26d5c2f0:::rbd_data.1250f5e25b80.000000000000000b:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:41.495 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.958596+0000", 2024-09-11T09:51:41.495 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 18.482803219000001, 2024-09-11T09:51:41.495 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.016477849999999999, 2024-09-11T09:51:41.495 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:41.495 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:41.495 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:41.495 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:41.495 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:51:41.495 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:51:41.495 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 52 2024-09-11T09:51:41.495 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.496 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:41.496 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.496 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:41.496 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.958596+0000", 2024-09-11T09:51:41.496 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:41.496 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.496 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.496 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:41.496 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.958596+0000", 2024-09-11T09:51:41.496 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:41.496 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.496 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.497 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:41.497 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.958597+0000", 2024-09-11T09:51:41.497 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1960000000000001e-06 2024-09-11T09:51:41.497 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.497 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.497 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:41.497 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.963157+0000", 2024-09-11T09:51:41.497 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0045603029999999999 2024-09-11T09:51:41.497 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.497 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.497 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:41.497 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.963171+0000", 2024-09-11T09:51:41.497 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3777999999999999e-05 2024-09-11T09:51:41.498 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.498 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.498 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:41.498 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.963181+0000", 2024-09-11T09:51:41.498 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.9010000000000006e-06 2024-09-11T09:51:41.498 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.498 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.498 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:41.498 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.963246+0000", 2024-09-11T09:51:41.498 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.5351000000000005e-05 2024-09-11T09:51:41.498 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.498 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.499 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:41.499 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.963419+0000", 2024-09-11T09:51:41.499 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00017299700000000001 2024-09-11T09:51:41.499 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.499 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.499 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:41.499 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.963570+0000", 2024-09-11T09:51:41.499 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00015066999999999999 2024-09-11T09:51:41.499 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.499 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.500 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:41.500 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.973881+0000", 2024-09-11T09:51:41.500 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.010311268 2024-09-11T09:51:41.500 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.500 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.500 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:41.500 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.974963+0000", 2024-09-11T09:51:41.500 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.001081493 2024-09-11T09:51:41.500 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.500 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.500 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:41.500 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.975012+0000", 2024-09-11T09:51:41.501 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.9471000000000001e-05 2024-09-11T09:51:41.501 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.501 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.501 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:41.501 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.975074+0000", 2024-09-11T09:51:41.501 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.1421999999999996e-05 2024-09-11T09:51:41.501 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:41.501 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:41.501 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:41.501 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.501 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.501 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4688.0:47 2.1 e53/15 2:95f53b3e:::rbd_data.1250f5e25b80.0000000000000008:head v 53'3, mlcod=53'1)", 2024-09-11T09:51:41.501 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.960861+0000", 2024-09-11T09:51:41.502 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 18.480537801000001, 2024-09-11T09:51:41.502 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.012346107, 2024-09-11T09:51:41.502 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:41.502 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:41.502 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:41.502 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:41.502 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.502 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:41.502 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.960861+0000", 2024-09-11T09:51:41.502 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:41.502 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.502 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.502 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:41.503 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.960861+0000", 2024-09-11T09:51:41.503 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:41.503 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.503 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.503 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:41.503 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.960863+0000", 2024-09-11T09:51:41.503 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7150000000000001e-06 2024-09-11T09:51:41.503 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.503 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.503 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:41.503 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.962428+0000", 2024-09-11T09:51:41.503 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.001564668 2024-09-11T09:51:41.503 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.504 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.504 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:41.504 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.962446+0000", 2024-09-11T09:51:41.504 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.8476999999999999e-05 2024-09-11T09:51:41.504 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.504 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.504 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:41.504 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.962458+0000", 2024-09-11T09:51:41.504 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1263999999999999e-05 2024-09-11T09:51:41.504 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.504 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.504 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:41.504 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.962575+0000", 2024-09-11T09:51:41.505 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000117025 2024-09-11T09:51:41.505 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.505 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.505 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:41.505 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.962601+0000", 2024-09-11T09:51:41.505 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.5984999999999999e-05 2024-09-11T09:51:41.505 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.505 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.505 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:41.505 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.973065+0000", 2024-09-11T09:51:41.505 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.010464753 2024-09-11T09:51:41.505 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.506 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.506 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:41.506 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.973207+0000", 2024-09-11T09:51:41.506 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00014222 2024-09-11T09:51:41.506 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:41.506 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:41.506 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:41.506 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.506 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.506 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:54 2.4 2.da8979c4 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:41.506 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.963207+0000", 2024-09-11T09:51:41.506 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 18.478192363000002, 2024-09-11T09:51:41.507 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.010641575, 2024-09-11T09:51:41.507 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:41.507 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:41.507 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "reached pg", 2024-09-11T09:51:41.507 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:41.507 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:51:41.507 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:51:41.507 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 54 2024-09-11T09:51:41.507 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.507 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:41.507 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.508 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:41.508 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.963207+0000", 2024-09-11T09:51:41.508 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:41.508 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.508 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.508 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:41.508 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.963207+0000", 2024-09-11T09:51:41.508 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:41.508 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.508 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.508 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:41.508 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.963208+0000", 2024-09-11T09:51:41.509 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2979999999999999e-06 2024-09-11T09:51:41.509 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.509 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.509 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:41.509 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.967419+0000", 2024-09-11T09:51:41.509 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0042111750000000002 2024-09-11T09:51:41.509 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.509 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.509 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:41.509 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.967439+0000", 2024-09-11T09:51:41.509 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.0129999999999999e-05 2024-09-11T09:51:41.509 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.510 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.510 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:41.510 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.967453+0000", 2024-09-11T09:51:41.510 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3781000000000001e-05 2024-09-11T09:51:41.510 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.510 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.510 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:41.510 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.973840+0000", 2024-09-11T09:51:41.510 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0063872269999999997 2024-09-11T09:51:41.510 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.510 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.510 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:41.510 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.973848+0000", 2024-09-11T09:51:41.511 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.9640000000000003e-06 2024-09-11T09:51:41.511 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:41.511 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:41.511 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:41.511 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.511 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.511 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:54 2.4 2:239e915b:::rbd_data.1250f5e25b80.000000000000000c:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:41.511 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.982401+0000", 2024-09-11T09:51:41.511 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 18.458998436000002, 2024-09-11T09:51:41.511 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.016817347, 2024-09-11T09:51:41.511 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:41.511 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:41.511 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:41.511 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:41.512 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:51:41.512 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:51:41.512 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 54 2024-09-11T09:51:41.512 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.512 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:41.512 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.512 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:41.512 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.982401+0000", 2024-09-11T09:51:41.512 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:41.512 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.512 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.512 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:41.513 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.982401+0000", 2024-09-11T09:51:41.513 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:41.513 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.513 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.513 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:41.513 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.982402+0000", 2024-09-11T09:51:41.513 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.8999999999999995e-07 2024-09-11T09:51:41.513 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.513 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.513 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:41.513 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.986535+0000", 2024-09-11T09:51:41.513 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0041333689999999996 2024-09-11T09:51:41.513 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.514 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.514 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:41.514 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.986555+0000", 2024-09-11T09:51:41.514 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.9851e-05 2024-09-11T09:51:41.514 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.514 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.514 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:41.514 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.986569+0000", 2024-09-11T09:51:41.514 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3784000000000001e-05 2024-09-11T09:51:41.514 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.514 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.514 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:41.514 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.986683+0000", 2024-09-11T09:51:41.515 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000114474 2024-09-11T09:51:41.515 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.515 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.515 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:41.515 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.986815+0000", 2024-09-11T09:51:41.515 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00013217799999999999 2024-09-11T09:51:41.515 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.515 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.515 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:41.515 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.986918+0000", 2024-09-11T09:51:41.515 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00010235599999999999 2024-09-11T09:51:41.515 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.515 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.516 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:41.516 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.998983+0000", 2024-09-11T09:51:41.516 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.012065352 2024-09-11T09:51:41.516 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.516 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.516 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:41.516 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.999081+0000", 2024-09-11T09:51:41.516 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.8091000000000007e-05 2024-09-11T09:51:41.516 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.516 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.516 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:41.516 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.999159+0000", 2024-09-11T09:51:41.516 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.7823000000000004e-05 2024-09-11T09:51:41.517 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.517 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.517 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:41.517 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.999218+0000", 2024-09-11T09:51:41.517 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.9178999999999997e-05 2024-09-11T09:51:41.517 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:41.517 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:41.517 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:41.517 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.517 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.517 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:56 2.4 2:2167012f:::rbd_data.1250f5e25b80.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:41.517 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.986661+0000", 2024-09-11T09:51:41.517 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 18.454737983000001, 2024-09-11T09:51:41.518 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.017353539000000001, 2024-09-11T09:51:41.518 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:41.518 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:41.518 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:41.518 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:41.518 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:51:41.518 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:51:41.518 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 56 2024-09-11T09:51:41.518 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.518 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:41.518 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.518 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:41.518 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.986661+0000", 2024-09-11T09:51:41.519 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:41.519 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.519 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.519 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:41.519 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.986661+0000", 2024-09-11T09:51:41.519 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:41.519 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.519 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.519 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:41.519 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.986663+0000", 2024-09-11T09:51:41.519 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.8989999999999999e-06 2024-09-11T09:51:41.519 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.519 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.519 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:41.520 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.990444+0000", 2024-09-11T09:51:41.520 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0037808569999999999 2024-09-11T09:51:41.520 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.520 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.520 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:41.520 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.990462+0000", 2024-09-11T09:51:41.520 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.8450000000000001e-05 2024-09-11T09:51:41.520 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.520 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.520 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:41.520 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.990476+0000", 2024-09-11T09:51:41.520 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3565e-05 2024-09-11T09:51:41.521 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.521 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.521 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:41.521 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.993582+0000", 2024-09-11T09:51:41.521 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0031059109999999998 2024-09-11T09:51:41.521 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.521 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.521 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:41.521 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.993686+0000", 2024-09-11T09:51:41.521 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000104424 2024-09-11T09:51:41.521 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.521 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.522 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:41.522 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.993799+0000", 2024-09-11T09:51:41.522 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000112985 2024-09-11T09:51:41.522 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.522 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.522 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:41.522 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.002449+0000", 2024-09-11T09:51:41.522 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.008650151 2024-09-11T09:51:41.522 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.522 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.522 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:41.522 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.003948+0000", 2024-09-11T09:51:41.522 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.001498636 2024-09-11T09:51:41.522 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.523 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.523 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:41.523 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.003985+0000", 2024-09-11T09:51:41.523 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.7054999999999997e-05 2024-09-11T09:51:41.523 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.523 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.523 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:41.523 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.004015+0000", 2024-09-11T09:51:41.523 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.9606000000000001e-05 2024-09-11T09:51:41.523 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:41.523 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:41.523 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:41.523 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.524 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.524 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:58 2.6 2:6277c644:::rbd_data.1250f5e25b80.000000000000000e:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:41.524 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.990540+0000", 2024-09-11T09:51:41.524 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 18.450859187999999, 2024-09-11T09:51:41.524 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.016232068999999998, 2024-09-11T09:51:41.524 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:41.524 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:41.524 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:41.524 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:41.524 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:51:41.524 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:51:41.525 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 58 2024-09-11T09:51:41.525 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.525 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:41.525 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.525 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:41.525 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.990540+0000", 2024-09-11T09:51:41.525 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:41.525 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.525 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.525 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:41.525 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.990540+0000", 2024-09-11T09:51:41.525 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:41.525 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.526 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.526 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:41.526 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.990541+0000", 2024-09-11T09:51:41.526 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2359999999999999e-06 2024-09-11T09:51:41.526 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.526 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.526 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:41.526 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.994626+0000", 2024-09-11T09:51:41.526 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.004084865 2024-09-11T09:51:41.526 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.526 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.526 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:41.526 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.994644+0000", 2024-09-11T09:51:41.527 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7866000000000001e-05 2024-09-11T09:51:41.527 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.527 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.527 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:41.527 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.994653+0000", 2024-09-11T09:51:41.527 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.2159999999999995e-06 2024-09-11T09:51:41.527 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.527 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.527 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:41.527 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.994732+0000", 2024-09-11T09:51:41.527 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.8954000000000003e-05 2024-09-11T09:51:41.527 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.528 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.528 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:41.528 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.994906+0000", 2024-09-11T09:51:41.528 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0001741 2024-09-11T09:51:41.528 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.528 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.528 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 3", 2024-09-11T09:51:41.528 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.995083+0000", 2024-09-11T09:51:41.528 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00017637100000000001 2024-09-11T09:51:41.528 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.528 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.528 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:41.529 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.003066+0000", 2024-09-11T09:51:41.529 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0079833360000000006 2024-09-11T09:51:41.529 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.529 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.529 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:41.529 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.006634+0000", 2024-09-11T09:51:41.529 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.003568309 2024-09-11T09:51:41.529 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.529 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.529 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:41.529 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.006716+0000", 2024-09-11T09:51:41.529 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.1377000000000004e-05 2024-09-11T09:51:41.530 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.530 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.530 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:41.530 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.006772+0000", 2024-09-11T09:51:41.530 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.6439e-05 2024-09-11T09:51:41.530 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:41.530 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:41.530 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:41.530 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.530 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.530 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:59 2.4 2:2906e6a7:::rbd_header.1250f5e25b80:head [watch reconnect cookie 140424358411408 gen 2] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:41.530 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.994684+0000", 2024-09-11T09:51:41.530 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 18.446715557000001, 2024-09-11T09:51:41.530 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0094134349999999995, 2024-09-11T09:51:41.531 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:41.531 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:41.531 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "started", 2024-09-11T09:51:41.531 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:41.531 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:51:41.531 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:51:41.531 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 59 2024-09-11T09:51:41.531 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.531 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:41.531 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.531 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:41.531 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.994684+0000", 2024-09-11T09:51:41.532 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:41.532 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.532 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.532 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:41.532 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.994684+0000", 2024-09-11T09:51:41.532 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:41.532 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.532 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.532 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:41.532 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.994686+0000", 2024-09-11T09:51:41.532 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.8159999999999998e-06 2024-09-11T09:51:41.532 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.532 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.533 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:41.533 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.994691+0000", 2024-09-11T09:51:41.533 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.5009999999999998e-06 2024-09-11T09:51:41.533 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.533 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.533 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:41.533 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.994698+0000", 2024-09-11T09:51:41.533 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.3039999999999999e-06 2024-09-11T09:51:41.533 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.533 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.534 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:41.534 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.994703+0000", 2024-09-11T09:51:41.534 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.5190000000000003e-06 2024-09-11T09:51:41.534 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.534 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.534 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:41.534 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.998678+0000", 2024-09-11T09:51:41.534 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0039752340000000002 2024-09-11T09:51:41.534 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.534 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.534 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:41.534 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.998748+0000", 2024-09-11T09:51:41.535 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.0114e-05 2024-09-11T09:51:41.535 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.535 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.535 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:41.535 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.004097+0000", 2024-09-11T09:51:41.535 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0053489469999999997 2024-09-11T09:51:41.535 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:41.535 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:41.535 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:41.535 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.535 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.535 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4694.0:33 2.4 2:3bfc818c:ns::rbd_data.1256a0c02879.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:41.535 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.084537+0000", 2024-09-11T09:51:41.536 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 18.356862129, 2024-09-11T09:51:41.536 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.022075773999999999, 2024-09-11T09:51:41.536 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:41.536 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:41.536 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:41.536 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:41.536 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4694", 2024-09-11T09:51:41.536 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:51:41.536 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 33 2024-09-11T09:51:41.536 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.536 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:41.536 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.537 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:41.537 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.084537+0000", 2024-09-11T09:51:41.537 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:41.537 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.537 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.537 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:41.537 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.084537+0000", 2024-09-11T09:51:41.537 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:41.537 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.537 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.537 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:41.537 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.084538+0000", 2024-09-11T09:51:41.537 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.5000000000000001e-07 2024-09-11T09:51:41.537 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.538 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.538 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:41.538 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.095984+0000", 2024-09-11T09:51:41.538 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.011446543 2024-09-11T09:51:41.538 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.538 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.538 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:41.538 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.096014+0000", 2024-09-11T09:51:41.538 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.9745000000000001e-05 2024-09-11T09:51:41.538 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.538 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.538 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:41.538 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.096030+0000", 2024-09-11T09:51:41.539 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6152999999999999e-05 2024-09-11T09:51:41.539 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.539 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.539 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:41.539 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.096113+0000", 2024-09-11T09:51:41.539 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.2731999999999993e-05 2024-09-11T09:51:41.539 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.539 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.539 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:41.539 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.096273+0000", 2024-09-11T09:51:41.539 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00015973699999999999 2024-09-11T09:51:41.539 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.539 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.540 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:41.540 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.096394+0000", 2024-09-11T09:51:41.540 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000120883 2024-09-11T09:51:41.540 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.540 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.540 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:41.540 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.106048+0000", 2024-09-11T09:51:41.540 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0096548020000000005 2024-09-11T09:51:41.540 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.540 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.540 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:41.540 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.106531+0000", 2024-09-11T09:51:41.540 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00048264299999999999 2024-09-11T09:51:41.541 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.541 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.541 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:41.541 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.106565+0000", 2024-09-11T09:51:41.541 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.4343e-05 2024-09-11T09:51:41.541 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.541 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.541 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:41.541 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.106613+0000", 2024-09-11T09:51:41.541 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.7342999999999997e-05 2024-09-11T09:51:41.541 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:41.541 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:41.541 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:41.542 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.542 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.542 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4694.0:32 2.1 e53/15 2:8a862b48:ns::rbd_data.1256a0c02879.0000000000000000:head v 53'8, mlcod=53'6)", 2024-09-11T09:51:41.542 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.091418+0000", 2024-09-11T09:51:41.542 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 18.34998096, 2024-09-11T09:51:41.542 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.013027372000000001, 2024-09-11T09:51:41.542 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:41.542 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:41.542 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:41.542 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:41.542 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.542 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:41.543 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.091418+0000", 2024-09-11T09:51:41.543 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:41.543 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.543 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.543 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:41.543 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.091418+0000", 2024-09-11T09:51:41.543 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:41.543 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.543 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.543 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:41.543 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.091420+0000", 2024-09-11T09:51:41.543 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6899999999999999e-06 2024-09-11T09:51:41.544 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.544 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.544 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:41.544 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.093086+0000", 2024-09-11T09:51:41.544 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0016662510000000001 2024-09-11T09:51:41.544 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.544 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.544 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:41.544 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.093107+0000", 2024-09-11T09:51:41.544 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.0999e-05 2024-09-11T09:51:41.544 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.544 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.544 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:41.545 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.093120+0000", 2024-09-11T09:51:41.545 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2544999999999999e-05 2024-09-11T09:51:41.545 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.545 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.545 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:41.545 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.093186+0000", 2024-09-11T09:51:41.545 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.6595999999999997e-05 2024-09-11T09:51:41.545 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.545 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.545 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:41.545 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.093208+0000", 2024-09-11T09:51:41.545 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.1624000000000002e-05 2024-09-11T09:51:41.545 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.546 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.546 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:41.546 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.104392+0000", 2024-09-11T09:51:41.546 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.011184265000000001 2024-09-11T09:51:41.546 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.546 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.546 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:41.546 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.104446+0000", 2024-09-11T09:51:41.546 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.3402000000000001e-05 2024-09-11T09:51:41.546 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:41.546 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:41.546 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:41.547 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.547 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.547 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4694.0:45 2.6 2:671fb830:ns::rbd_data.1256a0c02879.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:41.547 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.109629+0000", 2024-09-11T09:51:41.547 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 18.331770119000002, 2024-09-11T09:51:41.547 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.036403143999999998, 2024-09-11T09:51:41.547 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:41.547 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:41.547 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:41.547 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:41.547 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4694", 2024-09-11T09:51:41.547 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:51:41.547 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 45 2024-09-11T09:51:41.548 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.548 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:41.548 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.548 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:41.548 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.109629+0000", 2024-09-11T09:51:41.548 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:41.548 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.548 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.548 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:41.548 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.109629+0000", 2024-09-11T09:51:41.548 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:41.548 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.549 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.549 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:41.549 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.109630+0000", 2024-09-11T09:51:41.549 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1200000000000001e-06 2024-09-11T09:51:41.549 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.549 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.549 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:41.549 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.121370+0000", 2024-09-11T09:51:41.549 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.011739571000000001 2024-09-11T09:51:41.549 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.549 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.549 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:41.549 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.121391+0000", 2024-09-11T09:51:41.550 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.1175e-05 2024-09-11T09:51:41.550 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.550 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.550 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:41.550 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.121404+0000", 2024-09-11T09:51:41.550 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2807e-05 2024-09-11T09:51:41.550 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.550 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.550 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:41.550 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.121498+0000", 2024-09-11T09:51:41.550 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.3770999999999994e-05 2024-09-11T09:51:41.550 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.551 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.551 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:41.551 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.121612+0000", 2024-09-11T09:51:41.551 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000114666 2024-09-11T09:51:41.551 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.551 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.551 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 3", 2024-09-11T09:51:41.551 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.121711+0000", 2024-09-11T09:51:41.551 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.8647999999999995e-05 2024-09-11T09:51:41.551 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.551 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.551 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:41.551 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.126558+0000", 2024-09-11T09:51:41.552 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0048473769999999999 2024-09-11T09:51:41.552 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.552 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.552 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:41.552 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.145912+0000", 2024-09-11T09:51:41.552 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.019354281000000001 2024-09-11T09:51:41.552 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.552 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.552 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:41.552 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.145977+0000", 2024-09-11T09:51:41.552 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.4135999999999997e-05 2024-09-11T09:51:41.552 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.552 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.553 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:41.553 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.146032+0000", 2024-09-11T09:51:41.553 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.5591999999999999e-05 2024-09-11T09:51:41.553 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:41.553 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:41.553 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:41.553 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.553 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.553 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4694.0:41 2.2 e53/15 2:5e2505c4:ns::rbd_data.1256a0c02879.0000000000000003:head v 53'4, mlcod=18'2)", 2024-09-11T09:51:41.553 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.113199+0000", 2024-09-11T09:51:41.553 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 18.328199709, 2024-09-11T09:51:41.553 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.011404512, 2024-09-11T09:51:41.553 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:41.554 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:41.554 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:41.554 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:41.554 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.554 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:41.554 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.113199+0000", 2024-09-11T09:51:41.554 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:41.554 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.554 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.554 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:41.554 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.113199+0000", 2024-09-11T09:51:41.554 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:41.554 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.554 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.555 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:41.555 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.113200+0000", 2024-09-11T09:51:41.555 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.2799999999999995e-07 2024-09-11T09:51:41.555 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.555 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.555 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:41.555 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.118872+0000", 2024-09-11T09:51:41.555 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0056718089999999999 2024-09-11T09:51:41.555 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.555 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.555 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:41.555 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.118889+0000", 2024-09-11T09:51:41.556 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7428999999999999e-05 2024-09-11T09:51:41.556 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.556 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.556 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:41.556 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.118900+0000", 2024-09-11T09:51:41.556 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0243e-05 2024-09-11T09:51:41.556 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.556 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.556 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:41.556 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.118959+0000", 2024-09-11T09:51:41.556 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.9364000000000003e-05 2024-09-11T09:51:41.556 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.556 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.557 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:41.557 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.118976+0000", 2024-09-11T09:51:41.557 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7124000000000001e-05 2024-09-11T09:51:41.557 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.557 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.557 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:41.557 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.124557+0000", 2024-09-11T09:51:41.557 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0055812659999999997 2024-09-11T09:51:41.557 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.557 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.557 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:41.557 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.124604+0000", 2024-09-11T09:51:41.557 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.6548999999999998e-05 2024-09-11T09:51:41.557 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:41.558 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:41.558 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:41.558 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.558 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.558 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4694.0:43 2.0 e53/15 2:16c8ae84:ns::rbd_data.1256a0c02879.0000000000000005:head v 53'2, mlcod=0'0)", 2024-09-11T09:51:41.558 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.158559+0000", 2024-09-11T09:51:41.558 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 18.282840383, 2024-09-11T09:51:41.558 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.016713677999999999, 2024-09-11T09:51:41.558 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:41.558 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:41.558 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:41.558 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:41.559 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.559 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:41.559 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.158559+0000", 2024-09-11T09:51:41.559 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:41.559 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.559 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.559 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:41.559 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.158559+0000", 2024-09-11T09:51:41.559 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:41.559 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.559 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.559 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:41.559 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.158560+0000", 2024-09-11T09:51:41.559 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1459999999999999e-06 2024-09-11T09:51:41.560 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.560 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.560 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:41.560 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.166465+0000", 2024-09-11T09:51:41.560 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.007904767 2024-09-11T09:51:41.560 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.560 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.560 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:41.560 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.166491+0000", 2024-09-11T09:51:41.560 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.6308e-05 2024-09-11T09:51:41.561 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.561 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.561 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:41.561 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.166502+0000", 2024-09-11T09:51:41.561 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0736e-05 2024-09-11T09:51:41.561 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.561 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.561 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:41.561 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.166564+0000", 2024-09-11T09:51:41.561 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.2578999999999998e-05 2024-09-11T09:51:41.561 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.562 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.562 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:41.562 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.166585+0000", 2024-09-11T09:51:41.562 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.0395999999999999e-05 2024-09-11T09:51:41.562 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.562 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.562 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:41.562 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.175227+0000", 2024-09-11T09:51:41.562 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0086426420000000007 2024-09-11T09:51:41.563 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.563 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.563 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:41.563 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.175272+0000", 2024-09-11T09:51:41.563 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.5104e-05 2024-09-11T09:51:41.563 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:41.563 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:41.563 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:41.563 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.563 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.563 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4694.0:59 2.2 e53/15 2:41adefc9:ns::rbd_data.1256a0c02879.000000000000000e:head v 53'5, mlcod=53'3)", 2024-09-11T09:51:41.563 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.163389+0000", 2024-09-11T09:51:41.563 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 18.278010407, 2024-09-11T09:51:41.564 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.013095719, 2024-09-11T09:51:41.564 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:41.564 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:41.564 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:41.564 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:41.564 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.564 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:41.564 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.163389+0000", 2024-09-11T09:51:41.564 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:41.564 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.564 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.564 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:41.564 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.163389+0000", 2024-09-11T09:51:41.565 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:41.565 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.565 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.565 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:41.565 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.163390+0000", 2024-09-11T09:51:41.565 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.336e-06 2024-09-11T09:51:41.565 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.565 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.565 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:41.565 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.170335+0000", 2024-09-11T09:51:41.565 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0069451080000000002 2024-09-11T09:51:41.565 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.565 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.566 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:41.566 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.170354+0000", 2024-09-11T09:51:41.566 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.8423e-05 2024-09-11T09:51:41.566 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.566 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.566 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:41.566 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.170366+0000", 2024-09-11T09:51:41.566 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2235999999999999e-05 2024-09-11T09:51:41.566 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.566 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.566 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:41.566 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.170439+0000", 2024-09-11T09:51:41.567 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.2989000000000002e-05 2024-09-11T09:51:41.567 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.567 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.567 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:41.567 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.170460+0000", 2024-09-11T09:51:41.567 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.1211000000000001e-05 2024-09-11T09:51:41.567 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.567 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.567 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:41.567 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.176442+0000", 2024-09-11T09:51:41.567 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0059821969999999999 2024-09-11T09:51:41.567 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.568 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.568 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:41.568 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.176484+0000", 2024-09-11T09:51:41.568 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.2218999999999997e-05 2024-09-11T09:51:41.568 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:41.568 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:41.568 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:41.568 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:41.568 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:41.568 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:51:41.570 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-11T09:51:41.570 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:51:41.571+0000 7f3c96ec9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31909) UID: 0 2024-09-11T09:51:41.600 INFO:tasks.workunit.client.0.smithi107.stderr:+ DEV= 2024-09-11T09:51:41.601 INFO:tasks.workunit.client.0.smithi107.stderr:++ rbd snap ls rbd/testrbdnbd27121 2024-09-11T09:51:41.601 INFO:tasks.workunit.client.0.smithi107.stderr:++ awk '$2 == "snap" {print $1}' 2024-09-11T09:51:41.646 INFO:tasks.workunit.client.0.smithi107.stderr:+ SNAPID=3 2024-09-11T09:51:41.646 INFO:tasks.workunit.client.0.smithi107.stderr:++ _sudo rbd device --device-type nbd map --snap-id 3 rbd/testrbdnbd27121 2024-09-11T09:51:41.646 INFO:tasks.workunit.client.0.smithi107.stderr:++ local cmd 2024-09-11T09:51:41.646 INFO:tasks.workunit.client.0.smithi107.stderr:+++ id -u 2024-09-11T09:51:41.647 INFO:tasks.workunit.client.0.smithi107.stderr:++ '[' 1000 -eq 0 ']' 2024-09-11T09:51:41.647 INFO:tasks.workunit.client.0.smithi107.stderr:+++ which rbd 2024-09-11T09:51:41.649 INFO:tasks.workunit.client.0.smithi107.stderr:++ cmd=/usr/bin/rbd 2024-09-11T09:51:41.649 INFO:tasks.workunit.client.0.smithi107.stderr:++ shift 2024-09-11T09:51:41.649 INFO:tasks.workunit.client.0.smithi107.stderr:++ sudo -nE /usr/bin/rbd device --device-type nbd map --snap-id 3 rbd/testrbdnbd27121 2024-09-11T09:51:41.670 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:51:41.672+0000 7f3c96ec9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31909) UID: 0 2024-09-11T09:51:41.718 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:51:41.718 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [], 2024-09-11T09:51:41.718 INFO:teuthology.orchestra.run.smithi017.stdout: "num_ops": 0 2024-09-11T09:51:41.718 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:51:41.728 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-11T09:51:41.757 INFO:tasks.workunit.client.0.smithi107.stderr:+ DEV=/dev/nbd0 2024-09-11T09:51:41.757 INFO:tasks.workunit.client.0.smithi107.stderr:+ get_pid rbd 2024-09-11T09:51:41.758 INFO:tasks.workunit.client.0.smithi107.stderr:+ local pool=rbd 2024-09-11T09:51:41.758 INFO:tasks.workunit.client.0.smithi107.stderr:+ local ns= 2024-09-11T09:51:41.758 INFO:tasks.workunit.client.0.smithi107.stderr:++ rbd device --device-type nbd --format xml list 2024-09-11T09:51:41.758 INFO:tasks.workunit.client.0.smithi107.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd27121'\''][device='\''/dev/nbd0'\'']/id' 2024-09-11T09:51:41.764 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-11T09:51:41.772 INFO:tasks.ceph.osd.2.smithi017.stderr:2024-09-11T09:51:41.772+0000 7fd20b170640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31672) UID: 0 2024-09-11T09:51:41.821 INFO:tasks.workunit.client.0.smithi107.stderr:+ PID=28538 2024-09-11T09:51:41.821 INFO:tasks.workunit.client.0.smithi107.stderr:+ test -n 28538 2024-09-11T09:51:41.821 INFO:tasks.workunit.client.0.smithi107.stderr:+ ps -p 28538 -C rbd-nbd 2024-09-11T09:51:41.832 INFO:tasks.workunit.client.0.smithi107.stdout: PID TTY TIME CMD 2024-09-11T09:51:41.832 INFO:tasks.workunit.client.0.smithi107.stdout: 28538 ? 00:00:00 rbd-nbd 2024-09-11T09:51:41.833 INFO:tasks.workunit.client.0.smithi107.stderr:+ unmap_device '--snap-id 3 testrbdnbd27121' 28538 2024-09-11T09:51:41.833 INFO:tasks.workunit.client.0.smithi107.stderr:+ local 'args=--snap-id 3 testrbdnbd27121' 2024-09-11T09:51:41.833 INFO:tasks.workunit.client.0.smithi107.stderr:+ local pid=28538 2024-09-11T09:51:41.833 INFO:tasks.workunit.client.0.smithi107.stderr:+ _sudo rbd device --device-type nbd unmap --snap-id 3 testrbdnbd27121 2024-09-11T09:51:41.833 INFO:tasks.workunit.client.0.smithi107.stderr:+ local cmd 2024-09-11T09:51:41.833 INFO:tasks.workunit.client.0.smithi107.stderr:++ id -u 2024-09-11T09:51:41.834 INFO:tasks.workunit.client.0.smithi107.stderr:+ '[' 1000 -eq 0 ']' 2024-09-11T09:51:41.834 INFO:tasks.workunit.client.0.smithi107.stderr:++ which rbd 2024-09-11T09:51:41.835 INFO:tasks.workunit.client.0.smithi107.stderr:+ cmd=/usr/bin/rbd 2024-09-11T09:51:41.835 INFO:tasks.workunit.client.0.smithi107.stderr:+ shift 2024-09-11T09:51:41.835 INFO:tasks.workunit.client.0.smithi107.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd unmap --snap-id 3 testrbdnbd27121 2024-09-11T09:51:41.840 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:51:41.840 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [], 2024-09-11T09:51:41.840 INFO:teuthology.orchestra.run.smithi017.stdout: "complaint_time": 30, 2024-09-11T09:51:41.840 INFO:teuthology.orchestra.run.smithi017.stdout: "num_blocked_ops": 0 2024-09-11T09:51:41.840 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:51:41.849 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-11T09:51:41.893 INFO:tasks.ceph.osd.0.smithi017.stderr:2024-09-11T09:51:41.893+0000 7fb73c0f1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31665) UID: 0 2024-09-11T09:51:41.925 INFO:tasks.workunit.client.0.smithi107.stderr:+ rbd device --device-type nbd list 2024-09-11T09:51:41.925 INFO:tasks.workunit.client.0.smithi107.stderr:+ expect_false grep '^28538\b' 2024-09-11T09:51:41.925 INFO:tasks.workunit.client.0.smithi107.stderr:+ grep '^28538\b' 2024-09-11T09:51:41.970 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:51:41.970 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 20, 2024-09-11T09:51:41.970 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 600, 2024-09-11T09:51:41.970 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [ 2024-09-11T09:51:41.970 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.970 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:31 2.1 2:8654721b:::rbd_data.1250f5e25b80.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:41.970 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.896024+0000", 2024-09-11T09:51:41.970 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 19.072557216, 2024-09-11T09:51:41.971 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.023148401999999998, 2024-09-11T09:51:41.971 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:41.971 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:41.971 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:41.971 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:41.971 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:51:41.971 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:51:41.971 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 31 2024-09-11T09:51:41.971 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.971 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:41.971 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.972 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:41.972 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.896024+0000", 2024-09-11T09:51:41.972 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:41.972 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.972 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.972 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:41.972 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.896024+0000", 2024-09-11T09:51:41.972 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:41.972 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.973 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.973 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:41.973 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.896025+0000", 2024-09-11T09:51:41.973 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.54e-07 2024-09-11T09:51:41.973 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.973 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.973 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:41.973 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.904287+0000", 2024-09-11T09:51:41.973 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0082624729999999993 2024-09-11T09:51:41.973 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.973 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.973 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:41.974 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.904305+0000", 2024-09-11T09:51:41.974 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7912999999999999e-05 2024-09-11T09:51:41.974 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.974 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.974 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:41.974 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.904312+0000", 2024-09-11T09:51:41.974 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.0230000000000004e-06 2024-09-11T09:51:41.974 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.974 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.974 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:41.975 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.904395+0000", 2024-09-11T09:51:41.975 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.2355999999999996e-05 2024-09-11T09:51:41.975 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.975 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.975 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:41.975 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.904499+0000", 2024-09-11T09:51:41.975 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000104672 2024-09-11T09:51:41.975 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.975 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.975 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1", 2024-09-11T09:51:41.975 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.904594+0000", 2024-09-11T09:51:41.975 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.5129000000000003e-05 2024-09-11T09:51:41.975 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.976 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.976 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:41.976 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.918183+0000", 2024-09-11T09:51:41.976 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.013588355999999999 2024-09-11T09:51:41.976 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.976 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.976 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:41.976 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919053+0000", 2024-09-11T09:51:41.976 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00087066800000000001 2024-09-11T09:51:41.977 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.977 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.977 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:41.977 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919125+0000", 2024-09-11T09:51:41.977 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.1333000000000003e-05 2024-09-11T09:51:41.977 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.977 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.977 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:41.977 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919172+0000", 2024-09-11T09:51:41.977 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.7725000000000001e-05 2024-09-11T09:51:41.978 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:41.978 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:41.978 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:41.978 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.978 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.978 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4688.0:42 2.3 e53/15 2:c54742d8:::rbd_data.1250f5e25b80.0000000000000006:head v 53'2, mlcod=0'0)", 2024-09-11T09:51:41.978 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.939272+0000", 2024-09-11T09:51:41.978 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 19.029309231999999, 2024-09-11T09:51:41.978 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.010821958, 2024-09-11T09:51:41.978 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:41.978 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:41.979 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:41.979 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:41.979 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.979 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:41.979 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.939272+0000", 2024-09-11T09:51:41.979 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:41.979 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.979 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.979 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:41.979 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.939272+0000", 2024-09-11T09:51:41.979 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:41.979 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.980 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.980 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:41.980 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.939273+0000", 2024-09-11T09:51:41.980 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.7899999999999997e-07 2024-09-11T09:51:41.980 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.980 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.980 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:41.980 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.944403+0000", 2024-09-11T09:51:41.980 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0051302129999999998 2024-09-11T09:51:41.980 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.981 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.981 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:41.981 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.944413+0000", 2024-09-11T09:51:41.981 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0182e-05 2024-09-11T09:51:41.981 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.981 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.981 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:41.981 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.944420+0000", 2024-09-11T09:51:41.981 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.7000000000000002e-06 2024-09-11T09:51:41.981 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.981 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.981 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:41.981 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.944482+0000", 2024-09-11T09:51:41.982 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.1911999999999999e-05 2024-09-11T09:51:41.982 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.982 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.982 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:41.982 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.944508+0000", 2024-09-11T09:51:41.982 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.6582000000000001e-05 2024-09-11T09:51:41.982 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.982 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.982 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:41.982 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.950050+0000", 2024-09-11T09:51:41.982 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0055414080000000003 2024-09-11T09:51:41.982 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.983 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.983 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:41.983 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.950094+0000", 2024-09-11T09:51:41.983 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.4082000000000002e-05 2024-09-11T09:51:41.983 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:41.983 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:41.983 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:41.983 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.983 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.983 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:47 2.1 2:95f53b3e:::rbd_data.1250f5e25b80.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:41.983 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.948330+0000", 2024-09-11T09:51:41.983 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 19.020250888, 2024-09-11T09:51:41.983 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.025156603999999999, 2024-09-11T09:51:41.984 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:41.984 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:41.984 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:41.984 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:41.984 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:51:41.984 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:51:41.984 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 47 2024-09-11T09:51:41.984 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.984 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:41.984 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.984 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:41.985 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.948330+0000", 2024-09-11T09:51:41.985 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:41.985 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.985 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.985 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:41.985 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.948330+0000", 2024-09-11T09:51:41.985 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:41.985 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.985 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.985 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:41.985 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.948333+0000", 2024-09-11T09:51:41.985 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.481e-06 2024-09-11T09:51:41.986 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.986 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.986 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:41.986 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.960263+0000", 2024-09-11T09:51:41.986 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.011930671 2024-09-11T09:51:41.986 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.986 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.986 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:41.986 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.960289+0000", 2024-09-11T09:51:41.986 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.5573000000000001e-05 2024-09-11T09:51:41.986 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.986 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.986 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:41.986 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.960304+0000", 2024-09-11T09:51:41.987 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.4647000000000001e-05 2024-09-11T09:51:41.987 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.987 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.987 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:41.987 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.960390+0000", 2024-09-11T09:51:41.987 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.6088999999999995e-05 2024-09-11T09:51:41.987 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.987 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.987 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:41.987 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.960562+0000", 2024-09-11T09:51:41.987 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00017237400000000001 2024-09-11T09:51:41.987 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.988 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.988 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1", 2024-09-11T09:51:41.988 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.960725+0000", 2024-09-11T09:51:41.988 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00016321599999999999 2024-09-11T09:51:41.988 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.988 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.988 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:41.988 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.969465+0000", 2024-09-11T09:51:41.988 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0087396899999999996 2024-09-11T09:51:41.988 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.988 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.988 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:41.989 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.973393+0000", 2024-09-11T09:51:41.989 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0039274669999999996 2024-09-11T09:51:41.989 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.989 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.989 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:41.989 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.973443+0000", 2024-09-11T09:51:41.989 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.0340000000000003e-05 2024-09-11T09:51:41.989 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.989 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.989 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:41.989 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.973487+0000", 2024-09-11T09:51:41.990 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.4056e-05 2024-09-11T09:51:41.990 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:41.990 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:41.990 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:41.990 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.990 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.991 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:61 2.1 2:9af038b1:::rbd_data.1250f5e25b80.000000000000000f:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:41.991 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.975764+0000", 2024-09-11T09:51:41.991 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 18.992817364, 2024-09-11T09:51:41.991 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.016158891000000002, 2024-09-11T09:51:41.991 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:41.991 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:41.991 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:41.991 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:41.991 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:51:41.991 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:51:41.992 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 61 2024-09-11T09:51:41.992 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.992 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:41.992 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.992 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:41.992 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.975764+0000", 2024-09-11T09:51:41.992 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:41.992 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.992 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.992 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:41.992 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.975764+0000", 2024-09-11T09:51:41.992 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:41.993 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.993 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.993 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:41.993 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.975766+0000", 2024-09-11T09:51:41.993 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.767e-06 2024-09-11T09:51:41.993 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.993 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.993 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:41.993 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.983204+0000", 2024-09-11T09:51:41.993 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0074388400000000004 2024-09-11T09:51:41.994 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.994 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.994 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:41.994 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.983226+0000", 2024-09-11T09:51:41.994 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.1279000000000001e-05 2024-09-11T09:51:41.994 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.994 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.994 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:41.994 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.983238+0000", 2024-09-11T09:51:41.994 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2563e-05 2024-09-11T09:51:41.994 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.994 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.995 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:41.995 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.983318+0000", 2024-09-11T09:51:41.995 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.0075999999999997e-05 2024-09-11T09:51:41.995 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.995 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.995 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:41.995 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.983429+0000", 2024-09-11T09:51:41.995 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000110618 2024-09-11T09:51:41.995 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.995 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.996 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1", 2024-09-11T09:51:41.996 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.983525+0000", 2024-09-11T09:51:41.996 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.603e-05 2024-09-11T09:51:41.996 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.997 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.997 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:41.997 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.989004+0000", 2024-09-11T09:51:41.997 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0054785429999999998 2024-09-11T09:51:41.997 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.997 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.997 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:41.997 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.991788+0000", 2024-09-11T09:51:41.997 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0027845729999999998 2024-09-11T09:51:41.998 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.998 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.998 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:41.998 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.991870+0000", 2024-09-11T09:51:41.998 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.1567e-05 2024-09-11T09:51:41.998 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.998 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.998 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:41.998 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.991923+0000", 2024-09-11T09:51:41.998 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.3035000000000003e-05 2024-09-11T09:51:41.998 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:41.999 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:41.999 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:41.999 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:41.999 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:41.999 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4694.0:32 2.1 2:8a862b48:ns::rbd_data.1256a0c02879.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:41.999 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.084089+0000", 2024-09-11T09:51:41.999 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 18.88449189, 2024-09-11T09:51:41.999 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.020630553999999999, 2024-09-11T09:51:41.999 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:42.000 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:42.000 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:42.000 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:42.000 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4694", 2024-09-11T09:51:42.000 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:51:42.000 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 32 2024-09-11T09:51:42.000 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:42.000 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:42.000 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:42.000 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:42.000 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.084089+0000", 2024-09-11T09:51:42.000 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:42.000 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:42.001 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:42.001 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:42.001 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.084089+0000", 2024-09-11T09:51:42.001 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:42.001 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:42.001 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:42.001 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:42.001 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.084090+0000", 2024-09-11T09:51:42.001 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.2900000000000003e-07 2024-09-11T09:51:42.001 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:42.002 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:42.002 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:42.002 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.090866+0000", 2024-09-11T09:51:42.002 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0067756609999999997 2024-09-11T09:51:42.002 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:42.002 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:42.002 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:42.002 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.090891+0000", 2024-09-11T09:51:42.002 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.5119000000000001e-05 2024-09-11T09:51:42.003 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:42.003 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:42.003 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:42.003 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.090900+0000", 2024-09-11T09:51:42.003 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.4110000000000002e-06 2024-09-11T09:51:42.003 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:42.003 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:42.003 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:42.003 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.090968+0000", 2024-09-11T09:51:42.003 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.8041999999999997e-05 2024-09-11T09:51:42.003 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:42.004 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:42.004 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:42.004 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.091115+0000", 2024-09-11T09:51:42.004 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00014633499999999999 2024-09-11T09:51:42.004 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:42.004 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:42.004 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1", 2024-09-11T09:51:42.004 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.091284+0000", 2024-09-11T09:51:42.005 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000169836 2024-09-11T09:51:42.005 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:42.005 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:42.005 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:42.005 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.100896+0000", 2024-09-11T09:51:42.005 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0096118639999999995 2024-09-11T09:51:42.005 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:42.005 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:42.005 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:42.005 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.104604+0000", 2024-09-11T09:51:42.005 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0037074619999999999 2024-09-11T09:51:42.005 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:42.006 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:42.006 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:42.006 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.104669+0000", 2024-09-11T09:51:42.006 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.4836000000000001e-05 2024-09-11T09:51:42.006 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:42.006 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:42.006 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:42.006 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.104720+0000", 2024-09-11T09:51:42.006 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.1258999999999998e-05 2024-09-11T09:51:42.006 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:42.006 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:42.006 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:42.007 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:42.007 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:42.007 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4694.0:44 2.3 e53/15 2:cd1a7d1c:ns::rbd_data.1256a0c02879.0000000000000006:head v 53'3, mlcod=16'1)", 2024-09-11T09:51:42.007 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.120387+0000", 2024-09-11T09:51:42.007 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 18.848194237000001, 2024-09-11T09:51:42.007 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.011988468, 2024-09-11T09:51:42.007 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:42.007 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:42.007 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:42.007 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:42.007 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:42.007 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:42.007 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.120387+0000", 2024-09-11T09:51:42.008 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:42.008 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:42.008 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:42.008 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:42.008 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.120387+0000", 2024-09-11T09:51:42.008 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:42.008 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:42.008 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:42.008 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:42.008 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.120389+0000", 2024-09-11T09:51:42.008 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.018e-06 2024-09-11T09:51:42.008 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:42.008 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:42.009 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:42.009 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.127403+0000", 2024-09-11T09:51:42.009 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0070144769999999999 2024-09-11T09:51:42.009 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:42.009 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:42.009 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:42.009 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.127414+0000", 2024-09-11T09:51:42.009 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0991e-05 2024-09-11T09:51:42.009 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:42.009 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:42.009 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:42.009 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.127423+0000", 2024-09-11T09:51:42.009 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.8580000000000006e-06 2024-09-11T09:51:42.010 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:42.010 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:42.010 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:42.010 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.127468+0000", 2024-09-11T09:51:42.010 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.4830999999999998e-05 2024-09-11T09:51:42.010 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:42.010 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:42.010 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:42.010 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.127485+0000", 2024-09-11T09:51:42.010 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6838999999999999e-05 2024-09-11T09:51:42.011 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:42.011 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:42.011 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:42.011 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.132341+0000", 2024-09-11T09:51:42.011 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0048557959999999999 2024-09-11T09:51:42.011 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:42.011 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:42.011 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:42.011 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.132375+0000", 2024-09-11T09:51:42.011 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.4657999999999999e-05 2024-09-11T09:51:42.011 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:42.011 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:42.011 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:42.012 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:42.012 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:42.012 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4694.0:46 2.3 e53/15 2:ce0e1469:ns::rbd_data.1256a0c02879.0000000000000008:head v 53'4, mlcod=16'1)", 2024-09-11T09:51:42.012 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.129008+0000", 2024-09-11T09:51:42.012 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 18.839573650999998, 2024-09-11T09:51:42.012 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0087151220000000005, 2024-09-11T09:51:42.012 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:42.012 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:42.012 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:42.012 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:42.012 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:42.012 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:42.012 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.129008+0000", 2024-09-11T09:51:42.013 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:42.013 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:42.013 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:42.013 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:42.013 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.129008+0000", 2024-09-11T09:51:42.013 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:42.013 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:42.013 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:42.013 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:42.013 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.129009+0000", 2024-09-11T09:51:42.013 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.367e-06 2024-09-11T09:51:42.017 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:42.017 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:42.017 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:42.017 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.132839+0000", 2024-09-11T09:51:42.017 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0038297610000000001 2024-09-11T09:51:42.017 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:42.017 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:42.018 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:42.018 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.132849+0000", 2024-09-11T09:51:42.018 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0672e-05 2024-09-11T09:51:42.018 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:42.018 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:42.018 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:42.018 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.132856+0000", 2024-09-11T09:51:42.018 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.1380000000000004e-06 2024-09-11T09:51:42.018 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:42.018 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:42.018 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:42.018 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.132891+0000", 2024-09-11T09:51:42.018 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.5712e-05 2024-09-11T09:51:42.019 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:42.019 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:42.019 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:42.019 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.132907+0000", 2024-09-11T09:51:42.019 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6166000000000001e-05 2024-09-11T09:51:42.019 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:42.019 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:42.019 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:42.019 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.137681+0000", 2024-09-11T09:51:42.019 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0047735299999999998 2024-09-11T09:51:42.019 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:42.019 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:42.019 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:42.020 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.137723+0000", 2024-09-11T09:51:42.020 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.1776000000000001e-05 2024-09-11T09:51:42.020 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:42.020 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:42.020 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:42.020 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:42.020 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:42.020 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4694.0:48 2.3 e53/15 2:d2090b70:ns::rbd_data.1256a0c02879.0000000000000009:head v 53'5, mlcod=53'2)", 2024-09-11T09:51:42.020 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.135108+0000", 2024-09-11T09:51:42.020 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 18.833472753999999, 2024-09-11T09:51:42.020 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.010710679000000001, 2024-09-11T09:51:42.020 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:42.020 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:42.021 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:42.021 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:42.021 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:42.021 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:42.021 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.135108+0000", 2024-09-11T09:51:42.021 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:42.021 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:42.021 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:42.021 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:42.021 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.135108+0000", 2024-09-11T09:51:42.021 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:42.021 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:42.021 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:42.022 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:42.022 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.135110+0000", 2024-09-11T09:51:42.022 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.122e-06 2024-09-11T09:51:42.022 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:42.022 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:42.022 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:42.022 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.140876+0000", 2024-09-11T09:51:42.022 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.005766162 2024-09-11T09:51:42.022 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:42.022 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:42.022 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:42.022 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.140888+0000", 2024-09-11T09:51:42.022 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.201e-05 2024-09-11T09:51:42.022 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:42.023 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:42.023 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:42.023 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.140894+0000", 2024-09-11T09:51:42.023 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.1739999999999997e-06 2024-09-11T09:51:42.023 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:42.023 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:42.023 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:42.023 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.140971+0000", 2024-09-11T09:51:42.023 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.7495000000000003e-05 2024-09-11T09:51:42.023 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:42.023 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:42.023 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:42.023 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.140995+0000", 2024-09-11T09:51:42.023 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.3204999999999999e-05 2024-09-11T09:51:42.024 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:42.024 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:42.024 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:42.024 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.145757+0000", 2024-09-11T09:51:42.024 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0047627190000000003 2024-09-11T09:51:42.024 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:42.024 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:42.024 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:42.024 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.145819+0000", 2024-09-11T09:51:42.024 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.1791999999999994e-05 2024-09-11T09:51:42.024 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:42.024 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:42.024 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:42.025 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:42.025 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:42.025 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4694.0:50 2.1 2:83078786:ns::rbd_data.1256a0c02879.000000000000000a:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:42.025 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.141467+0000", 2024-09-11T09:51:42.025 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 18.827114030000001, 2024-09-11T09:51:42.025 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.012785889, 2024-09-11T09:51:42.025 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:42.025 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:42.026 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:42.026 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:42.026 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4694", 2024-09-11T09:51:42.026 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:51:42.026 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 50 2024-09-11T09:51:42.026 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:42.027 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:42.027 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:42.027 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:42.027 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.141467+0000", 2024-09-11T09:51:42.027 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:42.027 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:42.028 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:42.028 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:42.028 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.141467+0000", 2024-09-11T09:51:42.028 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:42.028 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:42.028 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:42.028 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:42.029 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.141468+0000", 2024-09-11T09:51:42.029 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1009999999999999e-06 2024-09-11T09:51:42.029 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:42.029 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:42.029 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:42.029 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.146432+0000", 2024-09-11T09:51:42.029 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0049640539999999999 2024-09-11T09:51:42.030 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:42.030 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:42.030 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:42.030 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.146446+0000", 2024-09-11T09:51:42.030 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3254e-05 2024-09-11T09:51:42.030 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:42.030 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:42.031 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:42.031 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.146452+0000", 2024-09-11T09:51:42.031 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.207e-06 2024-09-11T09:51:42.031 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:42.031 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:42.031 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:42.032 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.146510+0000", 2024-09-11T09:51:42.032 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.8322999999999997e-05 2024-09-11T09:51:42.032 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:42.032 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:42.032 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:42.032 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.146650+0000", 2024-09-11T09:51:42.033 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00013947100000000001 2024-09-11T09:51:42.033 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:42.033 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:42.033 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1", 2024-09-11T09:51:42.033 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.146779+0000", 2024-09-11T09:51:42.033 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000129059 2024-09-11T09:51:42.034 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:42.034 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:42.034 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:42.034 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.153778+0000", 2024-09-11T09:51:42.034 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0069997130000000003 2024-09-11T09:51:42.034 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:42.034 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:42.034 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:42.034 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.154180+0000", 2024-09-11T09:51:42.034 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00040159800000000003 2024-09-11T09:51:42.035 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:42.035 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:42.035 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:42.035 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.154218+0000", 2024-09-11T09:51:42.035 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.7795e-05 2024-09-11T09:51:42.035 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:42.035 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:42.035 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:42.035 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.154253+0000", 2024-09-11T09:51:42.035 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.5314000000000001e-05 2024-09-11T09:51:42.035 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:42.035 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:42.035 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:42.036 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:42.036 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:42.036 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4694.0:54 2.3 e53/15 2:c44f0253:ns::rbd_data.1256a0c02879.000000000000000b:head v 53'6, mlcod=53'4)", 2024-09-11T09:51:42.036 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.154926+0000", 2024-09-11T09:51:42.036 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 18.813655655000002, 2024-09-11T09:51:42.036 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0089497099999999996, 2024-09-11T09:51:42.036 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:42.036 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:42.036 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:42.036 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:42.036 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:42.036 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:42.036 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.154926+0000", 2024-09-11T09:51:42.037 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:42.037 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:42.037 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:42.037 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:42.037 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.154926+0000", 2024-09-11T09:51:42.037 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:42.037 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:42.037 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:42.037 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:42.037 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.154927+0000", 2024-09-11T09:51:42.037 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.7000000000000003e-07 2024-09-11T09:51:42.037 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:42.037 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:42.038 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:42.038 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.158712+0000", 2024-09-11T09:51:42.038 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0037855469999999998 2024-09-11T09:51:42.038 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:42.038 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:42.038 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:42.038 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.158726+0000", 2024-09-11T09:51:42.038 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3621000000000001e-05 2024-09-11T09:51:42.038 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:42.038 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:42.038 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:42.038 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.158732+0000", 2024-09-11T09:51:42.038 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.9089999999999997e-06 2024-09-11T09:51:42.039 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:42.039 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:42.039 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:42.039 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.158781+0000", 2024-09-11T09:51:42.039 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.8900000000000003e-05 2024-09-11T09:51:42.039 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:42.039 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:42.039 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:42.039 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.158805+0000", 2024-09-11T09:51:42.039 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.44e-05 2024-09-11T09:51:42.039 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:42.039 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:42.040 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:42.040 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.163826+0000", 2024-09-11T09:51:42.040 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0050206399999999998 2024-09-11T09:51:42.040 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:42.040 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:42.040 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:42.040 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.163875+0000", 2024-09-11T09:51:42.040 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.9722999999999998e-05 2024-09-11T09:51:42.040 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:42.040 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:42.040 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:42.040 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:42.040 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:42.041 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4694.0:56 2.3 e53/15 2:c462f029:ns::rbd_data.1256a0c02879.000000000000000c:head v 53'7, mlcod=53'4)", 2024-09-11T09:51:42.041 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.159549+0000", 2024-09-11T09:51:42.041 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 18.809031949000001, 2024-09-11T09:51:42.041 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.015591666000000001, 2024-09-11T09:51:42.041 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:42.041 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:42.041 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:42.041 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:42.041 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:42.041 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:42.041 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.159549+0000", 2024-09-11T09:51:42.041 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:42.041 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:42.042 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:42.042 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:42.042 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.159549+0000", 2024-09-11T09:51:42.042 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:42.042 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:42.042 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:42.042 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:42.042 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.159550+0000", 2024-09-11T09:51:42.042 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.1500000000000004e-07 2024-09-11T09:51:42.042 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:42.042 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:42.042 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:42.042 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.167834+0000", 2024-09-11T09:51:42.042 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0082840220000000003 2024-09-11T09:51:42.043 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:42.043 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:42.043 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:42.043 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.167857+0000", 2024-09-11T09:51:42.043 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.3337e-05 2024-09-11T09:51:42.043 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:42.043 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:42.043 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:42.043 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.167869+0000", 2024-09-11T09:51:42.043 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1556000000000001e-05 2024-09-11T09:51:42.043 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:42.043 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:42.043 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:42.044 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.167939+0000", 2024-09-11T09:51:42.044 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.0674999999999997e-05 2024-09-11T09:51:42.044 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:42.044 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:42.044 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:42.044 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.167967+0000", 2024-09-11T09:51:42.044 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.7322999999999999e-05 2024-09-11T09:51:42.044 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:42.044 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:42.044 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:42.044 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.175079+0000", 2024-09-11T09:51:42.044 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0071126150000000001 2024-09-11T09:51:42.045 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:42.045 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:42.045 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:42.045 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.175141+0000", 2024-09-11T09:51:42.045 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.1523e-05 2024-09-11T09:51:42.045 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:42.045 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:42.045 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:42.045 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:42.045 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:42.045 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.5002.0:25 2.1 2:8654721b:::rbd_data.1250f5e25b80.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,write 393216~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-09-11T09:51:42.045 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:38.367683+0000", 2024-09-11T09:51:42.045 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 3.600897953, 2024-09-11T09:51:42.045 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.005437641, 2024-09-11T09:51:42.046 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:42.046 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:42.046 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:42.046 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:42.046 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.5002", 2024-09-11T09:51:42.046 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/2204440692", 2024-09-11T09:51:42.046 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 25 2024-09-11T09:51:42.046 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:42.046 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:42.046 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:42.047 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:42.047 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.367683+0000", 2024-09-11T09:51:42.047 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:42.047 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:42.047 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:42.047 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:42.047 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.367683+0000", 2024-09-11T09:51:42.047 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:42.047 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:42.047 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:42.047 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:42.047 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.367684+0000", 2024-09-11T09:51:42.047 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.9699999999999995e-07 2024-09-11T09:51:42.048 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:42.048 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:42.048 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:42.048 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.367842+0000", 2024-09-11T09:51:42.048 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000158362 2024-09-11T09:51:42.048 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:42.048 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:42.048 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:42.048 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.367856+0000", 2024-09-11T09:51:42.048 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3360000000000001e-05 2024-09-11T09:51:42.048 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:42.048 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:42.048 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:42.049 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.367863+0000", 2024-09-11T09:51:42.049 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.1609999999999997e-06 2024-09-11T09:51:42.049 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:42.049 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:42.049 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:42.049 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.369486+0000", 2024-09-11T09:51:42.049 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0016228449999999999 2024-09-11T09:51:42.049 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:42.049 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:42.049 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:42.049 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.369538+0000", 2024-09-11T09:51:42.049 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.2138000000000001e-05 2024-09-11T09:51:42.050 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:42.050 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:42.050 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1", 2024-09-11T09:51:42.050 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.369620+0000", 2024-09-11T09:51:42.050 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.2235000000000001e-05 2024-09-11T09:51:42.050 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:42.050 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:42.050 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:42.050 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.372192+0000", 2024-09-11T09:51:42.050 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.002572024 2024-09-11T09:51:42.050 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:42.050 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:42.050 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:42.051 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.373045+0000", 2024-09-11T09:51:42.051 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00085311000000000002 2024-09-11T09:51:42.051 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:42.051 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:42.051 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:42.051 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.373086+0000", 2024-09-11T09:51:42.051 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.0731e-05 2024-09-11T09:51:42.051 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:42.051 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:42.051 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:42.051 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.373121+0000", 2024-09-11T09:51:42.051 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.4978000000000003e-05 2024-09-11T09:51:42.051 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:42.052 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:42.052 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:42.052 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:42.052 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:42.052 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.5002.0:26 2.1 2:8654721b:::rbd_data.1250f5e25b80.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,write 524288~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-09-11T09:51:42.052 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:38.367903+0000", 2024-09-11T09:51:42.052 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 3.6006778399999999, 2024-09-11T09:51:42.052 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0054161929999999997, 2024-09-11T09:51:42.052 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:42.052 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:42.053 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:42.053 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:42.053 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.5002", 2024-09-11T09:51:42.053 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/2204440692", 2024-09-11T09:51:42.053 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 26 2024-09-11T09:51:42.053 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:42.053 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:42.053 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:42.053 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:42.053 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.367903+0000", 2024-09-11T09:51:42.053 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:42.053 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:42.054 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:42.054 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:42.054 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.367903+0000", 2024-09-11T09:51:42.054 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:42.054 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:42.054 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:42.054 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:42.054 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.367904+0000", 2024-09-11T09:51:42.054 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.5199999999999996e-07 2024-09-11T09:51:42.054 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:42.054 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:42.054 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:42.054 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.368108+0000", 2024-09-11T09:51:42.055 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00020412499999999999 2024-09-11T09:51:42.055 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:42.055 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:42.055 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:42.055 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.368120+0000", 2024-09-11T09:51:42.055 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1503e-05 2024-09-11T09:51:42.055 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:42.055 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:42.055 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:42.055 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.368127+0000", 2024-09-11T09:51:42.055 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.4800000000000004e-06 2024-09-11T09:51:42.055 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:42.055 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:42.056 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:42.056 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.370401+0000", 2024-09-11T09:51:42.056 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0022742069999999999 2024-09-11T09:51:42.056 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:42.056 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:42.056 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:42.056 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.370494+0000", 2024-09-11T09:51:42.056 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.2161000000000001e-05 2024-09-11T09:51:42.056 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:42.057 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:42.057 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1", 2024-09-11T09:51:42.057 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.370644+0000", 2024-09-11T09:51:42.057 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00015055299999999999 2024-09-11T09:51:42.057 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:42.057 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:42.057 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:42.057 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.373144+0000", 2024-09-11T09:51:42.057 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.002499869 2024-09-11T09:51:42.057 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:42.057 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:42.057 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:42.057 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.373218+0000", 2024-09-11T09:51:42.058 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.4210999999999998e-05 2024-09-11T09:51:42.058 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:42.058 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:42.058 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:42.058 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.373274+0000", 2024-09-11T09:51:42.058 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.6069999999999997e-05 2024-09-11T09:51:42.058 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:42.058 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:42.058 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:42.058 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.373320+0000", 2024-09-11T09:51:42.058 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.5361999999999997e-05 2024-09-11T09:51:42.058 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:42.058 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:42.059 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:42.059 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:42.059 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:42.059 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.5002.0:27 2.1 2:8654721b:::rbd_data.1250f5e25b80.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,write 655360~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-09-11T09:51:42.059 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:38.368143+0000", 2024-09-11T09:51:42.059 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 3.6004383249999998, 2024-09-11T09:51:42.059 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0053157639999999997, 2024-09-11T09:51:42.059 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:42.060 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:42.060 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:42.060 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:42.060 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.5002", 2024-09-11T09:51:42.060 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/2204440692", 2024-09-11T09:51:42.060 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 27 2024-09-11T09:51:42.060 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:42.060 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:42.060 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:42.060 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:42.061 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.368143+0000", 2024-09-11T09:51:42.061 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:42.061 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:42.061 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:42.061 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:42.061 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.368143+0000", 2024-09-11T09:51:42.061 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:42.061 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:42.061 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:42.061 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:42.062 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.368144+0000", 2024-09-11T09:51:42.062 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.1800000000000005e-07 2024-09-11T09:51:42.062 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:42.062 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:42.062 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:42.062 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.368286+0000", 2024-09-11T09:51:42.062 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000141934 2024-09-11T09:51:42.062 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:42.062 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:42.062 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:42.062 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.368298+0000", 2024-09-11T09:51:42.063 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2926e-05 2024-09-11T09:51:42.063 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:42.063 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:42.063 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:42.063 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.368305+0000", 2024-09-11T09:51:42.063 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.7100000000000001e-06 2024-09-11T09:51:42.063 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:42.063 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:42.063 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:42.063 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.371358+0000", 2024-09-11T09:51:42.063 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0030531450000000002 2024-09-11T09:51:42.064 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:42.064 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:42.064 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:42.064 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.371448+0000", 2024-09-11T09:51:42.064 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.9899000000000001e-05 2024-09-11T09:51:42.064 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:42.064 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:42.064 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1", 2024-09-11T09:51:42.064 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.371593+0000", 2024-09-11T09:51:42.064 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000144948 2024-09-11T09:51:42.064 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:42.064 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:42.064 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:42.065 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.373387+0000", 2024-09-11T09:51:42.065 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0017935340000000001 2024-09-11T09:51:42.065 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:42.065 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:42.065 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:42.065 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.373397+0000", 2024-09-11T09:51:42.065 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0550999999999999e-05 2024-09-11T09:51:42.065 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:42.065 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:42.065 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:42.065 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.373428+0000", 2024-09-11T09:51:42.065 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.0731000000000001e-05 2024-09-11T09:51:42.065 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:42.066 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:42.066 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:42.066 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.373459+0000", 2024-09-11T09:51:42.066 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.0667999999999998e-05 2024-09-11T09:51:42.066 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:42.066 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:42.066 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:42.066 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:42.066 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:42.066 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.5002.0:28 2.1 2:8654721b:::rbd_data.1250f5e25b80.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,write 786432~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-09-11T09:51:42.066 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:38.368321+0000", 2024-09-11T09:51:42.066 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 3.600260628, 2024-09-11T09:51:42.067 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0060769780000000002, 2024-09-11T09:51:42.067 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:42.067 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:42.067 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:42.067 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:42.067 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.5002", 2024-09-11T09:51:42.067 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/2204440692", 2024-09-11T09:51:42.067 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 28 2024-09-11T09:51:42.067 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:42.067 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:42.067 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:42.067 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:42.067 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.368321+0000", 2024-09-11T09:51:42.068 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:42.068 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:42.068 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:42.068 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:42.068 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.368321+0000", 2024-09-11T09:51:42.068 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:42.068 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:42.068 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:42.068 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:42.068 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.368321+0000", 2024-09-11T09:51:42.068 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.6599999999999996e-07 2024-09-11T09:51:42.068 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:42.069 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:42.069 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:42.069 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.368452+0000", 2024-09-11T09:51:42.069 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000130858 2024-09-11T09:51:42.069 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:42.069 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:42.069 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:42.069 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.368477+0000", 2024-09-11T09:51:42.069 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.4627e-05 2024-09-11T09:51:42.069 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:42.070 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:42.070 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:42.070 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.368486+0000", 2024-09-11T09:51:42.070 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.2070000000000001e-06 2024-09-11T09:51:42.070 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:42.070 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:42.070 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:42.070 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.372253+0000", 2024-09-11T09:51:42.070 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0037666779999999999 2024-09-11T09:51:42.070 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:42.070 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:42.070 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:42.071 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.372346+0000", 2024-09-11T09:51:42.071 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.3222999999999998e-05 2024-09-11T09:51:42.071 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:42.071 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:42.071 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1", 2024-09-11T09:51:42.071 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.372489+0000", 2024-09-11T09:51:42.071 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00014329300000000001 2024-09-11T09:51:42.071 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:42.071 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:42.071 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:42.071 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.374251+0000", 2024-09-11T09:51:42.071 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0017618550000000001 2024-09-11T09:51:42.071 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:42.072 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:42.072 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:42.072 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.374293+0000", 2024-09-11T09:51:42.072 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.1891999999999998e-05 2024-09-11T09:51:42.072 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:42.072 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:42.072 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:42.072 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.374366+0000", 2024-09-11T09:51:42.072 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.3446000000000001e-05 2024-09-11T09:51:42.072 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:42.072 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:42.072 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:42.073 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.374398+0000", 2024-09-11T09:51:42.073 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.1232999999999997e-05 2024-09-11T09:51:42.073 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:42.073 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:42.073 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:42.073 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:42.073 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:42.073 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.5002.0:29 2.1 2:8654721b:::rbd_data.1250f5e25b80.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,write 917504~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-09-11T09:51:42.073 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:38.368498+0000", 2024-09-11T09:51:42.073 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 3.6000827900000001, 2024-09-11T09:51:42.073 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0069133370000000003, 2024-09-11T09:51:42.073 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:42.074 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:42.074 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:42.074 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:42.074 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.5002", 2024-09-11T09:51:42.074 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/2204440692", 2024-09-11T09:51:42.074 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 29 2024-09-11T09:51:42.074 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:42.074 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:42.074 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:42.074 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:42.074 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.368498+0000", 2024-09-11T09:51:42.074 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:42.074 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:42.075 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:42.075 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:42.075 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.368498+0000", 2024-09-11T09:51:42.075 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:42.075 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:42.075 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:42.075 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:42.075 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.368499+0000", 2024-09-11T09:51:42.075 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.2200000000000003e-07 2024-09-11T09:51:42.075 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:42.075 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:42.075 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:42.075 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.368604+0000", 2024-09-11T09:51:42.076 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000104627 2024-09-11T09:51:42.076 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:42.076 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:42.076 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:42.076 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.368614+0000", 2024-09-11T09:51:42.076 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0139999999999999e-05 2024-09-11T09:51:42.076 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:42.076 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:42.076 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:42.076 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.368620+0000", 2024-09-11T09:51:42.076 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.5100000000000004e-06 2024-09-11T09:51:42.076 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:42.076 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:42.077 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:42.077 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.373501+0000", 2024-09-11T09:51:42.077 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0048804749999999996 2024-09-11T09:51:42.077 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:42.077 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:42.077 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:42.077 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.373570+0000", 2024-09-11T09:51:42.077 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.8831000000000006e-05 2024-09-11T09:51:42.077 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:42.077 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:42.077 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1", 2024-09-11T09:51:42.077 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.373674+0000", 2024-09-11T09:51:42.078 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000103755 2024-09-11T09:51:42.078 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:42.078 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:42.078 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:42.078 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.374588+0000", 2024-09-11T09:51:42.078 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00091421700000000005 2024-09-11T09:51:42.078 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:42.078 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:42.078 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:42.078 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.375350+0000", 2024-09-11T09:51:42.078 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00076217699999999995 2024-09-11T09:51:42.078 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:42.078 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:42.078 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:42.079 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.375384+0000", 2024-09-11T09:51:42.079 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.4486000000000002e-05 2024-09-11T09:51:42.079 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:42.079 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:42.079 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:42.079 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.375412+0000", 2024-09-11T09:51:42.079 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.7396999999999999e-05 2024-09-11T09:51:42.079 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:42.079 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:42.079 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:42.079 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:42.079 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:42.079 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.5002.0:34 2.1 2:8654721b:::rbd_data.1250f5e25b80.0000000000000000:head [write 1572864~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-09-11T09:51:42.080 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:38.376433+0000", 2024-09-11T09:51:42.080 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 3.5921481179999999, 2024-09-11T09:51:42.080 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0059348530000000004, 2024-09-11T09:51:42.080 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:42.080 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:42.080 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:42.080 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:42.080 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.5002", 2024-09-11T09:51:42.081 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/2204440692", 2024-09-11T09:51:42.081 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 34 2024-09-11T09:51:42.081 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:42.081 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:42.081 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:42.081 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:42.081 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.376433+0000", 2024-09-11T09:51:42.081 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:42.081 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:42.081 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:42.081 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:42.081 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.376433+0000", 2024-09-11T09:51:42.081 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:42.082 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:42.082 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:42.082 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:42.082 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.376434+0000", 2024-09-11T09:51:42.082 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.0200000000000002e-07 2024-09-11T09:51:42.082 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:42.082 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:42.082 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:42.082 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.376505+0000", 2024-09-11T09:51:42.082 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.1273999999999996e-05 2024-09-11T09:51:42.082 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:42.082 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:42.082 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:42.083 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.376513+0000", 2024-09-11T09:51:42.083 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.7419999999999997e-06 2024-09-11T09:51:42.083 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:42.083 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:42.083 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:42.083 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.376517+0000", 2024-09-11T09:51:42.083 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.6e-06 2024-09-11T09:51:42.083 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:42.083 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:42.083 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:42.083 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.379223+0000", 2024-09-11T09:51:42.083 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0027053149999999998 2024-09-11T09:51:42.083 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:42.083 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:42.084 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:42.084 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.379273+0000", 2024-09-11T09:51:42.084 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.0927000000000002e-05 2024-09-11T09:51:42.084 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:42.084 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:42.084 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1", 2024-09-11T09:51:42.084 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.379353+0000", 2024-09-11T09:51:42.084 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.9431999999999994e-05 2024-09-11T09:51:42.084 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:42.085 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:42.085 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:42.085 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.382252+0000", 2024-09-11T09:51:42.085 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0028991569999999999 2024-09-11T09:51:42.085 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:42.085 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:42.086 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:42.086 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.382303+0000", 2024-09-11T09:51:42.086 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.058e-05 2024-09-11T09:51:42.086 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:42.086 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:42.086 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:42.086 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.382339+0000", 2024-09-11T09:51:42.086 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.6089e-05 2024-09-11T09:51:42.086 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:42.086 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:42.086 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:42.086 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.382368+0000", 2024-09-11T09:51:42.086 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.9235e-05 2024-09-11T09:51:42.087 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:42.087 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:42.087 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:42.087 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:42.087 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:42.087 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.5002.0:35 2.1 2:8654721b:::rbd_data.1250f5e25b80.0000000000000000:head [write 1703936~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-09-11T09:51:42.087 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:38.376525+0000", 2024-09-11T09:51:42.087 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 3.5920559499999998, 2024-09-11T09:51:42.087 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0060109220000000001, 2024-09-11T09:51:42.087 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:42.087 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:42.087 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:42.088 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:42.088 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.5002", 2024-09-11T09:51:42.088 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/2204440692", 2024-09-11T09:51:42.088 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 35 2024-09-11T09:51:42.088 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:42.088 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:42.088 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:42.088 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:42.088 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.376525+0000", 2024-09-11T09:51:42.088 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:42.088 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:42.088 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:42.089 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:42.089 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.376525+0000", 2024-09-11T09:51:42.089 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:42.089 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:42.089 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:42.089 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:42.089 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.376526+0000", 2024-09-11T09:51:42.089 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.32e-07 2024-09-11T09:51:42.089 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:42.089 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:42.090 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:42.090 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.376606+0000", 2024-09-11T09:51:42.090 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.0539999999999998e-05 2024-09-11T09:51:42.090 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:42.090 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:42.090 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:42.090 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.376615+0000", 2024-09-11T09:51:42.090 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.3310000000000004e-06 2024-09-11T09:51:42.090 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:42.090 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:42.091 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:42.091 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.376619+0000", 2024-09-11T09:51:42.091 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.9600000000000002e-06 2024-09-11T09:51:42.091 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:42.091 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:42.091 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:42.091 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.380083+0000", 2024-09-11T09:51:42.091 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.003464647 2024-09-11T09:51:42.091 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:42.092 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:42.092 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:42.092 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.380132+0000", 2024-09-11T09:51:42.092 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.8507000000000002e-05 2024-09-11T09:51:42.092 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:42.092 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:42.092 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1", 2024-09-11T09:51:42.092 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.380206+0000", 2024-09-11T09:51:42.092 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.4480000000000005e-05 2024-09-11T09:51:42.092 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:42.092 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:42.092 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:42.093 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.382427+0000", 2024-09-11T09:51:42.093 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00222091 2024-09-11T09:51:42.093 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:42.093 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:42.093 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:42.093 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.382443+0000", 2024-09-11T09:51:42.093 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.5586999999999998e-05 2024-09-11T09:51:42.093 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:42.093 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:42.093 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:42.093 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.382495+0000", 2024-09-11T09:51:42.093 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.2023e-05 2024-09-11T09:51:42.093 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:42.094 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:42.094 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:42.094 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.382536+0000", 2024-09-11T09:51:42.094 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.1505000000000003e-05 2024-09-11T09:51:42.094 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:42.094 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:42.094 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:42.094 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:42.094 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:42.094 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.5002.0:36 2.1 2:8654721b:::rbd_data.1250f5e25b80.0000000000000000:head [write 1835008~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-09-11T09:51:42.094 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:38.376629+0000", 2024-09-11T09:51:42.094 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 3.591952203, 2024-09-11T09:51:42.094 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.006032567, 2024-09-11T09:51:42.094 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:42.095 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:42.095 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:42.095 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:42.095 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.5002", 2024-09-11T09:51:42.095 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/2204440692", 2024-09-11T09:51:42.095 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 36 2024-09-11T09:51:42.095 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:42.095 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:42.095 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:42.095 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:42.095 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.376629+0000", 2024-09-11T09:51:42.095 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:42.095 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:42.096 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:42.096 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:42.096 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.376629+0000", 2024-09-11T09:51:42.096 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:42.096 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:42.096 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:42.096 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:42.096 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.376630+0000", 2024-09-11T09:51:42.096 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.8100000000000003e-07 2024-09-11T09:51:42.096 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:42.097 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:42.097 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:42.097 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.376831+0000", 2024-09-11T09:51:42.097 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000201576 2024-09-11T09:51:42.097 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:42.097 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:42.097 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:42.097 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.376843+0000", 2024-09-11T09:51:42.097 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2326e-05 2024-09-11T09:51:42.097 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:42.097 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:42.097 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:42.097 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.376850+0000", 2024-09-11T09:51:42.098 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.0859999999999996e-06 2024-09-11T09:51:42.098 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:42.098 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:42.098 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:42.098 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.380822+0000", 2024-09-11T09:51:42.098 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0039714520000000003 2024-09-11T09:51:42.098 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:42.098 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:42.098 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:42.098 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.380871+0000", 2024-09-11T09:51:42.098 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.9314999999999998e-05 2024-09-11T09:51:42.098 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:42.098 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:42.098 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1", 2024-09-11T09:51:42.099 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.380950+0000", 2024-09-11T09:51:42.099 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.8592999999999998e-05 2024-09-11T09:51:42.099 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:42.099 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:42.099 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:42.099 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.382544+0000", 2024-09-11T09:51:42.099 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.001594631 2024-09-11T09:51:42.099 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:42.100 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:42.100 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:42.100 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.382586+0000", 2024-09-11T09:51:42.100 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.1118999999999997e-05 2024-09-11T09:51:42.100 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:42.100 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:42.100 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:42.100 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.382628+0000", 2024-09-11T09:51:42.100 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.2548e-05 2024-09-11T09:51:42.100 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:42.100 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:42.100 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:42.100 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.382662+0000", 2024-09-11T09:51:42.101 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.3439999999999998e-05 2024-09-11T09:51:42.101 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:42.101 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:42.101 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:42.101 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:42.101 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:42.101 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.5002.0:37 2.1 2:8654721b:::rbd_data.1250f5e25b80.0000000000000000:head [write 1966080~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-09-11T09:51:42.101 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:38.376864+0000", 2024-09-11T09:51:42.102 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 3.5917173390000001, 2024-09-11T09:51:42.102 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0062377370000000001, 2024-09-11T09:51:42.102 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:42.102 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:42.102 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:42.102 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:42.102 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.5002", 2024-09-11T09:51:42.102 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/2204440692", 2024-09-11T09:51:42.102 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 37 2024-09-11T09:51:42.102 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:42.102 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:42.102 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:42.103 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:42.103 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.376864+0000", 2024-09-11T09:51:42.103 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:42.103 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:42.103 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:42.103 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:42.103 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.376864+0000", 2024-09-11T09:51:42.103 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:42.103 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:42.103 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:42.103 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:42.103 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.376865+0000", 2024-09-11T09:51:42.103 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.1600000000000001e-07 2024-09-11T09:51:42.104 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:42.104 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:42.104 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:42.104 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.376976+0000", 2024-09-11T09:51:42.104 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000111581 2024-09-11T09:51:42.104 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:42.104 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:42.104 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:42.104 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.376991+0000", 2024-09-11T09:51:42.104 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.4649e-05 2024-09-11T09:51:42.105 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:42.105 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:42.105 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:42.105 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.376996+0000", 2024-09-11T09:51:42.105 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.4920000000000001e-06 2024-09-11T09:51:42.105 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:42.105 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:42.105 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:42.105 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.381543+0000", 2024-09-11T09:51:42.105 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0045464839999999999 2024-09-11T09:51:42.105 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:42.105 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:42.105 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:42.106 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.381631+0000", 2024-09-11T09:51:42.106 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.865e-05 2024-09-11T09:51:42.106 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:42.106 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:42.106 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1", 2024-09-11T09:51:42.106 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.381777+0000", 2024-09-11T09:51:42.106 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00014590800000000001 2024-09-11T09:51:42.106 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:42.106 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:42.106 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:42.106 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.382671+0000", 2024-09-11T09:51:42.106 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00089374400000000003 2024-09-11T09:51:42.106 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:42.106 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:42.107 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:42.107 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.383050+0000", 2024-09-11T09:51:42.107 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00037862600000000002 2024-09-11T09:51:42.107 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:42.107 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:42.107 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:42.107 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.383079+0000", 2024-09-11T09:51:42.108 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.9541999999999999e-05 2024-09-11T09:51:42.108 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:42.108 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:42.108 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:42.108 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.383102+0000", 2024-09-11T09:51:42.108 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.2345e-05 2024-09-11T09:51:42.108 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:42.108 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:42.109 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:42.109 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:42.109 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:42.109 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:51:42.111 INFO:tasks.workunit.client.0.smithi107.stderr:+ return 0 2024-09-11T09:51:42.111 INFO:tasks.workunit.client.0.smithi107.stderr:+ ps -C rbd-nbd 2024-09-11T09:51:42.112 INFO:tasks.workunit.client.0.smithi107.stderr:+ expect_false grep '^ *28538\b' 2024-09-11T09:51:42.112 INFO:tasks.workunit.client.0.smithi107.stderr:+ grep '^ *28538\b' 2024-09-11T09:51:42.112 INFO:tasks.workunit.client.0.smithi107.stderr:+ return 0 2024-09-11T09:51:42.112 INFO:tasks.workunit.client.0.smithi107.stderr:+ sleep 0.5 2024-09-11T09:51:42.112 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-11T09:51:42.113 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-09-11T09:51:42.111+0000 7f7b6c7d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31904) UID: 0 2024-09-11T09:51:42.211 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-09-11T09:51:42.211+0000 7f7b6c7d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31904) UID: 0 2024-09-11T09:51:42.229 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-11T09:51:42.229 INFO:teuthology.orchestra.run.smithi096.stdout: "ops": [], 2024-09-11T09:51:42.229 INFO:teuthology.orchestra.run.smithi096.stdout: "num_ops": 0 2024-09-11T09:51:42.230 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-11T09:51:42.239 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-11T09:51:42.312 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:51:42.312+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:51:42.381 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-11T09:51:42.381 INFO:teuthology.orchestra.run.smithi096.stdout: "ops": [], 2024-09-11T09:51:42.381 INFO:teuthology.orchestra.run.smithi096.stdout: "complaint_time": 30, 2024-09-11T09:51:42.381 INFO:teuthology.orchestra.run.smithi096.stdout: "num_blocked_ops": 0 2024-09-11T09:51:42.382 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-11T09:51:42.390 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-11T09:51:42.434 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:51:42.434+0000 7f3c96ec9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31909) UID: 0 2024-09-11T09:51:42.484 INFO:tasks.workunit.client.0.smithi107.stderr:+ DEV= 2024-09-11T09:51:42.484 INFO:tasks.workunit.client.0.smithi107.stderr:+ rbd snap create rbd/ns/testrbdnbd27121@snap 2024-09-11T09:51:42.502 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-11T09:51:42.502 INFO:teuthology.orchestra.run.smithi096.stdout: "size": 20, 2024-09-11T09:51:42.502 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 600, 2024-09-11T09:51:42.502 INFO:teuthology.orchestra.run.smithi096.stdout: "ops": [ 2024-09-11T09:51:42.503 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.503 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:111 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:51:42.503 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.711668+0000", 2024-09-11T09:51:42.503 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 90.790460314000001, 2024-09-11T09:51:42.503 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0077145570000000004, 2024-09-11T09:51:42.503 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:42.503 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:42.503 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:42.503 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:42.503 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:51:42.503 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:51:42.503 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 111 2024-09-11T09:51:42.503 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.504 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:42.504 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.504 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:42.504 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711668+0000", 2024-09-11T09:51:42.504 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:42.504 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.504 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.504 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:42.504 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711668+0000", 2024-09-11T09:51:42.504 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:42.504 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.504 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.504 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:42.504 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711669+0000", 2024-09-11T09:51:42.505 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.99e-07 2024-09-11T09:51:42.505 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.505 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.505 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:42.505 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711673+0000", 2024-09-11T09:51:42.505 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.7440000000000001e-06 2024-09-11T09:51:42.505 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.505 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.505 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:42.505 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711686+0000", 2024-09-11T09:51:42.505 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.3317e-05 2024-09-11T09:51:42.505 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.505 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.506 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:42.506 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711697+0000", 2024-09-11T09:51:42.506 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1022000000000001e-05 2024-09-11T09:51:42.506 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.506 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.506 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:42.506 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.717065+0000", 2024-09-11T09:51:42.506 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.005367531 2024-09-11T09:51:42.506 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.506 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.506 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:42.506 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.717114+0000", 2024-09-11T09:51:42.506 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.9707999999999999e-05 2024-09-11T09:51:42.506 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.507 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.507 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:42.507 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.717190+0000", 2024-09-11T09:51:42.507 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.5394000000000004e-05 2024-09-11T09:51:42.507 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.507 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.507 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:42.507 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718793+0000", 2024-09-11T09:51:42.507 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001603653 2024-09-11T09:51:42.507 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.507 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.507 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:42.507 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719329+0000", 2024-09-11T09:51:42.508 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00053543799999999995 2024-09-11T09:51:42.508 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.508 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.508 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:42.508 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719359+0000", 2024-09-11T09:51:42.508 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.0420999999999999e-05 2024-09-11T09:51:42.508 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.508 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.508 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:42.508 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719383+0000", 2024-09-11T09:51:42.508 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.3730000000000001e-05 2024-09-11T09:51:42.508 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:42.508 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:42.509 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:42.509 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.509 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.509 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:112 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:51:42.509 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.711712+0000", 2024-09-11T09:51:42.509 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 90.790416754000006, 2024-09-11T09:51:42.509 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.008162084, 2024-09-11T09:51:42.509 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:42.509 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:42.509 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:42.509 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:42.509 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:51:42.509 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:51:42.509 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 112 2024-09-11T09:51:42.510 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.510 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:42.510 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.510 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:42.510 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711712+0000", 2024-09-11T09:51:42.510 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:42.510 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.510 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.510 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:42.510 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711712+0000", 2024-09-11T09:51:42.510 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:42.510 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.510 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.510 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:42.511 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711712+0000", 2024-09-11T09:51:42.511 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.6000000000000004e-07 2024-09-11T09:51:42.511 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.511 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.511 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:42.511 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711808+0000", 2024-09-11T09:51:42.511 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.5977999999999995e-05 2024-09-11T09:51:42.511 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.511 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.511 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:42.511 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711819+0000", 2024-09-11T09:51:42.511 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.0755e-05 2024-09-11T09:51:42.511 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.512 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.512 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:42.512 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711826+0000", 2024-09-11T09:51:42.512 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.7970000000000003e-06 2024-09-11T09:51:42.512 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.512 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.512 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:42.512 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.717435+0000", 2024-09-11T09:51:42.512 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0056090239999999998 2024-09-11T09:51:42.512 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.512 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.512 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:42.513 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.717496+0000", 2024-09-11T09:51:42.513 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.0543e-05 2024-09-11T09:51:42.513 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.513 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.513 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:42.513 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.717591+0000", 2024-09-11T09:51:42.513 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.5667000000000004e-05 2024-09-11T09:51:42.513 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.513 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.513 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:42.513 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718831+0000", 2024-09-11T09:51:42.514 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001240204 2024-09-11T09:51:42.514 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.514 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.514 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:42.514 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719819+0000", 2024-09-11T09:51:42.514 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00098751500000000001 2024-09-11T09:51:42.514 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.514 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.514 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:42.514 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719850+0000", 2024-09-11T09:51:42.514 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.1535999999999998e-05 2024-09-11T09:51:42.514 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.515 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.515 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:42.515 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719874+0000", 2024-09-11T09:51:42.515 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.3504999999999999e-05 2024-09-11T09:51:42.515 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:42.515 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:42.515 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:42.515 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.515 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.515 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:113 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:51:42.515 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.711842+0000", 2024-09-11T09:51:42.516 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 90.790286254999998, 2024-09-11T09:51:42.516 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0081160099999999999, 2024-09-11T09:51:42.516 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:42.516 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:42.516 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:42.516 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:42.516 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:51:42.516 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:51:42.516 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 113 2024-09-11T09:51:42.516 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.517 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:42.517 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.517 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:42.517 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711842+0000", 2024-09-11T09:51:42.517 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:42.517 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.517 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.517 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:42.517 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711842+0000", 2024-09-11T09:51:42.517 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:42.518 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.518 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.518 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:42.518 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711843+0000", 2024-09-11T09:51:42.518 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.4399999999999999e-07 2024-09-11T09:51:42.518 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.518 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.518 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:42.518 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711848+0000", 2024-09-11T09:51:42.518 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.4159999999999997e-06 2024-09-11T09:51:42.518 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.518 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.519 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:42.519 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711856+0000", 2024-09-11T09:51:42.519 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.7870000000000007e-06 2024-09-11T09:51:42.519 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.519 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.519 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:42.519 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711862+0000", 2024-09-11T09:51:42.519 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.3689999999999998e-06 2024-09-11T09:51:42.519 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.519 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.519 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:42.519 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.717965+0000", 2024-09-11T09:51:42.519 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0061028899999999997 2024-09-11T09:51:42.519 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.520 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.520 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:42.520 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718023+0000", 2024-09-11T09:51:42.520 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.8313000000000002e-05 2024-09-11T09:51:42.520 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.520 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.520 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:42.521 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718109+0000", 2024-09-11T09:51:42.521 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.6205000000000006e-05 2024-09-11T09:51:42.521 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.521 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.521 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:42.521 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719808+0000", 2024-09-11T09:51:42.521 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0016990060000000001 2024-09-11T09:51:42.522 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.522 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.522 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:42.522 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719883+0000", 2024-09-11T09:51:42.522 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.4624999999999998e-05 2024-09-11T09:51:42.522 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.523 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.523 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:42.523 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719910+0000", 2024-09-11T09:51:42.523 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.7512e-05 2024-09-11T09:51:42.523 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.524 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.524 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:42.524 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719958+0000", 2024-09-11T09:51:42.524 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.8143000000000003e-05 2024-09-11T09:51:42.524 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:42.524 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:42.525 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:42.525 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.525 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.525 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:51:42.525 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.711887+0000", 2024-09-11T09:51:42.525 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 90.790241640999994, 2024-09-11T09:51:42.526 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0086063589999999992, 2024-09-11T09:51:42.526 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:42.526 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:42.526 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:42.527 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:42.527 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:51:42.527 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:51:42.527 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 114 2024-09-11T09:51:42.527 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.527 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:42.528 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.528 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:42.528 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711887+0000", 2024-09-11T09:51:42.528 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:42.528 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.528 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.529 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:42.529 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711887+0000", 2024-09-11T09:51:42.529 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:42.529 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.529 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.530 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:42.530 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711887+0000", 2024-09-11T09:51:42.530 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.1500000000000005e-07 2024-09-11T09:51:42.530 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.530 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.530 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:42.531 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711897+0000", 2024-09-11T09:51:42.531 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.1900000000000001e-06 2024-09-11T09:51:42.531 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.531 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.531 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:42.531 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711902+0000", 2024-09-11T09:51:42.532 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.2349999999999999e-06 2024-09-11T09:51:42.532 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.532 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.532 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:42.532 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711906+0000", 2024-09-11T09:51:42.532 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.7129999999999999e-06 2024-09-11T09:51:42.533 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.533 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.533 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:42.533 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718429+0000", 2024-09-11T09:51:42.533 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0065231960000000002 2024-09-11T09:51:42.533 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.534 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.534 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:42.534 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718480+0000", 2024-09-11T09:51:42.534 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.1161e-05 2024-09-11T09:51:42.535 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.535 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.535 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:42.535 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718558+0000", 2024-09-11T09:51:42.535 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.7519000000000001e-05 2024-09-11T09:51:42.535 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.536 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.536 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:42.536 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720038+0000", 2024-09-11T09:51:42.536 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001480016 2024-09-11T09:51:42.536 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.536 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.537 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:42.537 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720417+0000", 2024-09-11T09:51:42.537 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00037943699999999999 2024-09-11T09:51:42.537 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.537 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.538 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:42.538 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720452+0000", 2024-09-11T09:51:42.538 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.4595000000000003e-05 2024-09-11T09:51:42.538 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.538 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.538 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:42.539 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720493+0000", 2024-09-11T09:51:42.539 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.1782000000000001e-05 2024-09-11T09:51:42.539 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:42.539 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:42.539 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:42.540 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.540 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.540 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:51:42.540 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.711912+0000", 2024-09-11T09:51:42.540 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 90.790216971999996, 2024-09-11T09:51:42.540 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0086864750000000008, 2024-09-11T09:51:42.541 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:42.541 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:42.541 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:42.541 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:42.541 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:51:42.541 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:51:42.542 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 115 2024-09-11T09:51:42.542 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.542 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:42.542 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.542 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:42.542 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711912+0000", 2024-09-11T09:51:42.543 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:42.543 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.543 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.543 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:42.543 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711912+0000", 2024-09-11T09:51:42.543 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:42.544 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.544 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.544 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:42.544 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711912+0000", 2024-09-11T09:51:42.544 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.3900000000000001e-07 2024-09-11T09:51:42.544 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.545 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.545 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:42.545 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711969+0000", 2024-09-11T09:51:42.545 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.7501999999999999e-05 2024-09-11T09:51:42.545 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.545 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.545 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:42.546 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711986+0000", 2024-09-11T09:51:42.546 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.6402999999999999e-05 2024-09-11T09:51:42.546 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.546 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.546 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:42.546 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711990+0000", 2024-09-11T09:51:42.546 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.929e-06 2024-09-11T09:51:42.547 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.547 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.547 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:42.547 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718861+0000", 2024-09-11T09:51:42.547 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0068710530000000002 2024-09-11T09:51:42.547 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.547 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.548 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:42.548 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718908+0000", 2024-09-11T09:51:42.548 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.7431e-05 2024-09-11T09:51:42.548 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.548 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.548 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:42.549 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718984+0000", 2024-09-11T09:51:42.549 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.5500999999999994e-05 2024-09-11T09:51:42.549 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.549 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.549 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:42.549 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720500+0000", 2024-09-11T09:51:42.549 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0015158820000000001 2024-09-11T09:51:42.549 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.550 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.550 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:42.550 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720536+0000", 2024-09-11T09:51:42.550 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.6118000000000002e-05 2024-09-11T09:51:42.550 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.550 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.550 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:42.551 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720570+0000", 2024-09-11T09:51:42.551 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.4387999999999997e-05 2024-09-11T09:51:42.551 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.551 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.551 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:42.551 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720598+0000", 2024-09-11T09:51:42.552 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.8028999999999999e-05 2024-09-11T09:51:42.552 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:42.552 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:42.552 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:42.552 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.552 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.552 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:51:42.553 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.711998+0000", 2024-09-11T09:51:42.553 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 90.790130793000003, 2024-09-11T09:51:42.553 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.010785813999999999, 2024-09-11T09:51:42.553 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:42.553 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:42.553 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:42.554 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:42.554 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:51:42.554 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:51:42.554 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 116 2024-09-11T09:51:42.554 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.554 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:42.554 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.555 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:42.555 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711998+0000", 2024-09-11T09:51:42.555 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:42.555 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.555 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.555 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:42.555 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711998+0000", 2024-09-11T09:51:42.556 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:42.556 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.556 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.556 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:42.556 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711998+0000", 2024-09-11T09:51:42.556 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.3300000000000003e-07 2024-09-11T09:51:42.557 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.557 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.557 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:42.557 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712001+0000", 2024-09-11T09:51:42.557 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.9349999999999999e-06 2024-09-11T09:51:42.557 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.557 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.557 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:42.557 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712006+0000", 2024-09-11T09:51:42.557 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.1630000000000004e-06 2024-09-11T09:51:42.557 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.557 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.557 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:42.558 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712010+0000", 2024-09-11T09:51:42.558 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.574e-06 2024-09-11T09:51:42.558 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.558 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.558 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:42.558 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719415+0000", 2024-09-11T09:51:42.558 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0074049099999999998 2024-09-11T09:51:42.558 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.558 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.558 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:42.558 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719465+0000", 2024-09-11T09:51:42.558 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.0129000000000001e-05 2024-09-11T09:51:42.558 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.558 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.559 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:42.559 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719545+0000", 2024-09-11T09:51:42.559 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.0253999999999994e-05 2024-09-11T09:51:42.559 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.559 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.559 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:42.559 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720636+0000", 2024-09-11T09:51:42.559 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0010909330000000001 2024-09-11T09:51:42.559 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.559 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.559 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:42.559 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.722707+0000", 2024-09-11T09:51:42.559 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0020713300000000001 2024-09-11T09:51:42.559 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.560 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.560 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:42.560 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.722743+0000", 2024-09-11T09:51:42.560 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.5351000000000001e-05 2024-09-11T09:51:42.560 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.560 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.560 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:42.560 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.722784+0000", 2024-09-11T09:51:42.560 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.0802e-05 2024-09-11T09:51:42.560 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:42.560 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:42.560 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:42.560 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.561 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.561 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:51:42.561 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.712014+0000", 2024-09-11T09:51:42.561 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 90.790114752999997, 2024-09-11T09:51:42.561 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.013420938, 2024-09-11T09:51:42.561 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:42.561 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:42.561 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:42.561 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:42.561 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:51:42.561 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:51:42.561 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 117 2024-09-11T09:51:42.561 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.562 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:42.562 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.562 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:42.562 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712014+0000", 2024-09-11T09:51:42.562 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:42.562 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.562 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.562 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:42.562 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712014+0000", 2024-09-11T09:51:42.562 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:42.562 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.562 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.562 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:42.562 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712014+0000", 2024-09-11T09:51:42.563 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.2000000000000001e-07 2024-09-11T09:51:42.563 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.563 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.563 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:42.563 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712016+0000", 2024-09-11T09:51:42.563 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.2309999999999999e-06 2024-09-11T09:51:42.563 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.563 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.563 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:42.563 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712020+0000", 2024-09-11T09:51:42.563 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.3830000000000001e-06 2024-09-11T09:51:42.563 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.563 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.563 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:42.564 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712023+0000", 2024-09-11T09:51:42.564 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.3340000000000002e-06 2024-09-11T09:51:42.564 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.564 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.564 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:42.564 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720083+0000", 2024-09-11T09:51:42.564 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0080601189999999993 2024-09-11T09:51:42.564 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.564 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.564 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:42.564 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720125+0000", 2024-09-11T09:51:42.564 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.2029e-05 2024-09-11T09:51:42.564 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.565 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.565 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:42.565 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720205+0000", 2024-09-11T09:51:42.565 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.0010000000000001e-05 2024-09-11T09:51:42.565 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.565 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.565 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:42.565 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720849+0000", 2024-09-11T09:51:42.565 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00064369899999999996 2024-09-11T09:51:42.565 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.565 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.565 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:42.565 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.725357+0000", 2024-09-11T09:51:42.565 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0045077379999999998 2024-09-11T09:51:42.565 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.566 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.566 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:42.566 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.725392+0000", 2024-09-11T09:51:42.566 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.5862999999999999e-05 2024-09-11T09:51:42.566 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.566 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.566 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:42.566 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.725435+0000", 2024-09-11T09:51:42.566 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.2311999999999997e-05 2024-09-11T09:51:42.566 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:42.566 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:42.566 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:42.566 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.567 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.567 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:159 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:51:42.567 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:12.452099+0000", 2024-09-11T09:51:42.567 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 90.050029164999998, 2024-09-11T09:51:42.567 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0069885950000000002, 2024-09-11T09:51:42.567 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:42.567 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:42.567 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:42.567 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:42.567 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:51:42.567 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:51:42.567 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 159 2024-09-11T09:51:42.567 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.567 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:42.568 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.568 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:42.568 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452099+0000", 2024-09-11T09:51:42.568 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:42.568 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.568 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.568 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:42.568 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452099+0000", 2024-09-11T09:51:42.568 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:42.568 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.568 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.568 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:42.568 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452100+0000", 2024-09-11T09:51:42.569 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.2800000000000003e-07 2024-09-11T09:51:42.569 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.569 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.569 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:42.569 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452102+0000", 2024-09-11T09:51:42.569 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.824e-06 2024-09-11T09:51:42.569 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.569 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.569 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:42.569 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452105+0000", 2024-09-11T09:51:42.569 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.0460000000000001e-06 2024-09-11T09:51:42.569 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.569 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.569 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:42.570 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452107+0000", 2024-09-11T09:51:42.570 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.543e-06 2024-09-11T09:51:42.570 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.570 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.570 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:42.570 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.455484+0000", 2024-09-11T09:51:42.570 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0033769759999999998 2024-09-11T09:51:42.570 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.570 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.570 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:42.570 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.455527+0000", 2024-09-11T09:51:42.570 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.2589999999999997e-05 2024-09-11T09:51:42.570 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.570 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.571 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:42.571 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.455594+0000", 2024-09-11T09:51:42.571 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.6775999999999999e-05 2024-09-11T09:51:42.571 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.571 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.571 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:42.571 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.456312+0000", 2024-09-11T09:51:42.571 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00071874700000000005 2024-09-11T09:51:42.571 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.571 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.571 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:42.571 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.459015+0000", 2024-09-11T09:51:42.571 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0027028540000000002 2024-09-11T09:51:42.571 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.572 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.572 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:42.572 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.459054+0000", 2024-09-11T09:51:42.572 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.8840000000000001e-05 2024-09-11T09:51:42.572 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.572 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.572 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:42.572 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.459088+0000", 2024-09-11T09:51:42.572 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.4071e-05 2024-09-11T09:51:42.572 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:42.572 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:42.572 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:42.572 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.573 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.573 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4688.0:38 2.5 2:abde2694:::rbd_data.1250f5e25b80.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:42.573 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:22.903926+0000", 2024-09-11T09:51:42.573 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 19.598202488999998, 2024-09-11T09:51:42.573 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.028303676, 2024-09-11T09:51:42.573 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:42.573 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:42.573 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:42.573 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:42.573 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4688", 2024-09-11T09:51:42.573 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:51:42.573 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 38 2024-09-11T09:51:42.573 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.573 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:42.574 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.574 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:42.574 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.903926+0000", 2024-09-11T09:51:42.574 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:42.574 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.574 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.574 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:42.574 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.903926+0000", 2024-09-11T09:51:42.574 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:42.574 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.574 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.574 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:42.574 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.903928+0000", 2024-09-11T09:51:42.575 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.4249999999999999e-06 2024-09-11T09:51:42.575 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.575 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.575 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:42.575 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.914774+0000", 2024-09-11T09:51:42.575 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.010846613 2024-09-11T09:51:42.575 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.575 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.575 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:42.575 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.914793+0000", 2024-09-11T09:51:42.575 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.8879999999999999e-05 2024-09-11T09:51:42.575 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.575 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.575 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:42.576 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.914801+0000", 2024-09-11T09:51:42.576 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.2230000000000007e-06 2024-09-11T09:51:42.576 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.576 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.576 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:42.576 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.914854+0000", 2024-09-11T09:51:42.576 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.2883999999999997e-05 2024-09-11T09:51:42.576 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.576 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.576 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:42.576 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.914951+0000", 2024-09-11T09:51:42.576 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.6731000000000006e-05 2024-09-11T09:51:42.576 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.576 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.577 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:42.577 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.915080+0000", 2024-09-11T09:51:42.577 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00012926999999999999 2024-09-11T09:51:42.577 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.577 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.577 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:42.577 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.931469+0000", 2024-09-11T09:51:42.577 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.016389107 2024-09-11T09:51:42.577 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.577 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.577 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:42.577 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.932118+0000", 2024-09-11T09:51:42.577 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00064915600000000002 2024-09-11T09:51:42.577 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.578 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.578 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:42.578 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.932177+0000", 2024-09-11T09:51:42.578 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.9032e-05 2024-09-11T09:51:42.578 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.578 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.578 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:42.578 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.932230+0000", 2024-09-11T09:51:42.578 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.2355000000000003e-05 2024-09-11T09:51:42.578 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:42.578 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:42.578 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:42.578 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.578 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.579 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_repop(client.4688.0:36 2.6 e53/15 2:61e1b442:::rbd_data.1250f5e25b80.0000000000000003:head v 53'2, mlcod=0'0)", 2024-09-11T09:51:42.579 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:22.915597+0000", 2024-09-11T09:51:42.579 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 19.586531939, 2024-09-11T09:51:42.579 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.020430798, 2024-09-11T09:51:42.579 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:42.579 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:42.579 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:42.579 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:42.579 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.579 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:42.579 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.915597+0000", 2024-09-11T09:51:42.579 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:42.579 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.579 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.580 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:42.580 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.915597+0000", 2024-09-11T09:51:42.580 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:42.580 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.580 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.580 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:42.580 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.915598+0000", 2024-09-11T09:51:42.580 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.7150000000000001e-06 2024-09-11T09:51:42.580 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.580 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.580 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:42.580 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921741+0000", 2024-09-11T09:51:42.580 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0061424339999999996 2024-09-11T09:51:42.581 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.581 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.581 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:42.581 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921757+0000", 2024-09-11T09:51:42.581 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.6144e-05 2024-09-11T09:51:42.581 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.581 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.581 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:42.581 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921765+0000", 2024-09-11T09:51:42.581 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.0809999999999994e-06 2024-09-11T09:51:42.581 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.581 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.581 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:42.581 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921826+0000", 2024-09-11T09:51:42.582 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.0878999999999998e-05 2024-09-11T09:51:42.582 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.582 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.582 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:42.582 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921849+0000", 2024-09-11T09:51:42.582 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.2966999999999999e-05 2024-09-11T09:51:42.582 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.582 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.582 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:42.582 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.935968+0000", 2024-09-11T09:51:42.582 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.014119191 2024-09-11T09:51:42.582 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.582 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.583 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:42.583 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.936027+0000", 2024-09-11T09:51:42.583 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.9386999999999999e-05 2024-09-11T09:51:42.583 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:42.583 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:42.583 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:42.583 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.583 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.583 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_repop(client.4688.0:39 2.6 e53/15 2:743e7616:::rbd_data.1250f5e25b80.0000000000000005:head v 53'3, mlcod=0'0)", 2024-09-11T09:51:42.583 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:22.921806+0000", 2024-09-11T09:51:42.583 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 19.580322889000001, 2024-09-11T09:51:42.583 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.015275018, 2024-09-11T09:51:42.583 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:42.583 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:42.584 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:42.584 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:42.584 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.584 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:42.584 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921806+0000", 2024-09-11T09:51:42.584 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:42.584 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.584 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.584 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:42.584 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921806+0000", 2024-09-11T09:51:42.584 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:42.584 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.584 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.585 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:42.585 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921806+0000", 2024-09-11T09:51:42.585 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.2200000000000003e-07 2024-09-11T09:51:42.585 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.585 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.585 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:42.585 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.925590+0000", 2024-09-11T09:51:42.585 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0037838949999999998 2024-09-11T09:51:42.585 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.585 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.585 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:42.585 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.925605+0000", 2024-09-11T09:51:42.585 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.4321999999999999e-05 2024-09-11T09:51:42.586 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.586 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.586 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:42.586 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.925613+0000", 2024-09-11T09:51:42.586 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.1980000000000001e-06 2024-09-11T09:51:42.586 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.586 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.586 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:42.586 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.925857+0000", 2024-09-11T09:51:42.586 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000244064 2024-09-11T09:51:42.586 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.586 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.587 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:42.587 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.925875+0000", 2024-09-11T09:51:42.587 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.838e-05 2024-09-11T09:51:42.587 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.587 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.587 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:42.587 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.937042+0000", 2024-09-11T09:51:42.587 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.011166416 2024-09-11T09:51:42.587 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.587 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.587 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:42.587 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.937081+0000", 2024-09-11T09:51:42.587 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.9020999999999998e-05 2024-09-11T09:51:42.587 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:42.588 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:42.588 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:42.588 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.588 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.588 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4688.0:49 2.5 2:b6dd9fc9:::rbd_data.1250f5e25b80.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:42.588 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:22.938548+0000", 2024-09-11T09:51:42.588 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 19.563580886, 2024-09-11T09:51:42.588 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.022766950000000001, 2024-09-11T09:51:42.588 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:42.588 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:42.588 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:42.588 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:42.588 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4688", 2024-09-11T09:51:42.588 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:51:42.589 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 49 2024-09-11T09:51:42.589 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.589 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:42.589 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.589 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:42.589 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.938548+0000", 2024-09-11T09:51:42.589 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:42.589 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.589 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.589 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:42.589 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.938548+0000", 2024-09-11T09:51:42.592 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:42.593 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.593 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.593 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:42.593 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.938550+0000", 2024-09-11T09:51:42.593 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.7710000000000001e-06 2024-09-11T09:51:42.593 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.593 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.593 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:42.593 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.944237+0000", 2024-09-11T09:51:42.594 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0056879790000000001 2024-09-11T09:51:42.594 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.594 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.594 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:42.594 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.944262+0000", 2024-09-11T09:51:42.594 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.4247e-05 2024-09-11T09:51:42.594 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.594 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.594 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:42.594 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.944274+0000", 2024-09-11T09:51:42.594 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.2639999999999999e-05 2024-09-11T09:51:42.594 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.594 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.594 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:42.595 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.944344+0000", 2024-09-11T09:51:42.595 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.9574999999999997e-05 2024-09-11T09:51:42.595 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.595 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.595 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:42.595 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.944474+0000", 2024-09-11T09:51:42.595 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000130464 2024-09-11T09:51:42.595 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.595 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.595 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:42.595 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.944585+0000", 2024-09-11T09:51:42.595 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000110096 2024-09-11T09:51:42.595 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.595 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.596 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:42.596 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.952724+0000", 2024-09-11T09:51:42.596 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0081398900000000003 2024-09-11T09:51:42.596 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.596 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.596 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:42.596 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.961219+0000", 2024-09-11T09:51:42.596 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0084949789999999997 2024-09-11T09:51:42.596 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.596 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.596 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:42.596 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.961280+0000", 2024-09-11T09:51:42.596 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.0404e-05 2024-09-11T09:51:42.597 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.597 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.597 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:42.597 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.961315+0000", 2024-09-11T09:51:42.597 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.4904999999999999e-05 2024-09-11T09:51:42.597 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:42.597 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:42.597 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:42.597 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.597 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.597 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_repop(client.4688.0:44 2.6 e53/15 2:67246a22:::rbd_data.1250f5e25b80.0000000000000007:head v 53'4, mlcod=53'2)", 2024-09-11T09:51:42.597 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:22.951721+0000", 2024-09-11T09:51:42.597 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 19.550407614000001, 2024-09-11T09:51:42.597 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.01080981, 2024-09-11T09:51:42.598 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:42.598 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:42.598 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:42.598 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:42.598 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.598 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:42.598 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.951721+0000", 2024-09-11T09:51:42.598 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:42.598 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.598 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.598 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:42.598 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.951721+0000", 2024-09-11T09:51:42.598 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:42.599 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.599 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.599 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:42.599 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.951722+0000", 2024-09-11T09:51:42.599 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.0279999999999999e-06 2024-09-11T09:51:42.599 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.599 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.599 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:42.599 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.955612+0000", 2024-09-11T09:51:42.599 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0038896920000000001 2024-09-11T09:51:42.599 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.599 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.599 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:42.600 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.955628+0000", 2024-09-11T09:51:42.600 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.5945000000000001e-05 2024-09-11T09:51:42.600 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.600 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.600 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:42.600 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.955635+0000", 2024-09-11T09:51:42.600 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.8080000000000005e-06 2024-09-11T09:51:42.600 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.600 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.600 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:42.600 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.955700+0000", 2024-09-11T09:51:42.600 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.4114000000000003e-05 2024-09-11T09:51:42.600 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.601 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.601 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:42.601 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.955724+0000", 2024-09-11T09:51:42.601 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.3989999999999999e-05 2024-09-11T09:51:42.601 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.601 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.601 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:42.601 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.962490+0000", 2024-09-11T09:51:42.601 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0067662599999999996 2024-09-11T09:51:42.601 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.601 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.601 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:42.601 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.962531+0000", 2024-09-11T09:51:42.601 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.0973000000000002e-05 2024-09-11T09:51:42.602 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:42.602 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:42.602 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:42.602 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.602 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.602 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_repop(client.4688.0:51 2.6 e53/15 2:6b5894fc:::rbd_data.1250f5e25b80.000000000000000a:head v 53'5, mlcod=53'2)", 2024-09-11T09:51:42.602 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:22.959030+0000", 2024-09-11T09:51:42.602 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 19.543098440000001, 2024-09-11T09:51:42.602 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.011503638, 2024-09-11T09:51:42.602 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:42.602 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:42.602 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:42.602 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:42.602 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.603 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:42.603 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.959030+0000", 2024-09-11T09:51:42.603 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:42.603 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.603 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.603 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:42.603 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.959030+0000", 2024-09-11T09:51:42.603 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:42.603 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.603 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.603 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:42.603 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.959032+0000", 2024-09-11T09:51:42.603 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.356e-06 2024-09-11T09:51:42.603 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.604 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.604 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:42.604 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.962899+0000", 2024-09-11T09:51:42.604 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.003867639 2024-09-11T09:51:42.604 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.604 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.604 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:42.604 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.962909+0000", 2024-09-11T09:51:42.604 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.0105999999999999e-05 2024-09-11T09:51:42.604 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.604 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.604 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:42.604 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.962915+0000", 2024-09-11T09:51:42.605 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.8869999999999997e-06 2024-09-11T09:51:42.605 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.605 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.605 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:42.605 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.962962+0000", 2024-09-11T09:51:42.605 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.6511999999999998e-05 2024-09-11T09:51:42.605 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.605 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.605 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:42.605 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.962989+0000", 2024-09-11T09:51:42.605 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.7702999999999999e-05 2024-09-11T09:51:42.605 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.605 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.606 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:42.606 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.970495+0000", 2024-09-11T09:51:42.606 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0075052449999999998 2024-09-11T09:51:42.606 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.606 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.606 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:42.606 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.970534+0000", 2024-09-11T09:51:42.606 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.9190000000000003e-05 2024-09-11T09:51:42.606 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:42.606 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:42.606 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:42.606 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.606 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.606 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_repop(client.4688.0:58 2.6 e53/15 2:6277c644:::rbd_data.1250f5e25b80.000000000000000e:head v 53'6, mlcod=53'4)", 2024-09-11T09:51:42.607 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:22.996199+0000", 2024-09-11T09:51:42.607 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 19.505929519999999, 2024-09-11T09:51:42.607 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.010145573999999999, 2024-09-11T09:51:42.607 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:42.607 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:42.607 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:42.607 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:42.607 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.607 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:42.607 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.996199+0000", 2024-09-11T09:51:42.607 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:42.607 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.607 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.608 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:42.608 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.996199+0000", 2024-09-11T09:51:42.608 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:42.608 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.608 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.608 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:42.608 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.996200+0000", 2024-09-11T09:51:42.608 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.85e-07 2024-09-11T09:51:42.608 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.608 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.608 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:42.608 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.999914+0000", 2024-09-11T09:51:42.608 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.003714406 2024-09-11T09:51:42.608 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.609 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.609 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:42.609 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.999928+0000", 2024-09-11T09:51:42.609 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.3149000000000001e-05 2024-09-11T09:51:42.609 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.609 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.609 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:42.609 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.999951+0000", 2024-09-11T09:51:42.609 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.3116e-05 2024-09-11T09:51:42.609 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.609 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.609 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:42.609 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.000003+0000", 2024-09-11T09:51:42.609 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.2169000000000001e-05 2024-09-11T09:51:42.610 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.610 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.610 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:42.610 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.000018+0000", 2024-09-11T09:51:42.610 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.5500000000000001e-05 2024-09-11T09:51:42.610 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.610 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.610 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:42.610 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.006303+0000", 2024-09-11T09:51:42.610 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0062846710000000004 2024-09-11T09:51:42.610 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.610 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.610 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:42.610 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.006345+0000", 2024-09-11T09:51:42.611 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.1678000000000003e-05 2024-09-11T09:51:42.611 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:42.611 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:42.611 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:42.611 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.611 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.611 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4694.0:42 2.5 2:a4fc00ec:ns::rbd_data.1256a0c02879.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:42.611 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.111570+0000", 2024-09-11T09:51:42.611 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 19.390558414000001, 2024-09-11T09:51:42.611 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.038002197000000001, 2024-09-11T09:51:42.611 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:42.611 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:42.611 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:42.611 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:42.612 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4694", 2024-09-11T09:51:42.612 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:51:42.612 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 42 2024-09-11T09:51:42.612 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.612 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:42.612 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.612 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:42.612 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.111570+0000", 2024-09-11T09:51:42.612 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:42.612 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.612 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.612 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:42.612 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.111570+0000", 2024-09-11T09:51:42.613 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:42.613 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.613 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.613 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:42.613 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.111572+0000", 2024-09-11T09:51:42.613 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.3909999999999999e-06 2024-09-11T09:51:42.613 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.613 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.613 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:42.613 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.135929+0000", 2024-09-11T09:51:42.613 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.024357410999999999 2024-09-11T09:51:42.613 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.613 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.614 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:42.614 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.135951+0000", 2024-09-11T09:51:42.614 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.1775000000000001e-05 2024-09-11T09:51:42.614 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.614 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.614 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:42.614 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.135962+0000", 2024-09-11T09:51:42.614 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.0903e-05 2024-09-11T09:51:42.614 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.614 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.614 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:42.614 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.136045+0000", 2024-09-11T09:51:42.614 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.3128999999999996e-05 2024-09-11T09:51:42.614 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.615 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.615 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:42.615 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.136229+0000", 2024-09-11T09:51:42.615 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00018393700000000001 2024-09-11T09:51:42.615 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.615 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.615 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:42.615 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.136394+0000", 2024-09-11T09:51:42.615 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000165527 2024-09-11T09:51:42.615 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.615 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.615 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:42.615 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.149320+0000", 2024-09-11T09:51:42.615 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.01292568 2024-09-11T09:51:42.616 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.616 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.616 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:42.616 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.149493+0000", 2024-09-11T09:51:42.616 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00017285999999999999 2024-09-11T09:51:42.616 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.616 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.616 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:42.616 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.149539+0000", 2024-09-11T09:51:42.616 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.6628000000000001e-05 2024-09-11T09:51:42.616 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.616 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.616 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:42.616 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.149572+0000", 2024-09-11T09:51:42.617 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.2956000000000001e-05 2024-09-11T09:51:42.617 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:42.617 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:42.617 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:42.617 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.617 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.617 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_repop(client.4694.0:45 2.6 e53/15 2:671fb830:ns::rbd_data.1256a0c02879.0000000000000007:head v 53'9, mlcod=53'7)", 2024-09-11T09:51:42.617 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.121833+0000", 2024-09-11T09:51:42.617 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 19.380296055999999, 2024-09-11T09:51:42.617 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.023876192000000001, 2024-09-11T09:51:42.617 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:42.617 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:42.617 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:42.618 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:42.618 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.618 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:42.618 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.121833+0000", 2024-09-11T09:51:42.618 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:42.618 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.618 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.618 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:42.618 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.121833+0000", 2024-09-11T09:51:42.618 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:42.618 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.618 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.618 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:42.618 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.121834+0000", 2024-09-11T09:51:42.619 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.471e-06 2024-09-11T09:51:42.619 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.619 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.619 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:42.619 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.128990+0000", 2024-09-11T09:51:42.619 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0071557879999999997 2024-09-11T09:51:42.619 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.619 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.619 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:42.619 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.129011+0000", 2024-09-11T09:51:42.619 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.0760000000000001e-05 2024-09-11T09:51:42.619 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.619 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.619 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:42.620 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.129022+0000", 2024-09-11T09:51:42.620 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1056000000000001e-05 2024-09-11T09:51:42.620 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.620 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.620 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:42.620 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.129097+0000", 2024-09-11T09:51:42.620 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.5082999999999999e-05 2024-09-11T09:51:42.620 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.620 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.620 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:42.620 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.129130+0000", 2024-09-11T09:51:42.620 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.3345e-05 2024-09-11T09:51:42.620 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.621 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.621 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:42.621 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.145657+0000", 2024-09-11T09:51:42.621 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.016527224 2024-09-11T09:51:42.621 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.621 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.621 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:42.621 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.145709+0000", 2024-09-11T09:51:42.621 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.1465000000000003e-05 2024-09-11T09:51:42.621 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:42.621 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:42.621 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:42.621 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.621 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.622 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4694.0:43 2.0 2:16c8ae84:ns::rbd_data.1256a0c02879.0000000000000005:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:42.622 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.135999+0000", 2024-09-11T09:51:42.622 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 19.366130106, 2024-09-11T09:51:42.622 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.039468309, 2024-09-11T09:51:42.622 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:42.622 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:42.622 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:42.622 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:42.622 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4694", 2024-09-11T09:51:42.622 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:51:42.622 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 43 2024-09-11T09:51:42.622 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.622 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:42.622 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.623 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:42.623 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.135999+0000", 2024-09-11T09:51:42.623 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:42.623 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.623 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.623 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:42.623 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.135999+0000", 2024-09-11T09:51:42.623 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:42.623 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.623 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.623 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:42.623 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.136000+0000", 2024-09-11T09:51:42.623 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.6330000000000001e-06 2024-09-11T09:51:42.623 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.624 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.624 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:42.624 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.157961+0000", 2024-09-11T09:51:42.624 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0219606 2024-09-11T09:51:42.624 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.624 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.624 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:42.624 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.157995+0000", 2024-09-11T09:51:42.624 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.4391999999999999e-05 2024-09-11T09:51:42.624 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.624 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.624 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:42.624 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.158011+0000", 2024-09-11T09:51:42.625 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.5794000000000002e-05 2024-09-11T09:51:42.625 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.625 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.625 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:42.625 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.158083+0000", 2024-09-11T09:51:42.625 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.2273000000000004e-05 2024-09-11T09:51:42.625 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.625 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.625 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:42.625 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.158239+0000", 2024-09-11T09:51:42.625 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000155372 2024-09-11T09:51:42.625 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.625 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.625 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 1", 2024-09-11T09:51:42.626 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.158403+0000", 2024-09-11T09:51:42.626 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00016408199999999999 2024-09-11T09:51:42.626 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.626 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.626 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:42.626 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.173860+0000", 2024-09-11T09:51:42.626 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.015457347 2024-09-11T09:51:42.626 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.626 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.626 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:42.626 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.175400+0000", 2024-09-11T09:51:42.626 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001540006 2024-09-11T09:51:42.626 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.627 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.627 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:42.627 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.175438+0000", 2024-09-11T09:51:42.627 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.8135e-05 2024-09-11T09:51:42.627 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.627 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.627 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:42.627 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.175467+0000", 2024-09-11T09:51:42.627 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.8674999999999999e-05 2024-09-11T09:51:42.627 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:42.627 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:42.627 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:42.627 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.627 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.628 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4694.0:57 2.0 2:04b5549c:ns::rbd_data.1256a0c02879.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:42.628 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.158037+0000", 2024-09-11T09:51:42.628 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 19.344091368000001, 2024-09-11T09:51:42.628 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.027748790999999998, 2024-09-11T09:51:42.628 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:42.628 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:42.628 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:42.628 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:42.628 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4694", 2024-09-11T09:51:42.628 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:51:42.628 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 57 2024-09-11T09:51:42.628 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.628 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:42.629 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.629 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:42.629 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.158037+0000", 2024-09-11T09:51:42.629 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:42.629 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.629 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.629 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:42.629 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.158037+0000", 2024-09-11T09:51:42.629 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:42.629 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.629 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.629 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:42.629 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.158039+0000", 2024-09-11T09:51:42.629 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.356e-06 2024-09-11T09:51:42.630 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.630 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.630 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:42.630 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.176367+0000", 2024-09-11T09:51:42.630 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.018328739 2024-09-11T09:51:42.630 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.630 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.630 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:42.630 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.176378+0000", 2024-09-11T09:51:42.630 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.0713999999999999e-05 2024-09-11T09:51:42.630 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.630 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.630 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:42.631 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.176383+0000", 2024-09-11T09:51:42.631 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.0209999999999999e-06 2024-09-11T09:51:42.631 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.631 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.631 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:42.631 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.176426+0000", 2024-09-11T09:51:42.631 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.2623999999999997e-05 2024-09-11T09:51:42.631 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.631 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.631 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:42.631 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.176552+0000", 2024-09-11T09:51:42.631 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000126076 2024-09-11T09:51:42.631 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.631 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.632 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 1", 2024-09-11T09:51:42.632 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.176665+0000", 2024-09-11T09:51:42.632 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000112995 2024-09-11T09:51:42.632 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.632 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.632 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:42.632 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.183639+0000", 2024-09-11T09:51:42.632 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0069740039999999998 2024-09-11T09:51:42.632 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.632 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.632 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:42.632 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.185689+0000", 2024-09-11T09:51:42.632 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.002049852 2024-09-11T09:51:42.633 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.633 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.633 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:42.633 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.185725+0000", 2024-09-11T09:51:42.633 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.6804999999999997e-05 2024-09-11T09:51:42.633 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.633 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.633 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:42.633 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.185786+0000", 2024-09-11T09:51:42.633 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.0605e-05 2024-09-11T09:51:42.633 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:42.633 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:42.633 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:42.633 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.634 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.634 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4694.0:60 2.0 2:054f2af4:ns::rbd_data.1256a0c02879.000000000000000f:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:42.634 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.186855+0000", 2024-09-11T09:51:42.634 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 19.315273351999998, 2024-09-11T09:51:42.634 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.013682913, 2024-09-11T09:51:42.634 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:42.634 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:42.634 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:42.634 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:42.634 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4694", 2024-09-11T09:51:42.634 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:51:42.634 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 60 2024-09-11T09:51:42.634 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.634 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:42.635 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.635 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:42.635 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.186855+0000", 2024-09-11T09:51:42.635 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:42.635 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.635 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.635 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:42.635 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.186855+0000", 2024-09-11T09:51:42.635 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:42.635 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.635 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.635 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:42.635 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.186856+0000", 2024-09-11T09:51:42.635 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.1200000000000001e-07 2024-09-11T09:51:42.636 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.636 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.636 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:42.636 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.191484+0000", 2024-09-11T09:51:42.636 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0046274159999999996 2024-09-11T09:51:42.636 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.636 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.636 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:42.636 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.191496+0000", 2024-09-11T09:51:42.636 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.2149e-05 2024-09-11T09:51:42.636 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.636 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.636 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:42.637 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.191502+0000", 2024-09-11T09:51:42.637 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.1890000000000004e-06 2024-09-11T09:51:42.637 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.637 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.637 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:42.637 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.191544+0000", 2024-09-11T09:51:42.637 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.1724999999999998e-05 2024-09-11T09:51:42.637 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.637 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.637 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:42.637 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.191641+0000", 2024-09-11T09:51:42.637 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.6939000000000001e-05 2024-09-11T09:51:42.637 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.637 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.638 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 1", 2024-09-11T09:51:42.638 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.191731+0000", 2024-09-11T09:51:42.638 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.0651999999999999e-05 2024-09-11T09:51:42.638 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.638 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.638 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:42.638 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.199852+0000", 2024-09-11T09:51:42.638 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0081204529999999997 2024-09-11T09:51:42.638 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.638 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.638 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:42.638 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.200461+0000", 2024-09-11T09:51:42.638 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00060900899999999998 2024-09-11T09:51:42.638 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.639 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.639 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:42.639 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.200497+0000", 2024-09-11T09:51:42.639 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.5800000000000003e-05 2024-09-11T09:51:42.639 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:42.639 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:42.639 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:42.639 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.200538+0000", 2024-09-11T09:51:42.639 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.1668999999999997e-05 2024-09-11T09:51:42.639 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:42.639 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:42.639 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:42.639 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:42.639 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:42.640 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-11T09:51:42.641 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-11T09:51:42.642 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:51:42.641+0000 7f3c96ec9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31909) UID: 0 2024-09-11T09:51:42.742 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-09-11T09:51:42.742+0000 7f7b6c7d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31904) UID: 0 2024-09-11T09:51:42.761 INFO:teuthology.orchestra.run.smithi017.stderr:nodeep-scrub is set 2024-09-11T09:51:42.770 INFO:tasks.workunit.client.0.smithi107.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2024-09-11T09:51:42.779 INFO:tasks.workunit.client.0.smithi107.stderr:++ _sudo rbd device --device-type nbd map rbd/ns/testrbdnbd27121@snap 2024-09-11T09:51:42.779 INFO:tasks.workunit.client.0.smithi107.stderr:++ local cmd 2024-09-11T09:51:42.779 INFO:tasks.workunit.client.0.smithi107.stderr:+++ id -u 2024-09-11T09:51:42.780 INFO:tasks.workunit.client.0.smithi107.stderr:++ '[' 1000 -eq 0 ']' 2024-09-11T09:51:42.781 INFO:tasks.workunit.client.0.smithi107.stderr:+++ which rbd 2024-09-11T09:51:42.783 INFO:tasks.workunit.client.0.smithi107.stderr:++ cmd=/usr/bin/rbd 2024-09-11T09:51:42.783 INFO:tasks.workunit.client.0.smithi107.stderr:++ shift 2024-09-11T09:51:42.783 INFO:tasks.workunit.client.0.smithi107.stderr:++ sudo -nE /usr/bin/rbd device --device-type nbd map rbd/ns/testrbdnbd27121@snap 2024-09-11T09:51:42.786 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-11T09:51:42.786 INFO:teuthology.orchestra.run.smithi096.stdout: "ops": [], 2024-09-11T09:51:42.786 INFO:teuthology.orchestra.run.smithi096.stdout: "num_ops": 0 2024-09-11T09:51:42.786 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-11T09:51:42.795 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-11T09:51:42.843 INFO:tasks.ceph.osd.2.smithi017.stderr:2024-09-11T09:51:42.843+0000 7fd20b170640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31672) UID: 0 2024-09-11T09:51:42.888 INFO:tasks.workunit.client.0.smithi107.stderr:+ DEV=/dev/nbd0 2024-09-11T09:51:42.888 INFO:tasks.workunit.client.0.smithi107.stderr:+ get_pid rbd ns 2024-09-11T09:51:42.888 INFO:tasks.workunit.client.0.smithi107.stderr:+ local pool=rbd 2024-09-11T09:51:42.888 INFO:tasks.workunit.client.0.smithi107.stderr:+ local ns=ns 2024-09-11T09:51:42.888 INFO:tasks.workunit.client.0.smithi107.stderr:++ rbd device --device-type nbd --format xml list 2024-09-11T09:51:42.888 INFO:tasks.workunit.client.0.smithi107.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\''ns'\''][image='\''testrbdnbd27121'\''][device='\''/dev/nbd0'\'']/id' 2024-09-11T09:51:42.940 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-11T09:51:42.941 INFO:teuthology.orchestra.run.smithi096.stdout: "ops": [], 2024-09-11T09:51:42.941 INFO:teuthology.orchestra.run.smithi096.stdout: "complaint_time": 30, 2024-09-11T09:51:42.941 INFO:teuthology.orchestra.run.smithi096.stdout: "num_blocked_ops": 0 2024-09-11T09:51:42.941 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-11T09:51:42.943 INFO:tasks.ceph.osd.2.smithi017.stderr:2024-09-11T09:51:42.944+0000 7fd20b170640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31672) UID: 0 2024-09-11T09:51:42.950 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-11T09:51:42.950 INFO:tasks.workunit.client.0.smithi107.stderr:+ PID=28639 2024-09-11T09:51:42.950 INFO:tasks.workunit.client.0.smithi107.stderr:+ test -n 28639 2024-09-11T09:51:42.951 INFO:tasks.workunit.client.0.smithi107.stderr:+ ps -p 28639 -C rbd-nbd 2024-09-11T09:51:42.961 INFO:tasks.workunit.client.0.smithi107.stdout: PID TTY TIME CMD 2024-09-11T09:51:42.961 INFO:tasks.workunit.client.0.smithi107.stdout: 28639 ? 00:00:00 rbd-nbd 2024-09-11T09:51:42.961 INFO:tasks.workunit.client.0.smithi107.stderr:+ unmap_device rbd/ns/testrbdnbd27121@snap 28639 2024-09-11T09:51:42.962 INFO:tasks.workunit.client.0.smithi107.stderr:+ local args=rbd/ns/testrbdnbd27121@snap 2024-09-11T09:51:42.962 INFO:tasks.workunit.client.0.smithi107.stderr:+ local pid=28639 2024-09-11T09:51:42.962 INFO:tasks.workunit.client.0.smithi107.stderr:+ _sudo rbd device --device-type nbd unmap rbd/ns/testrbdnbd27121@snap 2024-09-11T09:51:42.962 INFO:tasks.workunit.client.0.smithi107.stderr:+ local cmd 2024-09-11T09:51:42.962 INFO:tasks.workunit.client.0.smithi107.stderr:++ id -u 2024-09-11T09:51:42.962 INFO:tasks.workunit.client.0.smithi107.stderr:+ '[' 1000 -eq 0 ']' 2024-09-11T09:51:42.962 INFO:tasks.workunit.client.0.smithi107.stderr:++ which rbd 2024-09-11T09:51:42.964 INFO:tasks.workunit.client.0.smithi107.stderr:+ cmd=/usr/bin/rbd 2024-09-11T09:51:42.964 INFO:tasks.workunit.client.0.smithi107.stderr:+ shift 2024-09-11T09:51:42.964 INFO:tasks.workunit.client.0.smithi107.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd unmap rbd/ns/testrbdnbd27121@snap 2024-09-11T09:51:42.983 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T09:51:43.041 INFO:tasks.workunit.client.0.smithi107.stderr:+ rbd device --device-type nbd list 2024-09-11T09:51:43.041 INFO:tasks.workunit.client.0.smithi107.stderr:+ expect_false grep '^28639\b' 2024-09-11T09:51:43.041 INFO:tasks.workunit.client.0.smithi107.stderr:+ grep '^28639\b' 2024-09-11T09:51:43.044 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:51:43.045+0000 7f3c96ec9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31909) UID: 0 2024-09-11T09:51:43.055 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-11T09:51:43.056 INFO:teuthology.orchestra.run.smithi096.stdout: "size": 20, 2024-09-11T09:51:43.056 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 600, 2024-09-11T09:51:43.056 INFO:teuthology.orchestra.run.smithi096.stdout: "ops": [ 2024-09-11T09:51:43.056 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.056 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4230.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-11T09:51:43.056 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:18.490736+0000", 2024-09-11T09:51:43.056 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 84.564646957999997, 2024-09-11T09:51:43.056 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0059097569999999999, 2024-09-11T09:51:43.056 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:43.057 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:43.057 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:43.057 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:43.057 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4230", 2024-09-11T09:51:43.057 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.17:0/4101109576", 2024-09-11T09:51:43.057 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 1 2024-09-11T09:51:43.057 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.057 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:43.057 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.057 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:43.057 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.490736+0000", 2024-09-11T09:51:43.057 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:43.058 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.058 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.058 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:43.058 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.490736+0000", 2024-09-11T09:51:43.058 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:43.058 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.058 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.058 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:43.058 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.490742+0000", 2024-09-11T09:51:43.058 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.48e-06 2024-09-11T09:51:43.058 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.058 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.058 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:43.059 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.490749+0000", 2024-09-11T09:51:43.059 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.4569999999999999e-06 2024-09-11T09:51:43.059 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.059 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.059 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:43.059 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.490762+0000", 2024-09-11T09:51:43.059 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.2721000000000001e-05 2024-09-11T09:51:43.059 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.059 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.059 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:43.059 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.490774+0000", 2024-09-11T09:51:43.059 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1823e-05 2024-09-11T09:51:43.059 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.060 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.060 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:43.060 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.490846+0000", 2024-09-11T09:51:43.060 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.1878000000000005e-05 2024-09-11T09:51:43.060 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.060 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.060 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for peered", 2024-09-11T09:51:43.060 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.490857+0000", 2024-09-11T09:51:43.060 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1017e-05 2024-09-11T09:51:43.060 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.060 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.060 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:43.060 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.493856+0000", 2024-09-11T09:51:43.061 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00299885 2024-09-11T09:51:43.061 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.061 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.061 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:43.062 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.494020+0000", 2024-09-11T09:51:43.062 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00016436599999999999 2024-09-11T09:51:43.062 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.062 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.062 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 2", 2024-09-11T09:51:43.062 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.494186+0000", 2024-09-11T09:51:43.062 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00016562799999999999 2024-09-11T09:51:43.062 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.062 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.062 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:43.062 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.495407+0000", 2024-09-11T09:51:43.062 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0012217420000000001 2024-09-11T09:51:43.063 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.063 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.063 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:43.063 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.496579+0000", 2024-09-11T09:51:43.063 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001171844 2024-09-11T09:51:43.063 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.063 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.063 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:43.063 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.496620+0000", 2024-09-11T09:51:43.063 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.0479000000000003e-05 2024-09-11T09:51:43.063 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.063 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.063 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:43.064 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.496646+0000", 2024-09-11T09:51:43.064 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.6472e-05 2024-09-11T09:51:43.064 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:43.064 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:43.064 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:43.064 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.064 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.064 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4230.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-09-11T09:51:43.064 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:20.498634+0000", 2024-09-11T09:51:43.064 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 82.556748936999995, 2024-09-11T09:51:43.064 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0050648539999999997, 2024-09-11T09:51:43.064 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:43.064 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:43.065 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:43.065 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:43.065 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4230", 2024-09-11T09:51:43.065 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.17:0/4101109576", 2024-09-11T09:51:43.065 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 7 2024-09-11T09:51:43.065 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.065 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:43.065 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.065 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:43.065 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.498634+0000", 2024-09-11T09:51:43.066 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:43.066 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.066 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.066 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:43.066 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.498634+0000", 2024-09-11T09:51:43.066 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:43.066 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.066 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.066 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:43.066 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.498637+0000", 2024-09-11T09:51:43.066 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.1919999999999999e-06 2024-09-11T09:51:43.066 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.067 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.067 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:43.067 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.498648+0000", 2024-09-11T09:51:43.067 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1116e-05 2024-09-11T09:51:43.067 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.067 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.067 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:43.067 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.498666+0000", 2024-09-11T09:51:43.067 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.8411999999999999e-05 2024-09-11T09:51:43.067 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.067 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.067 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:43.067 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.498675+0000", 2024-09-11T09:51:43.067 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.4059999999999994e-06 2024-09-11T09:51:43.068 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.068 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.068 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:43.068 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.498745+0000", 2024-09-11T09:51:43.068 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.9764000000000005e-05 2024-09-11T09:51:43.068 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.068 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.068 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op must wait for map", 2024-09-11T09:51:43.068 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.498754+0000", 2024-09-11T09:51:43.068 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.5969999999999992e-06 2024-09-11T09:51:43.069 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.069 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.069 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:43.069 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.500385+0000", 2024-09-11T09:51:43.069 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0016308360000000001 2024-09-11T09:51:43.069 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.069 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.069 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:43.069 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.500473+0000", 2024-09-11T09:51:43.069 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.7966000000000004e-05 2024-09-11T09:51:43.069 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.069 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.070 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 1", 2024-09-11T09:51:43.070 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.500591+0000", 2024-09-11T09:51:43.070 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000117895 2024-09-11T09:51:43.070 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.070 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.070 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:43.070 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.501685+0000", 2024-09-11T09:51:43.070 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001094886 2024-09-11T09:51:43.070 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.070 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.070 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:43.070 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.503627+0000", 2024-09-11T09:51:43.070 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001941199 2024-09-11T09:51:43.071 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.071 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.071 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:43.071 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.503667+0000", 2024-09-11T09:51:43.071 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.0645000000000001e-05 2024-09-11T09:51:43.071 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.071 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.071 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:43.071 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.503699+0000", 2024-09-11T09:51:43.071 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.1940000000000003e-05 2024-09-11T09:51:43.071 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:43.071 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:43.071 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:43.072 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.072 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.072 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4688.0:32 2.2 2:45941fd2:::rbd_data.1250f5e25b80.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:43.072 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:22.896607+0000", 2024-09-11T09:51:43.072 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 20.158776333999999, 2024-09-11T09:51:43.072 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.038655179999999997, 2024-09-11T09:51:43.072 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:43.072 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:43.072 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:43.072 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:43.072 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4688", 2024-09-11T09:51:43.073 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:51:43.073 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 32 2024-09-11T09:51:43.073 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.073 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:43.073 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.073 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:43.073 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.896607+0000", 2024-09-11T09:51:43.073 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:43.073 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.073 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.073 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:43.073 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.896607+0000", 2024-09-11T09:51:43.073 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:43.073 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.074 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.074 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:43.074 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.896608+0000", 2024-09-11T09:51:43.074 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.0690000000000001e-06 2024-09-11T09:51:43.074 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.074 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.074 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:43.074 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.914461+0000", 2024-09-11T09:51:43.074 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.017852903 2024-09-11T09:51:43.074 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.074 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.074 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:43.074 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.914495+0000", 2024-09-11T09:51:43.075 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.4301000000000002e-05 2024-09-11T09:51:43.075 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.075 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.075 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:43.075 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.914522+0000", 2024-09-11T09:51:43.075 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.7127e-05 2024-09-11T09:51:43.075 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.075 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.075 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:43.075 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.914583+0000", 2024-09-11T09:51:43.075 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.0730999999999998e-05 2024-09-11T09:51:43.075 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.075 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.076 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:43.076 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.914709+0000", 2024-09-11T09:51:43.076 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00012590400000000001 2024-09-11T09:51:43.076 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.076 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.076 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 1", 2024-09-11T09:51:43.076 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.914809+0000", 2024-09-11T09:51:43.076 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.9882000000000004e-05 2024-09-11T09:51:43.076 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.076 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.076 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:43.076 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.932551+0000", 2024-09-11T09:51:43.076 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.017742048999999999 2024-09-11T09:51:43.077 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.077 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.077 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:43.077 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.935144+0000", 2024-09-11T09:51:43.077 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0025927630000000001 2024-09-11T09:51:43.077 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.077 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.077 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:43.077 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.935205+0000", 2024-09-11T09:51:43.077 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.1634e-05 2024-09-11T09:51:43.077 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.077 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.077 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:43.078 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.935262+0000", 2024-09-11T09:51:43.078 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.6817000000000002e-05 2024-09-11T09:51:43.078 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:43.078 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:43.078 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:43.078 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.078 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.078 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_repop(client.4688.0:40 2.7 e53/38 2:fe41411c:::rbd_object_map.1250f5e25b80:head v 53'9, mlcod=53'7)", 2024-09-11T09:51:43.078 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:22.920457+0000", 2024-09-11T09:51:43.078 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 20.134926242999999, 2024-09-11T09:51:43.078 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.011132817, 2024-09-11T09:51:43.078 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:43.079 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:43.079 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:43.079 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:43.079 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.079 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:43.079 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.920457+0000", 2024-09-11T09:51:43.079 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:43.079 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.079 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.079 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:43.079 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.920457+0000", 2024-09-11T09:51:43.079 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:43.079 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.080 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.080 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:43.080 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.920459+0000", 2024-09-11T09:51:43.080 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.6729999999999999e-06 2024-09-11T09:51:43.080 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.080 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.080 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:43.080 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.920469+0000", 2024-09-11T09:51:43.080 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.944e-06 2024-09-11T09:51:43.080 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.080 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.080 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:43.080 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.920495+0000", 2024-09-11T09:51:43.081 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.6401999999999999e-05 2024-09-11T09:51:43.081 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.081 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.081 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:43.081 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.920506+0000", 2024-09-11T09:51:43.081 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.133e-05 2024-09-11T09:51:43.081 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.081 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.081 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:43.081 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.920581+0000", 2024-09-11T09:51:43.081 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.4932999999999996e-05 2024-09-11T09:51:43.081 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.081 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.082 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:43.082 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.920612+0000", 2024-09-11T09:51:43.082 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.0537999999999998e-05 2024-09-11T09:51:43.082 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.082 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.082 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:43.082 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.931537+0000", 2024-09-11T09:51:43.082 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.010925239 2024-09-11T09:51:43.082 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.082 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.082 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:43.082 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.931590+0000", 2024-09-11T09:51:43.082 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.2757999999999999e-05 2024-09-11T09:51:43.083 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:43.083 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:43.083 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:43.083 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.083 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.083 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_repop(client.4688.0:41 2.7 e53/38 2:fe41411c:::rbd_object_map.1250f5e25b80:head v 53'10, mlcod=53'7)", 2024-09-11T09:51:43.083 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:22.921422+0000", 2024-09-11T09:51:43.083 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 20.133960826999999, 2024-09-11T09:51:43.083 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.013734942, 2024-09-11T09:51:43.083 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:43.083 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:43.083 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:43.083 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:43.084 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.084 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:43.084 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921422+0000", 2024-09-11T09:51:43.084 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:43.084 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.084 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.084 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:43.084 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921422+0000", 2024-09-11T09:51:43.084 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:43.084 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.084 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.084 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:43.084 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921426+0000", 2024-09-11T09:51:43.085 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.6150000000000001e-06 2024-09-11T09:51:43.085 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.085 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.085 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:43.085 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921435+0000", 2024-09-11T09:51:43.085 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.9840000000000007e-06 2024-09-11T09:51:43.085 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.085 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.085 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:43.085 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921444+0000", 2024-09-11T09:51:43.085 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.6000000000000007e-06 2024-09-11T09:51:43.085 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.085 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.086 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:43.086 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921450+0000", 2024-09-11T09:51:43.086 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.4690000000000002e-06 2024-09-11T09:51:43.086 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.086 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.086 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:43.086 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921511+0000", 2024-09-11T09:51:43.086 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.0779999999999997e-05 2024-09-11T09:51:43.086 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.086 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.086 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:43.086 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921531+0000", 2024-09-11T09:51:43.086 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.0302999999999999e-05 2024-09-11T09:51:43.087 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.087 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.087 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:43.087 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.935115+0000", 2024-09-11T09:51:43.087 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.013583471 2024-09-11T09:51:43.087 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.087 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.087 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:43.087 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.935157+0000", 2024-09-11T09:51:43.087 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.2719999999999998e-05 2024-09-11T09:51:43.087 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:43.087 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:43.087 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:43.088 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.088 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.088 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4688.0:42 2.3 2:c54742d8:::rbd_data.1250f5e25b80.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:43.088 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:22.932560+0000", 2024-09-11T09:51:43.088 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 20.122823084, 2024-09-11T09:51:43.088 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.017798198000000001, 2024-09-11T09:51:43.088 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:43.088 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:43.088 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:43.088 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:43.088 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4688", 2024-09-11T09:51:43.088 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:51:43.088 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 42 2024-09-11T09:51:43.089 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.089 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:43.089 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.089 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:43.089 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.932560+0000", 2024-09-11T09:51:43.089 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:43.089 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.089 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.089 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:43.089 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.932560+0000", 2024-09-11T09:51:43.089 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:43.089 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.089 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.089 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:43.090 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.932562+0000", 2024-09-11T09:51:43.090 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.17e-06 2024-09-11T09:51:43.090 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.090 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.090 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:43.090 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.938851+0000", 2024-09-11T09:51:43.090 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.006288971 2024-09-11T09:51:43.090 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.090 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.090 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:43.090 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.938865+0000", 2024-09-11T09:51:43.090 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.365e-05 2024-09-11T09:51:43.090 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.091 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.091 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:43.091 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.938871+0000", 2024-09-11T09:51:43.091 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.8850000000000004e-06 2024-09-11T09:51:43.091 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.091 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.091 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:43.091 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.938926+0000", 2024-09-11T09:51:43.091 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.4691999999999997e-05 2024-09-11T09:51:43.091 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.091 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.091 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:43.091 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.939059+0000", 2024-09-11T09:51:43.091 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00013310899999999999 2024-09-11T09:51:43.092 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.092 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.092 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 2", 2024-09-11T09:51:43.092 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.939171+0000", 2024-09-11T09:51:43.092 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000112718 2024-09-11T09:51:43.092 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.092 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.092 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:43.092 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.950262+0000", 2024-09-11T09:51:43.092 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.011091063 2024-09-11T09:51:43.092 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.092 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.093 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:43.093 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.950274+0000", 2024-09-11T09:51:43.093 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1963e-05 2024-09-11T09:51:43.093 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.093 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.093 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:43.093 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.950318+0000", 2024-09-11T09:51:43.093 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.3519999999999997e-05 2024-09-11T09:51:43.093 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.093 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.093 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:43.093 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.950358+0000", 2024-09-11T09:51:43.093 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.0457000000000002e-05 2024-09-11T09:51:43.093 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:43.094 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:43.094 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:43.094 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.094 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.094 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_repop(client.4694.0:34 2.7 e53/38 2:f80bfc0c:ns::rbd_data.1256a0c02879.0000000000000002:head v 53'25, mlcod=53'23)", 2024-09-11T09:51:43.094 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.094944+0000", 2024-09-11T09:51:43.094 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 19.960439692000001, 2024-09-11T09:51:43.094 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.010896785000000001, 2024-09-11T09:51:43.094 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:43.094 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:43.094 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:43.094 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:43.095 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.095 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:43.095 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.094944+0000", 2024-09-11T09:51:43.095 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:43.095 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.095 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.095 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:43.096 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.094944+0000", 2024-09-11T09:51:43.096 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:43.096 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.096 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.096 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:43.096 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.094944+0000", 2024-09-11T09:51:43.096 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.0699999999999996e-07 2024-09-11T09:51:43.097 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.097 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.097 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:43.097 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.099060+0000", 2024-09-11T09:51:43.097 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0041155580000000001 2024-09-11T09:51:43.097 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.097 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.097 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:43.097 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.099082+0000", 2024-09-11T09:51:43.097 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.2323999999999998e-05 2024-09-11T09:51:43.097 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.097 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.098 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:43.098 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.099094+0000", 2024-09-11T09:51:43.098 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1535000000000001e-05 2024-09-11T09:51:43.098 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.098 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.098 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:43.098 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.099160+0000", 2024-09-11T09:51:43.098 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.6449e-05 2024-09-11T09:51:43.098 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.098 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.098 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:43.098 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.099183+0000", 2024-09-11T09:51:43.098 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.3108e-05 2024-09-11T09:51:43.099 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.099 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.099 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:43.099 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.105776+0000", 2024-09-11T09:51:43.099 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0065922760000000002 2024-09-11T09:51:43.099 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.099 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.099 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:43.099 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.105840+0000", 2024-09-11T09:51:43.099 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.4727999999999994e-05 2024-09-11T09:51:43.099 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:43.099 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:43.099 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:43.099 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.099 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.100 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_repop(client.4694.0:35 2.7 e53/38 2:e8766325:ns::rbd_object_map.1256a0c02879:head v 53'26, mlcod=53'23)", 2024-09-11T09:51:43.100 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.100964+0000", 2024-09-11T09:51:43.100 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 19.954419164000001, 2024-09-11T09:51:43.100 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0049649359999999997, 2024-09-11T09:51:43.100 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:43.100 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:43.100 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:43.100 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:43.100 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.100 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:43.100 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.100964+0000", 2024-09-11T09:51:43.100 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:43.101 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.101 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.101 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:43.101 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.100964+0000", 2024-09-11T09:51:43.101 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:43.101 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.101 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.101 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:43.101 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.100966+0000", 2024-09-11T09:51:43.101 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.6330000000000001e-06 2024-09-11T09:51:43.101 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.101 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.101 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:43.102 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.100979+0000", 2024-09-11T09:51:43.102 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.2813000000000001e-05 2024-09-11T09:51:43.102 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.102 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.102 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:43.102 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.100992+0000", 2024-09-11T09:51:43.102 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.3414999999999999e-05 2024-09-11T09:51:43.102 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.102 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.102 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:43.102 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.101002+0000", 2024-09-11T09:51:43.102 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.0159e-05 2024-09-11T09:51:43.102 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.103 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.103 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:43.103 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.102365+0000", 2024-09-11T09:51:43.103 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001362912 2024-09-11T09:51:43.103 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.103 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.103 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:43.103 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.102382+0000", 2024-09-11T09:51:43.103 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.7433000000000001e-05 2024-09-11T09:51:43.103 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.103 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.103 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:43.104 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.105901+0000", 2024-09-11T09:51:43.104 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.003518476 2024-09-11T09:51:43.104 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.104 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.104 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:43.104 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.105929+0000", 2024-09-11T09:51:43.104 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.8095000000000001e-05 2024-09-11T09:51:43.104 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:43.104 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:43.104 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:43.104 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.104 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.104 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4694.0:41 2.2 2:5e2505c4:ns::rbd_data.1256a0c02879.0000000000000003:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:43.105 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.107511+0000", 2024-09-11T09:51:43.105 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 19.947872011000001, 2024-09-11T09:51:43.105 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.018571066000000001, 2024-09-11T09:51:43.105 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:43.105 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:43.105 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:43.105 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:43.105 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4694", 2024-09-11T09:51:43.105 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:51:43.105 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 41 2024-09-11T09:51:43.105 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.105 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:43.106 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.106 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:43.106 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.107511+0000", 2024-09-11T09:51:43.106 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:43.106 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.106 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.106 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:43.106 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.107511+0000", 2024-09-11T09:51:43.106 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:43.106 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.106 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.106 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:43.106 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.107513+0000", 2024-09-11T09:51:43.107 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.3489999999999999e-06 2024-09-11T09:51:43.107 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.107 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.107 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:43.107 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.112704+0000", 2024-09-11T09:51:43.107 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0051918509999999999 2024-09-11T09:51:43.107 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.107 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.107 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:43.107 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.112718+0000", 2024-09-11T09:51:43.107 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.359e-05 2024-09-11T09:51:43.107 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.107 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.108 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:43.108 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.112725+0000", 2024-09-11T09:51:43.108 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.9940000000000003e-06 2024-09-11T09:51:43.108 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.108 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.108 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:43.108 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.112788+0000", 2024-09-11T09:51:43.108 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.2447999999999996e-05 2024-09-11T09:51:43.108 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.108 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.108 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:43.108 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.112937+0000", 2024-09-11T09:51:43.108 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00014921499999999999 2024-09-11T09:51:43.109 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.109 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.109 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 1", 2024-09-11T09:51:43.109 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.113085+0000", 2024-09-11T09:51:43.109 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000148085 2024-09-11T09:51:43.109 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.109 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.109 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:43.109 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.124318+0000", 2024-09-11T09:51:43.109 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.011233158 2024-09-11T09:51:43.109 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.109 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.109 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:43.110 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.125957+0000", 2024-09-11T09:51:43.110 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001638701 2024-09-11T09:51:43.110 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.110 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.110 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:43.110 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.126029+0000", 2024-09-11T09:51:43.110 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.2125000000000005e-05 2024-09-11T09:51:43.110 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.110 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.110 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:43.110 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.126082+0000", 2024-09-11T09:51:43.110 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.3550000000000001e-05 2024-09-11T09:51:43.110 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:43.111 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:43.111 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:43.111 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.111 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.111 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4694.0:44 2.3 2:cd1a7d1c:ns::rbd_data.1256a0c02879.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:43.111 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.112744+0000", 2024-09-11T09:51:43.111 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 19.942639444000001, 2024-09-11T09:51:43.111 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.021672212999999999, 2024-09-11T09:51:43.111 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:43.111 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:43.111 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:43.111 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:43.111 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4694", 2024-09-11T09:51:43.112 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:51:43.112 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 44 2024-09-11T09:51:43.112 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.112 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:43.112 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.112 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:43.112 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.112744+0000", 2024-09-11T09:51:43.112 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:43.112 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.112 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.112 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:43.112 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.112744+0000", 2024-09-11T09:51:43.112 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:43.113 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.113 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.113 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:43.113 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.112745+0000", 2024-09-11T09:51:43.113 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.2900000000000002e-07 2024-09-11T09:51:43.113 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.113 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.113 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:43.113 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.119929+0000", 2024-09-11T09:51:43.113 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0071846230000000002 2024-09-11T09:51:43.113 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.113 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.113 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:43.114 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.119949+0000", 2024-09-11T09:51:43.114 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.9264000000000001e-05 2024-09-11T09:51:43.114 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.114 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.114 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:43.114 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.119959+0000", 2024-09-11T09:51:43.114 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.0465999999999999e-05 2024-09-11T09:51:43.114 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.114 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.114 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:43.114 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.120028+0000", 2024-09-11T09:51:43.114 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.9318000000000003e-05 2024-09-11T09:51:43.114 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.115 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.115 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:43.115 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.120160+0000", 2024-09-11T09:51:43.115 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000132015 2024-09-11T09:51:43.115 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.115 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.115 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 2", 2024-09-11T09:51:43.115 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.120260+0000", 2024-09-11T09:51:43.115 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.9397999999999999e-05 2024-09-11T09:51:43.115 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.115 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.115 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:43.115 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.134246+0000", 2024-09-11T09:51:43.116 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.013986721000000001 2024-09-11T09:51:43.116 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.116 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.116 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:43.116 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.134295+0000", 2024-09-11T09:51:43.116 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.8649000000000001e-05 2024-09-11T09:51:43.116 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.116 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.116 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:43.116 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.134361+0000", 2024-09-11T09:51:43.116 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.6081999999999995e-05 2024-09-11T09:51:43.116 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.116 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.116 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:43.117 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.134416+0000", 2024-09-11T09:51:43.117 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.4848000000000001e-05 2024-09-11T09:51:43.117 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:43.117 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:43.117 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:43.117 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.117 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.117 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4694.0:46 2.3 2:ce0e1469:ns::rbd_data.1256a0c02879.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:43.117 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.119993+0000", 2024-09-11T09:51:43.117 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 19.935389967999999, 2024-09-11T09:51:43.117 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.025840610999999999, 2024-09-11T09:51:43.117 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:43.118 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:43.118 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:43.118 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:43.118 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4694", 2024-09-11T09:51:43.118 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:51:43.118 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 46 2024-09-11T09:51:43.118 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.118 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:43.118 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.118 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:43.118 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.119993+0000", 2024-09-11T09:51:43.118 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:43.118 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.119 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.119 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:43.119 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.119993+0000", 2024-09-11T09:51:43.119 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:43.119 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.119 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.119 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:43.119 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.119995+0000", 2024-09-11T09:51:43.119 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1930000000000001e-06 2024-09-11T09:51:43.119 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.119 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.119 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:43.119 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.128458+0000", 2024-09-11T09:51:43.120 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0084632630000000004 2024-09-11T09:51:43.120 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.120 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.120 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:43.120 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.128478+0000", 2024-09-11T09:51:43.120 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.0353e-05 2024-09-11T09:51:43.120 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.120 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.120 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:43.120 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.128491+0000", 2024-09-11T09:51:43.120 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.2954e-05 2024-09-11T09:51:43.120 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.120 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.121 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:43.121 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.128558+0000", 2024-09-11T09:51:43.121 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.7076999999999995e-05 2024-09-11T09:51:43.121 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.121 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.121 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:43.121 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.128708+0000", 2024-09-11T09:51:43.121 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00014939599999999999 2024-09-11T09:51:43.121 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.121 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.121 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 2", 2024-09-11T09:51:43.121 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.128866+0000", 2024-09-11T09:51:43.121 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000158145 2024-09-11T09:51:43.121 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.122 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.122 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:43.122 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.141045+0000", 2024-09-11T09:51:43.122 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.012179331999999999 2024-09-11T09:51:43.122 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.122 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.122 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:43.122 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.145742+0000", 2024-09-11T09:51:43.122 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0046972050000000003 2024-09-11T09:51:43.122 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.122 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.122 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:43.122 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.145779+0000", 2024-09-11T09:51:43.123 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.6544000000000001e-05 2024-09-11T09:51:43.123 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.123 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.123 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:43.123 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.145834+0000", 2024-09-11T09:51:43.123 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.5148999999999997e-05 2024-09-11T09:51:43.123 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:43.123 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:43.123 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:43.123 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.123 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.123 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4694.0:48 2.3 2:d2090b70:ns::rbd_data.1256a0c02879.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:43.123 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.130540+0000", 2024-09-11T09:51:43.124 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 19.924843702, 2024-09-11T09:51:43.124 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.017167789999999999, 2024-09-11T09:51:43.124 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:43.124 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:43.124 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:43.124 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:43.124 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4694", 2024-09-11T09:51:43.124 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:51:43.124 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 48 2024-09-11T09:51:43.124 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.124 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:43.124 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.124 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:43.124 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.130540+0000", 2024-09-11T09:51:43.125 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:43.125 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.125 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.125 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:43.125 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.130540+0000", 2024-09-11T09:51:43.125 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:43.125 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.125 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.125 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:43.125 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.130542+0000", 2024-09-11T09:51:43.125 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.9209999999999999e-06 2024-09-11T09:51:43.125 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.126 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.126 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:43.126 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.134691+0000", 2024-09-11T09:51:43.126 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0041498739999999996 2024-09-11T09:51:43.126 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.126 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.126 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:43.126 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.134705+0000", 2024-09-11T09:51:43.126 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.3669000000000001e-05 2024-09-11T09:51:43.126 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.126 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.126 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:43.126 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.134712+0000", 2024-09-11T09:51:43.126 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.8410000000000002e-06 2024-09-11T09:51:43.127 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.127 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.127 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:43.127 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.134754+0000", 2024-09-11T09:51:43.127 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.2524999999999997e-05 2024-09-11T09:51:43.127 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.127 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.127 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:43.127 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.134875+0000", 2024-09-11T09:51:43.127 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000120332 2024-09-11T09:51:43.127 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.127 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.128 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 2", 2024-09-11T09:51:43.128 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.135005+0000", 2024-09-11T09:51:43.128 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00013024900000000001 2024-09-11T09:51:43.128 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.128 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.128 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:43.128 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.145971+0000", 2024-09-11T09:51:43.128 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.010966203000000001 2024-09-11T09:51:43.128 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.129 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.129 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:43.129 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.147600+0000", 2024-09-11T09:51:43.129 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001628382 2024-09-11T09:51:43.129 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.129 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.129 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:43.129 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.147635+0000", 2024-09-11T09:51:43.129 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.5420000000000003e-05 2024-09-11T09:51:43.129 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.129 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.129 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:43.130 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.147707+0000", 2024-09-11T09:51:43.130 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.2373999999999995e-05 2024-09-11T09:51:43.130 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:43.130 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:43.130 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:43.130 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.130 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.130 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_repop(client.4694.0:49 2.7 e53/38 2:e8766325:ns::rbd_object_map.1256a0c02879:head v 53'33, mlcod=53'31)", 2024-09-11T09:51:43.130 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.135391+0000", 2024-09-11T09:51:43.130 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 19.919992137000001, 2024-09-11T09:51:43.130 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0051189859999999998, 2024-09-11T09:51:43.130 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:43.130 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:43.131 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:43.131 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:43.131 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.131 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:43.131 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.135391+0000", 2024-09-11T09:51:43.131 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:43.131 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.131 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.131 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:43.131 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.135391+0000", 2024-09-11T09:51:43.131 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:43.131 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.131 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.131 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:43.132 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.135393+0000", 2024-09-11T09:51:43.132 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.4020000000000001e-06 2024-09-11T09:51:43.132 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.132 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.132 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:43.132 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.135400+0000", 2024-09-11T09:51:43.132 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.5129999999999999e-06 2024-09-11T09:51:43.132 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.132 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.132 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:43.132 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.135411+0000", 2024-09-11T09:51:43.132 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1038e-05 2024-09-11T09:51:43.132 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.133 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.133 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:43.133 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.135418+0000", 2024-09-11T09:51:43.133 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.1489999999999997e-06 2024-09-11T09:51:43.133 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.133 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.133 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:43.133 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.135470+0000", 2024-09-11T09:51:43.133 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.2021999999999998e-05 2024-09-11T09:51:43.133 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.133 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.133 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:43.133 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.135501+0000", 2024-09-11T09:51:43.134 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.0349999999999999e-05 2024-09-11T09:51:43.134 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.134 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.134 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:43.134 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.140453+0000", 2024-09-11T09:51:43.134 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0049522409999999996 2024-09-11T09:51:43.134 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.134 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.134 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:43.134 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.140510+0000", 2024-09-11T09:51:43.134 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.7271000000000001e-05 2024-09-11T09:51:43.134 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:43.134 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:43.135 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:43.135 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.135 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.135 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4694.0:54 2.3 2:c44f0253:ns::rbd_data.1256a0c02879.000000000000000b:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:43.135 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.150151+0000", 2024-09-11T09:51:43.135 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 19.905232286, 2024-09-11T09:51:43.135 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.015103484, 2024-09-11T09:51:43.135 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:43.135 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:43.135 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:43.135 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:43.135 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4694", 2024-09-11T09:51:43.136 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:51:43.136 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 54 2024-09-11T09:51:43.136 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.136 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:43.136 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.136 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:43.136 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.150151+0000", 2024-09-11T09:51:43.136 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:43.136 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.136 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.136 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:43.136 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.150151+0000", 2024-09-11T09:51:43.137 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:43.137 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.137 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.137 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:43.137 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.150152+0000", 2024-09-11T09:51:43.137 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.2550000000000001e-06 2024-09-11T09:51:43.137 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.137 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.137 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:43.137 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.154528+0000", 2024-09-11T09:51:43.137 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0043758219999999997 2024-09-11T09:51:43.137 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.137 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.138 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:43.138 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.154541+0000", 2024-09-11T09:51:43.138 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.3414999999999999e-05 2024-09-11T09:51:43.138 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.138 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.138 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:43.138 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.154551+0000", 2024-09-11T09:51:43.138 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.4350000000000003e-06 2024-09-11T09:51:43.138 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.138 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.138 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:43.138 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.154615+0000", 2024-09-11T09:51:43.138 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.3627000000000006e-05 2024-09-11T09:51:43.139 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.139 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.139 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:43.139 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.154717+0000", 2024-09-11T09:51:43.139 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000102729 2024-09-11T09:51:43.139 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.139 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.139 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 2", 2024-09-11T09:51:43.139 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.154826+0000", 2024-09-11T09:51:43.139 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000108409 2024-09-11T09:51:43.139 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.139 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.139 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:43.140 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.165094+0000", 2024-09-11T09:51:43.140 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.010268210999999999 2024-09-11T09:51:43.140 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.140 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.140 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:43.140 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.165138+0000", 2024-09-11T09:51:43.140 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.4078e-05 2024-09-11T09:51:43.140 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.140 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.140 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:43.140 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.165204+0000", 2024-09-11T09:51:43.140 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.5716000000000006e-05 2024-09-11T09:51:43.140 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.141 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.141 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:43.141 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.165254+0000", 2024-09-11T09:51:43.141 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.0787e-05 2024-09-11T09:51:43.141 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:43.141 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:43.141 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:43.141 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.141 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.141 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_repop(client.4694.0:55 2.7 e53/38 2:e8766325:ns::rbd_object_map.1256a0c02879:head v 53'37, mlcod=53'34)", 2024-09-11T09:51:43.141 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.150221+0000", 2024-09-11T09:51:43.141 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 19.905162421, 2024-09-11T09:51:43.141 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00530015, 2024-09-11T09:51:43.142 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:43.142 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:43.142 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:43.142 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:43.142 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.142 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:43.142 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.150221+0000", 2024-09-11T09:51:43.142 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:43.142 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.142 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.142 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:43.142 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.150221+0000", 2024-09-11T09:51:43.142 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:43.143 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.143 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.143 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:43.143 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.150221+0000", 2024-09-11T09:51:43.143 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.9100000000000004e-07 2024-09-11T09:51:43.143 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.143 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.143 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:43.143 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.150226+0000", 2024-09-11T09:51:43.143 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.2250000000000002e-06 2024-09-11T09:51:43.143 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.143 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.143 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:43.143 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.150251+0000", 2024-09-11T09:51:43.144 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.4987e-05 2024-09-11T09:51:43.144 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.144 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.144 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:43.144 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.150258+0000", 2024-09-11T09:51:43.144 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.0790000000000003e-06 2024-09-11T09:51:43.144 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.144 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.144 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:43.144 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.150305+0000", 2024-09-11T09:51:43.144 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.6879000000000003e-05 2024-09-11T09:51:43.144 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.144 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.145 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:43.145 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.150320+0000", 2024-09-11T09:51:43.145 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.5268999999999999e-05 2024-09-11T09:51:43.145 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.145 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.145 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:43.145 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.155493+0000", 2024-09-11T09:51:43.145 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0051731390000000002 2024-09-11T09:51:43.145 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.145 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.145 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:43.145 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.155521+0000", 2024-09-11T09:51:43.145 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.8081000000000001e-05 2024-09-11T09:51:43.145 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:43.146 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:43.146 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:43.146 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.146 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.146 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4694.0:56 2.3 2:c462f029:ns::rbd_data.1256a0c02879.000000000000000c:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:43.146 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.154576+0000", 2024-09-11T09:51:43.146 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 19.900807445000002, 2024-09-11T09:51:43.146 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.020781621, 2024-09-11T09:51:43.146 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:43.146 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:43.146 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:43.146 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:43.146 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4694", 2024-09-11T09:51:43.147 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:51:43.147 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 56 2024-09-11T09:51:43.147 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.147 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:43.147 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.147 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:43.147 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.154576+0000", 2024-09-11T09:51:43.147 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:43.147 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.147 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.147 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:43.147 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.154576+0000", 2024-09-11T09:51:43.147 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:43.148 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.148 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.148 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:43.148 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.154577+0000", 2024-09-11T09:51:43.148 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.2249999999999999e-06 2024-09-11T09:51:43.148 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.148 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.148 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:43.148 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.158783+0000", 2024-09-11T09:51:43.148 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0042063889999999996 2024-09-11T09:51:43.148 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.148 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.148 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:43.149 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.158803+0000", 2024-09-11T09:51:43.149 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.9576999999999999e-05 2024-09-11T09:51:43.149 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.149 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.149 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:43.149 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.158815+0000", 2024-09-11T09:51:43.149 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.2221999999999999e-05 2024-09-11T09:51:43.149 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.149 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.149 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:43.149 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.158902+0000", 2024-09-11T09:51:43.149 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.6390000000000005e-05 2024-09-11T09:51:43.149 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.150 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.150 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:43.150 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.159141+0000", 2024-09-11T09:51:43.150 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000238886 2024-09-11T09:51:43.150 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.150 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.150 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 2", 2024-09-11T09:51:43.150 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.159279+0000", 2024-09-11T09:51:43.150 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000138073 2024-09-11T09:51:43.150 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.150 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.150 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:43.150 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.173297+0000", 2024-09-11T09:51:43.150 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.014018113 2024-09-11T09:51:43.151 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.151 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.151 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:43.151 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.175278+0000", 2024-09-11T09:51:43.151 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0019817390000000002 2024-09-11T09:51:43.151 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.151 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.151 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:43.151 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.175316+0000", 2024-09-11T09:51:43.151 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.7829e-05 2024-09-11T09:51:43.151 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.151 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.151 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:43.152 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.175357+0000", 2024-09-11T09:51:43.152 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.1177999999999998e-05 2024-09-11T09:51:43.152 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:43.152 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:43.152 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:43.152 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.152 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.152 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4694.0:59 2.2 2:41adefc9:ns::rbd_data.1256a0c02879.000000000000000e:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:43.152 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.158843+0000", 2024-09-11T09:51:43.152 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 19.896539878999999, 2024-09-11T09:51:43.152 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.017874207, 2024-09-11T09:51:43.152 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:43.152 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:43.153 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:43.153 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:43.153 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4694", 2024-09-11T09:51:43.153 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:51:43.153 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 59 2024-09-11T09:51:43.153 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.153 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:43.153 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.153 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:43.153 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.158843+0000", 2024-09-11T09:51:43.153 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:43.153 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.153 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.154 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:43.154 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.158843+0000", 2024-09-11T09:51:43.154 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:43.154 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.154 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.154 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:43.154 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.158845+0000", 2024-09-11T09:51:43.154 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.7439999999999999e-06 2024-09-11T09:51:43.154 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.154 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.154 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:43.154 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.162713+0000", 2024-09-11T09:51:43.154 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0038678829999999999 2024-09-11T09:51:43.154 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.155 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.155 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:43.155 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.162728+0000", 2024-09-11T09:51:43.155 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.5114e-05 2024-09-11T09:51:43.155 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.155 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.155 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:43.155 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.162737+0000", 2024-09-11T09:51:43.155 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.1099999999999992e-06 2024-09-11T09:51:43.155 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.155 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.155 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:43.155 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.162799+0000", 2024-09-11T09:51:43.156 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.1934999999999996e-05 2024-09-11T09:51:43.156 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.156 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.156 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:43.156 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.162911+0000", 2024-09-11T09:51:43.156 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000111867 2024-09-11T09:51:43.156 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.156 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.156 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 1", 2024-09-11T09:51:43.156 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.163030+0000", 2024-09-11T09:51:43.156 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000119108 2024-09-11T09:51:43.156 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.156 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.157 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:43.157 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.174086+0000", 2024-09-11T09:51:43.157 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.01105543 2024-09-11T09:51:43.157 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.157 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.157 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:43.157 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.176614+0000", 2024-09-11T09:51:43.157 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0025284940000000001 2024-09-11T09:51:43.157 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.157 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.157 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:43.157 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.176672+0000", 2024-09-11T09:51:43.157 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.7892000000000001e-05 2024-09-11T09:51:43.157 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.158 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.158 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:43.158 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.176718+0000", 2024-09-11T09:51:43.158 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.5630000000000002e-05 2024-09-11T09:51:43.158 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:43.158 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:43.158 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:43.158 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.158 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.158 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.5002.0:89 2.3 2:c54742d8:::rbd_data.1250f5e25b80.0000000000000006:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-09-11T09:51:43.158 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:38.546016+0000", 2024-09-11T09:51:43.158 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 4.5093673330000001, 2024-09-11T09:51:43.158 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0049098550000000003, 2024-09-11T09:51:43.159 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:43.159 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:43.159 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:43.159 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:43.159 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.5002", 2024-09-11T09:51:43.159 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/2204440692", 2024-09-11T09:51:43.159 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 89 2024-09-11T09:51:43.159 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.159 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:43.159 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.159 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:43.159 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.546016+0000", 2024-09-11T09:51:43.159 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:43.159 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.160 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.160 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:43.160 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.546016+0000", 2024-09-11T09:51:43.160 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:43.160 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.160 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.160 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:43.160 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.546017+0000", 2024-09-11T09:51:43.160 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.3399999999999998e-07 2024-09-11T09:51:43.160 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.160 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.160 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:43.160 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.546839+0000", 2024-09-11T09:51:43.161 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00082222800000000004 2024-09-11T09:51:43.161 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.161 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.161 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:43.161 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.546851+0000", 2024-09-11T09:51:43.161 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.2075999999999999e-05 2024-09-11T09:51:43.161 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.161 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.161 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:43.161 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.546856+0000", 2024-09-11T09:51:43.161 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.4009999999999996e-06 2024-09-11T09:51:43.161 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.161 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.162 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:43.162 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.546897+0000", 2024-09-11T09:51:43.162 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.0113e-05 2024-09-11T09:51:43.162 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.162 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.162 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:43.162 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.546954+0000", 2024-09-11T09:51:43.162 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.7825000000000003e-05 2024-09-11T09:51:43.162 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.162 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.162 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 2", 2024-09-11T09:51:43.162 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.547055+0000", 2024-09-11T09:51:43.162 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000100269 2024-09-11T09:51:43.163 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.163 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.163 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:43.163 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.549675+0000", 2024-09-11T09:51:43.163 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0026206609999999998 2024-09-11T09:51:43.163 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.163 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.163 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:43.163 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.550858+0000", 2024-09-11T09:51:43.163 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0011825270000000001 2024-09-11T09:51:43.163 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.163 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.163 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:43.163 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.550894+0000", 2024-09-11T09:51:43.164 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.5896999999999999e-05 2024-09-11T09:51:43.164 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.164 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.164 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:43.164 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.550926+0000", 2024-09-11T09:51:43.164 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.2023999999999997e-05 2024-09-11T09:51:43.164 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:43.164 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:43.164 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:43.164 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.164 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.164 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.5002.0:91 2.3 2:c54742d8:::rbd_data.1250f5e25b80.0000000000000006:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-09-11T09:51:43.164 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:38.553743+0000", 2024-09-11T09:51:43.165 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 4.5016402900000001, 2024-09-11T09:51:43.165 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0051633629999999998, 2024-09-11T09:51:43.165 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:43.165 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:43.165 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:43.165 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:43.165 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.5002", 2024-09-11T09:51:43.165 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/2204440692", 2024-09-11T09:51:43.165 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 91 2024-09-11T09:51:43.165 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.165 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:43.165 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.165 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:43.166 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.553743+0000", 2024-09-11T09:51:43.166 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:43.166 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.166 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.166 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:43.166 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.553743+0000", 2024-09-11T09:51:43.166 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:43.166 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.166 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.166 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:43.166 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.553745+0000", 2024-09-11T09:51:43.166 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.5120000000000001e-06 2024-09-11T09:51:43.167 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.167 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.167 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:43.167 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.554588+0000", 2024-09-11T09:51:43.167 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00084352299999999999 2024-09-11T09:51:43.167 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.167 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.167 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:43.167 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.554604+0000", 2024-09-11T09:51:43.167 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.643e-05 2024-09-11T09:51:43.167 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.167 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.167 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:43.168 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.554613+0000", 2024-09-11T09:51:43.168 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.5669999999999995e-06 2024-09-11T09:51:43.168 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.168 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.168 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:43.168 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.554662+0000", 2024-09-11T09:51:43.168 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.8510000000000001e-05 2024-09-11T09:51:43.168 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.168 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.168 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:43.168 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.554732+0000", 2024-09-11T09:51:43.168 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.0297999999999997e-05 2024-09-11T09:51:43.168 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.169 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.169 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 2", 2024-09-11T09:51:43.169 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.554870+0000", 2024-09-11T09:51:43.169 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00013856599999999999 2024-09-11T09:51:43.169 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.169 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.169 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:43.169 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.557875+0000", 2024-09-11T09:51:43.169 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0030042760000000002 2024-09-11T09:51:43.169 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.169 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.169 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:43.169 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.558835+0000", 2024-09-11T09:51:43.170 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00096057900000000003 2024-09-11T09:51:43.170 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.170 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.170 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:43.170 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.558874+0000", 2024-09-11T09:51:43.170 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.8992999999999997e-05 2024-09-11T09:51:43.170 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.170 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.170 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:43.170 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.558906+0000", 2024-09-11T09:51:43.170 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.2109e-05 2024-09-11T09:51:43.170 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:43.170 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:43.171 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:43.171 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.171 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.171 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.5002.0:93 2.3 2:c54742d8:::rbd_data.1250f5e25b80.0000000000000006:head [write 2228224~917504] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e61)", 2024-09-11T09:51:43.171 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:38.562067+0000", 2024-09-11T09:51:43.171 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 4.4933166939999998, 2024-09-11T09:51:43.171 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0050551099999999998, 2024-09-11T09:51:43.171 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:43.171 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:43.171 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:43.171 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:43.171 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.5002", 2024-09-11T09:51:43.171 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/2204440692", 2024-09-11T09:51:43.172 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 93 2024-09-11T09:51:43.172 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.172 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:43.172 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.172 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:43.172 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.562067+0000", 2024-09-11T09:51:43.172 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:43.172 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.172 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.172 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:43.172 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.562067+0000", 2024-09-11T09:51:43.172 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:43.173 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.173 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.173 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:43.173 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.562068+0000", 2024-09-11T09:51:43.173 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.0669999999999999e-06 2024-09-11T09:51:43.173 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.173 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.173 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:43.173 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.563170+0000", 2024-09-11T09:51:43.173 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00110256 2024-09-11T09:51:43.173 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.173 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.173 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:43.174 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.563185+0000", 2024-09-11T09:51:43.174 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.4873e-05 2024-09-11T09:51:43.174 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.174 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.174 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:43.174 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.563193+0000", 2024-09-11T09:51:43.174 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.1319999999999994e-06 2024-09-11T09:51:43.174 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.174 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.174 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:43.174 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.563238+0000", 2024-09-11T09:51:43.174 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.4773e-05 2024-09-11T09:51:43.174 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.175 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.175 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:43.175 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.563290+0000", 2024-09-11T09:51:43.175 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.1690000000000001e-05 2024-09-11T09:51:43.175 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.175 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.175 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 2", 2024-09-11T09:51:43.175 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.563365+0000", 2024-09-11T09:51:43.175 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.5023000000000003e-05 2024-09-11T09:51:43.175 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.175 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.175 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:43.175 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.566552+0000", 2024-09-11T09:51:43.176 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0031869960000000001 2024-09-11T09:51:43.176 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.176 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.176 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:43.176 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.567048+0000", 2024-09-11T09:51:43.176 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00049605099999999996 2024-09-11T09:51:43.176 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.176 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.176 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:43.176 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.567095+0000", 2024-09-11T09:51:43.176 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.6823999999999998e-05 2024-09-11T09:51:43.176 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:43.176 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:43.177 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:43.177 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.567122+0000", 2024-09-11T09:51:43.177 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.7121e-05 2024-09-11T09:51:43.177 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:43.177 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:43.177 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:43.177 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:43.177 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:43.177 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-11T09:51:43.178 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-11T09:51:43.179 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T09:51:43.180 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-11T09:51:43.180 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T09:51:43.180 INFO:tasks.workunit.client.0.smithi107.stderr:+ return 0 2024-09-11T09:51:43.180 INFO:tasks.workunit.client.0.smithi107.stderr:+ ps -C rbd-nbd 2024-09-11T09:51:43.180 INFO:tasks.workunit.client.0.smithi107.stderr:+ expect_false grep '^ *28639\b' 2024-09-11T09:51:43.180 INFO:tasks.workunit.client.0.smithi107.stderr:+ grep '^ *28639\b' 2024-09-11T09:51:43.180 INFO:tasks.workunit.client.0.smithi107.stderr:+ return 0 2024-09-11T09:51:43.180 INFO:tasks.workunit.client.0.smithi107.stderr:+ sleep 0.5 2024-09-11T09:51:43.181 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-11T09:51:43.181 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-11T09:51:43.182 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T09:51:43.182 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-11T09:51:43.182 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T09:51:43.182 INFO:teuthology.orchestra.run.smithi017.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-09-11T09:51:43.183 INFO:tasks.ceph.osd.2.smithi017.stderr:2024-09-11T09:51:43.178+0000 7fd20b170640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31672) UID: 0 2024-09-11T09:51:43.183 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-11T09:51:43.183 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T09:51:43.185 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-11T09:51:43.186 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-11T09:51:43.234 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:51:43.235+0000 7f3a7f013640 -1 osd.4 54 log_to_monitors true 2024-09-11T09:51:43.279 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:51:43.280+0000 7f3c96ec9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31909) UID: 0 2024-09-11T09:51:43.290 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:51:43.290 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:51:43.299 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-11T09:51:43.380 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:51:43.380+0000 7f3c96ec9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31909) UID: 0 2024-09-11T09:51:43.406 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:51:43.406 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:51:43.414 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-11T09:51:43.481 INFO:tasks.ceph.osd.0.smithi017.stderr:2024-09-11T09:51:43.481+0000 7fb73c0f1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31665) UID: 0 2024-09-11T09:51:43.521 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:51:43.521 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:51:43.530 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-11T09:51:43.593 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-11T09:51:43.599 INFO:tasks.workunit.client.0.smithi107.stderr:+ DEV= 2024-09-11T09:51:43.600 INFO:tasks.workunit.client.0.smithi107.stderr:++ rbd snap ls rbd/ns/testrbdnbd27121 2024-09-11T09:51:43.600 INFO:tasks.workunit.client.0.smithi107.stderr:++ awk '$2 == "snap" {print $1}' 2024-09-11T09:51:43.614 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:51:43.614+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:51:43.644 INFO:tasks.workunit.client.0.smithi107.stderr:+ SNAPID=4 2024-09-11T09:51:43.644 INFO:tasks.workunit.client.0.smithi107.stderr:++ _sudo rbd device --device-type nbd map --snap-id 4 rbd/ns/testrbdnbd27121 2024-09-11T09:51:43.645 INFO:tasks.workunit.client.0.smithi107.stderr:++ local cmd 2024-09-11T09:51:43.645 INFO:tasks.workunit.client.0.smithi107.stderr:+++ id -u 2024-09-11T09:51:43.645 INFO:tasks.workunit.client.0.smithi107.stderr:++ '[' 1000 -eq 0 ']' 2024-09-11T09:51:43.646 INFO:tasks.workunit.client.0.smithi107.stderr:+++ which rbd 2024-09-11T09:51:43.647 INFO:tasks.workunit.client.0.smithi107.stderr:++ cmd=/usr/bin/rbd 2024-09-11T09:51:43.647 INFO:tasks.workunit.client.0.smithi107.stderr:++ shift 2024-09-11T09:51:43.648 INFO:tasks.workunit.client.0.smithi107.stderr:++ sudo -nE /usr/bin/rbd device --device-type nbd map --snap-id 4 rbd/ns/testrbdnbd27121 2024-09-11T09:51:43.678 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:51:43.678 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:51:43.682 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:51:43.683+0000 7f3c96ec9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31909) UID: 0 2024-09-11T09:51:43.686 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-11T09:51:43.696 INFO:teuthology.orchestra.run.smithi096.stderr:no valid command found; 10 closest matches: 2024-09-11T09:51:43.696 INFO:teuthology.orchestra.run.smithi096.stderr:0 2024-09-11T09:51:43.696 INFO:teuthology.orchestra.run.smithi096.stderr:1 2024-09-11T09:51:43.696 INFO:teuthology.orchestra.run.smithi096.stderr:2 2024-09-11T09:51:43.697 INFO:teuthology.orchestra.run.smithi096.stderr:abort 2024-09-11T09:51:43.697 INFO:teuthology.orchestra.run.smithi096.stderr:assert 2024-09-11T09:51:43.697 INFO:teuthology.orchestra.run.smithi096.stderr:bluefs debug_inject_read_zeros 2024-09-11T09:51:43.697 INFO:teuthology.orchestra.run.smithi096.stderr:bluefs files list 2024-09-11T09:51:43.697 INFO:teuthology.orchestra.run.smithi096.stderr:bluefs stats 2024-09-11T09:51:43.698 INFO:teuthology.orchestra.run.smithi096.stderr:bluestore allocator dump block 2024-09-11T09:51:43.698 INFO:teuthology.orchestra.run.smithi096.stderr:bluestore allocator fragmentation block 2024-09-11T09:51:43.698 INFO:teuthology.orchestra.run.smithi096.stderr:admin_socket: invalid command 2024-09-11T09:51:43.699 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-11T09:51:43.699 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-4, ['dump_ops_in_flight'] 2024-09-11T09:51:43.759 INFO:tasks.workunit.client.0.smithi107.stderr:+ DEV=/dev/nbd0 2024-09-11T09:51:43.760 INFO:tasks.workunit.client.0.smithi107.stderr:+ get_pid rbd ns 2024-09-11T09:51:43.760 INFO:tasks.workunit.client.0.smithi107.stderr:+ local pool=rbd 2024-09-11T09:51:43.760 INFO:tasks.workunit.client.0.smithi107.stderr:+ local ns=ns 2024-09-11T09:51:43.760 INFO:tasks.workunit.client.0.smithi107.stderr:++ rbd device --device-type nbd --format xml list 2024-09-11T09:51:43.760 INFO:tasks.workunit.client.0.smithi107.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\''ns'\''][image='\''testrbdnbd27121'\''][device='\''/dev/nbd0'\'']/id' 2024-09-11T09:51:43.779 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:51:43.780+0000 7f3a72715640 -1 osd.4 54 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-11T09:51:43.783 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:51:43.784+0000 7f3c96ec9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31909) UID: 0 2024-09-11T09:51:43.823 INFO:tasks.workunit.client.0.smithi107.stderr:+ PID=28727 2024-09-11T09:51:43.823 INFO:tasks.workunit.client.0.smithi107.stderr:+ test -n 28727 2024-09-11T09:51:43.823 INFO:tasks.workunit.client.0.smithi107.stderr:+ ps -p 28727 -C rbd-nbd 2024-09-11T09:51:43.834 INFO:tasks.workunit.client.0.smithi107.stdout: PID TTY TIME CMD 2024-09-11T09:51:43.834 INFO:tasks.workunit.client.0.smithi107.stdout: 28727 ? 00:00:00 rbd-nbd 2024-09-11T09:51:43.835 INFO:tasks.workunit.client.0.smithi107.stderr:+ unmap_device '--snap-id 4 rbd/ns/testrbdnbd27121' 28727 2024-09-11T09:51:43.835 INFO:tasks.workunit.client.0.smithi107.stderr:+ local 'args=--snap-id 4 rbd/ns/testrbdnbd27121' 2024-09-11T09:51:43.835 INFO:tasks.workunit.client.0.smithi107.stderr:+ local pid=28727 2024-09-11T09:51:43.835 INFO:tasks.workunit.client.0.smithi107.stderr:+ _sudo rbd device --device-type nbd unmap --snap-id 4 rbd/ns/testrbdnbd27121 2024-09-11T09:51:43.835 INFO:tasks.workunit.client.0.smithi107.stderr:+ local cmd 2024-09-11T09:51:43.835 INFO:tasks.workunit.client.0.smithi107.stderr:++ id -u 2024-09-11T09:51:43.836 INFO:tasks.workunit.client.0.smithi107.stderr:+ '[' 1000 -eq 0 ']' 2024-09-11T09:51:43.836 INFO:tasks.workunit.client.0.smithi107.stderr:++ which rbd 2024-09-11T09:51:43.838 INFO:tasks.workunit.client.0.smithi107.stderr:+ cmd=/usr/bin/rbd 2024-09-11T09:51:43.838 INFO:tasks.workunit.client.0.smithi107.stderr:+ shift 2024-09-11T09:51:43.838 INFO:tasks.workunit.client.0.smithi107.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd unmap --snap-id 4 rbd/ns/testrbdnbd27121 2024-09-11T09:51:43.838 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:51:43.838 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:51:43.847 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-11T09:51:43.892 INFO:tasks.ceph.osd.2.smithi017.stderr:2024-09-11T09:51:43.892+0000 7fd20b170640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31672) UID: 0 2024-09-11T09:51:43.922 INFO:tasks.workunit.client.0.smithi107.stderr:+ rbd device --device-type nbd list 2024-09-11T09:51:43.922 INFO:tasks.workunit.client.0.smithi107.stderr:+ expect_false grep '^28727\b' 2024-09-11T09:51:43.923 INFO:tasks.workunit.client.0.smithi107.stderr:+ grep '^28727\b' 2024-09-11T09:51:43.955 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:51:43.955 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:51:43.964 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-11T09:51:43.969 INFO:tasks.workunit.client.0.smithi107.stderr:+ return 0 2024-09-11T09:51:43.969 INFO:tasks.workunit.client.0.smithi107.stderr:+ ps -C rbd-nbd 2024-09-11T09:51:43.969 INFO:tasks.workunit.client.0.smithi107.stderr:+ expect_false grep '^ *28727\b' 2024-09-11T09:51:43.969 INFO:tasks.workunit.client.0.smithi107.stderr:+ grep '^ *28727\b' 2024-09-11T09:51:43.980 INFO:tasks.workunit.client.0.smithi107.stderr:+ return 0 2024-09-11T09:51:43.980 INFO:tasks.workunit.client.0.smithi107.stderr:+ sleep 0.5 2024-09-11T09:51:43.985 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:51:43.986+0000 7f3c96ec9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31909) UID: 0 2024-09-11T09:51:44.086 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:51:44.087+0000 7f3c96ec9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31909) UID: 0 2024-09-11T09:51:44.112 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:51:44.112 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:51:44.121 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-11T09:51:44.187 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:51:44.188+0000 7f3c96ec9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31909) UID: 0 2024-09-11T09:51:44.229 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:51:44.229 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:51:44.238 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-11T09:51:44.288 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:51:44.288+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:51:44.345 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:51:44.345 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:51:44.354 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-11T09:51:44.398 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-09-11T09:51:44.399+0000 7f7b6c7d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31904) UID: 0 2024-09-11T09:51:44.461 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:51:44.461 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:51:44.469 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-11T09:51:44.481 INFO:tasks.workunit.client.0.smithi107.stderr:+ DEV= 2024-09-11T09:51:44.482 INFO:tasks.workunit.client.0.smithi107.stderr:++ _sudo rbd device --device-type nbd map --pool rbd --namespace ns --image testrbdnbd27121 2024-09-11T09:51:44.482 INFO:tasks.workunit.client.0.smithi107.stderr:++ local cmd 2024-09-11T09:51:44.482 INFO:tasks.workunit.client.0.smithi107.stderr:+++ id -u 2024-09-11T09:51:44.483 INFO:tasks.workunit.client.0.smithi107.stderr:++ '[' 1000 -eq 0 ']' 2024-09-11T09:51:44.483 INFO:tasks.workunit.client.0.smithi107.stderr:+++ which rbd 2024-09-11T09:51:44.485 INFO:tasks.workunit.client.0.smithi107.stderr:++ cmd=/usr/bin/rbd 2024-09-11T09:51:44.485 INFO:tasks.workunit.client.0.smithi107.stderr:++ shift 2024-09-11T09:51:44.485 INFO:tasks.workunit.client.0.smithi107.stderr:++ sudo -nE /usr/bin/rbd device --device-type nbd map --pool rbd --namespace ns --image testrbdnbd27121 2024-09-11T09:51:44.510 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:51:44.511+0000 7f3c96ec9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31909) UID: 0 2024-09-11T09:51:44.591 INFO:tasks.ceph.osd.2.smithi017.stderr:2024-09-11T09:51:44.591+0000 7fd20b170640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31672) UID: 0 2024-09-11T09:51:44.596 INFO:tasks.workunit.client.0.smithi107.stderr:+ DEV=/dev/nbd0 2024-09-11T09:51:44.596 INFO:tasks.workunit.client.0.smithi107.stderr:+ get_pid rbd ns 2024-09-11T09:51:44.596 INFO:tasks.workunit.client.0.smithi107.stderr:+ local pool=rbd 2024-09-11T09:51:44.596 INFO:tasks.workunit.client.0.smithi107.stderr:+ local ns=ns 2024-09-11T09:51:44.596 INFO:tasks.workunit.client.0.smithi107.stderr:++ rbd device --device-type nbd --format xml list 2024-09-11T09:51:44.596 INFO:tasks.workunit.client.0.smithi107.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\''ns'\''][image='\''testrbdnbd27121'\''][device='\''/dev/nbd0'\'']/id' 2024-09-11T09:51:44.614 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:51:44.615 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:51:44.624 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-11T09:51:44.655 INFO:tasks.workunit.client.0.smithi107.stderr:+ PID=28793 2024-09-11T09:51:44.655 INFO:tasks.workunit.client.0.smithi107.stderr:+ test -n 28793 2024-09-11T09:51:44.656 INFO:tasks.workunit.client.0.smithi107.stderr:+ ps -p 28793 -C rbd-nbd 2024-09-11T09:51:44.666 INFO:tasks.workunit.client.0.smithi107.stdout: PID TTY TIME CMD 2024-09-11T09:51:44.666 INFO:tasks.workunit.client.0.smithi107.stdout: 28793 ? 00:00:00 rbd-nbd 2024-09-11T09:51:44.667 INFO:tasks.workunit.client.0.smithi107.stderr:+ unmap_device '--pool rbd --namespace ns --image testrbdnbd27121' 28793 2024-09-11T09:51:44.667 INFO:tasks.workunit.client.0.smithi107.stderr:+ local 'args=--pool rbd --namespace ns --image testrbdnbd27121' 2024-09-11T09:51:44.667 INFO:tasks.workunit.client.0.smithi107.stderr:+ local pid=28793 2024-09-11T09:51:44.667 INFO:tasks.workunit.client.0.smithi107.stderr:+ _sudo rbd device --device-type nbd unmap --pool rbd --namespace ns --image testrbdnbd27121 2024-09-11T09:51:44.667 INFO:tasks.workunit.client.0.smithi107.stderr:+ local cmd 2024-09-11T09:51:44.667 INFO:tasks.workunit.client.0.smithi107.stderr:++ id -u 2024-09-11T09:51:44.668 INFO:tasks.workunit.client.0.smithi107.stderr:+ '[' 1000 -eq 0 ']' 2024-09-11T09:51:44.668 INFO:tasks.workunit.client.0.smithi107.stderr:++ which rbd 2024-09-11T09:51:44.670 INFO:tasks.workunit.client.0.smithi107.stderr:+ cmd=/usr/bin/rbd 2024-09-11T09:51:44.670 INFO:tasks.workunit.client.0.smithi107.stderr:+ shift 2024-09-11T09:51:44.670 INFO:tasks.workunit.client.0.smithi107.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd unmap --pool rbd --namespace ns --image testrbdnbd27121 2024-09-11T09:51:44.692 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:51:44.693+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:51:44.726 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:51:44.726 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:51:44.735 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-11T09:51:44.751 INFO:tasks.workunit.client.0.smithi107.stderr:+ rbd device --device-type nbd list 2024-09-11T09:51:44.751 INFO:tasks.workunit.client.0.smithi107.stderr:+ expect_false grep '^28793\b' 2024-09-11T09:51:44.751 INFO:tasks.workunit.client.0.smithi107.stderr:+ grep '^28793\b' 2024-09-11T09:51:44.779 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-11T09:51:44.800 INFO:tasks.workunit.client.0.smithi107.stderr:+ return 0 2024-09-11T09:51:44.800 INFO:tasks.workunit.client.0.smithi107.stderr:+ ps -C rbd-nbd 2024-09-11T09:51:44.800 INFO:tasks.workunit.client.0.smithi107.stderr:+ expect_false grep '^ *28793\b' 2024-09-11T09:51:44.800 INFO:tasks.workunit.client.0.smithi107.stderr:+ grep '^ *28793\b' 2024-09-11T09:51:44.811 INFO:tasks.workunit.client.0.smithi107.stderr:+ return 0 2024-09-11T09:51:44.811 INFO:tasks.workunit.client.0.smithi107.stderr:+ sleep 0.5 2024-09-11T09:51:44.823 INFO:tasks.ceph.osd.2.smithi017.stderr:2024-09-11T09:51:44.823+0000 7fd20b170640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31672) UID: 0 2024-09-11T09:51:44.839 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:51:44.839 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:51:44.847 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-11T09:51:44.894 INFO:tasks.ceph.osd.2.smithi017.stderr:2024-09-11T09:51:44.894+0000 7fd20b170640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31672) UID: 0 2024-09-11T09:51:44.951 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:51:44.951 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:51:44.960 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-11T09:51:45.005 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-09-11T09:51:45.006+0000 7f7b6c7d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31904) UID: 0 2024-09-11T09:51:45.063 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:51:45.063 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:51:45.072 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-11T09:51:45.116 INFO:tasks.ceph.osd.0.smithi017.stderr:2024-09-11T09:51:45.116+0000 7fb73c0f1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31665) UID: 0 2024-09-11T09:51:45.179 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:51:45.180 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:51:45.187 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T09:51:45.188 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-11T09:51:45.197 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:51:45.197+0000 7f3c96ec9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31909) UID: 0 2024-09-11T09:51:45.298 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-09-11T09:51:45.299+0000 7f7b6c7d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31904) UID: 0 2024-09-11T09:51:45.313 INFO:tasks.workunit.client.0.smithi107.stderr:++ _sudo rbd device --device-type nbd map --pool rbd --namespace ns --image testrbdnbd27121 --snap snap 2024-09-11T09:51:45.313 INFO:tasks.workunit.client.0.smithi107.stderr:++ local cmd 2024-09-11T09:51:45.313 INFO:tasks.workunit.client.0.smithi107.stderr:+++ id -u 2024-09-11T09:51:45.314 INFO:tasks.workunit.client.0.smithi107.stderr:++ '[' 1000 -eq 0 ']' 2024-09-11T09:51:45.314 INFO:tasks.workunit.client.0.smithi107.stderr:+++ which rbd 2024-09-11T09:51:45.316 INFO:tasks.workunit.client.0.smithi107.stderr:++ cmd=/usr/bin/rbd 2024-09-11T09:51:45.316 INFO:tasks.workunit.client.0.smithi107.stderr:++ shift 2024-09-11T09:51:45.316 INFO:tasks.workunit.client.0.smithi107.stderr:++ sudo -nE /usr/bin/rbd device --device-type nbd map --pool rbd --namespace ns --image testrbdnbd27121 --snap snap 2024-09-11T09:51:45.338 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:51:45.338 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:51:45.347 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-11T09:51:45.353 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-11T09:51:45.353 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T09:51:45.373 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-11T09:51:45.373 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T09:51:45.389 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-11T09:51:45.390 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T09:51:45.399 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-09-11T09:51:45.400+0000 7f7b6c7d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31904) UID: 0 2024-09-11T09:51:45.407 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-11T09:51:45.408 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T09:51:45.427 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-11T09:51:45.427 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_enable_op_tracker = '' osd_max_backfills = '' osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_sleep = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' 2024-09-11T09:51:45.432 INFO:tasks.workunit.client.0.smithi107.stderr:+ DEV=/dev/nbd0 2024-09-11T09:51:45.432 INFO:tasks.workunit.client.0.smithi107.stderr:+ get_pid rbd ns 2024-09-11T09:51:45.432 INFO:tasks.workunit.client.0.smithi107.stderr:+ local pool=rbd 2024-09-11T09:51:45.432 INFO:tasks.workunit.client.0.smithi107.stderr:+ local ns=ns 2024-09-11T09:51:45.432 INFO:tasks.workunit.client.0.smithi107.stderr:++ rbd device --device-type nbd --format xml list 2024-09-11T09:51:45.432 INFO:tasks.workunit.client.0.smithi107.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\''ns'\''][image='\''testrbdnbd27121'\''][device='\''/dev/nbd0'\'']/id' 2024-09-11T09:51:45.445 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-11T09:51:45.445 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T09:51:45.461 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:51:45.462 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 20, 2024-09-11T09:51:45.462 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 600, 2024-09-11T09:51:45.462 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [ 2024-09-11T09:51:45.462 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.462 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4688.0:34 2.4 e53/32 2:2367385a:::rbd_data.1250f5e25b80.0000000000000002:head v 53'6, mlcod=53'4)", 2024-09-11T09:51:45.462 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.907415+0000", 2024-09-11T09:51:45.462 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 22.553812445999998, 2024-09-11T09:51:45.462 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.011285778999999999, 2024-09-11T09:51:45.462 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:45.462 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:45.462 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:45.462 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:45.462 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.463 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:45.463 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.907415+0000", 2024-09-11T09:51:45.463 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:45.463 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.463 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.463 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:45.463 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.907415+0000", 2024-09-11T09:51:45.463 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:45.463 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.463 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.463 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:45.463 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.907417+0000", 2024-09-11T09:51:45.463 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.2910000000000002e-06 2024-09-11T09:51:45.464 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.464 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.464 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:45.464 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.910477+0000", 2024-09-11T09:51:45.464 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0030597829999999999 2024-09-11T09:51:45.464 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.464 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.464 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:45.464 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.910501+0000", 2024-09-11T09:51:45.464 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.3332e-05 2024-09-11T09:51:45.464 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.464 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.464 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:45.464 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.910514+0000", 2024-09-11T09:51:45.464 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3957e-05 2024-09-11T09:51:45.465 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.465 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.465 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:45.465 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.910612+0000", 2024-09-11T09:51:45.465 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.7075000000000001e-05 2024-09-11T09:51:45.465 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.465 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.465 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:45.465 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.910640+0000", 2024-09-11T09:51:45.465 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.8768999999999998e-05 2024-09-11T09:51:45.465 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.465 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.465 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:45.466 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.918640+0000", 2024-09-11T09:51:45.466 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0079991899999999998 2024-09-11T09:51:45.466 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.466 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.466 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:45.466 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.918701+0000", 2024-09-11T09:51:45.466 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.1382000000000003e-05 2024-09-11T09:51:45.466 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:45.466 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:45.466 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:45.466 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.466 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.466 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4688.0:38 2.5 e53/32 2:abde2694:::rbd_data.1250f5e25b80.0000000000000004:head v 53'1, mlcod=0'0)", 2024-09-11T09:51:45.467 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.915432+0000", 2024-09-11T09:51:45.467 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 22.545796066000001, 2024-09-11T09:51:45.467 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.015849652999999998, 2024-09-11T09:51:45.467 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:45.467 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:45.467 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:45.467 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:45.467 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.467 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:45.467 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.915432+0000", 2024-09-11T09:51:45.467 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:45.467 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.467 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.468 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:45.468 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.915432+0000", 2024-09-11T09:51:45.468 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:45.468 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.468 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.468 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:45.468 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.915432+0000", 2024-09-11T09:51:45.468 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.91e-07 2024-09-11T09:51:45.468 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.468 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.468 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:45.468 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.924943+0000", 2024-09-11T09:51:45.468 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0095110630000000002 2024-09-11T09:51:45.469 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.469 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.469 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:45.469 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.924960+0000", 2024-09-11T09:51:45.469 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6915999999999999e-05 2024-09-11T09:51:45.469 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.469 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.469 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:45.469 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.924968+0000", 2024-09-11T09:51:45.469 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.6999999999999997e-06 2024-09-11T09:51:45.469 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.469 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.469 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:45.470 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.925025+0000", 2024-09-11T09:51:45.470 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.6949e-05 2024-09-11T09:51:45.470 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.470 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.470 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:45.470 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.925043+0000", 2024-09-11T09:51:45.470 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7784000000000001e-05 2024-09-11T09:51:45.470 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.470 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.470 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:45.470 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.931237+0000", 2024-09-11T09:51:45.470 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0061935389999999996 2024-09-11T09:51:45.470 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.471 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.471 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:45.471 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.931281+0000", 2024-09-11T09:51:45.471 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.4511000000000001e-05 2024-09-11T09:51:45.471 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:45.471 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:45.471 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:45.471 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.471 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.471 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:40 2.7 2:fe41411c:::rbd_object_map.1250f5e25b80:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:45.471 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.919652+0000", 2024-09-11T09:51:45.471 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 22.541575398999999, 2024-09-11T09:51:45.471 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.012187073, 2024-09-11T09:51:45.472 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:45.472 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:45.472 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:45.472 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:45.472 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:51:45.472 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:51:45.472 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 40 2024-09-11T09:51:45.472 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.472 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:45.472 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.472 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:45.472 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919652+0000", 2024-09-11T09:51:45.472 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:45.473 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.473 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.473 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:45.473 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919652+0000", 2024-09-11T09:51:45.473 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:45.473 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.473 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.473 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:45.473 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919654+0000", 2024-09-11T09:51:45.473 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.88e-06 2024-09-11T09:51:45.473 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.473 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.473 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:45.473 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919669+0000", 2024-09-11T09:51:45.474 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.5271e-05 2024-09-11T09:51:45.474 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.474 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.474 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:45.474 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919687+0000", 2024-09-11T09:51:45.474 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7448999999999998e-05 2024-09-11T09:51:45.474 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.474 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.474 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:45.474 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919695+0000", 2024-09-11T09:51:45.474 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.1149999999999994e-06 2024-09-11T09:51:45.474 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.474 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.475 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:45.475 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919757+0000", 2024-09-11T09:51:45.475 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.2348999999999995e-05 2024-09-11T09:51:45.475 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.475 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.475 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:45.475 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919875+0000", 2024-09-11T09:51:45.475 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00011750499999999999 2024-09-11T09:51:45.475 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.475 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.475 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 5", 2024-09-11T09:51:45.475 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.920201+0000", 2024-09-11T09:51:45.475 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00032628099999999999 2024-09-11T09:51:45.476 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.476 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.476 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:45.476 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.922089+0000", 2024-09-11T09:51:45.476 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.001887978 2024-09-11T09:51:45.476 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.476 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.476 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:45.476 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.931752+0000", 2024-09-11T09:51:45.476 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0096626309999999997 2024-09-11T09:51:45.476 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.476 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.477 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:45.477 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.931794+0000", 2024-09-11T09:51:45.477 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.2849999999999998e-05 2024-09-11T09:51:45.477 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.477 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.477 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:45.477 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.931839+0000", 2024-09-11T09:51:45.477 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.4764e-05 2024-09-11T09:51:45.477 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:45.477 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:45.477 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:45.477 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.477 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.478 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:41 2.7 2:fe41411c:::rbd_object_map.1250f5e25b80:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:45.478 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.919716+0000", 2024-09-11T09:51:45.478 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 22.541511555, 2024-09-11T09:51:45.478 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.015641605999999999, 2024-09-11T09:51:45.478 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:45.478 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:45.478 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:45.478 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:45.478 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:51:45.478 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:51:45.478 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 41 2024-09-11T09:51:45.478 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.478 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:45.478 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.479 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:45.479 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919716+0000", 2024-09-11T09:51:45.479 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:45.479 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.479 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.479 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:45.479 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919716+0000", 2024-09-11T09:51:45.479 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:45.479 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.479 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.479 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:45.479 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919716+0000", 2024-09-11T09:51:45.479 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.9999999999999998e-07 2024-09-11T09:51:45.480 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.480 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.480 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:45.480 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919720+0000", 2024-09-11T09:51:45.480 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.6440000000000002e-06 2024-09-11T09:51:45.480 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.480 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.480 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:45.480 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919729+0000", 2024-09-11T09:51:45.480 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.1759999999999999e-06 2024-09-11T09:51:45.480 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.480 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.481 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:45.481 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919736+0000", 2024-09-11T09:51:45.481 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.5390000000000004e-06 2024-09-11T09:51:45.481 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.481 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.481 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:45.481 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.920669+0000", 2024-09-11T09:51:45.481 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00093275599999999997 2024-09-11T09:51:45.481 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.481 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.481 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:45.481 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.920797+0000", 2024-09-11T09:51:45.481 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00012893400000000001 2024-09-11T09:51:45.481 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.482 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.482 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 5", 2024-09-11T09:51:45.482 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.921269+0000", 2024-09-11T09:51:45.482 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00047128600000000001 2024-09-11T09:51:45.482 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.482 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.482 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:45.482 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.923198+0000", 2024-09-11T09:51:45.482 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0019289369999999999 2024-09-11T09:51:45.482 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.482 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.482 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:45.482 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.935300+0000", 2024-09-11T09:51:45.483 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.012102392 2024-09-11T09:51:45.483 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.483 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.483 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:45.483 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.935332+0000", 2024-09-11T09:51:45.483 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.2292000000000002e-05 2024-09-11T09:51:45.483 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.483 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.483 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:45.483 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.935358+0000", 2024-09-11T09:51:45.483 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.525e-05 2024-09-11T09:51:45.483 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:45.484 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:45.484 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:45.484 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.484 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.484 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:48 2.7 2:fe41411c:::rbd_object_map.1250f5e25b80:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:45.484 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.938799+0000", 2024-09-11T09:51:45.484 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 22.522428445999999, 2024-09-11T09:51:45.484 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0041642340000000002, 2024-09-11T09:51:45.484 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:45.484 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:45.484 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:45.484 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:45.485 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:51:45.485 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:51:45.485 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 48 2024-09-11T09:51:45.485 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.485 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:45.485 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.485 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:45.485 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.938799+0000", 2024-09-11T09:51:45.485 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:45.485 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.485 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.485 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:45.486 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.938799+0000", 2024-09-11T09:51:45.486 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:45.486 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.486 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.486 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:45.486 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.938800+0000", 2024-09-11T09:51:45.486 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.71e-07 2024-09-11T09:51:45.486 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.486 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.486 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:45.486 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.938805+0000", 2024-09-11T09:51:45.486 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.3140000000000003e-06 2024-09-11T09:51:45.486 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.486 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.486 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:45.487 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.938816+0000", 2024-09-11T09:51:45.487 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0231000000000001e-05 2024-09-11T09:51:45.487 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.487 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.487 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:45.487 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.938822+0000", 2024-09-11T09:51:45.487 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.9550000000000003e-06 2024-09-11T09:51:45.487 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.487 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.487 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:45.487 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.938868+0000", 2024-09-11T09:51:45.487 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.5562999999999997e-05 2024-09-11T09:51:45.487 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.488 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.488 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:45.488 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.938918+0000", 2024-09-11T09:51:45.488 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.0317e-05 2024-09-11T09:51:45.488 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.488 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.488 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 5", 2024-09-11T09:51:45.488 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.939114+0000", 2024-09-11T09:51:45.488 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00019538300000000001 2024-09-11T09:51:45.488 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.488 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.488 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:45.489 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.939663+0000", 2024-09-11T09:51:45.489 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00054932199999999998 2024-09-11T09:51:45.489 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.489 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.489 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:45.489 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.942910+0000", 2024-09-11T09:51:45.489 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0032473839999999999 2024-09-11T09:51:45.489 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.489 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.489 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:45.489 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.942940+0000", 2024-09-11T09:51:45.489 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.9306999999999999e-05 2024-09-11T09:51:45.489 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.489 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.490 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:45.490 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.942963+0000", 2024-09-11T09:51:45.490 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.3587e-05 2024-09-11T09:51:45.490 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:45.490 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:45.490 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:45.490 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.490 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.490 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4688.0:49 2.5 e53/32 2:b6dd9fc9:::rbd_data.1250f5e25b80.0000000000000009:head v 53'2, mlcod=0'0)", 2024-09-11T09:51:45.490 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.944712+0000", 2024-09-11T09:51:45.490 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 22.516515606999999, 2024-09-11T09:51:45.490 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.016185069999999999, 2024-09-11T09:51:45.490 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:45.491 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:45.491 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:45.491 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:45.491 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.491 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:45.491 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.944712+0000", 2024-09-11T09:51:45.491 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:45.491 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.491 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.491 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:45.491 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.944712+0000", 2024-09-11T09:51:45.491 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:45.492 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.492 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.492 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:45.492 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.944713+0000", 2024-09-11T09:51:45.492 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0139999999999999e-06 2024-09-11T09:51:45.492 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.492 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.492 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:45.492 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.955518+0000", 2024-09-11T09:51:45.492 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.01080542 2024-09-11T09:51:45.492 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.492 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.493 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:45.493 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.955536+0000", 2024-09-11T09:51:45.493 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7818000000000001e-05 2024-09-11T09:51:45.493 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.493 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.493 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:45.493 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.955544+0000", 2024-09-11T09:51:45.493 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.1699999999999997e-06 2024-09-11T09:51:45.493 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.493 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.493 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:45.493 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.955607+0000", 2024-09-11T09:51:45.494 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.2451000000000002e-05 2024-09-11T09:51:45.494 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.494 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.494 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:45.494 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.955632+0000", 2024-09-11T09:51:45.494 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.5335000000000001e-05 2024-09-11T09:51:45.494 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.494 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.494 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:45.494 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.960826+0000", 2024-09-11T09:51:45.494 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0051938560000000002 2024-09-11T09:51:45.494 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.494 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.495 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:45.495 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.960897+0000", 2024-09-11T09:51:45.495 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.1006000000000004e-05 2024-09-11T09:51:45.495 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:45.495 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:45.495 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:45.495 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.495 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.495 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:53 2.7 2:fe41411c:::rbd_object_map.1250f5e25b80:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:45.495 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.951291+0000", 2024-09-11T09:51:45.495 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 22.509936617000001, 2024-09-11T09:51:45.495 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0044964999999999996, 2024-09-11T09:51:45.495 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:45.496 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:45.496 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:45.496 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:45.496 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:51:45.496 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:51:45.496 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 53 2024-09-11T09:51:45.496 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.496 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:45.496 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.496 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:45.496 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951291+0000", 2024-09-11T09:51:45.496 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:45.496 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.497 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.497 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:45.497 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951291+0000", 2024-09-11T09:51:45.497 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:45.497 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.497 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.497 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:45.497 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951293+0000", 2024-09-11T09:51:45.497 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7889999999999999e-06 2024-09-11T09:51:45.497 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.497 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.497 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:45.497 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951302+0000", 2024-09-11T09:51:45.498 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.9819999999999997e-06 2024-09-11T09:51:45.498 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.498 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.498 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:45.498 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951324+0000", 2024-09-11T09:51:45.498 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.2141e-05 2024-09-11T09:51:45.498 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.498 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.498 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:45.498 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951338+0000", 2024-09-11T09:51:45.498 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3750999999999999e-05 2024-09-11T09:51:45.498 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.498 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.499 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:45.499 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951409+0000", 2024-09-11T09:51:45.499 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.1658999999999999e-05 2024-09-11T09:51:45.499 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.499 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.499 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:45.499 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951517+0000", 2024-09-11T09:51:45.499 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00010746799999999999 2024-09-11T09:51:45.499 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.499 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.499 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 5", 2024-09-11T09:51:45.499 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951942+0000", 2024-09-11T09:51:45.499 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00042524300000000001 2024-09-11T09:51:45.499 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.500 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.500 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:45.500 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.952872+0000", 2024-09-11T09:51:45.500 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00092976300000000001 2024-09-11T09:51:45.500 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.500 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.500 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:45.500 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.955682+0000", 2024-09-11T09:51:45.500 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0028099459999999998 2024-09-11T09:51:45.500 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.500 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.501 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:45.501 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.955746+0000", 2024-09-11T09:51:45.501 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.4084999999999994e-05 2024-09-11T09:51:45.505 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.505 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.505 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:45.505 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.955787+0000", 2024-09-11T09:51:45.505 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.1672999999999999e-05 2024-09-11T09:51:45.505 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:45.505 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:45.505 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:45.505 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.506 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.506 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4688.0:52 2.4 e53/32 2:26d5c2f0:::rbd_data.1250f5e25b80.000000000000000b:head v 53'8, mlcod=53'6)", 2024-09-11T09:51:45.506 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.963677+0000", 2024-09-11T09:51:45.506 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 22.497551061999999, 2024-09-11T09:51:45.506 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0089437969999999999, 2024-09-11T09:51:45.506 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:45.506 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:45.506 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:45.506 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:45.506 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.506 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:45.506 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.963677+0000", 2024-09-11T09:51:45.507 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:45.507 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.507 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.507 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:45.507 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.963677+0000", 2024-09-11T09:51:45.507 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:45.507 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.507 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.507 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:45.507 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.963677+0000", 2024-09-11T09:51:45.507 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.7199999999999998e-07 2024-09-11T09:51:45.507 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.507 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.507 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:45.508 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.965506+0000", 2024-09-11T09:51:45.508 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.001829064 2024-09-11T09:51:45.508 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.508 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.508 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:45.508 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.965538+0000", 2024-09-11T09:51:45.508 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.1306999999999997e-05 2024-09-11T09:51:45.508 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.508 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.508 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:45.508 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.965553+0000", 2024-09-11T09:51:45.508 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.4924e-05 2024-09-11T09:51:45.509 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.509 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.509 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:45.509 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.965663+0000", 2024-09-11T09:51:45.509 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000110315 2024-09-11T09:51:45.509 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.509 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.509 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:45.510 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.965692+0000", 2024-09-11T09:51:45.510 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.934e-05 2024-09-11T09:51:45.510 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.510 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.510 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:45.510 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.972537+0000", 2024-09-11T09:51:45.510 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0068444969999999997 2024-09-11T09:51:45.510 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.510 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.510 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:45.510 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.972620+0000", 2024-09-11T09:51:45.510 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.3578000000000005e-05 2024-09-11T09:51:45.510 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:45.511 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:45.511 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:45.511 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.511 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.511 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4688.0:54 2.4 e53/32 2:239e915b:::rbd_data.1250f5e25b80.000000000000000c:head v 53'9, mlcod=53'7)", 2024-09-11T09:51:45.511 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.987028+0000", 2024-09-11T09:51:45.511 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 22.474199145, 2024-09-11T09:51:45.511 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0094155650000000007, 2024-09-11T09:51:45.511 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:45.511 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:45.511 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:45.511 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:45.511 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.512 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:45.512 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.987028+0000", 2024-09-11T09:51:45.512 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:45.512 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.512 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.512 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:45.512 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.987028+0000", 2024-09-11T09:51:45.512 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:45.512 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.512 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.512 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:45.512 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.987030+0000", 2024-09-11T09:51:45.512 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6130000000000001e-06 2024-09-11T09:51:45.513 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.513 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.513 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:45.513 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.988575+0000", 2024-09-11T09:51:45.513 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.001545193 2024-09-11T09:51:45.513 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.513 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.513 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:45.513 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.988604+0000", 2024-09-11T09:51:45.513 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.9173e-05 2024-09-11T09:51:45.513 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.514 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.517 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:45.517 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.988620+0000", 2024-09-11T09:51:45.518 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.5517999999999999e-05 2024-09-11T09:51:45.518 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.518 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.518 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:45.518 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.988704+0000", 2024-09-11T09:51:45.518 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.3588e-05 2024-09-11T09:51:45.518 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.518 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.518 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:45.518 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.988735+0000", 2024-09-11T09:51:45.518 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.1643000000000002e-05 2024-09-11T09:51:45.518 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.519 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.519 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:45.519 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.996383+0000", 2024-09-11T09:51:45.519 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0076478099999999997 2024-09-11T09:51:45.519 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.519 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.519 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:45.519 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.996444+0000", 2024-09-11T09:51:45.519 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.1026999999999997e-05 2024-09-11T09:51:45.519 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:45.519 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:45.519 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:45.519 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.520 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.520 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4688.0:56 2.4 e53/32 2:2167012f:::rbd_data.1250f5e25b80.000000000000000d:head v 53'10, mlcod=53'7)", 2024-09-11T09:51:45.520 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.993894+0000", 2024-09-11T09:51:45.520 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 22.467333603, 2024-09-11T09:51:45.520 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0099268559999999995, 2024-09-11T09:51:45.520 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:45.520 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:45.520 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:45.520 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:45.520 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.520 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:45.520 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.993894+0000", 2024-09-11T09:51:45.521 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:45.521 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.521 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.521 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:45.521 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.993894+0000", 2024-09-11T09:51:45.521 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:45.521 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.521 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.521 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:45.521 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.993896+0000", 2024-09-11T09:51:45.521 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6249999999999999e-06 2024-09-11T09:51:45.521 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.522 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.522 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:45.522 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.997606+0000", 2024-09-11T09:51:45.522 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.003710737 2024-09-11T09:51:45.522 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.522 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.522 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:45.522 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.997628+0000", 2024-09-11T09:51:45.522 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.1971e-05 2024-09-11T09:51:45.522 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.522 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.522 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:45.522 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.997640+0000", 2024-09-11T09:51:45.523 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1481000000000001e-05 2024-09-11T09:51:45.523 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.523 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.523 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:45.523 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.997712+0000", 2024-09-11T09:51:45.523 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.2255000000000005e-05 2024-09-11T09:51:45.523 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.523 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.523 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:45.523 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.997738+0000", 2024-09-11T09:51:45.523 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.6050999999999998e-05 2024-09-11T09:51:45.523 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.524 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.524 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:45.524 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.003779+0000", 2024-09-11T09:51:45.524 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0060410960000000001 2024-09-11T09:51:45.524 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.524 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.524 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:45.524 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.003821+0000", 2024-09-11T09:51:45.524 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.1640000000000001e-05 2024-09-11T09:51:45.524 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:45.524 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:45.524 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:45.524 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.524 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.525 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "pg_update_log_missing(2.4 epoch 53/32 rep_tid 16 entries 53'11 (0'0) error 2:2906e6a7:::rbd_header.1250f5e25b80:head by client.4688.0:59 0.000000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 53'7)", 2024-09-11T09:51:45.525 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.998956+0000", 2024-09-11T09:51:45.525 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 22.462271434000002, 2024-09-11T09:51:45.525 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.004934127, 2024-09-11T09:51:45.525 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:45.525 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:45.525 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "reached pg", 2024-09-11T09:51:45.525 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:45.525 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.525 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:45.525 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.998956+0000", 2024-09-11T09:51:45.525 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:45.526 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.526 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.526 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:45.526 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.998956+0000", 2024-09-11T09:51:45.526 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:45.526 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.526 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.526 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:45.526 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.998957+0000", 2024-09-11T09:51:45.526 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.139e-06 2024-09-11T09:51:45.526 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.526 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.526 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:45.526 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.998985+0000", 2024-09-11T09:51:45.527 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.7912e-05 2024-09-11T09:51:45.527 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.527 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.527 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:45.527 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.999011+0000", 2024-09-11T09:51:45.527 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.6061e-05 2024-09-11T09:51:45.527 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.527 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.527 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:45.527 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.999028+0000", 2024-09-11T09:51:45.527 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7065999999999999e-05 2024-09-11T09:51:45.527 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.527 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.528 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:45.528 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.001919+0000", 2024-09-11T09:51:45.528 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.002891167 2024-09-11T09:51:45.528 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.528 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.528 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:45.528 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.003890+0000", 2024-09-11T09:51:45.528 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0019707819999999999 2024-09-11T09:51:45.528 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:45.528 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:45.528 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:45.528 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.528 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.528 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4694.0:34 2.7 2:f80bfc0c:ns::rbd_data.1256a0c02879.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:45.529 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.085183+0000", 2024-09-11T09:51:45.529 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 22.376044920999998, 2024-09-11T09:51:45.529 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.020908599, 2024-09-11T09:51:45.529 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:45.529 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:45.529 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:45.529 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:45.529 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4694", 2024-09-11T09:51:45.529 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:51:45.529 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 34 2024-09-11T09:51:45.529 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.529 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:45.530 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.530 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:45.530 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.085183+0000", 2024-09-11T09:51:45.530 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:45.530 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.530 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.530 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:45.530 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.085183+0000", 2024-09-11T09:51:45.530 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:45.530 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.530 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.530 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:45.530 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.085183+0000", 2024-09-11T09:51:45.531 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.4300000000000002e-07 2024-09-11T09:51:45.531 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.531 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.531 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:45.531 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094337+0000", 2024-09-11T09:51:45.531 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0091532769999999996 2024-09-11T09:51:45.531 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.531 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.531 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:45.531 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094362+0000", 2024-09-11T09:51:45.531 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.5287999999999999e-05 2024-09-11T09:51:45.531 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.531 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.532 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:45.532 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094375+0000", 2024-09-11T09:51:45.532 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2446000000000001e-05 2024-09-11T09:51:45.532 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.532 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.532 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:45.532 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094455+0000", 2024-09-11T09:51:45.532 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.0092999999999994e-05 2024-09-11T09:51:45.532 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.532 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.532 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:45.532 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094618+0000", 2024-09-11T09:51:45.532 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000163747 2024-09-11T09:51:45.533 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.533 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.533 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 5", 2024-09-11T09:51:45.533 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094783+0000", 2024-09-11T09:51:45.533 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000165038 2024-09-11T09:51:45.533 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.533 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.533 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:45.533 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.105853+0000", 2024-09-11T09:51:45.533 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.011069294 2024-09-11T09:51:45.533 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.533 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.533 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:45.533 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.106004+0000", 2024-09-11T09:51:45.534 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00015123900000000001 2024-09-11T09:51:45.534 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.534 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.534 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:45.534 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.106062+0000", 2024-09-11T09:51:45.534 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.8205000000000003e-05 2024-09-11T09:51:45.534 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.534 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.534 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:45.534 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.106091+0000", 2024-09-11T09:51:45.534 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.9128999999999998e-05 2024-09-11T09:51:45.534 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:45.534 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:45.534 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:45.535 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.535 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.535 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4694.0:35 2.7 2:e8766325:ns::rbd_object_map.1256a0c02879:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:45.535 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.094405+0000", 2024-09-11T09:51:45.535 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 22.366822285000001, 2024-09-11T09:51:45.535 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.012623564, 2024-09-11T09:51:45.535 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:45.535 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:45.535 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:45.535 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:45.535 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4694", 2024-09-11T09:51:45.535 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:51:45.535 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 35 2024-09-11T09:51:45.536 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.536 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:45.536 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.536 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:45.536 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094405+0000", 2024-09-11T09:51:45.536 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:45.536 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.536 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.536 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:45.536 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094405+0000", 2024-09-11T09:51:45.536 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:45.536 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.536 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.537 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:45.537 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094407+0000", 2024-09-11T09:51:45.537 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.739e-06 2024-09-11T09:51:45.537 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.537 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.537 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:45.537 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094411+0000", 2024-09-11T09:51:45.537 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.3340000000000003e-06 2024-09-11T09:51:45.537 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.537 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.537 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:45.537 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094419+0000", 2024-09-11T09:51:45.537 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.8909999999999995e-06 2024-09-11T09:51:45.538 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.538 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.538 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:45.538 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094424+0000", 2024-09-11T09:51:45.538 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.7639999999999996e-06 2024-09-11T09:51:45.538 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.538 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.538 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:45.538 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.100209+0000", 2024-09-11T09:51:45.538 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0057850489999999996 2024-09-11T09:51:45.538 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.538 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.538 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:45.538 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.100341+0000", 2024-09-11T09:51:45.539 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000131438 2024-09-11T09:51:45.539 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.539 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.539 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 5", 2024-09-11T09:51:45.539 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.100805+0000", 2024-09-11T09:51:45.539 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00046471499999999998 2024-09-11T09:51:45.539 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.539 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.539 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:45.539 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.106148+0000", 2024-09-11T09:51:45.539 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0053426460000000004 2024-09-11T09:51:45.539 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.539 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.539 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:45.540 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.106929+0000", 2024-09-11T09:51:45.540 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00078161300000000001 2024-09-11T09:51:45.540 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.540 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.540 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:45.540 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.106985+0000", 2024-09-11T09:51:45.540 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.5223999999999998e-05 2024-09-11T09:51:45.540 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.540 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.540 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:45.540 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.107029+0000", 2024-09-11T09:51:45.540 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.4150999999999998e-05 2024-09-11T09:51:45.540 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:45.541 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:45.541 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:45.541 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.541 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.541 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4694.0:36 2.7 2:e8766325:ns::rbd_object_map.1256a0c02879:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:45.541 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.094452+0000", 2024-09-11T09:51:45.541 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 22.366775274999998, 2024-09-11T09:51:45.541 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.012680383, 2024-09-11T09:51:45.541 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:45.541 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:45.541 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:45.541 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:45.541 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4694", 2024-09-11T09:51:45.542 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:51:45.542 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 36 2024-09-11T09:51:45.542 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.542 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:45.542 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.542 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:45.542 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094452+0000", 2024-09-11T09:51:45.542 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:45.542 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.542 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.542 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:45.542 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094452+0000", 2024-09-11T09:51:45.542 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:45.542 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.543 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.543 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:45.543 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094453+0000", 2024-09-11T09:51:45.543 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.1800000000000002e-07 2024-09-11T09:51:45.543 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.543 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.543 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:45.543 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094455+0000", 2024-09-11T09:51:45.543 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.7020000000000001e-06 2024-09-11T09:51:45.543 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.543 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.543 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:45.543 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094463+0000", 2024-09-11T09:51:45.544 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.1999999999999997e-06 2024-09-11T09:51:45.544 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.544 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.544 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:45.544 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094467+0000", 2024-09-11T09:51:45.544 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.729e-06 2024-09-11T09:51:45.544 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.544 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.544 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:45.544 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.101269+0000", 2024-09-11T09:51:45.544 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0068018799999999997 2024-09-11T09:51:45.544 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.544 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.545 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:45.545 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.101353+0000", 2024-09-11T09:51:45.545 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.3827999999999998e-05 2024-09-11T09:51:45.545 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.545 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.545 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 5", 2024-09-11T09:51:45.545 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.101588+0000", 2024-09-11T09:51:45.545 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00023467099999999999 2024-09-11T09:51:45.545 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.545 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.545 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:45.545 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.106182+0000", 2024-09-11T09:51:45.545 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0045939489999999999 2024-09-11T09:51:45.546 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.546 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.546 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:45.546 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.107041+0000", 2024-09-11T09:51:45.546 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00085965800000000002 2024-09-11T09:51:45.546 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.546 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.546 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:45.546 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.107084+0000", 2024-09-11T09:51:45.546 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.261e-05 2024-09-11T09:51:45.546 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.546 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.546 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:45.546 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.107133+0000", 2024-09-11T09:51:45.547 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.8838000000000002e-05 2024-09-11T09:51:45.547 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:45.547 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:45.547 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:45.547 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.547 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.547 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4694.0:37 2.7 2:e8766325:ns::rbd_object_map.1256a0c02879:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:45.547 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.094473+0000", 2024-09-11T09:51:45.547 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 22.366754124, 2024-09-11T09:51:45.547 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.012778484999999999, 2024-09-11T09:51:45.547 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:45.547 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:45.547 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:45.548 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:45.548 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4694", 2024-09-11T09:51:45.548 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:51:45.548 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 37 2024-09-11T09:51:45.548 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.548 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:45.548 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.548 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:45.548 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094473+0000", 2024-09-11T09:51:45.548 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:45.548 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.548 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.548 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:45.549 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094473+0000", 2024-09-11T09:51:45.549 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:45.549 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.549 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.549 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:45.549 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094474+0000", 2024-09-11T09:51:45.549 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.9699999999999996e-07 2024-09-11T09:51:45.549 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.549 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.549 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:45.549 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094477+0000", 2024-09-11T09:51:45.549 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.151e-06 2024-09-11T09:51:45.549 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.550 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.550 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:45.550 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094483+0000", 2024-09-11T09:51:45.550 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.6330000000000001e-06 2024-09-11T09:51:45.550 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.550 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.550 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:45.550 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094487+0000", 2024-09-11T09:51:45.550 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.5399999999999997e-06 2024-09-11T09:51:45.550 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.550 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.550 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:45.550 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.101866+0000", 2024-09-11T09:51:45.550 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.007378768 2024-09-11T09:51:45.551 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.551 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.551 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:45.551 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.101960+0000", 2024-09-11T09:51:45.551 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.3861000000000001e-05 2024-09-11T09:51:45.551 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.551 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.551 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 5", 2024-09-11T09:51:45.551 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.102197+0000", 2024-09-11T09:51:45.551 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00023723200000000001 2024-09-11T09:51:45.551 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.551 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.551 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:45.552 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.106222+0000", 2024-09-11T09:51:45.552 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0040249689999999998 2024-09-11T09:51:45.552 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.552 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.552 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:45.552 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.107141+0000", 2024-09-11T09:51:45.552 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00091913900000000002 2024-09-11T09:51:45.552 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.552 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.552 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:45.552 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.107198+0000", 2024-09-11T09:51:45.552 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.6564999999999998e-05 2024-09-11T09:51:45.552 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.553 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.553 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:45.553 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.107252+0000", 2024-09-11T09:51:45.553 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.4030000000000003e-05 2024-09-11T09:51:45.553 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:45.553 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:45.553 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:45.553 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.553 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.553 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4694.0:33 2.4 e53/32 2:3bfc818c:ns::rbd_data.1256a0c02879.0000000000000001:head v 53'14, mlcod=53'12)", 2024-09-11T09:51:45.553 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.096496+0000", 2024-09-11T09:51:45.553 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 22.364731550999998, 2024-09-11T09:51:45.553 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0099139310000000008, 2024-09-11T09:51:45.554 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:45.554 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:45.554 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:45.554 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:45.554 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.554 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:45.554 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.096496+0000", 2024-09-11T09:51:45.554 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:45.554 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.554 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.554 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:45.554 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.096496+0000", 2024-09-11T09:51:45.554 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:45.554 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.555 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.555 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:45.555 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.096498+0000", 2024-09-11T09:51:45.555 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6309999999999999e-06 2024-09-11T09:51:45.555 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.555 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.555 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:45.555 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.098087+0000", 2024-09-11T09:51:45.555 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0015889019999999999 2024-09-11T09:51:45.555 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.555 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.555 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:45.555 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.098112+0000", 2024-09-11T09:51:45.556 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.5369000000000001e-05 2024-09-11T09:51:45.556 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.556 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.556 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:45.556 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.098127+0000", 2024-09-11T09:51:45.556 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.5218999999999999e-05 2024-09-11T09:51:45.556 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.556 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.556 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:45.556 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.098209+0000", 2024-09-11T09:51:45.556 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.2120999999999995e-05 2024-09-11T09:51:45.556 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.557 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.557 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:45.557 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.098237+0000", 2024-09-11T09:51:45.557 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.7657999999999998e-05 2024-09-11T09:51:45.557 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.557 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.557 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:45.557 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.106376+0000", 2024-09-11T09:51:45.557 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0081386749999999997 2024-09-11T09:51:45.557 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.557 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.557 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:45.557 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.106410+0000", 2024-09-11T09:51:45.558 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.4356000000000001e-05 2024-09-11T09:51:45.558 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:45.558 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:45.558 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:45.558 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.558 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.558 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4694.0:49 2.7 2:e8766325:ns::rbd_object_map.1256a0c02879:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:45.558 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.134806+0000", 2024-09-11T09:51:45.558 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 22.326421482000001, 2024-09-11T09:51:45.558 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0061629049999999998, 2024-09-11T09:51:45.558 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:45.558 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:45.558 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:45.559 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:45.559 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4694", 2024-09-11T09:51:45.559 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:51:45.559 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 49 2024-09-11T09:51:45.559 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.559 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:45.559 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.559 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:45.559 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.134806+0000", 2024-09-11T09:51:45.559 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:45.559 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.559 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.559 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:45.559 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.134806+0000", 2024-09-11T09:51:45.560 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:45.560 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.560 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.560 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:45.560 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.134808+0000", 2024-09-11T09:51:45.560 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.9870000000000002e-06 2024-09-11T09:51:45.560 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.560 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.560 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:45.560 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.134818+0000", 2024-09-11T09:51:45.560 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.5599999999999999e-06 2024-09-11T09:51:45.560 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.561 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.561 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:45.561 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.134835+0000", 2024-09-11T09:51:45.561 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7399000000000001e-05 2024-09-11T09:51:45.561 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.561 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.561 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:45.561 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.134845+0000", 2024-09-11T09:51:45.561 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0285000000000001e-05 2024-09-11T09:51:45.561 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.561 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.561 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:45.561 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.134911+0000", 2024-09-11T09:51:45.561 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.5654999999999994e-05 2024-09-11T09:51:45.562 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.562 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.562 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:45.562 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.135023+0000", 2024-09-11T09:51:45.562 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00011186799999999999 2024-09-11T09:51:45.562 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.562 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.562 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 5", 2024-09-11T09:51:45.562 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.135290+0000", 2024-09-11T09:51:45.562 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00026725300000000001 2024-09-11T09:51:45.562 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.562 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.562 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:45.563 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.136506+0000", 2024-09-11T09:51:45.563 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0012161139999999999 2024-09-11T09:51:45.563 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.563 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.563 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:45.563 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.140884+0000", 2024-09-11T09:51:45.563 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0043781990000000002 2024-09-11T09:51:45.563 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.563 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.563 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:45.563 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.140923+0000", 2024-09-11T09:51:45.563 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.8813999999999998e-05 2024-09-11T09:51:45.563 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.564 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.564 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:45.564 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.140969+0000", 2024-09-11T09:51:45.564 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.5770999999999999e-05 2024-09-11T09:51:45.564 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:45.564 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:45.564 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:45.564 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.564 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.564 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4694.0:42 2.5 e53/32 2:a4fc00ec:ns::rbd_data.1256a0c02879.0000000000000004:head v 53'3, mlcod=53'1)", 2024-09-11T09:51:45.564 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.136646+0000", 2024-09-11T09:51:45.564 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 22.324581175999999, 2024-09-11T09:51:45.564 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.012545501000000001, 2024-09-11T09:51:45.565 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:45.565 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:45.565 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:45.565 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:45.565 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.565 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:45.565 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.136646+0000", 2024-09-11T09:51:45.565 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:45.565 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.565 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.565 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:45.565 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.136646+0000", 2024-09-11T09:51:45.565 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:45.566 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.566 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.566 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:45.566 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.136647+0000", 2024-09-11T09:51:45.566 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.2900000000000002e-07 2024-09-11T09:51:45.566 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.566 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.566 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:45.566 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.143620+0000", 2024-09-11T09:51:45.566 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0069727440000000003 2024-09-11T09:51:45.566 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.566 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.566 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:45.567 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.143642+0000", 2024-09-11T09:51:45.567 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.1846000000000001e-05 2024-09-11T09:51:45.567 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.567 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.567 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:45.567 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.143654+0000", 2024-09-11T09:51:45.567 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2081e-05 2024-09-11T09:51:45.567 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.567 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.567 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:45.567 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.143717+0000", 2024-09-11T09:51:45.567 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.3443999999999998e-05 2024-09-11T09:51:45.567 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.567 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.568 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:45.568 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.143740+0000", 2024-09-11T09:51:45.568 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.2626999999999999e-05 2024-09-11T09:51:45.568 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.568 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.568 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:45.568 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.149142+0000", 2024-09-11T09:51:45.568 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0054017630000000004 2024-09-11T09:51:45.568 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.568 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.568 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:45.568 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.149192+0000", 2024-09-11T09:51:45.568 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.0067e-05 2024-09-11T09:51:45.569 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:45.569 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:45.569 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:45.569 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.569 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.569 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4694.0:53 2.7 2:e8766325:ns::rbd_object_map.1256a0c02879:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:45.569 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.147978+0000", 2024-09-11T09:51:45.569 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 22.313249691999999, 2024-09-11T09:51:45.569 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0044549769999999997, 2024-09-11T09:51:45.569 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:45.569 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:45.569 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:45.569 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:45.569 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4694", 2024-09-11T09:51:45.570 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:51:45.570 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 53 2024-09-11T09:51:45.570 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.570 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:45.570 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.570 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:45.570 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.147978+0000", 2024-09-11T09:51:45.570 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:45.570 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.570 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.570 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:45.570 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.147978+0000", 2024-09-11T09:51:45.570 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:45.570 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.571 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.571 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:45.571 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.147980+0000", 2024-09-11T09:51:45.571 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.8309999999999999e-06 2024-09-11T09:51:45.571 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.571 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.571 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:45.571 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.147989+0000", 2024-09-11T09:51:45.571 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.9639999999999992e-06 2024-09-11T09:51:45.571 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.571 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.571 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:45.571 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.148007+0000", 2024-09-11T09:51:45.572 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7884999999999999e-05 2024-09-11T09:51:45.572 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.572 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.572 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:45.572 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.148017+0000", 2024-09-11T09:51:45.572 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0895e-05 2024-09-11T09:51:45.572 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.572 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.572 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:45.572 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.148091+0000", 2024-09-11T09:51:45.572 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.3709999999999997e-05 2024-09-11T09:51:45.572 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.572 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.573 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:45.573 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.148189+0000", 2024-09-11T09:51:45.573 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.7403000000000002e-05 2024-09-11T09:51:45.573 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.573 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.573 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 5", 2024-09-11T09:51:45.573 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.148504+0000", 2024-09-11T09:51:45.573 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00031589900000000002 2024-09-11T09:51:45.573 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.573 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.573 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:45.573 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.149514+0000", 2024-09-11T09:51:45.573 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0010096 2024-09-11T09:51:45.574 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.574 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.574 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:45.574 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.152367+0000", 2024-09-11T09:51:45.574 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0028532309999999999 2024-09-11T09:51:45.574 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.574 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.574 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:45.574 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.152402+0000", 2024-09-11T09:51:45.574 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.4232e-05 2024-09-11T09:51:45.574 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.574 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.574 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:45.574 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.152433+0000", 2024-09-11T09:51:45.575 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.1327e-05 2024-09-11T09:51:45.575 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:45.575 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:45.575 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:45.575 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.575 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.575 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4694.0:55 2.7 2:e8766325:ns::rbd_object_map.1256a0c02879:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:45.575 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.149838+0000", 2024-09-11T09:51:45.575 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 22.311389959, 2024-09-11T09:51:45.575 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0060695159999999996, 2024-09-11T09:51:45.575 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:45.575 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:45.575 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:45.576 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:45.576 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4694", 2024-09-11T09:51:45.576 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:51:45.576 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 55 2024-09-11T09:51:45.576 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.576 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:45.576 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.576 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:45.576 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.149838+0000", 2024-09-11T09:51:45.576 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:45.576 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.576 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.576 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:45.576 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.149838+0000", 2024-09-11T09:51:45.577 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:45.577 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.577 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.577 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:45.577 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.149839+0000", 2024-09-11T09:51:45.577 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.119e-06 2024-09-11T09:51:45.577 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.577 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.577 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:45.577 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.149845+0000", 2024-09-11T09:51:45.577 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.7810000000000002e-06 2024-09-11T09:51:45.577 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.577 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.577 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:45.578 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.149856+0000", 2024-09-11T09:51:45.578 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1199999999999999e-05 2024-09-11T09:51:45.578 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.578 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.578 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:45.578 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.149863+0000", 2024-09-11T09:51:45.578 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.4109999999999999e-06 2024-09-11T09:51:45.578 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.578 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.578 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:45.578 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.149908+0000", 2024-09-11T09:51:45.578 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.5184e-05 2024-09-11T09:51:45.578 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.579 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.579 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:45.579 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.149964+0000", 2024-09-11T09:51:45.579 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.5395000000000001e-05 2024-09-11T09:51:45.579 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.579 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.579 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 5", 2024-09-11T09:51:45.579 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.150157+0000", 2024-09-11T09:51:45.579 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000193489 2024-09-11T09:51:45.579 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.579 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.579 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:45.579 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.152238+0000", 2024-09-11T09:51:45.579 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0020803480000000001 2024-09-11T09:51:45.580 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.580 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.580 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:45.580 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.155828+0000", 2024-09-11T09:51:45.580 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0035902210000000002 2024-09-11T09:51:45.580 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.580 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.580 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:45.580 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.155873+0000", 2024-09-11T09:51:45.580 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.5145999999999998e-05 2024-09-11T09:51:45.580 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.580 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.580 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:45.581 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.155907+0000", 2024-09-11T09:51:45.581 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.4221999999999999e-05 2024-09-11T09:51:45.581 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:45.581 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:45.581 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:45.581 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:45.581 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:45.581 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:51:45.582 INFO:tasks.workunit.client.0.smithi107.stderr:+ PID=28859 2024-09-11T09:51:45.583 INFO:tasks.workunit.client.0.smithi107.stderr:+ test -n 28859 2024-09-11T09:51:45.583 INFO:tasks.workunit.client.0.smithi107.stderr:+ ps -p 28859 -C rbd-nbd 2024-09-11T09:51:45.583 INFO:tasks.workunit.client.0.smithi107.stdout: PID TTY TIME CMD 2024-09-11T09:51:45.583 INFO:tasks.workunit.client.0.smithi107.stdout: 28859 ? 00:00:00 rbd-nbd 2024-09-11T09:51:45.584 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-11T09:51:45.585 INFO:tasks.workunit.client.0.smithi107.stderr:+ unmap_device '--pool rbd --namespace ns --image testrbdnbd27121 --snap snap' 28859 2024-09-11T09:51:45.585 INFO:tasks.workunit.client.0.smithi107.stderr:+ local 'args=--pool rbd --namespace ns --image testrbdnbd27121 --snap snap' 2024-09-11T09:51:45.585 INFO:tasks.workunit.client.0.smithi107.stderr:+ local pid=28859 2024-09-11T09:51:45.585 INFO:tasks.workunit.client.0.smithi107.stderr:+ _sudo rbd device --device-type nbd unmap --pool rbd --namespace ns --image testrbdnbd27121 --snap snap 2024-09-11T09:51:45.585 INFO:tasks.workunit.client.0.smithi107.stderr:+ local cmd 2024-09-11T09:51:45.585 INFO:tasks.workunit.client.0.smithi107.stderr:++ id -u 2024-09-11T09:51:45.585 INFO:tasks.workunit.client.0.smithi107.stderr:+ '[' 1000 -eq 0 ']' 2024-09-11T09:51:45.586 INFO:tasks.workunit.client.0.smithi107.stderr:++ which rbd 2024-09-11T09:51:45.586 INFO:tasks.workunit.client.0.smithi107.stderr:+ cmd=/usr/bin/rbd 2024-09-11T09:51:45.586 INFO:tasks.workunit.client.0.smithi107.stderr:+ shift 2024-09-11T09:51:45.586 INFO:tasks.workunit.client.0.smithi107.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd unmap --pool rbd --namespace ns --image testrbdnbd27121 --snap snap 2024-09-11T09:51:45.586 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-09-11T09:51:45.583+0000 7f7b6c7d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31904) UID: 0 2024-09-11T09:51:45.590 INFO:tasks.workunit.client.0.smithi107.stderr:+ rbd device --device-type nbd list 2024-09-11T09:51:45.590 INFO:tasks.workunit.client.0.smithi107.stderr:+ expect_false grep '^28859\b' 2024-09-11T09:51:45.590 INFO:tasks.workunit.client.0.smithi107.stderr:+ grep '^28859\b' 2024-09-11T09:51:45.636 INFO:tasks.workunit.client.0.smithi107.stderr:+ return 0 2024-09-11T09:51:45.637 INFO:tasks.workunit.client.0.smithi107.stderr:+ ps -C rbd-nbd 2024-09-11T09:51:45.637 INFO:tasks.workunit.client.0.smithi107.stderr:+ expect_false grep '^ *28859\b' 2024-09-11T09:51:45.637 INFO:tasks.workunit.client.0.smithi107.stderr:+ grep '^ *28859\b' 2024-09-11T09:51:45.648 INFO:tasks.workunit.client.0.smithi107.stderr:+ return 0 2024-09-11T09:51:45.648 INFO:tasks.workunit.client.0.smithi107.stderr:+ sleep 0.5 2024-09-11T09:51:45.683 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-09-11T09:51:45.684+0000 7f7b6c7d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31904) UID: 0 2024-09-11T09:51:45.732 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:51:45.733 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [], 2024-09-11T09:51:45.733 INFO:teuthology.orchestra.run.smithi017.stdout: "num_ops": 0 2024-09-11T09:51:45.733 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:51:45.741 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-11T09:51:45.769 INFO:teuthology.orchestra.run.smithi017.stderr:noscrub is unset 2024-09-11T09:51:45.784 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:51:45.785+0000 7f3c96ec9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31909) UID: 0 2024-09-11T09:51:45.851 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:51:45.852 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [], 2024-09-11T09:51:45.852 INFO:teuthology.orchestra.run.smithi017.stdout: "complaint_time": 30, 2024-09-11T09:51:45.852 INFO:teuthology.orchestra.run.smithi017.stdout: "num_blocked_ops": 0 2024-09-11T09:51:45.852 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:51:45.860 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-11T09:51:45.904 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:51:45.904+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:51:45.969 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:51:45.969 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 20, 2024-09-11T09:51:45.970 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 600, 2024-09-11T09:51:45.970 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [ 2024-09-11T09:51:45.970 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.970 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:34 2.4 2:2367385a:::rbd_data.1250f5e25b80.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:45.970 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.897717+0000", 2024-09-11T09:51:45.970 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 23.071549213000001, 2024-09-11T09:51:45.970 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.021323661000000001, 2024-09-11T09:51:45.970 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:45.970 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:45.970 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:45.970 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:45.970 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:51:45.971 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:51:45.971 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 34 2024-09-11T09:51:45.971 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.971 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:45.971 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.971 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:45.971 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.897717+0000", 2024-09-11T09:51:45.971 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:45.971 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.971 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.971 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:45.971 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.897717+0000", 2024-09-11T09:51:45.971 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:45.972 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.972 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.972 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:45.972 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.897718+0000", 2024-09-11T09:51:45.972 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.1800000000000004e-07 2024-09-11T09:51:45.972 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.972 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.972 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:45.972 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.905339+0000", 2024-09-11T09:51:45.972 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0076210239999999997 2024-09-11T09:51:45.972 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.972 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.973 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:45.973 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.905366+0000", 2024-09-11T09:51:45.973 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.6959000000000001e-05 2024-09-11T09:51:45.973 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.973 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.973 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:45.973 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.905375+0000", 2024-09-11T09:51:45.973 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.516e-06 2024-09-11T09:51:45.973 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.973 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.973 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:45.973 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.905475+0000", 2024-09-11T09:51:45.973 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.9738e-05 2024-09-11T09:51:45.974 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.974 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.974 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:45.974 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.905660+0000", 2024-09-11T09:51:45.974 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00018554199999999999 2024-09-11T09:51:45.974 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.974 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.974 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:45.974 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.905808+0000", 2024-09-11T09:51:45.974 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000147386 2024-09-11T09:51:45.974 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.974 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.974 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:45.974 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.918230+0000", 2024-09-11T09:51:45.975 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.012421690000000001 2024-09-11T09:51:45.975 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.975 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.975 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:45.975 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.918932+0000", 2024-09-11T09:51:45.975 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00070267300000000001 2024-09-11T09:51:45.975 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.975 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.975 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:45.975 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.918989+0000", 2024-09-11T09:51:45.975 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.7268999999999997e-05 2024-09-11T09:51:45.975 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.975 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.976 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:45.976 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919040+0000", 2024-09-11T09:51:45.976 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.0946000000000003e-05 2024-09-11T09:51:45.976 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:45.976 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:45.976 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:45.976 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.976 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.976 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4688.0:31 2.1 e53/15 2:8654721b:::rbd_data.1250f5e25b80.0000000000000000:head v 53'2, mlcod=0'0)", 2024-09-11T09:51:45.976 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.904661+0000", 2024-09-11T09:51:45.976 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 23.064604723999999, 2024-09-11T09:51:45.976 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.014152804999999999, 2024-09-11T09:51:45.976 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:45.977 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:45.977 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:45.977 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:45.977 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.977 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:45.977 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.904661+0000", 2024-09-11T09:51:45.977 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:45.977 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.977 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.977 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:45.977 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.904661+0000", 2024-09-11T09:51:45.977 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:45.978 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.978 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.978 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:45.978 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.904662+0000", 2024-09-11T09:51:45.978 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0759999999999999e-06 2024-09-11T09:51:45.978 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.978 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.978 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:45.978 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.907298+0000", 2024-09-11T09:51:45.978 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0026361240000000001 2024-09-11T09:51:45.978 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.978 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.978 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:45.978 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.907320+0000", 2024-09-11T09:51:45.979 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.1268999999999999e-05 2024-09-11T09:51:45.979 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.979 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.979 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:45.979 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.907331+0000", 2024-09-11T09:51:45.979 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1031e-05 2024-09-11T09:51:45.979 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.979 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.979 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:45.979 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.907403+0000", 2024-09-11T09:51:45.979 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.2571999999999996e-05 2024-09-11T09:51:45.979 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.979 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.980 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:45.980 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.907428+0000", 2024-09-11T09:51:45.980 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.4380999999999999e-05 2024-09-11T09:51:45.980 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.980 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.980 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:45.980 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.918750+0000", 2024-09-11T09:51:45.980 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.011322211 2024-09-11T09:51:45.980 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.980 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.980 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:45.980 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.918814+0000", 2024-09-11T09:51:45.980 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.4140999999999995e-05 2024-09-11T09:51:45.980 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:45.981 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:45.981 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:45.981 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.981 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.981 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:36 2.6 2:61e1b442:::rbd_data.1250f5e25b80.0000000000000003:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:45.981 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.905404+0000", 2024-09-11T09:51:45.981 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 23.063861852999999, 2024-09-11T09:51:45.981 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.030912479999999999, 2024-09-11T09:51:45.981 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:45.981 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:45.981 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:45.981 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:45.981 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:51:45.982 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:51:45.982 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 36 2024-09-11T09:51:45.982 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.982 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:45.982 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.982 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:45.982 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.905404+0000", 2024-09-11T09:51:45.982 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:45.982 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.982 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.982 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:45.982 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.905404+0000", 2024-09-11T09:51:45.982 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:45.982 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.983 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.983 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:45.983 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.905405+0000", 2024-09-11T09:51:45.983 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2270000000000001e-06 2024-09-11T09:51:45.983 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.983 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.983 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:45.983 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.914859+0000", 2024-09-11T09:51:45.983 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0094541190000000004 2024-09-11T09:51:45.983 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.983 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.983 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:45.983 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.914879+0000", 2024-09-11T09:51:45.984 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.0052999999999999e-05 2024-09-11T09:51:45.984 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.984 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.984 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:45.984 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.914897+0000", 2024-09-11T09:51:45.984 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7331000000000001e-05 2024-09-11T09:51:45.984 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.984 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.984 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:45.984 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.915019+0000", 2024-09-11T09:51:45.984 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00012168 2024-09-11T09:51:45.984 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.984 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.985 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:45.985 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.915269+0000", 2024-09-11T09:51:45.985 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00025054899999999998 2024-09-11T09:51:45.985 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.985 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.985 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 3", 2024-09-11T09:51:45.985 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.915479+0000", 2024-09-11T09:51:45.985 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00021017799999999999 2024-09-11T09:51:45.985 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.985 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.985 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:45.985 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.926070+0000", 2024-09-11T09:51:45.985 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.010590624999999999 2024-09-11T09:51:45.986 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.986 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.986 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:45.986 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.936198+0000", 2024-09-11T09:51:45.986 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.010128194 2024-09-11T09:51:45.986 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.986 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.986 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:45.986 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.936263+0000", 2024-09-11T09:51:45.986 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.5366999999999999e-05 2024-09-11T09:51:45.986 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.986 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.986 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:45.987 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.936317+0000", 2024-09-11T09:51:45.987 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.3156999999999999e-05 2024-09-11T09:51:45.987 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:45.987 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:45.987 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:45.987 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.987 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.987 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:39 2.6 2:743e7616:::rbd_data.1250f5e25b80.0000000000000005:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:45.987 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.914960+0000", 2024-09-11T09:51:45.987 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 23.054305550999999, 2024-09-11T09:51:45.987 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.022371518, 2024-09-11T09:51:45.988 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:45.988 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:45.988 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:45.988 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:45.988 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:51:45.988 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:51:45.988 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 39 2024-09-11T09:51:45.988 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.988 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:45.988 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.988 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:45.988 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.914960+0000", 2024-09-11T09:51:45.988 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:45.989 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.989 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.989 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:45.989 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.914960+0000", 2024-09-11T09:51:45.989 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:45.989 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.989 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.989 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:45.989 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.914963+0000", 2024-09-11T09:51:45.989 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.1119999999999999e-06 2024-09-11T09:51:45.989 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.989 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.990 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:45.990 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.920417+0000", 2024-09-11T09:51:45.990 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0054543969999999997 2024-09-11T09:51:45.990 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.990 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.990 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:45.990 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.920430+0000", 2024-09-11T09:51:45.990 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3414000000000001e-05 2024-09-11T09:51:45.990 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.990 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.990 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:45.990 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.920437+0000", 2024-09-11T09:51:45.990 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.6159999999999998e-06 2024-09-11T09:51:45.991 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.991 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.991 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:45.991 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.921325+0000", 2024-09-11T09:51:45.991 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000887731 2024-09-11T09:51:45.991 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.991 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.991 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:45.991 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.921422+0000", 2024-09-11T09:51:45.991 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.7201999999999995e-05 2024-09-11T09:51:45.991 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.991 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.991 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 3", 2024-09-11T09:51:45.992 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.921511+0000", 2024-09-11T09:51:45.992 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.8880000000000003e-05 2024-09-11T09:51:45.992 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.992 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.992 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:45.992 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.928092+0000", 2024-09-11T09:51:45.992 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0065816290000000003 2024-09-11T09:51:45.992 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.992 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.992 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:45.992 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.937240+0000", 2024-09-11T09:51:45.992 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0091481070000000008 2024-09-11T09:51:45.993 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.993 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.993 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:45.993 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.937294+0000", 2024-09-11T09:51:45.993 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.3100000000000003e-05 2024-09-11T09:51:45.993 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.993 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.993 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:45.993 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.937332+0000", 2024-09-11T09:51:45.993 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.8330000000000001e-05 2024-09-11T09:51:45.993 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:45.993 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:45.993 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:45.994 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.994 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.994 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4688.0:32 2.2 e53/15 2:45941fd2:::rbd_data.1250f5e25b80.0000000000000001:head v 53'3, mlcod=17'1)", 2024-09-11T09:51:45.994 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.916509+0000", 2024-09-11T09:51:45.994 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 23.052756468999998, 2024-09-11T09:51:45.994 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.015885318999999998, 2024-09-11T09:51:45.994 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:45.994 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:45.994 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:45.994 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:45.994 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.994 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:45.994 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.916509+0000", 2024-09-11T09:51:45.995 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:45.995 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.995 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.995 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:45.995 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.916509+0000", 2024-09-11T09:51:45.995 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:45.995 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.995 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.995 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:45.995 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.916511+0000", 2024-09-11T09:51:45.995 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.052e-06 2024-09-11T09:51:45.995 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.995 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.995 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:45.996 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.927019+0000", 2024-09-11T09:51:45.996 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.010508224 2024-09-11T09:51:45.996 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.996 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.996 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:45.996 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.927038+0000", 2024-09-11T09:51:45.996 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.9590999999999999e-05 2024-09-11T09:51:45.996 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.996 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.996 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:45.996 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.927050+0000", 2024-09-11T09:51:45.996 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1439e-05 2024-09-11T09:51:45.996 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.997 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.997 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:45.997 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.927115+0000", 2024-09-11T09:51:45.997 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.5301000000000004e-05 2024-09-11T09:51:45.997 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.997 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.997 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:45.997 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.927132+0000", 2024-09-11T09:51:45.997 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7297000000000001e-05 2024-09-11T09:51:45.997 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.997 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.997 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:45.998 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.932354+0000", 2024-09-11T09:51:45.998 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0052211870000000004 2024-09-11T09:51:45.998 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.998 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.998 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:45.998 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.932395+0000", 2024-09-11T09:51:45.998 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.1227999999999999e-05 2024-09-11T09:51:45.998 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:45.998 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:45.998 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:45.998 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:45.998 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:45.998 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:44 2.6 2:67246a22:::rbd_data.1250f5e25b80.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:45.999 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.943181+0000", 2024-09-11T09:51:45.999 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 23.026085257999998, 2024-09-11T09:51:45.999 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.022629371999999998, 2024-09-11T09:51:45.999 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:45.999 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:45.999 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:45.999 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:45.999 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:51:45.999 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:51:45.999 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 44 2024-09-11T09:51:46.000 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.000 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:46.000 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.000 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:46.000 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.943181+0000", 2024-09-11T09:51:46.000 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:46.000 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.000 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.000 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:46.000 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.943181+0000", 2024-09-11T09:51:46.000 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:46.000 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.000 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.001 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:46.001 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.943182+0000", 2024-09-11T09:51:46.001 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.0500000000000002e-07 2024-09-11T09:51:46.001 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.001 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.001 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:46.001 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951240+0000", 2024-09-11T09:51:46.001 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.008058529 2024-09-11T09:51:46.001 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.001 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.001 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:46.001 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951260+0000", 2024-09-11T09:51:46.002 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.9570000000000001e-05 2024-09-11T09:51:46.002 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.002 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.002 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:46.002 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951270+0000", 2024-09-11T09:51:46.002 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0018e-05 2024-09-11T09:51:46.002 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.002 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.002 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:46.002 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951349+0000", 2024-09-11T09:51:46.002 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.8801000000000007e-05 2024-09-11T09:51:46.002 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.002 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.003 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:46.003 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951483+0000", 2024-09-11T09:51:46.003 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00013427 2024-09-11T09:51:46.003 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.003 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.003 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 3", 2024-09-11T09:51:46.003 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951593+0000", 2024-09-11T09:51:46.003 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00011031000000000001 2024-09-11T09:51:46.003 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.003 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.003 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:46.003 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.957303+0000", 2024-09-11T09:51:46.003 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0057097959999999996 2024-09-11T09:51:46.003 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.004 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.004 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:46.004 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.965690+0000", 2024-09-11T09:51:46.004 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0083867420000000008 2024-09-11T09:51:46.004 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.004 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.004 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:46.004 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.965749+0000", 2024-09-11T09:51:46.004 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.8981999999999999e-05 2024-09-11T09:51:46.004 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.004 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.004 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:46.004 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.965810+0000", 2024-09-11T09:51:46.005 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.1449000000000001e-05 2024-09-11T09:51:46.005 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:46.005 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:46.005 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:46.005 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.005 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.005 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:51 2.6 2:6b5894fc:::rbd_data.1250f5e25b80.000000000000000a:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:46.005 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.951324+0000", 2024-09-11T09:51:46.005 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 23.017942184999999, 2024-09-11T09:51:46.005 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.021375273, 2024-09-11T09:51:46.005 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:46.005 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:46.005 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:46.005 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:46.006 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:51:46.006 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:51:46.006 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 51 2024-09-11T09:51:46.006 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.006 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:46.006 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.006 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:46.006 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951324+0000", 2024-09-11T09:51:46.006 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:46.006 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.006 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.006 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:46.006 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951324+0000", 2024-09-11T09:51:46.006 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:46.007 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.007 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.007 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:46.007 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951324+0000", 2024-09-11T09:51:46.007 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.39e-07 2024-09-11T09:51:46.007 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.007 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.007 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:46.007 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.958537+0000", 2024-09-11T09:51:46.007 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0072126769999999998 2024-09-11T09:51:46.007 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.007 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.007 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:46.007 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.958554+0000", 2024-09-11T09:51:46.008 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7232e-05 2024-09-11T09:51:46.008 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.008 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.008 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:46.008 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.958570+0000", 2024-09-11T09:51:46.008 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.5747999999999999e-05 2024-09-11T09:51:46.008 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.008 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.008 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:46.008 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.958638+0000", 2024-09-11T09:51:46.008 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.7810000000000003e-05 2024-09-11T09:51:46.008 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.008 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.009 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:46.009 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.958758+0000", 2024-09-11T09:51:46.009 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000120689 2024-09-11T09:51:46.009 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.009 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.009 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 3", 2024-09-11T09:51:46.009 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.958865+0000", 2024-09-11T09:51:46.009 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00010692 2024-09-11T09:51:46.009 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.009 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.009 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:46.009 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.970712+0000", 2024-09-11T09:51:46.010 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.011846327 2024-09-11T09:51:46.010 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.010 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.010 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:46.010 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.972603+0000", 2024-09-11T09:51:46.010 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0018918610000000001 2024-09-11T09:51:46.010 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.010 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.010 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:46.010 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.972653+0000", 2024-09-11T09:51:46.010 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.9762999999999998e-05 2024-09-11T09:51:46.011 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.011 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.011 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:46.011 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.972699+0000", 2024-09-11T09:51:46.011 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.5806999999999997e-05 2024-09-11T09:51:46.011 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:46.011 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:46.011 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:46.011 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.011 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.011 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:52 2.4 2:26d5c2f0:::rbd_data.1250f5e25b80.000000000000000b:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:46.011 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.958596+0000", 2024-09-11T09:51:46.011 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 23.01066999, 2024-09-11T09:51:46.012 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.016477849999999999, 2024-09-11T09:51:46.012 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:46.012 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:46.012 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:46.012 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:46.012 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:51:46.012 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:51:46.012 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 52 2024-09-11T09:51:46.012 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.012 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:46.012 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.012 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:46.013 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.958596+0000", 2024-09-11T09:51:46.013 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:46.013 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.013 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.013 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:46.013 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.958596+0000", 2024-09-11T09:51:46.013 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:46.013 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.013 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.013 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:46.013 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.958597+0000", 2024-09-11T09:51:46.013 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1960000000000001e-06 2024-09-11T09:51:46.013 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.014 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.014 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:46.014 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.963157+0000", 2024-09-11T09:51:46.014 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0045603029999999999 2024-09-11T09:51:46.014 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.014 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.014 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:46.014 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.963171+0000", 2024-09-11T09:51:46.014 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3777999999999999e-05 2024-09-11T09:51:46.014 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.014 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.014 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:46.015 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.963181+0000", 2024-09-11T09:51:46.015 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.9010000000000006e-06 2024-09-11T09:51:46.015 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.015 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.015 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:46.015 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.963246+0000", 2024-09-11T09:51:46.015 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.5351000000000005e-05 2024-09-11T09:51:46.015 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.015 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.015 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:46.015 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.963419+0000", 2024-09-11T09:51:46.015 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00017299700000000001 2024-09-11T09:51:46.016 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.016 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.016 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:46.016 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.963570+0000", 2024-09-11T09:51:46.016 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00015066999999999999 2024-09-11T09:51:46.016 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.016 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.016 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:46.016 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.973881+0000", 2024-09-11T09:51:46.016 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.010311268 2024-09-11T09:51:46.016 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.016 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.016 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:46.017 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.974963+0000", 2024-09-11T09:51:46.017 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.001081493 2024-09-11T09:51:46.017 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.017 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.017 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:46.017 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.975012+0000", 2024-09-11T09:51:46.017 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.9471000000000001e-05 2024-09-11T09:51:46.017 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.017 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.017 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:46.017 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.975074+0000", 2024-09-11T09:51:46.017 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.1421999999999996e-05 2024-09-11T09:51:46.017 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:46.018 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:46.018 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:46.018 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.018 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.018 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4688.0:47 2.1 e53/15 2:95f53b3e:::rbd_data.1250f5e25b80.0000000000000008:head v 53'3, mlcod=53'1)", 2024-09-11T09:51:46.018 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.960861+0000", 2024-09-11T09:51:46.018 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 23.008404572, 2024-09-11T09:51:46.018 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.012346107, 2024-09-11T09:51:46.018 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:46.018 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:46.018 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:46.018 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:46.018 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.019 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:46.019 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.960861+0000", 2024-09-11T09:51:46.019 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:46.019 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.019 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.019 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:46.019 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.960861+0000", 2024-09-11T09:51:46.019 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:46.019 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.019 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.019 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:46.019 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.960863+0000", 2024-09-11T09:51:46.019 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7150000000000001e-06 2024-09-11T09:51:46.019 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.020 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.020 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:46.020 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.962428+0000", 2024-09-11T09:51:46.020 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.001564668 2024-09-11T09:51:46.020 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.020 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.020 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:46.020 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.962446+0000", 2024-09-11T09:51:46.020 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.8476999999999999e-05 2024-09-11T09:51:46.020 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.020 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.020 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:46.020 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.962458+0000", 2024-09-11T09:51:46.021 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1263999999999999e-05 2024-09-11T09:51:46.021 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.021 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.021 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:46.021 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.962575+0000", 2024-09-11T09:51:46.021 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000117025 2024-09-11T09:51:46.021 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.021 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.021 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:46.021 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.962601+0000", 2024-09-11T09:51:46.021 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.5984999999999999e-05 2024-09-11T09:51:46.021 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.021 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.022 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:46.022 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.973065+0000", 2024-09-11T09:51:46.022 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.010464753 2024-09-11T09:51:46.022 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.022 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.022 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:46.022 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.973207+0000", 2024-09-11T09:51:46.022 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00014222 2024-09-11T09:51:46.022 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:46.022 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:46.022 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:46.022 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.022 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.023 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:54 2.4 2.da8979c4 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:46.023 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.963207+0000", 2024-09-11T09:51:46.023 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 23.006059134000001, 2024-09-11T09:51:46.023 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.010641575, 2024-09-11T09:51:46.023 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:46.023 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:46.023 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "reached pg", 2024-09-11T09:51:46.023 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:46.023 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:51:46.023 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:51:46.023 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 54 2024-09-11T09:51:46.023 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.023 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:46.023 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.024 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:46.024 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.963207+0000", 2024-09-11T09:51:46.024 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:46.024 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.024 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.024 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:46.024 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.963207+0000", 2024-09-11T09:51:46.024 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:46.024 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.024 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.024 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:46.024 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.963208+0000", 2024-09-11T09:51:46.024 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2979999999999999e-06 2024-09-11T09:51:46.024 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.025 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.025 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:46.025 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.967419+0000", 2024-09-11T09:51:46.025 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0042111750000000002 2024-09-11T09:51:46.025 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.025 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.025 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:46.025 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.967439+0000", 2024-09-11T09:51:46.025 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.0129999999999999e-05 2024-09-11T09:51:46.025 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.025 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.025 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:46.025 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.967453+0000", 2024-09-11T09:51:46.026 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3781000000000001e-05 2024-09-11T09:51:46.026 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.026 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.026 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:46.026 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.973840+0000", 2024-09-11T09:51:46.026 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0063872269999999997 2024-09-11T09:51:46.026 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.026 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.026 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:46.026 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.973848+0000", 2024-09-11T09:51:46.026 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.9640000000000003e-06 2024-09-11T09:51:46.026 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:46.026 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:46.027 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:46.027 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.027 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.027 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:54 2.4 2:239e915b:::rbd_data.1250f5e25b80.000000000000000c:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:46.027 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.982401+0000", 2024-09-11T09:51:46.027 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 22.986865207000001, 2024-09-11T09:51:46.027 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.016817347, 2024-09-11T09:51:46.027 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:46.027 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:46.027 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:46.027 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:46.027 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:51:46.028 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:51:46.028 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 54 2024-09-11T09:51:46.028 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.028 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:46.028 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.028 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:46.028 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.982401+0000", 2024-09-11T09:51:46.028 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:46.028 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.028 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.028 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:46.028 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.982401+0000", 2024-09-11T09:51:46.029 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:46.029 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.029 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.029 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:46.029 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.982402+0000", 2024-09-11T09:51:46.029 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.8999999999999995e-07 2024-09-11T09:51:46.029 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.029 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.029 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:46.029 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.986535+0000", 2024-09-11T09:51:46.029 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0041333689999999996 2024-09-11T09:51:46.029 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.030 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.030 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:46.030 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.986555+0000", 2024-09-11T09:51:46.030 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.9851e-05 2024-09-11T09:51:46.030 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.030 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.030 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:46.030 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.986569+0000", 2024-09-11T09:51:46.030 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3784000000000001e-05 2024-09-11T09:51:46.030 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.030 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.030 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:46.030 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.986683+0000", 2024-09-11T09:51:46.031 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000114474 2024-09-11T09:51:46.031 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.031 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.031 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:46.031 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.986815+0000", 2024-09-11T09:51:46.031 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00013217799999999999 2024-09-11T09:51:46.031 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.031 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.031 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:46.031 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.986918+0000", 2024-09-11T09:51:46.031 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00010235599999999999 2024-09-11T09:51:46.031 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.032 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.032 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:46.032 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.998983+0000", 2024-09-11T09:51:46.032 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.012065352 2024-09-11T09:51:46.032 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.032 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.032 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:46.032 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.999081+0000", 2024-09-11T09:51:46.032 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.8091000000000007e-05 2024-09-11T09:51:46.032 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.032 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.032 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:46.032 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.999159+0000", 2024-09-11T09:51:46.033 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.7823000000000004e-05 2024-09-11T09:51:46.033 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.033 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.033 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:46.033 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.999218+0000", 2024-09-11T09:51:46.033 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.9178999999999997e-05 2024-09-11T09:51:46.033 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:46.033 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:46.033 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:46.033 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.033 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.033 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:56 2.4 2:2167012f:::rbd_data.1250f5e25b80.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:46.034 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.986661+0000", 2024-09-11T09:51:46.034 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 22.982604754, 2024-09-11T09:51:46.034 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.017353539000000001, 2024-09-11T09:51:46.034 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:46.034 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:46.034 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:46.034 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:46.034 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:51:46.034 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:51:46.034 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 56 2024-09-11T09:51:46.034 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.034 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:46.034 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.035 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:46.035 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.986661+0000", 2024-09-11T09:51:46.035 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:46.035 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.035 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.035 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:46.035 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.986661+0000", 2024-09-11T09:51:46.035 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:46.035 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.035 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.035 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:46.035 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.986663+0000", 2024-09-11T09:51:46.035 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.8989999999999999e-06 2024-09-11T09:51:46.035 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.036 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.036 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:46.036 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.990444+0000", 2024-09-11T09:51:46.036 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0037808569999999999 2024-09-11T09:51:46.036 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.036 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.036 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:46.036 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.990462+0000", 2024-09-11T09:51:46.036 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.8450000000000001e-05 2024-09-11T09:51:46.036 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.036 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.036 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:46.036 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.990476+0000", 2024-09-11T09:51:46.037 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3565e-05 2024-09-11T09:51:46.037 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.037 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.037 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:46.037 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.993582+0000", 2024-09-11T09:51:46.037 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0031059109999999998 2024-09-11T09:51:46.037 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.037 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.037 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:46.037 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.993686+0000", 2024-09-11T09:51:46.037 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000104424 2024-09-11T09:51:46.037 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.037 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.038 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:46.038 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.993799+0000", 2024-09-11T09:51:46.038 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000112985 2024-09-11T09:51:46.038 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.038 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.038 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:46.038 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.002449+0000", 2024-09-11T09:51:46.038 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.008650151 2024-09-11T09:51:46.038 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.038 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.038 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:46.038 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.003948+0000", 2024-09-11T09:51:46.038 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.001498636 2024-09-11T09:51:46.039 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.039 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.039 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:46.039 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.003985+0000", 2024-09-11T09:51:46.039 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.7054999999999997e-05 2024-09-11T09:51:46.039 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.039 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.039 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:46.039 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.004015+0000", 2024-09-11T09:51:46.040 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.9606000000000001e-05 2024-09-11T09:51:46.040 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:46.040 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:46.040 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:46.040 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.040 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.040 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:58 2.6 2:6277c644:::rbd_data.1250f5e25b80.000000000000000e:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:46.040 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.990540+0000", 2024-09-11T09:51:46.040 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 22.978725958999998, 2024-09-11T09:51:46.040 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.016232068999999998, 2024-09-11T09:51:46.040 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:46.040 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:46.040 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:46.041 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:46.041 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:51:46.041 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:51:46.041 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 58 2024-09-11T09:51:46.041 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.041 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:46.041 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.041 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:46.041 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.990540+0000", 2024-09-11T09:51:46.041 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:46.041 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.041 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.041 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:46.041 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.990540+0000", 2024-09-11T09:51:46.042 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:46.042 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.042 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.042 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:46.042 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.990541+0000", 2024-09-11T09:51:46.042 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2359999999999999e-06 2024-09-11T09:51:46.042 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.042 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.042 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:46.042 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.994626+0000", 2024-09-11T09:51:46.042 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.004084865 2024-09-11T09:51:46.042 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.042 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.043 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:46.043 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.994644+0000", 2024-09-11T09:51:46.043 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7866000000000001e-05 2024-09-11T09:51:46.043 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.043 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.043 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:46.043 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.994653+0000", 2024-09-11T09:51:46.043 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.2159999999999995e-06 2024-09-11T09:51:46.043 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.043 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.043 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:46.043 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.994732+0000", 2024-09-11T09:51:46.043 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.8954000000000003e-05 2024-09-11T09:51:46.044 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.044 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.044 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:46.044 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.994906+0000", 2024-09-11T09:51:46.044 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0001741 2024-09-11T09:51:46.044 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.044 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.044 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 3", 2024-09-11T09:51:46.044 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.995083+0000", 2024-09-11T09:51:46.044 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00017637100000000001 2024-09-11T09:51:46.044 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.044 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.045 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:46.045 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.003066+0000", 2024-09-11T09:51:46.045 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0079833360000000006 2024-09-11T09:51:46.045 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.045 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.045 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:46.045 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.006634+0000", 2024-09-11T09:51:46.045 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.003568309 2024-09-11T09:51:46.045 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.045 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.045 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:46.045 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.006716+0000", 2024-09-11T09:51:46.045 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.1377000000000004e-05 2024-09-11T09:51:46.046 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.046 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.046 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:46.046 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.006772+0000", 2024-09-11T09:51:46.046 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.6439e-05 2024-09-11T09:51:46.046 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:46.046 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:46.046 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:46.046 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.046 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.046 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:59 2.4 2:2906e6a7:::rbd_header.1250f5e25b80:head [watch reconnect cookie 140424358411408 gen 2] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:46.046 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.994684+0000", 2024-09-11T09:51:46.046 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 22.974582328, 2024-09-11T09:51:46.046 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0094134349999999995, 2024-09-11T09:51:46.047 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:46.047 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:46.047 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "started", 2024-09-11T09:51:46.047 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:46.047 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:51:46.047 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:51:46.047 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 59 2024-09-11T09:51:46.047 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.047 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:46.047 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.047 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:46.047 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.994684+0000", 2024-09-11T09:51:46.047 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:46.048 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.048 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.048 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:46.048 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.994684+0000", 2024-09-11T09:51:46.048 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:46.048 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.048 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.048 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:46.048 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.994686+0000", 2024-09-11T09:51:46.048 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.8159999999999998e-06 2024-09-11T09:51:46.048 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.048 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.048 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:46.049 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.994691+0000", 2024-09-11T09:51:46.049 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.5009999999999998e-06 2024-09-11T09:51:46.049 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.049 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.049 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:46.049 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.994698+0000", 2024-09-11T09:51:46.049 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.3039999999999999e-06 2024-09-11T09:51:46.049 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.049 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.049 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:46.049 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.994703+0000", 2024-09-11T09:51:46.049 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.5190000000000003e-06 2024-09-11T09:51:46.049 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.050 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.050 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:46.050 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.998678+0000", 2024-09-11T09:51:46.050 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0039752340000000002 2024-09-11T09:51:46.050 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.050 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.050 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:46.050 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.998748+0000", 2024-09-11T09:51:46.050 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.0114e-05 2024-09-11T09:51:46.050 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.050 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.050 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:46.050 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.004097+0000", 2024-09-11T09:51:46.051 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0053489469999999997 2024-09-11T09:51:46.051 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:46.051 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:46.051 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:46.051 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.051 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.051 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4694.0:33 2.4 2:3bfc818c:ns::rbd_data.1256a0c02879.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:46.051 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.084537+0000", 2024-09-11T09:51:46.051 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 22.884728899999999, 2024-09-11T09:51:46.051 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.022075773999999999, 2024-09-11T09:51:46.051 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:46.051 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:46.051 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:46.052 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:46.052 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4694", 2024-09-11T09:51:46.052 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:51:46.052 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 33 2024-09-11T09:51:46.052 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.052 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:46.052 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.052 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:46.052 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.084537+0000", 2024-09-11T09:51:46.052 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:46.052 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.052 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.052 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:46.053 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.084537+0000", 2024-09-11T09:51:46.053 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:46.053 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.053 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.053 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:46.053 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.084538+0000", 2024-09-11T09:51:46.053 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.5000000000000001e-07 2024-09-11T09:51:46.053 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.053 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.053 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:46.053 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.095984+0000", 2024-09-11T09:51:46.053 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.011446543 2024-09-11T09:51:46.053 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.054 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.054 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:46.054 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.096014+0000", 2024-09-11T09:51:46.054 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.9745000000000001e-05 2024-09-11T09:51:46.054 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.054 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.054 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:46.054 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.096030+0000", 2024-09-11T09:51:46.054 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6152999999999999e-05 2024-09-11T09:51:46.054 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.054 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.054 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:46.054 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.096113+0000", 2024-09-11T09:51:46.054 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.2731999999999993e-05 2024-09-11T09:51:46.055 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.055 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.055 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:46.055 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.096273+0000", 2024-09-11T09:51:46.055 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00015973699999999999 2024-09-11T09:51:46.055 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.055 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.055 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:46.055 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.096394+0000", 2024-09-11T09:51:46.055 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000120883 2024-09-11T09:51:46.055 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.055 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.055 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:46.055 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.106048+0000", 2024-09-11T09:51:46.056 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0096548020000000005 2024-09-11T09:51:46.056 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.056 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.056 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:46.056 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.106531+0000", 2024-09-11T09:51:46.056 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00048264299999999999 2024-09-11T09:51:46.056 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.056 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.056 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:46.056 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.106565+0000", 2024-09-11T09:51:46.056 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.4343e-05 2024-09-11T09:51:46.056 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.056 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.057 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:46.057 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.106613+0000", 2024-09-11T09:51:46.057 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.7342999999999997e-05 2024-09-11T09:51:46.057 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:46.057 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:46.057 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:46.057 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.057 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.057 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4694.0:32 2.1 e53/15 2:8a862b48:ns::rbd_data.1256a0c02879.0000000000000000:head v 53'8, mlcod=53'6)", 2024-09-11T09:51:46.057 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.091418+0000", 2024-09-11T09:51:46.057 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 22.877847730999999, 2024-09-11T09:51:46.057 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.013027372000000001, 2024-09-11T09:51:46.057 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:46.057 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:46.058 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:46.058 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:46.058 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.058 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:46.058 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.091418+0000", 2024-09-11T09:51:46.058 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:46.058 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.058 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.058 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:46.058 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.091418+0000", 2024-09-11T09:51:46.058 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:46.058 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.058 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.059 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:46.059 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.091420+0000", 2024-09-11T09:51:46.059 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6899999999999999e-06 2024-09-11T09:51:46.059 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.059 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.059 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:46.059 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.093086+0000", 2024-09-11T09:51:46.059 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0016662510000000001 2024-09-11T09:51:46.059 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.059 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.059 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:46.059 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.093107+0000", 2024-09-11T09:51:46.059 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.0999e-05 2024-09-11T09:51:46.059 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.060 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.060 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:46.060 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.093120+0000", 2024-09-11T09:51:46.060 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2544999999999999e-05 2024-09-11T09:51:46.060 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.060 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.060 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:46.060 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.093186+0000", 2024-09-11T09:51:46.060 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.6595999999999997e-05 2024-09-11T09:51:46.060 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.060 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.060 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:46.060 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.093208+0000", 2024-09-11T09:51:46.060 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.1624000000000002e-05 2024-09-11T09:51:46.061 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.061 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.061 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:46.061 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.104392+0000", 2024-09-11T09:51:46.061 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.011184265000000001 2024-09-11T09:51:46.061 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.061 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.061 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:46.061 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.104446+0000", 2024-09-11T09:51:46.061 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.3402000000000001e-05 2024-09-11T09:51:46.061 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:46.061 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:46.061 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:46.062 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.062 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.062 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4694.0:45 2.6 2:671fb830:ns::rbd_data.1256a0c02879.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:46.062 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.109629+0000", 2024-09-11T09:51:46.062 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 22.859636890000001, 2024-09-11T09:51:46.062 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.036403143999999998, 2024-09-11T09:51:46.062 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:46.062 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:46.062 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:46.062 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:46.062 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4694", 2024-09-11T09:51:46.062 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:51:46.062 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 45 2024-09-11T09:51:46.062 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.063 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:46.063 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.063 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:46.063 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.109629+0000", 2024-09-11T09:51:46.063 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:46.063 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.063 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.063 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:46.063 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.109629+0000", 2024-09-11T09:51:46.063 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:46.063 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.063 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.063 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:46.064 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.109630+0000", 2024-09-11T09:51:46.064 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1200000000000001e-06 2024-09-11T09:51:46.064 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.064 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.064 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:46.064 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.121370+0000", 2024-09-11T09:51:46.064 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.011739571000000001 2024-09-11T09:51:46.064 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.064 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.064 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:46.064 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.121391+0000", 2024-09-11T09:51:46.064 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.1175e-05 2024-09-11T09:51:46.064 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.065 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.065 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:46.065 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.121404+0000", 2024-09-11T09:51:46.065 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2807e-05 2024-09-11T09:51:46.065 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.065 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.065 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:46.065 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.121498+0000", 2024-09-11T09:51:46.065 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.3770999999999994e-05 2024-09-11T09:51:46.065 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.065 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.065 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:46.065 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.121612+0000", 2024-09-11T09:51:46.065 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000114666 2024-09-11T09:51:46.066 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.066 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.066 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 3", 2024-09-11T09:51:46.066 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.121711+0000", 2024-09-11T09:51:46.066 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.8647999999999995e-05 2024-09-11T09:51:46.066 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.066 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.066 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:46.066 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.126558+0000", 2024-09-11T09:51:46.066 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0048473769999999999 2024-09-11T09:51:46.066 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.066 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.066 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:46.066 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.145912+0000", 2024-09-11T09:51:46.067 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.019354281000000001 2024-09-11T09:51:46.067 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.067 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.067 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:46.067 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.145977+0000", 2024-09-11T09:51:46.067 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.4135999999999997e-05 2024-09-11T09:51:46.067 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.067 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.067 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:46.067 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.146032+0000", 2024-09-11T09:51:46.067 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.5591999999999999e-05 2024-09-11T09:51:46.067 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:46.067 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:46.067 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:46.068 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.068 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.068 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4694.0:41 2.2 e53/15 2:5e2505c4:ns::rbd_data.1256a0c02879.0000000000000003:head v 53'4, mlcod=18'2)", 2024-09-11T09:51:46.068 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.113199+0000", 2024-09-11T09:51:46.068 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 22.856066479999999, 2024-09-11T09:51:46.068 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.011404512, 2024-09-11T09:51:46.068 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:46.068 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:46.068 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:46.068 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:46.068 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.068 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:46.068 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.113199+0000", 2024-09-11T09:51:46.068 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:46.069 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.069 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.069 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:46.069 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.113199+0000", 2024-09-11T09:51:46.069 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:46.069 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.069 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.069 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:46.069 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.113200+0000", 2024-09-11T09:51:46.069 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.2799999999999995e-07 2024-09-11T09:51:46.069 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.069 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.069 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:46.069 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.118872+0000", 2024-09-11T09:51:46.070 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0056718089999999999 2024-09-11T09:51:46.070 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.070 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.070 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:46.070 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.118889+0000", 2024-09-11T09:51:46.070 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7428999999999999e-05 2024-09-11T09:51:46.070 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.070 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.070 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:46.070 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.118900+0000", 2024-09-11T09:51:46.070 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0243e-05 2024-09-11T09:51:46.070 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.070 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.071 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:46.071 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.118959+0000", 2024-09-11T09:51:46.071 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.9364000000000003e-05 2024-09-11T09:51:46.071 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.071 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.071 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:46.071 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.118976+0000", 2024-09-11T09:51:46.071 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7124000000000001e-05 2024-09-11T09:51:46.071 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.071 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.071 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:46.071 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.124557+0000", 2024-09-11T09:51:46.071 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0055812659999999997 2024-09-11T09:51:46.071 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.072 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.072 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:46.072 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.124604+0000", 2024-09-11T09:51:46.072 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.6548999999999998e-05 2024-09-11T09:51:46.072 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:46.072 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:46.072 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:46.072 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.072 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.072 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4694.0:43 2.0 e53/15 2:16c8ae84:ns::rbd_data.1256a0c02879.0000000000000005:head v 53'2, mlcod=0'0)", 2024-09-11T09:51:46.072 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.158559+0000", 2024-09-11T09:51:46.072 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 22.810707153999999, 2024-09-11T09:51:46.072 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.016713677999999999, 2024-09-11T09:51:46.073 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:46.073 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:46.073 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:46.073 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:46.073 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.073 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:46.073 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.158559+0000", 2024-09-11T09:51:46.073 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:46.073 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.073 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.073 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:46.073 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.158559+0000", 2024-09-11T09:51:46.073 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:46.073 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.074 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.074 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:46.074 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.158560+0000", 2024-09-11T09:51:46.074 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1459999999999999e-06 2024-09-11T09:51:46.074 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.074 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.074 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:46.074 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.166465+0000", 2024-09-11T09:51:46.074 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.007904767 2024-09-11T09:51:46.074 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.074 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.074 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:46.074 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.166491+0000", 2024-09-11T09:51:46.075 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.6308e-05 2024-09-11T09:51:46.075 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.075 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.075 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:46.075 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.166502+0000", 2024-09-11T09:51:46.075 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0736e-05 2024-09-11T09:51:46.075 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.075 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.075 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:46.075 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.166564+0000", 2024-09-11T09:51:46.075 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.2578999999999998e-05 2024-09-11T09:51:46.075 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.075 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.075 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:46.076 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.166585+0000", 2024-09-11T09:51:46.076 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.0395999999999999e-05 2024-09-11T09:51:46.076 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.076 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.076 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:46.076 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.175227+0000", 2024-09-11T09:51:46.076 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0086426420000000007 2024-09-11T09:51:46.076 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.076 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.076 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:46.076 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.175272+0000", 2024-09-11T09:51:46.076 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.5104e-05 2024-09-11T09:51:46.077 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:46.077 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:46.077 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:46.077 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.077 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.077 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4694.0:59 2.2 e53/15 2:41adefc9:ns::rbd_data.1256a0c02879.000000000000000e:head v 53'5, mlcod=53'3)", 2024-09-11T09:51:46.077 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.163389+0000", 2024-09-11T09:51:46.077 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 22.805877177999999, 2024-09-11T09:51:46.077 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.013095719, 2024-09-11T09:51:46.077 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:46.077 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:46.077 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:46.077 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:46.077 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.078 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:46.078 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.163389+0000", 2024-09-11T09:51:46.078 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:46.078 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.078 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.078 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:46.078 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.163389+0000", 2024-09-11T09:51:46.078 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:46.078 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.078 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.078 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:46.078 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.163390+0000", 2024-09-11T09:51:46.078 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.336e-06 2024-09-11T09:51:46.079 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.079 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.079 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:46.079 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.170335+0000", 2024-09-11T09:51:46.079 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0069451080000000002 2024-09-11T09:51:46.079 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.079 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.079 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:46.079 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.170354+0000", 2024-09-11T09:51:46.079 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.8423e-05 2024-09-11T09:51:46.079 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.079 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.079 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:46.079 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.170366+0000", 2024-09-11T09:51:46.080 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2235999999999999e-05 2024-09-11T09:51:46.080 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.080 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.080 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:46.080 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.170439+0000", 2024-09-11T09:51:46.080 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.2989000000000002e-05 2024-09-11T09:51:46.080 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.080 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.080 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:46.080 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.170460+0000", 2024-09-11T09:51:46.080 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.1211000000000001e-05 2024-09-11T09:51:46.080 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.080 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.081 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:46.081 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.176442+0000", 2024-09-11T09:51:46.081 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0059821969999999999 2024-09-11T09:51:46.081 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.081 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.081 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:46.081 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.176484+0000", 2024-09-11T09:51:46.081 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.2218999999999997e-05 2024-09-11T09:51:46.081 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:46.081 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:46.081 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:46.081 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:46.081 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:46.081 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:51:46.083 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-11T09:51:46.083 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:51:46.082+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:51:46.149 INFO:tasks.workunit.client.0.smithi107.stderr:+ DEV= 2024-09-11T09:51:46.150 INFO:tasks.workunit.client.0.smithi107.stderr:++ _sudo rbd device --device-type nbd map rbd/testrbdnbd27121 2024-09-11T09:51:46.150 INFO:tasks.workunit.client.0.smithi107.stderr:++ local cmd 2024-09-11T09:51:46.150 INFO:tasks.workunit.client.0.smithi107.stderr:+++ id -u 2024-09-11T09:51:46.150 INFO:tasks.workunit.client.0.smithi107.stderr:++ '[' 1000 -eq 0 ']' 2024-09-11T09:51:46.151 INFO:tasks.workunit.client.0.smithi107.stderr:+++ which rbd 2024-09-11T09:51:46.152 INFO:tasks.workunit.client.0.smithi107.stderr:++ cmd=/usr/bin/rbd 2024-09-11T09:51:46.153 INFO:tasks.workunit.client.0.smithi107.stderr:++ shift 2024-09-11T09:51:46.153 INFO:tasks.workunit.client.0.smithi107.stderr:++ sudo -nE /usr/bin/rbd device --device-type nbd map rbd/testrbdnbd27121 2024-09-11T09:51:46.184 INFO:tasks.ceph.osd.0.smithi017.stderr:2024-09-11T09:51:46.184+0000 7fb73c0f1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31665) UID: 0 2024-09-11T09:51:46.233 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:51:46.233 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [], 2024-09-11T09:51:46.233 INFO:teuthology.orchestra.run.smithi017.stdout: "num_ops": 0 2024-09-11T09:51:46.233 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:51:46.242 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-11T09:51:46.267 INFO:tasks.workunit.client.0.smithi107.stderr:+ DEV=/dev/nbd0 2024-09-11T09:51:46.267 INFO:tasks.workunit.client.0.smithi107.stderr:+ get_pid rbd 2024-09-11T09:51:46.267 INFO:tasks.workunit.client.0.smithi107.stderr:+ local pool=rbd 2024-09-11T09:51:46.267 INFO:tasks.workunit.client.0.smithi107.stderr:+ local ns= 2024-09-11T09:51:46.268 INFO:tasks.workunit.client.0.smithi107.stderr:++ rbd device --device-type nbd --format xml list 2024-09-11T09:51:46.268 INFO:tasks.workunit.client.0.smithi107.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd27121'\''][device='\''/dev/nbd0'\'']/id' 2024-09-11T09:51:46.285 INFO:tasks.ceph.osd.0.smithi017.stderr:2024-09-11T09:51:46.285+0000 7fb73c0f1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31665) UID: 0 2024-09-11T09:51:46.329 INFO:tasks.workunit.client.0.smithi107.stderr:+ PID=28925 2024-09-11T09:51:46.329 INFO:tasks.workunit.client.0.smithi107.stderr:+ test -n 28925 2024-09-11T09:51:46.329 INFO:tasks.workunit.client.0.smithi107.stderr:+ ps -p 28925 -C rbd-nbd 2024-09-11T09:51:46.339 INFO:tasks.workunit.client.0.smithi107.stdout: PID TTY TIME CMD 2024-09-11T09:51:46.339 INFO:tasks.workunit.client.0.smithi107.stdout: 28925 ? 00:00:00 rbd-nbd 2024-09-11T09:51:46.340 INFO:tasks.workunit.client.0.smithi107.stderr:+ pid=28925 2024-09-11T09:51:46.340 INFO:tasks.workunit.client.0.smithi107.stderr:++ _sudo rbd device --device-type nbd map rbd/ns/testrbdnbd27121 2024-09-11T09:51:46.340 INFO:tasks.workunit.client.0.smithi107.stderr:++ local cmd 2024-09-11T09:51:46.340 INFO:tasks.workunit.client.0.smithi107.stderr:+++ id -u 2024-09-11T09:51:46.341 INFO:tasks.workunit.client.0.smithi107.stderr:++ '[' 1000 -eq 0 ']' 2024-09-11T09:51:46.341 INFO:tasks.workunit.client.0.smithi107.stderr:+++ which rbd 2024-09-11T09:51:46.343 INFO:tasks.workunit.client.0.smithi107.stderr:++ cmd=/usr/bin/rbd 2024-09-11T09:51:46.343 INFO:tasks.workunit.client.0.smithi107.stderr:++ shift 2024-09-11T09:51:46.343 INFO:tasks.workunit.client.0.smithi107.stderr:++ sudo -nE /usr/bin/rbd device --device-type nbd map rbd/ns/testrbdnbd27121 2024-09-11T09:51:46.386 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-09-11T09:51:46.386+0000 7f7b6c7d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31904) UID: 0 2024-09-11T09:51:46.389 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:51:46.389 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [], 2024-09-11T09:51:46.389 INFO:teuthology.orchestra.run.smithi017.stdout: "complaint_time": 30, 2024-09-11T09:51:46.389 INFO:teuthology.orchestra.run.smithi017.stdout: "num_blocked_ops": 0 2024-09-11T09:51:46.389 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:51:46.398 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-11T09:51:46.445 INFO:tasks.workunit.client.0.smithi107.stderr:+ DEV=/dev/nbd1 2024-09-11T09:51:46.446 INFO:tasks.workunit.client.0.smithi107.stderr:+ get_pid rbd ns 2024-09-11T09:51:46.446 INFO:tasks.workunit.client.0.smithi107.stderr:+ local pool=rbd 2024-09-11T09:51:46.446 INFO:tasks.workunit.client.0.smithi107.stderr:+ local ns=ns 2024-09-11T09:51:46.446 INFO:tasks.workunit.client.0.smithi107.stderr:++ rbd device --device-type nbd --format xml list 2024-09-11T09:51:46.446 INFO:tasks.workunit.client.0.smithi107.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\''ns'\''][image='\''testrbdnbd27121'\''][device='\''/dev/nbd1'\'']/id' 2024-09-11T09:51:46.486 INFO:tasks.ceph.osd.2.smithi017.stderr:2024-09-11T09:51:46.486+0000 7fd20b170640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31672) UID: 0 2024-09-11T09:51:46.510 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:51:46.510 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 20, 2024-09-11T09:51:46.510 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 600, 2024-09-11T09:51:46.510 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [ 2024-09-11T09:51:46.510 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.510 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:31 2.1 2:8654721b:::rbd_data.1250f5e25b80.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:46.510 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.896024+0000", 2024-09-11T09:51:46.510 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 23.613953645999999, 2024-09-11T09:51:46.511 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.023148401999999998, 2024-09-11T09:51:46.511 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:46.511 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:46.511 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:46.511 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:46.511 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:51:46.511 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:51:46.511 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 31 2024-09-11T09:51:46.511 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.511 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:46.511 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.511 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:46.512 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.896024+0000", 2024-09-11T09:51:46.512 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:46.512 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.512 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.512 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:46.512 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.896024+0000", 2024-09-11T09:51:46.512 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:46.512 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.512 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.512 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:46.512 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.896025+0000", 2024-09-11T09:51:46.512 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.54e-07 2024-09-11T09:51:46.512 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.513 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.513 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:46.513 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.904287+0000", 2024-09-11T09:51:46.513 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0082624729999999993 2024-09-11T09:51:46.513 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.513 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.513 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:46.513 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.904305+0000", 2024-09-11T09:51:46.513 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7912999999999999e-05 2024-09-11T09:51:46.513 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.513 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.513 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:46.514 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.904312+0000", 2024-09-11T09:51:46.514 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.0230000000000004e-06 2024-09-11T09:51:46.514 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.514 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.514 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:46.514 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.904395+0000", 2024-09-11T09:51:46.514 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.2355999999999996e-05 2024-09-11T09:51:46.514 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.514 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.514 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:46.514 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.904499+0000", 2024-09-11T09:51:46.514 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000104672 2024-09-11T09:51:46.514 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.515 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.515 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1", 2024-09-11T09:51:46.515 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.904594+0000", 2024-09-11T09:51:46.515 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.5129000000000003e-05 2024-09-11T09:51:46.515 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.515 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.515 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:46.515 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.918183+0000", 2024-09-11T09:51:46.515 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.013588355999999999 2024-09-11T09:51:46.515 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.515 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.515 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:46.516 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919053+0000", 2024-09-11T09:51:46.516 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00087066800000000001 2024-09-11T09:51:46.516 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.516 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.516 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:46.516 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919125+0000", 2024-09-11T09:51:46.516 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.1333000000000003e-05 2024-09-11T09:51:46.516 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.516 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.516 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:46.516 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919172+0000", 2024-09-11T09:51:46.516 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.7725000000000001e-05 2024-09-11T09:51:46.517 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:46.517 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:46.517 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:46.517 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.517 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.517 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4688.0:42 2.3 e53/15 2:c54742d8:::rbd_data.1250f5e25b80.0000000000000006:head v 53'2, mlcod=0'0)", 2024-09-11T09:51:46.517 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.939272+0000", 2024-09-11T09:51:46.517 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 23.570705662000002, 2024-09-11T09:51:46.517 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.010821958, 2024-09-11T09:51:46.517 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:46.517 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:46.518 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:46.518 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:46.518 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.518 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:46.518 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.939272+0000", 2024-09-11T09:51:46.518 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:46.518 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.518 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.518 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:46.518 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.939272+0000", 2024-09-11T09:51:46.518 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:46.519 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.519 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.519 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:46.519 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.939273+0000", 2024-09-11T09:51:46.519 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.7899999999999997e-07 2024-09-11T09:51:46.519 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.519 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.519 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:46.519 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.944403+0000", 2024-09-11T09:51:46.519 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0051302129999999998 2024-09-11T09:51:46.519 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.519 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.519 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:46.520 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.944413+0000", 2024-09-11T09:51:46.520 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0182e-05 2024-09-11T09:51:46.520 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.520 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.520 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:46.520 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.944420+0000", 2024-09-11T09:51:46.520 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.7000000000000002e-06 2024-09-11T09:51:46.520 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.520 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.520 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:46.520 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.944482+0000", 2024-09-11T09:51:46.520 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.1911999999999999e-05 2024-09-11T09:51:46.521 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.521 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.521 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:46.521 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.944508+0000", 2024-09-11T09:51:46.521 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.6582000000000001e-05 2024-09-11T09:51:46.521 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.521 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.521 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:46.521 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.950050+0000", 2024-09-11T09:51:46.521 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0055414080000000003 2024-09-11T09:51:46.521 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.521 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.521 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:46.522 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.950094+0000", 2024-09-11T09:51:46.522 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.4082000000000002e-05 2024-09-11T09:51:46.522 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:46.522 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:46.522 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:46.522 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.522 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.522 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:47 2.1 2:95f53b3e:::rbd_data.1250f5e25b80.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:46.522 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.948330+0000", 2024-09-11T09:51:46.522 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 23.561647317999999, 2024-09-11T09:51:46.522 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.025156603999999999, 2024-09-11T09:51:46.522 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:46.522 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:46.522 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:46.523 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:46.523 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:51:46.523 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:51:46.523 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 47 2024-09-11T09:51:46.523 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.523 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:46.523 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.523 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:46.523 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.948330+0000", 2024-09-11T09:51:46.523 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:46.523 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.523 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.523 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:46.524 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.948330+0000", 2024-09-11T09:51:46.524 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:46.524 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.524 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.524 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:46.524 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.948333+0000", 2024-09-11T09:51:46.524 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.481e-06 2024-09-11T09:51:46.524 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.524 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.524 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:46.524 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.960263+0000", 2024-09-11T09:51:46.524 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.011930671 2024-09-11T09:51:46.524 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.525 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.525 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:46.525 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.960289+0000", 2024-09-11T09:51:46.525 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.5573000000000001e-05 2024-09-11T09:51:46.525 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.525 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.525 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:46.525 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.960304+0000", 2024-09-11T09:51:46.525 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.4647000000000001e-05 2024-09-11T09:51:46.525 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.525 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.525 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:46.525 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.960390+0000", 2024-09-11T09:51:46.526 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.6088999999999995e-05 2024-09-11T09:51:46.526 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.526 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.526 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:46.526 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.960562+0000", 2024-09-11T09:51:46.526 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00017237400000000001 2024-09-11T09:51:46.526 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.526 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.526 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1", 2024-09-11T09:51:46.526 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.960725+0000", 2024-09-11T09:51:46.526 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00016321599999999999 2024-09-11T09:51:46.526 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.527 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.527 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:46.527 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.969465+0000", 2024-09-11T09:51:46.527 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0087396899999999996 2024-09-11T09:51:46.527 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.527 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.527 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:46.527 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.973393+0000", 2024-09-11T09:51:46.527 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0039274669999999996 2024-09-11T09:51:46.527 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.527 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.527 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:46.527 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.973443+0000", 2024-09-11T09:51:46.527 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.0340000000000003e-05 2024-09-11T09:51:46.528 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.528 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.528 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:46.528 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.973487+0000", 2024-09-11T09:51:46.528 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.4056e-05 2024-09-11T09:51:46.528 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:46.528 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:46.528 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:46.528 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.528 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.528 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:61 2.1 2:9af038b1:::rbd_data.1250f5e25b80.000000000000000f:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:46.528 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.975764+0000", 2024-09-11T09:51:46.528 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 23.534213793999999, 2024-09-11T09:51:46.529 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.016158891000000002, 2024-09-11T09:51:46.529 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:46.529 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:46.529 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:46.529 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:46.529 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:51:46.529 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:51:46.529 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 61 2024-09-11T09:51:46.529 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.530 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:46.530 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.530 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:46.530 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.975764+0000", 2024-09-11T09:51:46.530 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:46.530 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.530 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.530 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:46.530 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.975764+0000", 2024-09-11T09:51:46.530 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:46.530 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.530 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.530 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:46.530 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.975766+0000", 2024-09-11T09:51:46.531 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.767e-06 2024-09-11T09:51:46.531 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.531 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.531 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:46.531 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.983204+0000", 2024-09-11T09:51:46.531 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0074388400000000004 2024-09-11T09:51:46.531 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.531 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.531 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:46.531 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.983226+0000", 2024-09-11T09:51:46.531 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.1279000000000001e-05 2024-09-11T09:51:46.531 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.531 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.532 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:46.532 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.983238+0000", 2024-09-11T09:51:46.532 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2563e-05 2024-09-11T09:51:46.532 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.532 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.532 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:46.532 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.983318+0000", 2024-09-11T09:51:46.532 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.0075999999999997e-05 2024-09-11T09:51:46.532 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.532 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.532 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:46.532 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.983429+0000", 2024-09-11T09:51:46.532 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000110618 2024-09-11T09:51:46.533 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.533 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.533 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1", 2024-09-11T09:51:46.533 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.983525+0000", 2024-09-11T09:51:46.533 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.603e-05 2024-09-11T09:51:46.533 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.533 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.533 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:46.533 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.989004+0000", 2024-09-11T09:51:46.533 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0054785429999999998 2024-09-11T09:51:46.533 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.533 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.533 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:46.533 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.991788+0000", 2024-09-11T09:51:46.534 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0027845729999999998 2024-09-11T09:51:46.534 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.534 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.534 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:46.534 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.991870+0000", 2024-09-11T09:51:46.534 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.1567e-05 2024-09-11T09:51:46.534 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.534 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.534 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:46.534 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.991923+0000", 2024-09-11T09:51:46.534 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.3035000000000003e-05 2024-09-11T09:51:46.534 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:46.535 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:46.535 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:46.535 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.535 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.535 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4694.0:32 2.1 2:8a862b48:ns::rbd_data.1256a0c02879.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:46.535 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.084089+0000", 2024-09-11T09:51:46.535 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 23.425888319999999, 2024-09-11T09:51:46.535 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.020630553999999999, 2024-09-11T09:51:46.535 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:46.535 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:46.535 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:46.535 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:46.535 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4694", 2024-09-11T09:51:46.535 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:51:46.536 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 32 2024-09-11T09:51:46.536 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.536 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:46.536 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.536 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:46.536 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.084089+0000", 2024-09-11T09:51:46.536 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:46.536 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.536 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.536 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:46.536 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.084089+0000", 2024-09-11T09:51:46.536 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:46.536 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.537 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.537 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:46.537 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.084090+0000", 2024-09-11T09:51:46.537 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.2900000000000003e-07 2024-09-11T09:51:46.537 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.537 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.537 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:46.537 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.090866+0000", 2024-09-11T09:51:46.537 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0067756609999999997 2024-09-11T09:51:46.537 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.537 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.537 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:46.538 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.090891+0000", 2024-09-11T09:51:46.538 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.5119000000000001e-05 2024-09-11T09:51:46.538 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.538 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.538 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:46.538 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.090900+0000", 2024-09-11T09:51:46.538 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.4110000000000002e-06 2024-09-11T09:51:46.538 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.538 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.538 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:46.538 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.090968+0000", 2024-09-11T09:51:46.538 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.8041999999999997e-05 2024-09-11T09:51:46.538 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.538 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.539 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:46.539 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.091115+0000", 2024-09-11T09:51:46.539 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00014633499999999999 2024-09-11T09:51:46.539 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.539 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.539 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1", 2024-09-11T09:51:46.539 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.091284+0000", 2024-09-11T09:51:46.539 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000169836 2024-09-11T09:51:46.539 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.539 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.539 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:46.539 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.100896+0000", 2024-09-11T09:51:46.539 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0096118639999999995 2024-09-11T09:51:46.539 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.540 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.540 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:46.540 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.104604+0000", 2024-09-11T09:51:46.540 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0037074619999999999 2024-09-11T09:51:46.540 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.540 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.540 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:46.540 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.104669+0000", 2024-09-11T09:51:46.540 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.4836000000000001e-05 2024-09-11T09:51:46.540 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.540 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.540 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:46.540 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.104720+0000", 2024-09-11T09:51:46.541 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.1258999999999998e-05 2024-09-11T09:51:46.541 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:46.541 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:46.541 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:46.541 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.541 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.541 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4694.0:44 2.3 e53/15 2:cd1a7d1c:ns::rbd_data.1256a0c02879.0000000000000006:head v 53'3, mlcod=16'1)", 2024-09-11T09:51:46.541 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.120387+0000", 2024-09-11T09:51:46.542 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 23.389590667, 2024-09-11T09:51:46.542 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.011988468, 2024-09-11T09:51:46.542 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:46.542 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:46.542 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:46.542 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:46.542 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.542 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:46.542 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.120387+0000", 2024-09-11T09:51:46.542 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:46.542 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.542 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.542 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:46.543 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.120387+0000", 2024-09-11T09:51:46.543 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:46.543 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.543 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.543 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:46.543 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.120389+0000", 2024-09-11T09:51:46.543 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.018e-06 2024-09-11T09:51:46.543 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.543 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.543 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:46.543 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.127403+0000", 2024-09-11T09:51:46.544 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0070144769999999999 2024-09-11T09:51:46.544 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.544 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.544 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:46.544 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.127414+0000", 2024-09-11T09:51:46.544 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0991e-05 2024-09-11T09:51:46.544 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.544 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.544 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:46.544 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.127423+0000", 2024-09-11T09:51:46.544 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.8580000000000006e-06 2024-09-11T09:51:46.544 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.544 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.545 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:46.545 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.127468+0000", 2024-09-11T09:51:46.545 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.4830999999999998e-05 2024-09-11T09:51:46.545 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.545 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.545 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:46.545 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.127485+0000", 2024-09-11T09:51:46.545 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6838999999999999e-05 2024-09-11T09:51:46.545 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.545 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.545 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:46.545 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.132341+0000", 2024-09-11T09:51:46.545 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0048557959999999999 2024-09-11T09:51:46.546 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.546 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.546 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:46.546 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.132375+0000", 2024-09-11T09:51:46.546 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.4657999999999999e-05 2024-09-11T09:51:46.546 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:46.546 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:46.546 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:46.546 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.546 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.546 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4694.0:46 2.3 e53/15 2:ce0e1469:ns::rbd_data.1256a0c02879.0000000000000008:head v 53'4, mlcod=16'1)", 2024-09-11T09:51:46.546 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.129008+0000", 2024-09-11T09:51:46.546 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 23.380970081000001, 2024-09-11T09:51:46.547 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0087151220000000005, 2024-09-11T09:51:46.547 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:46.547 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:46.547 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:46.547 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:46.547 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.547 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:46.547 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.129008+0000", 2024-09-11T09:51:46.548 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:46.548 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.548 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.548 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:46.548 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.129008+0000", 2024-09-11T09:51:46.548 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:46.548 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.548 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.548 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:46.548 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.129009+0000", 2024-09-11T09:51:46.548 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.367e-06 2024-09-11T09:51:46.548 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.548 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.549 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:46.549 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.132839+0000", 2024-09-11T09:51:46.549 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0038297610000000001 2024-09-11T09:51:46.549 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.549 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.549 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:46.549 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.132849+0000", 2024-09-11T09:51:46.549 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0672e-05 2024-09-11T09:51:46.549 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.549 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.549 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:46.549 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.132856+0000", 2024-09-11T09:51:46.549 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.1380000000000004e-06 2024-09-11T09:51:46.550 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.550 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.550 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:46.550 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.132891+0000", 2024-09-11T09:51:46.550 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.5712e-05 2024-09-11T09:51:46.550 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.550 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.550 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:46.550 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.132907+0000", 2024-09-11T09:51:46.550 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6166000000000001e-05 2024-09-11T09:51:46.550 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.550 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.550 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:46.550 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.137681+0000", 2024-09-11T09:51:46.551 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0047735299999999998 2024-09-11T09:51:46.551 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.551 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.551 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:46.551 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.137723+0000", 2024-09-11T09:51:46.551 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.1776000000000001e-05 2024-09-11T09:51:46.551 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:46.551 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:46.551 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:46.551 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.551 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.551 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4694.0:48 2.3 e53/15 2:d2090b70:ns::rbd_data.1256a0c02879.0000000000000009:head v 53'5, mlcod=53'2)", 2024-09-11T09:51:46.551 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.135108+0000", 2024-09-11T09:51:46.551 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 23.374869184000001, 2024-09-11T09:51:46.552 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.010710679000000001, 2024-09-11T09:51:46.552 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:46.552 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:46.552 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:46.552 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:46.552 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.552 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:46.552 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.135108+0000", 2024-09-11T09:51:46.552 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:46.552 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.553 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.553 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:46.553 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.135108+0000", 2024-09-11T09:51:46.553 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:46.553 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.553 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.553 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:46.553 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.135110+0000", 2024-09-11T09:51:46.553 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.122e-06 2024-09-11T09:51:46.553 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.553 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.554 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:46.554 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.140876+0000", 2024-09-11T09:51:46.554 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.005766162 2024-09-11T09:51:46.554 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.554 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.554 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:46.554 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.140888+0000", 2024-09-11T09:51:46.554 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.201e-05 2024-09-11T09:51:46.554 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.554 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.554 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:46.554 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.140894+0000", 2024-09-11T09:51:46.554 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.1739999999999997e-06 2024-09-11T09:51:46.555 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.555 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.555 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:46.555 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.140971+0000", 2024-09-11T09:51:46.555 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.7495000000000003e-05 2024-09-11T09:51:46.555 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.555 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.555 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:46.555 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.140995+0000", 2024-09-11T09:51:46.555 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.3204999999999999e-05 2024-09-11T09:51:46.555 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.556 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.556 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:46.556 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.145757+0000", 2024-09-11T09:51:46.556 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0047627190000000003 2024-09-11T09:51:46.556 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.556 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.556 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:46.556 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.145819+0000", 2024-09-11T09:51:46.556 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.1791999999999994e-05 2024-09-11T09:51:46.556 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:46.556 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:46.556 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:46.556 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.557 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.557 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4694.0:50 2.1 2:83078786:ns::rbd_data.1256a0c02879.000000000000000a:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:46.557 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.141467+0000", 2024-09-11T09:51:46.557 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 23.36851046, 2024-09-11T09:51:46.557 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.012785889, 2024-09-11T09:51:46.557 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:46.557 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:46.557 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:46.557 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:46.557 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4694", 2024-09-11T09:51:46.557 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:51:46.558 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 50 2024-09-11T09:51:46.558 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.558 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:46.558 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.558 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:46.558 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.141467+0000", 2024-09-11T09:51:46.558 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:46.558 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.558 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.558 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:46.558 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.141467+0000", 2024-09-11T09:51:46.558 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:46.558 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.559 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.559 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:46.559 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.141468+0000", 2024-09-11T09:51:46.559 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1009999999999999e-06 2024-09-11T09:51:46.559 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.559 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.559 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:46.559 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.146432+0000", 2024-09-11T09:51:46.559 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0049640539999999999 2024-09-11T09:51:46.559 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.559 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.559 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:46.559 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.146446+0000", 2024-09-11T09:51:46.560 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3254e-05 2024-09-11T09:51:46.560 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.560 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.560 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:46.560 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.146452+0000", 2024-09-11T09:51:46.560 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.207e-06 2024-09-11T09:51:46.560 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.560 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.560 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:46.560 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.146510+0000", 2024-09-11T09:51:46.560 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.8322999999999997e-05 2024-09-11T09:51:46.560 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.560 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.561 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:46.561 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.146650+0000", 2024-09-11T09:51:46.561 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00013947100000000001 2024-09-11T09:51:46.561 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.561 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.561 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1", 2024-09-11T09:51:46.561 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.146779+0000", 2024-09-11T09:51:46.561 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000129059 2024-09-11T09:51:46.561 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.561 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.561 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:46.561 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.153778+0000", 2024-09-11T09:51:46.561 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0069997130000000003 2024-09-11T09:51:46.561 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.562 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.562 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:46.562 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.154180+0000", 2024-09-11T09:51:46.562 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00040159800000000003 2024-09-11T09:51:46.562 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.562 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.562 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:46.562 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.154218+0000", 2024-09-11T09:51:46.562 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.7795e-05 2024-09-11T09:51:46.562 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.562 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.562 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:46.563 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.154253+0000", 2024-09-11T09:51:46.563 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.5314000000000001e-05 2024-09-11T09:51:46.563 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:46.563 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:46.563 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:46.563 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.563 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.563 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4694.0:54 2.3 e53/15 2:c44f0253:ns::rbd_data.1256a0c02879.000000000000000b:head v 53'6, mlcod=53'4)", 2024-09-11T09:51:46.563 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.154926+0000", 2024-09-11T09:51:46.563 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 23.355052085000001, 2024-09-11T09:51:46.563 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0089497099999999996, 2024-09-11T09:51:46.563 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:46.563 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:46.563 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:46.564 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:46.564 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.564 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:46.564 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.154926+0000", 2024-09-11T09:51:46.564 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:46.564 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.564 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.564 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:46.564 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.154926+0000", 2024-09-11T09:51:46.564 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:46.564 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.564 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.565 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:46.565 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.154927+0000", 2024-09-11T09:51:46.565 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.7000000000000003e-07 2024-09-11T09:51:46.565 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.565 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.565 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:46.565 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.158712+0000", 2024-09-11T09:51:46.565 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0037855469999999998 2024-09-11T09:51:46.565 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.565 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.565 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:46.565 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.158726+0000", 2024-09-11T09:51:46.565 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3621000000000001e-05 2024-09-11T09:51:46.565 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.566 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.566 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:46.566 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.158732+0000", 2024-09-11T09:51:46.566 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.9089999999999997e-06 2024-09-11T09:51:46.566 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.566 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.566 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:46.566 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.158781+0000", 2024-09-11T09:51:46.566 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.8900000000000003e-05 2024-09-11T09:51:46.566 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.566 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.566 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:46.566 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.158805+0000", 2024-09-11T09:51:46.566 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.44e-05 2024-09-11T09:51:46.567 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.567 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.567 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:46.567 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.163826+0000", 2024-09-11T09:51:46.567 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0050206399999999998 2024-09-11T09:51:46.567 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.567 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.567 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:46.567 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.163875+0000", 2024-09-11T09:51:46.567 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.9722999999999998e-05 2024-09-11T09:51:46.567 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:46.567 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:46.567 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:46.567 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.568 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.568 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4694.0:56 2.3 e53/15 2:c462f029:ns::rbd_data.1256a0c02879.000000000000000c:head v 53'7, mlcod=53'4)", 2024-09-11T09:51:46.568 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.159549+0000", 2024-09-11T09:51:46.568 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 23.350428379, 2024-09-11T09:51:46.568 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.015591666000000001, 2024-09-11T09:51:46.568 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:46.568 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:46.568 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:46.568 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:46.568 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.568 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:46.568 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.159549+0000", 2024-09-11T09:51:46.568 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:46.569 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.569 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.569 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:46.569 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.159549+0000", 2024-09-11T09:51:46.569 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:46.569 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.569 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.569 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:46.569 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.159550+0000", 2024-09-11T09:51:46.569 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.1500000000000004e-07 2024-09-11T09:51:46.569 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.569 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.569 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:46.569 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.167834+0000", 2024-09-11T09:51:46.570 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0082840220000000003 2024-09-11T09:51:46.570 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.570 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.570 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:46.570 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.167857+0000", 2024-09-11T09:51:46.570 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.3337e-05 2024-09-11T09:51:46.570 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.570 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.570 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:46.570 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.167869+0000", 2024-09-11T09:51:46.570 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1556000000000001e-05 2024-09-11T09:51:46.570 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.570 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.570 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:46.571 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.167939+0000", 2024-09-11T09:51:46.571 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.0674999999999997e-05 2024-09-11T09:51:46.571 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.571 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.571 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:46.571 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.167967+0000", 2024-09-11T09:51:46.571 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.7322999999999999e-05 2024-09-11T09:51:46.571 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.571 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.571 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:46.571 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.175079+0000", 2024-09-11T09:51:46.571 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0071126150000000001 2024-09-11T09:51:46.571 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.572 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.572 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:46.572 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.175141+0000", 2024-09-11T09:51:46.572 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.1523e-05 2024-09-11T09:51:46.572 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:46.572 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:46.572 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:46.572 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.572 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.572 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.5002.0:25 2.1 2:8654721b:::rbd_data.1250f5e25b80.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,write 393216~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-09-11T09:51:46.572 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:38.367683+0000", 2024-09-11T09:51:46.572 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 8.1422943829999994, 2024-09-11T09:51:46.572 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.005437641, 2024-09-11T09:51:46.573 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:46.573 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:46.573 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:46.573 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:46.573 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.5002", 2024-09-11T09:51:46.573 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/2204440692", 2024-09-11T09:51:46.573 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 25 2024-09-11T09:51:46.573 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.573 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:46.573 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.573 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:46.573 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.367683+0000", 2024-09-11T09:51:46.573 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:46.574 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.574 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.574 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:46.574 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.367683+0000", 2024-09-11T09:51:46.574 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:46.574 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.574 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.574 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:46.574 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.367684+0000", 2024-09-11T09:51:46.574 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.9699999999999995e-07 2024-09-11T09:51:46.574 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.574 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.574 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:46.575 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.367842+0000", 2024-09-11T09:51:46.575 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000158362 2024-09-11T09:51:46.575 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.575 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.575 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:46.575 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.367856+0000", 2024-09-11T09:51:46.575 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3360000000000001e-05 2024-09-11T09:51:46.575 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.575 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.575 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:46.575 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.367863+0000", 2024-09-11T09:51:46.575 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.1609999999999997e-06 2024-09-11T09:51:46.576 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.576 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.576 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:46.576 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.369486+0000", 2024-09-11T09:51:46.576 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0016228449999999999 2024-09-11T09:51:46.576 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.576 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.576 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:46.576 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.369538+0000", 2024-09-11T09:51:46.576 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.2138000000000001e-05 2024-09-11T09:51:46.576 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.576 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.576 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1", 2024-09-11T09:51:46.576 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.369620+0000", 2024-09-11T09:51:46.577 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.2235000000000001e-05 2024-09-11T09:51:46.577 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.577 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.577 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:46.577 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.372192+0000", 2024-09-11T09:51:46.577 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.002572024 2024-09-11T09:51:46.577 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.577 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.577 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:46.577 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.373045+0000", 2024-09-11T09:51:46.577 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00085311000000000002 2024-09-11T09:51:46.577 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.577 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.578 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:46.578 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.373086+0000", 2024-09-11T09:51:46.578 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.0731e-05 2024-09-11T09:51:46.578 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.578 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.578 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:46.578 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.373121+0000", 2024-09-11T09:51:46.578 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.4978000000000003e-05 2024-09-11T09:51:46.578 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:46.578 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:46.578 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:46.578 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.578 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.579 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.5002.0:26 2.1 2:8654721b:::rbd_data.1250f5e25b80.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,write 524288~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-09-11T09:51:46.579 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:38.367903+0000", 2024-09-11T09:51:46.579 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 8.1420742700000002, 2024-09-11T09:51:46.579 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0054161929999999997, 2024-09-11T09:51:46.579 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:46.579 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:46.579 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:46.579 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:46.579 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.5002", 2024-09-11T09:51:46.579 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/2204440692", 2024-09-11T09:51:46.579 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 26 2024-09-11T09:51:46.579 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.579 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:46.579 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.580 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:46.580 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.367903+0000", 2024-09-11T09:51:46.580 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:46.580 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.580 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.580 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:46.580 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.367903+0000", 2024-09-11T09:51:46.580 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:46.580 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.580 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.580 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:46.580 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.367904+0000", 2024-09-11T09:51:46.581 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.5199999999999996e-07 2024-09-11T09:51:46.581 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.581 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.581 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:46.581 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.368108+0000", 2024-09-11T09:51:46.581 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00020412499999999999 2024-09-11T09:51:46.581 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.581 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.581 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:46.581 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.368120+0000", 2024-09-11T09:51:46.581 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1503e-05 2024-09-11T09:51:46.581 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.581 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.582 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:46.582 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.368127+0000", 2024-09-11T09:51:46.582 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.4800000000000004e-06 2024-09-11T09:51:46.582 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.582 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.582 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:46.582 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.370401+0000", 2024-09-11T09:51:46.582 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0022742069999999999 2024-09-11T09:51:46.582 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.582 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.582 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:46.582 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.370494+0000", 2024-09-11T09:51:46.582 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.2161000000000001e-05 2024-09-11T09:51:46.583 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.583 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.583 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1", 2024-09-11T09:51:46.583 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.370644+0000", 2024-09-11T09:51:46.583 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00015055299999999999 2024-09-11T09:51:46.583 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.583 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.583 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:46.583 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.373144+0000", 2024-09-11T09:51:46.583 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.002499869 2024-09-11T09:51:46.583 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.583 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.583 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:46.584 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.373218+0000", 2024-09-11T09:51:46.584 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.4210999999999998e-05 2024-09-11T09:51:46.584 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.584 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.584 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:46.584 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.373274+0000", 2024-09-11T09:51:46.584 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.6069999999999997e-05 2024-09-11T09:51:46.584 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.584 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.584 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:46.584 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.373320+0000", 2024-09-11T09:51:46.584 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.5361999999999997e-05 2024-09-11T09:51:46.584 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:46.585 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:46.585 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:46.585 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.585 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.585 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.5002.0:27 2.1 2:8654721b:::rbd_data.1250f5e25b80.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,write 655360~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-09-11T09:51:46.585 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:38.368143+0000", 2024-09-11T09:51:46.585 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 8.1418347549999996, 2024-09-11T09:51:46.585 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0053157639999999997, 2024-09-11T09:51:46.585 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:46.585 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:46.586 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:46.586 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:46.586 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.5002", 2024-09-11T09:51:46.586 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/2204440692", 2024-09-11T09:51:46.586 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 27 2024-09-11T09:51:46.586 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.586 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:46.586 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.586 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:46.586 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.368143+0000", 2024-09-11T09:51:46.586 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:46.586 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.586 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.587 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:46.587 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.368143+0000", 2024-09-11T09:51:46.587 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:46.587 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.587 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.587 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:46.587 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.368144+0000", 2024-09-11T09:51:46.587 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.1800000000000005e-07 2024-09-11T09:51:46.587 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.587 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.587 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:46.587 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.368286+0000", 2024-09-11T09:51:46.587 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000141934 2024-09-11T09:51:46.588 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.588 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.588 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:46.588 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.368298+0000", 2024-09-11T09:51:46.588 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2926e-05 2024-09-11T09:51:46.588 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.588 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.588 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:46.588 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.368305+0000", 2024-09-11T09:51:46.588 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.7100000000000001e-06 2024-09-11T09:51:46.588 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.588 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.588 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:46.589 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.371358+0000", 2024-09-11T09:51:46.589 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0030531450000000002 2024-09-11T09:51:46.589 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.589 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.589 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:46.589 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.371448+0000", 2024-09-11T09:51:46.589 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.9899000000000001e-05 2024-09-11T09:51:46.589 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.589 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.589 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1", 2024-09-11T09:51:46.589 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.371593+0000", 2024-09-11T09:51:46.589 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000144948 2024-09-11T09:51:46.589 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.590 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.590 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:46.590 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.373387+0000", 2024-09-11T09:51:46.590 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0017935340000000001 2024-09-11T09:51:46.590 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.590 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.590 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:46.590 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.373397+0000", 2024-09-11T09:51:46.590 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0550999999999999e-05 2024-09-11T09:51:46.590 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.590 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.590 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:46.590 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.373428+0000", 2024-09-11T09:51:46.590 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.0731000000000001e-05 2024-09-11T09:51:46.591 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.591 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.591 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:46.591 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.373459+0000", 2024-09-11T09:51:46.591 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.0667999999999998e-05 2024-09-11T09:51:46.591 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:46.591 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:46.591 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:46.591 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.591 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.591 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.5002.0:28 2.1 2:8654721b:::rbd_data.1250f5e25b80.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,write 786432~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-09-11T09:51:46.591 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:38.368321+0000", 2024-09-11T09:51:46.591 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 8.1416570579999998, 2024-09-11T09:51:46.592 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0060769780000000002, 2024-09-11T09:51:46.592 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:46.592 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:46.592 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:46.592 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:46.592 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.5002", 2024-09-11T09:51:46.592 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/2204440692", 2024-09-11T09:51:46.592 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 28 2024-09-11T09:51:46.592 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.592 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:46.592 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.592 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:46.592 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.368321+0000", 2024-09-11T09:51:46.592 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:46.593 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.593 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.593 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:46.593 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.368321+0000", 2024-09-11T09:51:46.593 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:46.593 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.593 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.593 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:46.593 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.368321+0000", 2024-09-11T09:51:46.593 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.6599999999999996e-07 2024-09-11T09:51:46.593 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.593 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.593 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:46.594 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.368452+0000", 2024-09-11T09:51:46.594 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000130858 2024-09-11T09:51:46.594 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.594 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.594 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:46.594 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.368477+0000", 2024-09-11T09:51:46.594 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.4627e-05 2024-09-11T09:51:46.594 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.594 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.594 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:46.594 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.368486+0000", 2024-09-11T09:51:46.594 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.2070000000000001e-06 2024-09-11T09:51:46.594 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.594 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.595 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:46.595 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.372253+0000", 2024-09-11T09:51:46.595 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0037666779999999999 2024-09-11T09:51:46.595 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.595 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.595 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:46.595 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.372346+0000", 2024-09-11T09:51:46.595 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.3222999999999998e-05 2024-09-11T09:51:46.595 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.595 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.595 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1", 2024-09-11T09:51:46.595 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.372489+0000", 2024-09-11T09:51:46.595 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00014329300000000001 2024-09-11T09:51:46.595 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.596 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.596 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:46.596 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.374251+0000", 2024-09-11T09:51:46.596 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0017618550000000001 2024-09-11T09:51:46.596 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.596 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.596 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:46.596 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.374293+0000", 2024-09-11T09:51:46.596 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.1891999999999998e-05 2024-09-11T09:51:46.596 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.596 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.596 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:46.596 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.374366+0000", 2024-09-11T09:51:46.597 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.3446000000000001e-05 2024-09-11T09:51:46.597 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.597 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.597 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:46.597 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.374398+0000", 2024-09-11T09:51:46.597 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.1232999999999997e-05 2024-09-11T09:51:46.597 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:46.597 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:46.597 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:46.597 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.597 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.597 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.5002.0:29 2.1 2:8654721b:::rbd_data.1250f5e25b80.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,write 917504~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-09-11T09:51:46.597 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:38.368498+0000", 2024-09-11T09:51:46.598 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 8.1414792200000008, 2024-09-11T09:51:46.598 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0069133370000000003, 2024-09-11T09:51:46.598 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:46.598 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:46.598 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:46.598 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:46.598 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.5002", 2024-09-11T09:51:46.598 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/2204440692", 2024-09-11T09:51:46.598 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 29 2024-09-11T09:51:46.598 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.598 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:46.599 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.599 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:46.599 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.368498+0000", 2024-09-11T09:51:46.599 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:46.599 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.599 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.599 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:46.599 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.368498+0000", 2024-09-11T09:51:46.599 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:46.599 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.599 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.599 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:46.600 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.368499+0000", 2024-09-11T09:51:46.600 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.2200000000000003e-07 2024-09-11T09:51:46.600 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.600 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.600 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:46.600 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.368604+0000", 2024-09-11T09:51:46.600 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000104627 2024-09-11T09:51:46.600 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.600 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.600 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:46.600 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.368614+0000", 2024-09-11T09:51:46.600 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0139999999999999e-05 2024-09-11T09:51:46.601 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.601 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.601 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:46.601 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.368620+0000", 2024-09-11T09:51:46.601 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.5100000000000004e-06 2024-09-11T09:51:46.601 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.601 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.601 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:46.601 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.373501+0000", 2024-09-11T09:51:46.601 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0048804749999999996 2024-09-11T09:51:46.601 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.601 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.601 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:46.601 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.373570+0000", 2024-09-11T09:51:46.602 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.8831000000000006e-05 2024-09-11T09:51:46.602 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.602 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.602 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1", 2024-09-11T09:51:46.602 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.373674+0000", 2024-09-11T09:51:46.602 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000103755 2024-09-11T09:51:46.602 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.602 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.602 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:46.602 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.374588+0000", 2024-09-11T09:51:46.602 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00091421700000000005 2024-09-11T09:51:46.602 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.602 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.603 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:46.603 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.375350+0000", 2024-09-11T09:51:46.603 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00076217699999999995 2024-09-11T09:51:46.603 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.603 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.603 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:46.603 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.375384+0000", 2024-09-11T09:51:46.603 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.4486000000000002e-05 2024-09-11T09:51:46.603 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.603 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.603 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:46.603 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.375412+0000", 2024-09-11T09:51:46.603 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.7396999999999999e-05 2024-09-11T09:51:46.604 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:46.604 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:46.604 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:46.604 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.604 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.604 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.5002.0:34 2.1 2:8654721b:::rbd_data.1250f5e25b80.0000000000000000:head [write 1572864~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-09-11T09:51:46.604 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:38.376433+0000", 2024-09-11T09:51:46.604 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 8.1335445479999997, 2024-09-11T09:51:46.604 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0059348530000000004, 2024-09-11T09:51:46.604 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:46.604 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:46.604 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:46.604 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:46.605 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.5002", 2024-09-11T09:51:46.605 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/2204440692", 2024-09-11T09:51:46.605 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 34 2024-09-11T09:51:46.605 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.605 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:46.605 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.605 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:46.605 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.376433+0000", 2024-09-11T09:51:46.605 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:46.605 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.605 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.605 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:46.605 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.376433+0000", 2024-09-11T09:51:46.605 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:46.606 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.606 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.606 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:46.606 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.376434+0000", 2024-09-11T09:51:46.606 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.0200000000000002e-07 2024-09-11T09:51:46.606 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.606 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.606 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:46.606 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.376505+0000", 2024-09-11T09:51:46.606 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.1273999999999996e-05 2024-09-11T09:51:46.606 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.606 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.606 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:46.607 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.376513+0000", 2024-09-11T09:51:46.607 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.7419999999999997e-06 2024-09-11T09:51:46.607 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.607 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.607 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:46.607 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.376517+0000", 2024-09-11T09:51:46.607 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.6e-06 2024-09-11T09:51:46.607 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.607 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.607 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:46.607 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.379223+0000", 2024-09-11T09:51:46.607 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0027053149999999998 2024-09-11T09:51:46.607 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.607 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.608 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:46.608 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.379273+0000", 2024-09-11T09:51:46.608 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.0927000000000002e-05 2024-09-11T09:51:46.608 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.608 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.608 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1", 2024-09-11T09:51:46.608 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.379353+0000", 2024-09-11T09:51:46.608 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.9431999999999994e-05 2024-09-11T09:51:46.608 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.608 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.608 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:46.608 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.382252+0000", 2024-09-11T09:51:46.608 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0028991569999999999 2024-09-11T09:51:46.609 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.609 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.609 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:46.609 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.382303+0000", 2024-09-11T09:51:46.609 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.058e-05 2024-09-11T09:51:46.609 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.609 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.609 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:46.609 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.382339+0000", 2024-09-11T09:51:46.609 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.6089e-05 2024-09-11T09:51:46.609 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.609 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.609 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:46.610 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.382368+0000", 2024-09-11T09:51:46.610 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.9235e-05 2024-09-11T09:51:46.610 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:46.610 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:46.610 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:46.610 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.610 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.610 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.5002.0:35 2.1 2:8654721b:::rbd_data.1250f5e25b80.0000000000000000:head [write 1703936~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-09-11T09:51:46.610 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:38.376525+0000", 2024-09-11T09:51:46.610 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 8.1334523799999996, 2024-09-11T09:51:46.610 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0060109220000000001, 2024-09-11T09:51:46.610 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:46.610 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:46.610 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:46.611 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:46.611 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.5002", 2024-09-11T09:51:46.611 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/2204440692", 2024-09-11T09:51:46.611 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 35 2024-09-11T09:51:46.611 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.611 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:46.611 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.611 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:46.611 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.376525+0000", 2024-09-11T09:51:46.611 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:46.611 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.611 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.611 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:46.611 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.376525+0000", 2024-09-11T09:51:46.612 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:46.612 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.612 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.612 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:46.612 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.376526+0000", 2024-09-11T09:51:46.612 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.32e-07 2024-09-11T09:51:46.612 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.612 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.613 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:46.613 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.376606+0000", 2024-09-11T09:51:46.613 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.0539999999999998e-05 2024-09-11T09:51:46.613 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.613 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.613 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:46.613 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.376615+0000", 2024-09-11T09:51:46.613 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.3310000000000004e-06 2024-09-11T09:51:46.613 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.613 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.613 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:46.613 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.376619+0000", 2024-09-11T09:51:46.613 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.9600000000000002e-06 2024-09-11T09:51:46.614 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.614 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.614 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:46.614 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.380083+0000", 2024-09-11T09:51:46.614 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.003464647 2024-09-11T09:51:46.614 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.614 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.614 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:46.614 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.380132+0000", 2024-09-11T09:51:46.614 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.8507000000000002e-05 2024-09-11T09:51:46.614 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.614 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.614 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1", 2024-09-11T09:51:46.615 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.380206+0000", 2024-09-11T09:51:46.615 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.4480000000000005e-05 2024-09-11T09:51:46.615 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.615 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.615 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:46.615 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.382427+0000", 2024-09-11T09:51:46.615 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00222091 2024-09-11T09:51:46.615 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.615 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.615 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:46.615 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.382443+0000", 2024-09-11T09:51:46.615 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.5586999999999998e-05 2024-09-11T09:51:46.615 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.615 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.616 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:46.616 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.382495+0000", 2024-09-11T09:51:46.616 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.2023e-05 2024-09-11T09:51:46.616 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.616 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.616 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:46.616 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.382536+0000", 2024-09-11T09:51:46.616 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.1505000000000003e-05 2024-09-11T09:51:46.616 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:46.616 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:46.616 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:46.616 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.616 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.617 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.5002.0:36 2.1 2:8654721b:::rbd_data.1250f5e25b80.0000000000000000:head [write 1835008~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-09-11T09:51:46.617 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:38.376629+0000", 2024-09-11T09:51:46.617 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 8.1333486330000007, 2024-09-11T09:51:46.617 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.006032567, 2024-09-11T09:51:46.617 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:46.617 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:46.617 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:46.617 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:46.617 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.5002", 2024-09-11T09:51:46.617 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/2204440692", 2024-09-11T09:51:46.617 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 36 2024-09-11T09:51:46.617 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.617 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:46.618 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.618 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:46.618 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.376629+0000", 2024-09-11T09:51:46.618 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:46.618 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.618 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.618 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:46.618 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.376629+0000", 2024-09-11T09:51:46.618 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:46.618 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.618 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.619 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:46.619 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.376630+0000", 2024-09-11T09:51:46.619 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.8100000000000003e-07 2024-09-11T09:51:46.619 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.619 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.619 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:46.619 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.376831+0000", 2024-09-11T09:51:46.619 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000201576 2024-09-11T09:51:46.619 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.619 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.619 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:46.619 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.376843+0000", 2024-09-11T09:51:46.619 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2326e-05 2024-09-11T09:51:46.620 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.620 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.620 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:46.620 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.376850+0000", 2024-09-11T09:51:46.620 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.0859999999999996e-06 2024-09-11T09:51:46.620 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.620 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.620 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:46.620 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.380822+0000", 2024-09-11T09:51:46.620 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0039714520000000003 2024-09-11T09:51:46.620 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.620 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.620 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:46.621 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.380871+0000", 2024-09-11T09:51:46.621 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.9314999999999998e-05 2024-09-11T09:51:46.621 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.621 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.621 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1", 2024-09-11T09:51:46.621 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.380950+0000", 2024-09-11T09:51:46.621 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.8592999999999998e-05 2024-09-11T09:51:46.621 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.621 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.621 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:46.621 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.382544+0000", 2024-09-11T09:51:46.621 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.001594631 2024-09-11T09:51:46.622 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.622 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.622 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:46.622 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.382586+0000", 2024-09-11T09:51:46.622 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.1118999999999997e-05 2024-09-11T09:51:46.622 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.622 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.622 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:46.622 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.382628+0000", 2024-09-11T09:51:46.622 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.2548e-05 2024-09-11T09:51:46.622 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.622 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.622 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:46.623 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.382662+0000", 2024-09-11T09:51:46.623 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.3439999999999998e-05 2024-09-11T09:51:46.623 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:46.623 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:46.623 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:46.623 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.623 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.623 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.5002.0:37 2.1 2:8654721b:::rbd_data.1250f5e25b80.0000000000000000:head [write 1966080~131072] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-09-11T09:51:46.623 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:38.376864+0000", 2024-09-11T09:51:46.623 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 8.1331137689999995, 2024-09-11T09:51:46.623 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0062377370000000001, 2024-09-11T09:51:46.623 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:46.623 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:46.624 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:46.624 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:46.624 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.5002", 2024-09-11T09:51:46.624 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/2204440692", 2024-09-11T09:51:46.624 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 37 2024-09-11T09:51:46.624 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.624 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:46.624 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.624 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:46.624 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.376864+0000", 2024-09-11T09:51:46.624 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:46.624 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.624 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.625 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:46.625 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.376864+0000", 2024-09-11T09:51:46.625 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:46.625 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.625 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.625 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:46.625 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.376865+0000", 2024-09-11T09:51:46.625 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.1600000000000001e-07 2024-09-11T09:51:46.625 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.625 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.625 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:46.625 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.376976+0000", 2024-09-11T09:51:46.625 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000111581 2024-09-11T09:51:46.626 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.626 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.626 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:46.626 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.376991+0000", 2024-09-11T09:51:46.626 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.4649e-05 2024-09-11T09:51:46.626 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.626 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.626 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:46.626 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.376996+0000", 2024-09-11T09:51:46.626 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.4920000000000001e-06 2024-09-11T09:51:46.626 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.626 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.626 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:46.626 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.381543+0000", 2024-09-11T09:51:46.627 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0045464839999999999 2024-09-11T09:51:46.627 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.627 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.627 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:46.627 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.381631+0000", 2024-09-11T09:51:46.627 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.865e-05 2024-09-11T09:51:46.627 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.627 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.627 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1", 2024-09-11T09:51:46.627 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.381777+0000", 2024-09-11T09:51:46.627 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00014590800000000001 2024-09-11T09:51:46.627 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.627 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.627 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:46.628 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.382671+0000", 2024-09-11T09:51:46.628 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00089374400000000003 2024-09-11T09:51:46.628 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.628 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.628 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:46.628 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.383050+0000", 2024-09-11T09:51:46.628 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00037862600000000002 2024-09-11T09:51:46.628 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.628 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.628 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:46.628 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.383079+0000", 2024-09-11T09:51:46.628 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.9541999999999999e-05 2024-09-11T09:51:46.628 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:46.628 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:46.629 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:46.629 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:38.383102+0000", 2024-09-11T09:51:46.629 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.2345e-05 2024-09-11T09:51:46.629 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:46.629 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:46.629 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:46.629 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:46.629 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:46.629 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:51:46.630 INFO:tasks.workunit.client.0.smithi107.stderr:+ PID=28973 2024-09-11T09:51:46.631 INFO:tasks.workunit.client.0.smithi107.stderr:+ test -n 28973 2024-09-11T09:51:46.631 INFO:tasks.workunit.client.0.smithi107.stderr:+ ps -p 28973 -C rbd-nbd 2024-09-11T09:51:46.631 INFO:tasks.workunit.client.0.smithi107.stdout: PID TTY TIME CMD 2024-09-11T09:51:46.631 INFO:tasks.workunit.client.0.smithi107.stdout: 28925 ? 00:00:00 rbd-nbd 2024-09-11T09:51:46.631 INFO:tasks.workunit.client.0.smithi107.stdout: 28973 ? 00:00:00 rbd-nbd 2024-09-11T09:51:46.632 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-11T09:51:46.632 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:51:46.631+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:51:46.634 INFO:tasks.workunit.client.0.smithi107.stderr:+ unmap_device rbd/ns/testrbdnbd27121 28973 2024-09-11T09:51:46.634 INFO:tasks.workunit.client.0.smithi107.stderr:+ local args=rbd/ns/testrbdnbd27121 2024-09-11T09:51:46.634 INFO:tasks.workunit.client.0.smithi107.stderr:+ local pid=28973 2024-09-11T09:51:46.634 INFO:tasks.workunit.client.0.smithi107.stderr:+ _sudo rbd device --device-type nbd unmap rbd/ns/testrbdnbd27121 2024-09-11T09:51:46.634 INFO:tasks.workunit.client.0.smithi107.stderr:+ local cmd 2024-09-11T09:51:46.634 INFO:tasks.workunit.client.0.smithi107.stderr:++ id -u 2024-09-11T09:51:46.634 INFO:tasks.workunit.client.0.smithi107.stderr:+ '[' 1000 -eq 0 ']' 2024-09-11T09:51:46.634 INFO:tasks.workunit.client.0.smithi107.stderr:++ which rbd 2024-09-11T09:51:46.634 INFO:tasks.workunit.client.0.smithi107.stderr:+ cmd=/usr/bin/rbd 2024-09-11T09:51:46.634 INFO:tasks.workunit.client.0.smithi107.stderr:+ shift 2024-09-11T09:51:46.634 INFO:tasks.workunit.client.0.smithi107.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd unmap rbd/ns/testrbdnbd27121 2024-09-11T09:51:46.634 INFO:tasks.workunit.client.0.smithi107.stderr:+ rbd device --device-type nbd list 2024-09-11T09:51:46.634 INFO:tasks.workunit.client.0.smithi107.stderr:+ expect_false grep '^28973\b' 2024-09-11T09:51:46.635 INFO:tasks.workunit.client.0.smithi107.stderr:+ grep '^28973\b' 2024-09-11T09:51:46.666 INFO:tasks.workunit.client.0.smithi107.stderr:+ return 0 2024-09-11T09:51:46.667 INFO:tasks.workunit.client.0.smithi107.stderr:+ ps -C rbd-nbd 2024-09-11T09:51:46.667 INFO:tasks.workunit.client.0.smithi107.stderr:+ expect_false grep '^ *28973\b' 2024-09-11T09:51:46.667 INFO:tasks.workunit.client.0.smithi107.stderr:+ grep '^ *28973\b' 2024-09-11T09:51:46.678 INFO:tasks.workunit.client.0.smithi107.stderr:+ return 0 2024-09-11T09:51:46.678 INFO:tasks.workunit.client.0.smithi107.stderr:+ sleep 0.5 2024-09-11T09:51:46.731 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:51:46.732+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:51:46.748 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-11T09:51:46.748 INFO:teuthology.orchestra.run.smithi096.stdout: "ops": [], 2024-09-11T09:51:46.748 INFO:teuthology.orchestra.run.smithi096.stdout: "num_ops": 0 2024-09-11T09:51:46.748 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-11T09:51:46.757 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-11T09:51:46.832 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-09-11T09:51:46.833+0000 7f7b6c7d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31904) UID: 0 2024-09-11T09:51:46.860 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-11T09:51:46.860 INFO:teuthology.orchestra.run.smithi096.stdout: "ops": [], 2024-09-11T09:51:46.860 INFO:teuthology.orchestra.run.smithi096.stdout: "complaint_time": 30, 2024-09-11T09:51:46.860 INFO:teuthology.orchestra.run.smithi096.stdout: "num_blocked_ops": 0 2024-09-11T09:51:46.860 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-11T09:51:46.868 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-11T09:51:46.934 INFO:tasks.ceph.osd.0.smithi017.stderr:2024-09-11T09:51:46.934+0000 7fb73c0f1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31665) UID: 0 2024-09-11T09:51:47.013 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-11T09:51:47.014 INFO:teuthology.orchestra.run.smithi096.stdout: "size": 20, 2024-09-11T09:51:47.014 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 600, 2024-09-11T09:51:47.014 INFO:teuthology.orchestra.run.smithi096.stdout: "ops": [ 2024-09-11T09:51:47.014 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.014 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:111 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:51:47.014 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.711668+0000", 2024-09-11T09:51:47.014 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 95.301597561999998, 2024-09-11T09:51:47.015 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0077145570000000004, 2024-09-11T09:51:47.015 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:47.015 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:47.015 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:47.015 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:47.015 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:51:47.015 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:51:47.015 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 111 2024-09-11T09:51:47.015 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.015 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:47.015 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.015 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:47.015 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711668+0000", 2024-09-11T09:51:47.016 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:47.016 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.016 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.016 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:47.016 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711668+0000", 2024-09-11T09:51:47.016 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:47.016 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.016 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.016 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:47.016 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711669+0000", 2024-09-11T09:51:47.016 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.99e-07 2024-09-11T09:51:47.016 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.016 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.017 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:47.017 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711673+0000", 2024-09-11T09:51:47.017 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.7440000000000001e-06 2024-09-11T09:51:47.017 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.017 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.017 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:47.017 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711686+0000", 2024-09-11T09:51:47.017 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.3317e-05 2024-09-11T09:51:47.017 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.017 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.017 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:47.017 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711697+0000", 2024-09-11T09:51:47.017 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1022000000000001e-05 2024-09-11T09:51:47.018 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.018 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.018 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:47.018 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.717065+0000", 2024-09-11T09:51:47.018 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.005367531 2024-09-11T09:51:47.018 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.018 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.018 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:47.018 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.717114+0000", 2024-09-11T09:51:47.018 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.9707999999999999e-05 2024-09-11T09:51:47.018 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.018 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.018 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:47.019 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.717190+0000", 2024-09-11T09:51:47.019 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.5394000000000004e-05 2024-09-11T09:51:47.019 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.019 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.019 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:47.019 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718793+0000", 2024-09-11T09:51:47.019 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001603653 2024-09-11T09:51:47.019 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.019 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.019 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:47.019 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719329+0000", 2024-09-11T09:51:47.019 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00053543799999999995 2024-09-11T09:51:47.020 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.020 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.020 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:47.020 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719359+0000", 2024-09-11T09:51:47.020 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.0420999999999999e-05 2024-09-11T09:51:47.020 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.020 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.020 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:47.020 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719383+0000", 2024-09-11T09:51:47.020 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.3730000000000001e-05 2024-09-11T09:51:47.020 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:47.020 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:47.020 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:47.021 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.021 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.021 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:112 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:51:47.021 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.711712+0000", 2024-09-11T09:51:47.021 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 95.301554002000003, 2024-09-11T09:51:47.021 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.008162084, 2024-09-11T09:51:47.021 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:47.021 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:47.021 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:47.021 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:47.021 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:51:47.021 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:51:47.021 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 112 2024-09-11T09:51:47.021 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.022 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:47.022 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.022 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:47.022 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711712+0000", 2024-09-11T09:51:47.022 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:47.022 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.022 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.022 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:47.022 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711712+0000", 2024-09-11T09:51:47.022 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:47.022 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.022 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.022 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:47.022 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711712+0000", 2024-09-11T09:51:47.023 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.6000000000000004e-07 2024-09-11T09:51:47.023 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.023 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.023 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:47.023 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711808+0000", 2024-09-11T09:51:47.023 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.5977999999999995e-05 2024-09-11T09:51:47.023 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.023 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.023 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:47.023 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711819+0000", 2024-09-11T09:51:47.023 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.0755e-05 2024-09-11T09:51:47.023 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.024 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.024 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:47.024 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711826+0000", 2024-09-11T09:51:47.024 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.7970000000000003e-06 2024-09-11T09:51:47.024 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.024 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.024 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:47.024 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.717435+0000", 2024-09-11T09:51:47.024 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0056090239999999998 2024-09-11T09:51:47.024 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.024 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.024 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:47.024 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.717496+0000", 2024-09-11T09:51:47.024 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.0543e-05 2024-09-11T09:51:47.025 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.025 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.025 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:47.025 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.717591+0000", 2024-09-11T09:51:47.025 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.5667000000000004e-05 2024-09-11T09:51:47.025 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.025 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.025 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:47.025 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718831+0000", 2024-09-11T09:51:47.025 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001240204 2024-09-11T09:51:47.025 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.025 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.025 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:47.026 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719819+0000", 2024-09-11T09:51:47.026 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00098751500000000001 2024-09-11T09:51:47.026 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.026 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.026 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:47.026 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719850+0000", 2024-09-11T09:51:47.026 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.1535999999999998e-05 2024-09-11T09:51:47.026 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.026 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.026 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:47.026 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719874+0000", 2024-09-11T09:51:47.027 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.3504999999999999e-05 2024-09-11T09:51:47.027 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:47.027 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:47.027 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:47.027 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.027 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.027 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:113 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:51:47.027 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.711842+0000", 2024-09-11T09:51:47.027 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 95.301423502999995, 2024-09-11T09:51:47.027 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0081160099999999999, 2024-09-11T09:51:47.027 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:47.027 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:47.027 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:47.028 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:47.028 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:51:47.028 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:51:47.028 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 113 2024-09-11T09:51:47.028 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.028 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:47.028 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.028 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:47.028 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711842+0000", 2024-09-11T09:51:47.028 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:47.028 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.028 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.028 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:47.029 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711842+0000", 2024-09-11T09:51:47.029 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:47.029 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.029 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.029 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:47.029 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711843+0000", 2024-09-11T09:51:47.029 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.4399999999999999e-07 2024-09-11T09:51:47.029 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.029 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.029 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:47.029 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711848+0000", 2024-09-11T09:51:47.029 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.4159999999999997e-06 2024-09-11T09:51:47.029 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.029 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.030 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:47.030 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711856+0000", 2024-09-11T09:51:47.030 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.7870000000000007e-06 2024-09-11T09:51:47.030 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.030 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.030 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:47.030 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711862+0000", 2024-09-11T09:51:47.030 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.3689999999999998e-06 2024-09-11T09:51:47.030 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.030 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.030 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:47.030 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.717965+0000", 2024-09-11T09:51:47.031 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0061028899999999997 2024-09-11T09:51:47.031 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.031 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.031 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:47.031 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718023+0000", 2024-09-11T09:51:47.031 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.8313000000000002e-05 2024-09-11T09:51:47.031 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.031 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.031 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:47.031 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718109+0000", 2024-09-11T09:51:47.031 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.6205000000000006e-05 2024-09-11T09:51:47.031 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.031 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.031 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:47.032 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719808+0000", 2024-09-11T09:51:47.032 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0016990060000000001 2024-09-11T09:51:47.032 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.032 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.032 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:47.032 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719883+0000", 2024-09-11T09:51:47.032 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.4624999999999998e-05 2024-09-11T09:51:47.032 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.032 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.032 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:47.032 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719910+0000", 2024-09-11T09:51:47.032 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.7512e-05 2024-09-11T09:51:47.032 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.033 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.033 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:47.033 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719958+0000", 2024-09-11T09:51:47.033 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.8143000000000003e-05 2024-09-11T09:51:47.033 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:47.033 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:47.033 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:47.033 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.033 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.033 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:51:47.033 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.711887+0000", 2024-09-11T09:51:47.033 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 95.301378889000006, 2024-09-11T09:51:47.033 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0086063589999999992, 2024-09-11T09:51:47.033 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:47.034 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:47.034 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:47.034 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:47.034 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:51:47.034 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:51:47.034 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 114 2024-09-11T09:51:47.034 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.034 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:47.034 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.034 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:47.034 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711887+0000", 2024-09-11T09:51:47.034 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:47.034 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.035 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.035 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:47.035 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711887+0000", 2024-09-11T09:51:47.035 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:47.035 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.035 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.035 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:47.035 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711887+0000", 2024-09-11T09:51:47.035 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.1500000000000005e-07 2024-09-11T09:51:47.035 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.035 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.035 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:47.035 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711897+0000", 2024-09-11T09:51:47.036 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.1900000000000001e-06 2024-09-11T09:51:47.036 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.036 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.036 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:47.036 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711902+0000", 2024-09-11T09:51:47.036 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.2349999999999999e-06 2024-09-11T09:51:47.036 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.036 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.036 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:47.036 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711906+0000", 2024-09-11T09:51:47.036 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.7129999999999999e-06 2024-09-11T09:51:47.036 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.036 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.036 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:47.037 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718429+0000", 2024-09-11T09:51:47.037 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0065231960000000002 2024-09-11T09:51:47.037 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.037 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.037 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:47.037 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718480+0000", 2024-09-11T09:51:47.037 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.1161e-05 2024-09-11T09:51:47.037 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.037 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.037 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:47.037 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718558+0000", 2024-09-11T09:51:47.037 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.7519000000000001e-05 2024-09-11T09:51:47.037 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.038 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.038 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:47.038 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720038+0000", 2024-09-11T09:51:47.038 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001480016 2024-09-11T09:51:47.038 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.038 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.038 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:47.038 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720417+0000", 2024-09-11T09:51:47.038 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00037943699999999999 2024-09-11T09:51:47.038 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.038 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.038 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:47.038 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720452+0000", 2024-09-11T09:51:47.039 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.4595000000000003e-05 2024-09-11T09:51:47.039 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.039 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.039 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:47.039 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720493+0000", 2024-09-11T09:51:47.039 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.1782000000000001e-05 2024-09-11T09:51:47.039 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:47.039 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:47.039 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:47.039 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.039 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.039 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:51:47.039 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.711912+0000", 2024-09-11T09:51:47.040 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 95.301354219999993, 2024-09-11T09:51:47.040 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0086864750000000008, 2024-09-11T09:51:47.040 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:47.040 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:47.040 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:47.040 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:47.040 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:51:47.040 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:51:47.040 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 115 2024-09-11T09:51:47.040 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.040 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:47.040 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.040 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:47.040 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711912+0000", 2024-09-11T09:51:47.041 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:47.041 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.041 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.041 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:47.041 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711912+0000", 2024-09-11T09:51:47.041 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:47.041 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.041 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.041 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:47.041 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711912+0000", 2024-09-11T09:51:47.041 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.3900000000000001e-07 2024-09-11T09:51:47.041 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.041 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.041 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:47.042 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711969+0000", 2024-09-11T09:51:47.042 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.7501999999999999e-05 2024-09-11T09:51:47.042 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.042 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.042 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:47.042 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711986+0000", 2024-09-11T09:51:47.042 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.6402999999999999e-05 2024-09-11T09:51:47.042 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.042 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.042 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:47.042 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711990+0000", 2024-09-11T09:51:47.042 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.929e-06 2024-09-11T09:51:47.042 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.043 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.043 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:47.043 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718861+0000", 2024-09-11T09:51:47.043 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0068710530000000002 2024-09-11T09:51:47.043 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.043 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.043 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:47.043 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718908+0000", 2024-09-11T09:51:47.043 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.7431e-05 2024-09-11T09:51:47.043 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.044 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.044 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:47.044 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718984+0000", 2024-09-11T09:51:47.044 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.5500999999999994e-05 2024-09-11T09:51:47.044 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.044 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.044 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:47.044 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720500+0000", 2024-09-11T09:51:47.044 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0015158820000000001 2024-09-11T09:51:47.045 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.045 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.045 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:47.045 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720536+0000", 2024-09-11T09:51:47.045 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.6118000000000002e-05 2024-09-11T09:51:47.045 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.045 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.045 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:47.045 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720570+0000", 2024-09-11T09:51:47.045 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.4387999999999997e-05 2024-09-11T09:51:47.046 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.046 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.046 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:47.046 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720598+0000", 2024-09-11T09:51:47.046 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.8028999999999999e-05 2024-09-11T09:51:47.046 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:47.046 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:47.046 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:47.046 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.047 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.047 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:51:47.047 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.711998+0000", 2024-09-11T09:51:47.047 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 95.301268041, 2024-09-11T09:51:47.047 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.010785813999999999, 2024-09-11T09:51:47.047 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:47.047 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:47.047 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:47.047 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:47.047 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:51:47.047 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:51:47.047 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 116 2024-09-11T09:51:47.047 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.047 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:47.048 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.048 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:47.048 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711998+0000", 2024-09-11T09:51:47.048 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:47.048 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.048 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.048 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:47.048 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711998+0000", 2024-09-11T09:51:47.048 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:47.048 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.048 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.048 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:47.048 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711998+0000", 2024-09-11T09:51:47.049 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.3300000000000003e-07 2024-09-11T09:51:47.049 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.049 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.049 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:47.049 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712001+0000", 2024-09-11T09:51:47.049 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.9349999999999999e-06 2024-09-11T09:51:47.049 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.049 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.049 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:47.049 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712006+0000", 2024-09-11T09:51:47.049 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.1630000000000004e-06 2024-09-11T09:51:47.049 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.049 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.049 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:47.050 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712010+0000", 2024-09-11T09:51:47.050 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.574e-06 2024-09-11T09:51:47.050 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.050 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.050 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:47.050 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719415+0000", 2024-09-11T09:51:47.050 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0074049099999999998 2024-09-11T09:51:47.050 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.050 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.050 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:47.050 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719465+0000", 2024-09-11T09:51:47.050 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.0129000000000001e-05 2024-09-11T09:51:47.050 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.051 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.051 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:47.051 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719545+0000", 2024-09-11T09:51:47.051 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.0253999999999994e-05 2024-09-11T09:51:47.051 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.051 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.051 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:47.051 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720636+0000", 2024-09-11T09:51:47.051 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0010909330000000001 2024-09-11T09:51:47.051 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.051 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.051 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:47.051 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.722707+0000", 2024-09-11T09:51:47.051 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0020713300000000001 2024-09-11T09:51:47.052 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.052 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.052 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:47.052 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.722743+0000", 2024-09-11T09:51:47.052 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.5351000000000001e-05 2024-09-11T09:51:47.052 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.052 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.052 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:47.052 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.722784+0000", 2024-09-11T09:51:47.052 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.0802e-05 2024-09-11T09:51:47.052 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:47.052 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:47.052 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:47.053 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.053 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.053 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:51:47.053 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.712014+0000", 2024-09-11T09:51:47.053 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 95.301252000999995, 2024-09-11T09:51:47.053 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.013420938, 2024-09-11T09:51:47.053 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:47.053 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:47.053 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:47.053 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:47.053 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:51:47.053 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:51:47.053 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 117 2024-09-11T09:51:47.054 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.054 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:47.054 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.054 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:47.054 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712014+0000", 2024-09-11T09:51:47.054 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:47.054 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.054 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.054 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:47.054 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712014+0000", 2024-09-11T09:51:47.054 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:47.054 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.054 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.054 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:47.055 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712014+0000", 2024-09-11T09:51:47.055 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.2000000000000001e-07 2024-09-11T09:51:47.055 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.055 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.055 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:47.055 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712016+0000", 2024-09-11T09:51:47.055 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.2309999999999999e-06 2024-09-11T09:51:47.055 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.055 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.055 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:47.055 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712020+0000", 2024-09-11T09:51:47.055 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.3830000000000001e-06 2024-09-11T09:51:47.055 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.056 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.056 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:47.056 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712023+0000", 2024-09-11T09:51:47.056 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.3340000000000002e-06 2024-09-11T09:51:47.056 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.056 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.056 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:47.056 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720083+0000", 2024-09-11T09:51:47.056 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0080601189999999993 2024-09-11T09:51:47.056 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.056 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.056 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:47.056 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720125+0000", 2024-09-11T09:51:47.057 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.2029e-05 2024-09-11T09:51:47.057 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.057 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.057 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:47.057 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720205+0000", 2024-09-11T09:51:47.057 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.0010000000000001e-05 2024-09-11T09:51:47.057 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.057 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.057 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:47.057 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720849+0000", 2024-09-11T09:51:47.057 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00064369899999999996 2024-09-11T09:51:47.057 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.057 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.058 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:47.058 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.725357+0000", 2024-09-11T09:51:47.058 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0045077379999999998 2024-09-11T09:51:47.058 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.058 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.058 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:47.058 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.725392+0000", 2024-09-11T09:51:47.058 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.5862999999999999e-05 2024-09-11T09:51:47.058 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.058 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.058 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:47.058 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.725435+0000", 2024-09-11T09:51:47.059 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.2311999999999997e-05 2024-09-11T09:51:47.059 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:47.059 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:47.059 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:47.059 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.059 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.059 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:159 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:51:47.059 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:12.452099+0000", 2024-09-11T09:51:47.059 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 94.561166412999995, 2024-09-11T09:51:47.059 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0069885950000000002, 2024-09-11T09:51:47.059 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:47.059 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:47.059 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:47.060 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:47.060 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:51:47.060 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:51:47.060 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 159 2024-09-11T09:51:47.060 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.060 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:47.060 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.060 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:47.060 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452099+0000", 2024-09-11T09:51:47.060 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:47.060 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.060 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.060 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:47.061 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452099+0000", 2024-09-11T09:51:47.061 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:47.061 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.061 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.061 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:47.061 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452100+0000", 2024-09-11T09:51:47.061 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.2800000000000003e-07 2024-09-11T09:51:47.061 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.061 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.061 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:47.061 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452102+0000", 2024-09-11T09:51:47.061 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.824e-06 2024-09-11T09:51:47.061 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.061 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.062 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:47.062 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452105+0000", 2024-09-11T09:51:47.062 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.0460000000000001e-06 2024-09-11T09:51:47.062 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.062 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.062 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:47.062 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452107+0000", 2024-09-11T09:51:47.062 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.543e-06 2024-09-11T09:51:47.062 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.062 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.062 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:47.062 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.455484+0000", 2024-09-11T09:51:47.062 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0033769759999999998 2024-09-11T09:51:47.062 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.063 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.063 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:47.063 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.455527+0000", 2024-09-11T09:51:47.063 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.2589999999999997e-05 2024-09-11T09:51:47.063 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.063 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.063 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:47.063 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.455594+0000", 2024-09-11T09:51:47.063 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.6775999999999999e-05 2024-09-11T09:51:47.063 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.063 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.063 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:47.063 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.456312+0000", 2024-09-11T09:51:47.064 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00071874700000000005 2024-09-11T09:51:47.064 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.064 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.064 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:47.064 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.459015+0000", 2024-09-11T09:51:47.064 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0027028540000000002 2024-09-11T09:51:47.064 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.064 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.064 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:47.064 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.459054+0000", 2024-09-11T09:51:47.064 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.8840000000000001e-05 2024-09-11T09:51:47.064 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.064 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.064 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:47.065 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.459088+0000", 2024-09-11T09:51:47.065 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.4071e-05 2024-09-11T09:51:47.065 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:47.065 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:47.065 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:47.065 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.065 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.065 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4688.0:38 2.5 2:abde2694:::rbd_data.1250f5e25b80.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:47.065 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:22.903926+0000", 2024-09-11T09:51:47.065 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 24.109339736999999, 2024-09-11T09:51:47.065 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.028303676, 2024-09-11T09:51:47.065 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:47.065 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:47.066 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:47.066 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:47.066 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4688", 2024-09-11T09:51:47.066 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:51:47.066 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 38 2024-09-11T09:51:47.066 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.066 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:47.066 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.066 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:47.066 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.903926+0000", 2024-09-11T09:51:47.066 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:47.066 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.066 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.066 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:47.067 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.903926+0000", 2024-09-11T09:51:47.067 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:47.067 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.067 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.067 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:47.067 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.903928+0000", 2024-09-11T09:51:47.067 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.4249999999999999e-06 2024-09-11T09:51:47.067 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.067 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.067 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:47.067 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.914774+0000", 2024-09-11T09:51:47.067 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.010846613 2024-09-11T09:51:47.067 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.067 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.068 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:47.068 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.914793+0000", 2024-09-11T09:51:47.068 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.8879999999999999e-05 2024-09-11T09:51:47.068 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.068 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.068 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:47.068 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.914801+0000", 2024-09-11T09:51:47.068 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.2230000000000007e-06 2024-09-11T09:51:47.068 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.068 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.068 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:47.068 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.914854+0000", 2024-09-11T09:51:47.068 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.2883999999999997e-05 2024-09-11T09:51:47.068 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.069 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.069 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:47.069 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.914951+0000", 2024-09-11T09:51:47.069 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.6731000000000006e-05 2024-09-11T09:51:47.069 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.069 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.069 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:47.069 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.915080+0000", 2024-09-11T09:51:47.069 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00012926999999999999 2024-09-11T09:51:47.069 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.069 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.069 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:47.069 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.931469+0000", 2024-09-11T09:51:47.069 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.016389107 2024-09-11T09:51:47.070 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.070 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.070 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:47.070 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.932118+0000", 2024-09-11T09:51:47.070 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00064915600000000002 2024-09-11T09:51:47.070 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.070 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.070 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:47.070 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.932177+0000", 2024-09-11T09:51:47.070 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.9032e-05 2024-09-11T09:51:47.070 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.070 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.071 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:47.071 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.932230+0000", 2024-09-11T09:51:47.071 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.2355000000000003e-05 2024-09-11T09:51:47.071 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:47.071 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:47.071 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:47.071 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.071 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.071 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_repop(client.4688.0:36 2.6 e53/15 2:61e1b442:::rbd_data.1250f5e25b80.0000000000000003:head v 53'2, mlcod=0'0)", 2024-09-11T09:51:47.072 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:22.915597+0000", 2024-09-11T09:51:47.072 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 24.097669187000001, 2024-09-11T09:51:47.072 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.020430798, 2024-09-11T09:51:47.072 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:47.072 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:47.072 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:47.072 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:47.072 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.072 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:47.072 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.915597+0000", 2024-09-11T09:51:47.072 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:47.072 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.072 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.073 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:47.073 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.915597+0000", 2024-09-11T09:51:47.073 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:47.073 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.073 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.073 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:47.073 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.915598+0000", 2024-09-11T09:51:47.073 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.7150000000000001e-06 2024-09-11T09:51:47.073 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.073 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.073 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:47.074 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921741+0000", 2024-09-11T09:51:47.074 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0061424339999999996 2024-09-11T09:51:47.074 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.074 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.074 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:47.074 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921757+0000", 2024-09-11T09:51:47.074 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.6144e-05 2024-09-11T09:51:47.074 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.074 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.074 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:47.074 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921765+0000", 2024-09-11T09:51:47.074 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.0809999999999994e-06 2024-09-11T09:51:47.074 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.074 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.075 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:47.075 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921826+0000", 2024-09-11T09:51:47.075 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.0878999999999998e-05 2024-09-11T09:51:47.075 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.075 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.075 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:47.075 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921849+0000", 2024-09-11T09:51:47.075 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.2966999999999999e-05 2024-09-11T09:51:47.075 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.075 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.075 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:47.075 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.935968+0000", 2024-09-11T09:51:47.075 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.014119191 2024-09-11T09:51:47.075 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.076 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.076 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:47.076 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.936027+0000", 2024-09-11T09:51:47.076 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.9386999999999999e-05 2024-09-11T09:51:47.076 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:47.076 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:47.076 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:47.076 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.076 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.076 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_repop(client.4688.0:39 2.6 e53/15 2:743e7616:::rbd_data.1250f5e25b80.0000000000000005:head v 53'3, mlcod=0'0)", 2024-09-11T09:51:47.076 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:22.921806+0000", 2024-09-11T09:51:47.076 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 24.091460136999999, 2024-09-11T09:51:47.076 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.015275018, 2024-09-11T09:51:47.077 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:47.077 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:47.077 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:47.077 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:47.077 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.077 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:47.077 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921806+0000", 2024-09-11T09:51:47.077 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:47.077 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.077 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.077 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:47.077 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921806+0000", 2024-09-11T09:51:47.077 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:47.078 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.078 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.078 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:47.078 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921806+0000", 2024-09-11T09:51:47.078 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.2200000000000003e-07 2024-09-11T09:51:47.078 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.078 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.078 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:47.078 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.925590+0000", 2024-09-11T09:51:47.078 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0037838949999999998 2024-09-11T09:51:47.078 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.078 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.078 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:47.078 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.925605+0000", 2024-09-11T09:51:47.079 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.4321999999999999e-05 2024-09-11T09:51:47.079 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.079 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.079 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:47.079 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.925613+0000", 2024-09-11T09:51:47.079 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.1980000000000001e-06 2024-09-11T09:51:47.079 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.079 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.079 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:47.079 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.925857+0000", 2024-09-11T09:51:47.079 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000244064 2024-09-11T09:51:47.079 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.079 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.079 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:47.080 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.925875+0000", 2024-09-11T09:51:47.080 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.838e-05 2024-09-11T09:51:47.080 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.080 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.080 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:47.080 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.937042+0000", 2024-09-11T09:51:47.080 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.011166416 2024-09-11T09:51:47.080 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.080 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.080 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:47.080 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.937081+0000", 2024-09-11T09:51:47.080 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.9020999999999998e-05 2024-09-11T09:51:47.080 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:47.081 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:47.081 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:47.081 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.081 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.081 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4688.0:49 2.5 2:b6dd9fc9:::rbd_data.1250f5e25b80.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:47.081 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:22.938548+0000", 2024-09-11T09:51:47.081 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 24.074718134000001, 2024-09-11T09:51:47.081 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.022766950000000001, 2024-09-11T09:51:47.081 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:47.081 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:47.081 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:47.081 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:47.081 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4688", 2024-09-11T09:51:47.082 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:51:47.082 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 49 2024-09-11T09:51:47.082 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.082 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:47.082 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.082 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:47.082 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.938548+0000", 2024-09-11T09:51:47.082 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:47.082 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.082 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.082 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:47.082 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.938548+0000", 2024-09-11T09:51:47.082 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:47.083 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.083 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.083 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:47.083 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.938550+0000", 2024-09-11T09:51:47.083 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.7710000000000001e-06 2024-09-11T09:51:47.083 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.083 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.083 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:47.083 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.944237+0000", 2024-09-11T09:51:47.083 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0056879790000000001 2024-09-11T09:51:47.083 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.083 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.083 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:47.083 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.944262+0000", 2024-09-11T09:51:47.084 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.4247e-05 2024-09-11T09:51:47.084 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.084 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.084 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:47.084 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.944274+0000", 2024-09-11T09:51:47.084 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.2639999999999999e-05 2024-09-11T09:51:47.084 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.084 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.084 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:47.084 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.944344+0000", 2024-09-11T09:51:47.084 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.9574999999999997e-05 2024-09-11T09:51:47.084 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.084 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.085 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:47.085 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.944474+0000", 2024-09-11T09:51:47.085 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000130464 2024-09-11T09:51:47.085 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.085 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.085 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:47.085 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.944585+0000", 2024-09-11T09:51:47.085 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000110096 2024-09-11T09:51:47.085 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.085 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.085 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:47.085 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.952724+0000", 2024-09-11T09:51:47.085 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0081398900000000003 2024-09-11T09:51:47.085 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.085 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.086 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:47.086 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.961219+0000", 2024-09-11T09:51:47.086 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0084949789999999997 2024-09-11T09:51:47.086 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.086 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.086 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:47.086 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.961280+0000", 2024-09-11T09:51:47.086 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.0404e-05 2024-09-11T09:51:47.086 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.086 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.086 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:47.086 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.961315+0000", 2024-09-11T09:51:47.086 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.4904999999999999e-05 2024-09-11T09:51:47.086 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:47.087 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:47.087 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:47.087 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.087 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.087 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_repop(client.4688.0:44 2.6 e53/15 2:67246a22:::rbd_data.1250f5e25b80.0000000000000007:head v 53'4, mlcod=53'2)", 2024-09-11T09:51:47.087 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:22.951721+0000", 2024-09-11T09:51:47.087 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 24.061544862000002, 2024-09-11T09:51:47.087 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.01080981, 2024-09-11T09:51:47.087 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:47.087 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:47.087 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:47.087 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:47.087 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.088 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:47.088 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.951721+0000", 2024-09-11T09:51:47.088 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:47.088 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.088 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.088 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:47.088 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.951721+0000", 2024-09-11T09:51:47.088 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:47.088 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.088 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.088 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:47.088 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.951722+0000", 2024-09-11T09:51:47.088 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.0279999999999999e-06 2024-09-11T09:51:47.088 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.089 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.089 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:47.089 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.955612+0000", 2024-09-11T09:51:47.089 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0038896920000000001 2024-09-11T09:51:47.089 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.089 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.089 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:47.089 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.955628+0000", 2024-09-11T09:51:47.089 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.5945000000000001e-05 2024-09-11T09:51:47.089 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.089 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.089 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:47.089 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.955635+0000", 2024-09-11T09:51:47.089 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.8080000000000005e-06 2024-09-11T09:51:47.090 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.090 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.090 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:47.090 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.955700+0000", 2024-09-11T09:51:47.090 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.4114000000000003e-05 2024-09-11T09:51:47.090 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.090 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.090 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:47.090 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.955724+0000", 2024-09-11T09:51:47.090 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.3989999999999999e-05 2024-09-11T09:51:47.090 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.090 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.090 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:47.091 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.962490+0000", 2024-09-11T09:51:47.091 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0067662599999999996 2024-09-11T09:51:47.091 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.091 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.091 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:47.091 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.962531+0000", 2024-09-11T09:51:47.091 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.0973000000000002e-05 2024-09-11T09:51:47.091 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:47.091 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:47.091 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:47.091 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.091 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.091 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_repop(client.4688.0:51 2.6 e53/15 2:6b5894fc:::rbd_data.1250f5e25b80.000000000000000a:head v 53'5, mlcod=53'2)", 2024-09-11T09:51:47.091 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:22.959030+0000", 2024-09-11T09:51:47.092 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 24.054235687999999, 2024-09-11T09:51:47.092 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.011503638, 2024-09-11T09:51:47.092 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:47.092 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:47.092 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:47.092 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:47.092 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.092 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:47.092 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.959030+0000", 2024-09-11T09:51:47.092 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:47.092 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.092 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.092 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:47.093 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.959030+0000", 2024-09-11T09:51:47.093 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:47.093 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.093 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.093 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:47.093 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.959032+0000", 2024-09-11T09:51:47.093 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.356e-06 2024-09-11T09:51:47.093 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.093 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.093 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:47.093 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.962899+0000", 2024-09-11T09:51:47.093 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.003867639 2024-09-11T09:51:47.093 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.094 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.094 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:47.094 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.962909+0000", 2024-09-11T09:51:47.094 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.0105999999999999e-05 2024-09-11T09:51:47.094 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.094 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.094 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:47.094 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.962915+0000", 2024-09-11T09:51:47.094 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.8869999999999997e-06 2024-09-11T09:51:47.094 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.094 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.094 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:47.094 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.962962+0000", 2024-09-11T09:51:47.095 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.6511999999999998e-05 2024-09-11T09:51:47.095 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.095 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.095 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:47.095 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.962989+0000", 2024-09-11T09:51:47.095 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.7702999999999999e-05 2024-09-11T09:51:47.095 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.095 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.095 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:47.095 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.970495+0000", 2024-09-11T09:51:47.095 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0075052449999999998 2024-09-11T09:51:47.095 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.095 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.095 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:47.096 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.970534+0000", 2024-09-11T09:51:47.096 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.9190000000000003e-05 2024-09-11T09:51:47.096 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:47.096 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:47.096 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:47.096 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.096 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.096 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_repop(client.4688.0:58 2.6 e53/15 2:6277c644:::rbd_data.1250f5e25b80.000000000000000e:head v 53'6, mlcod=53'4)", 2024-09-11T09:51:47.096 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:22.996199+0000", 2024-09-11T09:51:47.096 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 24.017066767999999, 2024-09-11T09:51:47.096 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.010145573999999999, 2024-09-11T09:51:47.096 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:47.096 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:47.096 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:47.097 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:47.097 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.097 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:47.097 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.996199+0000", 2024-09-11T09:51:47.097 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:47.097 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.097 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.097 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:47.097 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.996199+0000", 2024-09-11T09:51:47.097 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:47.097 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.097 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.097 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:47.098 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.996200+0000", 2024-09-11T09:51:47.098 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.85e-07 2024-09-11T09:51:47.098 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.098 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.098 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:47.098 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.999914+0000", 2024-09-11T09:51:47.098 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.003714406 2024-09-11T09:51:47.098 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.098 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.098 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:47.098 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.999928+0000", 2024-09-11T09:51:47.098 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.3149000000000001e-05 2024-09-11T09:51:47.098 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.098 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.099 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:47.099 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.999951+0000", 2024-09-11T09:51:47.099 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.3116e-05 2024-09-11T09:51:47.099 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.099 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.099 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:47.099 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.000003+0000", 2024-09-11T09:51:47.099 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.2169000000000001e-05 2024-09-11T09:51:47.099 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.099 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.099 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:47.099 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.000018+0000", 2024-09-11T09:51:47.099 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.5500000000000001e-05 2024-09-11T09:51:47.099 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.100 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.100 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:47.100 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.006303+0000", 2024-09-11T09:51:47.100 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0062846710000000004 2024-09-11T09:51:47.100 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.100 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.100 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:47.100 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.006345+0000", 2024-09-11T09:51:47.100 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.1678000000000003e-05 2024-09-11T09:51:47.100 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:47.100 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:47.100 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:47.100 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.101 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.101 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4694.0:42 2.5 2:a4fc00ec:ns::rbd_data.1256a0c02879.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:47.101 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.111570+0000", 2024-09-11T09:51:47.101 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 23.901695662000002, 2024-09-11T09:51:47.101 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.038002197000000001, 2024-09-11T09:51:47.101 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:47.101 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:47.101 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:47.101 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:47.101 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4694", 2024-09-11T09:51:47.101 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:51:47.101 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 42 2024-09-11T09:51:47.101 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.102 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:47.102 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.102 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:47.102 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.111570+0000", 2024-09-11T09:51:47.102 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:47.102 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.102 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.102 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:47.102 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.111570+0000", 2024-09-11T09:51:47.102 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:47.102 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.102 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.102 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:47.103 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.111572+0000", 2024-09-11T09:51:47.103 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.3909999999999999e-06 2024-09-11T09:51:47.103 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.103 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.103 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:47.103 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.135929+0000", 2024-09-11T09:51:47.103 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.024357410999999999 2024-09-11T09:51:47.103 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.103 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.103 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:47.103 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.135951+0000", 2024-09-11T09:51:47.103 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.1775000000000001e-05 2024-09-11T09:51:47.103 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.104 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.104 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:47.104 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.135962+0000", 2024-09-11T09:51:47.104 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.0903e-05 2024-09-11T09:51:47.104 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.104 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.104 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:47.104 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.136045+0000", 2024-09-11T09:51:47.104 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.3128999999999996e-05 2024-09-11T09:51:47.104 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.104 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.104 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:47.104 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.136229+0000", 2024-09-11T09:51:47.104 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00018393700000000001 2024-09-11T09:51:47.105 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.105 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.105 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:47.105 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.136394+0000", 2024-09-11T09:51:47.105 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000165527 2024-09-11T09:51:47.105 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.105 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.105 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:47.105 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.149320+0000", 2024-09-11T09:51:47.105 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.01292568 2024-09-11T09:51:47.105 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.105 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.105 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:47.106 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.149493+0000", 2024-09-11T09:51:47.106 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00017285999999999999 2024-09-11T09:51:47.106 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.106 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.106 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:47.106 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.149539+0000", 2024-09-11T09:51:47.106 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.6628000000000001e-05 2024-09-11T09:51:47.106 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.106 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.106 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:47.106 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.149572+0000", 2024-09-11T09:51:47.106 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.2956000000000001e-05 2024-09-11T09:51:47.106 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:47.106 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:47.107 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:47.107 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.107 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.107 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_repop(client.4694.0:45 2.6 e53/15 2:671fb830:ns::rbd_data.1256a0c02879.0000000000000007:head v 53'9, mlcod=53'7)", 2024-09-11T09:51:47.107 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.121833+0000", 2024-09-11T09:51:47.107 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 23.891433304, 2024-09-11T09:51:47.107 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.023876192000000001, 2024-09-11T09:51:47.107 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:47.107 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:47.107 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:47.107 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:47.107 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.107 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:47.107 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.121833+0000", 2024-09-11T09:51:47.108 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:47.108 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.108 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.108 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:47.108 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.121833+0000", 2024-09-11T09:51:47.108 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:47.108 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.108 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.108 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:47.108 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.121834+0000", 2024-09-11T09:51:47.108 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.471e-06 2024-09-11T09:51:47.108 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.108 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.108 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:47.109 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.128990+0000", 2024-09-11T09:51:47.109 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0071557879999999997 2024-09-11T09:51:47.109 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.109 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.109 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:47.109 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.129011+0000", 2024-09-11T09:51:47.109 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.0760000000000001e-05 2024-09-11T09:51:47.109 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.109 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.109 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:47.109 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.129022+0000", 2024-09-11T09:51:47.109 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1056000000000001e-05 2024-09-11T09:51:47.109 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.110 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.110 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:47.110 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.129097+0000", 2024-09-11T09:51:47.110 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.5082999999999999e-05 2024-09-11T09:51:47.110 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.110 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.110 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:47.110 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.129130+0000", 2024-09-11T09:51:47.110 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.3345e-05 2024-09-11T09:51:47.110 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.110 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.110 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:47.110 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.145657+0000", 2024-09-11T09:51:47.110 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.016527224 2024-09-11T09:51:47.111 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.111 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.111 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:47.111 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.145709+0000", 2024-09-11T09:51:47.111 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.1465000000000003e-05 2024-09-11T09:51:47.111 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:47.111 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:47.111 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:47.111 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.111 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.111 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4694.0:43 2.0 2:16c8ae84:ns::rbd_data.1256a0c02879.0000000000000005:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:47.111 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.135999+0000", 2024-09-11T09:51:47.111 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 23.877267354000001, 2024-09-11T09:51:47.112 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.039468309, 2024-09-11T09:51:47.112 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:47.112 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:47.112 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:47.112 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:47.112 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4694", 2024-09-11T09:51:47.112 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:51:47.112 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 43 2024-09-11T09:51:47.112 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.112 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:47.112 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.112 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:47.112 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.135999+0000", 2024-09-11T09:51:47.112 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:47.113 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.113 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.113 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:47.113 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.135999+0000", 2024-09-11T09:51:47.113 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:47.113 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.113 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.113 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:47.113 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.136000+0000", 2024-09-11T09:51:47.113 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.6330000000000001e-06 2024-09-11T09:51:47.113 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.113 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.113 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:47.114 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.157961+0000", 2024-09-11T09:51:47.114 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0219606 2024-09-11T09:51:47.114 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.114 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.114 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:47.114 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.157995+0000", 2024-09-11T09:51:47.114 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.4391999999999999e-05 2024-09-11T09:51:47.114 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.115 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.115 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:47.115 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.158011+0000", 2024-09-11T09:51:47.115 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.5794000000000002e-05 2024-09-11T09:51:47.115 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.115 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.115 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:47.115 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.158083+0000", 2024-09-11T09:51:47.115 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.2273000000000004e-05 2024-09-11T09:51:47.115 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.115 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.115 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:47.115 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.158239+0000", 2024-09-11T09:51:47.115 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000155372 2024-09-11T09:51:47.116 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.116 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.116 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 1", 2024-09-11T09:51:47.116 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.158403+0000", 2024-09-11T09:51:47.116 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00016408199999999999 2024-09-11T09:51:47.116 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.116 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.116 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:47.116 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.173860+0000", 2024-09-11T09:51:47.116 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.015457347 2024-09-11T09:51:47.116 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.116 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.117 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:47.117 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.175400+0000", 2024-09-11T09:51:47.117 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001540006 2024-09-11T09:51:47.117 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.117 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.117 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:47.117 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.175438+0000", 2024-09-11T09:51:47.117 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.8135e-05 2024-09-11T09:51:47.117 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.117 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.117 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:47.117 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.175467+0000", 2024-09-11T09:51:47.117 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.8674999999999999e-05 2024-09-11T09:51:47.117 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:47.118 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:47.118 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:47.118 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.118 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.118 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4694.0:57 2.0 2:04b5549c:ns::rbd_data.1256a0c02879.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:47.118 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.158037+0000", 2024-09-11T09:51:47.118 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 23.855228616000002, 2024-09-11T09:51:47.118 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.027748790999999998, 2024-09-11T09:51:47.118 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:47.118 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:47.118 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:47.118 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:47.118 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4694", 2024-09-11T09:51:47.118 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:51:47.119 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 57 2024-09-11T09:51:47.119 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.119 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:47.119 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.119 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:47.119 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.158037+0000", 2024-09-11T09:51:47.119 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:47.119 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.119 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.119 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:47.119 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.158037+0000", 2024-09-11T09:51:47.119 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:47.119 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.120 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.120 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:47.120 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.158039+0000", 2024-09-11T09:51:47.120 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.356e-06 2024-09-11T09:51:47.120 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.120 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.120 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:47.120 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.176367+0000", 2024-09-11T09:51:47.120 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.018328739 2024-09-11T09:51:47.120 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.120 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.120 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:47.120 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.176378+0000", 2024-09-11T09:51:47.120 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.0713999999999999e-05 2024-09-11T09:51:47.121 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.121 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.121 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:47.121 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.176383+0000", 2024-09-11T09:51:47.121 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.0209999999999999e-06 2024-09-11T09:51:47.121 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.121 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.121 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:47.121 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.176426+0000", 2024-09-11T09:51:47.121 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.2623999999999997e-05 2024-09-11T09:51:47.121 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.121 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.121 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:47.122 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.176552+0000", 2024-09-11T09:51:47.122 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000126076 2024-09-11T09:51:47.122 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.122 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.122 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 1", 2024-09-11T09:51:47.122 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.176665+0000", 2024-09-11T09:51:47.122 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000112995 2024-09-11T09:51:47.122 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.122 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.122 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:47.122 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.183639+0000", 2024-09-11T09:51:47.122 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0069740039999999998 2024-09-11T09:51:47.122 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.122 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.123 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:47.123 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.185689+0000", 2024-09-11T09:51:47.123 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.002049852 2024-09-11T09:51:47.123 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.123 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.123 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:47.123 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.185725+0000", 2024-09-11T09:51:47.123 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.6804999999999997e-05 2024-09-11T09:51:47.123 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.123 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.123 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:47.123 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.185786+0000", 2024-09-11T09:51:47.123 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.0605e-05 2024-09-11T09:51:47.123 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:47.124 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:47.124 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:47.124 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.124 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.124 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4694.0:60 2.0 2:054f2af4:ns::rbd_data.1256a0c02879.000000000000000f:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:47.124 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.186855+0000", 2024-09-11T09:51:47.124 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 23.826410599999999, 2024-09-11T09:51:47.124 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.013682913, 2024-09-11T09:51:47.124 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:47.124 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:47.124 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:47.124 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:47.124 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4694", 2024-09-11T09:51:47.125 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:51:47.125 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 60 2024-09-11T09:51:47.125 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.125 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:47.125 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.125 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:47.125 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.186855+0000", 2024-09-11T09:51:47.125 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:47.125 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.125 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.125 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:47.125 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.186855+0000", 2024-09-11T09:51:47.125 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:47.126 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.126 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.126 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:47.126 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.186856+0000", 2024-09-11T09:51:47.126 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.1200000000000001e-07 2024-09-11T09:51:47.126 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.126 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.126 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:47.126 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.191484+0000", 2024-09-11T09:51:47.126 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0046274159999999996 2024-09-11T09:51:47.126 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.126 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.126 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:47.126 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.191496+0000", 2024-09-11T09:51:47.127 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.2149e-05 2024-09-11T09:51:47.127 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.127 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.127 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:47.127 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.191502+0000", 2024-09-11T09:51:47.127 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.1890000000000004e-06 2024-09-11T09:51:47.127 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.127 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.127 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:47.127 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.191544+0000", 2024-09-11T09:51:47.127 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.1724999999999998e-05 2024-09-11T09:51:47.127 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.127 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.127 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:47.128 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.191641+0000", 2024-09-11T09:51:47.128 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.6939000000000001e-05 2024-09-11T09:51:47.128 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.128 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.128 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 1", 2024-09-11T09:51:47.128 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.191731+0000", 2024-09-11T09:51:47.128 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.0651999999999999e-05 2024-09-11T09:51:47.128 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.128 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.128 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:47.128 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.199852+0000", 2024-09-11T09:51:47.128 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0081204529999999997 2024-09-11T09:51:47.128 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.128 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.129 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:47.129 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.200461+0000", 2024-09-11T09:51:47.129 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00060900899999999998 2024-09-11T09:51:47.129 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.129 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.129 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:47.129 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.200497+0000", 2024-09-11T09:51:47.129 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.5800000000000003e-05 2024-09-11T09:51:47.129 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.129 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.129 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:47.129 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.200538+0000", 2024-09-11T09:51:47.129 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.1668999999999997e-05 2024-09-11T09:51:47.130 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:47.130 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:47.130 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:47.130 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:47.130 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:47.130 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-11T09:51:47.131 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-11T09:51:47.132 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:51:47.132+0000 7f3c96ec9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31909) UID: 0 2024-09-11T09:51:47.179 INFO:tasks.workunit.client.0.smithi107.stderr:+ DEV= 2024-09-11T09:51:47.179 INFO:tasks.workunit.client.0.smithi107.stderr:+ unmap_device rbd/testrbdnbd27121 28925 2024-09-11T09:51:47.179 INFO:tasks.workunit.client.0.smithi107.stderr:+ local args=rbd/testrbdnbd27121 2024-09-11T09:51:47.179 INFO:tasks.workunit.client.0.smithi107.stderr:+ local pid=28925 2024-09-11T09:51:47.179 INFO:tasks.workunit.client.0.smithi107.stderr:+ _sudo rbd device --device-type nbd unmap rbd/testrbdnbd27121 2024-09-11T09:51:47.179 INFO:tasks.workunit.client.0.smithi107.stderr:+ local cmd 2024-09-11T09:51:47.180 INFO:tasks.workunit.client.0.smithi107.stderr:++ id -u 2024-09-11T09:51:47.180 INFO:tasks.workunit.client.0.smithi107.stderr:+ '[' 1000 -eq 0 ']' 2024-09-11T09:51:47.180 INFO:tasks.workunit.client.0.smithi107.stderr:++ which rbd 2024-09-11T09:51:47.182 INFO:tasks.workunit.client.0.smithi107.stderr:+ cmd=/usr/bin/rbd 2024-09-11T09:51:47.182 INFO:tasks.workunit.client.0.smithi107.stderr:+ shift 2024-09-11T09:51:47.182 INFO:tasks.workunit.client.0.smithi107.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd unmap rbd/testrbdnbd27121 2024-09-11T09:51:47.233 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-09-11T09:51:47.234+0000 7f7b6c7d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31904) UID: 0 2024-09-11T09:51:47.261 INFO:tasks.workunit.client.0.smithi107.stderr:+ rbd device --device-type nbd list 2024-09-11T09:51:47.261 INFO:tasks.workunit.client.0.smithi107.stderr:+ expect_false grep '^28925\b' 2024-09-11T09:51:47.261 INFO:tasks.workunit.client.0.smithi107.stderr:+ grep '^28925\b' 2024-09-11T09:51:47.275 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-11T09:51:47.275 INFO:teuthology.orchestra.run.smithi096.stdout: "ops": [], 2024-09-11T09:51:47.275 INFO:teuthology.orchestra.run.smithi096.stdout: "num_ops": 0 2024-09-11T09:51:47.276 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-11T09:51:47.284 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-11T09:51:47.308 INFO:tasks.workunit.client.0.smithi107.stderr:+ return 0 2024-09-11T09:51:47.308 INFO:tasks.workunit.client.0.smithi107.stderr:+ ps -C rbd-nbd 2024-09-11T09:51:47.308 INFO:tasks.workunit.client.0.smithi107.stderr:+ expect_false grep '^ *28925\b' 2024-09-11T09:51:47.308 INFO:tasks.workunit.client.0.smithi107.stderr:+ grep '^ *28925\b' 2024-09-11T09:51:47.319 INFO:tasks.workunit.client.0.smithi107.stderr:+ return 0 2024-09-11T09:51:47.319 INFO:tasks.workunit.client.0.smithi107.stderr:+ sleep 0.5 2024-09-11T09:51:47.333 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:51:47.334+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:51:47.426 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-11T09:51:47.427 INFO:teuthology.orchestra.run.smithi096.stdout: "ops": [], 2024-09-11T09:51:47.427 INFO:teuthology.orchestra.run.smithi096.stdout: "complaint_time": 30, 2024-09-11T09:51:47.427 INFO:teuthology.orchestra.run.smithi096.stdout: "num_blocked_ops": 0 2024-09-11T09:51:47.427 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-11T09:51:47.434 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:51:47.435+0000 7f3c96ec9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31909) UID: 0 2024-09-11T09:51:47.435 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-11T09:51:47.457 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T09:51:47.536 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-09-11T09:51:47.536+0000 7f7b6c7d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31904) UID: 0 2024-09-11T09:51:47.580 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-11T09:51:47.580 INFO:teuthology.orchestra.run.smithi096.stdout: "size": 20, 2024-09-11T09:51:47.580 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 600, 2024-09-11T09:51:47.580 INFO:teuthology.orchestra.run.smithi096.stdout: "ops": [ 2024-09-11T09:51:47.580 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.580 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4230.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-11T09:51:47.580 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:18.490736+0000", 2024-09-11T09:51:47.580 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 89.088947535000003, 2024-09-11T09:51:47.581 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0059097569999999999, 2024-09-11T09:51:47.581 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:47.581 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:47.581 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:47.581 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:47.581 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4230", 2024-09-11T09:51:47.581 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.17:0/4101109576", 2024-09-11T09:51:47.581 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 1 2024-09-11T09:51:47.581 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.581 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:47.581 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.581 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:47.581 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.490736+0000", 2024-09-11T09:51:47.581 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:47.582 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.582 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.582 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:47.582 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.490736+0000", 2024-09-11T09:51:47.582 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:47.582 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.582 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.582 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:47.582 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.490742+0000", 2024-09-11T09:51:47.582 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.48e-06 2024-09-11T09:51:47.582 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.582 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.582 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:47.582 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.490749+0000", 2024-09-11T09:51:47.583 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.4569999999999999e-06 2024-09-11T09:51:47.583 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.583 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.583 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:47.583 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.490762+0000", 2024-09-11T09:51:47.583 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.2721000000000001e-05 2024-09-11T09:51:47.583 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.583 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.583 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:47.583 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.490774+0000", 2024-09-11T09:51:47.583 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1823e-05 2024-09-11T09:51:47.583 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.583 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.584 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:47.584 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.490846+0000", 2024-09-11T09:51:47.584 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.1878000000000005e-05 2024-09-11T09:51:47.584 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.584 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.584 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for peered", 2024-09-11T09:51:47.584 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.490857+0000", 2024-09-11T09:51:47.584 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1017e-05 2024-09-11T09:51:47.584 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.584 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.584 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:47.584 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.493856+0000", 2024-09-11T09:51:47.584 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00299885 2024-09-11T09:51:47.585 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.585 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.585 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:47.585 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.494020+0000", 2024-09-11T09:51:47.585 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00016436599999999999 2024-09-11T09:51:47.585 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.585 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.585 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 2", 2024-09-11T09:51:47.585 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.494186+0000", 2024-09-11T09:51:47.585 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00016562799999999999 2024-09-11T09:51:47.585 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.585 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.585 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:47.585 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.495407+0000", 2024-09-11T09:51:47.586 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0012217420000000001 2024-09-11T09:51:47.586 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.586 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.586 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:47.586 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.496579+0000", 2024-09-11T09:51:47.586 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001171844 2024-09-11T09:51:47.586 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.586 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.586 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:47.586 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.496620+0000", 2024-09-11T09:51:47.586 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.0479000000000003e-05 2024-09-11T09:51:47.586 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.586 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.586 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:47.587 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.496646+0000", 2024-09-11T09:51:47.587 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.6472e-05 2024-09-11T09:51:47.587 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:47.587 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:47.587 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:47.587 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.587 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.587 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4230.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-09-11T09:51:47.587 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:20.498634+0000", 2024-09-11T09:51:47.587 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 87.081049514, 2024-09-11T09:51:47.587 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0050648539999999997, 2024-09-11T09:51:47.587 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:47.587 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:47.588 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:47.588 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:47.588 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4230", 2024-09-11T09:51:47.588 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.17:0/4101109576", 2024-09-11T09:51:47.588 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 7 2024-09-11T09:51:47.588 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.588 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:47.588 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.588 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:47.588 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.498634+0000", 2024-09-11T09:51:47.588 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:47.588 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.588 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.589 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:47.589 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.498634+0000", 2024-09-11T09:51:47.589 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:47.589 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.589 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.589 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:47.589 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.498637+0000", 2024-09-11T09:51:47.589 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.1919999999999999e-06 2024-09-11T09:51:47.589 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.589 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.589 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:47.589 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.498648+0000", 2024-09-11T09:51:47.589 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1116e-05 2024-09-11T09:51:47.589 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.590 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.590 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:47.590 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.498666+0000", 2024-09-11T09:51:47.590 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.8411999999999999e-05 2024-09-11T09:51:47.590 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.590 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.590 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:47.590 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.498675+0000", 2024-09-11T09:51:47.590 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.4059999999999994e-06 2024-09-11T09:51:47.590 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.590 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.590 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:47.590 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.498745+0000", 2024-09-11T09:51:47.590 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.9764000000000005e-05 2024-09-11T09:51:47.591 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.591 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.591 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op must wait for map", 2024-09-11T09:51:47.591 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.498754+0000", 2024-09-11T09:51:47.591 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.5969999999999992e-06 2024-09-11T09:51:47.591 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.591 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.591 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:47.591 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.500385+0000", 2024-09-11T09:51:47.591 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0016308360000000001 2024-09-11T09:51:47.591 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.591 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.591 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:47.592 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.500473+0000", 2024-09-11T09:51:47.592 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.7966000000000004e-05 2024-09-11T09:51:47.592 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.592 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.592 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 1", 2024-09-11T09:51:47.592 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.500591+0000", 2024-09-11T09:51:47.592 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000117895 2024-09-11T09:51:47.592 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.592 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.592 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:47.592 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.501685+0000", 2024-09-11T09:51:47.592 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001094886 2024-09-11T09:51:47.592 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.593 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.593 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:47.593 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.503627+0000", 2024-09-11T09:51:47.593 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001941199 2024-09-11T09:51:47.593 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.593 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.593 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:47.593 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.503667+0000", 2024-09-11T09:51:47.593 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.0645000000000001e-05 2024-09-11T09:51:47.593 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.593 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.593 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:47.593 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.503699+0000", 2024-09-11T09:51:47.594 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.1940000000000003e-05 2024-09-11T09:51:47.594 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:47.594 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:47.594 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:47.594 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.594 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.594 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4688.0:32 2.2 2:45941fd2:::rbd_data.1250f5e25b80.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:47.594 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:22.896607+0000", 2024-09-11T09:51:47.594 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 24.683076911000001, 2024-09-11T09:51:47.594 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.038655179999999997, 2024-09-11T09:51:47.594 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:47.594 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:47.594 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:47.595 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:47.595 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4688", 2024-09-11T09:51:47.595 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:51:47.595 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 32 2024-09-11T09:51:47.595 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.595 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:47.595 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.595 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:47.595 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.896607+0000", 2024-09-11T09:51:47.595 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:47.595 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.595 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.595 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:47.595 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.896607+0000", 2024-09-11T09:51:47.596 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:47.596 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.596 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.596 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:47.596 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.896608+0000", 2024-09-11T09:51:47.596 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.0690000000000001e-06 2024-09-11T09:51:47.596 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.596 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.596 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:47.596 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.914461+0000", 2024-09-11T09:51:47.596 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.017852903 2024-09-11T09:51:47.596 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.596 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.596 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:47.597 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.914495+0000", 2024-09-11T09:51:47.597 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.4301000000000002e-05 2024-09-11T09:51:47.597 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.597 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.597 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:47.597 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.914522+0000", 2024-09-11T09:51:47.597 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.7127e-05 2024-09-11T09:51:47.597 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.597 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.597 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:47.597 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.914583+0000", 2024-09-11T09:51:47.597 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.0730999999999998e-05 2024-09-11T09:51:47.597 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.598 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.598 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:47.598 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.914709+0000", 2024-09-11T09:51:47.598 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00012590400000000001 2024-09-11T09:51:47.598 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.598 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.598 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 1", 2024-09-11T09:51:47.598 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.914809+0000", 2024-09-11T09:51:47.598 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.9882000000000004e-05 2024-09-11T09:51:47.598 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.598 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.598 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:47.598 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.932551+0000", 2024-09-11T09:51:47.598 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.017742048999999999 2024-09-11T09:51:47.599 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.599 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.599 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:47.599 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.935144+0000", 2024-09-11T09:51:47.599 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0025927630000000001 2024-09-11T09:51:47.599 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.599 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.599 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:47.599 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.935205+0000", 2024-09-11T09:51:47.599 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.1634e-05 2024-09-11T09:51:47.599 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.599 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.599 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:47.599 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.935262+0000", 2024-09-11T09:51:47.600 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.6817000000000002e-05 2024-09-11T09:51:47.600 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:47.600 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:47.600 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:47.600 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.600 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.600 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_repop(client.4688.0:40 2.7 e53/38 2:fe41411c:::rbd_object_map.1250f5e25b80:head v 53'9, mlcod=53'7)", 2024-09-11T09:51:47.600 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:22.920457+0000", 2024-09-11T09:51:47.600 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 24.659226820000001, 2024-09-11T09:51:47.600 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.011132817, 2024-09-11T09:51:47.600 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:47.600 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:47.600 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:47.601 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:47.601 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.601 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:47.601 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.920457+0000", 2024-09-11T09:51:47.601 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:47.601 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.601 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.601 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:47.601 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.920457+0000", 2024-09-11T09:51:47.601 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:47.601 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.601 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.601 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:47.601 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.920459+0000", 2024-09-11T09:51:47.602 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.6729999999999999e-06 2024-09-11T09:51:47.602 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.602 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.602 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:47.602 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.920469+0000", 2024-09-11T09:51:47.602 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.944e-06 2024-09-11T09:51:47.602 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.602 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.602 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:47.602 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.920495+0000", 2024-09-11T09:51:47.602 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.6401999999999999e-05 2024-09-11T09:51:47.602 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.602 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.602 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:47.603 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.920506+0000", 2024-09-11T09:51:47.603 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.133e-05 2024-09-11T09:51:47.603 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.603 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.603 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:47.603 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.920581+0000", 2024-09-11T09:51:47.603 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.4932999999999996e-05 2024-09-11T09:51:47.603 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.603 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.603 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:47.603 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.920612+0000", 2024-09-11T09:51:47.603 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.0537999999999998e-05 2024-09-11T09:51:47.603 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.603 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.604 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:47.604 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.931537+0000", 2024-09-11T09:51:47.604 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.010925239 2024-09-11T09:51:47.604 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.604 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.604 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:47.604 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.931590+0000", 2024-09-11T09:51:47.604 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.2757999999999999e-05 2024-09-11T09:51:47.604 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:47.604 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:47.604 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:47.604 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.604 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.605 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_repop(client.4688.0:41 2.7 e53/38 2:fe41411c:::rbd_object_map.1250f5e25b80:head v 53'10, mlcod=53'7)", 2024-09-11T09:51:47.605 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:22.921422+0000", 2024-09-11T09:51:47.605 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 24.658261404000001, 2024-09-11T09:51:47.605 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.013734942, 2024-09-11T09:51:47.605 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:47.605 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:47.605 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:47.605 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:47.605 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.605 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:47.605 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921422+0000", 2024-09-11T09:51:47.605 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:47.605 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.606 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.606 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:47.606 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921422+0000", 2024-09-11T09:51:47.606 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:47.606 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.606 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.606 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:47.606 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921426+0000", 2024-09-11T09:51:47.606 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.6150000000000001e-06 2024-09-11T09:51:47.606 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.606 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.606 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:47.606 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921435+0000", 2024-09-11T09:51:47.607 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.9840000000000007e-06 2024-09-11T09:51:47.607 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.607 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.607 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:47.607 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921444+0000", 2024-09-11T09:51:47.607 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.6000000000000007e-06 2024-09-11T09:51:47.607 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.607 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.607 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:47.607 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921450+0000", 2024-09-11T09:51:47.607 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.4690000000000002e-06 2024-09-11T09:51:47.607 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.607 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.608 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:47.608 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921511+0000", 2024-09-11T09:51:47.608 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.0779999999999997e-05 2024-09-11T09:51:47.608 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.608 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.608 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:47.608 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921531+0000", 2024-09-11T09:51:47.608 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.0302999999999999e-05 2024-09-11T09:51:47.608 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.608 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.608 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:47.608 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.935115+0000", 2024-09-11T09:51:47.608 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.013583471 2024-09-11T09:51:47.608 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.609 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.609 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:47.609 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.935157+0000", 2024-09-11T09:51:47.609 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.2719999999999998e-05 2024-09-11T09:51:47.609 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:47.609 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:47.609 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:47.609 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.609 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.609 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4688.0:42 2.3 2:c54742d8:::rbd_data.1250f5e25b80.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:47.609 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:22.932560+0000", 2024-09-11T09:51:47.609 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 24.647123660999998, 2024-09-11T09:51:47.609 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.017798198000000001, 2024-09-11T09:51:47.610 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:47.610 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:47.610 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:47.610 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:47.610 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4688", 2024-09-11T09:51:47.610 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:51:47.610 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 42 2024-09-11T09:51:47.610 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.610 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:47.610 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.610 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:47.610 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.932560+0000", 2024-09-11T09:51:47.610 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:47.610 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.611 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.611 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:47.611 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.932560+0000", 2024-09-11T09:51:47.611 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:47.611 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.611 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.611 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:47.611 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.932562+0000", 2024-09-11T09:51:47.611 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.17e-06 2024-09-11T09:51:47.611 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.611 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.611 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:47.611 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.938851+0000", 2024-09-11T09:51:47.611 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.006288971 2024-09-11T09:51:47.612 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.612 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.612 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:47.612 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.938865+0000", 2024-09-11T09:51:47.612 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.365e-05 2024-09-11T09:51:47.612 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.612 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.612 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:47.612 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.938871+0000", 2024-09-11T09:51:47.612 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.8850000000000004e-06 2024-09-11T09:51:47.612 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.612 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.612 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:47.612 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.938926+0000", 2024-09-11T09:51:47.613 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.4691999999999997e-05 2024-09-11T09:51:47.613 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.613 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.613 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:47.613 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.939059+0000", 2024-09-11T09:51:47.613 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00013310899999999999 2024-09-11T09:51:47.613 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.613 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.613 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 2", 2024-09-11T09:51:47.613 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.939171+0000", 2024-09-11T09:51:47.613 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000112718 2024-09-11T09:51:47.613 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.613 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.614 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:47.614 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.950262+0000", 2024-09-11T09:51:47.614 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.011091063 2024-09-11T09:51:47.614 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.614 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.614 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:47.614 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.950274+0000", 2024-09-11T09:51:47.614 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1963e-05 2024-09-11T09:51:47.614 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.614 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.614 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:47.614 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.950318+0000", 2024-09-11T09:51:47.614 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.3519999999999997e-05 2024-09-11T09:51:47.615 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.615 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.615 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:47.615 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.950358+0000", 2024-09-11T09:51:47.615 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.0457000000000002e-05 2024-09-11T09:51:47.615 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:47.615 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:47.615 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:47.615 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.615 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.615 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_repop(client.4694.0:34 2.7 e53/38 2:f80bfc0c:ns::rbd_data.1256a0c02879.0000000000000002:head v 53'25, mlcod=53'23)", 2024-09-11T09:51:47.615 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.094944+0000", 2024-09-11T09:51:47.615 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 24.484740269, 2024-09-11T09:51:47.616 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.010896785000000001, 2024-09-11T09:51:47.616 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:47.616 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:47.616 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:47.616 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:47.616 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.616 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:47.616 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.094944+0000", 2024-09-11T09:51:47.616 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:47.617 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.617 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.617 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:47.617 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.094944+0000", 2024-09-11T09:51:47.617 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:47.617 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.617 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.617 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:47.617 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.094944+0000", 2024-09-11T09:51:47.617 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.0699999999999996e-07 2024-09-11T09:51:47.617 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.617 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.617 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:47.618 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.099060+0000", 2024-09-11T09:51:47.618 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0041155580000000001 2024-09-11T09:51:47.618 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.618 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.618 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:47.618 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.099082+0000", 2024-09-11T09:51:47.618 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.2323999999999998e-05 2024-09-11T09:51:47.618 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.618 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.618 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:47.618 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.099094+0000", 2024-09-11T09:51:47.618 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1535000000000001e-05 2024-09-11T09:51:47.618 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.618 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.619 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:47.619 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.099160+0000", 2024-09-11T09:51:47.619 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.6449e-05 2024-09-11T09:51:47.619 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.619 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.619 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:47.619 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.099183+0000", 2024-09-11T09:51:47.619 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.3108e-05 2024-09-11T09:51:47.619 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.619 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.619 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:47.619 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.105776+0000", 2024-09-11T09:51:47.619 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0065922760000000002 2024-09-11T09:51:47.619 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.620 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.620 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:47.620 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.105840+0000", 2024-09-11T09:51:47.620 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.4727999999999994e-05 2024-09-11T09:51:47.620 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:47.620 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:47.620 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:47.620 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.620 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.620 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_repop(client.4694.0:35 2.7 e53/38 2:e8766325:ns::rbd_object_map.1256a0c02879:head v 53'26, mlcod=53'23)", 2024-09-11T09:51:47.620 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.100964+0000", 2024-09-11T09:51:47.620 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 24.478719740999999, 2024-09-11T09:51:47.620 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0049649359999999997, 2024-09-11T09:51:47.620 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:47.621 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:47.621 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:47.621 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:47.621 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.621 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:47.621 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.100964+0000", 2024-09-11T09:51:47.621 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:47.621 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.621 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.621 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:47.621 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.100964+0000", 2024-09-11T09:51:47.621 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:47.621 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.622 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.622 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:47.622 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.100966+0000", 2024-09-11T09:51:47.622 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.6330000000000001e-06 2024-09-11T09:51:47.622 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.622 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.622 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:47.622 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.100979+0000", 2024-09-11T09:51:47.622 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.2813000000000001e-05 2024-09-11T09:51:47.622 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.622 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.622 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:47.622 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.100992+0000", 2024-09-11T09:51:47.622 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.3414999999999999e-05 2024-09-11T09:51:47.623 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.623 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.623 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:47.623 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.101002+0000", 2024-09-11T09:51:47.623 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.0159e-05 2024-09-11T09:51:47.623 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.623 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.623 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:47.623 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.102365+0000", 2024-09-11T09:51:47.623 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001362912 2024-09-11T09:51:47.623 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.623 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.623 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:47.623 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.102382+0000", 2024-09-11T09:51:47.624 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.7433000000000001e-05 2024-09-11T09:51:47.624 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.624 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.624 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:47.624 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.105901+0000", 2024-09-11T09:51:47.624 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.003518476 2024-09-11T09:51:47.624 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.624 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.624 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:47.624 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.105929+0000", 2024-09-11T09:51:47.624 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.8095000000000001e-05 2024-09-11T09:51:47.624 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:47.624 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:47.625 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:47.625 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.625 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.625 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4694.0:41 2.2 2:5e2505c4:ns::rbd_data.1256a0c02879.0000000000000003:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:47.625 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.107511+0000", 2024-09-11T09:51:47.625 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 24.472172587999999, 2024-09-11T09:51:47.625 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.018571066000000001, 2024-09-11T09:51:47.625 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:47.625 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:47.625 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:47.625 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:47.625 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4694", 2024-09-11T09:51:47.625 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:51:47.625 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 41 2024-09-11T09:51:47.626 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.626 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:47.626 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.626 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:47.626 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.107511+0000", 2024-09-11T09:51:47.626 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:47.626 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.626 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.626 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:47.626 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.107511+0000", 2024-09-11T09:51:47.626 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:47.626 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.626 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.626 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:47.627 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.107513+0000", 2024-09-11T09:51:47.627 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.3489999999999999e-06 2024-09-11T09:51:47.627 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.627 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.627 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:47.627 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.112704+0000", 2024-09-11T09:51:47.627 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0051918509999999999 2024-09-11T09:51:47.627 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.627 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.627 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:47.627 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.112718+0000", 2024-09-11T09:51:47.627 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.359e-05 2024-09-11T09:51:47.627 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.628 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.628 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:47.628 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.112725+0000", 2024-09-11T09:51:47.628 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.9940000000000003e-06 2024-09-11T09:51:47.628 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.628 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.628 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:47.628 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.112788+0000", 2024-09-11T09:51:47.628 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.2447999999999996e-05 2024-09-11T09:51:47.628 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.628 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.628 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:47.628 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.112937+0000", 2024-09-11T09:51:47.629 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00014921499999999999 2024-09-11T09:51:47.629 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.629 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.629 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 1", 2024-09-11T09:51:47.629 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.113085+0000", 2024-09-11T09:51:47.629 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000148085 2024-09-11T09:51:47.629 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.629 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.629 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:47.629 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.124318+0000", 2024-09-11T09:51:47.629 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.011233158 2024-09-11T09:51:47.629 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.629 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.629 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:47.630 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.125957+0000", 2024-09-11T09:51:47.630 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001638701 2024-09-11T09:51:47.630 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.630 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.630 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:47.630 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.126029+0000", 2024-09-11T09:51:47.630 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.2125000000000005e-05 2024-09-11T09:51:47.630 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.630 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.630 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:47.630 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.126082+0000", 2024-09-11T09:51:47.630 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.3550000000000001e-05 2024-09-11T09:51:47.630 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:47.630 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:47.631 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:47.631 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.631 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.631 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4694.0:44 2.3 2:cd1a7d1c:ns::rbd_data.1256a0c02879.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:47.631 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.112744+0000", 2024-09-11T09:51:47.631 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 24.466940020999999, 2024-09-11T09:51:47.631 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.021672212999999999, 2024-09-11T09:51:47.631 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:47.631 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:47.631 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:47.631 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:47.631 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4694", 2024-09-11T09:51:47.631 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:51:47.632 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 44 2024-09-11T09:51:47.632 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.632 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:47.632 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.632 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:47.632 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.112744+0000", 2024-09-11T09:51:47.632 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:47.632 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.632 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.632 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:47.632 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.112744+0000", 2024-09-11T09:51:47.632 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:47.632 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.633 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.633 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:47.633 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.112745+0000", 2024-09-11T09:51:47.633 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.2900000000000002e-07 2024-09-11T09:51:47.633 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.633 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.633 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:47.633 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.119929+0000", 2024-09-11T09:51:47.633 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0071846230000000002 2024-09-11T09:51:47.633 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.633 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.633 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:47.633 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.119949+0000", 2024-09-11T09:51:47.634 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.9264000000000001e-05 2024-09-11T09:51:47.634 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.634 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.634 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:47.634 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.119959+0000", 2024-09-11T09:51:47.634 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.0465999999999999e-05 2024-09-11T09:51:47.634 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.634 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.634 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:47.634 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.120028+0000", 2024-09-11T09:51:47.634 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.9318000000000003e-05 2024-09-11T09:51:47.634 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.634 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.634 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:47.635 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.120160+0000", 2024-09-11T09:51:47.635 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000132015 2024-09-11T09:51:47.635 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.635 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.635 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 2", 2024-09-11T09:51:47.635 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.120260+0000", 2024-09-11T09:51:47.635 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.9397999999999999e-05 2024-09-11T09:51:47.635 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.635 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.635 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:47.635 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.134246+0000", 2024-09-11T09:51:47.635 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.013986721000000001 2024-09-11T09:51:47.635 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.635 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.635 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:47.636 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.134295+0000", 2024-09-11T09:51:47.636 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.8649000000000001e-05 2024-09-11T09:51:47.636 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.636 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.636 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:47.636 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.134361+0000", 2024-09-11T09:51:47.636 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.6081999999999995e-05 2024-09-11T09:51:47.636 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.636 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.636 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:47.636 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.134416+0000", 2024-09-11T09:51:47.636 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.4848000000000001e-05 2024-09-11T09:51:47.636 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:47.636 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:47.636 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:47.637 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.637 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.637 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4694.0:46 2.3 2:ce0e1469:ns::rbd_data.1256a0c02879.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:47.637 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.119993+0000", 2024-09-11T09:51:47.637 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 24.459690545000001, 2024-09-11T09:51:47.637 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.025840610999999999, 2024-09-11T09:51:47.637 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:47.637 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:47.637 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:47.637 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:47.637 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4694", 2024-09-11T09:51:47.637 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:51:47.638 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 46 2024-09-11T09:51:47.638 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.638 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:47.638 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.638 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:47.638 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.119993+0000", 2024-09-11T09:51:47.638 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:47.638 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.638 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.638 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:47.638 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.119993+0000", 2024-09-11T09:51:47.638 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:47.638 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.638 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.639 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:47.639 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.119995+0000", 2024-09-11T09:51:47.639 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1930000000000001e-06 2024-09-11T09:51:47.639 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.639 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.639 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:47.639 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.128458+0000", 2024-09-11T09:51:47.639 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0084632630000000004 2024-09-11T09:51:47.639 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.639 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.639 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:47.639 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.128478+0000", 2024-09-11T09:51:47.639 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.0353e-05 2024-09-11T09:51:47.639 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.640 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.640 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:47.640 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.128491+0000", 2024-09-11T09:51:47.640 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.2954e-05 2024-09-11T09:51:47.640 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.640 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.640 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:47.640 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.128558+0000", 2024-09-11T09:51:47.640 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.7076999999999995e-05 2024-09-11T09:51:47.640 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.640 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.640 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:47.640 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.128708+0000", 2024-09-11T09:51:47.641 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00014939599999999999 2024-09-11T09:51:47.641 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.641 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.641 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 2", 2024-09-11T09:51:47.641 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.128866+0000", 2024-09-11T09:51:47.641 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000158145 2024-09-11T09:51:47.641 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.641 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.641 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:47.641 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.141045+0000", 2024-09-11T09:51:47.641 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.012179331999999999 2024-09-11T09:51:47.641 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.641 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.642 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:47.642 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.145742+0000", 2024-09-11T09:51:47.642 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0046972050000000003 2024-09-11T09:51:47.642 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.642 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.642 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:47.642 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.145779+0000", 2024-09-11T09:51:47.642 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.6544000000000001e-05 2024-09-11T09:51:47.642 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.642 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.642 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:47.642 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.145834+0000", 2024-09-11T09:51:47.642 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.5148999999999997e-05 2024-09-11T09:51:47.643 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:47.643 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:47.643 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:47.643 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.643 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.643 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4694.0:48 2.3 2:d2090b70:ns::rbd_data.1256a0c02879.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:47.643 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.130540+0000", 2024-09-11T09:51:47.643 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 24.449144278999999, 2024-09-11T09:51:47.643 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.017167789999999999, 2024-09-11T09:51:47.643 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:47.643 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:47.643 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:47.643 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:47.644 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4694", 2024-09-11T09:51:47.644 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:51:47.644 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 48 2024-09-11T09:51:47.644 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.644 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:47.644 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.644 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:47.644 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.130540+0000", 2024-09-11T09:51:47.644 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:47.644 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.644 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.644 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:47.644 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.130540+0000", 2024-09-11T09:51:47.644 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:47.645 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.645 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.645 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:47.645 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.130542+0000", 2024-09-11T09:51:47.645 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.9209999999999999e-06 2024-09-11T09:51:47.645 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.645 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.645 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:47.645 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.134691+0000", 2024-09-11T09:51:47.645 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0041498739999999996 2024-09-11T09:51:47.645 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.645 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.645 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:47.646 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.134705+0000", 2024-09-11T09:51:47.646 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.3669000000000001e-05 2024-09-11T09:51:47.646 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.646 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.646 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:47.646 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.134712+0000", 2024-09-11T09:51:47.646 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.8410000000000002e-06 2024-09-11T09:51:47.646 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.646 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.646 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:47.646 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.134754+0000", 2024-09-11T09:51:47.646 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.2524999999999997e-05 2024-09-11T09:51:47.646 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.646 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.647 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:47.647 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.134875+0000", 2024-09-11T09:51:47.647 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000120332 2024-09-11T09:51:47.647 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.647 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.647 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 2", 2024-09-11T09:51:47.647 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.135005+0000", 2024-09-11T09:51:47.647 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00013024900000000001 2024-09-11T09:51:47.647 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.647 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.647 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:47.647 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.145971+0000", 2024-09-11T09:51:47.647 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.010966203000000001 2024-09-11T09:51:47.647 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.648 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.648 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:47.648 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.147600+0000", 2024-09-11T09:51:47.648 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001628382 2024-09-11T09:51:47.648 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.648 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.648 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:47.648 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.147635+0000", 2024-09-11T09:51:47.648 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.5420000000000003e-05 2024-09-11T09:51:47.648 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.648 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.648 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:47.648 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.147707+0000", 2024-09-11T09:51:47.649 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.2373999999999995e-05 2024-09-11T09:51:47.649 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:47.649 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:47.649 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:47.649 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.649 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.649 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_repop(client.4694.0:49 2.7 e53/38 2:e8766325:ns::rbd_object_map.1256a0c02879:head v 53'33, mlcod=53'31)", 2024-09-11T09:51:47.649 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.135391+0000", 2024-09-11T09:51:47.649 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 24.444292713999999, 2024-09-11T09:51:47.649 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0051189859999999998, 2024-09-11T09:51:47.649 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:47.649 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:47.649 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:47.650 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:47.650 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.650 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:47.650 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.135391+0000", 2024-09-11T09:51:47.650 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:47.650 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.650 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.650 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:47.650 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.135391+0000", 2024-09-11T09:51:47.650 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:47.650 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.650 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.650 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:47.650 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.135393+0000", 2024-09-11T09:51:47.651 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.4020000000000001e-06 2024-09-11T09:51:47.651 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.651 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.651 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:47.651 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.135400+0000", 2024-09-11T09:51:47.651 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.5129999999999999e-06 2024-09-11T09:51:47.651 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.651 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.651 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:47.651 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.135411+0000", 2024-09-11T09:51:47.651 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1038e-05 2024-09-11T09:51:47.651 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.651 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.652 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:47.652 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.135418+0000", 2024-09-11T09:51:47.652 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.1489999999999997e-06 2024-09-11T09:51:47.652 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.652 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.652 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:47.652 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.135470+0000", 2024-09-11T09:51:47.652 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.2021999999999998e-05 2024-09-11T09:51:47.652 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.652 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.652 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:47.652 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.135501+0000", 2024-09-11T09:51:47.652 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.0349999999999999e-05 2024-09-11T09:51:47.653 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.653 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.653 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:47.653 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.140453+0000", 2024-09-11T09:51:47.653 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0049522409999999996 2024-09-11T09:51:47.653 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.653 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.653 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:47.653 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.140510+0000", 2024-09-11T09:51:47.653 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.7271000000000001e-05 2024-09-11T09:51:47.653 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:47.653 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:47.653 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:47.653 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.654 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.654 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4694.0:54 2.3 2:c44f0253:ns::rbd_data.1256a0c02879.000000000000000b:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:47.654 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.150151+0000", 2024-09-11T09:51:47.654 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 24.429532862999999, 2024-09-11T09:51:47.654 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.015103484, 2024-09-11T09:51:47.654 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:47.654 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:47.654 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:47.654 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:47.654 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4694", 2024-09-11T09:51:47.654 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:51:47.654 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 54 2024-09-11T09:51:47.654 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.654 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:47.655 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.655 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:47.655 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.150151+0000", 2024-09-11T09:51:47.655 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:47.655 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.655 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.655 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:47.655 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.150151+0000", 2024-09-11T09:51:47.655 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:47.655 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.655 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.655 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:47.655 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.150152+0000", 2024-09-11T09:51:47.656 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.2550000000000001e-06 2024-09-11T09:51:47.656 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.656 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.656 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:47.656 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.154528+0000", 2024-09-11T09:51:47.656 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0043758219999999997 2024-09-11T09:51:47.656 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.656 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.656 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:47.656 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.154541+0000", 2024-09-11T09:51:47.656 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.3414999999999999e-05 2024-09-11T09:51:47.656 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.656 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.657 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:47.657 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.154551+0000", 2024-09-11T09:51:47.657 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.4350000000000003e-06 2024-09-11T09:51:47.657 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.657 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.657 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:47.657 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.154615+0000", 2024-09-11T09:51:47.657 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.3627000000000006e-05 2024-09-11T09:51:47.657 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.657 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.657 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:47.657 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.154717+0000", 2024-09-11T09:51:47.657 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000102729 2024-09-11T09:51:47.657 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.658 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.658 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 2", 2024-09-11T09:51:47.658 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.154826+0000", 2024-09-11T09:51:47.658 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000108409 2024-09-11T09:51:47.658 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.658 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.658 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:47.658 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.165094+0000", 2024-09-11T09:51:47.658 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.010268210999999999 2024-09-11T09:51:47.658 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.658 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.658 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:47.658 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.165138+0000", 2024-09-11T09:51:47.659 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.4078e-05 2024-09-11T09:51:47.659 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.659 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.659 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:47.659 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.165204+0000", 2024-09-11T09:51:47.659 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.5716000000000006e-05 2024-09-11T09:51:47.659 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.659 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.659 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:47.659 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.165254+0000", 2024-09-11T09:51:47.659 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.0787e-05 2024-09-11T09:51:47.660 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:47.660 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:47.660 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:47.660 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.660 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.660 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_repop(client.4694.0:55 2.7 e53/38 2:e8766325:ns::rbd_object_map.1256a0c02879:head v 53'37, mlcod=53'34)", 2024-09-11T09:51:47.660 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.150221+0000", 2024-09-11T09:51:47.660 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 24.429462997999998, 2024-09-11T09:51:47.660 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00530015, 2024-09-11T09:51:47.660 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:47.660 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:47.660 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:47.660 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:47.661 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.661 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:47.661 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.150221+0000", 2024-09-11T09:51:47.661 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:47.661 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.661 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.661 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:47.661 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.150221+0000", 2024-09-11T09:51:47.661 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:47.661 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.661 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.661 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:47.661 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.150221+0000", 2024-09-11T09:51:47.662 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.9100000000000004e-07 2024-09-11T09:51:47.662 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.662 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.662 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:47.662 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.150226+0000", 2024-09-11T09:51:47.662 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.2250000000000002e-06 2024-09-11T09:51:47.662 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.662 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.662 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:47.662 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.150251+0000", 2024-09-11T09:51:47.662 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.4987e-05 2024-09-11T09:51:47.662 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.662 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.662 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:47.662 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.150258+0000", 2024-09-11T09:51:47.663 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.0790000000000003e-06 2024-09-11T09:51:47.663 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.663 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.663 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:47.663 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.150305+0000", 2024-09-11T09:51:47.663 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.6879000000000003e-05 2024-09-11T09:51:47.663 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.663 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.663 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:47.663 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.150320+0000", 2024-09-11T09:51:47.663 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.5268999999999999e-05 2024-09-11T09:51:47.663 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.663 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.664 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:47.664 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.155493+0000", 2024-09-11T09:51:47.664 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0051731390000000002 2024-09-11T09:51:47.664 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.664 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.664 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:47.664 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.155521+0000", 2024-09-11T09:51:47.664 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.8081000000000001e-05 2024-09-11T09:51:47.664 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:47.664 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:47.664 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:47.664 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.664 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.664 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4694.0:56 2.3 2:c462f029:ns::rbd_data.1256a0c02879.000000000000000c:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:47.665 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.154576+0000", 2024-09-11T09:51:47.665 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 24.425108022, 2024-09-11T09:51:47.665 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.020781621, 2024-09-11T09:51:47.665 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:47.665 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:47.665 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:47.665 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:47.665 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4694", 2024-09-11T09:51:47.665 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:51:47.665 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 56 2024-09-11T09:51:47.665 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.665 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:47.666 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.666 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:47.666 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.154576+0000", 2024-09-11T09:51:47.666 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:47.666 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.666 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.666 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:47.666 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.154576+0000", 2024-09-11T09:51:47.666 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:47.666 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.666 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.666 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:47.666 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.154577+0000", 2024-09-11T09:51:47.666 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.2249999999999999e-06 2024-09-11T09:51:47.667 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.667 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.667 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:47.667 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.158783+0000", 2024-09-11T09:51:47.667 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0042063889999999996 2024-09-11T09:51:47.667 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.667 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.667 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:47.667 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.158803+0000", 2024-09-11T09:51:47.667 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.9576999999999999e-05 2024-09-11T09:51:47.667 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.667 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.667 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:47.667 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.158815+0000", 2024-09-11T09:51:47.668 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.2221999999999999e-05 2024-09-11T09:51:47.668 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.668 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.668 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:47.668 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.158902+0000", 2024-09-11T09:51:47.668 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.6390000000000005e-05 2024-09-11T09:51:47.668 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.668 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.668 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:47.668 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.159141+0000", 2024-09-11T09:51:47.668 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000238886 2024-09-11T09:51:47.668 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.668 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.668 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 2", 2024-09-11T09:51:47.669 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.159279+0000", 2024-09-11T09:51:47.669 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000138073 2024-09-11T09:51:47.669 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.669 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.669 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:47.669 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.173297+0000", 2024-09-11T09:51:47.669 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.014018113 2024-09-11T09:51:47.669 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.669 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.669 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:47.669 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.175278+0000", 2024-09-11T09:51:47.669 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0019817390000000002 2024-09-11T09:51:47.669 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.669 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.670 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:47.670 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.175316+0000", 2024-09-11T09:51:47.670 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.7829e-05 2024-09-11T09:51:47.670 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.670 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.670 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:47.670 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.175357+0000", 2024-09-11T09:51:47.670 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.1177999999999998e-05 2024-09-11T09:51:47.670 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:47.670 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:47.670 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:47.670 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.670 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.670 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4694.0:59 2.2 2:41adefc9:ns::rbd_data.1256a0c02879.000000000000000e:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:47.671 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.158843+0000", 2024-09-11T09:51:47.671 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 24.420840456000001, 2024-09-11T09:51:47.671 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.017874207, 2024-09-11T09:51:47.671 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:47.671 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:47.671 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:47.671 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:47.671 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4694", 2024-09-11T09:51:47.671 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:51:47.671 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 59 2024-09-11T09:51:47.671 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.671 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:47.671 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.671 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:47.671 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.158843+0000", 2024-09-11T09:51:47.672 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:47.672 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.672 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.672 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:47.672 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.158843+0000", 2024-09-11T09:51:47.672 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:47.672 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.672 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.672 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:47.672 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.158845+0000", 2024-09-11T09:51:47.672 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.7439999999999999e-06 2024-09-11T09:51:47.672 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.672 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.673 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:47.673 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.162713+0000", 2024-09-11T09:51:47.673 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0038678829999999999 2024-09-11T09:51:47.673 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.673 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.673 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:47.673 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.162728+0000", 2024-09-11T09:51:47.673 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.5114e-05 2024-09-11T09:51:47.673 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.673 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.673 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:47.673 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.162737+0000", 2024-09-11T09:51:47.673 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.1099999999999992e-06 2024-09-11T09:51:47.673 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.674 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.674 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:47.674 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.162799+0000", 2024-09-11T09:51:47.674 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.1934999999999996e-05 2024-09-11T09:51:47.674 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.674 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.674 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:47.674 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.162911+0000", 2024-09-11T09:51:47.674 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000111867 2024-09-11T09:51:47.674 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.674 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.674 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 1", 2024-09-11T09:51:47.674 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.163030+0000", 2024-09-11T09:51:47.674 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000119108 2024-09-11T09:51:47.675 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.675 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.675 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:47.675 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.174086+0000", 2024-09-11T09:51:47.675 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.01105543 2024-09-11T09:51:47.675 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.675 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.675 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:47.675 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.176614+0000", 2024-09-11T09:51:47.675 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0025284940000000001 2024-09-11T09:51:47.675 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.675 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.675 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:47.675 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.176672+0000", 2024-09-11T09:51:47.676 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.7892000000000001e-05 2024-09-11T09:51:47.676 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.676 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.676 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:47.676 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.176718+0000", 2024-09-11T09:51:47.676 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.5630000000000002e-05 2024-09-11T09:51:47.676 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:47.676 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:47.676 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:47.676 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.676 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.676 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.5002.0:89 2.3 2:c54742d8:::rbd_data.1250f5e25b80.0000000000000006:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-09-11T09:51:47.677 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:38.546016+0000", 2024-09-11T09:51:47.677 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 9.0336679100000001, 2024-09-11T09:51:47.677 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0049098550000000003, 2024-09-11T09:51:47.677 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:47.677 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:47.677 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:47.677 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:47.677 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.5002", 2024-09-11T09:51:47.677 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/2204440692", 2024-09-11T09:51:47.677 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 89 2024-09-11T09:51:47.677 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.677 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:47.677 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.677 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:47.678 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.546016+0000", 2024-09-11T09:51:47.678 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:47.678 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.678 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.678 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:47.678 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.546016+0000", 2024-09-11T09:51:47.678 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:47.678 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.678 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.678 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:47.678 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.546017+0000", 2024-09-11T09:51:47.678 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.3399999999999998e-07 2024-09-11T09:51:47.678 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.678 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.679 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:47.679 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.546839+0000", 2024-09-11T09:51:47.679 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00082222800000000004 2024-09-11T09:51:47.679 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.679 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.679 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:47.679 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.546851+0000", 2024-09-11T09:51:47.679 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.2075999999999999e-05 2024-09-11T09:51:47.679 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.679 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.679 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:47.679 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.546856+0000", 2024-09-11T09:51:47.679 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.4009999999999996e-06 2024-09-11T09:51:47.680 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.680 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.680 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:47.680 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.546897+0000", 2024-09-11T09:51:47.680 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.0113e-05 2024-09-11T09:51:47.680 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.680 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.680 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:47.680 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.546954+0000", 2024-09-11T09:51:47.680 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.7825000000000003e-05 2024-09-11T09:51:47.680 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.680 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.680 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 2", 2024-09-11T09:51:47.681 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.547055+0000", 2024-09-11T09:51:47.681 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000100269 2024-09-11T09:51:47.681 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.681 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.681 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:47.681 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.549675+0000", 2024-09-11T09:51:47.681 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0026206609999999998 2024-09-11T09:51:47.681 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.681 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.681 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:47.681 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.550858+0000", 2024-09-11T09:51:47.681 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0011825270000000001 2024-09-11T09:51:47.681 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.682 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.682 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:47.682 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.550894+0000", 2024-09-11T09:51:47.682 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.5896999999999999e-05 2024-09-11T09:51:47.682 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.682 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.682 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:47.682 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.550926+0000", 2024-09-11T09:51:47.682 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.2023999999999997e-05 2024-09-11T09:51:47.682 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:47.682 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:47.682 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:47.682 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.682 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.683 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.5002.0:91 2.3 2:c54742d8:::rbd_data.1250f5e25b80.0000000000000006:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-09-11T09:51:47.683 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:38.553743+0000", 2024-09-11T09:51:47.683 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 9.0259408669999992, 2024-09-11T09:51:47.683 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0051633629999999998, 2024-09-11T09:51:47.683 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:47.683 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:47.683 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:47.683 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:47.683 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.5002", 2024-09-11T09:51:47.683 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/2204440692", 2024-09-11T09:51:47.683 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 91 2024-09-11T09:51:47.683 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.683 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:47.683 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.684 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:47.684 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.553743+0000", 2024-09-11T09:51:47.684 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:47.684 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.684 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.684 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:47.684 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.553743+0000", 2024-09-11T09:51:47.684 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:47.684 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.684 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.684 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:47.684 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.553745+0000", 2024-09-11T09:51:47.684 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.5120000000000001e-06 2024-09-11T09:51:47.685 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.685 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.685 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:47.685 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.554588+0000", 2024-09-11T09:51:47.685 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00084352299999999999 2024-09-11T09:51:47.685 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.685 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.685 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:47.685 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.554604+0000", 2024-09-11T09:51:47.685 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.643e-05 2024-09-11T09:51:47.685 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.685 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.685 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:47.686 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.554613+0000", 2024-09-11T09:51:47.686 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.5669999999999995e-06 2024-09-11T09:51:47.686 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.686 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.686 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:47.686 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.554662+0000", 2024-09-11T09:51:47.686 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.8510000000000001e-05 2024-09-11T09:51:47.686 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.686 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.686 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:47.686 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.554732+0000", 2024-09-11T09:51:47.686 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.0297999999999997e-05 2024-09-11T09:51:47.686 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.686 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.687 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 2", 2024-09-11T09:51:47.687 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.554870+0000", 2024-09-11T09:51:47.687 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00013856599999999999 2024-09-11T09:51:47.687 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.687 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.687 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:47.687 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.557875+0000", 2024-09-11T09:51:47.687 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0030042760000000002 2024-09-11T09:51:47.687 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.687 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.687 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:47.687 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.558835+0000", 2024-09-11T09:51:47.687 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00096057900000000003 2024-09-11T09:51:47.687 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.688 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.688 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:47.688 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.558874+0000", 2024-09-11T09:51:47.688 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.8992999999999997e-05 2024-09-11T09:51:47.688 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.688 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.688 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:47.688 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.558906+0000", 2024-09-11T09:51:47.688 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.2109e-05 2024-09-11T09:51:47.688 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:47.688 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:47.688 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:47.688 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.688 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.689 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.5002.0:93 2.3 2:c54742d8:::rbd_data.1250f5e25b80.0000000000000006:head [write 2228224~917504] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e61)", 2024-09-11T09:51:47.689 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:38.562067+0000", 2024-09-11T09:51:47.689 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 9.0176172710000007, 2024-09-11T09:51:47.689 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0050551099999999998, 2024-09-11T09:51:47.689 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:47.689 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:47.689 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:47.689 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:47.689 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.5002", 2024-09-11T09:51:47.689 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/2204440692", 2024-09-11T09:51:47.689 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 93 2024-09-11T09:51:47.689 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.689 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:47.689 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.690 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:47.690 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.562067+0000", 2024-09-11T09:51:47.690 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:47.690 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.690 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.690 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:47.690 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.562067+0000", 2024-09-11T09:51:47.690 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:47.690 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.690 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.690 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:47.690 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.562068+0000", 2024-09-11T09:51:47.690 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.0669999999999999e-06 2024-09-11T09:51:47.690 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.691 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.691 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:47.691 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.563170+0000", 2024-09-11T09:51:47.691 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00110256 2024-09-11T09:51:47.691 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.691 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.691 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:47.691 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.563185+0000", 2024-09-11T09:51:47.691 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.4873e-05 2024-09-11T09:51:47.691 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.691 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.691 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:47.691 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.563193+0000", 2024-09-11T09:51:47.691 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.1319999999999994e-06 2024-09-11T09:51:47.692 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.692 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.692 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:47.692 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.563238+0000", 2024-09-11T09:51:47.692 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.4773e-05 2024-09-11T09:51:47.692 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.692 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.692 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:47.692 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.563290+0000", 2024-09-11T09:51:47.692 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.1690000000000001e-05 2024-09-11T09:51:47.692 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.692 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.692 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 2", 2024-09-11T09:51:47.692 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.563365+0000", 2024-09-11T09:51:47.693 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.5023000000000003e-05 2024-09-11T09:51:47.693 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.693 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.693 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:47.693 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.566552+0000", 2024-09-11T09:51:47.693 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0031869960000000001 2024-09-11T09:51:47.693 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.693 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.693 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:47.693 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.567048+0000", 2024-09-11T09:51:47.693 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00049605099999999996 2024-09-11T09:51:47.693 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.693 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.693 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:47.694 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.567095+0000", 2024-09-11T09:51:47.694 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.6823999999999998e-05 2024-09-11T09:51:47.694 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:47.694 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:47.694 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:47.694 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.567122+0000", 2024-09-11T09:51:47.694 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.7121e-05 2024-09-11T09:51:47.694 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:47.694 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:47.694 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:47.694 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:47.694 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:47.694 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-11T09:51:47.695 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-11T09:51:47.696 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T09:51:47.696 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-11T09:51:47.697 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T09:51:47.697 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:51:47.696+0000 7f3c96ec9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31909) UID: 0 2024-09-11T09:51:47.697 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-11T09:51:47.698 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-11T09:51:47.698 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T09:51:47.698 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-11T09:51:47.698 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T09:51:47.699 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-11T09:51:47.699 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T09:51:47.699 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-11T09:51:47.700 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T09:51:47.785 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-11T09:51:47.820 INFO:tasks.workunit.client.0.smithi107.stderr:+ '[' rbd = rbd ']' 2024-09-11T09:51:47.820 INFO:tasks.workunit.client.0.smithi107.stderr:++ _sudo rbd device --device-type nbd map testrbdnbd27121 2024-09-11T09:51:47.821 INFO:tasks.workunit.client.0.smithi107.stderr:++ local cmd 2024-09-11T09:51:47.821 INFO:tasks.workunit.client.0.smithi107.stderr:+++ id -u 2024-09-11T09:51:47.821 INFO:tasks.workunit.client.0.smithi107.stderr:++ '[' 1000 -eq 0 ']' 2024-09-11T09:51:47.822 INFO:tasks.workunit.client.0.smithi107.stderr:+++ which rbd 2024-09-11T09:51:47.824 INFO:tasks.workunit.client.0.smithi107.stderr:++ cmd=/usr/bin/rbd 2024-09-11T09:51:47.824 INFO:tasks.workunit.client.0.smithi107.stderr:++ shift 2024-09-11T09:51:47.824 INFO:tasks.workunit.client.0.smithi107.stderr:++ sudo -nE /usr/bin/rbd device --device-type nbd map testrbdnbd27121 2024-09-11T09:51:47.830 INFO:tasks.ceph.osd.0.smithi017.stderr:2024-09-11T09:51:47.830+0000 7fb73c0f1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31665) UID: 0 2024-09-11T09:51:47.848 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:51:47.848 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:51:47.857 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-11T09:51:47.898 INFO:tasks.ceph.osd.0.smithi017.stderr:2024-09-11T09:51:47.898+0000 7fb73c0f1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31665) UID: 0 2024-09-11T09:51:47.931 INFO:tasks.workunit.client.0.smithi107.stderr:+ DEV=/dev/nbd0 2024-09-11T09:51:47.931 INFO:tasks.workunit.client.0.smithi107.stderr:+ get_pid rbd 2024-09-11T09:51:47.931 INFO:tasks.workunit.client.0.smithi107.stderr:+ local pool=rbd 2024-09-11T09:51:47.931 INFO:tasks.workunit.client.0.smithi107.stderr:+ local ns= 2024-09-11T09:51:47.931 INFO:tasks.workunit.client.0.smithi107.stderr:++ rbd device --device-type nbd --format xml list 2024-09-11T09:51:47.931 INFO:tasks.workunit.client.0.smithi107.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd27121'\''][device='\''/dev/nbd0'\'']/id' 2024-09-11T09:51:47.992 INFO:tasks.workunit.client.0.smithi107.stderr:+ PID=29073 2024-09-11T09:51:47.992 INFO:tasks.workunit.client.0.smithi107.stderr:+ test -n 29073 2024-09-11T09:51:47.992 INFO:tasks.workunit.client.0.smithi107.stderr:+ ps -p 29073 -C rbd-nbd 2024-09-11T09:51:47.998 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:51:47.998+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:51:48.003 INFO:tasks.workunit.client.0.smithi107.stdout: PID TTY TIME CMD 2024-09-11T09:51:48.003 INFO:tasks.workunit.client.0.smithi107.stdout: 29073 ? 00:00:00 rbd-nbd 2024-09-11T09:51:48.004 INFO:tasks.workunit.client.0.smithi107.stderr:+ unmap_device testrbdnbd27121 29073 2024-09-11T09:51:48.004 INFO:tasks.workunit.client.0.smithi107.stderr:+ local args=testrbdnbd27121 2024-09-11T09:51:48.004 INFO:tasks.workunit.client.0.smithi107.stderr:+ local pid=29073 2024-09-11T09:51:48.004 INFO:tasks.workunit.client.0.smithi107.stderr:+ _sudo rbd device --device-type nbd unmap testrbdnbd27121 2024-09-11T09:51:48.004 INFO:tasks.workunit.client.0.smithi107.stderr:+ local cmd 2024-09-11T09:51:48.004 INFO:tasks.workunit.client.0.smithi107.stderr:++ id -u 2024-09-11T09:51:48.005 INFO:tasks.workunit.client.0.smithi107.stderr:+ '[' 1000 -eq 0 ']' 2024-09-11T09:51:48.005 INFO:tasks.workunit.client.0.smithi107.stderr:++ which rbd 2024-09-11T09:51:48.006 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:51:48.006 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:51:48.007 INFO:tasks.workunit.client.0.smithi107.stderr:+ cmd=/usr/bin/rbd 2024-09-11T09:51:48.007 INFO:tasks.workunit.client.0.smithi107.stderr:+ shift 2024-09-11T09:51:48.007 INFO:tasks.workunit.client.0.smithi107.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd unmap testrbdnbd27121 2024-09-11T09:51:48.015 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-11T09:51:48.090 INFO:tasks.workunit.client.0.smithi107.stderr:+ rbd device --device-type nbd list 2024-09-11T09:51:48.090 INFO:tasks.workunit.client.0.smithi107.stderr:+ expect_false grep '^29073\b' 2024-09-11T09:51:48.090 INFO:tasks.workunit.client.0.smithi107.stderr:+ grep '^29073\b' 2024-09-11T09:51:48.099 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:51:48.099+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:51:48.137 INFO:tasks.workunit.client.0.smithi107.stderr:+ return 0 2024-09-11T09:51:48.137 INFO:tasks.workunit.client.0.smithi107.stderr:+ ps -C rbd-nbd 2024-09-11T09:51:48.137 INFO:tasks.workunit.client.0.smithi107.stderr:+ expect_false grep '^ *29073\b' 2024-09-11T09:51:48.137 INFO:tasks.workunit.client.0.smithi107.stderr:+ grep '^ *29073\b' 2024-09-11T09:51:48.149 INFO:tasks.workunit.client.0.smithi107.stderr:+ return 0 2024-09-11T09:51:48.149 INFO:tasks.workunit.client.0.smithi107.stderr:+ sleep 0.5 2024-09-11T09:51:48.163 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:51:48.163 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:51:48.172 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-11T09:51:48.200 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:51:48.201+0000 7f3c96ec9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31909) UID: 0 2024-09-11T09:51:48.301 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:51:48.302+0000 7f3c96ec9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31909) UID: 0 2024-09-11T09:51:48.320 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:51:48.320 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:51:48.329 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-11T09:51:48.401 INFO:tasks.ceph.osd.2.smithi017.stderr:2024-09-11T09:51:48.401+0000 7fd20b170640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31672) UID: 0 2024-09-11T09:51:48.435 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:51:48.436 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:51:48.444 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-11T09:51:48.502 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-09-11T09:51:48.503+0000 7f7b6c7d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31904) UID: 0 2024-09-11T09:51:48.592 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:51:48.592 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:51:48.601 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-11T09:51:48.603 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:51:48.603+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:51:48.650 INFO:tasks.workunit.client.0.smithi107.stderr:+ DEV= 2024-09-11T09:51:48.650 INFO:tasks.workunit.client.0.smithi107.stderr:+ rbd config global set global rbd_default_pool new_default_pool27121 2024-09-11T09:51:48.691 INFO:tasks.workunit.client.0.smithi107.stderr:+ rbd create --size 10M testrbdnbd27121 2024-09-11T09:51:48.700 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-11T09:51:48.704 INFO:tasks.ceph.osd.0.smithi017.stderr:2024-09-11T09:51:48.703+0000 7fb73c0f1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31665) UID: 0 2024-09-11T09:51:48.710 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:51:48.710 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:51:48.720 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-11T09:51:48.735 INFO:tasks.workunit.client.0.smithi107.stderr:++ _sudo rbd device --device-type nbd map testrbdnbd27121 2024-09-11T09:51:48.735 INFO:tasks.workunit.client.0.smithi107.stderr:++ local cmd 2024-09-11T09:51:48.735 INFO:tasks.workunit.client.0.smithi107.stderr:+++ id -u 2024-09-11T09:51:48.736 INFO:tasks.workunit.client.0.smithi107.stderr:++ '[' 1000 -eq 0 ']' 2024-09-11T09:51:48.736 INFO:tasks.workunit.client.0.smithi107.stderr:+++ which rbd 2024-09-11T09:51:48.738 INFO:tasks.workunit.client.0.smithi107.stderr:++ cmd=/usr/bin/rbd 2024-09-11T09:51:48.738 INFO:tasks.workunit.client.0.smithi107.stderr:++ shift 2024-09-11T09:51:48.738 INFO:tasks.workunit.client.0.smithi107.stderr:++ sudo -nE /usr/bin/rbd device --device-type nbd map testrbdnbd27121 2024-09-11T09:51:48.784 INFO:teuthology.orchestra.run.smithi017.stderr:nodeep-scrub is unset 2024-09-11T09:51:48.806 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:51:48.806 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:51:48.842 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:51:48.842+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:51:48.844 INFO:tasks.workunit.client.0.smithi107.stderr:+ DEV=/dev/nbd0 2024-09-11T09:51:48.844 INFO:tasks.workunit.client.0.smithi107.stderr:+ get_pid new_default_pool27121 2024-09-11T09:51:48.844 INFO:tasks.workunit.client.0.smithi107.stderr:+ local pool=new_default_pool27121 2024-09-11T09:51:48.844 INFO:tasks.workunit.client.0.smithi107.stderr:+ local ns= 2024-09-11T09:51:48.845 INFO:tasks.workunit.client.0.smithi107.stderr:++ rbd device --device-type nbd --format xml list 2024-09-11T09:51:48.845 INFO:tasks.workunit.client.0.smithi107.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''new_default_pool27121'\''][namespace='\'''\''][image='\''testrbdnbd27121'\''][device='\''/dev/nbd0'\'']/id' 2024-09-11T09:51:48.874 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:51:48.875 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:51:48.884 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-11T09:51:48.904 INFO:tasks.workunit.client.0.smithi107.stderr:+ PID=29203 2024-09-11T09:51:48.904 INFO:tasks.workunit.client.0.smithi107.stderr:+ test -n 29203 2024-09-11T09:51:48.904 INFO:tasks.workunit.client.0.smithi107.stderr:+ ps -p 29203 -C rbd-nbd 2024-09-11T09:51:48.905 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:51:48.906+0000 7f3c96ec9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31909) UID: 0 2024-09-11T09:51:48.915 INFO:tasks.workunit.client.0.smithi107.stdout: PID TTY TIME CMD 2024-09-11T09:51:48.915 INFO:tasks.workunit.client.0.smithi107.stdout: 29203 ? 00:00:00 rbd-nbd 2024-09-11T09:51:48.916 INFO:tasks.workunit.client.0.smithi107.stderr:+ unmap_device testrbdnbd27121 29203 2024-09-11T09:51:48.916 INFO:tasks.workunit.client.0.smithi107.stderr:+ local args=testrbdnbd27121 2024-09-11T09:51:48.916 INFO:tasks.workunit.client.0.smithi107.stderr:+ local pid=29203 2024-09-11T09:51:48.916 INFO:tasks.workunit.client.0.smithi107.stderr:+ _sudo rbd device --device-type nbd unmap testrbdnbd27121 2024-09-11T09:51:48.916 INFO:tasks.workunit.client.0.smithi107.stderr:+ local cmd 2024-09-11T09:51:48.916 INFO:tasks.workunit.client.0.smithi107.stderr:++ id -u 2024-09-11T09:51:48.917 INFO:tasks.workunit.client.0.smithi107.stderr:+ '[' 1000 -eq 0 ']' 2024-09-11T09:51:48.917 INFO:tasks.workunit.client.0.smithi107.stderr:++ which rbd 2024-09-11T09:51:48.918 INFO:tasks.workunit.client.0.smithi107.stderr:+ cmd=/usr/bin/rbd 2024-09-11T09:51:48.918 INFO:tasks.workunit.client.0.smithi107.stderr:+ shift 2024-09-11T09:51:48.918 INFO:tasks.workunit.client.0.smithi107.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd unmap testrbdnbd27121 2024-09-11T09:51:49.002 INFO:tasks.workunit.client.0.smithi107.stderr:+ rbd device --device-type nbd list 2024-09-11T09:51:49.002 INFO:tasks.workunit.client.0.smithi107.stderr:+ expect_false grep '^29203\b' 2024-09-11T09:51:49.002 INFO:tasks.workunit.client.0.smithi107.stderr:+ grep '^29203\b' 2024-09-11T09:51:49.006 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:51:49.006+0000 7f3c96ec9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31909) UID: 0 2024-09-11T09:51:49.032 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:51:49.032 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:51:49.041 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-11T09:51:49.048 INFO:tasks.workunit.client.0.smithi107.stderr:+ return 0 2024-09-11T09:51:49.049 INFO:tasks.workunit.client.0.smithi107.stderr:+ ps -C rbd-nbd 2024-09-11T09:51:49.049 INFO:tasks.workunit.client.0.smithi107.stderr:+ expect_false grep '^ *29203\b' 2024-09-11T09:51:49.049 INFO:tasks.workunit.client.0.smithi107.stderr:+ grep '^ *29203\b' 2024-09-11T09:51:49.060 INFO:tasks.workunit.client.0.smithi107.stderr:+ return 0 2024-09-11T09:51:49.060 INFO:tasks.workunit.client.0.smithi107.stderr:+ sleep 0.5 2024-09-11T09:51:49.107 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-09-11T09:51:49.107+0000 7f7b6c7d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31904) UID: 0 2024-09-11T09:51:49.144 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:51:49.144 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:51:49.152 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-11T09:51:49.237 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-09-11T09:51:49.238+0000 7f7b6c7d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31904) UID: 0 2024-09-11T09:51:49.295 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:51:49.295 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:51:49.304 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-11T09:51:49.308 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:51:49.309+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:51:49.410 INFO:tasks.ceph.osd.2.smithi017.stderr:2024-09-11T09:51:49.410+0000 7fd20b170640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31672) UID: 0 2024-09-11T09:51:49.447 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:51:49.447 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:51:49.456 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-11T09:51:49.511 INFO:tasks.ceph.osd.0.smithi017.stderr:2024-09-11T09:51:49.511+0000 7fb73c0f1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31665) UID: 0 2024-09-11T09:51:49.559 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:51:49.559 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:51:49.561 INFO:tasks.workunit.client.0.smithi107.stderr:+ DEV= 2024-09-11T09:51:49.561 INFO:tasks.workunit.client.0.smithi107.stderr:+ rbd config global rm global rbd_default_pool 2024-09-11T09:51:49.568 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-11T09:51:49.600 INFO:tasks.workunit.client.0.smithi107.stderr:++ _sudo rbd device --device-type nbd map rbd/testrbdnbd27121 2024-09-11T09:51:49.600 INFO:tasks.workunit.client.0.smithi107.stderr:++ local cmd 2024-09-11T09:51:49.600 INFO:tasks.workunit.client.0.smithi107.stderr:+++ id -u 2024-09-11T09:51:49.601 INFO:tasks.workunit.client.0.smithi107.stderr:++ '[' 1000 -eq 0 ']' 2024-09-11T09:51:49.604 INFO:tasks.workunit.client.0.smithi107.stderr:+++ which rbd 2024-09-11T09:51:49.604 INFO:tasks.workunit.client.0.smithi107.stderr:++ cmd=/usr/bin/rbd 2024-09-11T09:51:49.605 INFO:tasks.workunit.client.0.smithi107.stderr:++ shift 2024-09-11T09:51:49.605 INFO:tasks.workunit.client.0.smithi107.stderr:++ sudo -nE /usr/bin/rbd device --device-type nbd map rbd/testrbdnbd27121 2024-09-11T09:51:49.612 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:51:49.613+0000 7f3c96ec9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31909) UID: 0 2024-09-11T09:51:49.671 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:51:49.671 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:51:49.680 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-11T09:51:49.701 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T09:51:49.712 INFO:tasks.workunit.client.0.smithi107.stderr:+ DEV=/dev/nbd0 2024-09-11T09:51:49.712 INFO:tasks.workunit.client.0.smithi107.stderr:+ get_pid rbd 2024-09-11T09:51:49.713 INFO:tasks.workunit.client.0.smithi107.stderr:+ local pool=rbd 2024-09-11T09:51:49.713 INFO:tasks.workunit.client.0.smithi107.stderr:+ local ns= 2024-09-11T09:51:49.713 INFO:tasks.workunit.client.0.smithi107.stderr:++ rbd device --device-type nbd --format xml list 2024-09-11T09:51:49.713 INFO:tasks.workunit.client.0.smithi107.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd27121'\''][device='\''/dev/nbd0'\'']/id' 2024-09-11T09:51:49.745 INFO:tasks.ceph.osd.0.smithi017.stderr:2024-09-11T09:51:49.745+0000 7fb73c0f1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31665) UID: 0 2024-09-11T09:51:49.774 INFO:tasks.workunit.client.0.smithi107.stderr:+ PID=29301 2024-09-11T09:51:49.775 INFO:tasks.workunit.client.0.smithi107.stderr:+ test -n 29301 2024-09-11T09:51:49.775 INFO:tasks.workunit.client.0.smithi107.stderr:+ ps -p 29301 -C rbd-nbd 2024-09-11T09:51:49.786 INFO:tasks.workunit.client.0.smithi107.stdout: PID TTY TIME CMD 2024-09-11T09:51:49.786 INFO:tasks.workunit.client.0.smithi107.stdout: 29301 ? 00:00:00 rbd-nbd 2024-09-11T09:51:49.787 INFO:tasks.workunit.client.0.smithi107.stderr:+ _sudo kill 29301 2024-09-11T09:51:49.787 INFO:tasks.workunit.client.0.smithi107.stderr:+ local cmd 2024-09-11T09:51:49.787 INFO:tasks.workunit.client.0.smithi107.stderr:++ id -u 2024-09-11T09:51:49.787 INFO:tasks.workunit.client.0.smithi107.stderr:+ '[' 1000 -eq 0 ']' 2024-09-11T09:51:49.788 INFO:tasks.workunit.client.0.smithi107.stderr:++ which kill 2024-09-11T09:51:49.789 INFO:tasks.workunit.client.0.smithi107.stderr:+ cmd=/usr/bin/kill 2024-09-11T09:51:49.789 INFO:tasks.workunit.client.0.smithi107.stderr:+ shift 2024-09-11T09:51:49.789 INFO:tasks.workunit.client.0.smithi107.stderr:+ sudo -nE /usr/bin/kill 29301 2024-09-11T09:51:49.810 INFO:tasks.workunit.client.0.smithi107.stderr:++ seq 10 2024-09-11T09:51:49.813 INFO:tasks.workunit.client.0.smithi107.stderr:+ for i in `seq 10` 2024-09-11T09:51:49.813 INFO:tasks.workunit.client.0.smithi107.stderr:+ rbd device --device-type nbd list 2024-09-11T09:51:49.814 INFO:tasks.workunit.client.0.smithi107.stderr:+ expect_false grep '^29301 *rbd *testrbdnbd27121' 2024-09-11T09:51:49.814 INFO:tasks.workunit.client.0.smithi107.stderr:+ grep '^29301 *rbd *testrbdnbd27121' 2024-09-11T09:51:49.814 INFO:tasks.ceph.osd.0.smithi017.stderr:2024-09-11T09:51:49.813+0000 7fb73c0f1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31665) UID: 0 2024-09-11T09:51:49.825 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:51:49.825 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:51:49.833 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-11T09:51:49.833 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T09:51:49.834 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-11T09:51:49.850 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-11T09:51:49.850 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T09:51:49.866 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-11T09:51:49.867 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T09:51:49.873 INFO:tasks.workunit.client.0.smithi107.stderr:+ return 0 2024-09-11T09:51:49.873 INFO:tasks.workunit.client.0.smithi107.stderr:+ break 2024-09-11T09:51:49.873 INFO:tasks.workunit.client.0.smithi107.stderr:+ rbd device --device-type nbd list 2024-09-11T09:51:49.873 INFO:tasks.workunit.client.0.smithi107.stderr:+ expect_false grep '^29301 *rbd *testrbdnbd27121' 2024-09-11T09:51:49.873 INFO:tasks.workunit.client.0.smithi107.stderr:+ grep '^29301 *rbd *testrbdnbd27121' 2024-09-11T09:51:49.883 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-11T09:51:49.883 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T09:51:49.899 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-11T09:51:49.899 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T09:51:49.914 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:51:49.914+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:51:49.916 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-11T09:51:49.916 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T09:51:49.933 INFO:tasks.workunit.client.0.smithi107.stderr:+ return 0 2024-09-11T09:51:49.934 INFO:tasks.workunit.client.0.smithi107.stderr:+ QUIESCE_HOOK=/tmp/tmp.rvrcDirNue/quiesce.sh 2024-09-11T09:51:49.934 INFO:tasks.workunit.client.0.smithi107.stderr:++ _sudo rbd device --device-type nbd map --quiesce --quiesce-hook /tmp/tmp.rvrcDirNue/quiesce.sh rbd/testrbdnbd27121 2024-09-11T09:51:49.934 INFO:tasks.workunit.client.0.smithi107.stderr:++ local cmd 2024-09-11T09:51:49.934 INFO:tasks.workunit.client.0.smithi107.stderr:+++ id -u 2024-09-11T09:51:49.935 INFO:tasks.workunit.client.0.smithi107.stderr:++ '[' 1000 -eq 0 ']' 2024-09-11T09:51:49.935 INFO:tasks.workunit.client.0.smithi107.stderr:+++ which rbd 2024-09-11T09:51:49.937 INFO:tasks.workunit.client.0.smithi107.stderr:++ cmd=/usr/bin/rbd 2024-09-11T09:51:49.937 INFO:tasks.workunit.client.0.smithi107.stderr:++ shift 2024-09-11T09:51:49.937 INFO:tasks.workunit.client.0.smithi107.stderr:++ sudo -nE /usr/bin/rbd device --device-type nbd map --quiesce --quiesce-hook /tmp/tmp.rvrcDirNue/quiesce.sh rbd/testrbdnbd27121 2024-09-11T09:51:49.938 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-11T09:51:49.938 INFO:teuthology.orchestra.run.smithi096.stdout: "ops": [], 2024-09-11T09:51:49.938 INFO:teuthology.orchestra.run.smithi096.stdout: "num_ops": 0 2024-09-11T09:51:49.938 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-11T09:51:49.947 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-11T09:51:50.016 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:51:50.017+0000 7f3a7b90a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 42475) UID: 0 2024-09-11T09:51:50.051 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-11T09:51:50.051 INFO:teuthology.orchestra.run.smithi096.stdout: "ops": [], 2024-09-11T09:51:50.052 INFO:teuthology.orchestra.run.smithi096.stdout: "complaint_time": 30, 2024-09-11T09:51:50.052 INFO:teuthology.orchestra.run.smithi096.stdout: "num_blocked_ops": 0 2024-09-11T09:51:50.052 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-11T09:51:50.054 INFO:tasks.workunit.client.0.smithi107.stderr:+ DEV=/dev/nbd1 2024-09-11T09:51:50.054 INFO:tasks.workunit.client.0.smithi107.stderr:+ get_pid rbd 2024-09-11T09:51:50.054 INFO:tasks.workunit.client.0.smithi107.stderr:+ local pool=rbd 2024-09-11T09:51:50.054 INFO:tasks.workunit.client.0.smithi107.stderr:+ local ns= 2024-09-11T09:51:50.054 INFO:tasks.workunit.client.0.smithi107.stderr:++ rbd device --device-type nbd --format xml list 2024-09-11T09:51:50.054 INFO:tasks.workunit.client.0.smithi107.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd27121'\''][device='\''/dev/nbd1'\'']/id' 2024-09-11T09:51:50.060 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-11T09:51:50.143 INFO:tasks.workunit.client.0.smithi107.stderr:+ PID=29367 2024-09-11T09:51:50.143 INFO:tasks.workunit.client.0.smithi107.stderr:+ test -n 29367 2024-09-11T09:51:50.143 INFO:tasks.workunit.client.0.smithi107.stderr:+ ps -p 29367 -C rbd-nbd 2024-09-11T09:51:50.145 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:51:50.146+0000 7f3a7b90a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 42475) UID: 0 2024-09-11T09:51:50.154 INFO:tasks.workunit.client.0.smithi107.stdout: PID TTY TIME CMD 2024-09-11T09:51:50.154 INFO:tasks.workunit.client.0.smithi107.stdout: 29367 ? 00:00:00 rbd-nbd 2024-09-11T09:51:50.155 INFO:tasks.workunit.client.0.smithi107.stderr:+ test '!' -e /tmp/tmp.rvrcDirNue/quiesce.sh 2024-09-11T09:51:50.155 INFO:tasks.workunit.client.0.smithi107.stderr:+ expect_false rbd snap create rbd/testrbdnbd27121@quiesce1 2024-09-11T09:51:50.156 INFO:tasks.workunit.client.0.smithi107.stderr:+ rbd snap create rbd/testrbdnbd27121@quiesce1 2024-09-11T09:51:50.201 INFO:tasks.workunit.client.0.smithi107.stderr: Creating snap: 10% complete...2024-09-11T09:51:50.202+0000 7f252a615640 -1 librbd::ImageWatcher: 0x7f25080074e0 failed to notify quiesce: (1) Operation not permitted 2024-09-11T09:51:50.202 INFO:tasks.workunit.client.0.smithi107.stderr:2024-09-11T09:51:50.202+0000 7f252a615640 -1 librbd::SnapshotCreateRequest: failed to notify quiesce: (1) Operation not permitted 2024-09-11T09:51:50.202 INFO:tasks.workunit.client.0.smithi107.stderr: Creating snap: 10% complete...failed. 2024-09-11T09:51:50.202 INFO:tasks.workunit.client.0.smithi107.stderr:rbd: failed to create snapshot: (1) Operation not permitted 2024-09-11T09:51:50.205 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-11T09:51:50.205 INFO:teuthology.orchestra.run.smithi096.stdout: "size": 20, 2024-09-11T09:51:50.205 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 600, 2024-09-11T09:51:50.206 INFO:teuthology.orchestra.run.smithi096.stdout: "ops": [ 2024-09-11T09:51:50.206 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:50.206 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "MOSDPGPush(3.3 67/66 [PushOp(3:c4c92e5a:::rbd_trash:head, version: 56'1, data_included: [], data_size: 0, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:c4c92e5a:::rbd_trash:head@56'1, size: 0, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 0, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-11T09:51:50.206 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:45.784642+0000", 2024-09-11T09:51:50.206 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 4.4224739770000001, 2024-09-11T09:51:50.206 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00049292200000000002, 2024-09-11T09:51:50.206 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:50.206 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:50.206 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "started", 2024-09-11T09:51:50.206 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:50.206 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:50.206 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:50.206 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:45.784642+0000", 2024-09-11T09:51:50.207 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:50.207 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:50.207 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:50.207 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:50.207 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:45.784642+0000", 2024-09-11T09:51:50.207 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:50.207 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:50.207 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:50.207 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:50.207 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:45.784643+0000", 2024-09-11T09:51:50.207 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.2599999999999998e-07 2024-09-11T09:51:50.207 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:50.207 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:50.207 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:50.208 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:45.784652+0000", 2024-09-11T09:51:50.208 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.064e-06 2024-09-11T09:51:50.208 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:50.208 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:50.208 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:50.208 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:45.784664+0000", 2024-09-11T09:51:50.208 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1831000000000001e-05 2024-09-11T09:51:50.208 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:50.208 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:50.208 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:50.208 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:45.784676+0000", 2024-09-11T09:51:50.208 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1996e-05 2024-09-11T09:51:50.208 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:50.209 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:50.209 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:50.209 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:45.784744+0000", 2024-09-11T09:51:50.209 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.8310000000000002e-05 2024-09-11T09:51:50.209 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:50.209 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:50.209 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:50.209 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:45.784758+0000", 2024-09-11T09:51:50.209 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.3998999999999999e-05 2024-09-11T09:51:50.209 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:50.209 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:50.209 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:50.209 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:45.785135+0000", 2024-09-11T09:51:50.209 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000377296 2024-09-11T09:51:50.210 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:50.210 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:50.210 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:50.210 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:50.210 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:50.210 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-11T09:51:50.211 INFO:tasks.workunit.client.0.smithi107.stderr:+ return 0 2024-09-11T09:51:50.211 INFO:tasks.workunit.client.0.smithi107.stderr:+ _sudo dd if=/tmp/tmp.rvrcDirNue/data of=/dev/nbd1 bs=1M count=1 oflag=direct 2024-09-11T09:51:50.211 INFO:tasks.workunit.client.0.smithi107.stderr:+ local cmd 2024-09-11T09:51:50.211 INFO:tasks.workunit.client.0.smithi107.stderr:++ id -u 2024-09-11T09:51:50.211 INFO:tasks.workunit.client.0.smithi107.stderr:+ '[' 1000 -eq 0 ']' 2024-09-11T09:51:50.211 INFO:tasks.workunit.client.0.smithi107.stderr:++ which dd 2024-09-11T09:51:50.212 INFO:tasks.workunit.client.0.smithi107.stderr:+ cmd=/usr/bin/dd 2024-09-11T09:51:50.212 INFO:tasks.workunit.client.0.smithi107.stderr:+ shift 2024-09-11T09:51:50.212 INFO:tasks.workunit.client.0.smithi107.stderr:+ sudo -nE /usr/bin/dd if=/tmp/tmp.rvrcDirNue/data of=/dev/nbd1 bs=1M count=1 oflag=direct 2024-09-11T09:51:50.214 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-11T09:51:50.217 INFO:tasks.ceph.osd.0.smithi017.stderr:2024-09-11T09:51:50.217+0000 7fb73c0f1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31665) UID: 0 2024-09-11T09:51:50.238 INFO:tasks.workunit.client.0.smithi107.stderr:1+0 records in 2024-09-11T09:51:50.238 INFO:tasks.workunit.client.0.smithi107.stderr:1+0 records out 2024-09-11T09:51:50.238 INFO:tasks.workunit.client.0.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0167802 s, 62.5 MB/s 2024-09-11T09:51:50.239 INFO:tasks.workunit.client.0.smithi107.stderr:+ touch /tmp/tmp.rvrcDirNue/quiesce.sh 2024-09-11T09:51:50.240 INFO:tasks.workunit.client.0.smithi107.stderr:+ chmod +x /tmp/tmp.rvrcDirNue/quiesce.sh 2024-09-11T09:51:50.241 INFO:tasks.workunit.client.0.smithi107.stderr:+ cat 2024-09-11T09:51:50.242 INFO:tasks.workunit.client.0.smithi107.stderr:+ rbd snap create rbd/testrbdnbd27121@quiesce1 2024-09-11T09:51:50.318 INFO:tasks.ceph.osd.0.smithi017.stderr:2024-09-11T09:51:50.318+0000 7fb73c0f1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31665) UID: 0 2024-09-11T09:51:50.329 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:51:50.329 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [], 2024-09-11T09:51:50.329 INFO:teuthology.orchestra.run.smithi017.stdout: "num_ops": 0 2024-09-11T09:51:50.329 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:51:50.338 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-11T09:51:50.419 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:51:50.420+0000 7f3c96ec9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31909) UID: 0 2024-09-11T09:51:50.485 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:51:50.485 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [], 2024-09-11T09:51:50.485 INFO:teuthology.orchestra.run.smithi017.stdout: "complaint_time": 30, 2024-09-11T09:51:50.486 INFO:teuthology.orchestra.run.smithi017.stdout: "num_blocked_ops": 0 2024-09-11T09:51:50.486 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:51:50.494 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-11T09:51:50.538 INFO:tasks.ceph.osd.0.smithi017.stderr:2024-09-11T09:51:50.538+0000 7fb73c0f1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31665) UID: 0 2024-09-11T09:51:50.602 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:51:50.603 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 20, 2024-09-11T09:51:50.603 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 600, 2024-09-11T09:51:50.603 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [ 2024-09-11T09:51:50.603 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.603 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4688.0:34 2.4 e53/32 2:2367385a:::rbd_data.1250f5e25b80.0000000000000002:head v 53'6, mlcod=53'4)", 2024-09-11T09:51:50.603 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.907415+0000", 2024-09-11T09:51:50.603 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 27.695190234999998, 2024-09-11T09:51:50.603 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.011285778999999999, 2024-09-11T09:51:50.603 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:50.603 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:50.603 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:50.604 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:50.604 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.604 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:50.604 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.907415+0000", 2024-09-11T09:51:50.604 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:50.604 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.604 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.604 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:50.604 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.907415+0000", 2024-09-11T09:51:50.604 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:50.604 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.604 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.604 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:50.605 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.907417+0000", 2024-09-11T09:51:50.605 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.2910000000000002e-06 2024-09-11T09:51:50.605 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.605 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.605 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:50.605 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.910477+0000", 2024-09-11T09:51:50.605 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0030597829999999999 2024-09-11T09:51:50.605 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.605 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.605 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:50.605 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.910501+0000", 2024-09-11T09:51:50.605 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.3332e-05 2024-09-11T09:51:50.606 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.606 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.606 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:50.606 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.910514+0000", 2024-09-11T09:51:50.606 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3957e-05 2024-09-11T09:51:50.606 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.606 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.606 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:50.606 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.910612+0000", 2024-09-11T09:51:50.606 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.7075000000000001e-05 2024-09-11T09:51:50.606 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.606 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.606 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:50.607 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.910640+0000", 2024-09-11T09:51:50.607 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.8768999999999998e-05 2024-09-11T09:51:50.607 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.607 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.607 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:50.607 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.918640+0000", 2024-09-11T09:51:50.607 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0079991899999999998 2024-09-11T09:51:50.607 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.607 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.607 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:50.607 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.918701+0000", 2024-09-11T09:51:50.607 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.1382000000000003e-05 2024-09-11T09:51:50.607 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:50.607 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:50.608 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:50.608 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.608 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.608 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4688.0:38 2.5 e53/32 2:abde2694:::rbd_data.1250f5e25b80.0000000000000004:head v 53'1, mlcod=0'0)", 2024-09-11T09:51:50.608 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.915432+0000", 2024-09-11T09:51:50.608 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 27.687173855000001, 2024-09-11T09:51:50.608 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.015849652999999998, 2024-09-11T09:51:50.608 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:50.608 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:50.608 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:50.608 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:50.608 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.608 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:50.608 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.915432+0000", 2024-09-11T09:51:50.609 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:50.609 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.609 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.609 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:50.609 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.915432+0000", 2024-09-11T09:51:50.609 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:50.609 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.609 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.609 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:50.609 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.915432+0000", 2024-09-11T09:51:50.609 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.91e-07 2024-09-11T09:51:50.609 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.609 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.609 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:50.610 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.924943+0000", 2024-09-11T09:51:50.610 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0095110630000000002 2024-09-11T09:51:50.610 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.610 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.610 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:50.610 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.924960+0000", 2024-09-11T09:51:50.610 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6915999999999999e-05 2024-09-11T09:51:50.610 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.610 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.610 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:50.610 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.924968+0000", 2024-09-11T09:51:50.610 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.6999999999999997e-06 2024-09-11T09:51:50.610 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.610 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.611 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:50.611 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.925025+0000", 2024-09-11T09:51:50.611 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.6949e-05 2024-09-11T09:51:50.611 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.611 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.611 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:50.611 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.925043+0000", 2024-09-11T09:51:50.611 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7784000000000001e-05 2024-09-11T09:51:50.611 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.611 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.611 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:50.611 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.931237+0000", 2024-09-11T09:51:50.611 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0061935389999999996 2024-09-11T09:51:50.611 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.611 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.612 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:50.612 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.931281+0000", 2024-09-11T09:51:50.612 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.4511000000000001e-05 2024-09-11T09:51:50.612 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:50.612 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:50.612 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:50.612 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.612 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.612 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:40 2.7 2:fe41411c:::rbd_object_map.1250f5e25b80:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:50.612 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.919652+0000", 2024-09-11T09:51:50.612 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 27.682953187999999, 2024-09-11T09:51:50.612 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.012187073, 2024-09-11T09:51:50.612 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:50.613 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:50.613 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:50.613 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:50.613 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:51:50.613 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:51:50.613 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 40 2024-09-11T09:51:50.613 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.613 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:50.613 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.613 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:50.613 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919652+0000", 2024-09-11T09:51:50.613 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:50.613 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.613 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.614 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:50.614 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919652+0000", 2024-09-11T09:51:50.614 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:50.614 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.614 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.614 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:50.614 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919654+0000", 2024-09-11T09:51:50.614 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.88e-06 2024-09-11T09:51:50.614 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.614 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.614 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:50.614 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919669+0000", 2024-09-11T09:51:50.615 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.5271e-05 2024-09-11T09:51:50.615 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.615 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.615 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:50.615 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919687+0000", 2024-09-11T09:51:50.615 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7448999999999998e-05 2024-09-11T09:51:50.615 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.615 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.615 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:50.615 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919695+0000", 2024-09-11T09:51:50.615 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.1149999999999994e-06 2024-09-11T09:51:50.615 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.616 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.616 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:50.616 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919757+0000", 2024-09-11T09:51:50.616 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.2348999999999995e-05 2024-09-11T09:51:50.616 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.616 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.616 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:50.616 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919875+0000", 2024-09-11T09:51:50.616 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00011750499999999999 2024-09-11T09:51:50.616 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.616 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.616 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 5", 2024-09-11T09:51:50.616 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.920201+0000", 2024-09-11T09:51:50.617 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00032628099999999999 2024-09-11T09:51:50.617 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.617 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.617 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:50.617 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.922089+0000", 2024-09-11T09:51:50.617 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.001887978 2024-09-11T09:51:50.617 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.617 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.617 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:50.617 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.931752+0000", 2024-09-11T09:51:50.617 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0096626309999999997 2024-09-11T09:51:50.617 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.617 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.618 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:50.618 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.931794+0000", 2024-09-11T09:51:50.618 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.2849999999999998e-05 2024-09-11T09:51:50.618 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.618 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.618 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:50.618 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.931839+0000", 2024-09-11T09:51:50.618 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.4764e-05 2024-09-11T09:51:50.618 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:50.618 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:50.618 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:50.618 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.618 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.618 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:41 2.7 2:fe41411c:::rbd_object_map.1250f5e25b80:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:50.619 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.919716+0000", 2024-09-11T09:51:50.619 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 27.682889343999999, 2024-09-11T09:51:50.619 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.015641605999999999, 2024-09-11T09:51:50.619 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:50.619 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:50.619 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:50.619 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:50.619 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:51:50.619 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:51:50.619 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 41 2024-09-11T09:51:50.619 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.619 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:50.619 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.619 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:50.620 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919716+0000", 2024-09-11T09:51:50.620 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:50.620 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.620 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.620 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:50.620 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919716+0000", 2024-09-11T09:51:50.620 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:50.620 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.620 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.620 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:50.620 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919716+0000", 2024-09-11T09:51:50.620 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.9999999999999998e-07 2024-09-11T09:51:50.620 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.621 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.621 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:50.621 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919720+0000", 2024-09-11T09:51:50.621 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.6440000000000002e-06 2024-09-11T09:51:50.621 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.621 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.621 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:50.621 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919729+0000", 2024-09-11T09:51:50.621 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.1759999999999999e-06 2024-09-11T09:51:50.621 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.621 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.621 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:50.621 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919736+0000", 2024-09-11T09:51:50.621 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.5390000000000004e-06 2024-09-11T09:51:50.621 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.622 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.622 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:50.622 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.920669+0000", 2024-09-11T09:51:50.622 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00093275599999999997 2024-09-11T09:51:50.622 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.622 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.622 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:50.622 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.920797+0000", 2024-09-11T09:51:50.622 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00012893400000000001 2024-09-11T09:51:50.622 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.622 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.622 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 5", 2024-09-11T09:51:50.622 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.921269+0000", 2024-09-11T09:51:50.623 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00047128600000000001 2024-09-11T09:51:50.623 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.623 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.623 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:50.623 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.923198+0000", 2024-09-11T09:51:50.623 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0019289369999999999 2024-09-11T09:51:50.623 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.623 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.623 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:50.623 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.935300+0000", 2024-09-11T09:51:50.623 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.012102392 2024-09-11T09:51:50.623 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.623 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.623 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:50.624 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.935332+0000", 2024-09-11T09:51:50.624 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.2292000000000002e-05 2024-09-11T09:51:50.624 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.624 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.624 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:50.624 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.935358+0000", 2024-09-11T09:51:50.624 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.525e-05 2024-09-11T09:51:50.624 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:50.624 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:50.624 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:50.624 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.624 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.624 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:48 2.7 2:fe41411c:::rbd_object_map.1250f5e25b80:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:50.624 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.938799+0000", 2024-09-11T09:51:50.625 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 27.663806234999999, 2024-09-11T09:51:50.625 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0041642340000000002, 2024-09-11T09:51:50.625 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:50.625 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:50.625 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:50.625 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:50.625 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:51:50.625 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:51:50.625 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 48 2024-09-11T09:51:50.625 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.625 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:50.625 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.626 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:50.626 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.938799+0000", 2024-09-11T09:51:50.626 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:50.626 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.626 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.626 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:50.626 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.938799+0000", 2024-09-11T09:51:50.626 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:50.626 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.626 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.626 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:50.626 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.938800+0000", 2024-09-11T09:51:50.626 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.71e-07 2024-09-11T09:51:50.626 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.627 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.627 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:50.627 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.938805+0000", 2024-09-11T09:51:50.627 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.3140000000000003e-06 2024-09-11T09:51:50.627 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.627 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.627 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:50.627 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.938816+0000", 2024-09-11T09:51:50.627 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0231000000000001e-05 2024-09-11T09:51:50.627 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.627 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.627 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:50.627 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.938822+0000", 2024-09-11T09:51:50.628 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.9550000000000003e-06 2024-09-11T09:51:50.628 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.628 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.628 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:50.628 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.938868+0000", 2024-09-11T09:51:50.628 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.5562999999999997e-05 2024-09-11T09:51:50.628 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.628 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.628 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:50.628 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.938918+0000", 2024-09-11T09:51:50.628 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.0317e-05 2024-09-11T09:51:50.628 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.629 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.629 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 5", 2024-09-11T09:51:50.629 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.939114+0000", 2024-09-11T09:51:50.629 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00019538300000000001 2024-09-11T09:51:50.629 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.629 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.629 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:50.629 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.939663+0000", 2024-09-11T09:51:50.629 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00054932199999999998 2024-09-11T09:51:50.629 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.629 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.629 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:50.629 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.942910+0000", 2024-09-11T09:51:50.630 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0032473839999999999 2024-09-11T09:51:50.630 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.630 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.630 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:50.630 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.942940+0000", 2024-09-11T09:51:50.630 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.9306999999999999e-05 2024-09-11T09:51:50.630 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.630 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.630 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:50.630 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.942963+0000", 2024-09-11T09:51:50.630 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.3587e-05 2024-09-11T09:51:50.630 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:50.630 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:50.630 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:50.631 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.631 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.631 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4688.0:49 2.5 e53/32 2:b6dd9fc9:::rbd_data.1250f5e25b80.0000000000000009:head v 53'2, mlcod=0'0)", 2024-09-11T09:51:50.631 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.944712+0000", 2024-09-11T09:51:50.631 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 27.657893395999999, 2024-09-11T09:51:50.631 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.016185069999999999, 2024-09-11T09:51:50.631 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:50.631 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:50.631 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:50.631 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:50.631 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.632 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:50.632 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.944712+0000", 2024-09-11T09:51:50.632 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:50.632 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.632 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.632 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:50.632 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.944712+0000", 2024-09-11T09:51:50.632 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:50.632 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.632 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.632 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:50.632 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.944713+0000", 2024-09-11T09:51:50.632 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0139999999999999e-06 2024-09-11T09:51:50.633 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.633 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.633 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:50.633 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.955518+0000", 2024-09-11T09:51:50.633 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.01080542 2024-09-11T09:51:50.633 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.633 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.633 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:50.633 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.955536+0000", 2024-09-11T09:51:50.633 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7818000000000001e-05 2024-09-11T09:51:50.633 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.633 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.633 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:50.634 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.955544+0000", 2024-09-11T09:51:50.634 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.1699999999999997e-06 2024-09-11T09:51:50.634 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.634 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.634 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:50.634 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.955607+0000", 2024-09-11T09:51:50.634 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.2451000000000002e-05 2024-09-11T09:51:50.634 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.634 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.634 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:50.634 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.955632+0000", 2024-09-11T09:51:50.634 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.5335000000000001e-05 2024-09-11T09:51:50.634 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.634 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.635 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:50.635 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.960826+0000", 2024-09-11T09:51:50.635 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0051938560000000002 2024-09-11T09:51:50.635 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.635 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.635 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:50.635 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.960897+0000", 2024-09-11T09:51:50.635 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.1006000000000004e-05 2024-09-11T09:51:50.635 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:50.635 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:50.635 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:50.635 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.635 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.635 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:53 2.7 2:fe41411c:::rbd_object_map.1250f5e25b80:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:50.636 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.951291+0000", 2024-09-11T09:51:50.636 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 27.651314406000001, 2024-09-11T09:51:50.636 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0044964999999999996, 2024-09-11T09:51:50.636 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:50.636 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:50.636 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:50.636 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:50.636 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:51:50.636 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:51:50.636 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 53 2024-09-11T09:51:50.636 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.636 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:50.636 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.637 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:50.637 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951291+0000", 2024-09-11T09:51:50.637 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:50.637 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.637 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.637 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:50.637 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951291+0000", 2024-09-11T09:51:50.637 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:50.637 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.637 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.637 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:50.637 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951293+0000", 2024-09-11T09:51:50.637 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7889999999999999e-06 2024-09-11T09:51:50.637 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.638 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.638 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:50.638 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951302+0000", 2024-09-11T09:51:50.638 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.9819999999999997e-06 2024-09-11T09:51:50.638 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.638 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.638 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:50.638 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951324+0000", 2024-09-11T09:51:50.638 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.2141e-05 2024-09-11T09:51:50.638 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.638 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.638 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:50.639 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951338+0000", 2024-09-11T09:51:50.639 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3750999999999999e-05 2024-09-11T09:51:50.639 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.639 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.639 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:50.639 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951409+0000", 2024-09-11T09:51:50.639 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.1658999999999999e-05 2024-09-11T09:51:50.639 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.639 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.639 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:50.639 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951517+0000", 2024-09-11T09:51:50.639 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00010746799999999999 2024-09-11T09:51:50.639 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.640 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.640 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 5", 2024-09-11T09:51:50.640 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951942+0000", 2024-09-11T09:51:50.640 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00042524300000000001 2024-09-11T09:51:50.640 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.640 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.640 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:50.640 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.952872+0000", 2024-09-11T09:51:50.640 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00092976300000000001 2024-09-11T09:51:50.640 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.640 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.640 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:50.640 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.955682+0000", 2024-09-11T09:51:50.640 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0028099459999999998 2024-09-11T09:51:50.641 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.641 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.641 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:50.641 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.955746+0000", 2024-09-11T09:51:50.641 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.4084999999999994e-05 2024-09-11T09:51:50.641 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.641 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.641 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:50.641 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.955787+0000", 2024-09-11T09:51:50.641 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.1672999999999999e-05 2024-09-11T09:51:50.641 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:50.641 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:50.641 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:50.641 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.642 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.642 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4688.0:52 2.4 e53/32 2:26d5c2f0:::rbd_data.1250f5e25b80.000000000000000b:head v 53'8, mlcod=53'6)", 2024-09-11T09:51:50.642 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.963677+0000", 2024-09-11T09:51:50.642 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 27.638928850999999, 2024-09-11T09:51:50.642 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0089437969999999999, 2024-09-11T09:51:50.642 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:50.642 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:50.642 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:50.642 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:50.642 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.642 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:50.642 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.963677+0000", 2024-09-11T09:51:50.642 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:50.643 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.643 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.643 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:50.643 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.963677+0000", 2024-09-11T09:51:50.643 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:50.643 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.643 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.643 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:50.643 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.963677+0000", 2024-09-11T09:51:50.643 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.7199999999999998e-07 2024-09-11T09:51:50.643 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.643 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.643 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:50.643 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.965506+0000", 2024-09-11T09:51:50.644 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.001829064 2024-09-11T09:51:50.644 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.644 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.644 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:50.644 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.965538+0000", 2024-09-11T09:51:50.644 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.1306999999999997e-05 2024-09-11T09:51:50.644 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.644 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.644 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:50.644 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.965553+0000", 2024-09-11T09:51:50.644 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.4924e-05 2024-09-11T09:51:50.644 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.645 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.645 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:50.645 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.965663+0000", 2024-09-11T09:51:50.645 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000110315 2024-09-11T09:51:50.645 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.645 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.645 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:50.645 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.965692+0000", 2024-09-11T09:51:50.645 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.934e-05 2024-09-11T09:51:50.645 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.645 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.645 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:50.645 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.972537+0000", 2024-09-11T09:51:50.646 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0068444969999999997 2024-09-11T09:51:50.646 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.646 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.646 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:50.646 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.972620+0000", 2024-09-11T09:51:50.646 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.3578000000000005e-05 2024-09-11T09:51:50.646 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:50.646 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:50.646 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:50.646 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.646 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.646 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4688.0:54 2.4 e53/32 2:239e915b:::rbd_data.1250f5e25b80.000000000000000c:head v 53'9, mlcod=53'7)", 2024-09-11T09:51:50.646 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.987028+0000", 2024-09-11T09:51:50.647 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 27.615576934, 2024-09-11T09:51:50.647 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0094155650000000007, 2024-09-11T09:51:50.647 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:50.647 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:50.647 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:50.647 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:50.647 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.647 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:50.647 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.987028+0000", 2024-09-11T09:51:50.647 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:50.647 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.647 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.647 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:50.648 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.987028+0000", 2024-09-11T09:51:50.648 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:50.648 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.648 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.648 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:50.648 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.987030+0000", 2024-09-11T09:51:50.648 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6130000000000001e-06 2024-09-11T09:51:50.648 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.648 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.648 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:50.648 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.988575+0000", 2024-09-11T09:51:50.648 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.001545193 2024-09-11T09:51:50.648 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.649 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.649 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:50.649 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.988604+0000", 2024-09-11T09:51:50.649 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.9173e-05 2024-09-11T09:51:50.649 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.649 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.649 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:50.649 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.988620+0000", 2024-09-11T09:51:50.649 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.5517999999999999e-05 2024-09-11T09:51:50.649 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.649 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.649 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:50.649 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.988704+0000", 2024-09-11T09:51:50.650 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.3588e-05 2024-09-11T09:51:50.650 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.650 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.650 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:50.650 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.988735+0000", 2024-09-11T09:51:50.650 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.1643000000000002e-05 2024-09-11T09:51:50.650 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.650 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.650 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:50.650 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.996383+0000", 2024-09-11T09:51:50.650 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0076478099999999997 2024-09-11T09:51:50.650 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.650 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.651 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:50.651 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.996444+0000", 2024-09-11T09:51:50.651 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.1026999999999997e-05 2024-09-11T09:51:50.651 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:50.651 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:50.651 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:50.651 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.651 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.651 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4688.0:56 2.4 e53/32 2:2167012f:::rbd_data.1250f5e25b80.000000000000000d:head v 53'10, mlcod=53'7)", 2024-09-11T09:51:50.651 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.993894+0000", 2024-09-11T09:51:50.651 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 27.608711392, 2024-09-11T09:51:50.651 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0099268559999999995, 2024-09-11T09:51:50.651 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:50.651 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:50.652 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:50.652 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:50.652 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.652 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:50.652 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.993894+0000", 2024-09-11T09:51:50.652 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:50.652 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.652 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.652 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:50.652 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.993894+0000", 2024-09-11T09:51:50.652 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:50.652 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.652 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.653 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:50.653 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.993896+0000", 2024-09-11T09:51:50.653 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6249999999999999e-06 2024-09-11T09:51:50.653 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.653 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.653 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:50.653 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.997606+0000", 2024-09-11T09:51:50.653 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.003710737 2024-09-11T09:51:50.653 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.653 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.653 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:50.653 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.997628+0000", 2024-09-11T09:51:50.654 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.1971e-05 2024-09-11T09:51:50.654 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.654 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.654 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:50.654 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.997640+0000", 2024-09-11T09:51:50.654 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1481000000000001e-05 2024-09-11T09:51:50.654 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.654 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.654 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:50.654 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.997712+0000", 2024-09-11T09:51:50.654 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.2255000000000005e-05 2024-09-11T09:51:50.655 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.655 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.655 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:50.655 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.997738+0000", 2024-09-11T09:51:50.655 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.6050999999999998e-05 2024-09-11T09:51:50.655 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.655 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.655 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:50.655 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.003779+0000", 2024-09-11T09:51:50.655 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0060410960000000001 2024-09-11T09:51:50.656 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.656 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.656 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:50.656 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.003821+0000", 2024-09-11T09:51:50.656 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.1640000000000001e-05 2024-09-11T09:51:50.656 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:50.656 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:50.656 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:50.656 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.656 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.656 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "pg_update_log_missing(2.4 epoch 53/32 rep_tid 16 entries 53'11 (0'0) error 2:2906e6a7:::rbd_header.1250f5e25b80:head by client.4688.0:59 0.000000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 53'7)", 2024-09-11T09:51:50.656 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.998956+0000", 2024-09-11T09:51:50.656 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 27.603649223000001, 2024-09-11T09:51:50.656 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.004934127, 2024-09-11T09:51:50.657 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:50.657 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:50.657 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "reached pg", 2024-09-11T09:51:50.657 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:50.657 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.657 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:50.657 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.998956+0000", 2024-09-11T09:51:50.657 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:50.657 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.657 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.657 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:50.657 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.998956+0000", 2024-09-11T09:51:50.657 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:50.657 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.658 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.658 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:50.658 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.998957+0000", 2024-09-11T09:51:50.658 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.139e-06 2024-09-11T09:51:50.658 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.658 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.658 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:50.658 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.998985+0000", 2024-09-11T09:51:50.658 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.7912e-05 2024-09-11T09:51:50.658 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.658 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.658 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:50.658 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.999011+0000", 2024-09-11T09:51:50.658 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.6061e-05 2024-09-11T09:51:50.658 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.659 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.659 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:50.659 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.999028+0000", 2024-09-11T09:51:50.659 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7065999999999999e-05 2024-09-11T09:51:50.659 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.659 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.659 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:50.659 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.001919+0000", 2024-09-11T09:51:50.659 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.002891167 2024-09-11T09:51:50.659 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.659 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.659 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:50.659 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.003890+0000", 2024-09-11T09:51:50.659 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0019707819999999999 2024-09-11T09:51:50.660 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:50.660 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:50.660 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:50.660 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.660 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.660 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4694.0:34 2.7 2:f80bfc0c:ns::rbd_data.1256a0c02879.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:50.660 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.085183+0000", 2024-09-11T09:51:50.660 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 27.517422709999998, 2024-09-11T09:51:50.660 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.020908599, 2024-09-11T09:51:50.660 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:50.660 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:50.660 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:50.660 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:50.660 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4694", 2024-09-11T09:51:50.661 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:51:50.661 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 34 2024-09-11T09:51:50.661 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.661 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:50.661 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.661 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:50.661 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.085183+0000", 2024-09-11T09:51:50.661 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:50.661 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.661 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.661 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:50.661 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.085183+0000", 2024-09-11T09:51:50.661 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:50.661 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.661 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.662 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:50.662 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.085183+0000", 2024-09-11T09:51:50.662 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.4300000000000002e-07 2024-09-11T09:51:50.662 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.662 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.662 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:50.662 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094337+0000", 2024-09-11T09:51:50.662 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0091532769999999996 2024-09-11T09:51:50.662 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.662 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.662 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:50.662 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094362+0000", 2024-09-11T09:51:50.662 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.5287999999999999e-05 2024-09-11T09:51:50.662 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.663 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.663 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:50.663 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094375+0000", 2024-09-11T09:51:50.663 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2446000000000001e-05 2024-09-11T09:51:50.663 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.663 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.663 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:50.663 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094455+0000", 2024-09-11T09:51:50.663 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.0092999999999994e-05 2024-09-11T09:51:50.663 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.663 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.663 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:50.663 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094618+0000", 2024-09-11T09:51:50.663 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000163747 2024-09-11T09:51:50.664 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.664 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.664 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 5", 2024-09-11T09:51:50.664 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094783+0000", 2024-09-11T09:51:50.664 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000165038 2024-09-11T09:51:50.664 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.664 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.664 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:50.664 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.105853+0000", 2024-09-11T09:51:50.664 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.011069294 2024-09-11T09:51:50.664 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.664 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.664 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:50.664 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.106004+0000", 2024-09-11T09:51:50.665 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00015123900000000001 2024-09-11T09:51:50.665 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.665 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.665 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:50.665 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.106062+0000", 2024-09-11T09:51:50.665 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.8205000000000003e-05 2024-09-11T09:51:50.665 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.665 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.665 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:50.665 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.106091+0000", 2024-09-11T09:51:50.665 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.9128999999999998e-05 2024-09-11T09:51:50.665 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:50.665 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:50.666 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:50.666 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.666 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.666 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4694.0:35 2.7 2:e8766325:ns::rbd_object_map.1256a0c02879:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:50.666 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.094405+0000", 2024-09-11T09:51:50.666 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 27.508200074000001, 2024-09-11T09:51:50.666 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.012623564, 2024-09-11T09:51:50.666 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:50.666 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:50.666 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:50.666 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:50.666 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4694", 2024-09-11T09:51:50.666 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:51:50.666 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 35 2024-09-11T09:51:50.666 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.667 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:50.667 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.667 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:50.667 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094405+0000", 2024-09-11T09:51:50.667 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:50.667 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.667 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.667 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:50.667 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094405+0000", 2024-09-11T09:51:50.667 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:50.667 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.667 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.667 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:50.668 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094407+0000", 2024-09-11T09:51:50.668 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.739e-06 2024-09-11T09:51:50.668 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.668 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.668 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:50.668 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094411+0000", 2024-09-11T09:51:50.668 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.3340000000000003e-06 2024-09-11T09:51:50.668 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.668 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.668 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:50.668 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094419+0000", 2024-09-11T09:51:50.668 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.8909999999999995e-06 2024-09-11T09:51:50.669 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.669 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.669 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:50.669 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094424+0000", 2024-09-11T09:51:50.669 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.7639999999999996e-06 2024-09-11T09:51:50.669 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.669 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.669 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:50.669 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.100209+0000", 2024-09-11T09:51:50.669 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0057850489999999996 2024-09-11T09:51:50.669 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.669 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.669 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:50.670 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.100341+0000", 2024-09-11T09:51:50.670 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000131438 2024-09-11T09:51:50.670 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.670 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.670 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 5", 2024-09-11T09:51:50.670 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.100805+0000", 2024-09-11T09:51:50.670 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00046471499999999998 2024-09-11T09:51:50.670 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.670 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.670 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:50.670 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.106148+0000", 2024-09-11T09:51:50.670 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0053426460000000004 2024-09-11T09:51:50.670 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.670 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.671 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:50.671 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.106929+0000", 2024-09-11T09:51:50.671 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00078161300000000001 2024-09-11T09:51:50.671 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.671 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.671 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:50.671 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.106985+0000", 2024-09-11T09:51:50.671 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.5223999999999998e-05 2024-09-11T09:51:50.671 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.671 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.671 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:50.671 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.107029+0000", 2024-09-11T09:51:50.671 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.4150999999999998e-05 2024-09-11T09:51:50.671 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:50.672 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:50.672 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:50.672 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.672 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.672 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4694.0:36 2.7 2:e8766325:ns::rbd_object_map.1256a0c02879:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:50.672 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.094452+0000", 2024-09-11T09:51:50.672 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 27.508153063999998, 2024-09-11T09:51:50.672 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.012680383, 2024-09-11T09:51:50.672 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:50.672 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:50.672 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:50.672 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:50.673 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4694", 2024-09-11T09:51:50.673 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:51:50.673 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 36 2024-09-11T09:51:50.673 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.673 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:50.673 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.673 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:50.673 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094452+0000", 2024-09-11T09:51:50.673 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:50.673 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.673 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.673 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:50.673 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094452+0000", 2024-09-11T09:51:50.673 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:50.673 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.674 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.674 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:50.674 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094453+0000", 2024-09-11T09:51:50.674 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.1800000000000002e-07 2024-09-11T09:51:50.674 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.674 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.674 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:50.674 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094455+0000", 2024-09-11T09:51:50.674 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.7020000000000001e-06 2024-09-11T09:51:50.674 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.674 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.674 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:50.674 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094463+0000", 2024-09-11T09:51:50.674 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.1999999999999997e-06 2024-09-11T09:51:50.674 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.675 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.675 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:50.675 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094467+0000", 2024-09-11T09:51:50.675 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.729e-06 2024-09-11T09:51:50.675 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.675 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.675 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:50.675 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.101269+0000", 2024-09-11T09:51:50.675 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0068018799999999997 2024-09-11T09:51:50.675 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.675 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.675 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:50.675 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.101353+0000", 2024-09-11T09:51:50.675 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.3827999999999998e-05 2024-09-11T09:51:50.675 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.676 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.676 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 5", 2024-09-11T09:51:50.676 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.101588+0000", 2024-09-11T09:51:50.676 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00023467099999999999 2024-09-11T09:51:50.676 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.676 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.676 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:50.676 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.106182+0000", 2024-09-11T09:51:50.676 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0045939489999999999 2024-09-11T09:51:50.676 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.676 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.676 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:50.676 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.107041+0000", 2024-09-11T09:51:50.677 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00085965800000000002 2024-09-11T09:51:50.677 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.677 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.677 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:50.677 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.107084+0000", 2024-09-11T09:51:50.677 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.261e-05 2024-09-11T09:51:50.677 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.677 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.677 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:50.677 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.107133+0000", 2024-09-11T09:51:50.677 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.8838000000000002e-05 2024-09-11T09:51:50.677 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:50.677 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:50.677 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:50.677 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.678 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.678 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4694.0:37 2.7 2:e8766325:ns::rbd_object_map.1256a0c02879:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:50.678 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.094473+0000", 2024-09-11T09:51:50.678 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 27.508131913, 2024-09-11T09:51:50.678 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.012778484999999999, 2024-09-11T09:51:50.678 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:50.678 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:50.678 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:50.678 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:50.678 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4694", 2024-09-11T09:51:50.678 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:51:50.678 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 37 2024-09-11T09:51:50.678 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.678 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:50.679 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.679 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:50.679 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094473+0000", 2024-09-11T09:51:50.679 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:50.679 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.679 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.679 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:50.679 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094473+0000", 2024-09-11T09:51:50.679 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:50.679 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.679 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.679 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:50.679 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094474+0000", 2024-09-11T09:51:50.679 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.9699999999999996e-07 2024-09-11T09:51:50.680 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.680 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.680 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:50.680 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094477+0000", 2024-09-11T09:51:50.680 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.151e-06 2024-09-11T09:51:50.680 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.680 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.680 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:50.680 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094483+0000", 2024-09-11T09:51:50.680 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.6330000000000001e-06 2024-09-11T09:51:50.680 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.680 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.680 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:50.680 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094487+0000", 2024-09-11T09:51:50.681 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.5399999999999997e-06 2024-09-11T09:51:50.681 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.681 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.681 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:50.681 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.101866+0000", 2024-09-11T09:51:50.681 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.007378768 2024-09-11T09:51:50.681 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.681 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.681 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:50.681 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.101960+0000", 2024-09-11T09:51:50.681 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.3861000000000001e-05 2024-09-11T09:51:50.681 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.681 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.681 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 5", 2024-09-11T09:51:50.681 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.102197+0000", 2024-09-11T09:51:50.682 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00023723200000000001 2024-09-11T09:51:50.682 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.682 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.682 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:50.682 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.106222+0000", 2024-09-11T09:51:50.682 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0040249689999999998 2024-09-11T09:51:50.682 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.682 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.682 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:50.682 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.107141+0000", 2024-09-11T09:51:50.682 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00091913900000000002 2024-09-11T09:51:50.682 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.682 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.683 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:50.683 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.107198+0000", 2024-09-11T09:51:50.683 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.6564999999999998e-05 2024-09-11T09:51:50.683 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.683 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.683 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:50.683 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.107252+0000", 2024-09-11T09:51:50.683 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.4030000000000003e-05 2024-09-11T09:51:50.683 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:50.683 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:50.683 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:50.683 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.683 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.684 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4694.0:33 2.4 e53/32 2:3bfc818c:ns::rbd_data.1256a0c02879.0000000000000001:head v 53'14, mlcod=53'12)", 2024-09-11T09:51:50.684 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.096496+0000", 2024-09-11T09:51:50.684 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 27.506109339999998, 2024-09-11T09:51:50.684 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0099139310000000008, 2024-09-11T09:51:50.684 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:50.684 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:50.684 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:50.684 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:50.684 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.684 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:50.684 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.096496+0000", 2024-09-11T09:51:50.684 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:50.684 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.685 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.685 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:50.685 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.096496+0000", 2024-09-11T09:51:50.685 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:50.685 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.685 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.685 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:50.685 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.096498+0000", 2024-09-11T09:51:50.685 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6309999999999999e-06 2024-09-11T09:51:50.685 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.685 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.685 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:50.685 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.098087+0000", 2024-09-11T09:51:50.686 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0015889019999999999 2024-09-11T09:51:50.686 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.686 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.686 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:50.686 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.098112+0000", 2024-09-11T09:51:50.686 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.5369000000000001e-05 2024-09-11T09:51:50.686 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.686 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.686 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:50.686 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.098127+0000", 2024-09-11T09:51:50.686 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.5218999999999999e-05 2024-09-11T09:51:50.686 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.687 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.687 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:50.687 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.098209+0000", 2024-09-11T09:51:50.687 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.2120999999999995e-05 2024-09-11T09:51:50.687 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.687 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.687 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:50.687 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.098237+0000", 2024-09-11T09:51:50.687 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.7657999999999998e-05 2024-09-11T09:51:50.687 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.687 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.687 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:50.688 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.106376+0000", 2024-09-11T09:51:50.688 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0081386749999999997 2024-09-11T09:51:50.688 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.688 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.688 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:50.688 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.106410+0000", 2024-09-11T09:51:50.688 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.4356000000000001e-05 2024-09-11T09:51:50.688 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:50.688 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:50.688 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:50.688 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.688 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.688 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4694.0:49 2.7 2:e8766325:ns::rbd_object_map.1256a0c02879:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:50.689 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.134806+0000", 2024-09-11T09:51:50.689 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 27.467799271000001, 2024-09-11T09:51:50.689 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0061629049999999998, 2024-09-11T09:51:50.689 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:50.689 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:50.689 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:50.689 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:50.689 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4694", 2024-09-11T09:51:50.689 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:51:50.689 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 49 2024-09-11T09:51:50.689 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.689 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:50.689 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.690 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:50.690 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.134806+0000", 2024-09-11T09:51:50.690 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:50.690 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.690 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.690 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:50.690 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.134806+0000", 2024-09-11T09:51:50.690 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:50.690 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.690 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.690 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:50.690 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.134808+0000", 2024-09-11T09:51:50.690 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.9870000000000002e-06 2024-09-11T09:51:50.691 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.691 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.691 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:50.691 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.134818+0000", 2024-09-11T09:51:50.691 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.5599999999999999e-06 2024-09-11T09:51:50.691 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.691 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.691 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:50.691 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.134835+0000", 2024-09-11T09:51:50.691 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7399000000000001e-05 2024-09-11T09:51:50.692 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.692 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.692 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:50.692 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.134845+0000", 2024-09-11T09:51:50.692 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0285000000000001e-05 2024-09-11T09:51:50.692 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.692 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.692 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:50.692 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.134911+0000", 2024-09-11T09:51:50.692 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.5654999999999994e-05 2024-09-11T09:51:50.692 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.692 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.692 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:50.692 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.135023+0000", 2024-09-11T09:51:50.693 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00011186799999999999 2024-09-11T09:51:50.693 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.693 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.693 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 5", 2024-09-11T09:51:50.693 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.135290+0000", 2024-09-11T09:51:50.693 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00026725300000000001 2024-09-11T09:51:50.693 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.693 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.693 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:50.693 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.136506+0000", 2024-09-11T09:51:50.693 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0012161139999999999 2024-09-11T09:51:50.693 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.693 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.694 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:50.694 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.140884+0000", 2024-09-11T09:51:50.694 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0043781990000000002 2024-09-11T09:51:50.694 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.694 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.694 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:50.694 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.140923+0000", 2024-09-11T09:51:50.694 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.8813999999999998e-05 2024-09-11T09:51:50.694 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.694 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.694 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:50.694 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.140969+0000", 2024-09-11T09:51:50.694 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.5770999999999999e-05 2024-09-11T09:51:50.694 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:50.695 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:50.695 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:50.695 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.695 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.695 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4694.0:42 2.5 e53/32 2:a4fc00ec:ns::rbd_data.1256a0c02879.0000000000000004:head v 53'3, mlcod=53'1)", 2024-09-11T09:51:50.695 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.136646+0000", 2024-09-11T09:51:50.695 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 27.465958964999999, 2024-09-11T09:51:50.695 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.012545501000000001, 2024-09-11T09:51:50.695 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:50.695 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:50.695 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:50.695 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:50.695 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.696 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:50.696 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.136646+0000", 2024-09-11T09:51:50.696 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:50.696 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.696 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.696 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:50.696 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.136646+0000", 2024-09-11T09:51:50.696 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:50.696 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.696 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.696 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:50.696 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.136647+0000", 2024-09-11T09:51:50.696 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.2900000000000002e-07 2024-09-11T09:51:50.697 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.697 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.697 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:50.697 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.143620+0000", 2024-09-11T09:51:50.697 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0069727440000000003 2024-09-11T09:51:50.697 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.697 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.697 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:50.697 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.143642+0000", 2024-09-11T09:51:50.697 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.1846000000000001e-05 2024-09-11T09:51:50.697 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.697 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.697 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:50.697 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.143654+0000", 2024-09-11T09:51:50.698 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2081e-05 2024-09-11T09:51:50.698 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.698 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.698 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:50.698 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.143717+0000", 2024-09-11T09:51:50.698 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.3443999999999998e-05 2024-09-11T09:51:50.698 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.698 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.698 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:50.698 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.143740+0000", 2024-09-11T09:51:50.698 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.2626999999999999e-05 2024-09-11T09:51:50.698 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.698 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.698 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:50.699 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.149142+0000", 2024-09-11T09:51:50.699 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0054017630000000004 2024-09-11T09:51:50.699 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.699 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.699 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:50.699 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.149192+0000", 2024-09-11T09:51:50.699 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.0067e-05 2024-09-11T09:51:50.699 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:50.699 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:50.699 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:50.699 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.699 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.700 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4694.0:53 2.7 2:e8766325:ns::rbd_object_map.1256a0c02879:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:50.700 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.147978+0000", 2024-09-11T09:51:50.700 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 27.454627480999999, 2024-09-11T09:51:50.700 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0044549769999999997, 2024-09-11T09:51:50.700 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:50.700 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:50.700 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:50.700 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:50.700 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4694", 2024-09-11T09:51:50.700 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:51:50.700 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 53 2024-09-11T09:51:50.700 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.700 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:50.701 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.701 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:50.701 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.147978+0000", 2024-09-11T09:51:50.701 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:50.701 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.701 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.701 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:50.701 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.147978+0000", 2024-09-11T09:51:50.701 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:50.701 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.701 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.701 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:50.701 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.147980+0000", 2024-09-11T09:51:50.702 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.8309999999999999e-06 2024-09-11T09:51:50.702 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.702 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.702 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:50.702 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.147989+0000", 2024-09-11T09:51:50.702 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.9639999999999992e-06 2024-09-11T09:51:50.702 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.702 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.702 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:50.702 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.148007+0000", 2024-09-11T09:51:50.702 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7884999999999999e-05 2024-09-11T09:51:50.702 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.702 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.702 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:50.703 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.148017+0000", 2024-09-11T09:51:50.703 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0895e-05 2024-09-11T09:51:50.703 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.703 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.703 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:50.703 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.148091+0000", 2024-09-11T09:51:50.703 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.3709999999999997e-05 2024-09-11T09:51:50.703 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.703 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.703 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:50.703 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.148189+0000", 2024-09-11T09:51:50.703 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.7403000000000002e-05 2024-09-11T09:51:50.704 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.704 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.704 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 5", 2024-09-11T09:51:50.704 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.148504+0000", 2024-09-11T09:51:50.704 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00031589900000000002 2024-09-11T09:51:50.704 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.704 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.704 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:50.704 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.149514+0000", 2024-09-11T09:51:50.704 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0010096 2024-09-11T09:51:50.704 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.704 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.704 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:50.705 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.152367+0000", 2024-09-11T09:51:50.705 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0028532309999999999 2024-09-11T09:51:50.705 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.705 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.705 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:50.705 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.152402+0000", 2024-09-11T09:51:50.705 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.4232e-05 2024-09-11T09:51:50.705 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.705 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.705 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:50.705 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.152433+0000", 2024-09-11T09:51:50.705 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.1327e-05 2024-09-11T09:51:50.705 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:50.705 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:50.705 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:50.706 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.706 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.706 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4694.0:55 2.7 2:e8766325:ns::rbd_object_map.1256a0c02879:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:50.706 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.149838+0000", 2024-09-11T09:51:50.706 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 27.452767747999999, 2024-09-11T09:51:50.706 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0060695159999999996, 2024-09-11T09:51:50.706 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:50.706 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:50.706 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:50.706 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:50.706 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4694", 2024-09-11T09:51:50.706 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:51:50.706 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 55 2024-09-11T09:51:50.706 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.707 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:50.707 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.707 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:50.707 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.149838+0000", 2024-09-11T09:51:50.707 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:50.707 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.707 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.707 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:50.707 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.149838+0000", 2024-09-11T09:51:50.707 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:50.707 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.707 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.707 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:50.707 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.149839+0000", 2024-09-11T09:51:50.707 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.119e-06 2024-09-11T09:51:50.708 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.708 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.708 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:50.708 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.149845+0000", 2024-09-11T09:51:50.708 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.7810000000000002e-06 2024-09-11T09:51:50.708 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.708 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.708 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:50.708 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.149856+0000", 2024-09-11T09:51:50.708 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1199999999999999e-05 2024-09-11T09:51:50.708 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.708 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.708 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:50.709 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.149863+0000", 2024-09-11T09:51:50.709 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.4109999999999999e-06 2024-09-11T09:51:50.709 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.709 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.709 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:50.709 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.149908+0000", 2024-09-11T09:51:50.709 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.5184e-05 2024-09-11T09:51:50.709 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.709 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.709 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:50.709 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.149964+0000", 2024-09-11T09:51:50.709 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.5395000000000001e-05 2024-09-11T09:51:50.709 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.709 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.710 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 5", 2024-09-11T09:51:50.710 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.150157+0000", 2024-09-11T09:51:50.710 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000193489 2024-09-11T09:51:50.710 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.710 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.710 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:50.710 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.152238+0000", 2024-09-11T09:51:50.710 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0020803480000000001 2024-09-11T09:51:50.710 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.710 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.710 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:50.710 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.155828+0000", 2024-09-11T09:51:50.710 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0035902210000000002 2024-09-11T09:51:50.710 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.710 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.711 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:50.711 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.155873+0000", 2024-09-11T09:51:50.711 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.5145999999999998e-05 2024-09-11T09:51:50.711 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:50.711 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:50.711 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:50.711 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.155907+0000", 2024-09-11T09:51:50.711 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.4221999999999999e-05 2024-09-11T09:51:50.711 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:50.711 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:50.711 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:50.711 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:50.711 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:50.711 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:51:50.713 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-11T09:51:50.713 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:51:50.713+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:51:50.800 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-11T09:51:50.801 INFO:tasks.workunit.client.0.smithi107.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2024-09-11T09:51:50.806 INFO:tasks.workunit.client.0.smithi107.stderr:+ _sudo dd if=/tmp/tmp.rvrcDirNue/data of=/dev/nbd1 bs=1M count=1 oflag=direct 2024-09-11T09:51:50.806 INFO:tasks.workunit.client.0.smithi107.stderr:+ local cmd 2024-09-11T09:51:50.807 INFO:tasks.workunit.client.0.smithi107.stderr:++ id -u 2024-09-11T09:51:50.807 INFO:tasks.workunit.client.0.smithi107.stderr:+ '[' 1000 -eq 0 ']' 2024-09-11T09:51:50.808 INFO:tasks.workunit.client.0.smithi107.stderr:++ which dd 2024-09-11T09:51:50.809 INFO:tasks.workunit.client.0.smithi107.stderr:+ cmd=/usr/bin/dd 2024-09-11T09:51:50.810 INFO:tasks.workunit.client.0.smithi107.stderr:+ shift 2024-09-11T09:51:50.810 INFO:tasks.workunit.client.0.smithi107.stderr:+ sudo -nE /usr/bin/dd if=/tmp/tmp.rvrcDirNue/data of=/dev/nbd1 bs=1M count=1 oflag=direct 2024-09-11T09:51:50.831 INFO:tasks.workunit.client.0.smithi107.stderr:1+0 records in 2024-09-11T09:51:50.832 INFO:tasks.workunit.client.0.smithi107.stderr:1+0 records out 2024-09-11T09:51:50.832 INFO:tasks.workunit.client.0.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0124051 s, 84.5 MB/s 2024-09-11T09:51:50.833 INFO:tasks.workunit.client.0.smithi107.stderr:++ cat /tmp/tmp.rvrcDirNue/quiesce 2024-09-11T09:51:50.834 INFO:tasks.workunit.client.0.smithi107.stderr:+ test /dev/nbd1 = /dev/nbd1 2024-09-11T09:51:50.834 INFO:tasks.workunit.client.0.smithi107.stderr:++ cat /tmp/tmp.rvrcDirNue/unquiesce 2024-09-11T09:51:50.835 INFO:tasks.workunit.client.0.smithi107.stderr:+ test /dev/nbd1 = /dev/nbd1 2024-09-11T09:51:50.835 INFO:tasks.workunit.client.0.smithi107.stderr:+ touch /tmp/tmp.rvrcDirNue/quiesce.sh 2024-09-11T09:51:50.836 INFO:tasks.workunit.client.0.smithi107.stderr:+ chmod +x /tmp/tmp.rvrcDirNue/quiesce.sh 2024-09-11T09:51:50.836 INFO:tasks.workunit.client.0.smithi107.stderr:+ cat 2024-09-11T09:51:50.837 INFO:tasks.workunit.client.0.smithi107.stderr:+ expect_false rbd snap create rbd/testrbdnbd27121@quiesce2 2024-09-11T09:51:50.837 INFO:tasks.workunit.client.0.smithi107.stderr:+ rbd snap create rbd/testrbdnbd27121@quiesce2 2024-09-11T09:51:50.848 INFO:tasks.ceph.osd.2.smithi017.stderr:2024-09-11T09:51:50.845+0000 7fd20b170640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31672) UID: 0 2024-09-11T09:51:50.870 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:51:50.870 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [], 2024-09-11T09:51:50.870 INFO:teuthology.orchestra.run.smithi017.stdout: "num_ops": 0 2024-09-11T09:51:50.870 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:51:50.880 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-11T09:51:50.890 INFO:tasks.workunit.client.0.smithi107.stderr: Creating snap: 10% complete... Creating snap: 10% complete...failed. 2024-09-11T09:51:50.890 INFO:tasks.workunit.client.0.smithi107.stderr:rbd: failed to create snapshot: (22) Invalid argument 2024-09-11T09:51:50.895 INFO:tasks.workunit.client.0.smithi107.stderr:+ return 0 2024-09-11T09:51:50.895 INFO:tasks.workunit.client.0.smithi107.stderr:+ _sudo dd if=/tmp/tmp.rvrcDirNue/data of=/dev/nbd1 bs=1M count=1 oflag=direct 2024-09-11T09:51:50.895 INFO:tasks.workunit.client.0.smithi107.stderr:+ local cmd 2024-09-11T09:51:50.895 INFO:tasks.workunit.client.0.smithi107.stderr:++ id -u 2024-09-11T09:51:50.896 INFO:tasks.workunit.client.0.smithi107.stderr:+ '[' 1000 -eq 0 ']' 2024-09-11T09:51:50.896 INFO:tasks.workunit.client.0.smithi107.stderr:++ which dd 2024-09-11T09:51:50.898 INFO:tasks.workunit.client.0.smithi107.stderr:+ cmd=/usr/bin/dd 2024-09-11T09:51:50.898 INFO:tasks.workunit.client.0.smithi107.stderr:+ shift 2024-09-11T09:51:50.898 INFO:tasks.workunit.client.0.smithi107.stderr:+ sudo -nE /usr/bin/dd if=/tmp/tmp.rvrcDirNue/data of=/dev/nbd1 bs=1M count=1 oflag=direct 2024-09-11T09:51:50.914 INFO:tasks.workunit.client.0.smithi107.stderr:1+0 records in 2024-09-11T09:51:50.914 INFO:tasks.workunit.client.0.smithi107.stderr:1+0 records out 2024-09-11T09:51:50.914 INFO:tasks.workunit.client.0.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00669289 s, 157 MB/s 2024-09-11T09:51:50.915 INFO:tasks.workunit.client.0.smithi107.stderr:+ cat 2024-09-11T09:51:50.916 INFO:tasks.workunit.client.0.smithi107.stderr:+ rbd snap create rbd/testrbdnbd27121@quiesce2 2024-09-11T09:51:50.921 INFO:tasks.ceph.osd.0.smithi017.stderr:2024-09-11T09:51:50.921+0000 7fb73c0f1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31665) UID: 0 2024-09-11T09:51:51.015 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-09-11T09:51:51.015+0000 7f7b6c7d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31904) UID: 0 2024-09-11T09:51:51.029 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:51:51.029 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [], 2024-09-11T09:51:51.030 INFO:teuthology.orchestra.run.smithi017.stdout: "complaint_time": 30, 2024-09-11T09:51:51.030 INFO:teuthology.orchestra.run.smithi017.stdout: "num_blocked_ops": 0 2024-09-11T09:51:51.030 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:51:51.038 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-11T09:51:51.116 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:51:51.116+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:51:51.148 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:51:51.148 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 20, 2024-09-11T09:51:51.148 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 600, 2024-09-11T09:51:51.148 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [ 2024-09-11T09:51:51.148 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.149 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:34 2.4 2:2367385a:::rbd_data.1250f5e25b80.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:51.149 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.897717+0000", 2024-09-11T09:51:51.149 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 28.250504881000001, 2024-09-11T09:51:51.149 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.021323661000000001, 2024-09-11T09:51:51.149 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:51.149 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:51.149 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:51.149 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:51.149 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:51:51.149 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:51:51.149 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 34 2024-09-11T09:51:51.149 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.150 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:51.150 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.150 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:51.150 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.897717+0000", 2024-09-11T09:51:51.150 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:51.150 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.150 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.150 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:51.150 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.897717+0000", 2024-09-11T09:51:51.150 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:51.150 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.150 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.151 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:51.151 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.897718+0000", 2024-09-11T09:51:51.151 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.1800000000000004e-07 2024-09-11T09:51:51.151 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.151 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.151 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:51.151 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.905339+0000", 2024-09-11T09:51:51.151 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0076210239999999997 2024-09-11T09:51:51.151 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.151 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.151 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:51.151 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.905366+0000", 2024-09-11T09:51:51.152 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.6959000000000001e-05 2024-09-11T09:51:51.152 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.152 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.152 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:51.152 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.905375+0000", 2024-09-11T09:51:51.152 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.516e-06 2024-09-11T09:51:51.152 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.152 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.152 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:51.152 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.905475+0000", 2024-09-11T09:51:51.152 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.9738e-05 2024-09-11T09:51:51.152 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.153 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.153 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:51.153 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.905660+0000", 2024-09-11T09:51:51.153 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00018554199999999999 2024-09-11T09:51:51.153 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.153 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.153 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:51.153 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.905808+0000", 2024-09-11T09:51:51.153 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000147386 2024-09-11T09:51:51.153 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.153 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.153 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:51.154 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.918230+0000", 2024-09-11T09:51:51.154 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.012421690000000001 2024-09-11T09:51:51.154 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.154 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.154 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:51.154 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.918932+0000", 2024-09-11T09:51:51.154 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00070267300000000001 2024-09-11T09:51:51.154 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.154 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.154 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:51.154 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.918989+0000", 2024-09-11T09:51:51.154 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.7268999999999997e-05 2024-09-11T09:51:51.154 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.155 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.155 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:51.155 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919040+0000", 2024-09-11T09:51:51.155 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.0946000000000003e-05 2024-09-11T09:51:51.155 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:51.155 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:51.155 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:51.155 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.155 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.155 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4688.0:31 2.1 e53/15 2:8654721b:::rbd_data.1250f5e25b80.0000000000000000:head v 53'2, mlcod=0'0)", 2024-09-11T09:51:51.155 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.904661+0000", 2024-09-11T09:51:51.156 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 28.243560391999999, 2024-09-11T09:51:51.156 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.014152804999999999, 2024-09-11T09:51:51.156 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:51.156 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:51.156 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:51.156 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:51.156 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.156 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:51.156 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.904661+0000", 2024-09-11T09:51:51.156 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:51.156 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.156 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.156 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:51.157 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.904661+0000", 2024-09-11T09:51:51.157 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:51.157 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.157 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.157 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:51.157 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.904662+0000", 2024-09-11T09:51:51.157 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0759999999999999e-06 2024-09-11T09:51:51.157 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.157 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.157 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:51.157 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.907298+0000", 2024-09-11T09:51:51.157 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0026361240000000001 2024-09-11T09:51:51.157 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.158 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.158 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:51.158 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.907320+0000", 2024-09-11T09:51:51.158 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.1268999999999999e-05 2024-09-11T09:51:51.158 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.158 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.158 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:51.158 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.907331+0000", 2024-09-11T09:51:51.158 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1031e-05 2024-09-11T09:51:51.158 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.158 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.158 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:51.158 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.907403+0000", 2024-09-11T09:51:51.158 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.2571999999999996e-05 2024-09-11T09:51:51.159 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.159 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.159 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:51.159 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.907428+0000", 2024-09-11T09:51:51.159 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.4380999999999999e-05 2024-09-11T09:51:51.159 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.159 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.159 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:51.159 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.918750+0000", 2024-09-11T09:51:51.159 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.011322211 2024-09-11T09:51:51.159 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.159 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.159 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:51.159 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.918814+0000", 2024-09-11T09:51:51.160 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.4140999999999995e-05 2024-09-11T09:51:51.160 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:51.160 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:51.160 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:51.160 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.160 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.160 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:36 2.6 2:61e1b442:::rbd_data.1250f5e25b80.0000000000000003:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:51.160 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.905404+0000", 2024-09-11T09:51:51.160 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 28.242817520999999, 2024-09-11T09:51:51.160 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.030912479999999999, 2024-09-11T09:51:51.160 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:51.160 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:51.160 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:51.160 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:51.160 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:51:51.161 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:51:51.161 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 36 2024-09-11T09:51:51.161 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.161 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:51.161 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.161 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:51.161 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.905404+0000", 2024-09-11T09:51:51.161 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:51.161 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.161 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.161 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:51.161 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.905404+0000", 2024-09-11T09:51:51.161 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:51.161 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.162 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.162 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:51.162 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.905405+0000", 2024-09-11T09:51:51.162 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2270000000000001e-06 2024-09-11T09:51:51.162 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.162 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.162 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:51.162 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.914859+0000", 2024-09-11T09:51:51.162 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0094541190000000004 2024-09-11T09:51:51.162 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.162 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.162 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:51.162 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.914879+0000", 2024-09-11T09:51:51.162 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.0052999999999999e-05 2024-09-11T09:51:51.163 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.163 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.163 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:51.163 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.914897+0000", 2024-09-11T09:51:51.163 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7331000000000001e-05 2024-09-11T09:51:51.163 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.163 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.163 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:51.163 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.915019+0000", 2024-09-11T09:51:51.163 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00012168 2024-09-11T09:51:51.163 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.163 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.163 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:51.164 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.915269+0000", 2024-09-11T09:51:51.164 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00025054899999999998 2024-09-11T09:51:51.164 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.164 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.164 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 3", 2024-09-11T09:51:51.164 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.915479+0000", 2024-09-11T09:51:51.164 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00021017799999999999 2024-09-11T09:51:51.164 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.164 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.164 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:51.164 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.926070+0000", 2024-09-11T09:51:51.164 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.010590624999999999 2024-09-11T09:51:51.164 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.164 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.165 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:51.165 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.936198+0000", 2024-09-11T09:51:51.165 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.010128194 2024-09-11T09:51:51.165 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.165 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.165 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:51.165 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.936263+0000", 2024-09-11T09:51:51.165 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.5366999999999999e-05 2024-09-11T09:51:51.165 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.165 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.165 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:51.165 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.936317+0000", 2024-09-11T09:51:51.165 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.3156999999999999e-05 2024-09-11T09:51:51.165 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:51.166 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:51.166 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:51.166 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.166 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.166 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:39 2.6 2:743e7616:::rbd_data.1250f5e25b80.0000000000000005:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:51.166 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.914960+0000", 2024-09-11T09:51:51.166 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 28.233261218999999, 2024-09-11T09:51:51.166 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.022371518, 2024-09-11T09:51:51.166 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:51.166 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:51.166 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:51.166 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:51.166 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:51:51.166 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:51:51.166 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 39 2024-09-11T09:51:51.167 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.167 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:51.167 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.167 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:51.167 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.914960+0000", 2024-09-11T09:51:51.167 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:51.167 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.167 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.167 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:51.167 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.914960+0000", 2024-09-11T09:51:51.167 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:51.167 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.168 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.168 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:51.168 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.914963+0000", 2024-09-11T09:51:51.168 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.1119999999999999e-06 2024-09-11T09:51:51.168 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.168 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.168 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:51.168 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.920417+0000", 2024-09-11T09:51:51.168 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0054543969999999997 2024-09-11T09:51:51.168 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.168 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.168 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:51.168 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.920430+0000", 2024-09-11T09:51:51.168 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3414000000000001e-05 2024-09-11T09:51:51.169 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.169 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.169 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:51.169 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.920437+0000", 2024-09-11T09:51:51.169 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.6159999999999998e-06 2024-09-11T09:51:51.169 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.169 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.169 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:51.169 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.921325+0000", 2024-09-11T09:51:51.169 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000887731 2024-09-11T09:51:51.169 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.169 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.169 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:51.169 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.921422+0000", 2024-09-11T09:51:51.170 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.7201999999999995e-05 2024-09-11T09:51:51.170 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.170 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.170 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 3", 2024-09-11T09:51:51.170 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.921511+0000", 2024-09-11T09:51:51.170 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.8880000000000003e-05 2024-09-11T09:51:51.170 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.170 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.170 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:51.170 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.928092+0000", 2024-09-11T09:51:51.170 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0065816290000000003 2024-09-11T09:51:51.170 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.170 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.170 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:51.170 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.937240+0000", 2024-09-11T09:51:51.171 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0091481070000000008 2024-09-11T09:51:51.171 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.171 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.171 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:51.171 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.937294+0000", 2024-09-11T09:51:51.171 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.3100000000000003e-05 2024-09-11T09:51:51.171 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.171 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.171 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:51.171 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.937332+0000", 2024-09-11T09:51:51.171 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.8330000000000001e-05 2024-09-11T09:51:51.171 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:51.172 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:51.172 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:51.172 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.172 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.172 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4688.0:32 2.2 e53/15 2:45941fd2:::rbd_data.1250f5e25b80.0000000000000001:head v 53'3, mlcod=17'1)", 2024-09-11T09:51:51.172 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.916509+0000", 2024-09-11T09:51:51.172 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 28.231712136999999, 2024-09-11T09:51:51.172 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.015885318999999998, 2024-09-11T09:51:51.172 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:51.172 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:51.172 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:51.172 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:51.172 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.172 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:51.173 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.916509+0000", 2024-09-11T09:51:51.173 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:51.173 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.173 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.173 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:51.173 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.916509+0000", 2024-09-11T09:51:51.173 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:51.173 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.173 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.173 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:51.173 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.916511+0000", 2024-09-11T09:51:51.173 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.052e-06 2024-09-11T09:51:51.173 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.173 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.174 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:51.174 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.927019+0000", 2024-09-11T09:51:51.174 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.010508224 2024-09-11T09:51:51.174 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.174 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.174 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:51.174 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.927038+0000", 2024-09-11T09:51:51.174 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.9590999999999999e-05 2024-09-11T09:51:51.174 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.174 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.174 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:51.174 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.927050+0000", 2024-09-11T09:51:51.174 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1439e-05 2024-09-11T09:51:51.175 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.175 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.175 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:51.175 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.927115+0000", 2024-09-11T09:51:51.175 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.5301000000000004e-05 2024-09-11T09:51:51.175 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.175 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.175 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:51.175 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.927132+0000", 2024-09-11T09:51:51.175 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7297000000000001e-05 2024-09-11T09:51:51.175 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.175 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.175 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:51.175 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.932354+0000", 2024-09-11T09:51:51.175 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0052211870000000004 2024-09-11T09:51:51.176 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.176 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.176 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:51.176 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.932395+0000", 2024-09-11T09:51:51.176 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.1227999999999999e-05 2024-09-11T09:51:51.176 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:51.176 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:51.176 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:51.176 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.176 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.176 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:44 2.6 2:67246a22:::rbd_data.1250f5e25b80.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:51.176 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.943181+0000", 2024-09-11T09:51:51.176 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 28.205040925999999, 2024-09-11T09:51:51.176 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.022629371999999998, 2024-09-11T09:51:51.177 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:51.177 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:51.177 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:51.177 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:51.177 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:51:51.177 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:51:51.177 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 44 2024-09-11T09:51:51.177 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.177 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:51.177 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.177 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:51.177 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.943181+0000", 2024-09-11T09:51:51.177 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:51.177 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.177 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.178 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:51.178 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.943181+0000", 2024-09-11T09:51:51.178 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:51.178 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.178 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.178 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:51.178 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.943182+0000", 2024-09-11T09:51:51.178 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.0500000000000002e-07 2024-09-11T09:51:51.178 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.178 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.178 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:51.178 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951240+0000", 2024-09-11T09:51:51.178 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.008058529 2024-09-11T09:51:51.178 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.179 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.179 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:51.179 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951260+0000", 2024-09-11T09:51:51.179 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.9570000000000001e-05 2024-09-11T09:51:51.179 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.179 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.179 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:51.179 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951270+0000", 2024-09-11T09:51:51.179 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0018e-05 2024-09-11T09:51:51.179 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.179 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.179 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:51.179 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951349+0000", 2024-09-11T09:51:51.179 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.8801000000000007e-05 2024-09-11T09:51:51.180 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.180 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.180 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:51.180 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951483+0000", 2024-09-11T09:51:51.180 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00013427 2024-09-11T09:51:51.180 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.180 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.180 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 3", 2024-09-11T09:51:51.180 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951593+0000", 2024-09-11T09:51:51.180 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00011031000000000001 2024-09-11T09:51:51.180 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.180 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.180 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:51.180 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.957303+0000", 2024-09-11T09:51:51.180 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0057097959999999996 2024-09-11T09:51:51.181 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.181 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.181 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:51.181 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.965690+0000", 2024-09-11T09:51:51.181 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0083867420000000008 2024-09-11T09:51:51.181 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.181 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.181 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:51.181 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.965749+0000", 2024-09-11T09:51:51.181 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.8981999999999999e-05 2024-09-11T09:51:51.181 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.181 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.181 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:51.181 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.965810+0000", 2024-09-11T09:51:51.182 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.1449000000000001e-05 2024-09-11T09:51:51.182 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:51.182 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:51.182 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:51.182 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.182 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.182 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:51 2.6 2:6b5894fc:::rbd_data.1250f5e25b80.000000000000000a:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:51.182 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.951324+0000", 2024-09-11T09:51:51.182 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 28.196897852999999, 2024-09-11T09:51:51.182 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.021375273, 2024-09-11T09:51:51.182 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:51.182 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:51.182 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:51.182 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:51.183 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:51:51.183 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:51:51.183 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 51 2024-09-11T09:51:51.183 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.183 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:51.183 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.183 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:51.183 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951324+0000", 2024-09-11T09:51:51.183 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:51.183 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.183 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.183 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:51.183 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951324+0000", 2024-09-11T09:51:51.183 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:51.183 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.184 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.184 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:51.184 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951324+0000", 2024-09-11T09:51:51.184 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.39e-07 2024-09-11T09:51:51.184 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.184 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.184 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:51.184 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.958537+0000", 2024-09-11T09:51:51.184 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0072126769999999998 2024-09-11T09:51:51.184 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.184 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.184 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:51.184 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.958554+0000", 2024-09-11T09:51:51.185 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7232e-05 2024-09-11T09:51:51.185 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.185 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.185 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:51.185 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.958570+0000", 2024-09-11T09:51:51.185 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.5747999999999999e-05 2024-09-11T09:51:51.185 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.185 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.185 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:51.185 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.958638+0000", 2024-09-11T09:51:51.185 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.7810000000000003e-05 2024-09-11T09:51:51.185 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.185 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.185 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:51.185 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.958758+0000", 2024-09-11T09:51:51.186 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000120689 2024-09-11T09:51:51.186 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.186 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.186 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 3", 2024-09-11T09:51:51.186 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.958865+0000", 2024-09-11T09:51:51.186 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00010692 2024-09-11T09:51:51.186 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.186 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.186 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:51.186 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.970712+0000", 2024-09-11T09:51:51.186 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.011846327 2024-09-11T09:51:51.186 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.186 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.187 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:51.187 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.972603+0000", 2024-09-11T09:51:51.187 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0018918610000000001 2024-09-11T09:51:51.187 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.187 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.187 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:51.187 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.972653+0000", 2024-09-11T09:51:51.187 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.9762999999999998e-05 2024-09-11T09:51:51.187 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.187 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.187 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:51.187 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.972699+0000", 2024-09-11T09:51:51.187 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.5806999999999997e-05 2024-09-11T09:51:51.187 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:51.187 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:51.187 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:51.188 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.188 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.188 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:52 2.4 2:26d5c2f0:::rbd_data.1250f5e25b80.000000000000000b:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:51.188 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.958596+0000", 2024-09-11T09:51:51.188 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 28.189625658000001, 2024-09-11T09:51:51.188 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.016477849999999999, 2024-09-11T09:51:51.188 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:51.188 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:51.188 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:51.188 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:51.188 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:51:51.188 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:51:51.188 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 52 2024-09-11T09:51:51.188 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.189 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:51.189 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.189 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:51.189 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.958596+0000", 2024-09-11T09:51:51.189 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:51.189 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.189 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.189 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:51.189 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.958596+0000", 2024-09-11T09:51:51.189 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:51.189 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.189 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.189 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:51.189 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.958597+0000", 2024-09-11T09:51:51.189 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1960000000000001e-06 2024-09-11T09:51:51.190 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.190 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.190 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:51.190 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.963157+0000", 2024-09-11T09:51:51.190 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0045603029999999999 2024-09-11T09:51:51.190 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.190 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.190 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:51.190 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.963171+0000", 2024-09-11T09:51:51.190 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3777999999999999e-05 2024-09-11T09:51:51.190 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.190 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.190 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:51.190 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.963181+0000", 2024-09-11T09:51:51.190 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.9010000000000006e-06 2024-09-11T09:51:51.191 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.191 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.191 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:51.191 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.963246+0000", 2024-09-11T09:51:51.191 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.5351000000000005e-05 2024-09-11T09:51:51.191 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.191 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.191 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:51.191 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.963419+0000", 2024-09-11T09:51:51.191 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00017299700000000001 2024-09-11T09:51:51.191 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.191 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.191 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:51.191 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.963570+0000", 2024-09-11T09:51:51.191 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00015066999999999999 2024-09-11T09:51:51.192 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.192 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.192 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:51.192 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.973881+0000", 2024-09-11T09:51:51.192 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.010311268 2024-09-11T09:51:51.192 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.192 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.192 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:51.192 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.974963+0000", 2024-09-11T09:51:51.192 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.001081493 2024-09-11T09:51:51.192 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.192 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.192 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:51.193 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.975012+0000", 2024-09-11T09:51:51.193 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.9471000000000001e-05 2024-09-11T09:51:51.193 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.193 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.193 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:51.193 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.975074+0000", 2024-09-11T09:51:51.193 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.1421999999999996e-05 2024-09-11T09:51:51.193 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:51.193 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:51.193 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:51.193 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.193 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.194 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4688.0:47 2.1 e53/15 2:95f53b3e:::rbd_data.1250f5e25b80.0000000000000008:head v 53'3, mlcod=53'1)", 2024-09-11T09:51:51.194 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.960861+0000", 2024-09-11T09:51:51.194 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 28.18736024, 2024-09-11T09:51:51.194 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.012346107, 2024-09-11T09:51:51.194 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:51.194 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:51.194 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:51.194 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:51.194 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.194 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:51.194 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.960861+0000", 2024-09-11T09:51:51.194 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:51.194 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.194 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.195 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:51.195 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.960861+0000", 2024-09-11T09:51:51.195 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:51.195 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.195 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.195 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:51.195 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.960863+0000", 2024-09-11T09:51:51.195 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7150000000000001e-06 2024-09-11T09:51:51.195 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.195 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.195 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:51.195 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.962428+0000", 2024-09-11T09:51:51.195 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.001564668 2024-09-11T09:51:51.195 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.196 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.196 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:51.196 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.962446+0000", 2024-09-11T09:51:51.196 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.8476999999999999e-05 2024-09-11T09:51:51.196 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.196 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.196 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:51.196 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.962458+0000", 2024-09-11T09:51:51.196 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1263999999999999e-05 2024-09-11T09:51:51.196 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.196 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.196 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:51.197 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.962575+0000", 2024-09-11T09:51:51.197 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000117025 2024-09-11T09:51:51.197 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.197 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.197 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:51.197 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.962601+0000", 2024-09-11T09:51:51.197 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.5984999999999999e-05 2024-09-11T09:51:51.197 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.197 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.197 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:51.197 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.973065+0000", 2024-09-11T09:51:51.197 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.010464753 2024-09-11T09:51:51.197 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.198 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.198 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:51.198 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.973207+0000", 2024-09-11T09:51:51.198 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00014222 2024-09-11T09:51:51.198 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:51.198 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:51.198 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:51.198 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.198 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.198 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:54 2.4 2.da8979c4 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:51.198 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.963207+0000", 2024-09-11T09:51:51.198 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 28.185014802000001, 2024-09-11T09:51:51.198 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.010641575, 2024-09-11T09:51:51.199 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:51.199 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:51.199 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "reached pg", 2024-09-11T09:51:51.199 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:51.199 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:51:51.199 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:51:51.199 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 54 2024-09-11T09:51:51.199 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.199 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:51.199 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.199 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:51.199 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.963207+0000", 2024-09-11T09:51:51.199 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:51.199 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.199 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.200 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:51.200 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.963207+0000", 2024-09-11T09:51:51.200 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:51.200 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.200 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.200 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:51.200 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.963208+0000", 2024-09-11T09:51:51.200 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2979999999999999e-06 2024-09-11T09:51:51.200 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.200 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.200 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:51.200 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.967419+0000", 2024-09-11T09:51:51.200 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0042111750000000002 2024-09-11T09:51:51.200 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.201 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.201 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:51.201 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.967439+0000", 2024-09-11T09:51:51.201 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.0129999999999999e-05 2024-09-11T09:51:51.201 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.201 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.201 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:51.201 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.967453+0000", 2024-09-11T09:51:51.201 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3781000000000001e-05 2024-09-11T09:51:51.201 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.201 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.201 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:51.201 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.973840+0000", 2024-09-11T09:51:51.202 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0063872269999999997 2024-09-11T09:51:51.202 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.202 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.202 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:51.202 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.973848+0000", 2024-09-11T09:51:51.202 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.9640000000000003e-06 2024-09-11T09:51:51.202 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:51.202 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:51.202 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:51.202 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.202 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.202 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:54 2.4 2:239e915b:::rbd_data.1250f5e25b80.000000000000000c:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:51.202 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.982401+0000", 2024-09-11T09:51:51.202 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 28.165820875000001, 2024-09-11T09:51:51.203 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.016817347, 2024-09-11T09:51:51.203 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:51.203 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:51.203 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:51.203 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:51.203 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:51:51.203 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:51:51.203 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 54 2024-09-11T09:51:51.203 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.203 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:51.203 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.203 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:51.203 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.982401+0000", 2024-09-11T09:51:51.204 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:51.204 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.204 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.204 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:51.204 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.982401+0000", 2024-09-11T09:51:51.204 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:51.204 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.204 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.204 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:51.204 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.982402+0000", 2024-09-11T09:51:51.204 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.8999999999999995e-07 2024-09-11T09:51:51.204 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.204 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.205 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:51.205 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.986535+0000", 2024-09-11T09:51:51.205 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0041333689999999996 2024-09-11T09:51:51.205 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.205 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.205 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:51.205 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.986555+0000", 2024-09-11T09:51:51.205 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.9851e-05 2024-09-11T09:51:51.205 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.205 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.205 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:51.205 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.986569+0000", 2024-09-11T09:51:51.205 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3784000000000001e-05 2024-09-11T09:51:51.205 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.205 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.206 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:51.206 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.986683+0000", 2024-09-11T09:51:51.206 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000114474 2024-09-11T09:51:51.206 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.206 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.206 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:51.206 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.986815+0000", 2024-09-11T09:51:51.206 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00013217799999999999 2024-09-11T09:51:51.206 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.206 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.206 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:51.206 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.986918+0000", 2024-09-11T09:51:51.206 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00010235599999999999 2024-09-11T09:51:51.206 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.206 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.207 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:51.207 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.998983+0000", 2024-09-11T09:51:51.207 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.012065352 2024-09-11T09:51:51.207 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.207 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.207 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:51.207 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.999081+0000", 2024-09-11T09:51:51.207 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.8091000000000007e-05 2024-09-11T09:51:51.207 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.207 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.207 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:51.207 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.999159+0000", 2024-09-11T09:51:51.207 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.7823000000000004e-05 2024-09-11T09:51:51.207 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.208 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.208 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:51.208 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.999218+0000", 2024-09-11T09:51:51.208 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.9178999999999997e-05 2024-09-11T09:51:51.208 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:51.208 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:51.208 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:51.208 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.208 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.208 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:56 2.4 2:2167012f:::rbd_data.1250f5e25b80.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:51.208 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.986661+0000", 2024-09-11T09:51:51.208 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 28.161560422000001, 2024-09-11T09:51:51.208 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.017353539000000001, 2024-09-11T09:51:51.209 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:51.209 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:51.209 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:51.209 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:51.209 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:51:51.209 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:51:51.209 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 56 2024-09-11T09:51:51.209 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.209 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:51.209 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.209 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:51.209 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.986661+0000", 2024-09-11T09:51:51.209 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:51.209 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.210 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.210 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:51.210 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.986661+0000", 2024-09-11T09:51:51.210 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:51.210 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.210 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.210 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:51.210 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.986663+0000", 2024-09-11T09:51:51.210 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.8989999999999999e-06 2024-09-11T09:51:51.210 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.210 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.210 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:51.210 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.990444+0000", 2024-09-11T09:51:51.210 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0037808569999999999 2024-09-11T09:51:51.210 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.210 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.211 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:51.211 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.990462+0000", 2024-09-11T09:51:51.211 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.8450000000000001e-05 2024-09-11T09:51:51.211 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.211 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.211 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:51.211 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.990476+0000", 2024-09-11T09:51:51.211 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3565e-05 2024-09-11T09:51:51.211 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.211 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.211 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:51.211 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.993582+0000", 2024-09-11T09:51:51.211 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0031059109999999998 2024-09-11T09:51:51.211 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.211 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.212 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:51.212 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.993686+0000", 2024-09-11T09:51:51.212 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000104424 2024-09-11T09:51:51.212 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.212 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.212 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:51.212 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.993799+0000", 2024-09-11T09:51:51.212 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000112985 2024-09-11T09:51:51.212 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.212 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.212 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:51.212 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.002449+0000", 2024-09-11T09:51:51.212 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.008650151 2024-09-11T09:51:51.212 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.213 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.213 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:51.213 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.003948+0000", 2024-09-11T09:51:51.213 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.001498636 2024-09-11T09:51:51.213 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.213 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.213 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:51.213 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.003985+0000", 2024-09-11T09:51:51.213 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.7054999999999997e-05 2024-09-11T09:51:51.213 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.213 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.213 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:51.213 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.004015+0000", 2024-09-11T09:51:51.213 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.9606000000000001e-05 2024-09-11T09:51:51.214 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:51.214 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:51.214 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:51.214 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.214 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.214 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:58 2.6 2:6277c644:::rbd_data.1250f5e25b80.000000000000000e:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:51.214 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.990540+0000", 2024-09-11T09:51:51.214 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 28.157681626999999, 2024-09-11T09:51:51.214 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.016232068999999998, 2024-09-11T09:51:51.214 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:51.214 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:51.214 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:51.214 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:51.214 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:51:51.215 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:51:51.215 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 58 2024-09-11T09:51:51.215 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.215 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:51.215 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.215 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:51.215 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.990540+0000", 2024-09-11T09:51:51.215 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:51.215 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.215 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.215 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:51.215 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.990540+0000", 2024-09-11T09:51:51.215 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:51.215 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.215 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.216 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:51.216 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.990541+0000", 2024-09-11T09:51:51.216 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2359999999999999e-06 2024-09-11T09:51:51.216 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.216 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.216 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:51.216 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.994626+0000", 2024-09-11T09:51:51.216 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.004084865 2024-09-11T09:51:51.216 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.216 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.216 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:51.216 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.994644+0000", 2024-09-11T09:51:51.216 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7866000000000001e-05 2024-09-11T09:51:51.216 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.217 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.217 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:51.217 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.994653+0000", 2024-09-11T09:51:51.217 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.2159999999999995e-06 2024-09-11T09:51:51.217 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.217 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.217 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:51.217 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.994732+0000", 2024-09-11T09:51:51.217 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.8954000000000003e-05 2024-09-11T09:51:51.217 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.217 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.217 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:51.217 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.994906+0000", 2024-09-11T09:51:51.217 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0001741 2024-09-11T09:51:51.218 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.218 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.218 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 3", 2024-09-11T09:51:51.218 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.995083+0000", 2024-09-11T09:51:51.218 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00017637100000000001 2024-09-11T09:51:51.218 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.218 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.218 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:51.218 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.003066+0000", 2024-09-11T09:51:51.218 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0079833360000000006 2024-09-11T09:51:51.218 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.218 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.218 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:51.218 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.006634+0000", 2024-09-11T09:51:51.218 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.003568309 2024-09-11T09:51:51.219 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.219 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.219 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:51.219 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.006716+0000", 2024-09-11T09:51:51.219 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.1377000000000004e-05 2024-09-11T09:51:51.219 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.219 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.219 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:51.219 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.006772+0000", 2024-09-11T09:51:51.219 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.6439e-05 2024-09-11T09:51:51.219 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:51.219 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:51.219 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:51.219 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.220 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.220 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:59 2.4 2:2906e6a7:::rbd_header.1250f5e25b80:head [watch reconnect cookie 140424358411408 gen 2] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:51.220 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.994684+0000", 2024-09-11T09:51:51.220 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 28.153537996000001, 2024-09-11T09:51:51.220 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0094134349999999995, 2024-09-11T09:51:51.220 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:51.220 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:51.220 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "started", 2024-09-11T09:51:51.220 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:51.220 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:51:51.220 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:51:51.220 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 59 2024-09-11T09:51:51.220 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.220 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:51.221 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.221 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:51.221 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.994684+0000", 2024-09-11T09:51:51.221 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:51.221 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.221 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.221 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:51.221 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.994684+0000", 2024-09-11T09:51:51.221 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:51.221 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.221 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.221 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:51.221 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.994686+0000", 2024-09-11T09:51:51.222 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.8159999999999998e-06 2024-09-11T09:51:51.222 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.222 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.222 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:51.222 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.994691+0000", 2024-09-11T09:51:51.222 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.5009999999999998e-06 2024-09-11T09:51:51.222 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.222 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.222 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:51.222 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.994698+0000", 2024-09-11T09:51:51.222 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.3039999999999999e-06 2024-09-11T09:51:51.222 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.222 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.222 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:51.223 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.994703+0000", 2024-09-11T09:51:51.223 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.5190000000000003e-06 2024-09-11T09:51:51.223 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.223 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.223 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:51.223 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.998678+0000", 2024-09-11T09:51:51.223 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0039752340000000002 2024-09-11T09:51:51.223 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.223 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.223 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:51.223 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.998748+0000", 2024-09-11T09:51:51.223 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.0114e-05 2024-09-11T09:51:51.223 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.223 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.223 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:51.224 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.004097+0000", 2024-09-11T09:51:51.224 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0053489469999999997 2024-09-11T09:51:51.224 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:51.224 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:51.224 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:51.224 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.224 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.224 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4694.0:33 2.4 2:3bfc818c:ns::rbd_data.1256a0c02879.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:51.224 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.084537+0000", 2024-09-11T09:51:51.224 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 28.063684567999999, 2024-09-11T09:51:51.224 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.022075773999999999, 2024-09-11T09:51:51.224 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:51.224 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:51.224 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:51.225 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:51.225 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4694", 2024-09-11T09:51:51.225 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:51:51.225 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 33 2024-09-11T09:51:51.225 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.225 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:51.225 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.225 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:51.225 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.084537+0000", 2024-09-11T09:51:51.225 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:51.225 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.225 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.225 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:51.225 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.084537+0000", 2024-09-11T09:51:51.226 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:51.226 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.226 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.226 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:51.226 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.084538+0000", 2024-09-11T09:51:51.226 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.5000000000000001e-07 2024-09-11T09:51:51.226 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.226 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.226 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:51.226 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.095984+0000", 2024-09-11T09:51:51.226 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.011446543 2024-09-11T09:51:51.226 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.226 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.226 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:51.226 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.096014+0000", 2024-09-11T09:51:51.227 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.9745000000000001e-05 2024-09-11T09:51:51.227 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.227 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.227 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:51.227 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.096030+0000", 2024-09-11T09:51:51.227 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6152999999999999e-05 2024-09-11T09:51:51.227 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.227 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.227 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:51.227 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.096113+0000", 2024-09-11T09:51:51.227 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.2731999999999993e-05 2024-09-11T09:51:51.227 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.227 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.227 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:51.227 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.096273+0000", 2024-09-11T09:51:51.228 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00015973699999999999 2024-09-11T09:51:51.228 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.228 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.228 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:51.228 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.096394+0000", 2024-09-11T09:51:51.228 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000120883 2024-09-11T09:51:51.228 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.228 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.228 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:51.228 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.106048+0000", 2024-09-11T09:51:51.228 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0096548020000000005 2024-09-11T09:51:51.228 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.228 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.228 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:51.229 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.106531+0000", 2024-09-11T09:51:51.229 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00048264299999999999 2024-09-11T09:51:51.229 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.229 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.229 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:51.229 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.106565+0000", 2024-09-11T09:51:51.229 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.4343e-05 2024-09-11T09:51:51.229 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.229 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.229 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:51.229 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.106613+0000", 2024-09-11T09:51:51.229 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.7342999999999997e-05 2024-09-11T09:51:51.230 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:51.230 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:51.230 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:51.230 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.230 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.230 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4694.0:32 2.1 e53/15 2:8a862b48:ns::rbd_data.1256a0c02879.0000000000000000:head v 53'8, mlcod=53'6)", 2024-09-11T09:51:51.230 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.091418+0000", 2024-09-11T09:51:51.230 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 28.056803399, 2024-09-11T09:51:51.230 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.013027372000000001, 2024-09-11T09:51:51.230 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:51.230 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:51.230 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:51.230 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:51.230 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.230 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:51.231 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.091418+0000", 2024-09-11T09:51:51.231 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:51.231 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.231 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.231 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:51.231 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.091418+0000", 2024-09-11T09:51:51.231 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:51.231 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.231 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.231 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:51.231 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.091420+0000", 2024-09-11T09:51:51.231 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6899999999999999e-06 2024-09-11T09:51:51.231 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.231 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.232 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:51.232 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.093086+0000", 2024-09-11T09:51:51.232 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0016662510000000001 2024-09-11T09:51:51.232 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.232 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.232 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:51.232 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.093107+0000", 2024-09-11T09:51:51.232 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.0999e-05 2024-09-11T09:51:51.232 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.232 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.232 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:51.232 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.093120+0000", 2024-09-11T09:51:51.232 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2544999999999999e-05 2024-09-11T09:51:51.232 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.233 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.233 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:51.233 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.093186+0000", 2024-09-11T09:51:51.233 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.6595999999999997e-05 2024-09-11T09:51:51.233 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.233 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.233 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:51.233 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.093208+0000", 2024-09-11T09:51:51.233 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.1624000000000002e-05 2024-09-11T09:51:51.233 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.233 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.233 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:51.233 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.104392+0000", 2024-09-11T09:51:51.234 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.011184265000000001 2024-09-11T09:51:51.234 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.234 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.234 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:51.234 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.104446+0000", 2024-09-11T09:51:51.234 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.3402000000000001e-05 2024-09-11T09:51:51.234 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:51.234 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:51.234 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:51.234 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.234 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.234 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4694.0:45 2.6 2:671fb830:ns::rbd_data.1256a0c02879.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:51.234 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.109629+0000", 2024-09-11T09:51:51.235 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 28.038592558000001, 2024-09-11T09:51:51.235 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.036403143999999998, 2024-09-11T09:51:51.235 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:51.235 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:51.235 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:51.235 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:51.235 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4694", 2024-09-11T09:51:51.235 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:51:51.235 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 45 2024-09-11T09:51:51.235 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.235 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:51.235 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.235 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:51.235 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.109629+0000", 2024-09-11T09:51:51.236 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:51.236 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.236 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.236 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:51.236 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.109629+0000", 2024-09-11T09:51:51.236 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:51.236 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.236 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.236 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:51.236 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.109630+0000", 2024-09-11T09:51:51.236 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1200000000000001e-06 2024-09-11T09:51:51.236 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.236 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.236 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:51.236 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.121370+0000", 2024-09-11T09:51:51.237 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.011739571000000001 2024-09-11T09:51:51.237 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.237 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.237 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:51.237 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.121391+0000", 2024-09-11T09:51:51.237 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.1175e-05 2024-09-11T09:51:51.237 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.237 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.237 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:51.237 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.121404+0000", 2024-09-11T09:51:51.237 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2807e-05 2024-09-11T09:51:51.237 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.237 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.237 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:51.238 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.121498+0000", 2024-09-11T09:51:51.238 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.3770999999999994e-05 2024-09-11T09:51:51.238 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.238 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.238 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:51.238 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.121612+0000", 2024-09-11T09:51:51.238 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000114666 2024-09-11T09:51:51.238 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.238 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.238 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 3", 2024-09-11T09:51:51.238 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.121711+0000", 2024-09-11T09:51:51.238 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.8647999999999995e-05 2024-09-11T09:51:51.238 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.238 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.238 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:51.239 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.126558+0000", 2024-09-11T09:51:51.239 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0048473769999999999 2024-09-11T09:51:51.239 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.239 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.239 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:51.239 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.145912+0000", 2024-09-11T09:51:51.239 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.019354281000000001 2024-09-11T09:51:51.239 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.239 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.239 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:51.239 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.145977+0000", 2024-09-11T09:51:51.239 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.4135999999999997e-05 2024-09-11T09:51:51.239 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.240 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.240 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:51.240 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.146032+0000", 2024-09-11T09:51:51.240 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.5591999999999999e-05 2024-09-11T09:51:51.240 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:51.240 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:51.240 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:51.240 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.240 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.240 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4694.0:41 2.2 e53/15 2:5e2505c4:ns::rbd_data.1256a0c02879.0000000000000003:head v 53'4, mlcod=18'2)", 2024-09-11T09:51:51.240 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.113199+0000", 2024-09-11T09:51:51.240 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 28.035022147999999, 2024-09-11T09:51:51.240 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.011404512, 2024-09-11T09:51:51.241 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:51.241 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:51.241 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:51.241 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:51.241 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.241 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:51.241 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.113199+0000", 2024-09-11T09:51:51.241 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:51.241 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.241 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.241 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:51.241 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.113199+0000", 2024-09-11T09:51:51.241 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:51.242 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.242 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.242 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:51.242 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.113200+0000", 2024-09-11T09:51:51.242 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.2799999999999995e-07 2024-09-11T09:51:51.242 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.242 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.242 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:51.242 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.118872+0000", 2024-09-11T09:51:51.242 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0056718089999999999 2024-09-11T09:51:51.242 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.242 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.242 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:51.242 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.118889+0000", 2024-09-11T09:51:51.243 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7428999999999999e-05 2024-09-11T09:51:51.243 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.243 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.243 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:51.243 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.118900+0000", 2024-09-11T09:51:51.243 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0243e-05 2024-09-11T09:51:51.243 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.243 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.243 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:51.243 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.118959+0000", 2024-09-11T09:51:51.243 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.9364000000000003e-05 2024-09-11T09:51:51.243 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.243 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.243 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:51.244 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.118976+0000", 2024-09-11T09:51:51.244 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7124000000000001e-05 2024-09-11T09:51:51.244 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.244 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.244 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:51.244 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.124557+0000", 2024-09-11T09:51:51.244 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0055812659999999997 2024-09-11T09:51:51.244 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.244 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.244 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:51.244 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.124604+0000", 2024-09-11T09:51:51.244 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.6548999999999998e-05 2024-09-11T09:51:51.244 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:51.245 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:51.245 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:51.245 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.245 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.245 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4694.0:43 2.0 e53/15 2:16c8ae84:ns::rbd_data.1256a0c02879.0000000000000005:head v 53'2, mlcod=0'0)", 2024-09-11T09:51:51.245 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.158559+0000", 2024-09-11T09:51:51.245 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 27.989662822, 2024-09-11T09:51:51.245 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.016713677999999999, 2024-09-11T09:51:51.245 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:51.245 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:51.245 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:51.245 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:51.246 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.246 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:51.246 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.158559+0000", 2024-09-11T09:51:51.246 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:51.246 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.246 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.246 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:51.246 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.158559+0000", 2024-09-11T09:51:51.246 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:51.246 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.246 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.246 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:51.246 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.158560+0000", 2024-09-11T09:51:51.246 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1459999999999999e-06 2024-09-11T09:51:51.247 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.247 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.247 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:51.247 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.166465+0000", 2024-09-11T09:51:51.247 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.007904767 2024-09-11T09:51:51.247 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.247 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.247 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:51.247 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.166491+0000", 2024-09-11T09:51:51.247 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.6308e-05 2024-09-11T09:51:51.247 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.247 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.247 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:51.247 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.166502+0000", 2024-09-11T09:51:51.247 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0736e-05 2024-09-11T09:51:51.248 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.248 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.248 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:51.248 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.166564+0000", 2024-09-11T09:51:51.248 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.2578999999999998e-05 2024-09-11T09:51:51.248 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.248 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.248 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:51.248 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.166585+0000", 2024-09-11T09:51:51.248 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.0395999999999999e-05 2024-09-11T09:51:51.248 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.248 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.248 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:51.248 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.175227+0000", 2024-09-11T09:51:51.249 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0086426420000000007 2024-09-11T09:51:51.249 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.249 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.249 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:51.249 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.175272+0000", 2024-09-11T09:51:51.249 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.5104e-05 2024-09-11T09:51:51.249 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:51.249 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:51.249 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:51.249 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.249 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.249 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4694.0:59 2.2 e53/15 2:41adefc9:ns::rbd_data.1256a0c02879.000000000000000e:head v 53'5, mlcod=53'3)", 2024-09-11T09:51:51.250 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.163389+0000", 2024-09-11T09:51:51.250 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 27.984832846, 2024-09-11T09:51:51.250 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.013095719, 2024-09-11T09:51:51.250 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:51.250 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:51.250 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:51.250 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:51.250 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.250 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:51.250 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.163389+0000", 2024-09-11T09:51:51.250 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:51.250 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.250 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.251 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:51.251 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.163389+0000", 2024-09-11T09:51:51.251 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:51.251 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.251 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.251 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:51.251 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.163390+0000", 2024-09-11T09:51:51.251 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.336e-06 2024-09-11T09:51:51.251 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.251 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.251 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:51.251 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.170335+0000", 2024-09-11T09:51:51.251 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0069451080000000002 2024-09-11T09:51:51.252 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.252 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.252 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:51.252 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.170354+0000", 2024-09-11T09:51:51.252 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.8423e-05 2024-09-11T09:51:51.252 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.252 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.252 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:51.252 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.170366+0000", 2024-09-11T09:51:51.252 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2235999999999999e-05 2024-09-11T09:51:51.252 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.252 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.252 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:51.253 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.170439+0000", 2024-09-11T09:51:51.253 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.2989000000000002e-05 2024-09-11T09:51:51.253 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.253 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.253 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:51.253 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.170460+0000", 2024-09-11T09:51:51.253 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.1211000000000001e-05 2024-09-11T09:51:51.253 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.253 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.253 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:51.253 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.176442+0000", 2024-09-11T09:51:51.253 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0059821969999999999 2024-09-11T09:51:51.253 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.253 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.254 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:51.254 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.176484+0000", 2024-09-11T09:51:51.254 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.2218999999999997e-05 2024-09-11T09:51:51.254 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:51.254 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:51.254 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:51.254 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:51.254 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:51.254 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:51:51.256 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-11T09:51:51.256 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:51:51.256+0000 7f3c96ec9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31909) UID: 0 2024-09-11T09:51:51.356 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:51:51.357+0000 7f3a7b90a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 42475) UID: 0 2024-09-11T09:51:51.404 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:51:51.404 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [], 2024-09-11T09:51:51.404 INFO:teuthology.orchestra.run.smithi017.stdout: "num_ops": 0 2024-09-11T09:51:51.404 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:51:51.412 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-11T09:51:51.457 INFO:tasks.ceph.osd.0.smithi017.stderr:2024-09-11T09:51:51.457+0000 7fb73c0f1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31665) UID: 0 2024-09-11T09:51:51.519 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:51:51.519 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [], 2024-09-11T09:51:51.519 INFO:teuthology.orchestra.run.smithi017.stdout: "complaint_time": 30, 2024-09-11T09:51:51.519 INFO:teuthology.orchestra.run.smithi017.stdout: "num_blocked_ops": 0 2024-09-11T09:51:51.520 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:51:51.528 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-11T09:51:51.558 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:51:51.558+0000 7f3c96ec9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31909) UID: 0 2024-09-11T09:51:51.659 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:51:51.660+0000 7f3a7b90a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 42475) UID: 0 2024-09-11T09:51:51.677 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:51:51.677 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 20, 2024-09-11T09:51:51.677 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 600, 2024-09-11T09:51:51.677 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [ 2024-09-11T09:51:51.677 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.677 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:31 2.1 2:8654721b:::rbd_data.1250f5e25b80.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:51.677 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.896024+0000", 2024-09-11T09:51:51.677 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 28.780880329999999, 2024-09-11T09:51:51.677 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.023148401999999998, 2024-09-11T09:51:51.678 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:51.678 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:51.678 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:51.678 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:51.678 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:51:51.678 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:51:51.678 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 31 2024-09-11T09:51:51.678 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.678 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:51.678 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.678 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:51.678 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.896024+0000", 2024-09-11T09:51:51.678 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:51.679 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.679 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.679 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:51.679 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.896024+0000", 2024-09-11T09:51:51.679 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:51.679 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.679 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.679 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:51.679 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.896025+0000", 2024-09-11T09:51:51.679 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.54e-07 2024-09-11T09:51:51.679 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.679 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.679 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:51.679 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.904287+0000", 2024-09-11T09:51:51.680 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0082624729999999993 2024-09-11T09:51:51.680 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.680 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.680 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:51.680 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.904305+0000", 2024-09-11T09:51:51.680 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7912999999999999e-05 2024-09-11T09:51:51.680 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.680 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.680 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:51.680 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.904312+0000", 2024-09-11T09:51:51.680 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.0230000000000004e-06 2024-09-11T09:51:51.680 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.680 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.681 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:51.681 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.904395+0000", 2024-09-11T09:51:51.681 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.2355999999999996e-05 2024-09-11T09:51:51.681 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.681 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.681 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:51.681 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.904499+0000", 2024-09-11T09:51:51.681 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000104672 2024-09-11T09:51:51.681 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.681 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.681 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1", 2024-09-11T09:51:51.681 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.904594+0000", 2024-09-11T09:51:51.681 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.5129000000000003e-05 2024-09-11T09:51:51.682 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.682 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.682 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:51.682 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.918183+0000", 2024-09-11T09:51:51.682 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.013588355999999999 2024-09-11T09:51:51.682 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.682 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.682 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:51.682 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919053+0000", 2024-09-11T09:51:51.682 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00087066800000000001 2024-09-11T09:51:51.682 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.682 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.682 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:51.683 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919125+0000", 2024-09-11T09:51:51.683 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.1333000000000003e-05 2024-09-11T09:51:51.683 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.683 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.683 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:51.683 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919172+0000", 2024-09-11T09:51:51.683 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.7725000000000001e-05 2024-09-11T09:51:51.683 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:51.683 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:51.683 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:51.683 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.683 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.683 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4688.0:42 2.3 e53/15 2:c54742d8:::rbd_data.1250f5e25b80.0000000000000006:head v 53'2, mlcod=0'0)", 2024-09-11T09:51:51.683 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.939272+0000", 2024-09-11T09:51:51.684 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 28.737632346000002, 2024-09-11T09:51:51.684 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.010821958, 2024-09-11T09:51:51.684 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:51.684 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:51.684 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:51.684 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:51.684 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.684 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:51.684 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.939272+0000", 2024-09-11T09:51:51.684 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:51.684 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.684 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.684 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:51.685 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.939272+0000", 2024-09-11T09:51:51.685 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:51.685 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.685 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.685 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:51.685 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.939273+0000", 2024-09-11T09:51:51.685 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.7899999999999997e-07 2024-09-11T09:51:51.685 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.685 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.685 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:51.685 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.944403+0000", 2024-09-11T09:51:51.685 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0051302129999999998 2024-09-11T09:51:51.685 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.685 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.686 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:51.686 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.944413+0000", 2024-09-11T09:51:51.686 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0182e-05 2024-09-11T09:51:51.686 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.686 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.686 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:51.686 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.944420+0000", 2024-09-11T09:51:51.686 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.7000000000000002e-06 2024-09-11T09:51:51.686 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.686 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.686 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:51.686 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.944482+0000", 2024-09-11T09:51:51.686 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.1911999999999999e-05 2024-09-11T09:51:51.687 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.687 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.687 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:51.687 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.944508+0000", 2024-09-11T09:51:51.687 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.6582000000000001e-05 2024-09-11T09:51:51.687 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.687 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.687 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:51.687 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.950050+0000", 2024-09-11T09:51:51.687 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0055414080000000003 2024-09-11T09:51:51.687 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.687 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.687 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:51.687 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.950094+0000", 2024-09-11T09:51:51.688 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.4082000000000002e-05 2024-09-11T09:51:51.688 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:51.688 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:51.688 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:51.688 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.688 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.688 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:47 2.1 2:95f53b3e:::rbd_data.1250f5e25b80.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:51.688 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.948330+0000", 2024-09-11T09:51:51.688 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 28.728574001999998, 2024-09-11T09:51:51.688 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.025156603999999999, 2024-09-11T09:51:51.688 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:51.688 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:51.688 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:51.689 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:51.689 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:51:51.689 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:51:51.689 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 47 2024-09-11T09:51:51.689 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.689 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:51.689 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.689 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:51.689 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.948330+0000", 2024-09-11T09:51:51.689 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:51.689 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.689 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.689 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:51.689 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.948330+0000", 2024-09-11T09:51:51.690 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:51.690 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.690 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.690 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:51.690 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.948333+0000", 2024-09-11T09:51:51.690 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.481e-06 2024-09-11T09:51:51.690 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.690 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.690 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:51.690 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.960263+0000", 2024-09-11T09:51:51.690 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.011930671 2024-09-11T09:51:51.690 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.690 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.691 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:51.691 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.960289+0000", 2024-09-11T09:51:51.691 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.5573000000000001e-05 2024-09-11T09:51:51.691 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.691 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.691 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:51.691 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.960304+0000", 2024-09-11T09:51:51.691 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.4647000000000001e-05 2024-09-11T09:51:51.691 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.691 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.691 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:51.691 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.960390+0000", 2024-09-11T09:51:51.691 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.6088999999999995e-05 2024-09-11T09:51:51.692 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.692 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.692 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:51.692 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.960562+0000", 2024-09-11T09:51:51.692 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00017237400000000001 2024-09-11T09:51:51.692 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.692 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.692 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1", 2024-09-11T09:51:51.692 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.960725+0000", 2024-09-11T09:51:51.692 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00016321599999999999 2024-09-11T09:51:51.692 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.692 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.692 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:51.692 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.969465+0000", 2024-09-11T09:51:51.693 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0087396899999999996 2024-09-11T09:51:51.693 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.693 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.693 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:51.693 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.973393+0000", 2024-09-11T09:51:51.693 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0039274669999999996 2024-09-11T09:51:51.693 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.693 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.693 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:51.693 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.973443+0000", 2024-09-11T09:51:51.693 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.0340000000000003e-05 2024-09-11T09:51:51.693 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.693 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.694 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:51.694 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.973487+0000", 2024-09-11T09:51:51.694 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.4056e-05 2024-09-11T09:51:51.694 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:51.694 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:51.694 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:51.694 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.694 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.694 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:61 2.1 2:9af038b1:::rbd_data.1250f5e25b80.000000000000000f:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:51.694 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.975764+0000", 2024-09-11T09:51:51.694 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 28.701140477999999, 2024-09-11T09:51:51.694 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.016158891000000002, 2024-09-11T09:51:51.694 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:51.695 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:51.695 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:51.695 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:51.695 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:51:51.695 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:51:51.695 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 61 2024-09-11T09:51:51.695 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.695 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:51.695 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.695 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:51.695 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.975764+0000", 2024-09-11T09:51:51.695 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:51.695 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.695 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.696 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:51.696 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.975764+0000", 2024-09-11T09:51:51.696 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:51.696 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.696 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.696 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:51.696 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.975766+0000", 2024-09-11T09:51:51.696 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.767e-06 2024-09-11T09:51:51.696 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.696 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.696 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:51.696 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.983204+0000", 2024-09-11T09:51:51.697 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0074388400000000004 2024-09-11T09:51:51.697 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.697 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.697 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:51.697 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.983226+0000", 2024-09-11T09:51:51.697 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.1279000000000001e-05 2024-09-11T09:51:51.697 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.697 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.697 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:51.697 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.983238+0000", 2024-09-11T09:51:51.697 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2563e-05 2024-09-11T09:51:51.697 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.697 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.698 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:51.698 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.983318+0000", 2024-09-11T09:51:51.698 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.0075999999999997e-05 2024-09-11T09:51:51.698 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.698 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.698 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:51.698 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.983429+0000", 2024-09-11T09:51:51.698 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000110618 2024-09-11T09:51:51.698 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.698 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.698 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1", 2024-09-11T09:51:51.698 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.983525+0000", 2024-09-11T09:51:51.698 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.603e-05 2024-09-11T09:51:51.698 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.699 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.699 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:51.699 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.989004+0000", 2024-09-11T09:51:51.699 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0054785429999999998 2024-09-11T09:51:51.699 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.699 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.699 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:51.699 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.991788+0000", 2024-09-11T09:51:51.699 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0027845729999999998 2024-09-11T09:51:51.699 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.699 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.699 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:51.699 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.991870+0000", 2024-09-11T09:51:51.699 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.1567e-05 2024-09-11T09:51:51.700 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.700 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.700 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:51.700 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.991923+0000", 2024-09-11T09:51:51.700 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.3035000000000003e-05 2024-09-11T09:51:51.700 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:51.700 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:51.700 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:51.700 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.700 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.700 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4694.0:32 2.1 2:8a862b48:ns::rbd_data.1256a0c02879.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:51.700 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.084089+0000", 2024-09-11T09:51:51.700 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 28.592815003999998, 2024-09-11T09:51:51.700 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.020630553999999999, 2024-09-11T09:51:51.701 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:51.701 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:51.701 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:51.701 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:51.701 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4694", 2024-09-11T09:51:51.701 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:51:51.701 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 32 2024-09-11T09:51:51.701 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.701 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:51.701 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.701 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:51.701 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.084089+0000", 2024-09-11T09:51:51.701 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:51.702 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.702 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.702 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:51.702 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.084089+0000", 2024-09-11T09:51:51.702 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:51.702 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.702 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.702 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:51.702 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.084090+0000", 2024-09-11T09:51:51.702 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.2900000000000003e-07 2024-09-11T09:51:51.702 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.702 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.702 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:51.702 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.090866+0000", 2024-09-11T09:51:51.703 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0067756609999999997 2024-09-11T09:51:51.703 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.703 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.703 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:51.703 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.090891+0000", 2024-09-11T09:51:51.703 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.5119000000000001e-05 2024-09-11T09:51:51.703 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.703 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.703 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:51.703 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.090900+0000", 2024-09-11T09:51:51.703 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.4110000000000002e-06 2024-09-11T09:51:51.703 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.703 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.703 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:51.704 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.090968+0000", 2024-09-11T09:51:51.704 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.8041999999999997e-05 2024-09-11T09:51:51.704 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.704 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.704 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:51.704 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.091115+0000", 2024-09-11T09:51:51.704 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00014633499999999999 2024-09-11T09:51:51.704 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.704 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.704 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1", 2024-09-11T09:51:51.704 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.091284+0000", 2024-09-11T09:51:51.704 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000169836 2024-09-11T09:51:51.704 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.704 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.705 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:51.705 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.100896+0000", 2024-09-11T09:51:51.705 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0096118639999999995 2024-09-11T09:51:51.705 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.705 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.705 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:51.705 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.104604+0000", 2024-09-11T09:51:51.705 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0037074619999999999 2024-09-11T09:51:51.705 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.705 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.705 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:51.705 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.104669+0000", 2024-09-11T09:51:51.705 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.4836000000000001e-05 2024-09-11T09:51:51.706 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.706 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.706 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:51.706 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.104720+0000", 2024-09-11T09:51:51.706 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.1258999999999998e-05 2024-09-11T09:51:51.706 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:51.706 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:51.706 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:51.706 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.706 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.706 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4694.0:44 2.3 e53/15 2:cd1a7d1c:ns::rbd_data.1256a0c02879.0000000000000006:head v 53'3, mlcod=16'1)", 2024-09-11T09:51:51.706 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.120387+0000", 2024-09-11T09:51:51.706 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 28.556517351, 2024-09-11T09:51:51.707 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.011988468, 2024-09-11T09:51:51.707 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:51.707 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:51.707 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:51.707 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:51.707 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.707 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:51.707 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.120387+0000", 2024-09-11T09:51:51.707 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:51.707 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.707 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.707 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:51.707 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.120387+0000", 2024-09-11T09:51:51.708 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:51.708 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.708 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.708 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:51.708 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.120389+0000", 2024-09-11T09:51:51.708 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.018e-06 2024-09-11T09:51:51.708 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.708 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.708 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:51.708 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.127403+0000", 2024-09-11T09:51:51.708 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0070144769999999999 2024-09-11T09:51:51.708 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.708 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.708 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:51.709 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.127414+0000", 2024-09-11T09:51:51.709 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0991e-05 2024-09-11T09:51:51.709 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.709 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.709 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:51.709 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.127423+0000", 2024-09-11T09:51:51.709 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.8580000000000006e-06 2024-09-11T09:51:51.709 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.709 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.709 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:51.709 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.127468+0000", 2024-09-11T09:51:51.709 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.4830999999999998e-05 2024-09-11T09:51:51.709 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.710 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.710 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:51.710 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.127485+0000", 2024-09-11T09:51:51.710 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6838999999999999e-05 2024-09-11T09:51:51.710 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.710 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.710 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:51.710 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.132341+0000", 2024-09-11T09:51:51.710 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0048557959999999999 2024-09-11T09:51:51.710 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.710 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.710 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:51.710 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.132375+0000", 2024-09-11T09:51:51.711 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.4657999999999999e-05 2024-09-11T09:51:51.711 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:51.711 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:51.711 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:51.711 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.711 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.711 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4694.0:46 2.3 e53/15 2:ce0e1469:ns::rbd_data.1256a0c02879.0000000000000008:head v 53'4, mlcod=16'1)", 2024-09-11T09:51:51.711 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.129008+0000", 2024-09-11T09:51:51.711 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 28.547896765000001, 2024-09-11T09:51:51.711 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0087151220000000005, 2024-09-11T09:51:51.711 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:51.711 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:51.711 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:51.712 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:51.712 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.712 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:51.712 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.129008+0000", 2024-09-11T09:51:51.712 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:51.712 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.712 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.712 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:51.712 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.129008+0000", 2024-09-11T09:51:51.712 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:51.712 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.712 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.713 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:51.713 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.129009+0000", 2024-09-11T09:51:51.713 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.367e-06 2024-09-11T09:51:51.713 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.713 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.713 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:51.713 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.132839+0000", 2024-09-11T09:51:51.713 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0038297610000000001 2024-09-11T09:51:51.713 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.713 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.713 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:51.713 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.132849+0000", 2024-09-11T09:51:51.713 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0672e-05 2024-09-11T09:51:51.714 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.714 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.714 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:51.714 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.132856+0000", 2024-09-11T09:51:51.714 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.1380000000000004e-06 2024-09-11T09:51:51.714 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.714 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.714 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:51.714 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.132891+0000", 2024-09-11T09:51:51.714 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.5712e-05 2024-09-11T09:51:51.714 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.714 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.714 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:51.714 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.132907+0000", 2024-09-11T09:51:51.715 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6166000000000001e-05 2024-09-11T09:51:51.715 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.715 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.715 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:51.715 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.137681+0000", 2024-09-11T09:51:51.715 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0047735299999999998 2024-09-11T09:51:51.715 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.715 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.716 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:51.716 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.137723+0000", 2024-09-11T09:51:51.716 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.1776000000000001e-05 2024-09-11T09:51:51.716 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:51.716 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:51.716 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:51.716 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.716 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.716 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4694.0:48 2.3 e53/15 2:d2090b70:ns::rbd_data.1256a0c02879.0000000000000009:head v 53'5, mlcod=53'2)", 2024-09-11T09:51:51.716 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.135108+0000", 2024-09-11T09:51:51.716 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 28.541795868000001, 2024-09-11T09:51:51.716 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.010710679000000001, 2024-09-11T09:51:51.716 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:51.717 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:51.717 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:51.717 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:51.717 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.717 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:51.717 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.135108+0000", 2024-09-11T09:51:51.717 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:51.717 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.717 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.717 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:51.717 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.135108+0000", 2024-09-11T09:51:51.717 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:51.717 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.718 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.718 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:51.718 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.135110+0000", 2024-09-11T09:51:51.718 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.122e-06 2024-09-11T09:51:51.718 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.718 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.718 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:51.718 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.140876+0000", 2024-09-11T09:51:51.718 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.005766162 2024-09-11T09:51:51.718 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.718 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.718 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:51.719 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.140888+0000", 2024-09-11T09:51:51.719 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.201e-05 2024-09-11T09:51:51.719 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.719 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.719 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:51.719 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.140894+0000", 2024-09-11T09:51:51.719 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.1739999999999997e-06 2024-09-11T09:51:51.719 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.719 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.719 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:51.719 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.140971+0000", 2024-09-11T09:51:51.719 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.7495000000000003e-05 2024-09-11T09:51:51.720 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.720 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.720 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:51.720 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.140995+0000", 2024-09-11T09:51:51.720 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.3204999999999999e-05 2024-09-11T09:51:51.720 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.720 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.720 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:51.720 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.145757+0000", 2024-09-11T09:51:51.720 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0047627190000000003 2024-09-11T09:51:51.720 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.720 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.720 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:51.721 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.145819+0000", 2024-09-11T09:51:51.721 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.1791999999999994e-05 2024-09-11T09:51:51.721 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:51.721 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:51.721 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:51.721 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.721 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.721 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4694.0:50 2.1 2:83078786:ns::rbd_data.1256a0c02879.000000000000000a:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:51.721 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.141467+0000", 2024-09-11T09:51:51.721 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 28.535437143999999, 2024-09-11T09:51:51.721 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.012785889, 2024-09-11T09:51:51.721 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:51.721 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:51.722 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:51.722 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:51.722 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4694", 2024-09-11T09:51:51.722 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:51:51.722 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 50 2024-09-11T09:51:51.722 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.722 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:51.722 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.722 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:51.722 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.141467+0000", 2024-09-11T09:51:51.722 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:51.722 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.722 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.722 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:51.723 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.141467+0000", 2024-09-11T09:51:51.723 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:51.723 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.723 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.723 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:51.723 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.141468+0000", 2024-09-11T09:51:51.723 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1009999999999999e-06 2024-09-11T09:51:51.723 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.723 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.723 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:51.723 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.146432+0000", 2024-09-11T09:51:51.723 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0049640539999999999 2024-09-11T09:51:51.723 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.723 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.724 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:51.724 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.146446+0000", 2024-09-11T09:51:51.724 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3254e-05 2024-09-11T09:51:51.724 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.724 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.724 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:51.724 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.146452+0000", 2024-09-11T09:51:51.724 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.207e-06 2024-09-11T09:51:51.724 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.724 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.724 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:51.724 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.146510+0000", 2024-09-11T09:51:51.724 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.8322999999999997e-05 2024-09-11T09:51:51.724 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.724 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.724 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:51.725 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.146650+0000", 2024-09-11T09:51:51.725 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00013947100000000001 2024-09-11T09:51:51.725 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.725 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.725 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1", 2024-09-11T09:51:51.725 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.146779+0000", 2024-09-11T09:51:51.725 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000129059 2024-09-11T09:51:51.725 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.725 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.725 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:51.725 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.153778+0000", 2024-09-11T09:51:51.725 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0069997130000000003 2024-09-11T09:51:51.725 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.725 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.726 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:51.726 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.154180+0000", 2024-09-11T09:51:51.726 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00040159800000000003 2024-09-11T09:51:51.726 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.726 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.726 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:51.726 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.154218+0000", 2024-09-11T09:51:51.726 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.7795e-05 2024-09-11T09:51:51.726 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.726 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.726 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:51.726 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.154253+0000", 2024-09-11T09:51:51.726 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.5314000000000001e-05 2024-09-11T09:51:51.726 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:51.726 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:51.727 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:51.727 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.727 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.727 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4694.0:54 2.3 e53/15 2:c44f0253:ns::rbd_data.1256a0c02879.000000000000000b:head v 53'6, mlcod=53'4)", 2024-09-11T09:51:51.727 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.154926+0000", 2024-09-11T09:51:51.727 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 28.521978769, 2024-09-11T09:51:51.727 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0089497099999999996, 2024-09-11T09:51:51.727 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:51.727 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:51.727 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:51.727 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:51.727 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.727 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:51.727 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.154926+0000", 2024-09-11T09:51:51.727 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:51.727 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.728 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.728 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:51.728 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.154926+0000", 2024-09-11T09:51:51.728 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:51.728 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.728 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.728 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:51.728 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.154927+0000", 2024-09-11T09:51:51.728 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.7000000000000003e-07 2024-09-11T09:51:51.728 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.728 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.728 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:51.728 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.158712+0000", 2024-09-11T09:51:51.728 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0037855469999999998 2024-09-11T09:51:51.728 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.729 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.729 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:51.729 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.158726+0000", 2024-09-11T09:51:51.729 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3621000000000001e-05 2024-09-11T09:51:51.729 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.729 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.729 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:51.729 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.158732+0000", 2024-09-11T09:51:51.729 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.9089999999999997e-06 2024-09-11T09:51:51.729 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.729 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.729 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:51.729 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.158781+0000", 2024-09-11T09:51:51.729 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.8900000000000003e-05 2024-09-11T09:51:51.729 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.730 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.730 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:51.730 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.158805+0000", 2024-09-11T09:51:51.730 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.44e-05 2024-09-11T09:51:51.730 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.730 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.730 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:51.730 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.163826+0000", 2024-09-11T09:51:51.730 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0050206399999999998 2024-09-11T09:51:51.730 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.730 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.731 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:51.731 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.163875+0000", 2024-09-11T09:51:51.731 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.9722999999999998e-05 2024-09-11T09:51:51.731 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:51.731 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:51.731 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:51.731 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.731 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.731 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4694.0:56 2.3 e53/15 2:c462f029:ns::rbd_data.1256a0c02879.000000000000000c:head v 53'7, mlcod=53'4)", 2024-09-11T09:51:51.731 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.159549+0000", 2024-09-11T09:51:51.731 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 28.517355063, 2024-09-11T09:51:51.731 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.015591666000000001, 2024-09-11T09:51:51.731 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:51.731 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:51.731 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:51.732 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:51.732 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.732 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:51.732 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.159549+0000", 2024-09-11T09:51:51.732 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:51.732 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.732 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.732 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:51.732 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.159549+0000", 2024-09-11T09:51:51.732 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:51.732 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.732 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.732 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:51.732 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.159550+0000", 2024-09-11T09:51:51.732 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.1500000000000004e-07 2024-09-11T09:51:51.733 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.733 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.733 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:51.733 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.167834+0000", 2024-09-11T09:51:51.733 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0082840220000000003 2024-09-11T09:51:51.733 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.733 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.733 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:51.733 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.167857+0000", 2024-09-11T09:51:51.733 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.3337e-05 2024-09-11T09:51:51.733 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.733 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.733 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:51.733 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.167869+0000", 2024-09-11T09:51:51.733 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1556000000000001e-05 2024-09-11T09:51:51.734 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.734 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.734 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:51.734 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.167939+0000", 2024-09-11T09:51:51.734 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.0674999999999997e-05 2024-09-11T09:51:51.734 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.734 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.734 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:51.734 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.167967+0000", 2024-09-11T09:51:51.734 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.7322999999999999e-05 2024-09-11T09:51:51.734 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.734 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.734 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:51.734 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.175079+0000", 2024-09-11T09:51:51.734 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0071126150000000001 2024-09-11T09:51:51.735 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.735 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.735 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:51.735 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.175141+0000", 2024-09-11T09:51:51.735 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.1523e-05 2024-09-11T09:51:51.735 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:51.735 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:51.735 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:51.735 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.735 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.735 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.5314.0:31 2.1 2:8654721b:::rbd_data.1250f5e25b80.0000000000000000:head [write 393216~131072] snapc 3=[3] ondisk+write+known_if_redirected+supports_pool_eio e68)", 2024-09-11T09:51:51.735 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:50.230850+0000", 2024-09-11T09:51:51.735 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 1.4460543020000001, 2024-09-11T09:51:51.735 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0073772539999999998, 2024-09-11T09:51:51.735 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:51.735 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:51.736 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:51.736 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:51.736 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.5314", 2024-09-11T09:51:51.736 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3853995732", 2024-09-11T09:51:51.736 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 31 2024-09-11T09:51:51.736 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.736 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:51.736 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.736 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:51.736 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.230850+0000", 2024-09-11T09:51:51.736 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:51.736 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.736 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.736 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:51.736 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.230850+0000", 2024-09-11T09:51:51.737 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:51.737 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.737 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.737 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:51.737 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.230850+0000", 2024-09-11T09:51:51.737 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.32e-07 2024-09-11T09:51:51.737 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.737 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.737 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:51.737 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.230908+0000", 2024-09-11T09:51:51.737 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.7698000000000002e-05 2024-09-11T09:51:51.737 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.737 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.737 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:51.738 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.230915+0000", 2024-09-11T09:51:51.738 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.81e-06 2024-09-11T09:51:51.738 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.738 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.738 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:51.738 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.230920+0000", 2024-09-11T09:51:51.738 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.9660000000000004e-06 2024-09-11T09:51:51.738 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.738 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.738 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:51.738 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.235055+0000", 2024-09-11T09:51:51.738 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0041354629999999998 2024-09-11T09:51:51.738 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.738 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.738 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:51.738 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.235145+0000", 2024-09-11T09:51:51.739 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.9362000000000003e-05 2024-09-11T09:51:51.739 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.739 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.739 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1", 2024-09-11T09:51:51.739 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.235282+0000", 2024-09-11T09:51:51.739 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00013710999999999999 2024-09-11T09:51:51.739 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.739 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.739 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:51.739 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.237335+0000", 2024-09-11T09:51:51.739 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.002053165 2024-09-11T09:51:51.739 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.739 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.739 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:51.739 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.238106+0000", 2024-09-11T09:51:51.740 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000771015 2024-09-11T09:51:51.740 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.740 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.740 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:51.740 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.238165+0000", 2024-09-11T09:51:51.740 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.8588000000000002e-05 2024-09-11T09:51:51.740 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.740 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.740 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:51.740 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.238227+0000", 2024-09-11T09:51:51.740 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.2644999999999994e-05 2024-09-11T09:51:51.740 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:51.740 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:51.740 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:51.740 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.741 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.741 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.5314.0:32 2.1 2:8654721b:::rbd_data.1250f5e25b80.0000000000000000:head [write 524288~131072] snapc 3=[3] ondisk+write+known_if_redirected+supports_pool_eio e68)", 2024-09-11T09:51:51.741 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:50.230932+0000", 2024-09-11T09:51:51.741 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 1.4459726769999999, 2024-09-11T09:51:51.741 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0074882129999999996, 2024-09-11T09:51:51.741 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:51.741 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:51.741 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:51.741 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:51.741 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.5314", 2024-09-11T09:51:51.741 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3853995732", 2024-09-11T09:51:51.741 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 32 2024-09-11T09:51:51.741 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.741 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:51.741 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.741 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:51.742 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.230932+0000", 2024-09-11T09:51:51.742 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:51.742 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.742 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.742 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:51.742 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.230932+0000", 2024-09-11T09:51:51.742 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:51.742 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.742 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.742 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:51.742 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.230932+0000", 2024-09-11T09:51:51.742 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.9999999999999999e-07 2024-09-11T09:51:51.742 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.742 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.743 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:51.743 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.231022+0000", 2024-09-11T09:51:51.743 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.0439999999999995e-05 2024-09-11T09:51:51.743 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.743 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.743 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:51.743 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.231030+0000", 2024-09-11T09:51:51.743 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.3350000000000001e-06 2024-09-11T09:51:51.743 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.743 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.743 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:51.743 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.231034+0000", 2024-09-11T09:51:51.743 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.9759999999999997e-06 2024-09-11T09:51:51.743 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.743 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.744 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:51.744 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.235837+0000", 2024-09-11T09:51:51.744 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0048031910000000001 2024-09-11T09:51:51.744 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.744 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.744 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:51.744 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.235904+0000", 2024-09-11T09:51:51.744 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.7181999999999995e-05 2024-09-11T09:51:51.744 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.744 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.744 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1", 2024-09-11T09:51:51.744 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.236040+0000", 2024-09-11T09:51:51.744 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00013545199999999999 2024-09-11T09:51:51.744 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.744 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.744 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:51.745 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.238242+0000", 2024-09-11T09:51:51.745 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.002202809 2024-09-11T09:51:51.745 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.745 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.745 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:51.745 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.238301+0000", 2024-09-11T09:51:51.745 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.8573999999999999e-05 2024-09-11T09:51:51.745 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.745 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.745 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:51.745 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.238378+0000", 2024-09-11T09:51:51.745 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.7323999999999994e-05 2024-09-11T09:51:51.745 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.745 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.745 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:51.746 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.238420+0000", 2024-09-11T09:51:51.746 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.163e-05 2024-09-11T09:51:51.746 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:51.746 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:51.746 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:51.746 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.746 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.746 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.5314.0:33 2.1 2:8654721b:::rbd_data.1250f5e25b80.0000000000000000:head [write 655360~131072] snapc 3=[3] ondisk+write+known_if_redirected+supports_pool_eio e68)", 2024-09-11T09:51:51.746 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:50.231041+0000", 2024-09-11T09:51:51.746 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 1.4458635179999999, 2024-09-11T09:51:51.746 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0075627139999999999, 2024-09-11T09:51:51.746 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:51.746 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:51.746 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:51.746 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:51.747 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.5314", 2024-09-11T09:51:51.747 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3853995732", 2024-09-11T09:51:51.747 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 33 2024-09-11T09:51:51.747 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.747 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:51.747 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.747 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:51.747 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.231041+0000", 2024-09-11T09:51:51.747 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:51.747 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.747 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.747 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:51.747 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.231041+0000", 2024-09-11T09:51:51.747 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:51.747 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.748 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.748 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:51.748 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.231041+0000", 2024-09-11T09:51:51.748 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.9499999999999998e-07 2024-09-11T09:51:51.748 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.748 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.748 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:51.748 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.231111+0000", 2024-09-11T09:51:51.748 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.0166999999999994e-05 2024-09-11T09:51:51.748 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.748 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.748 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:51.748 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.231122+0000", 2024-09-11T09:51:51.748 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0518e-05 2024-09-11T09:51:51.748 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.749 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.749 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:51.749 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.231128+0000", 2024-09-11T09:51:51.749 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.3509999999999998e-06 2024-09-11T09:51:51.749 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.749 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.749 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:51.749 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.236622+0000", 2024-09-11T09:51:51.749 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0054934989999999998 2024-09-11T09:51:51.749 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.749 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.749 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:51.749 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.236719+0000", 2024-09-11T09:51:51.749 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.7497999999999994e-05 2024-09-11T09:51:51.749 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.749 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.750 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1", 2024-09-11T09:51:51.750 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.236854+0000", 2024-09-11T09:51:51.750 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000134535 2024-09-11T09:51:51.750 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.750 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.750 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:51.750 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.238491+0000", 2024-09-11T09:51:51.750 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0016368610000000001 2024-09-11T09:51:51.750 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.750 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.750 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:51.750 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.238504+0000", 2024-09-11T09:51:51.750 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3927999999999999e-05 2024-09-11T09:51:51.751 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.751 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.751 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:51.751 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.238561+0000", 2024-09-11T09:51:51.751 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.6029999999999997e-05 2024-09-11T09:51:51.751 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.751 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.751 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:51.751 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.238604+0000", 2024-09-11T09:51:51.751 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.3031999999999997e-05 2024-09-11T09:51:51.751 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:51.751 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:51.751 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:51.751 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.751 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.751 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.5314.0:34 2.1 2:8654721b:::rbd_data.1250f5e25b80.0000000000000000:head [write 786432~131072] snapc 3=[3] ondisk+write+known_if_redirected+supports_pool_eio e68)", 2024-09-11T09:51:51.752 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:50.231139+0000", 2024-09-11T09:51:51.752 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 1.445764915, 2024-09-11T09:51:51.752 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0083689690000000004, 2024-09-11T09:51:51.752 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:51.752 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:51.752 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:51.752 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:51.752 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.5314", 2024-09-11T09:51:51.752 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3853995732", 2024-09-11T09:51:51.752 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 34 2024-09-11T09:51:51.752 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.752 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:51.752 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.752 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:51.752 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.231139+0000", 2024-09-11T09:51:51.753 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:51.753 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.753 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.753 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:51.753 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.231139+0000", 2024-09-11T09:51:51.753 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:51.753 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.753 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.753 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:51.753 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.231140+0000", 2024-09-11T09:51:51.753 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.1699999999999998e-07 2024-09-11T09:51:51.753 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.753 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.753 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:51.754 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.231247+0000", 2024-09-11T09:51:51.754 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000107088 2024-09-11T09:51:51.754 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.754 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.754 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:51.754 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.231260+0000", 2024-09-11T09:51:51.754 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.289e-05 2024-09-11T09:51:51.754 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.754 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.754 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:51.754 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.231272+0000", 2024-09-11T09:51:51.754 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2371999999999999e-05 2024-09-11T09:51:51.754 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.754 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.754 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:51.754 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.237380+0000", 2024-09-11T09:51:51.755 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.006107683 2024-09-11T09:51:51.755 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.755 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.755 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:51.755 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.237434+0000", 2024-09-11T09:51:51.755 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.4143e-05 2024-09-11T09:51:51.755 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.755 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.755 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1", 2024-09-11T09:51:51.755 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.237541+0000", 2024-09-11T09:51:51.755 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00010676800000000001 2024-09-11T09:51:51.755 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.755 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.755 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:51.755 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.239390+0000", 2024-09-11T09:51:51.756 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.001848861 2024-09-11T09:51:51.756 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.756 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.756 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:51.756 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.239409+0000", 2024-09-11T09:51:51.756 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.9505e-05 2024-09-11T09:51:51.756 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.756 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.756 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:51.756 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.239463+0000", 2024-09-11T09:51:51.756 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.3838000000000002e-05 2024-09-11T09:51:51.756 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.756 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.756 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:51.756 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.239508+0000", 2024-09-11T09:51:51.757 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.5303999999999999e-05 2024-09-11T09:51:51.757 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:51.757 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:51.757 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:51.757 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.757 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.757 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.5314.0:35 2.1 2:8654721b:::rbd_data.1250f5e25b80.0000000000000000:head [write 917504~131072] snapc 3=[3] ondisk+write+known_if_redirected+supports_pool_eio e68)", 2024-09-11T09:51:51.757 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:50.231287+0000", 2024-09-11T09:51:51.757 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 1.445617581, 2024-09-11T09:51:51.757 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0086930730000000008, 2024-09-11T09:51:51.757 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:51.757 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:51.757 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:51.758 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:51.758 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.5314", 2024-09-11T09:51:51.758 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3853995732", 2024-09-11T09:51:51.758 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 35 2024-09-11T09:51:51.758 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.758 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:51.758 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.758 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:51.758 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.231287+0000", 2024-09-11T09:51:51.758 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:51.758 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.758 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.758 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:51.758 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.231287+0000", 2024-09-11T09:51:51.758 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:51.759 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.759 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.759 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:51.759 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.231287+0000", 2024-09-11T09:51:51.759 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.7700000000000004e-07 2024-09-11T09:51:51.759 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.759 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.759 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:51.759 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.231354+0000", 2024-09-11T09:51:51.759 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.6304999999999996e-05 2024-09-11T09:51:51.759 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.759 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.759 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:51.759 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.231367+0000", 2024-09-11T09:51:51.760 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2952999999999999e-05 2024-09-11T09:51:51.760 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.760 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.760 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:51.760 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.231374+0000", 2024-09-11T09:51:51.760 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.0469999999999996e-06 2024-09-11T09:51:51.760 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.760 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.760 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:51.760 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.238650+0000", 2024-09-11T09:51:51.760 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0072760280000000004 2024-09-11T09:51:51.760 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.760 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.760 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:51.761 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.238732+0000", 2024-09-11T09:51:51.761 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.2194000000000006e-05 2024-09-11T09:51:51.761 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.761 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.761 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1", 2024-09-11T09:51:51.761 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.238876+0000", 2024-09-11T09:51:51.761 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00014458 2024-09-11T09:51:51.761 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.761 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.761 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:51.761 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.239739+0000", 2024-09-11T09:51:51.761 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00086244499999999999 2024-09-11T09:51:51.761 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.761 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.761 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:51.762 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.239917+0000", 2024-09-11T09:51:51.762 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00017777299999999999 2024-09-11T09:51:51.762 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.762 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.762 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:51.762 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.239954+0000", 2024-09-11T09:51:51.762 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.7379999999999998e-05 2024-09-11T09:51:51.762 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.762 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.762 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:51.762 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.239980+0000", 2024-09-11T09:51:51.762 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.5791000000000001e-05 2024-09-11T09:51:51.762 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:51.762 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:51.762 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:51.763 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.763 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.763 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.5314.0:70 2.1 2:8654721b:::rbd_data.1250f5e25b80.0000000000000000:head [write 524288~131072] snapc 5=[5,3] ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-09-11T09:51:51.763 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:50.824556+0000", 2024-09-11T09:51:51.763 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 0.85234878000000003, 2024-09-11T09:51:51.763 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0072794469999999997, 2024-09-11T09:51:51.763 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:51.763 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:51.763 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:51.763 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:51.763 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.5314", 2024-09-11T09:51:51.763 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3853995732", 2024-09-11T09:51:51.763 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 70 2024-09-11T09:51:51.763 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.764 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:51.764 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.764 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:51.764 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.824556+0000", 2024-09-11T09:51:51.764 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:51.764 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.764 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.764 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:51.764 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.824556+0000", 2024-09-11T09:51:51.764 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:51.764 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.764 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.764 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:51.764 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.824564+0000", 2024-09-11T09:51:51.764 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.2409999999999995e-06 2024-09-11T09:51:51.765 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.765 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.765 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:51.765 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.824613+0000", 2024-09-11T09:51:51.765 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.9579000000000001e-05 2024-09-11T09:51:51.765 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.765 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.765 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:51.765 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.824638+0000", 2024-09-11T09:51:51.765 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.4510000000000001e-05 2024-09-11T09:51:51.765 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.765 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.765 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:51.765 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.824653+0000", 2024-09-11T09:51:51.766 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.4831000000000001e-05 2024-09-11T09:51:51.766 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.766 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.766 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:51.766 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.829072+0000", 2024-09-11T09:51:51.766 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0044195700000000003 2024-09-11T09:51:51.766 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.766 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.766 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:51.766 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.829166+0000", 2024-09-11T09:51:51.766 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.3212000000000001e-05 2024-09-11T09:51:51.766 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.766 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.766 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1", 2024-09-11T09:51:51.767 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.829309+0000", 2024-09-11T09:51:51.767 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00014301900000000001 2024-09-11T09:51:51.767 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.767 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.767 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:51.767 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.831074+0000", 2024-09-11T09:51:51.767 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.001765305 2024-09-11T09:51:51.767 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.767 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.767 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:51.767 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.831751+0000", 2024-09-11T09:51:51.767 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00067694000000000001 2024-09-11T09:51:51.767 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.767 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.767 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:51.768 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.831803+0000", 2024-09-11T09:51:51.768 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.2441000000000002e-05 2024-09-11T09:51:51.768 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.768 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.768 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:51.768 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.831835+0000", 2024-09-11T09:51:51.768 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.1798999999999998e-05 2024-09-11T09:51:51.768 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:51.768 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:51.768 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:51.768 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.768 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.768 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.5314.0:71 2.1 2:8654721b:::rbd_data.1250f5e25b80.0000000000000000:head [write 655360~131072] snapc 5=[5,3] ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-09-11T09:51:51.768 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:50.824662+0000", 2024-09-11T09:51:51.768 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 0.85224266400000004, 2024-09-11T09:51:51.769 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0082172219999999997, 2024-09-11T09:51:51.769 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:51.769 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:51.769 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:51.769 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:51.769 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.5314", 2024-09-11T09:51:51.769 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3853995732", 2024-09-11T09:51:51.769 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 71 2024-09-11T09:51:51.769 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.769 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:51.769 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.769 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:51.769 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.824662+0000", 2024-09-11T09:51:51.769 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:51.769 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.770 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.770 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:51.770 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.824662+0000", 2024-09-11T09:51:51.770 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:51.770 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.770 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.770 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:51.770 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.824662+0000", 2024-09-11T09:51:51.770 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.0699999999999998e-07 2024-09-11T09:51:51.770 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.770 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.770 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:51.770 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.824859+0000", 2024-09-11T09:51:51.770 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00019725799999999999 2024-09-11T09:51:51.770 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.771 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.771 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:51.771 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.824906+0000", 2024-09-11T09:51:51.771 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.7162e-05 2024-09-11T09:51:51.771 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.771 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.771 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:51.771 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.824926+0000", 2024-09-11T09:51:51.771 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.0038999999999999e-05 2024-09-11T09:51:51.771 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.771 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.771 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:51.771 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.830313+0000", 2024-09-11T09:51:51.771 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0053868409999999999 2024-09-11T09:51:51.771 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.771 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.772 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:51.772 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.830359+0000", 2024-09-11T09:51:51.772 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.5370000000000001e-05 2024-09-11T09:51:51.772 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.772 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.772 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1", 2024-09-11T09:51:51.772 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.830460+0000", 2024-09-11T09:51:51.772 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00010171299999999999 2024-09-11T09:51:51.772 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.772 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.772 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:51.772 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.831843+0000", 2024-09-11T09:51:51.772 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.001382501 2024-09-11T09:51:51.772 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.773 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.773 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:51.773 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.832769+0000", 2024-09-11T09:51:51.773 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00092638099999999999 2024-09-11T09:51:51.773 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.773 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.773 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:51.773 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.832835+0000", 2024-09-11T09:51:51.773 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.6106999999999996e-05 2024-09-11T09:51:51.773 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.773 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.773 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:51.773 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.832879+0000", 2024-09-11T09:51:51.773 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.3543e-05 2024-09-11T09:51:51.773 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:51.774 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:51.774 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:51.774 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.774 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.774 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.5314.0:72 2.1 2:8654721b:::rbd_data.1250f5e25b80.0000000000000000:head [write 786432~131072] snapc 5=[5,3] ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-09-11T09:51:51.774 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:50.824938+0000", 2024-09-11T09:51:51.774 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 0.85196636199999998, 2024-09-11T09:51:51.774 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0082389799999999999, 2024-09-11T09:51:51.774 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:51.774 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:51.774 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:51.774 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:51.774 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.5314", 2024-09-11T09:51:51.774 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3853995732", 2024-09-11T09:51:51.774 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 72 2024-09-11T09:51:51.775 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.775 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:51.775 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.775 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:51.775 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.824938+0000", 2024-09-11T09:51:51.775 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:51.775 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.775 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.775 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:51.775 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.824938+0000", 2024-09-11T09:51:51.775 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:51.775 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.775 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.775 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:51.776 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.824939+0000", 2024-09-11T09:51:51.776 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.6199999999999997e-07 2024-09-11T09:51:51.776 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.776 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.776 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:51.776 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.825028+0000", 2024-09-11T09:51:51.776 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.9105999999999997e-05 2024-09-11T09:51:51.776 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.776 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.776 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:51.776 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.825042+0000", 2024-09-11T09:51:51.776 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3760999999999999e-05 2024-09-11T09:51:51.776 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.776 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.777 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:51.777 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.825061+0000", 2024-09-11T09:51:51.777 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.9088000000000001e-05 2024-09-11T09:51:51.777 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.777 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.777 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:51.777 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.831122+0000", 2024-09-11T09:51:51.777 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.006061299 2024-09-11T09:51:51.777 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.777 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.777 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:51.777 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.831195+0000", 2024-09-11T09:51:51.777 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.2957e-05 2024-09-11T09:51:51.777 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.778 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.778 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1", 2024-09-11T09:51:51.778 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.831301+0000", 2024-09-11T09:51:51.778 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000106049 2024-09-11T09:51:51.778 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.778 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.778 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:51.778 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.832893+0000", 2024-09-11T09:51:51.778 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0015923770000000001 2024-09-11T09:51:51.778 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.778 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.778 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:51.778 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.833076+0000", 2024-09-11T09:51:51.778 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000182855 2024-09-11T09:51:51.779 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.779 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.779 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:51.779 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.833137+0000", 2024-09-11T09:51:51.779 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.1193999999999997e-05 2024-09-11T09:51:51.779 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.779 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.779 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:51.779 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.833177+0000", 2024-09-11T09:51:51.779 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.9632000000000003e-05 2024-09-11T09:51:51.779 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:51.779 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:51.779 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:51.779 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.780 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.780 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.5314.0:73 2.1 2:8654721b:::rbd_data.1250f5e25b80.0000000000000000:head [write 917504~131072] snapc 5=[5,3] ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-09-11T09:51:51.780 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:50.825071+0000", 2024-09-11T09:51:51.780 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 0.85183285500000006, 2024-09-11T09:51:51.780 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0084039160000000009, 2024-09-11T09:51:51.780 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:51.780 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:51.780 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:51.780 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:51.780 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.5314", 2024-09-11T09:51:51.780 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3853995732", 2024-09-11T09:51:51.780 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 73 2024-09-11T09:51:51.780 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.781 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:51.781 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.781 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:51.781 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.825071+0000", 2024-09-11T09:51:51.781 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:51.781 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.781 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.781 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:51.781 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.825071+0000", 2024-09-11T09:51:51.781 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:51.781 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.781 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.781 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:51.781 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.825072+0000", 2024-09-11T09:51:51.782 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.5499999999999998e-07 2024-09-11T09:51:51.782 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.782 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.782 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:51.782 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.825124+0000", 2024-09-11T09:51:51.782 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.2197000000000002e-05 2024-09-11T09:51:51.782 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.782 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.782 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:51.782 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.825132+0000", 2024-09-11T09:51:51.782 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.0290000000000005e-06 2024-09-11T09:51:51.782 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.782 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.782 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:51.783 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.825136+0000", 2024-09-11T09:51:51.783 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.9380000000000002e-06 2024-09-11T09:51:51.783 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.783 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.783 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:51.783 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.832001+0000", 2024-09-11T09:51:51.783 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0068650949999999999 2024-09-11T09:51:51.783 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.783 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.783 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:51.783 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.832091+0000", 2024-09-11T09:51:51.783 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.9524999999999994e-05 2024-09-11T09:51:51.783 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.783 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.784 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1", 2024-09-11T09:51:51.784 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.832234+0000", 2024-09-11T09:51:51.784 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000142783 2024-09-11T09:51:51.784 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.784 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.784 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:51.784 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.833198+0000", 2024-09-11T09:51:51.784 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00096476899999999998 2024-09-11T09:51:51.784 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.784 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.784 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:51.784 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.833386+0000", 2024-09-11T09:51:51.784 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00018772 2024-09-11T09:51:51.784 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.785 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.785 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:51.785 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.833440+0000", 2024-09-11T09:51:51.785 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.4228999999999999e-05 2024-09-11T09:51:51.785 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:51.785 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:51.785 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:51.785 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.833475+0000", 2024-09-11T09:51:51.785 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.5076000000000001e-05 2024-09-11T09:51:51.785 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:51.785 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:51.785 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:51.785 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:51.786 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:51.786 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:51:51.787 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-11T09:51:51.788 INFO:tasks.ceph.osd.2.smithi017.stderr:2024-09-11T09:51:51.787+0000 7fd20b170640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31672) UID: 0 2024-09-11T09:51:51.792 INFO:teuthology.orchestra.run.smithi017.stderr:noscrub is set 2024-09-11T09:51:51.888 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:51:51.888+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:51:51.901 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-11T09:51:51.901 INFO:teuthology.orchestra.run.smithi096.stdout: "ops": [], 2024-09-11T09:51:51.901 INFO:teuthology.orchestra.run.smithi096.stdout: "num_ops": 0 2024-09-11T09:51:51.901 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-11T09:51:51.910 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-11T09:51:51.927 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T09:51:51.989 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:51:51.990+0000 7f3a7b90a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 42475) UID: 0 2024-09-11T09:51:52.014 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-11T09:51:52.015 INFO:teuthology.orchestra.run.smithi096.stdout: "ops": [], 2024-09-11T09:51:52.015 INFO:teuthology.orchestra.run.smithi096.stdout: "complaint_time": 30, 2024-09-11T09:51:52.015 INFO:teuthology.orchestra.run.smithi096.stdout: "num_blocked_ops": 0 2024-09-11T09:51:52.015 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-11T09:51:52.023 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-11T09:51:52.050 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-11T09:51:52.050 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T09:51:52.067 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-11T09:51:52.067 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T09:51:52.083 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-11T09:51:52.084 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T09:51:52.089 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:51:52.090+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:51:52.100 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-11T09:51:52.100 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T09:51:52.117 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-11T09:51:52.117 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T09:51:52.133 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-11T09:51:52.547 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T09:51:52.548 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:51:52.548 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:51:52.549 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-11T09:51:52.549 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:51:52.549+0000 7f3c96ec9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31909) UID: 0 2024-09-11T09:51:52.649 INFO:tasks.ceph.osd.0.smithi017.stderr:2024-09-11T09:51:52.650+0000 7fb73c0f1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31665) UID: 0 2024-09-11T09:51:52.691 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:51:52.691 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:51:52.700 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-11T09:51:52.750 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:51:52.750+0000 7f3c96ec9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31909) UID: 0 2024-09-11T09:51:52.803 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:51:52.804 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:51:52.813 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-11T09:51:52.853 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-09-11T09:51:52.854+0000 7f7b6c7d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31904) UID: 0 2024-09-11T09:51:52.952 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:51:52.952+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:51:52.955 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:51:52.956 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:51:52.964 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-11T09:51:53.053 INFO:tasks.ceph.osd.0.smithi017.stderr:2024-09-11T09:51:53.053+0000 7fb73c0f1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31665) UID: 0 2024-09-11T09:51:53.068 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:51:53.069 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:51:53.077 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-11T09:51:53.154 INFO:tasks.ceph.osd.2.smithi017.stderr:2024-09-11T09:51:53.155+0000 7fd20b170640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31672) UID: 0 2024-09-11T09:51:53.179 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:51:53.179 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:51:53.188 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-11T09:51:53.255 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:51:53.256+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:51:53.292 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:51:53.292 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:51:53.301 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-11T09:51:53.356 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:51:53.356+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:51:53.423 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:51:53.423 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:51:53.432 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-11T09:51:53.457 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:51:53.457+0000 7f3a7b90a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 42475) UID: 0 2024-09-11T09:51:53.558 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:51:53.558+0000 7f3a7b90a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 42475) UID: 0 2024-09-11T09:51:53.581 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:51:53.582 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:51:53.591 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-11T09:51:53.658 INFO:tasks.ceph.osd.0.smithi017.stderr:2024-09-11T09:51:53.658+0000 7fb73c0f1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31665) UID: 0 2024-09-11T09:51:53.698 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:51:53.698 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:51:53.708 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-11T09:51:53.792 INFO:tasks.ceph.osd.0.smithi017.stderr:2024-09-11T09:51:53.792+0000 7fb73c0f1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31665) UID: 0 2024-09-11T09:51:53.805 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-11T09:51:53.816 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 5, 0] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 5, 4] 2024-09-11T09:51:53.816 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-11T09:51:53.816 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-09-11T09:51:53.817 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump -f json-pretty 2024-09-11T09:51:53.858 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:51:53.858 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:51:53.861 INFO:tasks.ceph.osd.0.smithi017.stderr:2024-09-11T09:51:53.861+0000 7fb73c0f1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31665) UID: 0 2024-09-11T09:51:53.868 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-11T09:51:53.962 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-09-11T09:51:53.962+0000 7f7b6c7d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31904) UID: 0 2024-09-11T09:51:54.032 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:51:54.032 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:51:54.042 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-11T09:51:54.085 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:51:54.085+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:51:54.151 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:51:54.151 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:51:54.160 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-11T09:51:54.165 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-11T09:51:54.165 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:51:54.165 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 70, 2024-09-11T09:51:54.165 INFO:teuthology.orchestra.run.smithi017.stdout: "fsid": "a2f1daec-d3b0-4bf5-82c8-72ecbb93d4e4", 2024-09-11T09:51:54.165 INFO:teuthology.orchestra.run.smithi017.stdout: "created": "2024-09-11T09:50:00.165804+0000", 2024-09-11T09:51:54.165 INFO:teuthology.orchestra.run.smithi017.stdout: "modified": "2024-09-11T09:51:51.790908+0000", 2024-09-11T09:51:54.165 INFO:teuthology.orchestra.run.smithi017.stdout: "last_up_change": "2024-09-11T09:51:44.764990+0000", 2024-09-11T09:51:54.165 INFO:teuthology.orchestra.run.smithi017.stdout: "last_in_change": "2024-09-11T09:50:45.581301+0000", 2024-09-11T09:51:54.165 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": "noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-11T09:51:54.165 INFO:teuthology.orchestra.run.smithi017.stdout: "flags_num": 5801984, 2024-09-11T09:51:54.166 INFO:teuthology.orchestra.run.smithi017.stdout: "flags_set": [ 2024-09-11T09:51:54.166 INFO:teuthology.orchestra.run.smithi017.stdout: "noscrub", 2024-09-11T09:51:54.166 INFO:teuthology.orchestra.run.smithi017.stdout: "pglog_hardlimit", 2024-09-11T09:51:54.166 INFO:teuthology.orchestra.run.smithi017.stdout: "purged_snapdirs", 2024-09-11T09:51:54.166 INFO:teuthology.orchestra.run.smithi017.stdout: "recovery_deletes", 2024-09-11T09:51:54.166 INFO:teuthology.orchestra.run.smithi017.stdout: "sortbitwise" 2024-09-11T09:51:54.166 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-11T09:51:54.166 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_version": 6, 2024-09-11T09:51:54.166 INFO:teuthology.orchestra.run.smithi017.stdout: "full_ratio": 0.94999998807907104, 2024-09-11T09:51:54.166 INFO:teuthology.orchestra.run.smithi017.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-11T09:51:54.166 INFO:teuthology.orchestra.run.smithi017.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-11T09:51:54.166 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_snapshot": "", 2024-09-11T09:51:54.166 INFO:teuthology.orchestra.run.smithi017.stdout: "pool_max": 3, 2024-09-11T09:51:54.166 INFO:teuthology.orchestra.run.smithi017.stdout: "max_osd": 6, 2024-09-11T09:51:54.166 INFO:teuthology.orchestra.run.smithi017.stdout: "require_min_compat_client": "luminous", 2024-09-11T09:51:54.167 INFO:teuthology.orchestra.run.smithi017.stdout: "min_compat_client": "luminous", 2024-09-11T09:51:54.167 INFO:teuthology.orchestra.run.smithi017.stdout: "require_osd_release": "squid", 2024-09-11T09:51:54.167 INFO:teuthology.orchestra.run.smithi017.stdout: "allow_crimson": false, 2024-09-11T09:51:54.167 INFO:teuthology.orchestra.run.smithi017.stdout: "pools": [ 2024-09-11T09:51:54.167 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:54.167 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 1, 2024-09-11T09:51:54.167 INFO:teuthology.orchestra.run.smithi017.stdout: "pool_name": ".mgr", 2024-09-11T09:51:54.167 INFO:teuthology.orchestra.run.smithi017.stdout: "create_time": "2024-09-11T09:50:10.457184+0000", 2024-09-11T09:51:54.167 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 1, 2024-09-11T09:51:54.167 INFO:teuthology.orchestra.run.smithi017.stdout: "flags_names": "hashpspool", 2024-09-11T09:51:54.167 INFO:teuthology.orchestra.run.smithi017.stdout: "type": 1, 2024-09-11T09:51:54.167 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 2, 2024-09-11T09:51:54.167 INFO:teuthology.orchestra.run.smithi017.stdout: "min_size": 1, 2024-09-11T09:51:54.168 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_rule": 0, 2024-09-11T09:51:54.168 INFO:teuthology.orchestra.run.smithi017.stdout: "peering_crush_bucket_count": 0, 2024-09-11T09:51:54.168 INFO:teuthology.orchestra.run.smithi017.stdout: "peering_crush_bucket_target": 0, 2024-09-11T09:51:54.168 INFO:teuthology.orchestra.run.smithi017.stdout: "peering_crush_bucket_barrier": 0, 2024-09-11T09:51:54.168 INFO:teuthology.orchestra.run.smithi017.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-11T09:51:54.168 INFO:teuthology.orchestra.run.smithi017.stdout: "is_stretch_pool": false, 2024-09-11T09:51:54.168 INFO:teuthology.orchestra.run.smithi017.stdout: "object_hash": 2, 2024-09-11T09:51:54.168 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_autoscale_mode": "off", 2024-09-11T09:51:54.168 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_num": 1, 2024-09-11T09:51:54.169 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_placement_num": 1, 2024-09-11T09:51:54.169 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_placement_num_target": 1, 2024-09-11T09:51:54.169 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_num_target": 1, 2024-09-11T09:51:54.169 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_num_pending": 1, 2024-09-11T09:51:54.169 INFO:teuthology.orchestra.run.smithi017.stdout: "last_pg_merge_meta": { 2024-09-11T09:51:54.169 INFO:teuthology.orchestra.run.smithi017.stdout: "source_pgid": "0.0", 2024-09-11T09:51:54.169 INFO:teuthology.orchestra.run.smithi017.stdout: "ready_epoch": 0, 2024-09-11T09:51:54.169 INFO:teuthology.orchestra.run.smithi017.stdout: "last_epoch_started": 0, 2024-09-11T09:51:54.169 INFO:teuthology.orchestra.run.smithi017.stdout: "last_epoch_clean": 0, 2024-09-11T09:51:54.169 INFO:teuthology.orchestra.run.smithi017.stdout: "source_version": "0'0", 2024-09-11T09:51:54.169 INFO:teuthology.orchestra.run.smithi017.stdout: "target_version": "0'0" 2024-09-11T09:51:54.169 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:54.169 INFO:teuthology.orchestra.run.smithi017.stdout: "last_change": "14", 2024-09-11T09:51:54.170 INFO:teuthology.orchestra.run.smithi017.stdout: "last_force_op_resend": "0", 2024-09-11T09:51:54.170 INFO:teuthology.orchestra.run.smithi017.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-11T09:51:54.170 INFO:teuthology.orchestra.run.smithi017.stdout: "last_force_op_resend_preluminous": "0", 2024-09-11T09:51:54.170 INFO:teuthology.orchestra.run.smithi017.stdout: "auid": 0, 2024-09-11T09:51:54.170 INFO:teuthology.orchestra.run.smithi017.stdout: "snap_mode": "selfmanaged", 2024-09-11T09:51:54.170 INFO:teuthology.orchestra.run.smithi017.stdout: "snap_seq": 0, 2024-09-11T09:51:54.170 INFO:teuthology.orchestra.run.smithi017.stdout: "snap_epoch": 0, 2024-09-11T09:51:54.170 INFO:teuthology.orchestra.run.smithi017.stdout: "pool_snaps": [], 2024-09-11T09:51:54.170 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_snaps": "[]", 2024-09-11T09:51:54.170 INFO:teuthology.orchestra.run.smithi017.stdout: "quota_max_bytes": 0, 2024-09-11T09:51:54.170 INFO:teuthology.orchestra.run.smithi017.stdout: "quota_max_objects": 0, 2024-09-11T09:51:54.170 INFO:teuthology.orchestra.run.smithi017.stdout: "tiers": [], 2024-09-11T09:51:54.170 INFO:teuthology.orchestra.run.smithi017.stdout: "tier_of": -1, 2024-09-11T09:51:54.171 INFO:teuthology.orchestra.run.smithi017.stdout: "read_tier": -1, 2024-09-11T09:51:54.171 INFO:teuthology.orchestra.run.smithi017.stdout: "write_tier": -1, 2024-09-11T09:51:54.171 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_mode": "none", 2024-09-11T09:51:54.171 INFO:teuthology.orchestra.run.smithi017.stdout: "target_max_bytes": 0, 2024-09-11T09:51:54.171 INFO:teuthology.orchestra.run.smithi017.stdout: "target_max_objects": 0, 2024-09-11T09:51:54.171 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-11T09:51:54.171 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-11T09:51:54.171 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-11T09:51:54.171 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_min_flush_age": 0, 2024-09-11T09:51:54.171 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_min_evict_age": 0, 2024-09-11T09:51:54.171 INFO:teuthology.orchestra.run.smithi017.stdout: "erasure_code_profile": "", 2024-09-11T09:51:54.171 INFO:teuthology.orchestra.run.smithi017.stdout: "hit_set_params": { 2024-09-11T09:51:54.171 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "none" 2024-09-11T09:51:54.171 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:54.172 INFO:teuthology.orchestra.run.smithi017.stdout: "hit_set_period": 0, 2024-09-11T09:51:54.172 INFO:teuthology.orchestra.run.smithi017.stdout: "hit_set_count": 0, 2024-09-11T09:51:54.172 INFO:teuthology.orchestra.run.smithi017.stdout: "use_gmt_hitset": true, 2024-09-11T09:51:54.172 INFO:teuthology.orchestra.run.smithi017.stdout: "min_read_recency_for_promote": 0, 2024-09-11T09:51:54.172 INFO:teuthology.orchestra.run.smithi017.stdout: "min_write_recency_for_promote": 0, 2024-09-11T09:51:54.172 INFO:teuthology.orchestra.run.smithi017.stdout: "hit_set_grade_decay_rate": 0, 2024-09-11T09:51:54.172 INFO:teuthology.orchestra.run.smithi017.stdout: "hit_set_search_last_n": 0, 2024-09-11T09:51:54.172 INFO:teuthology.orchestra.run.smithi017.stdout: "grade_table": [], 2024-09-11T09:51:54.172 INFO:teuthology.orchestra.run.smithi017.stdout: "stripe_width": 0, 2024-09-11T09:51:54.172 INFO:teuthology.orchestra.run.smithi017.stdout: "expected_num_objects": 0, 2024-09-11T09:51:54.172 INFO:teuthology.orchestra.run.smithi017.stdout: "fast_read": false, 2024-09-11T09:51:54.172 INFO:teuthology.orchestra.run.smithi017.stdout: "options": { 2024-09-11T09:51:54.172 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_num_max": 32, 2024-09-11T09:51:54.173 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_num_min": 1 2024-09-11T09:51:54.173 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:54.173 INFO:teuthology.orchestra.run.smithi017.stdout: "application_metadata": { 2024-09-11T09:51:54.173 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr": {} 2024-09-11T09:51:54.173 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:54.173 INFO:teuthology.orchestra.run.smithi017.stdout: "read_balance": { 2024-09-11T09:51:54.173 INFO:teuthology.orchestra.run.smithi017.stdout: "score_type": "Fair distribution", 2024-09-11T09:51:54.173 INFO:teuthology.orchestra.run.smithi017.stdout: "score_acting": 6.059999942779541, 2024-09-11T09:51:54.173 INFO:teuthology.orchestra.run.smithi017.stdout: "score_stable": 6.059999942779541, 2024-09-11T09:51:54.173 INFO:teuthology.orchestra.run.smithi017.stdout: "optimal_score": 0.33000001311302185, 2024-09-11T09:51:54.173 INFO:teuthology.orchestra.run.smithi017.stdout: "raw_score_acting": 2, 2024-09-11T09:51:54.173 INFO:teuthology.orchestra.run.smithi017.stdout: "raw_score_stable": 2, 2024-09-11T09:51:54.173 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity_weighted": 1, 2024-09-11T09:51:54.174 INFO:teuthology.orchestra.run.smithi017.stdout: "average_primary_affinity": 1, 2024-09-11T09:51:54.174 INFO:teuthology.orchestra.run.smithi017.stdout: "average_primary_affinity_weighted": 1 2024-09-11T09:51:54.174 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:54.174 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:54.174 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:54.174 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 2, 2024-09-11T09:51:54.174 INFO:teuthology.orchestra.run.smithi017.stdout: "pool_name": "rbd", 2024-09-11T09:51:54.174 INFO:teuthology.orchestra.run.smithi017.stdout: "create_time": "2024-09-11T09:50:16.993202+0000", 2024-09-11T09:51:54.174 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 8193, 2024-09-11T09:51:54.174 INFO:teuthology.orchestra.run.smithi017.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-11T09:51:54.174 INFO:teuthology.orchestra.run.smithi017.stdout: "type": 1, 2024-09-11T09:51:54.174 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 2, 2024-09-11T09:51:54.174 INFO:teuthology.orchestra.run.smithi017.stdout: "min_size": 1, 2024-09-11T09:51:54.174 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_rule": 0, 2024-09-11T09:51:54.174 INFO:teuthology.orchestra.run.smithi017.stdout: "peering_crush_bucket_count": 0, 2024-09-11T09:51:54.175 INFO:teuthology.orchestra.run.smithi017.stdout: "peering_crush_bucket_target": 0, 2024-09-11T09:51:54.175 INFO:teuthology.orchestra.run.smithi017.stdout: "peering_crush_bucket_barrier": 0, 2024-09-11T09:51:54.175 INFO:teuthology.orchestra.run.smithi017.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-11T09:51:54.175 INFO:teuthology.orchestra.run.smithi017.stdout: "is_stretch_pool": false, 2024-09-11T09:51:54.175 INFO:teuthology.orchestra.run.smithi017.stdout: "object_hash": 2, 2024-09-11T09:51:54.175 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_autoscale_mode": "off", 2024-09-11T09:51:54.175 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_num": 8, 2024-09-11T09:51:54.175 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_placement_num": 8, 2024-09-11T09:51:54.175 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_placement_num_target": 8, 2024-09-11T09:51:54.175 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_num_target": 8, 2024-09-11T09:51:54.175 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_num_pending": 8, 2024-09-11T09:51:54.175 INFO:teuthology.orchestra.run.smithi017.stdout: "last_pg_merge_meta": { 2024-09-11T09:51:54.175 INFO:teuthology.orchestra.run.smithi017.stdout: "source_pgid": "0.0", 2024-09-11T09:51:54.175 INFO:teuthology.orchestra.run.smithi017.stdout: "ready_epoch": 0, 2024-09-11T09:51:54.176 INFO:teuthology.orchestra.run.smithi017.stdout: "last_epoch_started": 0, 2024-09-11T09:51:54.176 INFO:teuthology.orchestra.run.smithi017.stdout: "last_epoch_clean": 0, 2024-09-11T09:51:54.176 INFO:teuthology.orchestra.run.smithi017.stdout: "source_version": "0'0", 2024-09-11T09:51:54.176 INFO:teuthology.orchestra.run.smithi017.stdout: "target_version": "0'0" 2024-09-11T09:51:54.176 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:54.176 INFO:teuthology.orchestra.run.smithi017.stdout: "last_change": "69", 2024-09-11T09:51:54.176 INFO:teuthology.orchestra.run.smithi017.stdout: "last_force_op_resend": "0", 2024-09-11T09:51:54.176 INFO:teuthology.orchestra.run.smithi017.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-11T09:51:54.176 INFO:teuthology.orchestra.run.smithi017.stdout: "last_force_op_resend_preluminous": "0", 2024-09-11T09:51:54.176 INFO:teuthology.orchestra.run.smithi017.stdout: "auid": 0, 2024-09-11T09:51:54.176 INFO:teuthology.orchestra.run.smithi017.stdout: "snap_mode": "selfmanaged", 2024-09-11T09:51:54.176 INFO:teuthology.orchestra.run.smithi017.stdout: "snap_seq": 5, 2024-09-11T09:51:54.176 INFO:teuthology.orchestra.run.smithi017.stdout: "snap_epoch": 69, 2024-09-11T09:51:54.176 INFO:teuthology.orchestra.run.smithi017.stdout: "pool_snaps": [], 2024-09-11T09:51:54.177 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_snaps": "[]", 2024-09-11T09:51:54.177 INFO:teuthology.orchestra.run.smithi017.stdout: "quota_max_bytes": 0, 2024-09-11T09:51:54.177 INFO:teuthology.orchestra.run.smithi017.stdout: "quota_max_objects": 0, 2024-09-11T09:51:54.177 INFO:teuthology.orchestra.run.smithi017.stdout: "tiers": [], 2024-09-11T09:51:54.177 INFO:teuthology.orchestra.run.smithi017.stdout: "tier_of": -1, 2024-09-11T09:51:54.177 INFO:teuthology.orchestra.run.smithi017.stdout: "read_tier": -1, 2024-09-11T09:51:54.177 INFO:teuthology.orchestra.run.smithi017.stdout: "write_tier": -1, 2024-09-11T09:51:54.177 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_mode": "none", 2024-09-11T09:51:54.177 INFO:teuthology.orchestra.run.smithi017.stdout: "target_max_bytes": 0, 2024-09-11T09:51:54.177 INFO:teuthology.orchestra.run.smithi017.stdout: "target_max_objects": 0, 2024-09-11T09:51:54.177 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-11T09:51:54.178 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-11T09:51:54.178 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-11T09:51:54.178 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_min_flush_age": 0, 2024-09-11T09:51:54.178 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_min_evict_age": 0, 2024-09-11T09:51:54.178 INFO:teuthology.orchestra.run.smithi017.stdout: "erasure_code_profile": "", 2024-09-11T09:51:54.178 INFO:teuthology.orchestra.run.smithi017.stdout: "hit_set_params": { 2024-09-11T09:51:54.178 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "none" 2024-09-11T09:51:54.178 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:54.178 INFO:teuthology.orchestra.run.smithi017.stdout: "hit_set_period": 0, 2024-09-11T09:51:54.178 INFO:teuthology.orchestra.run.smithi017.stdout: "hit_set_count": 0, 2024-09-11T09:51:54.178 INFO:teuthology.orchestra.run.smithi017.stdout: "use_gmt_hitset": true, 2024-09-11T09:51:54.178 INFO:teuthology.orchestra.run.smithi017.stdout: "min_read_recency_for_promote": 0, 2024-09-11T09:51:54.178 INFO:teuthology.orchestra.run.smithi017.stdout: "min_write_recency_for_promote": 0, 2024-09-11T09:51:54.178 INFO:teuthology.orchestra.run.smithi017.stdout: "hit_set_grade_decay_rate": 0, 2024-09-11T09:51:54.179 INFO:teuthology.orchestra.run.smithi017.stdout: "hit_set_search_last_n": 0, 2024-09-11T09:51:54.179 INFO:teuthology.orchestra.run.smithi017.stdout: "grade_table": [], 2024-09-11T09:51:54.179 INFO:teuthology.orchestra.run.smithi017.stdout: "stripe_width": 0, 2024-09-11T09:51:54.179 INFO:teuthology.orchestra.run.smithi017.stdout: "expected_num_objects": 0, 2024-09-11T09:51:54.179 INFO:teuthology.orchestra.run.smithi017.stdout: "fast_read": false, 2024-09-11T09:51:54.179 INFO:teuthology.orchestra.run.smithi017.stdout: "options": {}, 2024-09-11T09:51:54.179 INFO:teuthology.orchestra.run.smithi017.stdout: "application_metadata": { 2024-09-11T09:51:54.179 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd": {} 2024-09-11T09:51:54.179 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:54.179 INFO:teuthology.orchestra.run.smithi017.stdout: "read_balance": { 2024-09-11T09:51:54.179 INFO:teuthology.orchestra.run.smithi017.stdout: "score_type": "Fair distribution", 2024-09-11T09:51:54.179 INFO:teuthology.orchestra.run.smithi017.stdout: "score_acting": 1.5099999904632568, 2024-09-11T09:51:54.179 INFO:teuthology.orchestra.run.smithi017.stdout: "score_stable": 1.5099999904632568, 2024-09-11T09:51:54.179 INFO:teuthology.orchestra.run.smithi017.stdout: "optimal_score": 0.82999998331069946, 2024-09-11T09:51:54.180 INFO:teuthology.orchestra.run.smithi017.stdout: "raw_score_acting": 1.25, 2024-09-11T09:51:54.180 INFO:teuthology.orchestra.run.smithi017.stdout: "raw_score_stable": 1.25, 2024-09-11T09:51:54.180 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity_weighted": 1, 2024-09-11T09:51:54.180 INFO:teuthology.orchestra.run.smithi017.stdout: "average_primary_affinity": 1, 2024-09-11T09:51:54.180 INFO:teuthology.orchestra.run.smithi017.stdout: "average_primary_affinity_weighted": 1 2024-09-11T09:51:54.180 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:54.180 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:54.180 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:54.180 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 3, 2024-09-11T09:51:54.180 INFO:teuthology.orchestra.run.smithi017.stdout: "pool_name": "new_default_pool27121", 2024-09-11T09:51:54.180 INFO:teuthology.orchestra.run.smithi017.stdout: "create_time": "2024-09-11T09:51:23.493016+0000", 2024-09-11T09:51:54.180 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 8193, 2024-09-11T09:51:54.180 INFO:teuthology.orchestra.run.smithi017.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-11T09:51:54.181 INFO:teuthology.orchestra.run.smithi017.stdout: "type": 1, 2024-09-11T09:51:54.181 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 2, 2024-09-11T09:51:54.181 INFO:teuthology.orchestra.run.smithi017.stdout: "min_size": 1, 2024-09-11T09:51:54.181 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_rule": 0, 2024-09-11T09:51:54.181 INFO:teuthology.orchestra.run.smithi017.stdout: "peering_crush_bucket_count": 0, 2024-09-11T09:51:54.181 INFO:teuthology.orchestra.run.smithi017.stdout: "peering_crush_bucket_target": 0, 2024-09-11T09:51:54.181 INFO:teuthology.orchestra.run.smithi017.stdout: "peering_crush_bucket_barrier": 0, 2024-09-11T09:51:54.181 INFO:teuthology.orchestra.run.smithi017.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-11T09:51:54.181 INFO:teuthology.orchestra.run.smithi017.stdout: "is_stretch_pool": false, 2024-09-11T09:51:54.181 INFO:teuthology.orchestra.run.smithi017.stdout: "object_hash": 2, 2024-09-11T09:51:54.181 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_autoscale_mode": "off", 2024-09-11T09:51:54.181 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_num": 8, 2024-09-11T09:51:54.181 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_placement_num": 8, 2024-09-11T09:51:54.182 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_placement_num_target": 8, 2024-09-11T09:51:54.182 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_num_target": 8, 2024-09-11T09:51:54.182 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_num_pending": 8, 2024-09-11T09:51:54.182 INFO:teuthology.orchestra.run.smithi017.stdout: "last_pg_merge_meta": { 2024-09-11T09:51:54.182 INFO:teuthology.orchestra.run.smithi017.stdout: "source_pgid": "0.0", 2024-09-11T09:51:54.182 INFO:teuthology.orchestra.run.smithi017.stdout: "ready_epoch": 0, 2024-09-11T09:51:54.182 INFO:teuthology.orchestra.run.smithi017.stdout: "last_epoch_started": 0, 2024-09-11T09:51:54.182 INFO:teuthology.orchestra.run.smithi017.stdout: "last_epoch_clean": 0, 2024-09-11T09:51:54.182 INFO:teuthology.orchestra.run.smithi017.stdout: "source_version": "0'0", 2024-09-11T09:51:54.182 INFO:teuthology.orchestra.run.smithi017.stdout: "target_version": "0'0" 2024-09-11T09:51:54.182 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:54.182 INFO:teuthology.orchestra.run.smithi017.stdout: "last_change": "58", 2024-09-11T09:51:54.182 INFO:teuthology.orchestra.run.smithi017.stdout: "last_force_op_resend": "0", 2024-09-11T09:51:54.182 INFO:teuthology.orchestra.run.smithi017.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-11T09:51:54.183 INFO:teuthology.orchestra.run.smithi017.stdout: "last_force_op_resend_preluminous": "0", 2024-09-11T09:51:54.183 INFO:teuthology.orchestra.run.smithi017.stdout: "auid": 0, 2024-09-11T09:51:54.183 INFO:teuthology.orchestra.run.smithi017.stdout: "snap_mode": "selfmanaged", 2024-09-11T09:51:54.183 INFO:teuthology.orchestra.run.smithi017.stdout: "snap_seq": 2, 2024-09-11T09:51:54.183 INFO:teuthology.orchestra.run.smithi017.stdout: "snap_epoch": 58, 2024-09-11T09:51:54.183 INFO:teuthology.orchestra.run.smithi017.stdout: "pool_snaps": [], 2024-09-11T09:51:54.183 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_snaps": "[]", 2024-09-11T09:51:54.183 INFO:teuthology.orchestra.run.smithi017.stdout: "quota_max_bytes": 0, 2024-09-11T09:51:54.183 INFO:teuthology.orchestra.run.smithi017.stdout: "quota_max_objects": 0, 2024-09-11T09:51:54.183 INFO:teuthology.orchestra.run.smithi017.stdout: "tiers": [], 2024-09-11T09:51:54.183 INFO:teuthology.orchestra.run.smithi017.stdout: "tier_of": -1, 2024-09-11T09:51:54.183 INFO:teuthology.orchestra.run.smithi017.stdout: "read_tier": -1, 2024-09-11T09:51:54.183 INFO:teuthology.orchestra.run.smithi017.stdout: "write_tier": -1, 2024-09-11T09:51:54.183 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_mode": "none", 2024-09-11T09:51:54.184 INFO:teuthology.orchestra.run.smithi017.stdout: "target_max_bytes": 0, 2024-09-11T09:51:54.184 INFO:teuthology.orchestra.run.smithi017.stdout: "target_max_objects": 0, 2024-09-11T09:51:54.184 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-11T09:51:54.184 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-11T09:51:54.184 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-11T09:51:54.184 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_min_flush_age": 0, 2024-09-11T09:51:54.184 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_min_evict_age": 0, 2024-09-11T09:51:54.184 INFO:teuthology.orchestra.run.smithi017.stdout: "erasure_code_profile": "", 2024-09-11T09:51:54.184 INFO:teuthology.orchestra.run.smithi017.stdout: "hit_set_params": { 2024-09-11T09:51:54.184 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "none" 2024-09-11T09:51:54.184 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:54.184 INFO:teuthology.orchestra.run.smithi017.stdout: "hit_set_period": 0, 2024-09-11T09:51:54.185 INFO:teuthology.orchestra.run.smithi017.stdout: "hit_set_count": 0, 2024-09-11T09:51:54.185 INFO:teuthology.orchestra.run.smithi017.stdout: "use_gmt_hitset": true, 2024-09-11T09:51:54.185 INFO:teuthology.orchestra.run.smithi017.stdout: "min_read_recency_for_promote": 0, 2024-09-11T09:51:54.185 INFO:teuthology.orchestra.run.smithi017.stdout: "min_write_recency_for_promote": 0, 2024-09-11T09:51:54.185 INFO:teuthology.orchestra.run.smithi017.stdout: "hit_set_grade_decay_rate": 0, 2024-09-11T09:51:54.185 INFO:teuthology.orchestra.run.smithi017.stdout: "hit_set_search_last_n": 0, 2024-09-11T09:51:54.185 INFO:teuthology.orchestra.run.smithi017.stdout: "grade_table": [], 2024-09-11T09:51:54.185 INFO:teuthology.orchestra.run.smithi017.stdout: "stripe_width": 0, 2024-09-11T09:51:54.185 INFO:teuthology.orchestra.run.smithi017.stdout: "expected_num_objects": 0, 2024-09-11T09:51:54.185 INFO:teuthology.orchestra.run.smithi017.stdout: "fast_read": false, 2024-09-11T09:51:54.185 INFO:teuthology.orchestra.run.smithi017.stdout: "options": {}, 2024-09-11T09:51:54.185 INFO:teuthology.orchestra.run.smithi017.stdout: "application_metadata": { 2024-09-11T09:51:54.185 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd": {} 2024-09-11T09:51:54.186 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:54.186 INFO:teuthology.orchestra.run.smithi017.stdout: "read_balance": { 2024-09-11T09:51:54.186 INFO:teuthology.orchestra.run.smithi017.stdout: "score_type": "Fair distribution", 2024-09-11T09:51:54.186 INFO:teuthology.orchestra.run.smithi017.stdout: "score_acting": 2.2699999809265137, 2024-09-11T09:51:54.186 INFO:teuthology.orchestra.run.smithi017.stdout: "score_stable": 2.2699999809265137, 2024-09-11T09:51:54.186 INFO:teuthology.orchestra.run.smithi017.stdout: "optimal_score": 0.82999998331069946, 2024-09-11T09:51:54.186 INFO:teuthology.orchestra.run.smithi017.stdout: "raw_score_acting": 1.8799999952316284, 2024-09-11T09:51:54.186 INFO:teuthology.orchestra.run.smithi017.stdout: "raw_score_stable": 1.8799999952316284, 2024-09-11T09:51:54.186 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity_weighted": 1, 2024-09-11T09:51:54.186 INFO:teuthology.orchestra.run.smithi017.stdout: "average_primary_affinity": 1, 2024-09-11T09:51:54.186 INFO:teuthology.orchestra.run.smithi017.stdout: "average_primary_affinity_weighted": 1 2024-09-11T09:51:54.186 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:54.186 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:54.186 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-11T09:51:54.187 INFO:teuthology.orchestra.run.smithi017.stdout: "osds": [ 2024-09-11T09:51:54.187 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:54.187 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 0, 2024-09-11T09:51:54.187 INFO:teuthology.orchestra.run.smithi017.stdout: "uuid": "4829b7a1-d607-4a56-b6d2-55036cb8acf8", 2024-09-11T09:51:54.187 INFO:teuthology.orchestra.run.smithi017.stdout: "up": 1, 2024-09-11T09:51:54.187 INFO:teuthology.orchestra.run.smithi017.stdout: "in": 1, 2024-09-11T09:51:54.187 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 1, 2024-09-11T09:51:54.187 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity": 1, 2024-09-11T09:51:54.187 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_begin": 0, 2024-09-11T09:51:54.187 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_end": 0, 2024-09-11T09:51:54.187 INFO:teuthology.orchestra.run.smithi017.stdout: "up_from": 12, 2024-09-11T09:51:54.187 INFO:teuthology.orchestra.run.smithi017.stdout: "up_thru": 66, 2024-09-11T09:51:54.187 INFO:teuthology.orchestra.run.smithi017.stdout: "down_at": 0, 2024-09-11T09:51:54.187 INFO:teuthology.orchestra.run.smithi017.stdout: "lost_at": 0, 2024-09-11T09:51:54.188 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-11T09:51:54.188 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-11T09:51:54.188 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:54.188 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-11T09:51:54.188 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6808", 2024-09-11T09:51:54.188 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 4154449929 2024-09-11T09:51:54.188 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:54.188 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:54.188 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-11T09:51:54.188 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6809", 2024-09-11T09:51:54.188 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 4154449929 2024-09-11T09:51:54.188 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:54.188 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:54.188 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:54.189 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addrs": { 2024-09-11T09:51:54.189 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-11T09:51:54.189 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:54.189 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-11T09:51:54.189 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6810", 2024-09-11T09:51:54.189 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 4154449929 2024-09-11T09:51:54.189 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:54.189 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:54.189 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-11T09:51:54.189 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6811", 2024-09-11T09:51:54.189 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 4154449929 2024-09-11T09:51:54.189 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:54.189 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:54.189 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:54.190 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addrs": { 2024-09-11T09:51:54.190 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-11T09:51:54.190 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:54.190 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-11T09:51:54.190 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6814", 2024-09-11T09:51:54.190 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 4154449929 2024-09-11T09:51:54.190 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:54.190 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:54.190 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-11T09:51:54.190 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6815", 2024-09-11T09:51:54.190 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 4154449929 2024-09-11T09:51:54.190 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:54.191 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:54.191 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:54.191 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addrs": { 2024-09-11T09:51:54.191 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-11T09:51:54.191 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:54.191 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-11T09:51:54.191 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6812", 2024-09-11T09:51:54.191 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 4154449929 2024-09-11T09:51:54.191 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:54.191 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:54.191 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-11T09:51:54.191 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6813", 2024-09-11T09:51:54.191 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 4154449929 2024-09-11T09:51:54.191 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:54.192 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:54.192 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:54.192 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6809/4154449929", 2024-09-11T09:51:54.192 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addr": "172.21.15.17:6811/4154449929", 2024-09-11T09:51:54.192 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addr": "172.21.15.17:6815/4154449929", 2024-09-11T09:51:54.192 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addr": "172.21.15.17:6813/4154449929", 2024-09-11T09:51:54.192 INFO:teuthology.orchestra.run.smithi017.stdout: "state": [ 2024-09-11T09:51:54.192 INFO:teuthology.orchestra.run.smithi017.stdout: "exists", 2024-09-11T09:51:54.192 INFO:teuthology.orchestra.run.smithi017.stdout: "up" 2024-09-11T09:51:54.192 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:54.192 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:54.192 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:54.192 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 1, 2024-09-11T09:51:54.193 INFO:teuthology.orchestra.run.smithi017.stdout: "uuid": "ab1d5df0-3c0e-44cd-88a0-96b07964148b", 2024-09-11T09:51:54.193 INFO:teuthology.orchestra.run.smithi017.stdout: "up": 1, 2024-09-11T09:51:54.193 INFO:teuthology.orchestra.run.smithi017.stdout: "in": 1, 2024-09-11T09:51:54.193 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 1, 2024-09-11T09:51:54.193 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity": 1, 2024-09-11T09:51:54.193 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_begin": 0, 2024-09-11T09:51:54.193 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_end": 0, 2024-09-11T09:51:54.193 INFO:teuthology.orchestra.run.smithi017.stdout: "up_from": 11, 2024-09-11T09:51:54.193 INFO:teuthology.orchestra.run.smithi017.stdout: "up_thru": 54, 2024-09-11T09:51:54.193 INFO:teuthology.orchestra.run.smithi017.stdout: "down_at": 0, 2024-09-11T09:51:54.193 INFO:teuthology.orchestra.run.smithi017.stdout: "lost_at": 0, 2024-09-11T09:51:54.193 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-11T09:51:54.193 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-11T09:51:54.194 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:54.194 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-11T09:51:54.194 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6800", 2024-09-11T09:51:54.194 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 4139511644 2024-09-11T09:51:54.194 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:54.194 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:54.194 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-11T09:51:54.194 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6801", 2024-09-11T09:51:54.194 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 4139511644 2024-09-11T09:51:54.194 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:54.194 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:54.194 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:54.194 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addrs": { 2024-09-11T09:51:54.195 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-11T09:51:54.195 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:54.195 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-11T09:51:54.195 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6802", 2024-09-11T09:51:54.195 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 4139511644 2024-09-11T09:51:54.195 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:54.195 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:54.195 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-11T09:51:54.195 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6803", 2024-09-11T09:51:54.195 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 4139511644 2024-09-11T09:51:54.195 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:54.195 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:54.195 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:54.196 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addrs": { 2024-09-11T09:51:54.196 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-11T09:51:54.196 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:54.196 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-11T09:51:54.196 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6806", 2024-09-11T09:51:54.196 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 4139511644 2024-09-11T09:51:54.196 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:54.196 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:54.196 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-11T09:51:54.196 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6807", 2024-09-11T09:51:54.196 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 4139511644 2024-09-11T09:51:54.196 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:54.196 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:54.196 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:54.197 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addrs": { 2024-09-11T09:51:54.197 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-11T09:51:54.197 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:54.197 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-11T09:51:54.197 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6804", 2024-09-11T09:51:54.197 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 4139511644 2024-09-11T09:51:54.197 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:54.197 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:54.197 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-11T09:51:54.197 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6805", 2024-09-11T09:51:54.197 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 4139511644 2024-09-11T09:51:54.197 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:54.197 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:54.198 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:54.198 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6801/4139511644", 2024-09-11T09:51:54.198 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addr": "172.21.15.17:6803/4139511644", 2024-09-11T09:51:54.198 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addr": "172.21.15.17:6807/4139511644", 2024-09-11T09:51:54.198 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addr": "172.21.15.17:6805/4139511644", 2024-09-11T09:51:54.198 INFO:teuthology.orchestra.run.smithi017.stdout: "state": [ 2024-09-11T09:51:54.198 INFO:teuthology.orchestra.run.smithi017.stdout: "exists", 2024-09-11T09:51:54.198 INFO:teuthology.orchestra.run.smithi017.stdout: "up" 2024-09-11T09:51:54.198 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:54.198 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:54.198 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:54.198 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 2, 2024-09-11T09:51:54.198 INFO:teuthology.orchestra.run.smithi017.stdout: "uuid": "4c8e50db-b49c-4d44-a913-47b5b30f8699", 2024-09-11T09:51:54.199 INFO:teuthology.orchestra.run.smithi017.stdout: "up": 1, 2024-09-11T09:51:54.199 INFO:teuthology.orchestra.run.smithi017.stdout: "in": 1, 2024-09-11T09:51:54.199 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 1, 2024-09-11T09:51:54.199 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity": 1, 2024-09-11T09:51:54.199 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_begin": 0, 2024-09-11T09:51:54.199 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_end": 0, 2024-09-11T09:51:54.199 INFO:teuthology.orchestra.run.smithi017.stdout: "up_from": 11, 2024-09-11T09:51:54.199 INFO:teuthology.orchestra.run.smithi017.stdout: "up_thru": 54, 2024-09-11T09:51:54.199 INFO:teuthology.orchestra.run.smithi017.stdout: "down_at": 0, 2024-09-11T09:51:54.199 INFO:teuthology.orchestra.run.smithi017.stdout: "lost_at": 0, 2024-09-11T09:51:54.199 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-11T09:51:54.199 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-11T09:51:54.199 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:54.200 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-11T09:51:54.200 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6816", 2024-09-11T09:51:54.200 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1151282506 2024-09-11T09:51:54.200 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:54.200 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:54.200 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-11T09:51:54.200 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6817", 2024-09-11T09:51:54.200 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1151282506 2024-09-11T09:51:54.200 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:54.200 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:54.200 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:54.200 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addrs": { 2024-09-11T09:51:54.200 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-11T09:51:54.201 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:54.201 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-11T09:51:54.201 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6818", 2024-09-11T09:51:54.201 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1151282506 2024-09-11T09:51:54.201 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:54.201 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:54.201 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-11T09:51:54.201 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6819", 2024-09-11T09:51:54.201 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1151282506 2024-09-11T09:51:54.201 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:54.201 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:54.201 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:54.201 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addrs": { 2024-09-11T09:51:54.202 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-11T09:51:54.202 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:54.202 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-11T09:51:54.202 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6822", 2024-09-11T09:51:54.202 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1151282506 2024-09-11T09:51:54.202 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:54.202 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:54.202 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-11T09:51:54.202 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6823", 2024-09-11T09:51:54.202 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1151282506 2024-09-11T09:51:54.202 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:54.202 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:54.202 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:54.203 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addrs": { 2024-09-11T09:51:54.203 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-11T09:51:54.203 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:54.203 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-11T09:51:54.203 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6820", 2024-09-11T09:51:54.203 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1151282506 2024-09-11T09:51:54.203 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:54.203 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:54.203 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-11T09:51:54.203 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6821", 2024-09-11T09:51:54.203 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1151282506 2024-09-11T09:51:54.203 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:54.203 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:54.204 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:54.204 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6817/1151282506", 2024-09-11T09:51:54.204 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addr": "172.21.15.17:6819/1151282506", 2024-09-11T09:51:54.204 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addr": "172.21.15.17:6823/1151282506", 2024-09-11T09:51:54.204 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addr": "172.21.15.17:6821/1151282506", 2024-09-11T09:51:54.204 INFO:teuthology.orchestra.run.smithi017.stdout: "state": [ 2024-09-11T09:51:54.204 INFO:teuthology.orchestra.run.smithi017.stdout: "exists", 2024-09-11T09:51:54.204 INFO:teuthology.orchestra.run.smithi017.stdout: "up" 2024-09-11T09:51:54.204 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:54.204 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:54.204 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:54.204 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 3, 2024-09-11T09:51:54.204 INFO:teuthology.orchestra.run.smithi017.stdout: "uuid": "37abc55b-a1eb-49d5-b1ff-b507ee2cb5a6", 2024-09-11T09:51:54.205 INFO:teuthology.orchestra.run.smithi017.stdout: "up": 1, 2024-09-11T09:51:54.205 INFO:teuthology.orchestra.run.smithi017.stdout: "in": 1, 2024-09-11T09:51:54.205 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 1, 2024-09-11T09:51:54.205 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity": 1, 2024-09-11T09:51:54.205 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_begin": 0, 2024-09-11T09:51:54.205 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_end": 0, 2024-09-11T09:51:54.205 INFO:teuthology.orchestra.run.smithi017.stdout: "up_from": 12, 2024-09-11T09:51:54.205 INFO:teuthology.orchestra.run.smithi017.stdout: "up_thru": 35, 2024-09-11T09:51:54.205 INFO:teuthology.orchestra.run.smithi017.stdout: "down_at": 0, 2024-09-11T09:51:54.205 INFO:teuthology.orchestra.run.smithi017.stdout: "lost_at": 0, 2024-09-11T09:51:54.205 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-11T09:51:54.205 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-11T09:51:54.205 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:54.206 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-11T09:51:54.206 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.96:6816", 2024-09-11T09:51:54.206 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 332854807 2024-09-11T09:51:54.206 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:54.206 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:54.206 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-11T09:51:54.206 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.96:6817", 2024-09-11T09:51:54.206 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 332854807 2024-09-11T09:51:54.206 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:54.206 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:54.206 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:54.206 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addrs": { 2024-09-11T09:51:54.206 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-11T09:51:54.206 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:54.206 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-11T09:51:54.207 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.96:6818", 2024-09-11T09:51:54.207 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 332854807 2024-09-11T09:51:54.207 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:54.207 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:54.207 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-11T09:51:54.207 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.96:6819", 2024-09-11T09:51:54.207 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 332854807 2024-09-11T09:51:54.207 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:54.207 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:54.207 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:54.207 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addrs": { 2024-09-11T09:51:54.207 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-11T09:51:54.207 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:54.208 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-11T09:51:54.208 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.96:6822", 2024-09-11T09:51:54.208 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 332854807 2024-09-11T09:51:54.208 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:54.208 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:54.208 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-11T09:51:54.208 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.96:6823", 2024-09-11T09:51:54.208 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 332854807 2024-09-11T09:51:54.208 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:54.208 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:54.208 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:54.208 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addrs": { 2024-09-11T09:51:54.208 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-11T09:51:54.209 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:54.209 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-11T09:51:54.209 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.96:6820", 2024-09-11T09:51:54.209 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 332854807 2024-09-11T09:51:54.209 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:54.209 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:54.209 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-11T09:51:54.209 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.96:6821", 2024-09-11T09:51:54.209 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 332854807 2024-09-11T09:51:54.209 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:54.209 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:54.209 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:54.209 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.96:6817/332854807", 2024-09-11T09:51:54.210 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addr": "172.21.15.96:6819/332854807", 2024-09-11T09:51:54.210 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addr": "172.21.15.96:6823/332854807", 2024-09-11T09:51:54.210 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addr": "172.21.15.96:6821/332854807", 2024-09-11T09:51:54.210 INFO:teuthology.orchestra.run.smithi017.stdout: "state": [ 2024-09-11T09:51:54.210 INFO:teuthology.orchestra.run.smithi017.stdout: "exists", 2024-09-11T09:51:54.210 INFO:teuthology.orchestra.run.smithi017.stdout: "up" 2024-09-11T09:51:54.210 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:54.210 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:54.210 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:54.210 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 4, 2024-09-11T09:51:54.210 INFO:teuthology.orchestra.run.smithi017.stdout: "uuid": "067d9586-4f5a-4d17-96c5-86a7b8bcd114", 2024-09-11T09:51:54.210 INFO:teuthology.orchestra.run.smithi017.stdout: "up": 1, 2024-09-11T09:51:54.210 INFO:teuthology.orchestra.run.smithi017.stdout: "in": 1, 2024-09-11T09:51:54.210 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 1, 2024-09-11T09:51:54.211 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity": 1, 2024-09-11T09:51:54.211 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_begin": 11, 2024-09-11T09:51:54.211 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_end": 54, 2024-09-11T09:51:54.211 INFO:teuthology.orchestra.run.smithi017.stdout: "up_from": 66, 2024-09-11T09:51:54.211 INFO:teuthology.orchestra.run.smithi017.stdout: "up_thru": 66, 2024-09-11T09:51:54.211 INFO:teuthology.orchestra.run.smithi017.stdout: "down_at": 55, 2024-09-11T09:51:54.211 INFO:teuthology.orchestra.run.smithi017.stdout: "lost_at": 0, 2024-09-11T09:51:54.211 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-11T09:51:54.211 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-11T09:51:54.211 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:54.211 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-11T09:51:54.211 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.96:6800", 2024-09-11T09:51:54.211 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3374675329 2024-09-11T09:51:54.212 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:54.212 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:54.212 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-11T09:51:54.212 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.96:6801", 2024-09-11T09:51:54.212 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3374675329 2024-09-11T09:51:54.212 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:54.212 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:54.212 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:54.212 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addrs": { 2024-09-11T09:51:54.212 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-11T09:51:54.212 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:54.212 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-11T09:51:54.212 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.96:6802", 2024-09-11T09:51:54.212 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3374675329 2024-09-11T09:51:54.213 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:54.213 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:54.213 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-11T09:51:54.213 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.96:6803", 2024-09-11T09:51:54.213 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3374675329 2024-09-11T09:51:54.213 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:54.213 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:54.213 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:54.213 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addrs": { 2024-09-11T09:51:54.213 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-11T09:51:54.213 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:54.213 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-11T09:51:54.213 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.96:6806", 2024-09-11T09:51:54.213 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3374675329 2024-09-11T09:51:54.214 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:54.214 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:54.214 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-11T09:51:54.214 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.96:6807", 2024-09-11T09:51:54.214 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3374675329 2024-09-11T09:51:54.214 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:54.214 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:54.214 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:54.214 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addrs": { 2024-09-11T09:51:54.214 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-11T09:51:54.214 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:54.214 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-11T09:51:54.214 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.96:6804", 2024-09-11T09:51:54.215 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3374675329 2024-09-11T09:51:54.215 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:54.215 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:54.215 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-11T09:51:54.215 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.96:6805", 2024-09-11T09:51:54.215 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3374675329 2024-09-11T09:51:54.215 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:54.215 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:54.215 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:54.215 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.96:6801/3374675329", 2024-09-11T09:51:54.215 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addr": "172.21.15.96:6803/3374675329", 2024-09-11T09:51:54.216 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addr": "172.21.15.96:6807/3374675329", 2024-09-11T09:51:54.216 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addr": "172.21.15.96:6805/3374675329", 2024-09-11T09:51:54.216 INFO:teuthology.orchestra.run.smithi017.stdout: "state": [ 2024-09-11T09:51:54.216 INFO:teuthology.orchestra.run.smithi017.stdout: "exists", 2024-09-11T09:51:54.216 INFO:teuthology.orchestra.run.smithi017.stdout: "up" 2024-09-11T09:51:54.216 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:54.216 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:54.216 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:54.216 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 5, 2024-09-11T09:51:54.216 INFO:teuthology.orchestra.run.smithi017.stdout: "uuid": "8ff7a724-0a3a-4aa2-baf1-fd52f88d476c", 2024-09-11T09:51:54.216 INFO:teuthology.orchestra.run.smithi017.stdout: "up": 1, 2024-09-11T09:51:54.216 INFO:teuthology.orchestra.run.smithi017.stdout: "in": 1, 2024-09-11T09:51:54.216 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 1, 2024-09-11T09:51:54.216 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity": 1, 2024-09-11T09:51:54.217 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_begin": 0, 2024-09-11T09:51:54.217 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_end": 0, 2024-09-11T09:51:54.217 INFO:teuthology.orchestra.run.smithi017.stdout: "up_from": 11, 2024-09-11T09:51:54.217 INFO:teuthology.orchestra.run.smithi017.stdout: "up_thru": 54, 2024-09-11T09:51:54.217 INFO:teuthology.orchestra.run.smithi017.stdout: "down_at": 0, 2024-09-11T09:51:54.217 INFO:teuthology.orchestra.run.smithi017.stdout: "lost_at": 0, 2024-09-11T09:51:54.217 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-11T09:51:54.217 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-11T09:51:54.217 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:54.217 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-11T09:51:54.217 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.96:6808", 2024-09-11T09:51:54.217 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 509935370 2024-09-11T09:51:54.217 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:54.217 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:54.218 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-11T09:51:54.218 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.96:6809", 2024-09-11T09:51:54.218 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 509935370 2024-09-11T09:51:54.218 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:54.218 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:54.218 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:54.218 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addrs": { 2024-09-11T09:51:54.218 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-11T09:51:54.218 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:54.218 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-11T09:51:54.218 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.96:6810", 2024-09-11T09:51:54.218 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 509935370 2024-09-11T09:51:54.218 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:54.218 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:54.219 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-11T09:51:54.219 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.96:6811", 2024-09-11T09:51:54.219 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 509935370 2024-09-11T09:51:54.219 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:54.219 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:54.219 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:54.219 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addrs": { 2024-09-11T09:51:54.219 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-11T09:51:54.219 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:54.219 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-11T09:51:54.219 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.96:6814", 2024-09-11T09:51:54.219 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 509935370 2024-09-11T09:51:54.219 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:54.219 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:54.220 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-11T09:51:54.220 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.96:6815", 2024-09-11T09:51:54.220 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 509935370 2024-09-11T09:51:54.220 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:54.220 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:54.220 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:54.220 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addrs": { 2024-09-11T09:51:54.220 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-11T09:51:54.220 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:54.220 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-11T09:51:54.220 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.96:6812", 2024-09-11T09:51:54.220 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 509935370 2024-09-11T09:51:54.220 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:54.220 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:54.221 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-11T09:51:54.221 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.96:6813", 2024-09-11T09:51:54.221 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 509935370 2024-09-11T09:51:54.221 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:54.221 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:54.221 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:54.221 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.96:6809/509935370", 2024-09-11T09:51:54.221 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addr": "172.21.15.96:6811/509935370", 2024-09-11T09:51:54.221 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addr": "172.21.15.96:6815/509935370", 2024-09-11T09:51:54.221 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addr": "172.21.15.96:6813/509935370", 2024-09-11T09:51:54.221 INFO:teuthology.orchestra.run.smithi017.stdout: "state": [ 2024-09-11T09:51:54.221 INFO:teuthology.orchestra.run.smithi017.stdout: "exists", 2024-09-11T09:51:54.221 INFO:teuthology.orchestra.run.smithi017.stdout: "up" 2024-09-11T09:51:54.222 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:54.222 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:54.222 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-11T09:51:54.222 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_xinfo": [ 2024-09-11T09:51:54.222 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:54.222 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 0, 2024-09-11T09:51:54.222 INFO:teuthology.orchestra.run.smithi017.stdout: "down_stamp": "0.000000", 2024-09-11T09:51:54.222 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_probability": 0, 2024-09-11T09:51:54.222 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_interval": 0, 2024-09-11T09:51:54.222 INFO:teuthology.orchestra.run.smithi017.stdout: "features": 4540701547738038271, 2024-09-11T09:51:54.222 INFO:teuthology.orchestra.run.smithi017.stdout: "old_weight": 0, 2024-09-11T09:51:54.222 INFO:teuthology.orchestra.run.smithi017.stdout: "last_purged_snaps_scrub": "2024-09-11T09:50:08.513270+0000", 2024-09-11T09:51:54.222 INFO:teuthology.orchestra.run.smithi017.stdout: "dead_epoch": 0 2024-09-11T09:51:54.222 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:54.222 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:54.223 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 1, 2024-09-11T09:51:54.223 INFO:teuthology.orchestra.run.smithi017.stdout: "down_stamp": "0.000000", 2024-09-11T09:51:54.223 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_probability": 0, 2024-09-11T09:51:54.223 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_interval": 0, 2024-09-11T09:51:54.223 INFO:teuthology.orchestra.run.smithi017.stdout: "features": 4540701547738038271, 2024-09-11T09:51:54.223 INFO:teuthology.orchestra.run.smithi017.stdout: "old_weight": 0, 2024-09-11T09:51:54.223 INFO:teuthology.orchestra.run.smithi017.stdout: "last_purged_snaps_scrub": "2024-09-11T09:50:07.522863+0000", 2024-09-11T09:51:54.223 INFO:teuthology.orchestra.run.smithi017.stdout: "dead_epoch": 0 2024-09-11T09:51:54.223 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:54.223 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:54.223 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 2, 2024-09-11T09:51:54.223 INFO:teuthology.orchestra.run.smithi017.stdout: "down_stamp": "0.000000", 2024-09-11T09:51:54.223 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_probability": 0, 2024-09-11T09:51:54.223 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_interval": 0, 2024-09-11T09:51:54.224 INFO:teuthology.orchestra.run.smithi017.stdout: "features": 4540701547738038271, 2024-09-11T09:51:54.224 INFO:teuthology.orchestra.run.smithi017.stdout: "old_weight": 0, 2024-09-11T09:51:54.224 INFO:teuthology.orchestra.run.smithi017.stdout: "last_purged_snaps_scrub": "2024-09-11T09:50:08.134364+0000", 2024-09-11T09:51:54.224 INFO:teuthology.orchestra.run.smithi017.stdout: "dead_epoch": 0 2024-09-11T09:51:54.224 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:54.224 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:54.224 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 3, 2024-09-11T09:51:54.224 INFO:teuthology.orchestra.run.smithi017.stdout: "down_stamp": "0.000000", 2024-09-11T09:51:54.224 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_probability": 0, 2024-09-11T09:51:54.224 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_interval": 0, 2024-09-11T09:51:54.224 INFO:teuthology.orchestra.run.smithi017.stdout: "features": 4540701547738038271, 2024-09-11T09:51:54.224 INFO:teuthology.orchestra.run.smithi017.stdout: "old_weight": 0, 2024-09-11T09:51:54.224 INFO:teuthology.orchestra.run.smithi017.stdout: "last_purged_snaps_scrub": "2024-09-11T09:50:08.622366+0000", 2024-09-11T09:51:54.224 INFO:teuthology.orchestra.run.smithi017.stdout: "dead_epoch": 0 2024-09-11T09:51:54.225 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:54.225 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:54.225 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 4, 2024-09-11T09:51:54.225 INFO:teuthology.orchestra.run.smithi017.stdout: "down_stamp": "0.000000", 2024-09-11T09:51:54.225 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_probability": 0, 2024-09-11T09:51:54.225 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_interval": 0, 2024-09-11T09:51:54.225 INFO:teuthology.orchestra.run.smithi017.stdout: "features": 4540701547738038271, 2024-09-11T09:51:54.225 INFO:teuthology.orchestra.run.smithi017.stdout: "old_weight": 0, 2024-09-11T09:51:54.225 INFO:teuthology.orchestra.run.smithi017.stdout: "last_purged_snaps_scrub": "2024-09-11T09:50:08.156142+0000", 2024-09-11T09:51:54.225 INFO:teuthology.orchestra.run.smithi017.stdout: "dead_epoch": 55 2024-09-11T09:51:54.225 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:54.225 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:54.225 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 5, 2024-09-11T09:51:54.226 INFO:teuthology.orchestra.run.smithi017.stdout: "down_stamp": "0.000000", 2024-09-11T09:51:54.226 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_probability": 0, 2024-09-11T09:51:54.226 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_interval": 0, 2024-09-11T09:51:54.226 INFO:teuthology.orchestra.run.smithi017.stdout: "features": 4540701547738038271, 2024-09-11T09:51:54.226 INFO:teuthology.orchestra.run.smithi017.stdout: "old_weight": 0, 2024-09-11T09:51:54.226 INFO:teuthology.orchestra.run.smithi017.stdout: "last_purged_snaps_scrub": "2024-09-11T09:50:08.108116+0000", 2024-09-11T09:51:54.226 INFO:teuthology.orchestra.run.smithi017.stdout: "dead_epoch": 0 2024-09-11T09:51:54.226 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:54.226 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-11T09:51:54.226 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_upmap": [], 2024-09-11T09:51:54.226 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_upmap_items": [ 2024-09-11T09:51:54.226 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:54.226 INFO:teuthology.orchestra.run.smithi017.stdout: "pgid": "2.7", 2024-09-11T09:51:54.227 INFO:teuthology.orchestra.run.smithi017.stdout: "mappings": [ 2024-09-11T09:51:54.227 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:54.227 INFO:teuthology.orchestra.run.smithi017.stdout: "from": 4, 2024-09-11T09:51:54.227 INFO:teuthology.orchestra.run.smithi017.stdout: "to": 5 2024-09-11T09:51:54.227 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:54.227 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:54.227 INFO:teuthology.orchestra.run.smithi017.stdout: "from": 3, 2024-09-11T09:51:54.227 INFO:teuthology.orchestra.run.smithi017.stdout: "to": 0 2024-09-11T09:51:54.227 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:54.227 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:54.227 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:54.227 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-11T09:51:54.227 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_upmap_primaries": [], 2024-09-11T09:51:54.227 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_temp": [], 2024-09-11T09:51:54.227 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_temp": [], 2024-09-11T09:51:54.227 INFO:teuthology.orchestra.run.smithi017.stdout: "blocklist": {}, 2024-09-11T09:51:54.228 INFO:teuthology.orchestra.run.smithi017.stdout: "range_blocklist": {}, 2024-09-11T09:51:54.228 INFO:teuthology.orchestra.run.smithi017.stdout: "erasure_code_profiles": { 2024-09-11T09:51:54.228 INFO:teuthology.orchestra.run.smithi017.stdout: "default": { 2024-09-11T09:51:54.228 INFO:teuthology.orchestra.run.smithi017.stdout: "crush-failure-domain": "osd", 2024-09-11T09:51:54.228 INFO:teuthology.orchestra.run.smithi017.stdout: "k": "2", 2024-09-11T09:51:54.228 INFO:teuthology.orchestra.run.smithi017.stdout: "m": "1", 2024-09-11T09:51:54.228 INFO:teuthology.orchestra.run.smithi017.stdout: "plugin": "jerasure", 2024-09-11T09:51:54.228 INFO:teuthology.orchestra.run.smithi017.stdout: "technique": "reed_sol_van" 2024-09-11T09:51:54.228 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:54.228 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:54.228 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_snaps_queue": [ 2024-09-11T09:51:54.228 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:54.228 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 3, 2024-09-11T09:51:54.229 INFO:teuthology.orchestra.run.smithi017.stdout: "snaps": [ 2024-09-11T09:51:54.229 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:54.229 INFO:teuthology.orchestra.run.smithi017.stdout: "begin": 2, 2024-09-11T09:51:54.229 INFO:teuthology.orchestra.run.smithi017.stdout: "length": 1 2024-09-11T09:51:54.229 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:54.229 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:54.229 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:54.229 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-11T09:51:54.229 INFO:teuthology.orchestra.run.smithi017.stdout: "new_removed_snaps": [], 2024-09-11T09:51:54.229 INFO:teuthology.orchestra.run.smithi017.stdout: "new_purged_snaps": [], 2024-09-11T09:51:54.229 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_node_flags": {}, 2024-09-11T09:51:54.229 INFO:teuthology.orchestra.run.smithi017.stdout: "device_class_flags": {}, 2024-09-11T09:51:54.229 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": { 2024-09-11T09:51:54.229 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode_enabled": false, 2024-09-11T09:51:54.230 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_bucket_count": 0, 2024-09-11T09:51:54.230 INFO:teuthology.orchestra.run.smithi017.stdout: "degraded_stretch_mode": 0, 2024-09-11T09:51:54.230 INFO:teuthology.orchestra.run.smithi017.stdout: "recovering_stretch_mode": 0, 2024-09-11T09:51:54.230 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode_bucket": 0 2024-09-11T09:51:54.230 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:54.230 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:51:54.230 INFO:tasks.ceph.osd.0.smithi017.stderr:2024-09-11T09:51:54.164+0000 7fb73c0f1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31665) UID: 0 2024-09-11T09:51:54.232 INFO:tasks.thrashosds.thrasher:j is {'epoch': 70, 'fsid': 'a2f1daec-d3b0-4bf5-82c8-72ecbb93d4e4', 'created': '2024-09-11T09:50:00.165804+0000', 'modified': '2024-09-11T09:51:51.790908+0000', 'last_up_change': '2024-09-11T09:51:44.764990+0000', 'last_in_change': '2024-09-11T09:50:45.581301+0000', 'flags': 'noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5801984, 'flags_set': ['noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 6, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 3, 'max_osd': 6, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-11T09:50:10.457184+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '14', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 6.059999942779541, 'score_stable': 6.059999942779541, 'optimal_score': 0.33000001311302185, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2024-09-11T09:50:16.993202+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '69', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 5, 'snap_epoch': 69, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5099999904632568, 'score_stable': 1.5099999904632568, 'optimal_score': 0.8299999833106995, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'new_default_pool27121', 'create_time': '2024-09-11T09:51:23.493016+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '58', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 58, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.2699999809265137, 'score_stable': 2.2699999809265137, 'optimal_score': 0.8299999833106995, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}], 'osds': [{'osd': 0, 'uuid': '4829b7a1-d607-4a56-b6d2-55036cb8acf8', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 12, 'up_thru': 66, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6808', 'nonce': 4154449929}, {'type': 'v1', 'addr': '172.21.15.17:6809', 'nonce': 4154449929}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6810', 'nonce': 4154449929}, {'type': 'v1', 'addr': '172.21.15.17:6811', 'nonce': 4154449929}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6814', 'nonce': 4154449929}, {'type': 'v1', 'addr': '172.21.15.17:6815', 'nonce': 4154449929}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6812', 'nonce': 4154449929}, {'type': 'v1', 'addr': '172.21.15.17:6813', 'nonce': 4154449929}]}, 'public_addr': '172.21.15.17:6809/4154449929', 'cluster_addr': '172.21.15.17:6811/4154449929', 'heartbeat_back_addr': '172.21.15.17:6815/4154449929', 'heartbeat_front_addr': '172.21.15.17:6813/4154449929', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'ab1d5df0-3c0e-44cd-88a0-96b07964148b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 54, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6800', 'nonce': 4139511644}, {'type': 'v1', 'addr': '172.21.15.17:6801', 'nonce': 4139511644}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6802', 'nonce': 4139511644}, {'type': 'v1', 'addr': '172.21.15.17:6803', 'nonce': 4139511644}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6806', 'nonce': 4139511644}, {'type': 'v1', 'addr': '172.21.15.17:6807', 'nonce': 4139511644}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6804', 'nonce': 4139511644}, {'type': 'v1', 'addr': '172.21.15.17:6805', 'nonce': 4139511644}]}, 'public_addr': '172.21.15.17:6801/4139511644', 'cluster_addr': '172.21.15.17:6803/4139511644', 'heartbeat_back_addr': '172.21.15.17:6807/4139511644', 'heartbeat_front_addr': '172.21.15.17:6805/4139511644', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '4c8e50db-b49c-4d44-a913-47b5b30f8699', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 54, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6816', 'nonce': 1151282506}, {'type': 'v1', 'addr': '172.21.15.17:6817', 'nonce': 1151282506}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6818', 'nonce': 1151282506}, {'type': 'v1', 'addr': '172.21.15.17:6819', 'nonce': 1151282506}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6822', 'nonce': 1151282506}, {'type': 'v1', 'addr': '172.21.15.17:6823', 'nonce': 1151282506}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6820', 'nonce': 1151282506}, {'type': 'v1', 'addr': '172.21.15.17:6821', 'nonce': 1151282506}]}, 'public_addr': '172.21.15.17:6817/1151282506', 'cluster_addr': '172.21.15.17:6819/1151282506', 'heartbeat_back_addr': '172.21.15.17:6823/1151282506', 'heartbeat_front_addr': '172.21.15.17:6821/1151282506', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '37abc55b-a1eb-49d5-b1ff-b507ee2cb5a6', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 12, 'up_thru': 35, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.96:6816', 'nonce': 332854807}, {'type': 'v1', 'addr': '172.21.15.96:6817', 'nonce': 332854807}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.96:6818', 'nonce': 332854807}, {'type': 'v1', 'addr': '172.21.15.96:6819', 'nonce': 332854807}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.96:6822', 'nonce': 332854807}, {'type': 'v1', 'addr': '172.21.15.96:6823', 'nonce': 332854807}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.96:6820', 'nonce': 332854807}, {'type': 'v1', 'addr': '172.21.15.96:6821', 'nonce': 332854807}]}, 'public_addr': '172.21.15.96:6817/332854807', 'cluster_addr': '172.21.15.96:6819/332854807', 'heartbeat_back_addr': '172.21.15.96:6823/332854807', 'heartbeat_front_addr': '172.21.15.96:6821/332854807', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '067d9586-4f5a-4d17-96c5-86a7b8bcd114', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 11, 'last_clean_end': 54, 'up_from': 66, 'up_thru': 66, 'down_at': 55, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.96:6800', 'nonce': 3374675329}, {'type': 'v1', 'addr': '172.21.15.96:6801', 'nonce': 3374675329}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.96:6802', 'nonce': 3374675329}, {'type': 'v1', 'addr': '172.21.15.96:6803', 'nonce': 3374675329}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.96:6806', 'nonce': 3374675329}, {'type': 'v1', 'addr': '172.21.15.96:6807', 'nonce': 3374675329}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.96:6804', 'nonce': 3374675329}, {'type': 'v1', 'addr': '172.21.15.96:6805', 'nonce': 3374675329}]}, 'public_addr': '172.21.15.96:6801/3374675329', 'cluster_addr': '172.21.15.96:6803/3374675329', 'heartbeat_back_addr': '172.21.15.96:6807/3374675329', 'heartbeat_front_addr': '172.21.15.96:6805/3374675329', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '8ff7a724-0a3a-4aa2-baf1-fd52f88d476c', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 54, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.96:6808', 'nonce': 509935370}, {'type': 'v1', 'addr': '172.21.15.96:6809', 'nonce': 509935370}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.96:6810', 'nonce': 509935370}, {'type': 'v1', 'addr': '172.21.15.96:6811', 'nonce': 509935370}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.96:6814', 'nonce': 509935370}, {'type': 'v1', 'addr': '172.21.15.96:6815', 'nonce': 509935370}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.96:6812', 'nonce': 509935370}, {'type': 'v1', 'addr': '172.21.15.96:6813', 'nonce': 509935370}]}, 'public_addr': '172.21.15.96:6809/509935370', 'cluster_addr': '172.21.15.96:6811/509935370', 'heartbeat_back_addr': '172.21.15.96:6815/509935370', 'heartbeat_front_addr': '172.21.15.96:6813/509935370', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-11T09:50:08.513270+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-11T09:50:07.522863+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-11T09:50:08.134364+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-11T09:50:08.622366+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-11T09:50:08.156142+0000', 'dead_epoch': 55}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-11T09:50:08.108116+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [{'pgid': '2.7', 'mappings': [{'from': 4, 'to': 5}, {'from': 3, 'to': 0}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [{'pool': 3, 'snaps': [{'begin': 2, 'length': 1}]}], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-11T09:51:54.232 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T09:51:54.273 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:51:54.273+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:51:54.337 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:51:54.338 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:51:54.347 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-11T09:51:54.391 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:51:54.391+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:51:54.457 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:51:54.458 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:51:54.466 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:51:54.467+0000 7f3c96ec9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31909) UID: 0 2024-09-11T09:51:54.467 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-11T09:51:54.548 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T09:51:54.565 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-11T09:51:54.565 INFO:teuthology.orchestra.run.smithi017.stderr:dumped all 2024-09-11T09:51:54.566 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:51:54.566+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:51:54.578 INFO:teuthology.orchestra.run.smithi017.stdout:{"pg_ready":true,"pg_map":{"version":117,"stamp":"2024-09-11T09:51:54.432794+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":135856983,"num_objects":56,"num_object_clones":1,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56,"num_whiteouts":0,"num_read":3730,"num_read_kb":163657,"num_write":738,"num_write_kb":265781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":688,"ondisk_log_size":688,"up":34,"acting":34,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":34,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":457248,"kb_used_data":269028,"kb_used_omap":12,"kb_used_meta":188083,"kb_avail":565773792,"statfs":{"total":579820584960,"available":579352363008,"internally_reserved":0,"allocated":275484672,"data_stored":279801174,"data_compressed":66628,"data_compressed_allocated":90112,"data_compressed_original":1163264,"omap_allocated":12403,"internal_metadata":192597901},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1048665,"num_objects":6,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":549,"num_read_kb":5011,"num_write":24,"num_write_kb":1034,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001187"},"pg_stats":[{"pgid":"1.0","version":"13'117","reported_seq":297,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-09-11T09:51:50.792695+0000","last_change":"2024-09-11T09:50:49.606072+0000","last_active":"2024-09-11T09:51:50.792695+0000","last_peered":"2024-09-11T09:51:50.792695+0000","last_clean":"2024-09-11T09:51:50.792695+0000","last_became_active":"2024-09-11T09:50:49.605824+0000","last_became_peered":"2024-09-11T09:50:49.605824+0000","last_unstale":"2024-09-11T09:51:50.792695+0000","last_undegraded":"2024-09-11T09:51:50.792695+0000","last_fullsized":"2024-09-11T09:51:50.792695+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:50:10.457262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:50:10.457262+0000","last_clean_scrub_stamp":"2024-09-11T09:50:10.457262+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:02:29.081582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"61'24","reported_seq":187,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-11T09:51:49.232169+0000","last_change":"2024-09-11T09:50:20.500687+0000","last_active":"2024-09-11T09:51:49.232169+0000","last_peered":"2024-09-11T09:51:49.232169+0000","last_clean":"2024-09-11T09:51:49.232169+0000","last_became_active":"2024-09-11T09:50:18.493650+0000","last_became_peered":"2024-09-11T09:50:18.493650+0000","last_unstale":"2024-09-11T09:51:49.232169+0000","last_undegraded":"2024-09-11T09:51:49.232169+0000","last_fullsized":"2024-09-11T09:51:49.232169+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_clean_scrub_stamp":"2024-09-11T09:50:17.477890+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:31:37.523854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041261100000000001,"stat_sum":{"num_bytes":25165824,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":12296,"num_write":23,"num_write_kb":32768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"67'3","reported_seq":48,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-11T09:51:48.827610+0000","last_change":"2024-09-11T09:51:27.719936+0000","last_active":"2024-09-11T09:51:48.827610+0000","last_peered":"2024-09-11T09:51:48.827610+0000","last_clean":"2024-09-11T09:51:48.827610+0000","last_became_active":"2024-09-11T09:51:24.714496+0000","last_became_peered":"2024-09-11T09:51:24.714496+0000","last_unstale":"2024-09-11T09:51:48.827610+0000","last_undegraded":"2024-09-11T09:51:48.827610+0000","last_fullsized":"2024-09-11T09:51:48.827610+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:51:23.699749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:51:23.699749+0000","last_clean_scrub_stamp":"2024-09-11T09:51:23.699749+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T20:58:07.190652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00068421499999999995,"stat_sum":{"num_bytes":54,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":5,"num_read_kb":4,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"63'5","reported_seq":283,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-09-11T09:51:50.792768+0000","last_change":"2024-09-11T09:50:20.498284+0000","last_active":"2024-09-11T09:51:50.792768+0000","last_peered":"2024-09-11T09:51:50.792768+0000","last_clean":"2024-09-11T09:51:50.792768+0000","last_became_active":"2024-09-11T09:50:18.493210+0000","last_became_peered":"2024-09-11T09:50:18.493210+0000","last_unstale":"2024-09-11T09:51:50.792768+0000","last_undegraded":"2024-09-11T09:51:50.792768+0000","last_fullsized":"2024-09-11T09:51:50.792768+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_clean_scrub_stamp":"2024-09-11T09:50:17.477890+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T19:13:17.609245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079749499999999995,"stat_sum":{"num_bytes":12582950,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":157,"num_read_kb":2850,"num_write":4,"num_write_kb":12289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"67'1","reported_seq":47,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-09-11T09:51:50.796810+0000","last_change":"2024-09-11T09:51:45.784453+0000","last_active":"2024-09-11T09:51:50.796810+0000","last_peered":"2024-09-11T09:51:50.796810+0000","last_clean":"2024-09-11T09:51:50.796810+0000","last_became_active":"2024-09-11T09:51:45.783693+0000","last_became_peered":"2024-09-11T09:51:45.783693+0000","last_unstale":"2024-09-11T09:51:50.796810+0000","last_undegraded":"2024-09-11T09:51:50.796810+0000","last_fullsized":"2024-09-11T09:51:50.796810+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:51:23.699749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:51:23.699749+0000","last_clean_scrub_stamp":"2024-09-11T09:51:23.699749+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T19:39:05.478650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041844700000000001,"stat_sum":{"num_bytes":16,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":5,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"68'113","reported_seq":1251,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-11T09:51:50.276543+0000","last_change":"2024-09-11T09:50:21.030566+0000","last_active":"2024-09-11T09:51:50.276543+0000","last_peered":"2024-09-11T09:51:50.276543+0000","last_clean":"2024-09-11T09:51:50.276543+0000","last_became_active":"2024-09-11T09:50:18.493109+0000","last_became_peered":"2024-09-11T09:50:18.493109+0000","last_unstale":"2024-09-11T09:51:50.276543+0000","last_undegraded":"2024-09-11T09:51:50.276543+0000","last_fullsized":"2024-09-11T09:51:50.276543+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_clean_scrub_stamp":"2024-09-11T09:50:17.477890+0000","objects_scrubbed":0,"log_size":113,"log_dups_size":0,"ondisk_log_size":113,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T18:10:55.614574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041008499999999997,"stat_sum":{"num_bytes":22020150,"num_objects":9,"num_object_clones":1,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":1223,"num_read_kb":40328,"num_write":98,"num_write_kb":46086,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":41,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-11T09:51:48.787947+0000","last_change":"2024-09-11T09:51:27.719437+0000","last_active":"2024-09-11T09:51:48.787947+0000","last_peered":"2024-09-11T09:51:48.787947+0000","last_clean":"2024-09-11T09:51:48.787947+0000","last_became_active":"2024-09-11T09:51:24.714263+0000","last_became_peered":"2024-09-11T09:51:24.714263+0000","last_unstale":"2024-09-11T09:51:48.787947+0000","last_undegraded":"2024-09-11T09:51:48.787947+0000","last_fullsized":"2024-09-11T09:51:48.787947+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:51:23.699749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:51:23.699749+0000","last_clean_scrub_stamp":"2024-09-11T09:51:23.699749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:06:36.882446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045627500000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"61'22","reported_seq":260,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-11T09:51:50.193847+0000","last_change":"2024-09-11T09:50:20.501218+0000","last_active":"2024-09-11T09:51:50.193847+0000","last_peered":"2024-09-11T09:51:50.193847+0000","last_clean":"2024-09-11T09:51:50.193847+0000","last_became_active":"2024-09-11T09:50:18.493165+0000","last_became_peered":"2024-09-11T09:50:18.493165+0000","last_unstale":"2024-09-11T09:51:50.193847+0000","last_undegraded":"2024-09-11T09:51:50.193847+0000","last_fullsized":"2024-09-11T09:51:50.193847+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_clean_scrub_stamp":"2024-09-11T09:50:17.477890+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T18:27:11.182608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00094463900000000005,"stat_sum":{"num_bytes":12582931,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":118,"num_read_kb":8312,"num_write":22,"num_write_kb":20482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"56'1","reported_seq":25,"reported_epoch":70,"state":"active+clean","last_fresh":"2024-09-11T09:51:53.039963+0000","last_change":"2024-09-11T09:51:45.785990+0000","last_active":"2024-09-11T09:51:53.039963+0000","last_peered":"2024-09-11T09:51:53.039963+0000","last_clean":"2024-09-11T09:51:53.039963+0000","last_became_active":"2024-09-11T09:51:45.783450+0000","last_became_peered":"2024-09-11T09:51:45.783450+0000","last_unstale":"2024-09-11T09:51:53.039963+0000","last_undegraded":"2024-09-11T09:51:53.039963+0000","last_fullsized":"2024-09-11T09:51:53.039963+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:51:23.699749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:51:23.699749+0000","last_clean_scrub_stamp":"2024-09-11T09:51:23.699749+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T14:28:40.036215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040505599999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.5","version":"0'0","reported_seq":41,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-11T09:51:49.232066+0000","last_change":"2024-09-11T09:51:27.729396+0000","last_active":"2024-09-11T09:51:49.232066+0000","last_peered":"2024-09-11T09:51:49.232066+0000","last_clean":"2024-09-11T09:51:49.232066+0000","last_became_active":"2024-09-11T09:51:24.715777+0000","last_became_peered":"2024-09-11T09:51:24.715777+0000","last_unstale":"2024-09-11T09:51:49.232066+0000","last_undegraded":"2024-09-11T09:51:49.232066+0000","last_fullsized":"2024-09-11T09:51:49.232066+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:51:23.699749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:51:23.699749+0000","last_clean_scrub_stamp":"2024-09-11T09:51:23.699749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:56:41.450651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026452099999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"68'153","reported_seq":854,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-11T09:51:50.288275+0000","last_change":"2024-09-11T09:50:46.593637+0000","last_active":"2024-09-11T09:51:50.288275+0000","last_peered":"2024-09-11T09:51:50.288275+0000","last_clean":"2024-09-11T09:51:50.288275+0000","last_became_active":"2024-09-11T09:50:46.593286+0000","last_became_peered":"2024-09-11T09:50:46.593286+0000","last_unstale":"2024-09-11T09:51:50.288275+0000","last_undegraded":"2024-09-11T09:51:50.288275+0000","last_fullsized":"2024-09-11T09:51:50.288275+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_clean_scrub_stamp":"2024-09-11T09:50:17.477890+0000","objects_scrubbed":0,"log_size":153,"log_dups_size":0,"ondisk_log_size":153,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T14:18:51.757602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00050337500000000002,"stat_sum":{"num_bytes":20971520,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1187,"num_read_kb":33727,"num_write":99,"num_write_kb":53255,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"68'4","reported_seq":55,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-11T09:51:48.830401+0000","last_change":"2024-09-11T09:51:27.720403+0000","last_active":"2024-09-11T09:51:48.830401+0000","last_peered":"2024-09-11T09:51:48.830401+0000","last_clean":"2024-09-11T09:51:48.830401+0000","last_became_active":"2024-09-11T09:51:24.714345+0000","last_became_peered":"2024-09-11T09:51:24.714345+0000","last_unstale":"2024-09-11T09:51:48.830401+0000","last_undegraded":"2024-09-11T09:51:48.830401+0000","last_fullsized":"2024-09-11T09:51:48.830401+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:51:23.699749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:51:23.699749+0000","last_clean_scrub_stamp":"2024-09-11T09:51:23.699749+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T20:24:54.033307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040133200000000001,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":28,"num_read_kb":25,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"61'37","reported_seq":240,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-09-11T09:51:50.792757+0000","last_change":"2024-09-11T09:50:46.593530+0000","last_active":"2024-09-11T09:51:50.792757+0000","last_peered":"2024-09-11T09:51:50.792757+0000","last_clean":"2024-09-11T09:51:50.792757+0000","last_became_active":"2024-09-11T09:50:46.593241+0000","last_became_peered":"2024-09-11T09:50:46.593241+0000","last_unstale":"2024-09-11T09:51:50.792757+0000","last_undegraded":"2024-09-11T09:51:50.792757+0000","last_fullsized":"2024-09-11T09:51:50.792757+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_clean_scrub_stamp":"2024-09-11T09:50:17.477890+0000","objects_scrubbed":0,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T14:48:26.677306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045546699999999998,"stat_sum":{"num_bytes":12582912,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":70,"num_read_kb":24576,"num_write":37,"num_write_kb":28672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":41,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-11T09:51:48.829858+0000","last_change":"2024-09-11T09:51:27.870068+0000","last_active":"2024-09-11T09:51:48.829858+0000","last_peered":"2024-09-11T09:51:48.829858+0000","last_clean":"2024-09-11T09:51:48.829858+0000","last_became_active":"2024-09-11T09:51:24.716325+0000","last_became_peered":"2024-09-11T09:51:24.716325+0000","last_unstale":"2024-09-11T09:51:48.829858+0000","last_undegraded":"2024-09-11T09:51:48.829858+0000","last_fullsized":"2024-09-11T09:51:48.829858+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:51:23.699749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:51:23.699749+0000","last_clean_scrub_stamp":"2024-09-11T09:51:23.699749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T19:44:13.062941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035292500000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"61'94","reported_seq":405,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-11T09:51:48.788370+0000","last_change":"2024-09-11T09:50:20.497446+0000","last_active":"2024-09-11T09:51:48.788370+0000","last_peered":"2024-09-11T09:51:48.788370+0000","last_clean":"2024-09-11T09:51:48.788370+0000","last_became_active":"2024-09-11T09:50:18.497481+0000","last_became_peered":"2024-09-11T09:50:18.497481+0000","last_unstale":"2024-09-11T09:51:48.788370+0000","last_undegraded":"2024-09-11T09:51:48.788370+0000","last_fullsized":"2024-09-11T09:51:48.788370+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_clean_scrub_stamp":"2024-09-11T09:50:17.477890+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T20:57:55.714584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00050697600000000002,"stat_sum":{"num_bytes":25165840,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":203,"num_read_kb":40974,"num_write":94,"num_write_kb":65539,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":43,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-09-11T09:51:50.796928+0000","last_change":"2024-09-11T09:51:27.720314+0000","last_active":"2024-09-11T09:51:50.796928+0000","last_peered":"2024-09-11T09:51:50.796928+0000","last_clean":"2024-09-11T09:51:50.796928+0000","last_became_active":"2024-09-11T09:51:24.712056+0000","last_became_peered":"2024-09-11T09:51:24.712056+0000","last_unstale":"2024-09-11T09:51:50.796928+0000","last_undegraded":"2024-09-11T09:51:50.796928+0000","last_fullsized":"2024-09-11T09:51:50.796928+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:51:23.699749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:51:23.699749+0000","last_clean_scrub_stamp":"2024-09-11T09:51:23.699749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T20:17:54.842299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00038419600000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.7","version":"69'114","reported_seq":210,"reported_epoch":69,"state":"active+clean","last_fresh":"2024-09-11T09:51:50.800194+0000","last_change":"2024-09-11T09:50:54.231326+0000","last_active":"2024-09-11T09:51:50.800194+0000","last_peered":"2024-09-11T09:51:50.800194+0000","last_clean":"2024-09-11T09:51:50.800194+0000","last_became_active":"2024-09-11T09:50:54.231067+0000","last_became_peered":"2024-09-11T09:50:54.231067+0000","last_unstale":"2024-09-11T09:51:50.800194+0000","last_undegraded":"2024-09-11T09:51:50.800194+0000","last_fullsized":"2024-09-11T09:51:50.800194+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_clean_scrub_stamp":"2024-09-11T09:50:17.477890+0000","objects_scrubbed":0,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T20:46:09.904149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00046094899999999999,"stat_sum":{"num_bytes":4194418,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":575,"num_read_kb":464,"num_write":196,"num_write_kb":4268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":8,"stat_sum":{"num_bytes":70,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":38,"num_read_kb":30,"num_write":6,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":16,"acting":16,"num_store_stats":5},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":135266545,"num_objects":48,"num_object_clones":1,"num_object_copies":96,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":3576,"num_read_kb":163527,"num_write":573,"num_write_kb":263359,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":270594048,"data_stored":276824508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":562,"ondisk_log_size":562,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1771104,"data_compressed":100760,"data_compressed_allocated":135168,"data_compressed_original":1753088,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":12,"seq":51539607576,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":81196,"kb_used_data":50004,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94290644,"statfs":{"total":96636764160,"available":96553619456,"internally_reserved":0,"allocated":51204096,"data_stored":51224103,"data_compressed":32496,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":12,"seq":51539607575,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":73040,"kb_used_data":37692,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94298800,"statfs":{"total":96636764160,"available":96561971200,"internally_reserved":0,"allocated":38596608,"data_stored":38635885,"data_compressed":34132,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":11,"seq":47244640280,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":68776,"kb_used_data":41748,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94303064,"statfs":{"total":96636764160,"available":96566337536,"internally_reserved":0,"allocated":42749952,"data_stored":42239878,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":66,"seq":283467841539,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28912,"kb_used_data":796,"kb_used_omap":4,"kb_used_meta":28091,"kb_avail":94342928,"statfs":{"total":96636764160,"available":96607158272,"internally_reserved":0,"allocated":815104,"data_stored":307055,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":28765872},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640279,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":78044,"kb_used_data":46856,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94293796,"statfs":{"total":96636764160,"available":96556847104,"internally_reserved":0,"allocated":47980544,"data_stored":50628391,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640279,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":127280,"kb_used_data":91932,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94244560,"statfs":{"total":96636764160,"available":96506429440,"internally_reserved":0,"allocated":94138368,"data_stored":96765862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":34132,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":34132,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32496,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":37761024,"data_stored":37748850,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":93343744,"data_stored":96469119,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":47194112,"data_stored":50331702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":50339840,"data_stored":50331702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":41955328,"data_stored":41943135,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":16,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":54,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":16,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":54,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T09:51:54.579 INFO:tasks.thrashosds.thrasher:Setting 2.1 to [2, 1] 2024-09-11T09:51:54.580 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '2.1', '2', '1'] 2024-09-11T09:51:54.580 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pg-upmap 2.1 2 1 2024-09-11T09:51:54.618 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:51:54.618 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:51:54.627 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-11T09:51:54.672 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:51:54.673+0000 7f3a7b90a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 42475) UID: 0 2024-09-11T09:51:54.692 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-11T09:51:54.692 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T09:51:54.710 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-11T09:51:54.710 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T09:51:54.727 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-11T09:51:54.728 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T09:51:54.743 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:51:54.743 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:51:54.745 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-11T09:51:54.745 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T09:51:54.753 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-11T09:51:54.761 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-11T09:51:54.761 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T09:51:54.778 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-11T09:51:54.779 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T09:51:54.793 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:51:54.794+0000 7f3a7b90a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 42475) UID: 0 2024-09-11T09:51:54.799 INFO:teuthology.orchestra.run.smithi017.stderr:nodeep-scrub is set 2024-09-11T09:51:54.869 INFO:tasks.ceph.osd.0.smithi017.stderr:2024-09-11T09:51:54.869+0000 7fb73c0f1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31665) UID: 0 2024-09-11T09:51:54.896 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-11T09:51:54.896 INFO:teuthology.orchestra.run.smithi096.stdout: "ops": [], 2024-09-11T09:51:54.896 INFO:teuthology.orchestra.run.smithi096.stdout: "complaint_time": 30, 2024-09-11T09:51:54.896 INFO:teuthology.orchestra.run.smithi096.stdout: "num_blocked_ops": 0 2024-09-11T09:51:54.897 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-11T09:51:54.905 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-11T09:51:54.954 INFO:teuthology.orchestra.run.smithi017.stderr:set 2.1 pg_upmap mapping to [2,1] 2024-09-11T09:51:54.970 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-09-11T09:51:54.971+0000 7f7b6c7d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31904) UID: 0 2024-09-11T09:51:55.011 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-11T09:51:55.011 INFO:teuthology.orchestra.run.smithi096.stdout: "size": 20, 2024-09-11T09:51:55.011 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 600, 2024-09-11T09:51:55.011 INFO:teuthology.orchestra.run.smithi096.stdout: "ops": [ 2024-09-11T09:51:55.011 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.011 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:111 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:51:55.011 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.711668+0000", 2024-09-11T09:51:55.011 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 103.29909264, 2024-09-11T09:51:55.011 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0077145570000000004, 2024-09-11T09:51:55.011 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:55.011 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:55.012 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:55.012 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:55.012 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:51:55.012 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:51:55.012 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 111 2024-09-11T09:51:55.012 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.012 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:55.012 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.012 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:55.012 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711668+0000", 2024-09-11T09:51:55.012 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:55.012 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.012 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.012 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:55.013 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711668+0000", 2024-09-11T09:51:55.013 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:55.013 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.013 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.013 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:55.013 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711669+0000", 2024-09-11T09:51:55.013 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.99e-07 2024-09-11T09:51:55.013 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.013 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.013 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:55.013 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711673+0000", 2024-09-11T09:51:55.013 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.7440000000000001e-06 2024-09-11T09:51:55.013 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.014 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.014 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:55.014 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711686+0000", 2024-09-11T09:51:55.014 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.3317e-05 2024-09-11T09:51:55.014 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.014 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.014 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:55.014 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711697+0000", 2024-09-11T09:51:55.014 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1022000000000001e-05 2024-09-11T09:51:55.014 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.014 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.014 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:55.015 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.717065+0000", 2024-09-11T09:51:55.015 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.005367531 2024-09-11T09:51:55.015 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.015 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.015 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:55.015 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.717114+0000", 2024-09-11T09:51:55.015 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.9707999999999999e-05 2024-09-11T09:51:55.015 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.015 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.015 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:55.015 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.717190+0000", 2024-09-11T09:51:55.015 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.5394000000000004e-05 2024-09-11T09:51:55.015 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.015 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.015 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:55.016 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718793+0000", 2024-09-11T09:51:55.016 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001603653 2024-09-11T09:51:55.016 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.016 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.016 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:55.016 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719329+0000", 2024-09-11T09:51:55.016 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00053543799999999995 2024-09-11T09:51:55.016 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.016 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.016 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:55.016 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719359+0000", 2024-09-11T09:51:55.016 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.0420999999999999e-05 2024-09-11T09:51:55.017 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.017 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.017 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:55.017 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719383+0000", 2024-09-11T09:51:55.017 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.3730000000000001e-05 2024-09-11T09:51:55.017 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:55.017 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:55.017 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:55.017 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.017 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.017 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:112 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:51:55.017 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.711712+0000", 2024-09-11T09:51:55.017 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 103.29904908, 2024-09-11T09:51:55.017 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.008162084, 2024-09-11T09:51:55.018 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:55.018 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:55.018 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:55.018 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:55.018 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:51:55.018 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:51:55.018 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 112 2024-09-11T09:51:55.018 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.018 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:55.018 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.018 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:55.018 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711712+0000", 2024-09-11T09:51:55.018 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:55.018 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.019 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.019 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:55.019 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711712+0000", 2024-09-11T09:51:55.019 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:55.019 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.019 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.019 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:55.019 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711712+0000", 2024-09-11T09:51:55.019 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.6000000000000004e-07 2024-09-11T09:51:55.019 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.019 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.019 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:55.019 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711808+0000", 2024-09-11T09:51:55.019 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.5977999999999995e-05 2024-09-11T09:51:55.020 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.020 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.020 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:55.020 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711819+0000", 2024-09-11T09:51:55.020 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.0755e-05 2024-09-11T09:51:55.020 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.020 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.020 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:55.020 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711826+0000", 2024-09-11T09:51:55.020 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.7970000000000003e-06 2024-09-11T09:51:55.020 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.020 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.020 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:55.020 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.717435+0000", 2024-09-11T09:51:55.020 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0056090239999999998 2024-09-11T09:51:55.021 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.021 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.021 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:55.021 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.717496+0000", 2024-09-11T09:51:55.021 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.0543e-05 2024-09-11T09:51:55.021 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.021 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.021 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:55.021 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.717591+0000", 2024-09-11T09:51:55.021 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.5667000000000004e-05 2024-09-11T09:51:55.021 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.021 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.021 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:55.022 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718831+0000", 2024-09-11T09:51:55.022 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001240204 2024-09-11T09:51:55.022 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.022 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.022 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:55.022 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719819+0000", 2024-09-11T09:51:55.022 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00098751500000000001 2024-09-11T09:51:55.022 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.022 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.022 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:55.022 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719850+0000", 2024-09-11T09:51:55.022 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.1535999999999998e-05 2024-09-11T09:51:55.022 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.022 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.022 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:55.023 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719874+0000", 2024-09-11T09:51:55.023 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.3504999999999999e-05 2024-09-11T09:51:55.023 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:55.023 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:55.023 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:55.023 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.023 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.023 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:113 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:51:55.023 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.711842+0000", 2024-09-11T09:51:55.023 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 103.298918581, 2024-09-11T09:51:55.024 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0081160099999999999, 2024-09-11T09:51:55.024 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:55.024 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:55.024 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:55.024 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:55.024 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:51:55.024 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:51:55.024 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 113 2024-09-11T09:51:55.024 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.024 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:55.024 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.024 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:55.024 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711842+0000", 2024-09-11T09:51:55.024 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:55.025 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.025 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.025 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:55.025 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711842+0000", 2024-09-11T09:51:55.025 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:55.025 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.025 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.025 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:55.025 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711843+0000", 2024-09-11T09:51:55.025 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.4399999999999999e-07 2024-09-11T09:51:55.025 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.025 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.025 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:55.026 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711848+0000", 2024-09-11T09:51:55.026 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.4159999999999997e-06 2024-09-11T09:51:55.026 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.026 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.026 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:55.026 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711856+0000", 2024-09-11T09:51:55.026 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.7870000000000007e-06 2024-09-11T09:51:55.026 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.026 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.026 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:55.026 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711862+0000", 2024-09-11T09:51:55.026 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.3689999999999998e-06 2024-09-11T09:51:55.026 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.026 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.027 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:55.027 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.717965+0000", 2024-09-11T09:51:55.027 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0061028899999999997 2024-09-11T09:51:55.027 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.027 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.027 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:55.027 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718023+0000", 2024-09-11T09:51:55.027 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.8313000000000002e-05 2024-09-11T09:51:55.027 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.027 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.027 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:55.027 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718109+0000", 2024-09-11T09:51:55.027 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.6205000000000006e-05 2024-09-11T09:51:55.027 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.028 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.028 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:55.028 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719808+0000", 2024-09-11T09:51:55.028 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0016990060000000001 2024-09-11T09:51:55.028 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.028 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.028 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:55.028 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719883+0000", 2024-09-11T09:51:55.028 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.4624999999999998e-05 2024-09-11T09:51:55.028 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.028 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.028 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:55.028 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719910+0000", 2024-09-11T09:51:55.028 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.7512e-05 2024-09-11T09:51:55.028 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.029 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.029 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:55.029 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719958+0000", 2024-09-11T09:51:55.029 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.8143000000000003e-05 2024-09-11T09:51:55.029 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:55.029 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:55.029 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:55.029 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.029 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.029 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:51:55.029 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.711887+0000", 2024-09-11T09:51:55.029 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 103.29887396700001, 2024-09-11T09:51:55.029 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0086063589999999992, 2024-09-11T09:51:55.029 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:55.030 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:55.030 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:55.030 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:55.030 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:51:55.030 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:51:55.030 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 114 2024-09-11T09:51:55.030 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.030 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:55.030 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.030 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:55.030 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711887+0000", 2024-09-11T09:51:55.030 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:55.031 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.031 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.031 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:55.031 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711887+0000", 2024-09-11T09:51:55.031 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:55.031 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.031 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.031 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:55.031 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711887+0000", 2024-09-11T09:51:55.031 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.1500000000000005e-07 2024-09-11T09:51:55.031 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.031 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.031 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:55.031 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711897+0000", 2024-09-11T09:51:55.032 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.1900000000000001e-06 2024-09-11T09:51:55.032 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.032 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.032 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:55.032 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711902+0000", 2024-09-11T09:51:55.032 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.2349999999999999e-06 2024-09-11T09:51:55.032 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.032 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.032 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:55.032 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711906+0000", 2024-09-11T09:51:55.032 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.7129999999999999e-06 2024-09-11T09:51:55.032 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.032 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.032 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:55.033 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718429+0000", 2024-09-11T09:51:55.033 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0065231960000000002 2024-09-11T09:51:55.033 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.033 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.033 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:55.033 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718480+0000", 2024-09-11T09:51:55.033 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.1161e-05 2024-09-11T09:51:55.033 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.033 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.033 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:55.033 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718558+0000", 2024-09-11T09:51:55.033 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.7519000000000001e-05 2024-09-11T09:51:55.033 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.034 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.034 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:55.034 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720038+0000", 2024-09-11T09:51:55.034 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001480016 2024-09-11T09:51:55.034 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.034 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.034 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:55.034 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720417+0000", 2024-09-11T09:51:55.034 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00037943699999999999 2024-09-11T09:51:55.034 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.034 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.034 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:55.034 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720452+0000", 2024-09-11T09:51:55.035 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.4595000000000003e-05 2024-09-11T09:51:55.035 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.035 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.035 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:55.035 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720493+0000", 2024-09-11T09:51:55.035 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.1782000000000001e-05 2024-09-11T09:51:55.035 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:55.035 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:55.035 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:55.035 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.035 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.035 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:51:55.035 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.711912+0000", 2024-09-11T09:51:55.035 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 103.29884929799999, 2024-09-11T09:51:55.035 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0086864750000000008, 2024-09-11T09:51:55.036 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:55.036 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:55.036 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:55.036 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:55.036 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:51:55.036 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:51:55.036 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 115 2024-09-11T09:51:55.036 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.036 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:55.036 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.036 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:55.036 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711912+0000", 2024-09-11T09:51:55.036 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:55.037 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.037 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.037 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:55.037 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711912+0000", 2024-09-11T09:51:55.037 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:55.037 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.037 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.037 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:55.037 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711912+0000", 2024-09-11T09:51:55.037 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.3900000000000001e-07 2024-09-11T09:51:55.037 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.037 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.037 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:55.037 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711969+0000", 2024-09-11T09:51:55.038 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.7501999999999999e-05 2024-09-11T09:51:55.038 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.038 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.038 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:55.038 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711986+0000", 2024-09-11T09:51:55.038 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.6402999999999999e-05 2024-09-11T09:51:55.038 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.038 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.038 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:55.038 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711990+0000", 2024-09-11T09:51:55.038 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.929e-06 2024-09-11T09:51:55.038 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.038 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.039 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:55.039 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718861+0000", 2024-09-11T09:51:55.039 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0068710530000000002 2024-09-11T09:51:55.039 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.039 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.039 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:55.039 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718908+0000", 2024-09-11T09:51:55.039 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.7431e-05 2024-09-11T09:51:55.039 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.039 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.039 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:55.039 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718984+0000", 2024-09-11T09:51:55.039 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.5500999999999994e-05 2024-09-11T09:51:55.039 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.040 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.040 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:55.040 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720500+0000", 2024-09-11T09:51:55.040 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0015158820000000001 2024-09-11T09:51:55.040 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.040 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.040 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:55.040 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720536+0000", 2024-09-11T09:51:55.040 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.6118000000000002e-05 2024-09-11T09:51:55.040 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.040 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.040 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:55.040 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720570+0000", 2024-09-11T09:51:55.041 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.4387999999999997e-05 2024-09-11T09:51:55.041 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.041 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.041 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:55.041 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720598+0000", 2024-09-11T09:51:55.041 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.8028999999999999e-05 2024-09-11T09:51:55.041 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:55.041 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:55.041 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:55.041 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.041 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.041 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:51:55.041 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.711998+0000", 2024-09-11T09:51:55.041 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 103.298763119, 2024-09-11T09:51:55.042 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.010785813999999999, 2024-09-11T09:51:55.042 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:55.042 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:55.042 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:55.042 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:55.042 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:51:55.042 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:51:55.042 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 116 2024-09-11T09:51:55.042 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.042 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:55.042 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.042 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:55.042 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711998+0000", 2024-09-11T09:51:55.042 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:55.042 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.043 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.043 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:55.043 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711998+0000", 2024-09-11T09:51:55.043 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:55.043 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.043 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.043 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:55.043 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711998+0000", 2024-09-11T09:51:55.043 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.3300000000000003e-07 2024-09-11T09:51:55.044 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.044 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.044 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:55.044 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712001+0000", 2024-09-11T09:51:55.044 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.9349999999999999e-06 2024-09-11T09:51:55.044 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.044 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.044 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:55.044 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712006+0000", 2024-09-11T09:51:55.044 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.1630000000000004e-06 2024-09-11T09:51:55.044 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.044 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.044 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:55.044 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712010+0000", 2024-09-11T09:51:55.045 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.574e-06 2024-09-11T09:51:55.045 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.045 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.045 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:55.045 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719415+0000", 2024-09-11T09:51:55.045 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0074049099999999998 2024-09-11T09:51:55.045 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.045 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.045 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:55.045 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719465+0000", 2024-09-11T09:51:55.045 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.0129000000000001e-05 2024-09-11T09:51:55.045 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.045 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.045 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:55.046 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719545+0000", 2024-09-11T09:51:55.046 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.0253999999999994e-05 2024-09-11T09:51:55.046 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.046 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.046 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:55.046 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720636+0000", 2024-09-11T09:51:55.046 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0010909330000000001 2024-09-11T09:51:55.046 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.046 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.046 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:55.046 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.722707+0000", 2024-09-11T09:51:55.046 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0020713300000000001 2024-09-11T09:51:55.046 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.046 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.046 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:55.047 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.722743+0000", 2024-09-11T09:51:55.047 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.5351000000000001e-05 2024-09-11T09:51:55.047 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.047 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.047 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:55.047 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.722784+0000", 2024-09-11T09:51:55.047 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.0802e-05 2024-09-11T09:51:55.047 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:55.047 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:55.047 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:55.047 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.047 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.047 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:51:55.047 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.712014+0000", 2024-09-11T09:51:55.047 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 103.29874707899999, 2024-09-11T09:51:55.048 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.013420938, 2024-09-11T09:51:55.048 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:55.048 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:55.048 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:55.048 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:55.048 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:51:55.048 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:51:55.048 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 117 2024-09-11T09:51:55.048 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.048 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:55.048 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.048 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:55.048 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712014+0000", 2024-09-11T09:51:55.049 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:55.049 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.049 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.049 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:55.049 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712014+0000", 2024-09-11T09:51:55.049 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:55.049 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.049 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.049 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:55.049 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712014+0000", 2024-09-11T09:51:55.049 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.2000000000000001e-07 2024-09-11T09:51:55.049 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.049 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.049 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:55.050 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712016+0000", 2024-09-11T09:51:55.050 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.2309999999999999e-06 2024-09-11T09:51:55.050 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.050 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.050 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:55.050 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712020+0000", 2024-09-11T09:51:55.050 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.3830000000000001e-06 2024-09-11T09:51:55.050 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.050 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.050 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:55.050 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712023+0000", 2024-09-11T09:51:55.050 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.3340000000000002e-06 2024-09-11T09:51:55.050 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.050 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.050 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:55.051 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720083+0000", 2024-09-11T09:51:55.051 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0080601189999999993 2024-09-11T09:51:55.051 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.051 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.051 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:55.051 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720125+0000", 2024-09-11T09:51:55.051 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.2029e-05 2024-09-11T09:51:55.051 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.051 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.051 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:55.051 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720205+0000", 2024-09-11T09:51:55.051 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.0010000000000001e-05 2024-09-11T09:51:55.051 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.052 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.052 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:55.052 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720849+0000", 2024-09-11T09:51:55.052 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00064369899999999996 2024-09-11T09:51:55.052 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.052 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.052 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:55.052 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.725357+0000", 2024-09-11T09:51:55.052 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0045077379999999998 2024-09-11T09:51:55.052 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.052 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.052 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:55.052 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.725392+0000", 2024-09-11T09:51:55.053 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.5862999999999999e-05 2024-09-11T09:51:55.053 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.053 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.053 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:55.053 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.725435+0000", 2024-09-11T09:51:55.053 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.2311999999999997e-05 2024-09-11T09:51:55.053 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:55.053 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:55.053 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:55.053 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.053 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.053 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:159 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:51:55.053 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:12.452099+0000", 2024-09-11T09:51:55.054 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 102.558661491, 2024-09-11T09:51:55.054 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0069885950000000002, 2024-09-11T09:51:55.054 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:55.054 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:55.054 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:55.054 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:55.054 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:51:55.054 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:51:55.054 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 159 2024-09-11T09:51:55.055 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.055 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:55.055 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.055 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:55.055 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452099+0000", 2024-09-11T09:51:55.055 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:55.055 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.055 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.055 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:55.056 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452099+0000", 2024-09-11T09:51:55.056 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:55.056 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.056 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.056 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:55.056 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452100+0000", 2024-09-11T09:51:55.056 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.2800000000000003e-07 2024-09-11T09:51:55.056 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.056 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.056 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:55.056 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452102+0000", 2024-09-11T09:51:55.056 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.824e-06 2024-09-11T09:51:55.056 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.057 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.057 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:55.057 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452105+0000", 2024-09-11T09:51:55.057 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.0460000000000001e-06 2024-09-11T09:51:55.057 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.057 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.057 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:55.057 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452107+0000", 2024-09-11T09:51:55.057 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.543e-06 2024-09-11T09:51:55.057 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.057 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.057 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:55.057 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.455484+0000", 2024-09-11T09:51:55.058 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0033769759999999998 2024-09-11T09:51:55.058 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.058 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.058 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:55.058 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.455527+0000", 2024-09-11T09:51:55.058 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.2589999999999997e-05 2024-09-11T09:51:55.058 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.058 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.058 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:55.058 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.455594+0000", 2024-09-11T09:51:55.058 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.6775999999999999e-05 2024-09-11T09:51:55.058 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.058 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.059 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:55.059 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.456312+0000", 2024-09-11T09:51:55.059 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00071874700000000005 2024-09-11T09:51:55.059 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.059 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.059 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:55.059 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.459015+0000", 2024-09-11T09:51:55.059 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0027028540000000002 2024-09-11T09:51:55.059 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.059 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.059 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:55.059 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.459054+0000", 2024-09-11T09:51:55.059 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.8840000000000001e-05 2024-09-11T09:51:55.059 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.060 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.060 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:55.060 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.459088+0000", 2024-09-11T09:51:55.060 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.4071e-05 2024-09-11T09:51:55.060 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:55.060 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:55.060 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:55.060 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.060 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.060 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4688.0:38 2.5 2:abde2694:::rbd_data.1250f5e25b80.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:55.060 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:22.903926+0000", 2024-09-11T09:51:55.060 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 32.106834814999999, 2024-09-11T09:51:55.060 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.028303676, 2024-09-11T09:51:55.061 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:55.061 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:55.061 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:55.061 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:55.061 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4688", 2024-09-11T09:51:55.061 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:51:55.061 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 38 2024-09-11T09:51:55.061 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.061 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:55.061 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.061 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:55.061 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.903926+0000", 2024-09-11T09:51:55.061 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:55.062 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.062 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.062 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:55.062 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.903926+0000", 2024-09-11T09:51:55.062 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:55.062 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.062 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.062 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:55.062 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.903928+0000", 2024-09-11T09:51:55.062 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.4249999999999999e-06 2024-09-11T09:51:55.062 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.062 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.062 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:55.063 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.914774+0000", 2024-09-11T09:51:55.063 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.010846613 2024-09-11T09:51:55.063 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.063 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.063 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:55.063 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.914793+0000", 2024-09-11T09:51:55.063 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.8879999999999999e-05 2024-09-11T09:51:55.063 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.063 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.063 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:55.063 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.914801+0000", 2024-09-11T09:51:55.063 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.2230000000000007e-06 2024-09-11T09:51:55.063 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.064 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.064 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:55.064 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.914854+0000", 2024-09-11T09:51:55.064 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.2883999999999997e-05 2024-09-11T09:51:55.064 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.064 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.064 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:55.064 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.914951+0000", 2024-09-11T09:51:55.064 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.6731000000000006e-05 2024-09-11T09:51:55.064 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.064 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.064 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:55.064 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.915080+0000", 2024-09-11T09:51:55.064 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00012926999999999999 2024-09-11T09:51:55.065 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.065 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.065 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:55.065 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.931469+0000", 2024-09-11T09:51:55.065 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.016389107 2024-09-11T09:51:55.065 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.065 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.065 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:55.065 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.932118+0000", 2024-09-11T09:51:55.065 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00064915600000000002 2024-09-11T09:51:55.065 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.065 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.065 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:55.066 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.932177+0000", 2024-09-11T09:51:55.066 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.9032e-05 2024-09-11T09:51:55.066 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.066 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.066 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:55.066 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.932230+0000", 2024-09-11T09:51:55.066 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.2355000000000003e-05 2024-09-11T09:51:55.066 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:55.066 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:55.066 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:55.066 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.066 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.066 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_repop(client.4688.0:36 2.6 e53/15 2:61e1b442:::rbd_data.1250f5e25b80.0000000000000003:head v 53'2, mlcod=0'0)", 2024-09-11T09:51:55.066 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:22.915597+0000", 2024-09-11T09:51:55.067 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 32.095164265000001, 2024-09-11T09:51:55.067 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.020430798, 2024-09-11T09:51:55.067 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:55.067 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:55.067 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:55.067 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:55.067 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.067 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:55.067 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.915597+0000", 2024-09-11T09:51:55.067 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:55.067 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.067 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.067 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:55.067 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.915597+0000", 2024-09-11T09:51:55.068 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:55.068 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.068 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.068 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:55.068 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.915598+0000", 2024-09-11T09:51:55.068 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.7150000000000001e-06 2024-09-11T09:51:55.068 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.068 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.068 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:55.068 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921741+0000", 2024-09-11T09:51:55.068 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0061424339999999996 2024-09-11T09:51:55.068 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.069 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.069 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:55.069 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921757+0000", 2024-09-11T09:51:55.069 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.6144e-05 2024-09-11T09:51:55.069 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.069 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.069 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:55.069 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921765+0000", 2024-09-11T09:51:55.069 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.0809999999999994e-06 2024-09-11T09:51:55.069 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.069 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.069 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:55.069 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921826+0000", 2024-09-11T09:51:55.070 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.0878999999999998e-05 2024-09-11T09:51:55.070 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.070 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.070 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:55.070 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921849+0000", 2024-09-11T09:51:55.070 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.2966999999999999e-05 2024-09-11T09:51:55.070 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.070 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.070 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:55.070 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.935968+0000", 2024-09-11T09:51:55.070 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.014119191 2024-09-11T09:51:55.070 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.071 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.071 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:55.071 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.936027+0000", 2024-09-11T09:51:55.071 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.9386999999999999e-05 2024-09-11T09:51:55.071 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:55.071 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:55.071 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:55.071 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.071 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.071 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_repop(client.4688.0:39 2.6 e53/15 2:743e7616:::rbd_data.1250f5e25b80.0000000000000005:head v 53'3, mlcod=0'0)", 2024-09-11T09:51:55.071 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:22.921806+0000", 2024-09-11T09:51:55.071 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 32.088955214999999, 2024-09-11T09:51:55.071 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.015275018, 2024-09-11T09:51:55.071 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:55.072 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:55.072 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:55.072 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:55.072 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.072 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:55.072 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921806+0000", 2024-09-11T09:51:55.072 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:55.072 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.072 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.072 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:55.072 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921806+0000", 2024-09-11T09:51:55.072 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:55.072 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.073 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.073 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:55.073 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921806+0000", 2024-09-11T09:51:55.073 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.2200000000000003e-07 2024-09-11T09:51:55.073 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.073 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.073 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:55.073 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.925590+0000", 2024-09-11T09:51:55.073 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0037838949999999998 2024-09-11T09:51:55.073 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.073 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.073 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:55.073 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.925605+0000", 2024-09-11T09:51:55.074 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.4321999999999999e-05 2024-09-11T09:51:55.074 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.074 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.074 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:55.074 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.925613+0000", 2024-09-11T09:51:55.074 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.1980000000000001e-06 2024-09-11T09:51:55.074 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.074 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.074 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:55.074 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.925857+0000", 2024-09-11T09:51:55.074 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000244064 2024-09-11T09:51:55.074 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.075 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.075 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:55.075 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.925875+0000", 2024-09-11T09:51:55.075 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.838e-05 2024-09-11T09:51:55.075 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.075 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.075 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:55.075 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.937042+0000", 2024-09-11T09:51:55.075 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.011166416 2024-09-11T09:51:55.075 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.075 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.075 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:55.075 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.937081+0000", 2024-09-11T09:51:55.075 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.9020999999999998e-05 2024-09-11T09:51:55.076 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:55.076 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:55.076 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:55.076 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.076 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.076 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4688.0:49 2.5 2:b6dd9fc9:::rbd_data.1250f5e25b80.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:55.076 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:22.938548+0000", 2024-09-11T09:51:55.076 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 32.072213212000001, 2024-09-11T09:51:55.076 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.022766950000000001, 2024-09-11T09:51:55.076 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:55.076 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:55.076 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:55.076 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:55.077 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4688", 2024-09-11T09:51:55.077 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:51:55.077 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 49 2024-09-11T09:51:55.077 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.077 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:55.077 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.077 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:55.077 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.938548+0000", 2024-09-11T09:51:55.077 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:55.077 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.077 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.077 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:55.077 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.938548+0000", 2024-09-11T09:51:55.078 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:55.078 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.078 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.078 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:55.078 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.938550+0000", 2024-09-11T09:51:55.078 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.7710000000000001e-06 2024-09-11T09:51:55.078 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.078 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.078 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:55.078 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.944237+0000", 2024-09-11T09:51:55.078 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0056879790000000001 2024-09-11T09:51:55.078 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.078 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.079 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:55.079 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.944262+0000", 2024-09-11T09:51:55.079 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.4247e-05 2024-09-11T09:51:55.079 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.079 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.079 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:55.079 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.944274+0000", 2024-09-11T09:51:55.079 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.2639999999999999e-05 2024-09-11T09:51:55.079 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.079 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.079 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:55.079 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.944344+0000", 2024-09-11T09:51:55.079 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.9574999999999997e-05 2024-09-11T09:51:55.080 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.080 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.080 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:55.080 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.944474+0000", 2024-09-11T09:51:55.080 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000130464 2024-09-11T09:51:55.080 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.080 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.080 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:55.080 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.944585+0000", 2024-09-11T09:51:55.080 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000110096 2024-09-11T09:51:55.080 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.080 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.080 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:55.081 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.952724+0000", 2024-09-11T09:51:55.081 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0081398900000000003 2024-09-11T09:51:55.081 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.081 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.081 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:55.081 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.961219+0000", 2024-09-11T09:51:55.081 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0084949789999999997 2024-09-11T09:51:55.081 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.081 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.081 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:55.081 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.961280+0000", 2024-09-11T09:51:55.081 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.0404e-05 2024-09-11T09:51:55.081 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.081 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.082 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:55.082 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.961315+0000", 2024-09-11T09:51:55.082 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.4904999999999999e-05 2024-09-11T09:51:55.082 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:55.082 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:55.082 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:55.082 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.082 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.082 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_repop(client.4688.0:44 2.6 e53/15 2:67246a22:::rbd_data.1250f5e25b80.0000000000000007:head v 53'4, mlcod=53'2)", 2024-09-11T09:51:55.082 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:22.951721+0000", 2024-09-11T09:51:55.082 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 32.059039939999998, 2024-09-11T09:51:55.082 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.01080981, 2024-09-11T09:51:55.082 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:55.082 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:55.083 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:55.083 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:55.083 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.083 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:55.083 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.951721+0000", 2024-09-11T09:51:55.083 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:55.083 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.083 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.083 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:55.083 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.951721+0000", 2024-09-11T09:51:55.083 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:55.083 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.083 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.084 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:55.084 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.951722+0000", 2024-09-11T09:51:55.084 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.0279999999999999e-06 2024-09-11T09:51:55.084 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.084 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.084 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:55.084 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.955612+0000", 2024-09-11T09:51:55.084 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0038896920000000001 2024-09-11T09:51:55.084 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.084 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.084 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:55.084 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.955628+0000", 2024-09-11T09:51:55.084 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.5945000000000001e-05 2024-09-11T09:51:55.085 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.085 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.085 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:55.085 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.955635+0000", 2024-09-11T09:51:55.085 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.8080000000000005e-06 2024-09-11T09:51:55.085 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.085 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.085 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:55.085 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.955700+0000", 2024-09-11T09:51:55.085 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.4114000000000003e-05 2024-09-11T09:51:55.085 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.085 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.086 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:55.086 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.955724+0000", 2024-09-11T09:51:55.086 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.3989999999999999e-05 2024-09-11T09:51:55.086 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.086 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.086 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:55.086 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.962490+0000", 2024-09-11T09:51:55.086 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0067662599999999996 2024-09-11T09:51:55.086 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.086 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.086 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:55.086 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.962531+0000", 2024-09-11T09:51:55.086 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.0973000000000002e-05 2024-09-11T09:51:55.087 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:55.087 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:55.087 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:55.087 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.087 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.087 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_repop(client.4688.0:51 2.6 e53/15 2:6b5894fc:::rbd_data.1250f5e25b80.000000000000000a:head v 53'5, mlcod=53'2)", 2024-09-11T09:51:55.087 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:22.959030+0000", 2024-09-11T09:51:55.087 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 32.051730765999999, 2024-09-11T09:51:55.087 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.011503638, 2024-09-11T09:51:55.087 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:55.087 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:55.087 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:55.087 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:55.088 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.088 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:55.088 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.959030+0000", 2024-09-11T09:51:55.088 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:55.088 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.088 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.088 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:55.088 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.959030+0000", 2024-09-11T09:51:55.088 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:55.088 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.088 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.088 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:55.088 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.959032+0000", 2024-09-11T09:51:55.089 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.356e-06 2024-09-11T09:51:55.089 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.089 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.089 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:55.089 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.962899+0000", 2024-09-11T09:51:55.089 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.003867639 2024-09-11T09:51:55.089 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.089 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.089 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:55.089 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.962909+0000", 2024-09-11T09:51:55.089 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.0105999999999999e-05 2024-09-11T09:51:55.089 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.089 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.090 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:55.090 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.962915+0000", 2024-09-11T09:51:55.090 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.8869999999999997e-06 2024-09-11T09:51:55.090 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.090 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.090 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:55.090 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.962962+0000", 2024-09-11T09:51:55.090 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.6511999999999998e-05 2024-09-11T09:51:55.090 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.090 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.090 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:55.090 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.962989+0000", 2024-09-11T09:51:55.090 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.7702999999999999e-05 2024-09-11T09:51:55.091 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.091 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.091 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:55.091 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.970495+0000", 2024-09-11T09:51:55.091 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0075052449999999998 2024-09-11T09:51:55.091 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.091 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.091 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:55.091 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.970534+0000", 2024-09-11T09:51:55.091 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.9190000000000003e-05 2024-09-11T09:51:55.091 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:55.091 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:55.091 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:55.091 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.092 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.092 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_repop(client.4688.0:58 2.6 e53/15 2:6277c644:::rbd_data.1250f5e25b80.000000000000000e:head v 53'6, mlcod=53'4)", 2024-09-11T09:51:55.092 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:22.996199+0000", 2024-09-11T09:51:55.092 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 32.014561845999999, 2024-09-11T09:51:55.092 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.010145573999999999, 2024-09-11T09:51:55.092 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:55.092 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:55.092 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:55.092 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:55.092 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.092 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:55.092 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.996199+0000", 2024-09-11T09:51:55.092 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:55.093 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.093 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.093 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:55.093 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.996199+0000", 2024-09-11T09:51:55.093 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:55.093 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.093 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.093 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:55.093 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.996200+0000", 2024-09-11T09:51:55.093 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.85e-07 2024-09-11T09:51:55.093 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.093 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.093 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:55.094 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.999914+0000", 2024-09-11T09:51:55.094 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.003714406 2024-09-11T09:51:55.094 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.094 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.094 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:55.094 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.999928+0000", 2024-09-11T09:51:55.094 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.3149000000000001e-05 2024-09-11T09:51:55.094 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.094 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.094 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:55.094 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.999951+0000", 2024-09-11T09:51:55.094 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.3116e-05 2024-09-11T09:51:55.094 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.095 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.095 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:55.095 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.000003+0000", 2024-09-11T09:51:55.095 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.2169000000000001e-05 2024-09-11T09:51:55.095 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.095 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.095 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:55.095 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.000018+0000", 2024-09-11T09:51:55.095 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.5500000000000001e-05 2024-09-11T09:51:55.095 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.095 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.095 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:55.095 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.006303+0000", 2024-09-11T09:51:55.095 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0062846710000000004 2024-09-11T09:51:55.096 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.096 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.096 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:55.096 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.006345+0000", 2024-09-11T09:51:55.096 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.1678000000000003e-05 2024-09-11T09:51:55.096 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:55.096 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:55.096 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:55.096 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.096 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.096 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4694.0:42 2.5 2:a4fc00ec:ns::rbd_data.1256a0c02879.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:55.096 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.111570+0000", 2024-09-11T09:51:55.096 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 31.899190740000002, 2024-09-11T09:51:55.097 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.038002197000000001, 2024-09-11T09:51:55.097 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:55.097 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:55.097 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:55.097 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:55.097 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4694", 2024-09-11T09:51:55.097 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:51:55.097 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 42 2024-09-11T09:51:55.097 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.097 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:55.097 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.097 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:55.097 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.111570+0000", 2024-09-11T09:51:55.098 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:55.098 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.098 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.098 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:55.098 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.111570+0000", 2024-09-11T09:51:55.098 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:55.098 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.098 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.098 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:55.098 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.111572+0000", 2024-09-11T09:51:55.098 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.3909999999999999e-06 2024-09-11T09:51:55.098 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.098 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.099 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:55.099 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.135929+0000", 2024-09-11T09:51:55.099 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.024357410999999999 2024-09-11T09:51:55.099 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.099 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.099 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:55.099 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.135951+0000", 2024-09-11T09:51:55.099 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.1775000000000001e-05 2024-09-11T09:51:55.099 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.099 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.099 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:55.099 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.135962+0000", 2024-09-11T09:51:55.099 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.0903e-05 2024-09-11T09:51:55.099 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.100 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.100 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:55.100 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.136045+0000", 2024-09-11T09:51:55.100 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.3128999999999996e-05 2024-09-11T09:51:55.100 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.100 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.100 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:55.100 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.136229+0000", 2024-09-11T09:51:55.100 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00018393700000000001 2024-09-11T09:51:55.100 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.100 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.100 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:55.100 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.136394+0000", 2024-09-11T09:51:55.101 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000165527 2024-09-11T09:51:55.101 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.101 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.101 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:55.101 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.149320+0000", 2024-09-11T09:51:55.101 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.01292568 2024-09-11T09:51:55.101 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.101 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.101 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:55.101 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.149493+0000", 2024-09-11T09:51:55.101 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00017285999999999999 2024-09-11T09:51:55.101 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.101 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.101 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:55.102 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.149539+0000", 2024-09-11T09:51:55.102 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.6628000000000001e-05 2024-09-11T09:51:55.102 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.102 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.102 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:55.102 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.149572+0000", 2024-09-11T09:51:55.102 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.2956000000000001e-05 2024-09-11T09:51:55.102 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:55.102 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:55.102 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:55.102 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.102 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.102 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_repop(client.4694.0:45 2.6 e53/15 2:671fb830:ns::rbd_data.1256a0c02879.0000000000000007:head v 53'9, mlcod=53'7)", 2024-09-11T09:51:55.102 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.121833+0000", 2024-09-11T09:51:55.103 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 31.888928382, 2024-09-11T09:51:55.103 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.023876192000000001, 2024-09-11T09:51:55.103 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:55.103 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:55.103 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:55.103 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:55.103 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.103 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:55.103 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.121833+0000", 2024-09-11T09:51:55.103 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:55.103 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.103 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.103 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:55.104 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.121833+0000", 2024-09-11T09:51:55.104 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:55.104 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.104 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.104 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:55.104 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.121834+0000", 2024-09-11T09:51:55.104 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.471e-06 2024-09-11T09:51:55.104 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.104 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.104 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:55.104 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.128990+0000", 2024-09-11T09:51:55.104 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0071557879999999997 2024-09-11T09:51:55.104 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.104 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.105 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:55.105 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.129011+0000", 2024-09-11T09:51:55.105 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.0760000000000001e-05 2024-09-11T09:51:55.105 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.105 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.105 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:55.105 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.129022+0000", 2024-09-11T09:51:55.105 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1056000000000001e-05 2024-09-11T09:51:55.105 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.105 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.105 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:55.105 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.129097+0000", 2024-09-11T09:51:55.105 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.5082999999999999e-05 2024-09-11T09:51:55.106 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.106 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.106 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:55.106 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.129130+0000", 2024-09-11T09:51:55.106 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.3345e-05 2024-09-11T09:51:55.106 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.106 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.106 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:55.106 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.145657+0000", 2024-09-11T09:51:55.106 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.016527224 2024-09-11T09:51:55.106 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.106 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.106 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:55.107 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.145709+0000", 2024-09-11T09:51:55.107 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.1465000000000003e-05 2024-09-11T09:51:55.107 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:55.107 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:55.107 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:55.107 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.107 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.107 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4694.0:43 2.0 2:16c8ae84:ns::rbd_data.1256a0c02879.0000000000000005:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:55.107 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.135999+0000", 2024-09-11T09:51:55.107 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 31.874762432000001, 2024-09-11T09:51:55.107 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.039468309, 2024-09-11T09:51:55.107 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:55.107 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:55.107 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:55.108 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:55.108 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4694", 2024-09-11T09:51:55.108 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:51:55.108 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 43 2024-09-11T09:51:55.108 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.108 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:55.108 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.108 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:55.108 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.135999+0000", 2024-09-11T09:51:55.108 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:55.108 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.108 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.108 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:55.109 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.135999+0000", 2024-09-11T09:51:55.109 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:55.109 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.109 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.109 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:55.109 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.136000+0000", 2024-09-11T09:51:55.109 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.6330000000000001e-06 2024-09-11T09:51:55.109 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.109 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.109 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:55.109 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.157961+0000", 2024-09-11T09:51:55.109 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0219606 2024-09-11T09:51:55.109 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.109 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.110 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:55.110 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.157995+0000", 2024-09-11T09:51:55.110 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.4391999999999999e-05 2024-09-11T09:51:55.110 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.110 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.110 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:55.110 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.158011+0000", 2024-09-11T09:51:55.110 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.5794000000000002e-05 2024-09-11T09:51:55.110 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.110 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.110 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:55.110 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.158083+0000", 2024-09-11T09:51:55.110 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.2273000000000004e-05 2024-09-11T09:51:55.111 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.111 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.111 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:55.111 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.158239+0000", 2024-09-11T09:51:55.111 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000155372 2024-09-11T09:51:55.111 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.111 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.111 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 1", 2024-09-11T09:51:55.111 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.158403+0000", 2024-09-11T09:51:55.111 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00016408199999999999 2024-09-11T09:51:55.111 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.111 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.111 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:55.111 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.173860+0000", 2024-09-11T09:51:55.112 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.015457347 2024-09-11T09:51:55.112 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.112 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.112 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:55.112 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.175400+0000", 2024-09-11T09:51:55.112 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001540006 2024-09-11T09:51:55.112 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.112 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.112 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:55.112 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.175438+0000", 2024-09-11T09:51:55.112 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.8135e-05 2024-09-11T09:51:55.112 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.112 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.113 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:55.113 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.175467+0000", 2024-09-11T09:51:55.113 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.8674999999999999e-05 2024-09-11T09:51:55.113 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:55.113 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:55.113 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:55.113 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.113 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.113 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4694.0:57 2.0 2:04b5549c:ns::rbd_data.1256a0c02879.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:55.113 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.158037+0000", 2024-09-11T09:51:55.113 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 31.852723694000002, 2024-09-11T09:51:55.113 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.027748790999999998, 2024-09-11T09:51:55.113 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:55.113 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:55.114 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:55.114 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:55.114 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4694", 2024-09-11T09:51:55.114 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:51:55.114 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 57 2024-09-11T09:51:55.114 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.114 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:55.114 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.114 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:55.114 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.158037+0000", 2024-09-11T09:51:55.114 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:55.114 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.114 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.115 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:55.115 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.158037+0000", 2024-09-11T09:51:55.115 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:55.115 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.115 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.115 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:55.115 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.158039+0000", 2024-09-11T09:51:55.115 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.356e-06 2024-09-11T09:51:55.115 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.115 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.115 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:55.115 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.176367+0000", 2024-09-11T09:51:55.115 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.018328739 2024-09-11T09:51:55.115 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.116 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.116 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:55.116 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.176378+0000", 2024-09-11T09:51:55.116 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.0713999999999999e-05 2024-09-11T09:51:55.116 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.116 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.116 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:55.116 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.176383+0000", 2024-09-11T09:51:55.116 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.0209999999999999e-06 2024-09-11T09:51:55.116 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.116 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.116 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:55.116 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.176426+0000", 2024-09-11T09:51:55.117 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.2623999999999997e-05 2024-09-11T09:51:55.117 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.117 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.117 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:55.117 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.176552+0000", 2024-09-11T09:51:55.117 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000126076 2024-09-11T09:51:55.117 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.117 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.117 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 1", 2024-09-11T09:51:55.117 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.176665+0000", 2024-09-11T09:51:55.117 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000112995 2024-09-11T09:51:55.117 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.117 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.117 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:55.118 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.183639+0000", 2024-09-11T09:51:55.118 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0069740039999999998 2024-09-11T09:51:55.118 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.118 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.118 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:55.118 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.185689+0000", 2024-09-11T09:51:55.118 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.002049852 2024-09-11T09:51:55.118 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.118 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.118 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:55.118 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.185725+0000", 2024-09-11T09:51:55.118 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.6804999999999997e-05 2024-09-11T09:51:55.118 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.119 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.119 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:55.119 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.185786+0000", 2024-09-11T09:51:55.119 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.0605e-05 2024-09-11T09:51:55.119 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:55.119 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:55.119 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:55.119 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.119 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.119 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4694.0:60 2.0 2:054f2af4:ns::rbd_data.1256a0c02879.000000000000000f:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:55.119 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.186855+0000", 2024-09-11T09:51:55.119 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 31.823905677999999, 2024-09-11T09:51:55.119 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.013682913, 2024-09-11T09:51:55.119 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:55.120 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:55.120 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:55.120 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:55.120 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4694", 2024-09-11T09:51:55.120 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:51:55.120 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 60 2024-09-11T09:51:55.120 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.120 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:55.120 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.120 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:55.120 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.186855+0000", 2024-09-11T09:51:55.120 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:55.120 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.120 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.121 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:55.121 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.186855+0000", 2024-09-11T09:51:55.121 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:55.121 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.121 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.121 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:55.121 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.186856+0000", 2024-09-11T09:51:55.121 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.1200000000000001e-07 2024-09-11T09:51:55.121 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.121 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.121 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:55.121 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.191484+0000", 2024-09-11T09:51:55.121 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0046274159999999996 2024-09-11T09:51:55.121 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.122 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.122 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:55.122 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.191496+0000", 2024-09-11T09:51:55.122 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.2149e-05 2024-09-11T09:51:55.122 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.122 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.122 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:55.122 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.191502+0000", 2024-09-11T09:51:55.122 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.1890000000000004e-06 2024-09-11T09:51:55.122 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.122 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.122 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:55.122 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.191544+0000", 2024-09-11T09:51:55.123 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.1724999999999998e-05 2024-09-11T09:51:55.123 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.123 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.123 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:55.123 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.191641+0000", 2024-09-11T09:51:55.123 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.6939000000000001e-05 2024-09-11T09:51:55.123 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.123 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.123 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 1", 2024-09-11T09:51:55.123 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.191731+0000", 2024-09-11T09:51:55.123 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.0651999999999999e-05 2024-09-11T09:51:55.123 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.123 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.123 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:55.124 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.199852+0000", 2024-09-11T09:51:55.124 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0081204529999999997 2024-09-11T09:51:55.124 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.124 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.124 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:55.124 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.200461+0000", 2024-09-11T09:51:55.124 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00060900899999999998 2024-09-11T09:51:55.124 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.124 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.124 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:55.124 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.200497+0000", 2024-09-11T09:51:55.124 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.5800000000000003e-05 2024-09-11T09:51:55.124 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.125 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.125 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:55.125 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.200538+0000", 2024-09-11T09:51:55.125 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.1668999999999997e-05 2024-09-11T09:51:55.125 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:55.125 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:55.125 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:55.125 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:55.125 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:55.125 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-11T09:51:55.127 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-11T09:51:55.127 INFO:tasks.ceph.osd.2.smithi017.stderr:2024-09-11T09:51:55.127+0000 7fd20b170640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31672) UID: 0 2024-09-11T09:51:55.227 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:51:55.228+0000 7f3c96ec9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31909) UID: 0 2024-09-11T09:51:55.271 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-11T09:51:55.271 INFO:teuthology.orchestra.run.smithi096.stdout: "ops": [], 2024-09-11T09:51:55.271 INFO:teuthology.orchestra.run.smithi096.stdout: "num_ops": 0 2024-09-11T09:51:55.271 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-11T09:51:55.280 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-11T09:51:55.328 INFO:tasks.ceph.osd.2.smithi017.stderr:2024-09-11T09:51:55.329+0000 7fd20b170640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31672) UID: 0 2024-09-11T09:51:55.422 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-11T09:51:55.423 INFO:teuthology.orchestra.run.smithi096.stdout: "ops": [], 2024-09-11T09:51:55.423 INFO:teuthology.orchestra.run.smithi096.stdout: "complaint_time": 30, 2024-09-11T09:51:55.423 INFO:teuthology.orchestra.run.smithi096.stdout: "num_blocked_ops": 0 2024-09-11T09:51:55.423 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-11T09:51:55.429 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:51:55.430+0000 7f3c96ec9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31909) UID: 0 2024-09-11T09:51:55.431 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-11T09:51:55.530 INFO:tasks.ceph.osd.2.smithi017.stderr:2024-09-11T09:51:55.531+0000 7fd20b170640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31672) UID: 0 2024-09-11T09:51:55.575 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-11T09:51:55.576 INFO:teuthology.orchestra.run.smithi096.stdout: "size": 20, 2024-09-11T09:51:55.576 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 600, 2024-09-11T09:51:55.576 INFO:teuthology.orchestra.run.smithi096.stdout: "ops": [ 2024-09-11T09:51:55.576 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.576 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4230.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-11T09:51:55.576 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:18.490736+0000", 2024-09-11T09:51:55.576 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 97.084731567999995, 2024-09-11T09:51:55.576 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0059097569999999999, 2024-09-11T09:51:55.576 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:55.576 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:55.576 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:55.576 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:55.576 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4230", 2024-09-11T09:51:55.577 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.17:0/4101109576", 2024-09-11T09:51:55.577 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 1 2024-09-11T09:51:55.577 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.577 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:55.577 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.577 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:55.577 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.490736+0000", 2024-09-11T09:51:55.577 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:55.577 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.577 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.577 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:55.577 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.490736+0000", 2024-09-11T09:51:55.577 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:55.577 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.577 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.578 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:55.578 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.490742+0000", 2024-09-11T09:51:55.578 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.48e-06 2024-09-11T09:51:55.578 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.578 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.578 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:55.578 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.490749+0000", 2024-09-11T09:51:55.578 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.4569999999999999e-06 2024-09-11T09:51:55.578 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.578 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.578 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:55.578 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.490762+0000", 2024-09-11T09:51:55.578 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.2721000000000001e-05 2024-09-11T09:51:55.578 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.579 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.579 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:55.579 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.490774+0000", 2024-09-11T09:51:55.579 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1823e-05 2024-09-11T09:51:55.579 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.579 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.579 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:55.579 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.490846+0000", 2024-09-11T09:51:55.579 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.1878000000000005e-05 2024-09-11T09:51:55.579 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.579 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.580 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for peered", 2024-09-11T09:51:55.580 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.490857+0000", 2024-09-11T09:51:55.580 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1017e-05 2024-09-11T09:51:55.580 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.580 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.580 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:55.580 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.493856+0000", 2024-09-11T09:51:55.580 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00299885 2024-09-11T09:51:55.580 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.580 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.580 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:55.580 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.494020+0000", 2024-09-11T09:51:55.580 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00016436599999999999 2024-09-11T09:51:55.580 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.581 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.581 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 2", 2024-09-11T09:51:55.581 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.494186+0000", 2024-09-11T09:51:55.581 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00016562799999999999 2024-09-11T09:51:55.581 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.581 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.581 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:55.581 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.495407+0000", 2024-09-11T09:51:55.581 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0012217420000000001 2024-09-11T09:51:55.581 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.581 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.581 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:55.581 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.496579+0000", 2024-09-11T09:51:55.581 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001171844 2024-09-11T09:51:55.582 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.582 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.582 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:55.582 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.496620+0000", 2024-09-11T09:51:55.582 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.0479000000000003e-05 2024-09-11T09:51:55.582 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.582 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.582 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:55.582 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.496646+0000", 2024-09-11T09:51:55.582 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.6472e-05 2024-09-11T09:51:55.582 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:55.582 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:55.582 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:55.582 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.583 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.583 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4230.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-09-11T09:51:55.583 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:20.498634+0000", 2024-09-11T09:51:55.583 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 95.076833547000007, 2024-09-11T09:51:55.583 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0050648539999999997, 2024-09-11T09:51:55.583 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:55.583 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:55.583 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:55.583 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:55.583 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4230", 2024-09-11T09:51:55.583 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.17:0/4101109576", 2024-09-11T09:51:55.583 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 7 2024-09-11T09:51:55.583 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.583 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:55.583 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.584 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:55.584 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.498634+0000", 2024-09-11T09:51:55.584 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:55.584 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.584 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.584 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:55.584 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.498634+0000", 2024-09-11T09:51:55.584 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:55.584 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.584 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.584 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:55.584 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.498637+0000", 2024-09-11T09:51:55.584 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.1919999999999999e-06 2024-09-11T09:51:55.584 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.585 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.585 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:55.585 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.498648+0000", 2024-09-11T09:51:55.585 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1116e-05 2024-09-11T09:51:55.585 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.585 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.585 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:55.585 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.498666+0000", 2024-09-11T09:51:55.585 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.8411999999999999e-05 2024-09-11T09:51:55.585 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.585 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.585 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:55.585 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.498675+0000", 2024-09-11T09:51:55.585 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.4059999999999994e-06 2024-09-11T09:51:55.585 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.586 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.586 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:55.586 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.498745+0000", 2024-09-11T09:51:55.586 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.9764000000000005e-05 2024-09-11T09:51:55.586 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.586 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.586 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op must wait for map", 2024-09-11T09:51:55.586 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.498754+0000", 2024-09-11T09:51:55.586 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.5969999999999992e-06 2024-09-11T09:51:55.586 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.586 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.586 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:55.586 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.500385+0000", 2024-09-11T09:51:55.586 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0016308360000000001 2024-09-11T09:51:55.587 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.587 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.587 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:55.587 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.500473+0000", 2024-09-11T09:51:55.587 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.7966000000000004e-05 2024-09-11T09:51:55.587 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.587 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.587 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 1", 2024-09-11T09:51:55.587 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.500591+0000", 2024-09-11T09:51:55.587 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000117895 2024-09-11T09:51:55.587 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.587 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.587 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:55.587 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.501685+0000", 2024-09-11T09:51:55.588 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001094886 2024-09-11T09:51:55.588 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.588 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.588 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:55.588 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.503627+0000", 2024-09-11T09:51:55.588 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001941199 2024-09-11T09:51:55.588 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.588 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.588 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:55.588 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.503667+0000", 2024-09-11T09:51:55.588 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.0645000000000001e-05 2024-09-11T09:51:55.588 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.588 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.588 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:55.589 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.503699+0000", 2024-09-11T09:51:55.589 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.1940000000000003e-05 2024-09-11T09:51:55.589 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:55.589 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:55.589 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:55.589 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.589 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.589 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4688.0:32 2.2 2:45941fd2:::rbd_data.1250f5e25b80.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:55.589 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:22.896607+0000", 2024-09-11T09:51:55.589 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 32.678860944, 2024-09-11T09:51:55.589 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.038655179999999997, 2024-09-11T09:51:55.589 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:55.589 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:55.590 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:55.590 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:55.590 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4688", 2024-09-11T09:51:55.590 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:51:55.590 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 32 2024-09-11T09:51:55.590 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.590 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:55.590 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.590 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:55.590 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.896607+0000", 2024-09-11T09:51:55.590 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:55.590 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.590 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.590 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:55.590 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.896607+0000", 2024-09-11T09:51:55.591 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:55.591 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.591 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.591 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:55.591 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.896608+0000", 2024-09-11T09:51:55.591 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.0690000000000001e-06 2024-09-11T09:51:55.591 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.591 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.591 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:55.591 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.914461+0000", 2024-09-11T09:51:55.591 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.017852903 2024-09-11T09:51:55.591 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.591 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.591 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:55.591 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.914495+0000", 2024-09-11T09:51:55.592 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.4301000000000002e-05 2024-09-11T09:51:55.592 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.592 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.592 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:55.592 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.914522+0000", 2024-09-11T09:51:55.592 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.7127e-05 2024-09-11T09:51:55.592 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.592 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.592 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:55.592 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.914583+0000", 2024-09-11T09:51:55.592 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.0730999999999998e-05 2024-09-11T09:51:55.592 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.592 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.593 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:55.593 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.914709+0000", 2024-09-11T09:51:55.593 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00012590400000000001 2024-09-11T09:51:55.593 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.593 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.593 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 1", 2024-09-11T09:51:55.593 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.914809+0000", 2024-09-11T09:51:55.593 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.9882000000000004e-05 2024-09-11T09:51:55.593 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.593 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.593 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:55.593 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.932551+0000", 2024-09-11T09:51:55.593 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.017742048999999999 2024-09-11T09:51:55.593 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.594 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.594 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:55.594 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.935144+0000", 2024-09-11T09:51:55.594 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0025927630000000001 2024-09-11T09:51:55.594 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.594 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.594 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:55.594 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.935205+0000", 2024-09-11T09:51:55.594 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.1634e-05 2024-09-11T09:51:55.594 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.594 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.594 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:55.594 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.935262+0000", 2024-09-11T09:51:55.594 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.6817000000000002e-05 2024-09-11T09:51:55.595 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:55.595 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:55.595 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:55.595 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.595 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.595 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_repop(client.4688.0:40 2.7 e53/38 2:fe41411c:::rbd_object_map.1250f5e25b80:head v 53'9, mlcod=53'7)", 2024-09-11T09:51:55.595 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:22.920457+0000", 2024-09-11T09:51:55.595 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 32.655010853, 2024-09-11T09:51:55.595 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.011132817, 2024-09-11T09:51:55.595 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:55.595 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:55.595 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:55.595 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:55.595 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.595 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:55.596 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.920457+0000", 2024-09-11T09:51:55.596 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:55.596 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.596 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.596 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:55.596 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.920457+0000", 2024-09-11T09:51:55.596 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:55.596 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.596 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.596 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:55.596 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.920459+0000", 2024-09-11T09:51:55.596 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.6729999999999999e-06 2024-09-11T09:51:55.596 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.596 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.597 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:55.597 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.920469+0000", 2024-09-11T09:51:55.597 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.944e-06 2024-09-11T09:51:55.597 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.597 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.597 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:55.597 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.920495+0000", 2024-09-11T09:51:55.597 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.6401999999999999e-05 2024-09-11T09:51:55.597 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.597 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.597 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:55.597 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.920506+0000", 2024-09-11T09:51:55.597 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.133e-05 2024-09-11T09:51:55.597 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.597 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.598 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:55.598 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.920581+0000", 2024-09-11T09:51:55.598 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.4932999999999996e-05 2024-09-11T09:51:55.598 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.598 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.598 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:55.598 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.920612+0000", 2024-09-11T09:51:55.598 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.0537999999999998e-05 2024-09-11T09:51:55.598 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.598 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.598 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:55.598 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.931537+0000", 2024-09-11T09:51:55.598 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.010925239 2024-09-11T09:51:55.598 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.598 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.599 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:55.599 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.931590+0000", 2024-09-11T09:51:55.599 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.2757999999999999e-05 2024-09-11T09:51:55.599 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:55.599 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:55.599 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:55.599 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.599 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.599 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_repop(client.4688.0:41 2.7 e53/38 2:fe41411c:::rbd_object_map.1250f5e25b80:head v 53'10, mlcod=53'7)", 2024-09-11T09:51:55.599 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:22.921422+0000", 2024-09-11T09:51:55.599 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 32.654045437000001, 2024-09-11T09:51:55.599 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.013734942, 2024-09-11T09:51:55.599 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:55.599 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:55.599 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:55.600 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:55.600 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.600 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:55.600 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921422+0000", 2024-09-11T09:51:55.600 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:55.600 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.600 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.600 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:55.600 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921422+0000", 2024-09-11T09:51:55.600 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:55.600 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.600 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.600 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:55.600 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921426+0000", 2024-09-11T09:51:55.600 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.6150000000000001e-06 2024-09-11T09:51:55.601 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.601 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.601 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:55.601 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921435+0000", 2024-09-11T09:51:55.601 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.9840000000000007e-06 2024-09-11T09:51:55.601 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.601 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.601 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:55.601 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921444+0000", 2024-09-11T09:51:55.601 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.6000000000000007e-06 2024-09-11T09:51:55.601 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.601 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.601 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:55.601 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921450+0000", 2024-09-11T09:51:55.601 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.4690000000000002e-06 2024-09-11T09:51:55.602 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.602 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.602 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:55.602 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921511+0000", 2024-09-11T09:51:55.602 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.0779999999999997e-05 2024-09-11T09:51:55.602 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.602 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.602 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:55.602 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921531+0000", 2024-09-11T09:51:55.602 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.0302999999999999e-05 2024-09-11T09:51:55.602 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.602 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.602 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:55.602 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.935115+0000", 2024-09-11T09:51:55.603 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.013583471 2024-09-11T09:51:55.603 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.603 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.603 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:55.603 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.935157+0000", 2024-09-11T09:51:55.603 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.2719999999999998e-05 2024-09-11T09:51:55.603 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:55.603 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:55.603 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:55.603 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.603 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.603 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4688.0:42 2.3 2:c54742d8:::rbd_data.1250f5e25b80.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:55.603 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:22.932560+0000", 2024-09-11T09:51:55.603 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 32.642907694000002, 2024-09-11T09:51:55.603 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.017798198000000001, 2024-09-11T09:51:55.604 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:55.604 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:55.604 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:55.604 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:55.604 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4688", 2024-09-11T09:51:55.604 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:51:55.604 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 42 2024-09-11T09:51:55.604 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.604 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:55.604 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.604 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:55.604 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.932560+0000", 2024-09-11T09:51:55.604 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:55.604 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.605 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.605 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:55.605 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.932560+0000", 2024-09-11T09:51:55.605 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:55.605 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.605 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.605 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:55.605 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.932562+0000", 2024-09-11T09:51:55.605 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.17e-06 2024-09-11T09:51:55.605 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.605 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.605 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:55.605 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.938851+0000", 2024-09-11T09:51:55.605 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.006288971 2024-09-11T09:51:55.605 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.606 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.606 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:55.606 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.938865+0000", 2024-09-11T09:51:55.606 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.365e-05 2024-09-11T09:51:55.606 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.606 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.606 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:55.606 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.938871+0000", 2024-09-11T09:51:55.606 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.8850000000000004e-06 2024-09-11T09:51:55.606 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.606 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.606 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:55.606 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.938926+0000", 2024-09-11T09:51:55.606 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.4691999999999997e-05 2024-09-11T09:51:55.606 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.607 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.607 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:55.607 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.939059+0000", 2024-09-11T09:51:55.607 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00013310899999999999 2024-09-11T09:51:55.607 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.607 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.607 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 2", 2024-09-11T09:51:55.607 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.939171+0000", 2024-09-11T09:51:55.607 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000112718 2024-09-11T09:51:55.607 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.607 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.607 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:55.607 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.950262+0000", 2024-09-11T09:51:55.607 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.011091063 2024-09-11T09:51:55.608 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.608 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.608 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:55.608 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.950274+0000", 2024-09-11T09:51:55.608 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1963e-05 2024-09-11T09:51:55.608 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.608 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.608 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:55.608 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.950318+0000", 2024-09-11T09:51:55.608 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.3519999999999997e-05 2024-09-11T09:51:55.608 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.608 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.608 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:55.608 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.950358+0000", 2024-09-11T09:51:55.609 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.0457000000000002e-05 2024-09-11T09:51:55.609 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:55.609 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:55.609 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:55.609 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.609 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.609 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_repop(client.4694.0:34 2.7 e53/38 2:f80bfc0c:ns::rbd_data.1256a0c02879.0000000000000002:head v 53'25, mlcod=53'23)", 2024-09-11T09:51:55.609 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.094944+0000", 2024-09-11T09:51:55.609 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 32.480524301999999, 2024-09-11T09:51:55.609 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.010896785000000001, 2024-09-11T09:51:55.609 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:55.609 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:55.609 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:55.609 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:55.609 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.610 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:55.610 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.094944+0000", 2024-09-11T09:51:55.610 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:55.610 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.610 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.610 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:55.610 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.094944+0000", 2024-09-11T09:51:55.610 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:55.610 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.610 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.610 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:55.610 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.094944+0000", 2024-09-11T09:51:55.610 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.0699999999999996e-07 2024-09-11T09:51:55.610 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.610 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.611 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:55.611 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.099060+0000", 2024-09-11T09:51:55.611 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0041155580000000001 2024-09-11T09:51:55.611 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.611 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.611 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:55.611 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.099082+0000", 2024-09-11T09:51:55.611 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.2323999999999998e-05 2024-09-11T09:51:55.611 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.611 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.611 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:55.611 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.099094+0000", 2024-09-11T09:51:55.611 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1535000000000001e-05 2024-09-11T09:51:55.611 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.612 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.612 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:55.612 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.099160+0000", 2024-09-11T09:51:55.612 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.6449e-05 2024-09-11T09:51:55.612 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.612 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.612 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:55.612 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.099183+0000", 2024-09-11T09:51:55.612 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.3108e-05 2024-09-11T09:51:55.612 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.612 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.612 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:55.612 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.105776+0000", 2024-09-11T09:51:55.613 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0065922760000000002 2024-09-11T09:51:55.613 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.613 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.613 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:55.613 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.105840+0000", 2024-09-11T09:51:55.613 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.4727999999999994e-05 2024-09-11T09:51:55.613 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:55.613 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:55.613 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:55.613 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.613 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.613 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_repop(client.4694.0:35 2.7 e53/38 2:e8766325:ns::rbd_object_map.1256a0c02879:head v 53'26, mlcod=53'23)", 2024-09-11T09:51:55.613 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.100964+0000", 2024-09-11T09:51:55.613 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 32.474503773999999, 2024-09-11T09:51:55.613 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0049649359999999997, 2024-09-11T09:51:55.614 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:55.614 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:55.614 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:55.614 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:55.614 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.614 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:55.614 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.100964+0000", 2024-09-11T09:51:55.614 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:55.614 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.614 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.614 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:55.614 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.100964+0000", 2024-09-11T09:51:55.614 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:55.615 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.615 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.615 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:55.615 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.100966+0000", 2024-09-11T09:51:55.615 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.6330000000000001e-06 2024-09-11T09:51:55.615 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.615 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.615 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:55.615 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.100979+0000", 2024-09-11T09:51:55.615 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.2813000000000001e-05 2024-09-11T09:51:55.615 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.615 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.615 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:55.615 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.100992+0000", 2024-09-11T09:51:55.615 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.3414999999999999e-05 2024-09-11T09:51:55.615 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.616 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.616 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:55.616 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.101002+0000", 2024-09-11T09:51:55.616 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.0159e-05 2024-09-11T09:51:55.616 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.616 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.616 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:55.616 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.102365+0000", 2024-09-11T09:51:55.616 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001362912 2024-09-11T09:51:55.616 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.616 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.616 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:55.616 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.102382+0000", 2024-09-11T09:51:55.616 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.7433000000000001e-05 2024-09-11T09:51:55.617 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.617 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.617 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:55.617 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.105901+0000", 2024-09-11T09:51:55.617 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.003518476 2024-09-11T09:51:55.617 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.617 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.617 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:55.617 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.105929+0000", 2024-09-11T09:51:55.617 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.8095000000000001e-05 2024-09-11T09:51:55.617 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:55.617 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:55.617 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:55.618 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.618 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.618 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4694.0:41 2.2 2:5e2505c4:ns::rbd_data.1256a0c02879.0000000000000003:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:55.618 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.107511+0000", 2024-09-11T09:51:55.618 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 32.467956620999999, 2024-09-11T09:51:55.618 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.018571066000000001, 2024-09-11T09:51:55.618 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:55.618 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:55.618 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:55.618 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:55.618 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4694", 2024-09-11T09:51:55.618 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:51:55.618 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 41 2024-09-11T09:51:55.619 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.619 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:55.619 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.619 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:55.619 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.107511+0000", 2024-09-11T09:51:55.619 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:55.619 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.619 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.619 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:55.619 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.107511+0000", 2024-09-11T09:51:55.619 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:55.619 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.619 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.619 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:55.620 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.107513+0000", 2024-09-11T09:51:55.620 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.3489999999999999e-06 2024-09-11T09:51:55.620 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.620 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.620 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:55.620 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.112704+0000", 2024-09-11T09:51:55.620 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0051918509999999999 2024-09-11T09:51:55.620 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.620 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.620 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:55.620 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.112718+0000", 2024-09-11T09:51:55.620 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.359e-05 2024-09-11T09:51:55.620 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.620 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.621 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:55.621 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.112725+0000", 2024-09-11T09:51:55.621 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.9940000000000003e-06 2024-09-11T09:51:55.621 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.621 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.621 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:55.621 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.112788+0000", 2024-09-11T09:51:55.621 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.2447999999999996e-05 2024-09-11T09:51:55.621 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.621 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.621 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:55.621 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.112937+0000", 2024-09-11T09:51:55.621 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00014921499999999999 2024-09-11T09:51:55.621 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.622 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.622 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 1", 2024-09-11T09:51:55.622 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.113085+0000", 2024-09-11T09:51:55.622 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000148085 2024-09-11T09:51:55.622 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.622 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.622 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:55.622 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.124318+0000", 2024-09-11T09:51:55.622 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.011233158 2024-09-11T09:51:55.622 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.622 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.622 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:55.622 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.125957+0000", 2024-09-11T09:51:55.623 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001638701 2024-09-11T09:51:55.623 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.623 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.623 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:55.623 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.126029+0000", 2024-09-11T09:51:55.623 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.2125000000000005e-05 2024-09-11T09:51:55.623 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.623 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.623 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:55.623 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.126082+0000", 2024-09-11T09:51:55.623 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.3550000000000001e-05 2024-09-11T09:51:55.623 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:55.623 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:55.624 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:55.624 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.624 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.624 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4694.0:44 2.3 2:cd1a7d1c:ns::rbd_data.1256a0c02879.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:55.624 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.112744+0000", 2024-09-11T09:51:55.624 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 32.462724053999999, 2024-09-11T09:51:55.624 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.021672212999999999, 2024-09-11T09:51:55.624 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:55.624 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:55.624 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:55.624 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:55.624 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4694", 2024-09-11T09:51:55.624 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:51:55.625 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 44 2024-09-11T09:51:55.625 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.625 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:55.625 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.625 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:55.625 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.112744+0000", 2024-09-11T09:51:55.625 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:55.625 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.625 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.625 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:55.625 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.112744+0000", 2024-09-11T09:51:55.625 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:55.625 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.625 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.626 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:55.626 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.112745+0000", 2024-09-11T09:51:55.626 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.2900000000000002e-07 2024-09-11T09:51:55.626 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.626 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.626 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:55.626 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.119929+0000", 2024-09-11T09:51:55.626 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0071846230000000002 2024-09-11T09:51:55.626 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.626 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.626 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:55.626 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.119949+0000", 2024-09-11T09:51:55.626 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.9264000000000001e-05 2024-09-11T09:51:55.627 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.627 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.627 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:55.627 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.119959+0000", 2024-09-11T09:51:55.627 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.0465999999999999e-05 2024-09-11T09:51:55.627 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.627 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.627 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:55.627 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.120028+0000", 2024-09-11T09:51:55.627 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.9318000000000003e-05 2024-09-11T09:51:55.627 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.627 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.627 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:55.628 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.120160+0000", 2024-09-11T09:51:55.628 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000132015 2024-09-11T09:51:55.628 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.628 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.628 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 2", 2024-09-11T09:51:55.628 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.120260+0000", 2024-09-11T09:51:55.628 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.9397999999999999e-05 2024-09-11T09:51:55.628 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.628 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.628 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:55.628 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.134246+0000", 2024-09-11T09:51:55.628 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.013986721000000001 2024-09-11T09:51:55.628 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.629 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.629 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:55.629 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.134295+0000", 2024-09-11T09:51:55.629 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.8649000000000001e-05 2024-09-11T09:51:55.629 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.629 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.629 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:55.629 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.134361+0000", 2024-09-11T09:51:55.629 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.6081999999999995e-05 2024-09-11T09:51:55.629 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.629 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.629 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:55.629 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.134416+0000", 2024-09-11T09:51:55.630 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.4848000000000001e-05 2024-09-11T09:51:55.630 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:55.630 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:55.630 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:55.630 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.630 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.630 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4694.0:46 2.3 2:ce0e1469:ns::rbd_data.1256a0c02879.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:55.630 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.119993+0000", 2024-09-11T09:51:55.630 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 32.455474578, 2024-09-11T09:51:55.630 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.025840610999999999, 2024-09-11T09:51:55.630 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:55.630 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:55.630 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:55.630 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:55.631 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4694", 2024-09-11T09:51:55.631 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:51:55.631 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 46 2024-09-11T09:51:55.631 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.631 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:55.631 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.631 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:55.631 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.119993+0000", 2024-09-11T09:51:55.631 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:55.631 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.631 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.631 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:55.631 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.119993+0000", 2024-09-11T09:51:55.632 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:55.632 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.632 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.632 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:55.632 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.119995+0000", 2024-09-11T09:51:55.632 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1930000000000001e-06 2024-09-11T09:51:55.632 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.632 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.632 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:55.632 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.128458+0000", 2024-09-11T09:51:55.632 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0084632630000000004 2024-09-11T09:51:55.632 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.633 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.633 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:55.633 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.128478+0000", 2024-09-11T09:51:55.633 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.0353e-05 2024-09-11T09:51:55.633 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.633 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.633 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:55.633 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.128491+0000", 2024-09-11T09:51:55.633 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.2954e-05 2024-09-11T09:51:55.633 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.633 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.633 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:55.633 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.128558+0000", 2024-09-11T09:51:55.634 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.7076999999999995e-05 2024-09-11T09:51:55.634 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.634 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.634 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:55.634 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.128708+0000", 2024-09-11T09:51:55.634 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00014939599999999999 2024-09-11T09:51:55.634 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.634 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.634 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 2", 2024-09-11T09:51:55.634 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.128866+0000", 2024-09-11T09:51:55.634 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000158145 2024-09-11T09:51:55.634 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.634 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.634 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:55.635 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.141045+0000", 2024-09-11T09:51:55.635 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.012179331999999999 2024-09-11T09:51:55.635 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.635 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.635 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:55.635 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.145742+0000", 2024-09-11T09:51:55.635 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0046972050000000003 2024-09-11T09:51:55.635 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.635 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.635 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:55.635 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.145779+0000", 2024-09-11T09:51:55.635 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.6544000000000001e-05 2024-09-11T09:51:55.635 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.635 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.636 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:55.636 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.145834+0000", 2024-09-11T09:51:55.636 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.5148999999999997e-05 2024-09-11T09:51:55.636 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:55.636 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:55.636 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:55.636 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.636 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.636 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4694.0:48 2.3 2:d2090b70:ns::rbd_data.1256a0c02879.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:55.636 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.130540+0000", 2024-09-11T09:51:55.636 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 32.444928312000002, 2024-09-11T09:51:55.636 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.017167789999999999, 2024-09-11T09:51:55.636 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:55.637 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:55.637 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:55.637 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:55.637 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4694", 2024-09-11T09:51:55.637 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:51:55.637 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 48 2024-09-11T09:51:55.637 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.637 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:55.637 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.637 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:55.637 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.130540+0000", 2024-09-11T09:51:55.637 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:55.637 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.637 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.638 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:55.638 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.130540+0000", 2024-09-11T09:51:55.638 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:55.638 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.638 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.638 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:55.638 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.130542+0000", 2024-09-11T09:51:55.638 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.9209999999999999e-06 2024-09-11T09:51:55.638 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.638 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.638 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:55.638 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.134691+0000", 2024-09-11T09:51:55.638 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0041498739999999996 2024-09-11T09:51:55.638 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.638 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.639 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:55.639 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.134705+0000", 2024-09-11T09:51:55.639 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.3669000000000001e-05 2024-09-11T09:51:55.639 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.639 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.639 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:55.639 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.134712+0000", 2024-09-11T09:51:55.639 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.8410000000000002e-06 2024-09-11T09:51:55.639 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.639 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.639 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:55.639 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.134754+0000", 2024-09-11T09:51:55.639 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.2524999999999997e-05 2024-09-11T09:51:55.639 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.640 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.640 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:55.640 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.134875+0000", 2024-09-11T09:51:55.640 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000120332 2024-09-11T09:51:55.640 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.640 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.640 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 2", 2024-09-11T09:51:55.640 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.135005+0000", 2024-09-11T09:51:55.640 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00013024900000000001 2024-09-11T09:51:55.640 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.640 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.640 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:55.640 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.145971+0000", 2024-09-11T09:51:55.640 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.010966203000000001 2024-09-11T09:51:55.641 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.641 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.641 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:55.641 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.147600+0000", 2024-09-11T09:51:55.641 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001628382 2024-09-11T09:51:55.641 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.641 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.641 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:55.641 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.147635+0000", 2024-09-11T09:51:55.641 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.5420000000000003e-05 2024-09-11T09:51:55.641 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.641 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.641 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:55.642 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.147707+0000", 2024-09-11T09:51:55.642 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.2373999999999995e-05 2024-09-11T09:51:55.642 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:55.642 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:55.642 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:55.642 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.642 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.642 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_repop(client.4694.0:49 2.7 e53/38 2:e8766325:ns::rbd_object_map.1256a0c02879:head v 53'33, mlcod=53'31)", 2024-09-11T09:51:55.642 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.135391+0000", 2024-09-11T09:51:55.642 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 32.440076746999999, 2024-09-11T09:51:55.642 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0051189859999999998, 2024-09-11T09:51:55.642 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:55.642 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:55.642 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:55.643 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:55.643 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.643 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:55.643 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.135391+0000", 2024-09-11T09:51:55.643 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:55.643 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.643 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.643 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:55.643 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.135391+0000", 2024-09-11T09:51:55.643 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:55.643 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.643 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.643 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:55.643 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.135393+0000", 2024-09-11T09:51:55.644 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.4020000000000001e-06 2024-09-11T09:51:55.644 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.644 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.644 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:55.644 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.135400+0000", 2024-09-11T09:51:55.644 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.5129999999999999e-06 2024-09-11T09:51:55.644 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.644 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.644 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:55.644 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.135411+0000", 2024-09-11T09:51:55.644 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1038e-05 2024-09-11T09:51:55.644 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.644 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.645 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:55.645 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.135418+0000", 2024-09-11T09:51:55.645 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.1489999999999997e-06 2024-09-11T09:51:55.645 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.645 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.645 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:55.645 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.135470+0000", 2024-09-11T09:51:55.645 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.2021999999999998e-05 2024-09-11T09:51:55.645 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.645 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.646 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:55.646 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.135501+0000", 2024-09-11T09:51:55.646 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.0349999999999999e-05 2024-09-11T09:51:55.646 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.646 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.646 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:55.646 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.140453+0000", 2024-09-11T09:51:55.646 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0049522409999999996 2024-09-11T09:51:55.646 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.646 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.646 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:55.646 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.140510+0000", 2024-09-11T09:51:55.646 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.7271000000000001e-05 2024-09-11T09:51:55.646 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:55.647 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:55.647 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:55.647 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.647 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.647 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4694.0:54 2.3 2:c44f0253:ns::rbd_data.1256a0c02879.000000000000000b:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:55.647 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.150151+0000", 2024-09-11T09:51:55.647 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 32.425316895999998, 2024-09-11T09:51:55.647 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.015103484, 2024-09-11T09:51:55.647 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:55.647 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:55.647 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:55.647 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:55.647 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4694", 2024-09-11T09:51:55.647 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:51:55.648 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 54 2024-09-11T09:51:55.648 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.648 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:55.648 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.648 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:55.648 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.150151+0000", 2024-09-11T09:51:55.648 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:55.648 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.648 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.648 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:55.648 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.150151+0000", 2024-09-11T09:51:55.648 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:55.648 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.649 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.649 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:55.649 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.150152+0000", 2024-09-11T09:51:55.649 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.2550000000000001e-06 2024-09-11T09:51:55.649 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.649 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.649 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:55.649 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.154528+0000", 2024-09-11T09:51:55.649 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0043758219999999997 2024-09-11T09:51:55.649 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.649 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.649 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:55.649 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.154541+0000", 2024-09-11T09:51:55.649 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.3414999999999999e-05 2024-09-11T09:51:55.650 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.650 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.650 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:55.650 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.154551+0000", 2024-09-11T09:51:55.650 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.4350000000000003e-06 2024-09-11T09:51:55.650 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.650 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.650 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:55.650 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.154615+0000", 2024-09-11T09:51:55.650 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.3627000000000006e-05 2024-09-11T09:51:55.650 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.650 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.651 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:55.651 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.154717+0000", 2024-09-11T09:51:55.651 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000102729 2024-09-11T09:51:55.651 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.651 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.651 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 2", 2024-09-11T09:51:55.651 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.154826+0000", 2024-09-11T09:51:55.651 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000108409 2024-09-11T09:51:55.651 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.651 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.651 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:55.651 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.165094+0000", 2024-09-11T09:51:55.651 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.010268210999999999 2024-09-11T09:51:55.651 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.652 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.652 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:55.652 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.165138+0000", 2024-09-11T09:51:55.652 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.4078e-05 2024-09-11T09:51:55.652 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.652 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.652 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:55.652 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.165204+0000", 2024-09-11T09:51:55.652 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.5716000000000006e-05 2024-09-11T09:51:55.652 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.652 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.652 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:55.652 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.165254+0000", 2024-09-11T09:51:55.652 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.0787e-05 2024-09-11T09:51:55.653 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:55.653 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:55.653 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:55.653 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.653 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.653 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_repop(client.4694.0:55 2.7 e53/38 2:e8766325:ns::rbd_object_map.1256a0c02879:head v 53'37, mlcod=53'34)", 2024-09-11T09:51:55.653 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.150221+0000", 2024-09-11T09:51:55.653 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 32.425247030999998, 2024-09-11T09:51:55.653 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00530015, 2024-09-11T09:51:55.653 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:55.653 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:55.653 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:55.653 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:55.654 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.654 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:55.654 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.150221+0000", 2024-09-11T09:51:55.654 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:55.654 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.654 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.654 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:55.654 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.150221+0000", 2024-09-11T09:51:55.654 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:55.654 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.654 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.654 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:55.654 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.150221+0000", 2024-09-11T09:51:55.654 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.9100000000000004e-07 2024-09-11T09:51:55.655 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.655 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.655 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:55.655 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.150226+0000", 2024-09-11T09:51:55.655 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.2250000000000002e-06 2024-09-11T09:51:55.655 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.655 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.655 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:55.655 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.150251+0000", 2024-09-11T09:51:55.655 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.4987e-05 2024-09-11T09:51:55.655 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.655 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.655 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:55.655 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.150258+0000", 2024-09-11T09:51:55.655 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.0790000000000003e-06 2024-09-11T09:51:55.656 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.656 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.656 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:55.656 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.150305+0000", 2024-09-11T09:51:55.656 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.6879000000000003e-05 2024-09-11T09:51:55.656 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.656 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.656 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:55.656 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.150320+0000", 2024-09-11T09:51:55.656 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.5268999999999999e-05 2024-09-11T09:51:55.656 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.656 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.656 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:55.656 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.155493+0000", 2024-09-11T09:51:55.656 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0051731390000000002 2024-09-11T09:51:55.657 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.657 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.657 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:55.657 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.155521+0000", 2024-09-11T09:51:55.657 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.8081000000000001e-05 2024-09-11T09:51:55.657 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:55.657 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:55.657 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:55.657 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.657 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.657 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4694.0:56 2.3 2:c462f029:ns::rbd_data.1256a0c02879.000000000000000c:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:55.657 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.154576+0000", 2024-09-11T09:51:55.657 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 32.420892055000003, 2024-09-11T09:51:55.657 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.020781621, 2024-09-11T09:51:55.658 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:55.658 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:55.658 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:55.658 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:55.658 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4694", 2024-09-11T09:51:55.658 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:51:55.658 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 56 2024-09-11T09:51:55.658 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.658 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:55.658 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.658 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:55.658 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.154576+0000", 2024-09-11T09:51:55.658 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:55.659 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.659 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.659 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:55.659 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.154576+0000", 2024-09-11T09:51:55.659 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:55.659 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.659 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.659 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:55.659 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.154577+0000", 2024-09-11T09:51:55.659 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.2249999999999999e-06 2024-09-11T09:51:55.659 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.659 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.659 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:55.659 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.158783+0000", 2024-09-11T09:51:55.660 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0042063889999999996 2024-09-11T09:51:55.660 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.660 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.660 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:55.660 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.158803+0000", 2024-09-11T09:51:55.660 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.9576999999999999e-05 2024-09-11T09:51:55.660 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.660 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.660 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:55.660 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.158815+0000", 2024-09-11T09:51:55.660 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.2221999999999999e-05 2024-09-11T09:51:55.660 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.660 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.660 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:55.661 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.158902+0000", 2024-09-11T09:51:55.661 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.6390000000000005e-05 2024-09-11T09:51:55.661 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.661 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.661 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:55.661 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.159141+0000", 2024-09-11T09:51:55.661 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000238886 2024-09-11T09:51:55.661 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.661 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.661 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 2", 2024-09-11T09:51:55.661 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.159279+0000", 2024-09-11T09:51:55.661 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000138073 2024-09-11T09:51:55.661 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.661 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.661 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:55.662 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.173297+0000", 2024-09-11T09:51:55.662 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.014018113 2024-09-11T09:51:55.662 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.662 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.662 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:55.662 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.175278+0000", 2024-09-11T09:51:55.662 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0019817390000000002 2024-09-11T09:51:55.662 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.662 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.662 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:55.662 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.175316+0000", 2024-09-11T09:51:55.662 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.7829e-05 2024-09-11T09:51:55.663 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.663 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.663 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:55.663 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.175357+0000", 2024-09-11T09:51:55.663 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.1177999999999998e-05 2024-09-11T09:51:55.663 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:55.663 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:55.663 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:55.663 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.663 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.663 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4694.0:59 2.2 2:41adefc9:ns::rbd_data.1256a0c02879.000000000000000e:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:55.663 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.158843+0000", 2024-09-11T09:51:55.663 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 32.416624489, 2024-09-11T09:51:55.663 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.017874207, 2024-09-11T09:51:55.663 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:55.663 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:55.664 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:55.664 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:55.664 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4694", 2024-09-11T09:51:55.664 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:51:55.664 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 59 2024-09-11T09:51:55.664 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.664 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:55.664 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.664 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:55.664 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.158843+0000", 2024-09-11T09:51:55.664 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:55.664 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.664 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.664 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:55.665 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.158843+0000", 2024-09-11T09:51:55.665 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:55.665 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.665 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.665 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:55.665 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.158845+0000", 2024-09-11T09:51:55.665 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.7439999999999999e-06 2024-09-11T09:51:55.665 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.665 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.665 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:55.665 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.162713+0000", 2024-09-11T09:51:55.665 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0038678829999999999 2024-09-11T09:51:55.665 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.665 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.666 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:55.666 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.162728+0000", 2024-09-11T09:51:55.666 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.5114e-05 2024-09-11T09:51:55.666 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.666 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.666 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:55.666 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.162737+0000", 2024-09-11T09:51:55.666 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.1099999999999992e-06 2024-09-11T09:51:55.666 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.666 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.666 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:55.666 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.162799+0000", 2024-09-11T09:51:55.667 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.1934999999999996e-05 2024-09-11T09:51:55.667 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.667 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.667 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:55.667 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.162911+0000", 2024-09-11T09:51:55.667 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000111867 2024-09-11T09:51:55.667 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.667 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.667 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 1", 2024-09-11T09:51:55.667 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.163030+0000", 2024-09-11T09:51:55.667 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000119108 2024-09-11T09:51:55.667 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.667 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.667 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:55.668 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.174086+0000", 2024-09-11T09:51:55.668 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.01105543 2024-09-11T09:51:55.668 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.668 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.668 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:55.668 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.176614+0000", 2024-09-11T09:51:55.668 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0025284940000000001 2024-09-11T09:51:55.668 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.668 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.668 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:55.668 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.176672+0000", 2024-09-11T09:51:55.668 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.7892000000000001e-05 2024-09-11T09:51:55.668 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.668 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.669 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:55.669 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.176718+0000", 2024-09-11T09:51:55.669 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.5630000000000002e-05 2024-09-11T09:51:55.669 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:55.669 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:55.669 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:55.669 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.669 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.669 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.5002.0:89 2.3 2:c54742d8:::rbd_data.1250f5e25b80.0000000000000006:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-09-11T09:51:55.669 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:38.546016+0000", 2024-09-11T09:51:55.669 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 17.029451943000002, 2024-09-11T09:51:55.669 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0049098550000000003, 2024-09-11T09:51:55.670 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:55.670 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:55.670 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:55.670 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:55.670 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.5002", 2024-09-11T09:51:55.670 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/2204440692", 2024-09-11T09:51:55.670 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 89 2024-09-11T09:51:55.670 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.670 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:55.670 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.670 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:55.670 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.546016+0000", 2024-09-11T09:51:55.670 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:55.671 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.671 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.671 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:55.671 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.546016+0000", 2024-09-11T09:51:55.671 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:55.671 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.671 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.671 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:55.671 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.546017+0000", 2024-09-11T09:51:55.671 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.3399999999999998e-07 2024-09-11T09:51:55.671 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.671 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.672 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:55.672 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.546839+0000", 2024-09-11T09:51:55.672 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00082222800000000004 2024-09-11T09:51:55.672 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.672 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.672 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:55.672 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.546851+0000", 2024-09-11T09:51:55.672 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.2075999999999999e-05 2024-09-11T09:51:55.672 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.672 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.672 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:55.672 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.546856+0000", 2024-09-11T09:51:55.672 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.4009999999999996e-06 2024-09-11T09:51:55.673 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.673 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.673 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:55.673 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.546897+0000", 2024-09-11T09:51:55.673 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.0113e-05 2024-09-11T09:51:55.673 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.673 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.673 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:55.673 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.546954+0000", 2024-09-11T09:51:55.673 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.7825000000000003e-05 2024-09-11T09:51:55.673 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.673 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.673 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 2", 2024-09-11T09:51:55.674 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.547055+0000", 2024-09-11T09:51:55.674 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000100269 2024-09-11T09:51:55.674 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.674 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.674 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:55.674 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.549675+0000", 2024-09-11T09:51:55.674 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0026206609999999998 2024-09-11T09:51:55.674 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.674 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.674 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:55.674 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.550858+0000", 2024-09-11T09:51:55.674 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0011825270000000001 2024-09-11T09:51:55.674 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.674 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.675 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:55.675 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.550894+0000", 2024-09-11T09:51:55.675 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.5896999999999999e-05 2024-09-11T09:51:55.675 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.675 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.675 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:55.675 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.550926+0000", 2024-09-11T09:51:55.675 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.2023999999999997e-05 2024-09-11T09:51:55.675 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:55.675 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:55.675 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:55.675 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.675 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.676 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.5002.0:91 2.3 2:c54742d8:::rbd_data.1250f5e25b80.0000000000000006:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-09-11T09:51:55.676 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:38.553743+0000", 2024-09-11T09:51:55.676 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 17.021724899999999, 2024-09-11T09:51:55.676 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0051633629999999998, 2024-09-11T09:51:55.676 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:55.676 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:55.676 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:55.676 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:55.676 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.5002", 2024-09-11T09:51:55.676 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/2204440692", 2024-09-11T09:51:55.676 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 91 2024-09-11T09:51:55.676 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.676 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:55.676 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.677 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:55.677 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.553743+0000", 2024-09-11T09:51:55.677 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:55.677 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.677 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.677 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:55.677 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.553743+0000", 2024-09-11T09:51:55.677 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:55.677 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.677 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.677 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:55.677 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.553745+0000", 2024-09-11T09:51:55.677 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.5120000000000001e-06 2024-09-11T09:51:55.678 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.678 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.678 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:55.678 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.554588+0000", 2024-09-11T09:51:55.678 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00084352299999999999 2024-09-11T09:51:55.678 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.678 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.678 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:55.678 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.554604+0000", 2024-09-11T09:51:55.678 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.643e-05 2024-09-11T09:51:55.678 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.678 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.678 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:55.678 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.554613+0000", 2024-09-11T09:51:55.679 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.5669999999999995e-06 2024-09-11T09:51:55.679 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.679 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.679 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:55.679 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.554662+0000", 2024-09-11T09:51:55.679 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.8510000000000001e-05 2024-09-11T09:51:55.679 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.679 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.679 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:55.679 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.554732+0000", 2024-09-11T09:51:55.679 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.0297999999999997e-05 2024-09-11T09:51:55.679 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.679 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.679 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 2", 2024-09-11T09:51:55.680 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.554870+0000", 2024-09-11T09:51:55.680 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00013856599999999999 2024-09-11T09:51:55.680 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.680 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.680 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:55.680 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.557875+0000", 2024-09-11T09:51:55.680 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0030042760000000002 2024-09-11T09:51:55.680 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.680 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.680 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:55.680 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.558835+0000", 2024-09-11T09:51:55.680 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00096057900000000003 2024-09-11T09:51:55.680 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.681 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.681 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:55.681 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.558874+0000", 2024-09-11T09:51:55.681 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.8992999999999997e-05 2024-09-11T09:51:55.681 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.681 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.681 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:55.681 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.558906+0000", 2024-09-11T09:51:55.681 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.2109e-05 2024-09-11T09:51:55.681 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:55.681 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:55.681 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:55.682 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.682 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.682 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.5002.0:93 2.3 2:c54742d8:::rbd_data.1250f5e25b80.0000000000000006:head [write 2228224~917504] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e61)", 2024-09-11T09:51:55.682 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:38.562067+0000", 2024-09-11T09:51:55.682 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 17.013401303999999, 2024-09-11T09:51:55.682 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0050551099999999998, 2024-09-11T09:51:55.682 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:55.682 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:55.682 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:55.682 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:51:55.682 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.5002", 2024-09-11T09:51:55.682 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/2204440692", 2024-09-11T09:51:55.682 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 93 2024-09-11T09:51:55.682 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.683 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:55.683 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.683 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:55.683 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.562067+0000", 2024-09-11T09:51:55.683 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:55.683 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.683 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.683 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:55.683 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.562067+0000", 2024-09-11T09:51:55.683 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:55.683 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.683 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.683 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:55.683 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.562068+0000", 2024-09-11T09:51:55.684 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.0669999999999999e-06 2024-09-11T09:51:55.684 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.684 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.684 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:55.684 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.563170+0000", 2024-09-11T09:51:55.684 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00110256 2024-09-11T09:51:55.684 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.684 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.684 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:55.684 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.563185+0000", 2024-09-11T09:51:55.684 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.4873e-05 2024-09-11T09:51:55.684 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.684 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.684 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:55.684 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.563193+0000", 2024-09-11T09:51:55.685 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.1319999999999994e-06 2024-09-11T09:51:55.685 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.685 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.685 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:55.685 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.563238+0000", 2024-09-11T09:51:55.685 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.4773e-05 2024-09-11T09:51:55.685 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.685 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.685 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:55.685 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.563290+0000", 2024-09-11T09:51:55.685 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.1690000000000001e-05 2024-09-11T09:51:55.685 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.685 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.685 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 2", 2024-09-11T09:51:55.685 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.563365+0000", 2024-09-11T09:51:55.686 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.5023000000000003e-05 2024-09-11T09:51:55.686 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.686 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.686 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:55.686 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.566552+0000", 2024-09-11T09:51:55.686 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0031869960000000001 2024-09-11T09:51:55.686 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.686 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.686 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:51:55.686 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.567048+0000", 2024-09-11T09:51:55.686 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00049605099999999996 2024-09-11T09:51:55.686 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.686 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.686 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:51:55.686 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.567095+0000", 2024-09-11T09:51:55.687 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.6823999999999998e-05 2024-09-11T09:51:55.687 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:55.687 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:55.687 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:55.687 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.567122+0000", 2024-09-11T09:51:55.687 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.7121e-05 2024-09-11T09:51:55.687 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:55.687 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:55.687 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:55.687 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:55.687 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:55.687 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-11T09:51:55.689 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-11T09:51:55.689 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:51:55.689+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:51:55.789 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-09-11T09:51:55.790+0000 7f7b6c7d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31904) UID: 0 2024-09-11T09:51:55.792 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-11T09:51:55.792 INFO:teuthology.orchestra.run.smithi096.stdout: "ops": [], 2024-09-11T09:51:55.793 INFO:teuthology.orchestra.run.smithi096.stdout: "num_ops": 0 2024-09-11T09:51:55.793 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-11T09:51:55.801 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-11T09:51:55.891 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:51:55.891+0000 7f3c96ec9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31909) UID: 0 2024-09-11T09:51:55.944 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-11T09:51:55.944 INFO:teuthology.orchestra.run.smithi096.stdout: "ops": [], 2024-09-11T09:51:55.944 INFO:teuthology.orchestra.run.smithi096.stdout: "complaint_time": 30, 2024-09-11T09:51:55.944 INFO:teuthology.orchestra.run.smithi096.stdout: "num_blocked_ops": 0 2024-09-11T09:51:55.945 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-11T09:51:55.953 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-11T09:51:55.992 INFO:tasks.ceph.osd.0.smithi017.stderr:2024-09-11T09:51:55.992+0000 7fb73c0f1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31665) UID: 0 2024-09-11T09:51:56.092 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-09-11T09:51:56.093+0000 7f7b6c7d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31904) UID: 0 2024-09-11T09:51:56.096 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-11T09:51:56.096 INFO:teuthology.orchestra.run.smithi096.stdout: "size": 20, 2024-09-11T09:51:56.096 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 600, 2024-09-11T09:51:56.096 INFO:teuthology.orchestra.run.smithi096.stdout: "ops": [ 2024-09-11T09:51:56.097 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:56.097 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "MOSDPGPush(3.3 67/66 [PushOp(3:c4c92e5a:::rbd_trash:head, version: 56'1, data_included: [], data_size: 0, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:c4c92e5a:::rbd_trash:head@56'1, size: 0, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 0, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-11T09:51:56.097 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:45.784642+0000", 2024-09-11T09:51:56.097 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 10.313311369999999, 2024-09-11T09:51:56.097 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00049292200000000002, 2024-09-11T09:51:56.097 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:51:56.097 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:51:56.097 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "started", 2024-09-11T09:51:56.097 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:51:56.097 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:56.098 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:51:56.098 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:45.784642+0000", 2024-09-11T09:51:56.098 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:56.098 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:56.098 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:56.098 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:51:56.098 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:45.784642+0000", 2024-09-11T09:51:56.098 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:51:56.098 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:56.098 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:56.098 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:51:56.098 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:45.784643+0000", 2024-09-11T09:51:56.098 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.2599999999999998e-07 2024-09-11T09:51:56.098 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:56.099 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:56.099 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:51:56.099 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:45.784652+0000", 2024-09-11T09:51:56.099 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.064e-06 2024-09-11T09:51:56.099 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:56.099 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:56.099 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:51:56.099 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:45.784664+0000", 2024-09-11T09:51:56.099 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1831000000000001e-05 2024-09-11T09:51:56.099 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:56.099 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:56.099 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:51:56.099 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:45.784676+0000", 2024-09-11T09:51:56.099 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1996e-05 2024-09-11T09:51:56.099 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:56.100 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:56.100 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:51:56.100 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:45.784744+0000", 2024-09-11T09:51:56.100 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.8310000000000002e-05 2024-09-11T09:51:56.100 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:56.100 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:56.100 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:51:56.100 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:45.784758+0000", 2024-09-11T09:51:56.100 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.3998999999999999e-05 2024-09-11T09:51:56.100 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:51:56.100 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:51:56.100 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:51:56.100 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:45.785135+0000", 2024-09-11T09:51:56.100 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000377296 2024-09-11T09:51:56.100 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:56.101 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:56.101 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:56.101 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:51:56.101 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:51:56.101 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-11T09:51:56.105 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-11T09:51:56.194 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:51:56.193+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:51:56.227 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:51:56.227 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [], 2024-09-11T09:51:56.227 INFO:teuthology.orchestra.run.smithi017.stdout: "num_ops": 0 2024-09-11T09:51:56.227 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:51:56.236 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-11T09:51:56.320 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:51:56.321+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:51:56.385 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:51:56.386 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [], 2024-09-11T09:51:56.386 INFO:teuthology.orchestra.run.smithi017.stdout: "complaint_time": 30, 2024-09-11T09:51:56.386 INFO:teuthology.orchestra.run.smithi017.stdout: "num_blocked_ops": 0 2024-09-11T09:51:56.386 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:51:56.394 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-11T09:51:56.435 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:51:56.435+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:51:56.496 INFO:tasks.ceph.osd.2.smithi017.stderr:2024-09-11T09:51:56.496+0000 7fd20b170640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31672) UID: 0 2024-09-11T09:51:56.543 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:51:56.543 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 20, 2024-09-11T09:51:56.543 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 600, 2024-09-11T09:51:56.543 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [ 2024-09-11T09:51:56.543 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.543 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4688.0:34 2.4 e53/32 2:2367385a:::rbd_data.1250f5e25b80.0000000000000002:head v 53'6, mlcod=53'4)", 2024-09-11T09:51:56.543 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.907415+0000", 2024-09-11T09:51:56.543 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 33.635529124000001, 2024-09-11T09:51:56.544 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.011285778999999999, 2024-09-11T09:51:56.544 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:56.544 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:56.544 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:56.544 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:56.544 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.544 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:56.544 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.907415+0000", 2024-09-11T09:51:56.544 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:56.544 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.544 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.544 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:56.544 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.907415+0000", 2024-09-11T09:51:56.544 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:56.545 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.545 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.545 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:56.545 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.907417+0000", 2024-09-11T09:51:56.545 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.2910000000000002e-06 2024-09-11T09:51:56.545 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.545 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.545 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:56.545 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.910477+0000", 2024-09-11T09:51:56.545 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0030597829999999999 2024-09-11T09:51:56.545 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.545 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.545 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:56.545 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.910501+0000", 2024-09-11T09:51:56.545 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.3332e-05 2024-09-11T09:51:56.546 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.546 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.546 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:56.546 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.910514+0000", 2024-09-11T09:51:56.546 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3957e-05 2024-09-11T09:51:56.546 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.546 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.546 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:56.546 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.910612+0000", 2024-09-11T09:51:56.546 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.7075000000000001e-05 2024-09-11T09:51:56.546 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.546 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.546 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:56.546 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.910640+0000", 2024-09-11T09:51:56.547 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.8768999999999998e-05 2024-09-11T09:51:56.547 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.547 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.547 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:56.547 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.918640+0000", 2024-09-11T09:51:56.547 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0079991899999999998 2024-09-11T09:51:56.547 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.547 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.547 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:56.547 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.918701+0000", 2024-09-11T09:51:56.547 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.1382000000000003e-05 2024-09-11T09:51:56.547 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:56.547 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:56.547 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:56.548 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.548 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.548 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4688.0:38 2.5 e53/32 2:abde2694:::rbd_data.1250f5e25b80.0000000000000004:head v 53'1, mlcod=0'0)", 2024-09-11T09:51:56.548 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.915432+0000", 2024-09-11T09:51:56.548 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 33.627512744000001, 2024-09-11T09:51:56.548 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.015849652999999998, 2024-09-11T09:51:56.548 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:56.548 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:56.548 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:56.548 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:56.548 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.548 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:56.549 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.915432+0000", 2024-09-11T09:51:56.549 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:56.549 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.549 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.549 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:56.549 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.915432+0000", 2024-09-11T09:51:56.549 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:56.549 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.549 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.549 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:56.549 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.915432+0000", 2024-09-11T09:51:56.549 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.91e-07 2024-09-11T09:51:56.549 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.550 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.550 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:56.550 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.924943+0000", 2024-09-11T09:51:56.550 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0095110630000000002 2024-09-11T09:51:56.550 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.550 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.550 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:56.550 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.924960+0000", 2024-09-11T09:51:56.550 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6915999999999999e-05 2024-09-11T09:51:56.550 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.550 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.550 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:56.550 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.924968+0000", 2024-09-11T09:51:56.550 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.6999999999999997e-06 2024-09-11T09:51:56.551 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.551 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.551 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:56.551 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.925025+0000", 2024-09-11T09:51:56.551 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.6949e-05 2024-09-11T09:51:56.551 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.551 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.551 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:56.551 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.925043+0000", 2024-09-11T09:51:56.551 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7784000000000001e-05 2024-09-11T09:51:56.551 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.551 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.551 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:56.552 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.931237+0000", 2024-09-11T09:51:56.552 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0061935389999999996 2024-09-11T09:51:56.552 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.552 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.552 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:56.552 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.931281+0000", 2024-09-11T09:51:56.552 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.4511000000000001e-05 2024-09-11T09:51:56.552 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:56.552 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:56.552 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:56.552 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.552 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.552 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:40 2.7 2:fe41411c:::rbd_object_map.1250f5e25b80:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:56.553 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.919652+0000", 2024-09-11T09:51:56.553 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 33.623292077000002, 2024-09-11T09:51:56.553 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.012187073, 2024-09-11T09:51:56.553 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:56.553 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:56.553 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:56.553 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:56.553 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:51:56.553 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:51:56.553 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 40 2024-09-11T09:51:56.553 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.553 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:56.554 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.554 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:56.554 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919652+0000", 2024-09-11T09:51:56.554 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:56.554 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.554 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.554 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:56.554 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919652+0000", 2024-09-11T09:51:56.554 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:56.554 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.554 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.554 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:56.554 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919654+0000", 2024-09-11T09:51:56.554 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.88e-06 2024-09-11T09:51:56.555 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.555 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.555 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:56.555 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919669+0000", 2024-09-11T09:51:56.555 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.5271e-05 2024-09-11T09:51:56.555 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.555 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.555 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:56.555 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919687+0000", 2024-09-11T09:51:56.555 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7448999999999998e-05 2024-09-11T09:51:56.555 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.555 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.555 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:56.555 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919695+0000", 2024-09-11T09:51:56.556 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.1149999999999994e-06 2024-09-11T09:51:56.556 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.556 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.556 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:56.556 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919757+0000", 2024-09-11T09:51:56.556 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.2348999999999995e-05 2024-09-11T09:51:56.556 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.556 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.556 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:56.556 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919875+0000", 2024-09-11T09:51:56.556 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00011750499999999999 2024-09-11T09:51:56.556 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.556 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.556 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 5", 2024-09-11T09:51:56.557 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.920201+0000", 2024-09-11T09:51:56.557 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00032628099999999999 2024-09-11T09:51:56.557 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.557 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.557 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:56.557 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.922089+0000", 2024-09-11T09:51:56.557 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.001887978 2024-09-11T09:51:56.557 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.557 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.557 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:56.557 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.931752+0000", 2024-09-11T09:51:56.557 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0096626309999999997 2024-09-11T09:51:56.558 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.558 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.558 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:56.558 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.931794+0000", 2024-09-11T09:51:56.558 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.2849999999999998e-05 2024-09-11T09:51:56.558 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.558 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.558 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:56.558 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.931839+0000", 2024-09-11T09:51:56.558 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.4764e-05 2024-09-11T09:51:56.558 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:56.558 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:56.558 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:56.558 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.558 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.559 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:41 2.7 2:fe41411c:::rbd_object_map.1250f5e25b80:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:56.559 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.919716+0000", 2024-09-11T09:51:56.559 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 33.623228232999999, 2024-09-11T09:51:56.559 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.015641605999999999, 2024-09-11T09:51:56.559 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:56.559 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:56.559 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:56.559 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:56.559 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:51:56.559 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:51:56.559 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 41 2024-09-11T09:51:56.559 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.559 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:56.560 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.560 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:56.560 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919716+0000", 2024-09-11T09:51:56.560 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:56.560 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.560 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.560 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:56.560 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919716+0000", 2024-09-11T09:51:56.560 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:56.560 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.560 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.560 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:56.560 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919716+0000", 2024-09-11T09:51:56.561 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.9999999999999998e-07 2024-09-11T09:51:56.561 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.561 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.561 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:56.561 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919720+0000", 2024-09-11T09:51:56.561 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.6440000000000002e-06 2024-09-11T09:51:56.561 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.561 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.561 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:56.561 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919729+0000", 2024-09-11T09:51:56.561 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.1759999999999999e-06 2024-09-11T09:51:56.561 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.561 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.562 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:56.562 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919736+0000", 2024-09-11T09:51:56.562 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.5390000000000004e-06 2024-09-11T09:51:56.562 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.562 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.562 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:56.562 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.920669+0000", 2024-09-11T09:51:56.562 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00093275599999999997 2024-09-11T09:51:56.562 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.562 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.562 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:56.562 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.920797+0000", 2024-09-11T09:51:56.562 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00012893400000000001 2024-09-11T09:51:56.563 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.563 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.563 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 5", 2024-09-11T09:51:56.563 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.921269+0000", 2024-09-11T09:51:56.563 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00047128600000000001 2024-09-11T09:51:56.563 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.563 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.563 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:56.563 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.923198+0000", 2024-09-11T09:51:56.563 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0019289369999999999 2024-09-11T09:51:56.563 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.563 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.563 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:56.564 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.935300+0000", 2024-09-11T09:51:56.564 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.012102392 2024-09-11T09:51:56.564 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.564 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.564 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:56.564 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.935332+0000", 2024-09-11T09:51:56.564 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.2292000000000002e-05 2024-09-11T09:51:56.564 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.564 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.564 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:56.564 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.935358+0000", 2024-09-11T09:51:56.564 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.525e-05 2024-09-11T09:51:56.564 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:56.565 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:56.565 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:56.565 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.565 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.565 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:48 2.7 2:fe41411c:::rbd_object_map.1250f5e25b80:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:56.565 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.938799+0000", 2024-09-11T09:51:56.565 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 33.604145123999999, 2024-09-11T09:51:56.565 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0041642340000000002, 2024-09-11T09:51:56.565 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:56.565 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:56.565 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:56.565 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:56.565 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:51:56.566 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:51:56.566 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 48 2024-09-11T09:51:56.566 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.566 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:56.566 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.567 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:56.567 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.938799+0000", 2024-09-11T09:51:56.567 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:56.567 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.568 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.568 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:56.568 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.938799+0000", 2024-09-11T09:51:56.568 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:56.568 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.568 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.568 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:56.568 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.938800+0000", 2024-09-11T09:51:56.568 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.71e-07 2024-09-11T09:51:56.568 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.568 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.568 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:56.568 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.938805+0000", 2024-09-11T09:51:56.569 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.3140000000000003e-06 2024-09-11T09:51:56.569 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.569 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.569 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:56.569 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.938816+0000", 2024-09-11T09:51:56.569 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0231000000000001e-05 2024-09-11T09:51:56.569 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.569 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.569 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:56.569 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.938822+0000", 2024-09-11T09:51:56.569 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.9550000000000003e-06 2024-09-11T09:51:56.569 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.569 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.569 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:56.570 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.938868+0000", 2024-09-11T09:51:56.570 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.5562999999999997e-05 2024-09-11T09:51:56.570 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.570 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.570 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:56.570 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.938918+0000", 2024-09-11T09:51:56.570 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.0317e-05 2024-09-11T09:51:56.570 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.570 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.570 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 5", 2024-09-11T09:51:56.570 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.939114+0000", 2024-09-11T09:51:56.570 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00019538300000000001 2024-09-11T09:51:56.570 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.570 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.571 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:56.571 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.939663+0000", 2024-09-11T09:51:56.571 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00054932199999999998 2024-09-11T09:51:56.571 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.571 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.571 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:56.571 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.942910+0000", 2024-09-11T09:51:56.571 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0032473839999999999 2024-09-11T09:51:56.571 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.571 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.571 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:56.571 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.942940+0000", 2024-09-11T09:51:56.571 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.9306999999999999e-05 2024-09-11T09:51:56.571 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.571 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.572 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:56.572 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.942963+0000", 2024-09-11T09:51:56.572 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.3587e-05 2024-09-11T09:51:56.572 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:56.572 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:56.572 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:56.572 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.572 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.572 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4688.0:49 2.5 e53/32 2:b6dd9fc9:::rbd_data.1250f5e25b80.0000000000000009:head v 53'2, mlcod=0'0)", 2024-09-11T09:51:56.572 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.944712+0000", 2024-09-11T09:51:56.572 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 33.598232285000002, 2024-09-11T09:51:56.572 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.016185069999999999, 2024-09-11T09:51:56.572 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:56.572 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:56.573 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:56.573 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:56.573 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.573 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:56.573 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.944712+0000", 2024-09-11T09:51:56.573 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:56.573 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.573 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.573 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:56.573 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.944712+0000", 2024-09-11T09:51:56.573 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:56.573 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.573 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.574 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:56.574 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.944713+0000", 2024-09-11T09:51:56.574 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0139999999999999e-06 2024-09-11T09:51:56.574 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.574 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.574 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:56.574 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.955518+0000", 2024-09-11T09:51:56.574 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.01080542 2024-09-11T09:51:56.574 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.574 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.574 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:56.574 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.955536+0000", 2024-09-11T09:51:56.574 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7818000000000001e-05 2024-09-11T09:51:56.574 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.574 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.575 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:56.575 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.955544+0000", 2024-09-11T09:51:56.575 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.1699999999999997e-06 2024-09-11T09:51:56.575 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.575 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.575 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:56.575 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.955607+0000", 2024-09-11T09:51:56.575 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.2451000000000002e-05 2024-09-11T09:51:56.575 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.575 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.575 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:56.575 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.955632+0000", 2024-09-11T09:51:56.575 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.5335000000000001e-05 2024-09-11T09:51:56.575 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.575 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.575 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:56.576 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.960826+0000", 2024-09-11T09:51:56.576 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0051938560000000002 2024-09-11T09:51:56.576 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.576 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.576 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:56.576 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.960897+0000", 2024-09-11T09:51:56.576 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.1006000000000004e-05 2024-09-11T09:51:56.576 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:56.576 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:56.576 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:56.576 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.576 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.576 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:53 2.7 2:fe41411c:::rbd_object_map.1250f5e25b80:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:56.576 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.951291+0000", 2024-09-11T09:51:56.577 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 33.591653295, 2024-09-11T09:51:56.577 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0044964999999999996, 2024-09-11T09:51:56.577 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:56.577 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:56.577 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:56.577 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:56.577 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:51:56.577 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:51:56.577 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 53 2024-09-11T09:51:56.577 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.577 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:56.577 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.577 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:56.577 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951291+0000", 2024-09-11T09:51:56.577 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:56.578 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.578 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.578 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:56.578 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951291+0000", 2024-09-11T09:51:56.578 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:56.578 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.578 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.578 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:56.578 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951293+0000", 2024-09-11T09:51:56.578 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7889999999999999e-06 2024-09-11T09:51:56.578 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.578 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.579 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:56.579 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951302+0000", 2024-09-11T09:51:56.579 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.9819999999999997e-06 2024-09-11T09:51:56.579 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.579 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.579 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:56.579 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951324+0000", 2024-09-11T09:51:56.579 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.2141e-05 2024-09-11T09:51:56.579 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.579 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.579 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:56.579 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951338+0000", 2024-09-11T09:51:56.579 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3750999999999999e-05 2024-09-11T09:51:56.579 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.580 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.580 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:56.580 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951409+0000", 2024-09-11T09:51:56.580 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.1658999999999999e-05 2024-09-11T09:51:56.580 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.580 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.580 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:56.580 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951517+0000", 2024-09-11T09:51:56.580 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00010746799999999999 2024-09-11T09:51:56.580 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.580 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.580 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 5", 2024-09-11T09:51:56.581 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951942+0000", 2024-09-11T09:51:56.581 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00042524300000000001 2024-09-11T09:51:56.581 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.581 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.581 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:56.581 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.952872+0000", 2024-09-11T09:51:56.581 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00092976300000000001 2024-09-11T09:51:56.581 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.581 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.581 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:56.581 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.955682+0000", 2024-09-11T09:51:56.581 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0028099459999999998 2024-09-11T09:51:56.581 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.582 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.582 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:56.582 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.955746+0000", 2024-09-11T09:51:56.582 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.4084999999999994e-05 2024-09-11T09:51:56.582 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.582 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.582 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:56.582 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.955787+0000", 2024-09-11T09:51:56.582 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.1672999999999999e-05 2024-09-11T09:51:56.582 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:56.582 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:56.582 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:56.582 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.583 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.583 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4688.0:52 2.4 e53/32 2:26d5c2f0:::rbd_data.1250f5e25b80.000000000000000b:head v 53'8, mlcod=53'6)", 2024-09-11T09:51:56.583 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.963677+0000", 2024-09-11T09:51:56.583 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 33.579267739999999, 2024-09-11T09:51:56.583 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0089437969999999999, 2024-09-11T09:51:56.583 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:56.583 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:56.583 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:56.583 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:56.583 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.583 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:56.583 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.963677+0000", 2024-09-11T09:51:56.583 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:56.584 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.584 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.584 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:56.584 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.963677+0000", 2024-09-11T09:51:56.584 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:56.584 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.584 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.584 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:56.584 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.963677+0000", 2024-09-11T09:51:56.584 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.7199999999999998e-07 2024-09-11T09:51:56.584 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.584 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.584 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:56.584 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.965506+0000", 2024-09-11T09:51:56.585 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.001829064 2024-09-11T09:51:56.585 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.585 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.585 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:56.585 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.965538+0000", 2024-09-11T09:51:56.585 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.1306999999999997e-05 2024-09-11T09:51:56.585 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.585 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.585 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:56.585 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.965553+0000", 2024-09-11T09:51:56.585 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.4924e-05 2024-09-11T09:51:56.585 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.585 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.585 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:56.586 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.965663+0000", 2024-09-11T09:51:56.586 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000110315 2024-09-11T09:51:56.586 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.586 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.586 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:56.586 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.965692+0000", 2024-09-11T09:51:56.586 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.934e-05 2024-09-11T09:51:56.586 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.586 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.586 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:56.586 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.972537+0000", 2024-09-11T09:51:56.586 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0068444969999999997 2024-09-11T09:51:56.586 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.586 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.586 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:56.587 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.972620+0000", 2024-09-11T09:51:56.587 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.3578000000000005e-05 2024-09-11T09:51:56.587 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:56.587 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:56.587 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:56.587 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.587 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.587 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4688.0:54 2.4 e53/32 2:239e915b:::rbd_data.1250f5e25b80.000000000000000c:head v 53'9, mlcod=53'7)", 2024-09-11T09:51:56.587 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.987028+0000", 2024-09-11T09:51:56.587 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 33.555915822999999, 2024-09-11T09:51:56.587 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0094155650000000007, 2024-09-11T09:51:56.587 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:56.587 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:56.587 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:56.587 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:56.588 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.588 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:56.588 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.987028+0000", 2024-09-11T09:51:56.588 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:56.588 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.588 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.588 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:56.588 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.987028+0000", 2024-09-11T09:51:56.588 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:56.588 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.588 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.588 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:56.588 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.987030+0000", 2024-09-11T09:51:56.588 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6130000000000001e-06 2024-09-11T09:51:56.589 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.589 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.589 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:56.589 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.988575+0000", 2024-09-11T09:51:56.589 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.001545193 2024-09-11T09:51:56.589 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.589 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.589 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:56.589 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.988604+0000", 2024-09-11T09:51:56.589 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.9173e-05 2024-09-11T09:51:56.589 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.589 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.589 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:56.589 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.988620+0000", 2024-09-11T09:51:56.590 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.5517999999999999e-05 2024-09-11T09:51:56.590 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.590 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.590 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:56.590 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.988704+0000", 2024-09-11T09:51:56.590 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.3588e-05 2024-09-11T09:51:56.590 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.590 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.590 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:56.590 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.988735+0000", 2024-09-11T09:51:56.590 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.1643000000000002e-05 2024-09-11T09:51:56.590 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.590 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.591 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:56.591 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.996383+0000", 2024-09-11T09:51:56.591 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0076478099999999997 2024-09-11T09:51:56.591 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.591 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.591 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:56.591 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.996444+0000", 2024-09-11T09:51:56.591 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.1026999999999997e-05 2024-09-11T09:51:56.591 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:56.591 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:56.591 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:56.591 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.591 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.591 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4688.0:56 2.4 e53/32 2:2167012f:::rbd_data.1250f5e25b80.000000000000000d:head v 53'10, mlcod=53'7)", 2024-09-11T09:51:56.591 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.993894+0000", 2024-09-11T09:51:56.592 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 33.549050281, 2024-09-11T09:51:56.592 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0099268559999999995, 2024-09-11T09:51:56.592 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:56.592 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:56.592 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:56.592 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:56.592 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.592 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:56.592 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.993894+0000", 2024-09-11T09:51:56.592 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:56.592 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.592 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.592 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:56.592 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.993894+0000", 2024-09-11T09:51:56.593 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:56.593 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.593 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.593 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:56.593 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.993896+0000", 2024-09-11T09:51:56.593 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6249999999999999e-06 2024-09-11T09:51:56.593 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.593 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.593 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:56.593 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.997606+0000", 2024-09-11T09:51:56.593 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.003710737 2024-09-11T09:51:56.593 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.593 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.593 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:56.594 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.997628+0000", 2024-09-11T09:51:56.594 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.1971e-05 2024-09-11T09:51:56.594 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.594 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.594 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:56.594 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.997640+0000", 2024-09-11T09:51:56.594 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1481000000000001e-05 2024-09-11T09:51:56.594 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.594 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.594 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:56.594 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.997712+0000", 2024-09-11T09:51:56.594 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.2255000000000005e-05 2024-09-11T09:51:56.594 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.594 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.595 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:56.595 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.997738+0000", 2024-09-11T09:51:56.595 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.6050999999999998e-05 2024-09-11T09:51:56.595 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.595 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.595 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:56.595 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.003779+0000", 2024-09-11T09:51:56.595 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0060410960000000001 2024-09-11T09:51:56.595 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.595 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.595 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:56.595 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.003821+0000", 2024-09-11T09:51:56.595 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.1640000000000001e-05 2024-09-11T09:51:56.595 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:56.595 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:56.596 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:56.596 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.596 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.596 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "pg_update_log_missing(2.4 epoch 53/32 rep_tid 16 entries 53'11 (0'0) error 2:2906e6a7:::rbd_header.1250f5e25b80:head by client.4688.0:59 0.000000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 53'7)", 2024-09-11T09:51:56.596 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.998956+0000", 2024-09-11T09:51:56.596 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 33.543988112000001, 2024-09-11T09:51:56.596 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.004934127, 2024-09-11T09:51:56.596 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:56.596 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:56.596 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "reached pg", 2024-09-11T09:51:56.596 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:56.596 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.596 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:56.597 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.998956+0000", 2024-09-11T09:51:56.597 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:56.597 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.597 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.597 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:56.597 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.998956+0000", 2024-09-11T09:51:56.597 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:56.597 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.597 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.597 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:56.597 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.998957+0000", 2024-09-11T09:51:56.597 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.139e-06 2024-09-11T09:51:56.597 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.597 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.598 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:56.598 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.998985+0000", 2024-09-11T09:51:56.598 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.7912e-05 2024-09-11T09:51:56.598 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.598 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.598 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:56.598 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.999011+0000", 2024-09-11T09:51:56.598 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.6061e-05 2024-09-11T09:51:56.598 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.598 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.598 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:56.598 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.999028+0000", 2024-09-11T09:51:56.598 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7065999999999999e-05 2024-09-11T09:51:56.598 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.598 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.599 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:56.599 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.001919+0000", 2024-09-11T09:51:56.599 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.002891167 2024-09-11T09:51:56.599 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.599 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.599 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:56.599 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.003890+0000", 2024-09-11T09:51:56.599 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0019707819999999999 2024-09-11T09:51:56.599 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:56.599 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:56.599 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:56.599 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.599 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.599 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4694.0:34 2.7 2:f80bfc0c:ns::rbd_data.1256a0c02879.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:56.600 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.085183+0000", 2024-09-11T09:51:56.600 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 33.457761599000001, 2024-09-11T09:51:56.600 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.020908599, 2024-09-11T09:51:56.600 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:56.600 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:56.600 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:56.600 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:56.600 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4694", 2024-09-11T09:51:56.600 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:51:56.600 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 34 2024-09-11T09:51:56.600 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.600 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:56.600 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.600 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:56.601 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.085183+0000", 2024-09-11T09:51:56.601 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:56.601 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.601 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.601 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:56.601 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.085183+0000", 2024-09-11T09:51:56.601 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:56.601 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.601 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.601 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:56.601 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.085183+0000", 2024-09-11T09:51:56.601 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.4300000000000002e-07 2024-09-11T09:51:56.601 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.601 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.601 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:56.602 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094337+0000", 2024-09-11T09:51:56.602 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0091532769999999996 2024-09-11T09:51:56.602 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.602 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.602 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:56.602 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094362+0000", 2024-09-11T09:51:56.602 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.5287999999999999e-05 2024-09-11T09:51:56.602 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.602 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.602 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:56.602 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094375+0000", 2024-09-11T09:51:56.602 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2446000000000001e-05 2024-09-11T09:51:56.602 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.602 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.603 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:56.603 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094455+0000", 2024-09-11T09:51:56.603 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.0092999999999994e-05 2024-09-11T09:51:56.603 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.603 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.603 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:56.603 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094618+0000", 2024-09-11T09:51:56.603 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000163747 2024-09-11T09:51:56.603 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.603 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.603 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 5", 2024-09-11T09:51:56.603 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094783+0000", 2024-09-11T09:51:56.603 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000165038 2024-09-11T09:51:56.603 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.603 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.604 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:56.604 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.105853+0000", 2024-09-11T09:51:56.604 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.011069294 2024-09-11T09:51:56.604 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.604 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.604 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:56.604 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.106004+0000", 2024-09-11T09:51:56.604 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00015123900000000001 2024-09-11T09:51:56.604 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.604 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.604 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:56.604 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.106062+0000", 2024-09-11T09:51:56.604 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.8205000000000003e-05 2024-09-11T09:51:56.604 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.605 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.605 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:56.605 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.106091+0000", 2024-09-11T09:51:56.605 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.9128999999999998e-05 2024-09-11T09:51:56.605 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:56.605 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:56.605 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:56.605 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.605 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.605 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4694.0:35 2.7 2:e8766325:ns::rbd_object_map.1256a0c02879:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:56.605 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.094405+0000", 2024-09-11T09:51:56.605 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 33.448538962999997, 2024-09-11T09:51:56.605 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.012623564, 2024-09-11T09:51:56.605 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:56.606 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:56.606 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:56.606 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:56.606 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4694", 2024-09-11T09:51:56.606 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:51:56.606 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 35 2024-09-11T09:51:56.606 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.606 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:56.606 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.606 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:56.606 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094405+0000", 2024-09-11T09:51:56.606 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:56.606 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.606 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.607 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:56.607 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094405+0000", 2024-09-11T09:51:56.607 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:56.607 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.607 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.607 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:56.607 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094407+0000", 2024-09-11T09:51:56.607 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.739e-06 2024-09-11T09:51:56.607 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.607 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.607 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:56.607 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094411+0000", 2024-09-11T09:51:56.607 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.3340000000000003e-06 2024-09-11T09:51:56.607 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.607 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.608 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:56.608 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094419+0000", 2024-09-11T09:51:56.608 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.8909999999999995e-06 2024-09-11T09:51:56.608 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.608 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.608 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:56.608 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094424+0000", 2024-09-11T09:51:56.608 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.7639999999999996e-06 2024-09-11T09:51:56.608 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.608 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.608 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:56.608 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.100209+0000", 2024-09-11T09:51:56.608 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0057850489999999996 2024-09-11T09:51:56.608 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.609 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.609 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:56.609 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.100341+0000", 2024-09-11T09:51:56.609 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000131438 2024-09-11T09:51:56.609 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.609 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.609 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 5", 2024-09-11T09:51:56.609 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.100805+0000", 2024-09-11T09:51:56.609 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00046471499999999998 2024-09-11T09:51:56.609 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.609 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.609 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:56.609 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.106148+0000", 2024-09-11T09:51:56.609 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0053426460000000004 2024-09-11T09:51:56.610 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.610 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.610 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:56.610 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.106929+0000", 2024-09-11T09:51:56.610 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00078161300000000001 2024-09-11T09:51:56.610 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.610 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.610 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:56.610 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.106985+0000", 2024-09-11T09:51:56.610 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.5223999999999998e-05 2024-09-11T09:51:56.610 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.610 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.610 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:56.610 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.107029+0000", 2024-09-11T09:51:56.610 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.4150999999999998e-05 2024-09-11T09:51:56.611 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:56.611 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:56.611 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:56.611 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.611 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.611 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4694.0:36 2.7 2:e8766325:ns::rbd_object_map.1256a0c02879:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:56.611 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.094452+0000", 2024-09-11T09:51:56.611 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 33.448491953000001, 2024-09-11T09:51:56.611 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.012680383, 2024-09-11T09:51:56.611 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:56.611 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:56.611 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:56.611 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:56.611 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4694", 2024-09-11T09:51:56.611 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:51:56.612 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 36 2024-09-11T09:51:56.612 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.612 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:56.612 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.612 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:56.612 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094452+0000", 2024-09-11T09:51:56.612 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:56.612 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.612 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.612 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:56.612 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094452+0000", 2024-09-11T09:51:56.612 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:56.612 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.613 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.613 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:56.613 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094453+0000", 2024-09-11T09:51:56.613 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.1800000000000002e-07 2024-09-11T09:51:56.613 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.613 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.613 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:56.613 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094455+0000", 2024-09-11T09:51:56.613 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.7020000000000001e-06 2024-09-11T09:51:56.613 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.613 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.613 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:56.613 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094463+0000", 2024-09-11T09:51:56.614 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.1999999999999997e-06 2024-09-11T09:51:56.614 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.614 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.614 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:56.614 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094467+0000", 2024-09-11T09:51:56.614 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.729e-06 2024-09-11T09:51:56.614 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.614 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.614 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:56.614 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.101269+0000", 2024-09-11T09:51:56.614 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0068018799999999997 2024-09-11T09:51:56.614 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.615 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.615 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:56.615 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.101353+0000", 2024-09-11T09:51:56.615 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.3827999999999998e-05 2024-09-11T09:51:56.615 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.615 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.615 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 5", 2024-09-11T09:51:56.615 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.101588+0000", 2024-09-11T09:51:56.615 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00023467099999999999 2024-09-11T09:51:56.615 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.615 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.615 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:56.615 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.106182+0000", 2024-09-11T09:51:56.615 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0045939489999999999 2024-09-11T09:51:56.616 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.616 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.616 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:56.616 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.107041+0000", 2024-09-11T09:51:56.616 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00085965800000000002 2024-09-11T09:51:56.616 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.616 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.616 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:56.616 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.107084+0000", 2024-09-11T09:51:56.616 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.261e-05 2024-09-11T09:51:56.616 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.616 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.616 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:56.617 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.107133+0000", 2024-09-11T09:51:56.617 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.8838000000000002e-05 2024-09-11T09:51:56.617 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:56.617 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:56.617 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:56.617 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.617 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.617 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4694.0:37 2.7 2:e8766325:ns::rbd_object_map.1256a0c02879:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:56.617 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.094473+0000", 2024-09-11T09:51:56.617 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 33.448470802000003, 2024-09-11T09:51:56.617 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.012778484999999999, 2024-09-11T09:51:56.617 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:56.617 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:56.618 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:56.618 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:56.618 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4694", 2024-09-11T09:51:56.618 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:51:56.618 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 37 2024-09-11T09:51:56.618 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.618 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:56.618 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.618 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:56.618 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094473+0000", 2024-09-11T09:51:56.618 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:56.618 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.618 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.619 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:56.619 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094473+0000", 2024-09-11T09:51:56.619 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:56.619 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.619 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.619 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:56.619 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094474+0000", 2024-09-11T09:51:56.619 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.9699999999999996e-07 2024-09-11T09:51:56.619 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.619 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.619 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:56.619 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094477+0000", 2024-09-11T09:51:56.619 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.151e-06 2024-09-11T09:51:56.619 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.619 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.620 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:56.620 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094483+0000", 2024-09-11T09:51:56.620 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.6330000000000001e-06 2024-09-11T09:51:56.620 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.620 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.620 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:56.620 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094487+0000", 2024-09-11T09:51:56.620 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.5399999999999997e-06 2024-09-11T09:51:56.620 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.620 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.620 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:56.620 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.101866+0000", 2024-09-11T09:51:56.620 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.007378768 2024-09-11T09:51:56.621 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.621 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.621 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:56.621 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.101960+0000", 2024-09-11T09:51:56.621 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.3861000000000001e-05 2024-09-11T09:51:56.621 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.621 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.621 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 5", 2024-09-11T09:51:56.621 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.102197+0000", 2024-09-11T09:51:56.621 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00023723200000000001 2024-09-11T09:51:56.621 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.621 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.621 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:56.622 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.106222+0000", 2024-09-11T09:51:56.622 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0040249689999999998 2024-09-11T09:51:56.622 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.622 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.622 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:56.622 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.107141+0000", 2024-09-11T09:51:56.622 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00091913900000000002 2024-09-11T09:51:56.622 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.622 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.622 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:56.622 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.107198+0000", 2024-09-11T09:51:56.622 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.6564999999999998e-05 2024-09-11T09:51:56.622 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.622 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.623 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:56.623 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.107252+0000", 2024-09-11T09:51:56.623 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.4030000000000003e-05 2024-09-11T09:51:56.623 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:56.623 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:56.623 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:56.623 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.623 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.623 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4694.0:33 2.4 e53/32 2:3bfc818c:ns::rbd_data.1256a0c02879.0000000000000001:head v 53'14, mlcod=53'12)", 2024-09-11T09:51:56.623 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.096496+0000", 2024-09-11T09:51:56.623 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 33.446448228999998, 2024-09-11T09:51:56.623 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0099139310000000008, 2024-09-11T09:51:56.623 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:56.623 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:56.624 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:56.624 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:56.624 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.624 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:56.624 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.096496+0000", 2024-09-11T09:51:56.624 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:56.624 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.624 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.624 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:56.624 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.096496+0000", 2024-09-11T09:51:56.624 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:56.624 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.624 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.624 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:56.624 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.096498+0000", 2024-09-11T09:51:56.625 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6309999999999999e-06 2024-09-11T09:51:56.625 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.625 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.625 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:56.625 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.098087+0000", 2024-09-11T09:51:56.625 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0015889019999999999 2024-09-11T09:51:56.625 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.625 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.625 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:56.625 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.098112+0000", 2024-09-11T09:51:56.625 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.5369000000000001e-05 2024-09-11T09:51:56.625 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.625 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.625 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:56.625 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.098127+0000", 2024-09-11T09:51:56.626 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.5218999999999999e-05 2024-09-11T09:51:56.626 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.626 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.626 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:56.626 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.098209+0000", 2024-09-11T09:51:56.626 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.2120999999999995e-05 2024-09-11T09:51:56.626 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.626 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.626 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:56.626 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.098237+0000", 2024-09-11T09:51:56.626 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.7657999999999998e-05 2024-09-11T09:51:56.626 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.626 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.626 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:56.626 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.106376+0000", 2024-09-11T09:51:56.627 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0081386749999999997 2024-09-11T09:51:56.627 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.627 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.627 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:56.627 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.106410+0000", 2024-09-11T09:51:56.627 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.4356000000000001e-05 2024-09-11T09:51:56.627 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:56.627 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:56.627 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:56.627 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.627 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.627 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4694.0:49 2.7 2:e8766325:ns::rbd_object_map.1256a0c02879:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:56.627 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.134806+0000", 2024-09-11T09:51:56.627 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 33.40813816, 2024-09-11T09:51:56.627 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0061629049999999998, 2024-09-11T09:51:56.628 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:56.628 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:56.628 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:56.628 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:56.628 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4694", 2024-09-11T09:51:56.628 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:51:56.628 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 49 2024-09-11T09:51:56.628 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.628 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:56.628 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.628 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:56.628 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.134806+0000", 2024-09-11T09:51:56.628 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:56.628 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.628 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.629 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:56.629 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.134806+0000", 2024-09-11T09:51:56.629 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:56.629 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.629 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.629 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:56.629 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.134808+0000", 2024-09-11T09:51:56.629 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.9870000000000002e-06 2024-09-11T09:51:56.629 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.629 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.629 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:56.629 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.134818+0000", 2024-09-11T09:51:56.629 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.5599999999999999e-06 2024-09-11T09:51:56.629 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.629 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.630 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:56.630 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.134835+0000", 2024-09-11T09:51:56.630 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7399000000000001e-05 2024-09-11T09:51:56.630 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.630 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.630 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:56.630 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.134845+0000", 2024-09-11T09:51:56.630 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0285000000000001e-05 2024-09-11T09:51:56.630 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.630 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.630 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:56.630 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.134911+0000", 2024-09-11T09:51:56.630 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.5654999999999994e-05 2024-09-11T09:51:56.630 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.630 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.631 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:56.631 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.135023+0000", 2024-09-11T09:51:56.631 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00011186799999999999 2024-09-11T09:51:56.631 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.631 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.631 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 5", 2024-09-11T09:51:56.631 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.135290+0000", 2024-09-11T09:51:56.631 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00026725300000000001 2024-09-11T09:51:56.631 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.631 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.631 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:56.631 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.136506+0000", 2024-09-11T09:51:56.631 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0012161139999999999 2024-09-11T09:51:56.631 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.631 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.632 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:56.632 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.140884+0000", 2024-09-11T09:51:56.632 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0043781990000000002 2024-09-11T09:51:56.632 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.632 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.632 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:56.632 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.140923+0000", 2024-09-11T09:51:56.632 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.8813999999999998e-05 2024-09-11T09:51:56.632 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.632 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.632 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:56.632 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.140969+0000", 2024-09-11T09:51:56.632 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.5770999999999999e-05 2024-09-11T09:51:56.633 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:56.633 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:56.633 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:56.633 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.633 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.633 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4694.0:42 2.5 e53/32 2:a4fc00ec:ns::rbd_data.1256a0c02879.0000000000000004:head v 53'3, mlcod=53'1)", 2024-09-11T09:51:56.633 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.136646+0000", 2024-09-11T09:51:56.633 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 33.406297854000002, 2024-09-11T09:51:56.633 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.012545501000000001, 2024-09-11T09:51:56.633 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:56.633 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:56.633 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:56.633 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:56.633 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.633 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:56.634 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.136646+0000", 2024-09-11T09:51:56.634 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:56.634 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.634 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.634 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:56.634 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.136646+0000", 2024-09-11T09:51:56.634 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:56.634 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.634 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.634 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:56.634 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.136647+0000", 2024-09-11T09:51:56.634 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.2900000000000002e-07 2024-09-11T09:51:56.634 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.634 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.634 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:56.635 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.143620+0000", 2024-09-11T09:51:56.635 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0069727440000000003 2024-09-11T09:51:56.635 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.635 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.635 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:56.635 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.143642+0000", 2024-09-11T09:51:56.635 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.1846000000000001e-05 2024-09-11T09:51:56.635 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.635 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.635 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:56.635 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.143654+0000", 2024-09-11T09:51:56.635 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2081e-05 2024-09-11T09:51:56.635 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.635 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.635 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:56.636 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.143717+0000", 2024-09-11T09:51:56.636 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.3443999999999998e-05 2024-09-11T09:51:56.636 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.636 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.636 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:56.636 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.143740+0000", 2024-09-11T09:51:56.636 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.2626999999999999e-05 2024-09-11T09:51:56.636 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.636 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.636 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:56.636 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.149142+0000", 2024-09-11T09:51:56.636 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0054017630000000004 2024-09-11T09:51:56.636 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.636 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.636 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:56.637 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.149192+0000", 2024-09-11T09:51:56.637 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.0067e-05 2024-09-11T09:51:56.637 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:56.637 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:56.637 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:56.637 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.637 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.637 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4694.0:53 2.7 2:e8766325:ns::rbd_object_map.1256a0c02879:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:56.637 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.147978+0000", 2024-09-11T09:51:56.637 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 33.394966369999999, 2024-09-11T09:51:56.637 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0044549769999999997, 2024-09-11T09:51:56.637 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:56.637 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:56.637 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:56.637 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:56.638 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4694", 2024-09-11T09:51:56.638 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:51:56.638 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 53 2024-09-11T09:51:56.638 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.638 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:56.638 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.638 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:56.638 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.147978+0000", 2024-09-11T09:51:56.638 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:56.638 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.638 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.638 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:56.638 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.147978+0000", 2024-09-11T09:51:56.639 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:56.639 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.639 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.639 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:56.639 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.147980+0000", 2024-09-11T09:51:56.639 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.8309999999999999e-06 2024-09-11T09:51:56.639 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.639 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.639 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:56.639 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.147989+0000", 2024-09-11T09:51:56.639 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.9639999999999992e-06 2024-09-11T09:51:56.639 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.639 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.639 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:56.640 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.148007+0000", 2024-09-11T09:51:56.640 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7884999999999999e-05 2024-09-11T09:51:56.640 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.640 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.640 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:56.640 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.148017+0000", 2024-09-11T09:51:56.640 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0895e-05 2024-09-11T09:51:56.640 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.640 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.640 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:56.640 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.148091+0000", 2024-09-11T09:51:56.640 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.3709999999999997e-05 2024-09-11T09:51:56.640 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.640 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.640 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:56.641 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.148189+0000", 2024-09-11T09:51:56.641 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.7403000000000002e-05 2024-09-11T09:51:56.641 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.641 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.641 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 5", 2024-09-11T09:51:56.641 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.148504+0000", 2024-09-11T09:51:56.641 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00031589900000000002 2024-09-11T09:51:56.641 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.641 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.641 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:56.641 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.149514+0000", 2024-09-11T09:51:56.641 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0010096 2024-09-11T09:51:56.641 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.642 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.642 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:56.642 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.152367+0000", 2024-09-11T09:51:56.642 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0028532309999999999 2024-09-11T09:51:56.642 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.642 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.642 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:56.642 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.152402+0000", 2024-09-11T09:51:56.642 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.4232e-05 2024-09-11T09:51:56.642 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.642 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.642 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:56.642 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.152433+0000", 2024-09-11T09:51:56.643 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.1327e-05 2024-09-11T09:51:56.643 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:56.643 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:56.643 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:56.643 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.643 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.643 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4694.0:55 2.7 2:e8766325:ns::rbd_object_map.1256a0c02879:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:56.643 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.149838+0000", 2024-09-11T09:51:56.643 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 33.393106637000002, 2024-09-11T09:51:56.643 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0060695159999999996, 2024-09-11T09:51:56.643 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:56.643 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:56.643 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:56.643 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:56.644 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4694", 2024-09-11T09:51:56.644 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:51:56.644 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 55 2024-09-11T09:51:56.644 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.644 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:56.644 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.644 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:56.644 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.149838+0000", 2024-09-11T09:51:56.644 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:56.644 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.644 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.644 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:56.644 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.149838+0000", 2024-09-11T09:51:56.644 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:56.645 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.645 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.645 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:56.645 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.149839+0000", 2024-09-11T09:51:56.645 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.119e-06 2024-09-11T09:51:56.645 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.645 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.645 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:56.645 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.149845+0000", 2024-09-11T09:51:56.645 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.7810000000000002e-06 2024-09-11T09:51:56.645 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.645 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.645 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:56.646 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.149856+0000", 2024-09-11T09:51:56.646 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1199999999999999e-05 2024-09-11T09:51:56.646 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.646 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.646 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:56.646 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.149863+0000", 2024-09-11T09:51:56.646 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.4109999999999999e-06 2024-09-11T09:51:56.646 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.646 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.646 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:56.646 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.149908+0000", 2024-09-11T09:51:56.646 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.5184e-05 2024-09-11T09:51:56.646 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.646 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.646 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:56.647 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.149964+0000", 2024-09-11T09:51:56.647 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.5395000000000001e-05 2024-09-11T09:51:56.647 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.647 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.647 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 5", 2024-09-11T09:51:56.647 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.150157+0000", 2024-09-11T09:51:56.647 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000193489 2024-09-11T09:51:56.647 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.647 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.647 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:56.647 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.152238+0000", 2024-09-11T09:51:56.647 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0020803480000000001 2024-09-11T09:51:56.647 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.648 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.648 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:56.648 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.155828+0000", 2024-09-11T09:51:56.648 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0035902210000000002 2024-09-11T09:51:56.648 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.648 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.648 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:56.648 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.155873+0000", 2024-09-11T09:51:56.648 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.5145999999999998e-05 2024-09-11T09:51:56.648 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:56.648 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:56.648 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:56.648 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.155907+0000", 2024-09-11T09:51:56.649 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.4221999999999999e-05 2024-09-11T09:51:56.649 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:56.649 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:56.649 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:56.649 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:56.649 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:56.649 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:51:56.650 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-11T09:51:56.651 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:51:56.651+0000 7f3a7b90a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 42475) UID: 0 2024-09-11T09:51:56.751 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:51:56.752+0000 7f3a7b90a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 42475) UID: 0 2024-09-11T09:51:56.790 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T09:51:56.800 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:51:56.800 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [], 2024-09-11T09:51:56.800 INFO:teuthology.orchestra.run.smithi017.stdout: "num_ops": 0 2024-09-11T09:51:56.800 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:51:56.810 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-11T09:51:56.815 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-11T09:51:56.851 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-09-11T09:51:56.852+0000 7f7b6c7d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31904) UID: 0 2024-09-11T09:51:56.915 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-11T09:51:56.915 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T09:51:56.933 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-11T09:51:56.933 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T09:51:56.951 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-11T09:51:56.951 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T09:51:56.952 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:51:56.952+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:51:56.965 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:51:56.965 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:51:56.969 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-11T09:51:56.970 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T09:51:56.975 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-11T09:51:57.003 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-11T09:51:57.003 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T09:51:57.035 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-11T09:51:57.035 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T09:51:57.053 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:51:57.054+0000 7f3c96ec9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31909) UID: 0 2024-09-11T09:51:57.112 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:51:57.112 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:51:57.121 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-11T09:51:57.154 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:51:57.154+0000 7f3c96ec9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31909) UID: 0 2024-09-11T09:51:57.229 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:51:57.229 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:51:57.238 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-11T09:51:57.282 INFO:tasks.ceph.osd.2.smithi017.stderr:2024-09-11T09:51:57.283+0000 7fd20b170640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31672) UID: 0 2024-09-11T09:51:57.346 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:51:57.346 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:51:57.355 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-11T09:51:57.355 INFO:tasks.ceph.osd.0.smithi017.stderr:2024-09-11T09:51:57.355+0000 7fb73c0f1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31665) UID: 0 2024-09-11T09:51:57.456 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:51:57.457+0000 7f3c96ec9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31909) UID: 0 2024-09-11T09:51:57.503 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:51:57.503 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:51:57.512 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-11T09:51:57.557 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-09-11T09:51:57.558+0000 7f7b6c7d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31904) UID: 0 2024-09-11T09:51:57.628 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:51:57.628 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:51:57.637 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-11T09:51:57.678 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-09-11T09:51:57.679+0000 7f7b6c7d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31904) UID: 0 2024-09-11T09:51:57.758 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:51:57.759+0000 7f3a7b90a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 42475) UID: 0 2024-09-11T09:51:57.782 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:51:57.782 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:51:57.791 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-11T09:51:57.803 INFO:teuthology.orchestra.run.smithi017.stderr:noscrub is unset 2024-09-11T09:51:57.874 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:51:57.875+0000 7f3a7b90a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 42475) UID: 0 2024-09-11T09:51:57.933 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:51:57.934 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:51:57.943 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-11T09:51:57.960 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:51:57.961+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:51:58.060 INFO:tasks.ceph.osd.0.smithi017.stderr:2024-09-11T09:51:58.061+0000 7fb73c0f1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31665) UID: 0 2024-09-11T09:51:58.086 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:51:58.086 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:51:58.095 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-11T09:51:58.161 INFO:tasks.ceph.osd.2.smithi017.stderr:2024-09-11T09:51:58.162+0000 7fd20b170640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31672) UID: 0 2024-09-11T09:51:58.199 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:51:58.199 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:51:58.208 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-11T09:51:58.262 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:51:58.262+0000 7f3c96ec9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31909) UID: 0 2024-09-11T09:51:58.311 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:51:58.312 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:51:58.320 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-11T09:51:58.362 INFO:tasks.ceph.osd.0.smithi017.stderr:2024-09-11T09:51:58.363+0000 7fb73c0f1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31665) UID: 0 2024-09-11T09:51:58.463 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:51:58.463+0000 7f3c96ec9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31909) UID: 0 2024-09-11T09:51:58.464 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:51:58.464 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:51:58.473 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-11T09:51:58.563 INFO:tasks.ceph.osd.0.smithi017.stderr:2024-09-11T09:51:58.564+0000 7fb73c0f1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31665) UID: 0 2024-09-11T09:51:58.616 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:51:58.617 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:51:58.625 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-11T09:51:58.664 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:51:58.665+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:51:58.728 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:51:58.728 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:51:58.737 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-11T09:51:58.781 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:51:58.782+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:51:58.821 INFO:tasks.workunit.client.0.smithi107.stderr: Creating snap: 10% complete... Creating snap: 20% complete... Creating snap: 100% complete...done. 2024-09-11T09:51:58.827 INFO:tasks.workunit.client.0.smithi107.stderr:+ _sudo dd if=/tmp/tmp.rvrcDirNue/data of=/dev/nbd1 bs=1M count=1 oflag=direct 2024-09-11T09:51:58.827 INFO:tasks.workunit.client.0.smithi107.stderr:+ local cmd 2024-09-11T09:51:58.827 INFO:tasks.workunit.client.0.smithi107.stderr:++ id -u 2024-09-11T09:51:58.828 INFO:tasks.workunit.client.0.smithi107.stderr:+ '[' 1000 -eq 0 ']' 2024-09-11T09:51:58.828 INFO:tasks.workunit.client.0.smithi107.stderr:++ which dd 2024-09-11T09:51:58.830 INFO:tasks.workunit.client.0.smithi107.stderr:+ cmd=/usr/bin/dd 2024-09-11T09:51:58.830 INFO:tasks.workunit.client.0.smithi107.stderr:+ shift 2024-09-11T09:51:58.831 INFO:tasks.workunit.client.0.smithi107.stderr:+ sudo -nE /usr/bin/dd if=/tmp/tmp.rvrcDirNue/data of=/dev/nbd1 bs=1M count=1 oflag=direct 2024-09-11T09:51:58.853 INFO:tasks.workunit.client.0.smithi107.stderr:1+0 records in 2024-09-11T09:51:58.853 INFO:tasks.workunit.client.0.smithi107.stderr:1+0 records out 2024-09-11T09:51:58.853 INFO:tasks.workunit.client.0.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0120452 s, 87.1 MB/s 2024-09-11T09:51:58.855 INFO:tasks.workunit.client.0.smithi107.stderr:+ unmap_device /dev/nbd1 29367 2024-09-11T09:51:58.855 INFO:tasks.workunit.client.0.smithi107.stderr:+ local args=/dev/nbd1 2024-09-11T09:51:58.855 INFO:tasks.workunit.client.0.smithi107.stderr:+ local pid=29367 2024-09-11T09:51:58.855 INFO:tasks.workunit.client.0.smithi107.stderr:+ _sudo rbd device --device-type nbd unmap /dev/nbd1 2024-09-11T09:51:58.855 INFO:tasks.workunit.client.0.smithi107.stderr:+ local cmd 2024-09-11T09:51:58.855 INFO:tasks.workunit.client.0.smithi107.stderr:++ id -u 2024-09-11T09:51:58.856 INFO:tasks.workunit.client.0.smithi107.stderr:+ '[' 1000 -eq 0 ']' 2024-09-11T09:51:58.856 INFO:tasks.workunit.client.0.smithi107.stderr:++ which rbd 2024-09-11T09:51:58.858 INFO:tasks.workunit.client.0.smithi107.stderr:+ cmd=/usr/bin/rbd 2024-09-11T09:51:58.858 INFO:tasks.workunit.client.0.smithi107.stderr:+ shift 2024-09-11T09:51:58.859 INFO:tasks.workunit.client.0.smithi107.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd unmap /dev/nbd1 2024-09-11T09:51:58.863 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:51:58.863 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:51:58.866 INFO:tasks.ceph.osd.2.smithi017.stderr:2024-09-11T09:51:58.867+0000 7fd20b170640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31672) UID: 0 2024-09-11T09:51:58.873 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-11T09:51:58.966 INFO:tasks.ceph.osd.2.smithi017.stderr:2024-09-11T09:51:58.967+0000 7fd20b170640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31672) UID: 0 2024-09-11T09:51:59.021 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:51:59.021 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:51:59.030 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-11T09:51:59.051 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T09:51:59.071 INFO:tasks.ceph.osd.2.smithi017.stderr:2024-09-11T09:51:59.072+0000 7fd20b170640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31672) UID: 0 2024-09-11T09:51:59.168 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-09-11T09:51:59.169+0000 7f7b6c7d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31904) UID: 0 2024-09-11T09:51:59.180 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:51:59.180 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:51:59.190 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-11T09:51:59.194 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-11T09:51:59.194 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T09:51:59.212 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-11T09:51:59.212 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T09:51:59.229 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-11T09:51:59.229 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T09:51:59.247 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-11T09:51:59.247 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T09:51:59.264 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-11T09:51:59.264 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T09:51:59.269 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:51:59.270+0000 7f3a7b90a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 42475) UID: 0 2024-09-11T09:51:59.281 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-11T09:51:59.281 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T09:51:59.300 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:51:59.300 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [], 2024-09-11T09:51:59.300 INFO:teuthology.orchestra.run.smithi017.stdout: "num_ops": 0 2024-09-11T09:51:59.300 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:51:59.309 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-11T09:51:59.370 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:51:59.371+0000 7f3a7b90a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 42475) UID: 0 2024-09-11T09:51:59.456 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:51:59.457 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [], 2024-09-11T09:51:59.457 INFO:teuthology.orchestra.run.smithi017.stdout: "complaint_time": 30, 2024-09-11T09:51:59.457 INFO:teuthology.orchestra.run.smithi017.stdout: "num_blocked_ops": 0 2024-09-11T09:51:59.457 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:51:59.465 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-11T09:51:59.509 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:51:59.510+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:51:59.572 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:51:59.572+0000 7f3a7b90a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 42475) UID: 0 2024-09-11T09:51:59.575 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:51:59.575 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 20, 2024-09-11T09:51:59.575 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 600, 2024-09-11T09:51:59.575 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [ 2024-09-11T09:51:59.575 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.576 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:34 2.4 2:2367385a:::rbd_data.1250f5e25b80.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:59.576 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.897717+0000", 2024-09-11T09:51:59.576 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 36.676599320999998, 2024-09-11T09:51:59.576 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.021323661000000001, 2024-09-11T09:51:59.577 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:59.577 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:59.577 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:59.577 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:59.577 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:51:59.577 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:51:59.578 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 34 2024-09-11T09:51:59.578 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.578 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:59.578 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.578 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:59.578 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.897717+0000", 2024-09-11T09:51:59.579 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:59.579 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.579 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.579 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:59.579 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.897717+0000", 2024-09-11T09:51:59.579 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:59.580 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.580 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.580 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:59.580 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.897718+0000", 2024-09-11T09:51:59.580 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.1800000000000004e-07 2024-09-11T09:51:59.580 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.581 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.581 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:59.581 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.905339+0000", 2024-09-11T09:51:59.581 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0076210239999999997 2024-09-11T09:51:59.581 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.581 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.582 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:59.582 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.905366+0000", 2024-09-11T09:51:59.582 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.6959000000000001e-05 2024-09-11T09:51:59.582 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.582 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.582 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:59.583 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.905375+0000", 2024-09-11T09:51:59.583 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.516e-06 2024-09-11T09:51:59.583 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.583 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.583 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:59.583 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.905475+0000", 2024-09-11T09:51:59.584 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.9738e-05 2024-09-11T09:51:59.584 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.584 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.584 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:59.584 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.905660+0000", 2024-09-11T09:51:59.585 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00018554199999999999 2024-09-11T09:51:59.585 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.585 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.585 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:59.585 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.905808+0000", 2024-09-11T09:51:59.586 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000147386 2024-09-11T09:51:59.586 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.586 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.586 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:59.586 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.918230+0000", 2024-09-11T09:51:59.586 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.012421690000000001 2024-09-11T09:51:59.587 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.587 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.587 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:59.587 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.918932+0000", 2024-09-11T09:51:59.587 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00070267300000000001 2024-09-11T09:51:59.588 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.588 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.588 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:59.588 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.918989+0000", 2024-09-11T09:51:59.588 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.7268999999999997e-05 2024-09-11T09:51:59.589 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.589 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.589 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:59.589 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919040+0000", 2024-09-11T09:51:59.589 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.0946000000000003e-05 2024-09-11T09:51:59.590 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:59.590 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:59.590 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:59.590 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.590 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.591 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4688.0:31 2.1 e53/15 2:8654721b:::rbd_data.1250f5e25b80.0000000000000000:head v 53'2, mlcod=0'0)", 2024-09-11T09:51:59.591 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.904661+0000", 2024-09-11T09:51:59.591 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 36.669654831999999, 2024-09-11T09:51:59.592 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.014152804999999999, 2024-09-11T09:51:59.592 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:59.592 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:59.592 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:59.593 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:59.593 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.593 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:59.593 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.904661+0000", 2024-09-11T09:51:59.593 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:59.594 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.594 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.594 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:59.594 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.904661+0000", 2024-09-11T09:51:59.594 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:59.595 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.595 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.595 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:59.595 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.904662+0000", 2024-09-11T09:51:59.595 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0759999999999999e-06 2024-09-11T09:51:59.595 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.596 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.596 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:59.596 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.907298+0000", 2024-09-11T09:51:59.596 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0026361240000000001 2024-09-11T09:51:59.596 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.597 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.597 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:59.597 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.907320+0000", 2024-09-11T09:51:59.597 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.1268999999999999e-05 2024-09-11T09:51:59.597 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.597 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.598 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:59.598 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.907331+0000", 2024-09-11T09:51:59.598 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1031e-05 2024-09-11T09:51:59.598 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.598 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.598 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:59.599 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.907403+0000", 2024-09-11T09:51:59.599 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.2571999999999996e-05 2024-09-11T09:51:59.599 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.599 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.599 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:59.599 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.907428+0000", 2024-09-11T09:51:59.600 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.4380999999999999e-05 2024-09-11T09:51:59.600 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.600 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.600 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:59.600 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.918750+0000", 2024-09-11T09:51:59.601 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.011322211 2024-09-11T09:51:59.601 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.601 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.601 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:59.601 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.918814+0000", 2024-09-11T09:51:59.601 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.4140999999999995e-05 2024-09-11T09:51:59.602 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:59.602 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:59.602 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:59.602 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.602 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.603 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:36 2.6 2:61e1b442:::rbd_data.1250f5e25b80.0000000000000003:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:59.603 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.905404+0000", 2024-09-11T09:51:59.603 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 36.668911960999999, 2024-09-11T09:51:59.603 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.030912479999999999, 2024-09-11T09:51:59.603 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:59.604 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:59.604 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:59.604 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:59.604 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:51:59.604 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:51:59.605 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 36 2024-09-11T09:51:59.605 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.605 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:59.605 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.605 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:59.605 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.905404+0000", 2024-09-11T09:51:59.606 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:59.606 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.606 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.606 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:59.606 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.905404+0000", 2024-09-11T09:51:59.607 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:59.607 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.607 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.607 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:59.607 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.905405+0000", 2024-09-11T09:51:59.607 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2270000000000001e-06 2024-09-11T09:51:59.608 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.608 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.608 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:59.608 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.914859+0000", 2024-09-11T09:51:59.608 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0094541190000000004 2024-09-11T09:51:59.608 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.609 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.609 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:59.609 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.914879+0000", 2024-09-11T09:51:59.609 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.0052999999999999e-05 2024-09-11T09:51:59.609 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.610 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.610 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:59.610 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.914897+0000", 2024-09-11T09:51:59.610 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7331000000000001e-05 2024-09-11T09:51:59.610 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.610 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.611 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:59.611 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.915019+0000", 2024-09-11T09:51:59.611 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00012168 2024-09-11T09:51:59.611 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.611 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.611 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:59.612 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.915269+0000", 2024-09-11T09:51:59.612 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00025054899999999998 2024-09-11T09:51:59.612 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.612 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.612 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 3", 2024-09-11T09:51:59.613 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.915479+0000", 2024-09-11T09:51:59.613 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00021017799999999999 2024-09-11T09:51:59.613 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.613 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.613 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:59.613 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.926070+0000", 2024-09-11T09:51:59.614 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.010590624999999999 2024-09-11T09:51:59.614 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.614 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.614 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:59.614 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.936198+0000", 2024-09-11T09:51:59.615 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.010128194 2024-09-11T09:51:59.615 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.615 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.615 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:59.615 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.936263+0000", 2024-09-11T09:51:59.616 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.5366999999999999e-05 2024-09-11T09:51:59.616 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.616 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.616 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:59.616 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.936317+0000", 2024-09-11T09:51:59.616 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.3156999999999999e-05 2024-09-11T09:51:59.617 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:59.617 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:59.617 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:59.617 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.617 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.618 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:39 2.6 2:743e7616:::rbd_data.1250f5e25b80.0000000000000005:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:59.618 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.914960+0000", 2024-09-11T09:51:59.618 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 36.659355658999999, 2024-09-11T09:51:59.618 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.022371518, 2024-09-11T09:51:59.618 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:59.619 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:59.619 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:59.619 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:59.619 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:51:59.619 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:51:59.619 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 39 2024-09-11T09:51:59.620 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.620 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:59.620 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.620 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:59.620 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.914960+0000", 2024-09-11T09:51:59.620 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:59.621 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.621 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.621 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:59.621 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.914960+0000", 2024-09-11T09:51:59.621 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:59.621 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.621 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.621 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:59.621 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.914963+0000", 2024-09-11T09:51:59.621 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.1119999999999999e-06 2024-09-11T09:51:59.621 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.622 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.622 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:59.622 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.920417+0000", 2024-09-11T09:51:59.622 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0054543969999999997 2024-09-11T09:51:59.622 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.622 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.622 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:59.622 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.920430+0000", 2024-09-11T09:51:59.622 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3414000000000001e-05 2024-09-11T09:51:59.622 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.622 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.622 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:59.622 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.920437+0000", 2024-09-11T09:51:59.623 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.6159999999999998e-06 2024-09-11T09:51:59.623 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.623 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.623 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:59.623 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.921325+0000", 2024-09-11T09:51:59.623 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000887731 2024-09-11T09:51:59.623 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.623 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.623 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:59.623 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.921422+0000", 2024-09-11T09:51:59.623 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.7201999999999995e-05 2024-09-11T09:51:59.623 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.624 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.624 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 3", 2024-09-11T09:51:59.624 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.921511+0000", 2024-09-11T09:51:59.624 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.8880000000000003e-05 2024-09-11T09:51:59.624 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.624 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.624 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:59.624 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.928092+0000", 2024-09-11T09:51:59.624 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0065816290000000003 2024-09-11T09:51:59.624 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.624 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.624 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:59.624 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.937240+0000", 2024-09-11T09:51:59.625 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0091481070000000008 2024-09-11T09:51:59.625 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.625 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.625 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:59.625 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.937294+0000", 2024-09-11T09:51:59.625 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.3100000000000003e-05 2024-09-11T09:51:59.625 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.625 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.625 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:59.625 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.937332+0000", 2024-09-11T09:51:59.625 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.8330000000000001e-05 2024-09-11T09:51:59.625 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:59.626 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:59.626 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:59.626 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.626 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.626 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4688.0:32 2.2 e53/15 2:45941fd2:::rbd_data.1250f5e25b80.0000000000000001:head v 53'3, mlcod=17'1)", 2024-09-11T09:51:59.626 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.916509+0000", 2024-09-11T09:51:59.626 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 36.657806577000002, 2024-09-11T09:51:59.626 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.015885318999999998, 2024-09-11T09:51:59.626 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:59.626 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:59.626 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:59.626 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:59.626 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.626 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:59.627 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.916509+0000", 2024-09-11T09:51:59.627 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:59.627 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.627 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.627 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:59.627 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.916509+0000", 2024-09-11T09:51:59.627 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:59.627 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.627 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.627 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:59.627 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.916511+0000", 2024-09-11T09:51:59.627 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.052e-06 2024-09-11T09:51:59.627 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.628 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.628 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:59.628 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.927019+0000", 2024-09-11T09:51:59.628 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.010508224 2024-09-11T09:51:59.628 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.628 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.628 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:59.628 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.927038+0000", 2024-09-11T09:51:59.628 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.9590999999999999e-05 2024-09-11T09:51:59.628 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.628 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.628 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:59.628 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.927050+0000", 2024-09-11T09:51:59.629 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1439e-05 2024-09-11T09:51:59.629 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.629 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.629 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:59.629 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.927115+0000", 2024-09-11T09:51:59.629 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.5301000000000004e-05 2024-09-11T09:51:59.629 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.629 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.629 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:59.629 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.927132+0000", 2024-09-11T09:51:59.629 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7297000000000001e-05 2024-09-11T09:51:59.629 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.629 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.629 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:59.630 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.932354+0000", 2024-09-11T09:51:59.630 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0052211870000000004 2024-09-11T09:51:59.630 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.630 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.630 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:59.630 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.932395+0000", 2024-09-11T09:51:59.630 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.1227999999999999e-05 2024-09-11T09:51:59.630 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:59.630 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:59.630 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:59.630 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.630 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.630 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:44 2.6 2:67246a22:::rbd_data.1250f5e25b80.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:59.631 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.943181+0000", 2024-09-11T09:51:59.631 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 36.631135366000002, 2024-09-11T09:51:59.631 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.022629371999999998, 2024-09-11T09:51:59.631 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:59.631 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:59.631 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:59.631 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:59.631 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:51:59.631 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:51:59.631 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 44 2024-09-11T09:51:59.631 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.631 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:59.631 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.631 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:59.632 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.943181+0000", 2024-09-11T09:51:59.632 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:59.632 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.632 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.632 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:59.632 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.943181+0000", 2024-09-11T09:51:59.632 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:59.632 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.632 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.632 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:59.632 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.943182+0000", 2024-09-11T09:51:59.632 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.0500000000000002e-07 2024-09-11T09:51:59.632 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.633 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.633 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:59.633 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951240+0000", 2024-09-11T09:51:59.633 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.008058529 2024-09-11T09:51:59.633 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.633 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.633 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:59.633 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951260+0000", 2024-09-11T09:51:59.633 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.9570000000000001e-05 2024-09-11T09:51:59.633 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.633 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.633 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:59.633 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951270+0000", 2024-09-11T09:51:59.634 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0018e-05 2024-09-11T09:51:59.634 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.634 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.634 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:59.634 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951349+0000", 2024-09-11T09:51:59.634 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.8801000000000007e-05 2024-09-11T09:51:59.634 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.634 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.634 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:59.634 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951483+0000", 2024-09-11T09:51:59.634 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00013427 2024-09-11T09:51:59.634 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.634 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.634 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 3", 2024-09-11T09:51:59.635 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951593+0000", 2024-09-11T09:51:59.635 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00011031000000000001 2024-09-11T09:51:59.635 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.635 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.635 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:59.635 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.957303+0000", 2024-09-11T09:51:59.635 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0057097959999999996 2024-09-11T09:51:59.635 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.635 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.635 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:59.635 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.965690+0000", 2024-09-11T09:51:59.635 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0083867420000000008 2024-09-11T09:51:59.635 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.635 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.636 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:59.636 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.965749+0000", 2024-09-11T09:51:59.636 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.8981999999999999e-05 2024-09-11T09:51:59.636 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.636 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.636 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:59.636 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.965810+0000", 2024-09-11T09:51:59.636 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.1449000000000001e-05 2024-09-11T09:51:59.636 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:59.636 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:59.636 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:59.636 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.636 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.637 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:51 2.6 2:6b5894fc:::rbd_data.1250f5e25b80.000000000000000a:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:59.637 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.951324+0000", 2024-09-11T09:51:59.637 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 36.622992293000003, 2024-09-11T09:51:59.637 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.021375273, 2024-09-11T09:51:59.637 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:59.637 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:59.637 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:59.637 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:59.637 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:51:59.637 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:51:59.637 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 51 2024-09-11T09:51:59.637 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.638 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:59.638 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.638 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:59.638 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951324+0000", 2024-09-11T09:51:59.638 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:59.638 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.638 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.638 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:59.638 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951324+0000", 2024-09-11T09:51:59.638 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:59.638 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.638 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.638 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:59.638 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951324+0000", 2024-09-11T09:51:59.639 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.39e-07 2024-09-11T09:51:59.639 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.639 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.639 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:59.639 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.958537+0000", 2024-09-11T09:51:59.639 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0072126769999999998 2024-09-11T09:51:59.639 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.639 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.639 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:59.639 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.958554+0000", 2024-09-11T09:51:59.639 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7232e-05 2024-09-11T09:51:59.639 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.639 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.640 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:59.640 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.958570+0000", 2024-09-11T09:51:59.640 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.5747999999999999e-05 2024-09-11T09:51:59.640 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.640 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.640 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:59.640 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.958638+0000", 2024-09-11T09:51:59.640 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.7810000000000003e-05 2024-09-11T09:51:59.640 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.640 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.640 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:59.640 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.958758+0000", 2024-09-11T09:51:59.640 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000120689 2024-09-11T09:51:59.641 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.641 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.641 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 3", 2024-09-11T09:51:59.641 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.958865+0000", 2024-09-11T09:51:59.641 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00010692 2024-09-11T09:51:59.641 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.641 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.641 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:59.641 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.970712+0000", 2024-09-11T09:51:59.641 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.011846327 2024-09-11T09:51:59.641 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.641 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.641 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:59.641 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.972603+0000", 2024-09-11T09:51:59.642 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0018918610000000001 2024-09-11T09:51:59.642 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.642 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.642 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:59.642 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.972653+0000", 2024-09-11T09:51:59.642 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.9762999999999998e-05 2024-09-11T09:51:59.642 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.642 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.642 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:59.642 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.972699+0000", 2024-09-11T09:51:59.642 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.5806999999999997e-05 2024-09-11T09:51:59.642 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:59.642 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:59.643 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:59.643 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.643 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.643 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:52 2.4 2:26d5c2f0:::rbd_data.1250f5e25b80.000000000000000b:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:59.643 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.958596+0000", 2024-09-11T09:51:59.643 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 36.615720097999997, 2024-09-11T09:51:59.643 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.016477849999999999, 2024-09-11T09:51:59.643 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:59.643 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:59.643 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:59.643 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:59.643 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:51:59.643 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:51:59.643 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 52 2024-09-11T09:51:59.644 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.644 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:59.644 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.644 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:59.644 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.958596+0000", 2024-09-11T09:51:59.644 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:59.644 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.644 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.644 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:59.644 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.958596+0000", 2024-09-11T09:51:59.644 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:59.644 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.644 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.645 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:59.645 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.958597+0000", 2024-09-11T09:51:59.645 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1960000000000001e-06 2024-09-11T09:51:59.645 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.645 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.645 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:59.645 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.963157+0000", 2024-09-11T09:51:59.645 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0045603029999999999 2024-09-11T09:51:59.645 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.645 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.645 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:59.645 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.963171+0000", 2024-09-11T09:51:59.645 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3777999999999999e-05 2024-09-11T09:51:59.645 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.646 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.646 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:59.646 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.963181+0000", 2024-09-11T09:51:59.646 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.9010000000000006e-06 2024-09-11T09:51:59.646 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.646 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.646 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:59.646 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.963246+0000", 2024-09-11T09:51:59.646 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.5351000000000005e-05 2024-09-11T09:51:59.646 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.646 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.646 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:59.647 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.963419+0000", 2024-09-11T09:51:59.647 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00017299700000000001 2024-09-11T09:51:59.647 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.647 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.647 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:59.647 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.963570+0000", 2024-09-11T09:51:59.647 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00015066999999999999 2024-09-11T09:51:59.647 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.647 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.647 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:59.647 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.973881+0000", 2024-09-11T09:51:59.647 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.010311268 2024-09-11T09:51:59.647 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.647 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.648 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:59.648 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.974963+0000", 2024-09-11T09:51:59.648 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.001081493 2024-09-11T09:51:59.648 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.648 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.648 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:59.648 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.975012+0000", 2024-09-11T09:51:59.648 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.9471000000000001e-05 2024-09-11T09:51:59.648 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.648 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.648 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:59.648 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.975074+0000", 2024-09-11T09:51:59.648 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.1421999999999996e-05 2024-09-11T09:51:59.649 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:59.649 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:59.649 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:59.649 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.649 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.649 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4688.0:47 2.1 e53/15 2:95f53b3e:::rbd_data.1250f5e25b80.0000000000000008:head v 53'3, mlcod=53'1)", 2024-09-11T09:51:59.649 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.960861+0000", 2024-09-11T09:51:59.649 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 36.613454679999997, 2024-09-11T09:51:59.649 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.012346107, 2024-09-11T09:51:59.649 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:59.649 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:59.649 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:59.649 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:59.649 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.650 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:59.650 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.960861+0000", 2024-09-11T09:51:59.650 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:59.650 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.650 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.650 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:59.650 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.960861+0000", 2024-09-11T09:51:59.650 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:59.650 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.650 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.650 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:59.650 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.960863+0000", 2024-09-11T09:51:59.650 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7150000000000001e-06 2024-09-11T09:51:59.651 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.651 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.651 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:59.651 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.962428+0000", 2024-09-11T09:51:59.651 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.001564668 2024-09-11T09:51:59.651 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.651 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.651 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:59.651 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.962446+0000", 2024-09-11T09:51:59.651 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.8476999999999999e-05 2024-09-11T09:51:59.651 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.651 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.651 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:59.651 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.962458+0000", 2024-09-11T09:51:59.652 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1263999999999999e-05 2024-09-11T09:51:59.652 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.652 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.652 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:59.652 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.962575+0000", 2024-09-11T09:51:59.652 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000117025 2024-09-11T09:51:59.652 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.652 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.652 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:59.652 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.962601+0000", 2024-09-11T09:51:59.652 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.5984999999999999e-05 2024-09-11T09:51:59.652 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.652 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.653 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:59.653 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.973065+0000", 2024-09-11T09:51:59.653 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.010464753 2024-09-11T09:51:59.653 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.653 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.653 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:59.653 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.973207+0000", 2024-09-11T09:51:59.653 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00014222 2024-09-11T09:51:59.653 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:59.653 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:59.653 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:59.653 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.653 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.654 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:54 2.4 2.da8979c4 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:59.654 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.963207+0000", 2024-09-11T09:51:59.654 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 36.611109241999998, 2024-09-11T09:51:59.654 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.010641575, 2024-09-11T09:51:59.654 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:59.654 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:59.654 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "reached pg", 2024-09-11T09:51:59.654 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:59.654 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:51:59.654 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:51:59.654 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 54 2024-09-11T09:51:59.654 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.654 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:59.654 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.655 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:59.655 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.963207+0000", 2024-09-11T09:51:59.655 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:59.655 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.655 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.655 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:59.655 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.963207+0000", 2024-09-11T09:51:59.655 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:59.655 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.655 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.655 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:59.655 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.963208+0000", 2024-09-11T09:51:59.655 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2979999999999999e-06 2024-09-11T09:51:59.656 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.656 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.656 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:59.656 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.967419+0000", 2024-09-11T09:51:59.656 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0042111750000000002 2024-09-11T09:51:59.656 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.656 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.656 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:59.656 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.967439+0000", 2024-09-11T09:51:59.656 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.0129999999999999e-05 2024-09-11T09:51:59.656 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.656 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.656 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:59.657 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.967453+0000", 2024-09-11T09:51:59.657 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3781000000000001e-05 2024-09-11T09:51:59.657 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.657 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.657 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:59.657 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.973840+0000", 2024-09-11T09:51:59.657 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0063872269999999997 2024-09-11T09:51:59.657 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.657 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.657 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:59.657 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.973848+0000", 2024-09-11T09:51:59.657 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.9640000000000003e-06 2024-09-11T09:51:59.657 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:59.658 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:59.658 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:59.658 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.658 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.658 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:54 2.4 2:239e915b:::rbd_data.1250f5e25b80.000000000000000c:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:59.658 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.982401+0000", 2024-09-11T09:51:59.658 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 36.591915315000001, 2024-09-11T09:51:59.658 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.016817347, 2024-09-11T09:51:59.658 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:59.658 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:59.658 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:59.658 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:59.658 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:51:59.659 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:51:59.659 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 54 2024-09-11T09:51:59.659 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.659 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:59.659 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.659 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:59.659 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.982401+0000", 2024-09-11T09:51:59.659 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:59.659 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.659 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.660 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:59.660 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.982401+0000", 2024-09-11T09:51:59.660 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:59.660 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.660 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.660 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:59.660 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.982402+0000", 2024-09-11T09:51:59.660 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.8999999999999995e-07 2024-09-11T09:51:59.660 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.661 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.661 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:59.661 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.986535+0000", 2024-09-11T09:51:59.661 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0041333689999999996 2024-09-11T09:51:59.661 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.661 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.661 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:59.661 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.986555+0000", 2024-09-11T09:51:59.661 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.9851e-05 2024-09-11T09:51:59.661 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.661 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.661 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:59.661 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.986569+0000", 2024-09-11T09:51:59.662 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3784000000000001e-05 2024-09-11T09:51:59.662 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.662 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.662 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:59.662 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.986683+0000", 2024-09-11T09:51:59.662 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000114474 2024-09-11T09:51:59.662 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.662 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.662 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:59.662 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.986815+0000", 2024-09-11T09:51:59.662 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00013217799999999999 2024-09-11T09:51:59.662 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.662 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.662 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:59.663 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.986918+0000", 2024-09-11T09:51:59.663 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00010235599999999999 2024-09-11T09:51:59.663 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.663 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.663 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:59.663 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.998983+0000", 2024-09-11T09:51:59.663 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.012065352 2024-09-11T09:51:59.663 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.663 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.663 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:59.663 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.999081+0000", 2024-09-11T09:51:59.663 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.8091000000000007e-05 2024-09-11T09:51:59.663 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.663 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.664 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:59.664 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.999159+0000", 2024-09-11T09:51:59.664 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.7823000000000004e-05 2024-09-11T09:51:59.664 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.664 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.664 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:59.664 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.999218+0000", 2024-09-11T09:51:59.664 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.9178999999999997e-05 2024-09-11T09:51:59.664 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:59.664 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:59.664 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:59.664 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.664 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.665 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:56 2.4 2:2167012f:::rbd_data.1250f5e25b80.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:59.665 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.986661+0000", 2024-09-11T09:51:59.665 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 36.587654862000001, 2024-09-11T09:51:59.665 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.017353539000000001, 2024-09-11T09:51:59.665 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:59.665 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:59.665 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:59.665 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:59.665 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:51:59.665 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:51:59.665 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 56 2024-09-11T09:51:59.665 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.665 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:59.665 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.666 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:59.666 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.986661+0000", 2024-09-11T09:51:59.666 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:59.666 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.666 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.666 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:59.666 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.986661+0000", 2024-09-11T09:51:59.666 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:59.666 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.666 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.666 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:59.666 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.986663+0000", 2024-09-11T09:51:59.666 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.8989999999999999e-06 2024-09-11T09:51:59.667 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.667 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.667 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:59.667 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.990444+0000", 2024-09-11T09:51:59.667 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0037808569999999999 2024-09-11T09:51:59.667 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.667 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.667 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:59.667 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.990462+0000", 2024-09-11T09:51:59.667 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.8450000000000001e-05 2024-09-11T09:51:59.667 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.667 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.667 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:59.667 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.990476+0000", 2024-09-11T09:51:59.668 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3565e-05 2024-09-11T09:51:59.668 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.668 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.668 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:59.668 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.993582+0000", 2024-09-11T09:51:59.668 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0031059109999999998 2024-09-11T09:51:59.668 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.668 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.668 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:59.668 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.993686+0000", 2024-09-11T09:51:59.668 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000104424 2024-09-11T09:51:59.668 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.668 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.669 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:59.669 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.993799+0000", 2024-09-11T09:51:59.669 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000112985 2024-09-11T09:51:59.669 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.669 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.669 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:59.669 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.002449+0000", 2024-09-11T09:51:59.669 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.008650151 2024-09-11T09:51:59.669 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.669 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.669 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:59.669 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.003948+0000", 2024-09-11T09:51:59.669 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.001498636 2024-09-11T09:51:59.670 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.670 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.670 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:59.670 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.003985+0000", 2024-09-11T09:51:59.670 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.7054999999999997e-05 2024-09-11T09:51:59.670 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.670 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.670 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:59.670 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.004015+0000", 2024-09-11T09:51:59.670 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.9606000000000001e-05 2024-09-11T09:51:59.670 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:59.670 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:59.671 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:59.671 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.671 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.671 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:58 2.6 2:6277c644:::rbd_data.1250f5e25b80.000000000000000e:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:59.671 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.990540+0000", 2024-09-11T09:51:59.671 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 36.583776067000002, 2024-09-11T09:51:59.671 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.016232068999999998, 2024-09-11T09:51:59.671 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:59.671 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:59.671 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:59.671 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:59.671 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:51:59.671 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:51:59.671 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 58 2024-09-11T09:51:59.672 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.672 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:59.672 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.672 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:59.672 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.990540+0000", 2024-09-11T09:51:59.672 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:59.672 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.672 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.672 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:59.672 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.990540+0000", 2024-09-11T09:51:59.672 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:59.672 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.672 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.673 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:59.673 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.990541+0000", 2024-09-11T09:51:59.673 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2359999999999999e-06 2024-09-11T09:51:59.673 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.673 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.673 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:59.673 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.994626+0000", 2024-09-11T09:51:59.673 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.004084865 2024-09-11T09:51:59.673 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.673 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.673 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:59.673 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.994644+0000", 2024-09-11T09:51:59.673 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7866000000000001e-05 2024-09-11T09:51:59.674 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.674 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.674 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:59.674 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.994653+0000", 2024-09-11T09:51:59.674 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.2159999999999995e-06 2024-09-11T09:51:59.674 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.674 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.674 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:59.674 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.994732+0000", 2024-09-11T09:51:59.674 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.8954000000000003e-05 2024-09-11T09:51:59.674 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.674 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.675 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:59.675 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.994906+0000", 2024-09-11T09:51:59.675 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0001741 2024-09-11T09:51:59.675 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.675 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.675 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 3", 2024-09-11T09:51:59.675 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.995083+0000", 2024-09-11T09:51:59.675 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00017637100000000001 2024-09-11T09:51:59.675 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.675 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.675 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:59.675 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.003066+0000", 2024-09-11T09:51:59.675 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0079833360000000006 2024-09-11T09:51:59.676 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.676 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.676 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:59.676 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.006634+0000", 2024-09-11T09:51:59.676 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.003568309 2024-09-11T09:51:59.676 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.676 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.676 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:59.676 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.006716+0000", 2024-09-11T09:51:59.676 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.1377000000000004e-05 2024-09-11T09:51:59.676 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.676 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.676 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:59.676 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.006772+0000", 2024-09-11T09:51:59.677 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.6439e-05 2024-09-11T09:51:59.677 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:59.677 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:59.677 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:59.677 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.677 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.677 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:59 2.4 2:2906e6a7:::rbd_header.1250f5e25b80:head [watch reconnect cookie 140424358411408 gen 2] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:59.677 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.994684+0000", 2024-09-11T09:51:59.677 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 36.579632435999997, 2024-09-11T09:51:59.677 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0094134349999999995, 2024-09-11T09:51:59.677 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:59.677 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:59.677 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "started", 2024-09-11T09:51:59.678 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:59.678 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:51:59.678 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:51:59.678 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 59 2024-09-11T09:51:59.678 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.678 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:59.678 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.678 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:59.678 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.994684+0000", 2024-09-11T09:51:59.678 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:59.678 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.678 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.678 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:59.678 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.994684+0000", 2024-09-11T09:51:59.679 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:59.679 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.679 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.679 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:59.679 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.994686+0000", 2024-09-11T09:51:59.679 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.8159999999999998e-06 2024-09-11T09:51:59.679 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.679 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.679 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:59.679 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.994691+0000", 2024-09-11T09:51:59.679 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.5009999999999998e-06 2024-09-11T09:51:59.679 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.679 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.679 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:59.680 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.994698+0000", 2024-09-11T09:51:59.680 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.3039999999999999e-06 2024-09-11T09:51:59.680 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.680 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.680 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:59.680 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.994703+0000", 2024-09-11T09:51:59.680 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.5190000000000003e-06 2024-09-11T09:51:59.680 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.680 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.680 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:59.680 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.998678+0000", 2024-09-11T09:51:59.680 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0039752340000000002 2024-09-11T09:51:59.680 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.681 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.681 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:59.681 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.998748+0000", 2024-09-11T09:51:59.681 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.0114e-05 2024-09-11T09:51:59.681 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.681 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.681 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:59.681 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.004097+0000", 2024-09-11T09:51:59.681 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0053489469999999997 2024-09-11T09:51:59.681 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:59.681 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:59.681 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:59.681 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.681 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.682 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4694.0:33 2.4 2:3bfc818c:ns::rbd_data.1256a0c02879.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:59.682 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.084537+0000", 2024-09-11T09:51:59.682 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 36.489779007999999, 2024-09-11T09:51:59.682 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.022075773999999999, 2024-09-11T09:51:59.682 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:59.682 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:59.682 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:59.682 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:59.682 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4694", 2024-09-11T09:51:59.682 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:51:59.682 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 33 2024-09-11T09:51:59.682 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.682 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:59.682 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.682 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:59.683 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.084537+0000", 2024-09-11T09:51:59.683 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:59.683 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.683 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.683 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:59.683 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.084537+0000", 2024-09-11T09:51:59.683 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:59.683 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.683 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.683 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:59.683 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.084538+0000", 2024-09-11T09:51:59.683 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.5000000000000001e-07 2024-09-11T09:51:59.683 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.683 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.683 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:59.684 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.095984+0000", 2024-09-11T09:51:59.684 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.011446543 2024-09-11T09:51:59.684 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.684 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.684 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:59.684 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.096014+0000", 2024-09-11T09:51:59.684 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.9745000000000001e-05 2024-09-11T09:51:59.684 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.684 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.684 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:59.684 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.096030+0000", 2024-09-11T09:51:59.684 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6152999999999999e-05 2024-09-11T09:51:59.684 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.685 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.685 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:59.685 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.096113+0000", 2024-09-11T09:51:59.685 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.2731999999999993e-05 2024-09-11T09:51:59.685 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.685 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.685 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:59.685 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.096273+0000", 2024-09-11T09:51:59.685 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00015973699999999999 2024-09-11T09:51:59.685 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.685 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.685 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 0", 2024-09-11T09:51:59.686 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.096394+0000", 2024-09-11T09:51:59.686 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000120883 2024-09-11T09:51:59.686 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.686 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.686 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:59.686 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.106048+0000", 2024-09-11T09:51:59.686 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0096548020000000005 2024-09-11T09:51:59.686 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.686 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.686 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:59.686 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.106531+0000", 2024-09-11T09:51:59.686 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00048264299999999999 2024-09-11T09:51:59.686 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.686 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.687 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:59.687 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.106565+0000", 2024-09-11T09:51:59.687 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.4343e-05 2024-09-11T09:51:59.687 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.687 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.687 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:59.687 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.106613+0000", 2024-09-11T09:51:59.687 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.7342999999999997e-05 2024-09-11T09:51:59.687 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:59.687 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:59.687 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:59.687 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.687 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.687 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4694.0:32 2.1 e53/15 2:8a862b48:ns::rbd_data.1256a0c02879.0000000000000000:head v 53'8, mlcod=53'6)", 2024-09-11T09:51:59.687 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.091418+0000", 2024-09-11T09:51:59.688 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 36.482897839000003, 2024-09-11T09:51:59.688 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.013027372000000001, 2024-09-11T09:51:59.688 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:59.688 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:59.688 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:59.688 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:59.688 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.688 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:59.688 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.091418+0000", 2024-09-11T09:51:59.688 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:59.688 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.688 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.688 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:59.689 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.091418+0000", 2024-09-11T09:51:59.689 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:59.689 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.689 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.689 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:59.689 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.091420+0000", 2024-09-11T09:51:59.689 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6899999999999999e-06 2024-09-11T09:51:59.689 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.689 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.689 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:59.689 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.093086+0000", 2024-09-11T09:51:59.689 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0016662510000000001 2024-09-11T09:51:59.690 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.690 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.690 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:59.690 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.093107+0000", 2024-09-11T09:51:59.690 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.0999e-05 2024-09-11T09:51:59.690 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.690 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.690 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:59.690 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.093120+0000", 2024-09-11T09:51:59.690 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2544999999999999e-05 2024-09-11T09:51:59.690 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.690 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.690 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:59.690 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.093186+0000", 2024-09-11T09:51:59.691 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.6595999999999997e-05 2024-09-11T09:51:59.691 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.691 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.691 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:59.691 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.093208+0000", 2024-09-11T09:51:59.691 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.1624000000000002e-05 2024-09-11T09:51:59.691 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.691 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.691 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:59.691 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.104392+0000", 2024-09-11T09:51:59.691 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.011184265000000001 2024-09-11T09:51:59.691 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.691 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.691 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:59.691 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.104446+0000", 2024-09-11T09:51:59.692 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.3402000000000001e-05 2024-09-11T09:51:59.692 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:59.692 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:59.692 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:59.692 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.692 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.692 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4694.0:45 2.6 2:671fb830:ns::rbd_data.1256a0c02879.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:51:59.692 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.109629+0000", 2024-09-11T09:51:59.692 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 36.464686997999998, 2024-09-11T09:51:59.692 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.036403143999999998, 2024-09-11T09:51:59.692 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:59.692 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:59.692 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:59.692 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:51:59.693 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4694", 2024-09-11T09:51:59.693 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:51:59.693 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 45 2024-09-11T09:51:59.693 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.693 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:59.693 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.693 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:59.693 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.109629+0000", 2024-09-11T09:51:59.693 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:59.694 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.694 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.694 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:59.694 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.109629+0000", 2024-09-11T09:51:59.694 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:59.694 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.694 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.694 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:59.694 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.109630+0000", 2024-09-11T09:51:59.694 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1200000000000001e-06 2024-09-11T09:51:59.694 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.694 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.694 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:59.695 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.121370+0000", 2024-09-11T09:51:59.695 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.011739571000000001 2024-09-11T09:51:59.695 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.695 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.695 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:59.695 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.121391+0000", 2024-09-11T09:51:59.695 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.1175e-05 2024-09-11T09:51:59.695 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.695 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.695 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:59.695 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.121404+0000", 2024-09-11T09:51:59.695 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2807e-05 2024-09-11T09:51:59.695 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.696 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.696 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:59.696 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.121498+0000", 2024-09-11T09:51:59.696 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.3770999999999994e-05 2024-09-11T09:51:59.696 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.696 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.696 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:59.696 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.121612+0000", 2024-09-11T09:51:59.696 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000114666 2024-09-11T09:51:59.696 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.696 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.696 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 3", 2024-09-11T09:51:59.696 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.121711+0000", 2024-09-11T09:51:59.697 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.8647999999999995e-05 2024-09-11T09:51:59.697 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.697 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.697 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:51:59.697 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.126558+0000", 2024-09-11T09:51:59.697 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0048473769999999999 2024-09-11T09:51:59.697 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.697 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.697 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:51:59.697 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.145912+0000", 2024-09-11T09:51:59.697 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.019354281000000001 2024-09-11T09:51:59.697 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.697 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.697 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:59.698 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.145977+0000", 2024-09-11T09:51:59.698 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.4135999999999997e-05 2024-09-11T09:51:59.698 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.698 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.698 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:59.698 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.146032+0000", 2024-09-11T09:51:59.698 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.5591999999999999e-05 2024-09-11T09:51:59.698 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:59.698 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:59.698 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:59.698 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.698 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.698 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4694.0:41 2.2 e53/15 2:5e2505c4:ns::rbd_data.1256a0c02879.0000000000000003:head v 53'4, mlcod=18'2)", 2024-09-11T09:51:59.698 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.113199+0000", 2024-09-11T09:51:59.698 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 36.461116588000003, 2024-09-11T09:51:59.699 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.011404512, 2024-09-11T09:51:59.699 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:59.699 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:59.699 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:59.699 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:59.699 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.699 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:59.699 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.113199+0000", 2024-09-11T09:51:59.699 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:59.699 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.699 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.699 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:59.699 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.113199+0000", 2024-09-11T09:51:59.699 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:59.700 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.700 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.700 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:59.700 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.113200+0000", 2024-09-11T09:51:59.700 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.2799999999999995e-07 2024-09-11T09:51:59.700 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.700 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.700 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:59.700 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.118872+0000", 2024-09-11T09:51:59.700 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0056718089999999999 2024-09-11T09:51:59.700 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.700 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.700 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:59.700 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.118889+0000", 2024-09-11T09:51:59.701 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7428999999999999e-05 2024-09-11T09:51:59.701 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.701 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.701 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:59.701 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.118900+0000", 2024-09-11T09:51:59.701 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0243e-05 2024-09-11T09:51:59.701 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.701 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.701 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:59.701 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.118959+0000", 2024-09-11T09:51:59.701 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.9364000000000003e-05 2024-09-11T09:51:59.701 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.701 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.702 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:59.702 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.118976+0000", 2024-09-11T09:51:59.702 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7124000000000001e-05 2024-09-11T09:51:59.702 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.702 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.702 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:59.702 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.124557+0000", 2024-09-11T09:51:59.702 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0055812659999999997 2024-09-11T09:51:59.702 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.702 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.702 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:59.702 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.124604+0000", 2024-09-11T09:51:59.702 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.6548999999999998e-05 2024-09-11T09:51:59.702 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:59.702 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:59.703 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:59.703 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.703 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.703 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4694.0:43 2.0 e53/15 2:16c8ae84:ns::rbd_data.1256a0c02879.0000000000000005:head v 53'2, mlcod=0'0)", 2024-09-11T09:51:59.703 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.158559+0000", 2024-09-11T09:51:59.703 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 36.415757262, 2024-09-11T09:51:59.703 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.016713677999999999, 2024-09-11T09:51:59.703 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:59.703 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:59.703 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:59.703 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:59.703 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.704 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:59.704 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.158559+0000", 2024-09-11T09:51:59.704 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:59.704 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.704 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.704 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:59.704 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.158559+0000", 2024-09-11T09:51:59.704 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:59.704 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.704 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.704 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:59.704 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.158560+0000", 2024-09-11T09:51:59.704 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1459999999999999e-06 2024-09-11T09:51:59.704 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.705 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.705 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:59.705 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.166465+0000", 2024-09-11T09:51:59.705 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.007904767 2024-09-11T09:51:59.705 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.705 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.705 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:59.705 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.166491+0000", 2024-09-11T09:51:59.705 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.6308e-05 2024-09-11T09:51:59.705 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.705 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.705 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:59.706 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.166502+0000", 2024-09-11T09:51:59.706 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0736e-05 2024-09-11T09:51:59.706 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.706 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.706 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:59.706 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.166564+0000", 2024-09-11T09:51:59.706 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.2578999999999998e-05 2024-09-11T09:51:59.706 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.706 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.706 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:59.706 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.166585+0000", 2024-09-11T09:51:59.706 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.0395999999999999e-05 2024-09-11T09:51:59.706 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.706 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.707 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:59.707 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.175227+0000", 2024-09-11T09:51:59.707 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0086426420000000007 2024-09-11T09:51:59.707 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.707 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.707 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:59.707 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.175272+0000", 2024-09-11T09:51:59.707 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.5104e-05 2024-09-11T09:51:59.707 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:59.707 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:59.707 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:59.707 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.707 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.707 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4694.0:59 2.2 e53/15 2:41adefc9:ns::rbd_data.1256a0c02879.000000000000000e:head v 53'5, mlcod=53'3)", 2024-09-11T09:51:59.708 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.163389+0000", 2024-09-11T09:51:59.708 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 36.410927286000003, 2024-09-11T09:51:59.708 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.013095719, 2024-09-11T09:51:59.708 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:51:59.708 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:51:59.708 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:51:59.708 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:51:59.708 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.708 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:51:59.708 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.163389+0000", 2024-09-11T09:51:59.708 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:59.708 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.708 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.708 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:51:59.709 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.163389+0000", 2024-09-11T09:51:59.709 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:51:59.709 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.709 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.709 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:51:59.709 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.163390+0000", 2024-09-11T09:51:59.709 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.336e-06 2024-09-11T09:51:59.709 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.709 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.709 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:51:59.709 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.170335+0000", 2024-09-11T09:51:59.709 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0069451080000000002 2024-09-11T09:51:59.709 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.710 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.710 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:51:59.710 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.170354+0000", 2024-09-11T09:51:59.710 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.8423e-05 2024-09-11T09:51:59.710 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.710 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.710 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:51:59.710 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.170366+0000", 2024-09-11T09:51:59.710 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2235999999999999e-05 2024-09-11T09:51:59.710 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.710 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.710 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:51:59.710 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.170439+0000", 2024-09-11T09:51:59.710 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.2989000000000002e-05 2024-09-11T09:51:59.710 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.711 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.711 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:51:59.711 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.170460+0000", 2024-09-11T09:51:59.711 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.1211000000000001e-05 2024-09-11T09:51:59.711 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.711 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.711 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:51:59.711 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.176442+0000", 2024-09-11T09:51:59.711 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0059821969999999999 2024-09-11T09:51:59.711 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:51:59.711 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:51:59.711 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:51:59.711 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.176484+0000", 2024-09-11T09:51:59.711 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.2218999999999997e-05 2024-09-11T09:51:59.711 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:59.711 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:59.712 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:59.712 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:51:59.712 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:51:59.712 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:51:59.713 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-11T09:51:59.714 INFO:tasks.ceph.osd.2.smithi017.stderr:2024-09-11T09:51:59.714+0000 7fd20b170640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31672) UID: 0 2024-09-11T09:51:59.814 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:51:59.814+0000 7f3c96ec9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31909) UID: 0 2024-09-11T09:51:59.818 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-11T09:51:59.862 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:51:59.862 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [], 2024-09-11T09:51:59.862 INFO:teuthology.orchestra.run.smithi017.stdout: "num_ops": 0 2024-09-11T09:51:59.862 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:51:59.871 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-11T09:51:59.915 INFO:tasks.ceph.osd.0.smithi017.stderr:2024-09-11T09:51:59.916+0000 7fb73c0f1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31665) UID: 0 2024-09-11T09:51:59.968 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 5, 0] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 5, 4] 2024-09-11T09:51:59.969 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-11T09:51:59.969 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-09-11T09:51:59.969 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump -f json-pretty 2024-09-11T09:51:59.980 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:51:59.980 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [], 2024-09-11T09:51:59.980 INFO:teuthology.orchestra.run.smithi017.stdout: "complaint_time": 30, 2024-09-11T09:51:59.980 INFO:teuthology.orchestra.run.smithi017.stdout: "num_blocked_ops": 0 2024-09-11T09:51:59.980 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:51:59.990 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-11T09:52:00.016 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:52:00.016+0000 7f3c96ec9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31909) UID: 0 2024-09-11T09:52:00.116 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:52:00.117+0000 7f3a7b90a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 42475) UID: 0 2024-09-11T09:52:00.142 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:52:00.143 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 20, 2024-09-11T09:52:00.143 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 600, 2024-09-11T09:52:00.143 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [ 2024-09-11T09:52:00.143 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.143 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:31 2.1 2:8654721b:::rbd_data.1250f5e25b80.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:00.143 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.896024+0000", 2024-09-11T09:52:00.143 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 37.246535731000002, 2024-09-11T09:52:00.143 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.023148401999999998, 2024-09-11T09:52:00.143 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:00.143 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:00.143 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:00.143 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:00.144 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:52:00.144 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:52:00.144 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 31 2024-09-11T09:52:00.144 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.144 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:00.144 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.144 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:00.144 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.896024+0000", 2024-09-11T09:52:00.144 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:00.144 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.144 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.144 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:00.144 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.896024+0000", 2024-09-11T09:52:00.145 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:00.145 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.145 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.145 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:00.145 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.896025+0000", 2024-09-11T09:52:00.145 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.54e-07 2024-09-11T09:52:00.145 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.145 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.145 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:00.145 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.904287+0000", 2024-09-11T09:52:00.145 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0082624729999999993 2024-09-11T09:52:00.145 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.146 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.146 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:00.146 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.904305+0000", 2024-09-11T09:52:00.146 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7912999999999999e-05 2024-09-11T09:52:00.146 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.146 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.146 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:00.146 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.904312+0000", 2024-09-11T09:52:00.146 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.0230000000000004e-06 2024-09-11T09:52:00.146 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.146 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.146 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:00.147 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.904395+0000", 2024-09-11T09:52:00.147 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.2355999999999996e-05 2024-09-11T09:52:00.147 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.147 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.147 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:00.147 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.904499+0000", 2024-09-11T09:52:00.147 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000104672 2024-09-11T09:52:00.147 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.147 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.147 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1", 2024-09-11T09:52:00.147 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.904594+0000", 2024-09-11T09:52:00.147 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.5129000000000003e-05 2024-09-11T09:52:00.147 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.148 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.148 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:00.148 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.918183+0000", 2024-09-11T09:52:00.148 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.013588355999999999 2024-09-11T09:52:00.148 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.148 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.148 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:00.148 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919053+0000", 2024-09-11T09:52:00.148 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00087066800000000001 2024-09-11T09:52:00.148 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.148 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.148 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:00.149 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919125+0000", 2024-09-11T09:52:00.149 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.1333000000000003e-05 2024-09-11T09:52:00.149 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.149 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.149 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:00.149 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919172+0000", 2024-09-11T09:52:00.149 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.7725000000000001e-05 2024-09-11T09:52:00.149 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:00.149 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:00.149 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:00.149 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.149 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.149 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4688.0:42 2.3 e53/15 2:c54742d8:::rbd_data.1250f5e25b80.0000000000000006:head v 53'2, mlcod=0'0)", 2024-09-11T09:52:00.150 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.939272+0000", 2024-09-11T09:52:00.150 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 37.203287746999997, 2024-09-11T09:52:00.150 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.010821958, 2024-09-11T09:52:00.150 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:00.150 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:00.150 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:00.150 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:00.150 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.150 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:00.150 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.939272+0000", 2024-09-11T09:52:00.150 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:00.150 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.151 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.151 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:00.151 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.939272+0000", 2024-09-11T09:52:00.151 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:00.151 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.151 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.151 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:00.151 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.939273+0000", 2024-09-11T09:52:00.151 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.7899999999999997e-07 2024-09-11T09:52:00.151 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.151 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.151 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:00.152 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.944403+0000", 2024-09-11T09:52:00.152 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0051302129999999998 2024-09-11T09:52:00.152 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.152 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.152 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:00.152 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.944413+0000", 2024-09-11T09:52:00.152 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0182e-05 2024-09-11T09:52:00.152 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.152 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.152 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:00.152 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.944420+0000", 2024-09-11T09:52:00.152 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.7000000000000002e-06 2024-09-11T09:52:00.152 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.153 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.153 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:00.153 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.944482+0000", 2024-09-11T09:52:00.153 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.1911999999999999e-05 2024-09-11T09:52:00.153 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.153 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.153 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:00.153 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.944508+0000", 2024-09-11T09:52:00.153 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.6582000000000001e-05 2024-09-11T09:52:00.153 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.153 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.153 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:00.154 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.950050+0000", 2024-09-11T09:52:00.154 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0055414080000000003 2024-09-11T09:52:00.154 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.154 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.154 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:00.154 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.950094+0000", 2024-09-11T09:52:00.154 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.4082000000000002e-05 2024-09-11T09:52:00.154 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:00.154 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:00.154 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:00.154 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.154 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.154 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:47 2.1 2:95f53b3e:::rbd_data.1250f5e25b80.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:00.155 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.948330+0000", 2024-09-11T09:52:00.155 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 37.194229403000001, 2024-09-11T09:52:00.155 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.025156603999999999, 2024-09-11T09:52:00.155 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:00.155 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:00.155 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:00.155 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:00.155 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:52:00.155 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:52:00.155 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 47 2024-09-11T09:52:00.155 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.155 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:00.155 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.155 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:00.156 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.948330+0000", 2024-09-11T09:52:00.156 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:00.156 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.156 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.156 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:00.156 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.948330+0000", 2024-09-11T09:52:00.156 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:00.156 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.156 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.156 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:00.156 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.948333+0000", 2024-09-11T09:52:00.156 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.481e-06 2024-09-11T09:52:00.156 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.156 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.157 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:00.157 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.960263+0000", 2024-09-11T09:52:00.157 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.011930671 2024-09-11T09:52:00.157 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.157 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.157 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:00.157 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.960289+0000", 2024-09-11T09:52:00.157 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.5573000000000001e-05 2024-09-11T09:52:00.157 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.157 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.157 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:00.157 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.960304+0000", 2024-09-11T09:52:00.157 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.4647000000000001e-05 2024-09-11T09:52:00.158 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.158 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.158 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:00.158 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.960390+0000", 2024-09-11T09:52:00.158 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.6088999999999995e-05 2024-09-11T09:52:00.158 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.158 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.158 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:00.158 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.960562+0000", 2024-09-11T09:52:00.158 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00017237400000000001 2024-09-11T09:52:00.158 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.158 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.159 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1", 2024-09-11T09:52:00.159 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.960725+0000", 2024-09-11T09:52:00.159 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00016321599999999999 2024-09-11T09:52:00.159 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.159 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.159 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:00.159 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.969465+0000", 2024-09-11T09:52:00.159 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0087396899999999996 2024-09-11T09:52:00.159 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.159 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.159 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:00.159 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.973393+0000", 2024-09-11T09:52:00.159 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0039274669999999996 2024-09-11T09:52:00.160 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.160 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.160 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:00.160 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.973443+0000", 2024-09-11T09:52:00.160 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.0340000000000003e-05 2024-09-11T09:52:00.160 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.160 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.160 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:00.160 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.973487+0000", 2024-09-11T09:52:00.160 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.4056e-05 2024-09-11T09:52:00.160 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:00.160 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:00.161 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:00.161 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.161 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.161 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:61 2.1 2:9af038b1:::rbd_data.1250f5e25b80.000000000000000f:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:00.161 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.975764+0000", 2024-09-11T09:52:00.161 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 37.166795878999999, 2024-09-11T09:52:00.161 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.016158891000000002, 2024-09-11T09:52:00.161 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:00.161 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:00.162 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:00.162 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:00.162 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:52:00.162 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:52:00.162 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 61 2024-09-11T09:52:00.162 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.162 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:00.162 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.162 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:00.162 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.975764+0000", 2024-09-11T09:52:00.162 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:00.162 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.162 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.163 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:00.163 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.975764+0000", 2024-09-11T09:52:00.163 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:00.163 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.163 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.163 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:00.163 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.975766+0000", 2024-09-11T09:52:00.163 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.767e-06 2024-09-11T09:52:00.163 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.163 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.163 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:00.163 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.983204+0000", 2024-09-11T09:52:00.163 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0074388400000000004 2024-09-11T09:52:00.164 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.164 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.164 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:00.164 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.983226+0000", 2024-09-11T09:52:00.164 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.1279000000000001e-05 2024-09-11T09:52:00.164 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.164 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.164 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:00.164 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.983238+0000", 2024-09-11T09:52:00.164 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2563e-05 2024-09-11T09:52:00.164 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.164 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.164 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:00.165 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.983318+0000", 2024-09-11T09:52:00.165 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.0075999999999997e-05 2024-09-11T09:52:00.165 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.165 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.165 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:00.165 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.983429+0000", 2024-09-11T09:52:00.165 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000110618 2024-09-11T09:52:00.165 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.165 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.165 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1", 2024-09-11T09:52:00.165 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.983525+0000", 2024-09-11T09:52:00.165 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.603e-05 2024-09-11T09:52:00.165 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.166 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.166 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:00.166 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.989004+0000", 2024-09-11T09:52:00.166 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0054785429999999998 2024-09-11T09:52:00.166 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.166 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.166 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:00.166 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.991788+0000", 2024-09-11T09:52:00.166 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0027845729999999998 2024-09-11T09:52:00.166 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.166 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.166 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:00.166 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.991870+0000", 2024-09-11T09:52:00.167 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.1567e-05 2024-09-11T09:52:00.167 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.167 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.167 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:00.167 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.991923+0000", 2024-09-11T09:52:00.167 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.3035000000000003e-05 2024-09-11T09:52:00.167 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:00.167 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:00.167 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:00.167 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.167 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.167 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4694.0:32 2.1 2:8a862b48:ns::rbd_data.1256a0c02879.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:00.167 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.084089+0000", 2024-09-11T09:52:00.167 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 37.058470405000001, 2024-09-11T09:52:00.168 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.020630553999999999, 2024-09-11T09:52:00.168 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:00.168 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:00.168 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:00.168 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:00.168 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4694", 2024-09-11T09:52:00.168 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:52:00.168 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 32 2024-09-11T09:52:00.168 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.168 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:00.168 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.168 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:00.168 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.084089+0000", 2024-09-11T09:52:00.168 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:00.169 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.169 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.169 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:00.169 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.084089+0000", 2024-09-11T09:52:00.169 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:00.169 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.169 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.169 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:00.169 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.084090+0000", 2024-09-11T09:52:00.169 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.2900000000000003e-07 2024-09-11T09:52:00.169 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.169 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.170 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:00.170 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.090866+0000", 2024-09-11T09:52:00.170 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0067756609999999997 2024-09-11T09:52:00.170 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.170 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.170 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:00.170 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.090891+0000", 2024-09-11T09:52:00.170 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.5119000000000001e-05 2024-09-11T09:52:00.170 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.170 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.170 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:00.170 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.090900+0000", 2024-09-11T09:52:00.170 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.4110000000000002e-06 2024-09-11T09:52:00.171 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.171 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.171 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:00.171 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.090968+0000", 2024-09-11T09:52:00.171 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.8041999999999997e-05 2024-09-11T09:52:00.171 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.171 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.171 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:00.171 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.091115+0000", 2024-09-11T09:52:00.171 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00014633499999999999 2024-09-11T09:52:00.171 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.171 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.171 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1", 2024-09-11T09:52:00.172 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.091284+0000", 2024-09-11T09:52:00.172 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000169836 2024-09-11T09:52:00.172 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.172 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.172 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:00.172 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.100896+0000", 2024-09-11T09:52:00.172 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0096118639999999995 2024-09-11T09:52:00.172 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.172 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.172 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:00.172 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.104604+0000", 2024-09-11T09:52:00.172 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0037074619999999999 2024-09-11T09:52:00.172 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.173 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.173 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:00.173 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.104669+0000", 2024-09-11T09:52:00.173 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.4836000000000001e-05 2024-09-11T09:52:00.173 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.173 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.173 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:00.173 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.104720+0000", 2024-09-11T09:52:00.173 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.1258999999999998e-05 2024-09-11T09:52:00.173 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:00.173 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:00.173 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:00.173 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.173 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.174 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4694.0:44 2.3 e53/15 2:cd1a7d1c:ns::rbd_data.1256a0c02879.0000000000000006:head v 53'3, mlcod=16'1)", 2024-09-11T09:52:00.174 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.120387+0000", 2024-09-11T09:52:00.174 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 37.022172752000003, 2024-09-11T09:52:00.174 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.011988468, 2024-09-11T09:52:00.174 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:00.174 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:00.174 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:00.174 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:00.174 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.174 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:00.174 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.120387+0000", 2024-09-11T09:52:00.174 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:00.175 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.175 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.175 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:00.175 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.120387+0000", 2024-09-11T09:52:00.175 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:00.175 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.175 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.175 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:00.175 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.120389+0000", 2024-09-11T09:52:00.175 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.018e-06 2024-09-11T09:52:00.175 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.175 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.176 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:00.176 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.127403+0000", 2024-09-11T09:52:00.176 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0070144769999999999 2024-09-11T09:52:00.176 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.176 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.176 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:00.176 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.127414+0000", 2024-09-11T09:52:00.176 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0991e-05 2024-09-11T09:52:00.176 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.176 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.176 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:00.176 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.127423+0000", 2024-09-11T09:52:00.176 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.8580000000000006e-06 2024-09-11T09:52:00.177 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.177 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.177 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:00.177 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.127468+0000", 2024-09-11T09:52:00.177 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.4830999999999998e-05 2024-09-11T09:52:00.177 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.177 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.177 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:00.177 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.127485+0000", 2024-09-11T09:52:00.178 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6838999999999999e-05 2024-09-11T09:52:00.178 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.178 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.178 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:00.178 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.132341+0000", 2024-09-11T09:52:00.178 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0048557959999999999 2024-09-11T09:52:00.178 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.178 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.178 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:00.178 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.132375+0000", 2024-09-11T09:52:00.178 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.4657999999999999e-05 2024-09-11T09:52:00.178 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:00.178 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:00.179 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:00.179 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.179 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.179 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4694.0:46 2.3 e53/15 2:ce0e1469:ns::rbd_data.1256a0c02879.0000000000000008:head v 53'4, mlcod=16'1)", 2024-09-11T09:52:00.179 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.129008+0000", 2024-09-11T09:52:00.179 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 37.013552165999997, 2024-09-11T09:52:00.179 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0087151220000000005, 2024-09-11T09:52:00.179 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:00.179 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:00.179 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:00.179 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:00.179 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.180 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:00.180 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.129008+0000", 2024-09-11T09:52:00.180 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:00.180 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.180 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.180 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:00.180 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.129008+0000", 2024-09-11T09:52:00.180 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:00.180 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.180 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.180 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:00.180 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.129009+0000", 2024-09-11T09:52:00.181 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.367e-06 2024-09-11T09:52:00.181 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.181 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.181 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:00.181 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.132839+0000", 2024-09-11T09:52:00.181 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0038297610000000001 2024-09-11T09:52:00.181 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.181 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.181 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:00.181 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.132849+0000", 2024-09-11T09:52:00.181 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0672e-05 2024-09-11T09:52:00.181 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.181 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.182 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:00.182 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.132856+0000", 2024-09-11T09:52:00.182 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.1380000000000004e-06 2024-09-11T09:52:00.182 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.182 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.182 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:00.182 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.132891+0000", 2024-09-11T09:52:00.182 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.5712e-05 2024-09-11T09:52:00.182 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.182 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.182 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:00.182 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.132907+0000", 2024-09-11T09:52:00.182 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6166000000000001e-05 2024-09-11T09:52:00.183 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.183 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.183 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:00.183 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.137681+0000", 2024-09-11T09:52:00.183 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0047735299999999998 2024-09-11T09:52:00.183 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.183 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.183 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:00.183 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.137723+0000", 2024-09-11T09:52:00.183 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.1776000000000001e-05 2024-09-11T09:52:00.183 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:00.183 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:00.183 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:00.184 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.184 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.184 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4694.0:48 2.3 e53/15 2:d2090b70:ns::rbd_data.1256a0c02879.0000000000000009:head v 53'5, mlcod=53'2)", 2024-09-11T09:52:00.184 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.135108+0000", 2024-09-11T09:52:00.184 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 37.007451269000001, 2024-09-11T09:52:00.184 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.010710679000000001, 2024-09-11T09:52:00.184 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:00.184 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:00.184 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:00.184 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:00.184 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.184 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:00.185 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.135108+0000", 2024-09-11T09:52:00.185 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:00.185 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.185 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.185 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:00.185 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.135108+0000", 2024-09-11T09:52:00.185 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:00.185 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.185 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.185 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:00.185 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.135110+0000", 2024-09-11T09:52:00.185 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.122e-06 2024-09-11T09:52:00.185 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.186 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.186 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:00.186 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.140876+0000", 2024-09-11T09:52:00.186 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.005766162 2024-09-11T09:52:00.186 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.186 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.186 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:00.186 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.140888+0000", 2024-09-11T09:52:00.186 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.201e-05 2024-09-11T09:52:00.186 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.186 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.186 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:00.186 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.140894+0000", 2024-09-11T09:52:00.186 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.1739999999999997e-06 2024-09-11T09:52:00.186 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.187 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.187 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:00.187 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.140971+0000", 2024-09-11T09:52:00.187 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.7495000000000003e-05 2024-09-11T09:52:00.187 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.187 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.187 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:00.187 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.140995+0000", 2024-09-11T09:52:00.187 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.3204999999999999e-05 2024-09-11T09:52:00.187 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.187 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.187 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:00.187 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.145757+0000", 2024-09-11T09:52:00.188 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0047627190000000003 2024-09-11T09:52:00.188 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.188 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.188 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:00.188 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.145819+0000", 2024-09-11T09:52:00.188 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.1791999999999994e-05 2024-09-11T09:52:00.188 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:00.188 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:00.188 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:00.188 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.188 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.188 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4694.0:50 2.1 2:83078786:ns::rbd_data.1256a0c02879.000000000000000a:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:00.189 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.141467+0000", 2024-09-11T09:52:00.189 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 37.001092544999999, 2024-09-11T09:52:00.189 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.012785889, 2024-09-11T09:52:00.189 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:00.189 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:00.189 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:00.189 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:00.189 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4694", 2024-09-11T09:52:00.189 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:52:00.189 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 50 2024-09-11T09:52:00.189 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.189 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:00.189 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.190 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:00.190 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.141467+0000", 2024-09-11T09:52:00.190 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:00.190 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.190 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.190 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:00.190 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.141467+0000", 2024-09-11T09:52:00.190 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:00.190 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.190 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.190 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:00.190 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.141468+0000", 2024-09-11T09:52:00.191 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1009999999999999e-06 2024-09-11T09:52:00.191 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.191 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.191 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:00.191 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.146432+0000", 2024-09-11T09:52:00.191 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0049640539999999999 2024-09-11T09:52:00.191 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.191 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.191 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:00.191 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.146446+0000", 2024-09-11T09:52:00.191 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3254e-05 2024-09-11T09:52:00.191 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.191 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.192 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:00.192 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.146452+0000", 2024-09-11T09:52:00.192 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.207e-06 2024-09-11T09:52:00.192 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.192 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.192 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:00.192 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.146510+0000", 2024-09-11T09:52:00.192 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.8322999999999997e-05 2024-09-11T09:52:00.192 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.192 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.192 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:00.192 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.146650+0000", 2024-09-11T09:52:00.192 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00013947100000000001 2024-09-11T09:52:00.193 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.193 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.193 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1", 2024-09-11T09:52:00.193 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.146779+0000", 2024-09-11T09:52:00.193 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000129059 2024-09-11T09:52:00.193 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.193 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.193 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:00.193 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.153778+0000", 2024-09-11T09:52:00.193 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0069997130000000003 2024-09-11T09:52:00.193 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.193 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.193 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:00.194 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.154180+0000", 2024-09-11T09:52:00.194 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00040159800000000003 2024-09-11T09:52:00.194 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.194 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.194 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:00.194 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.154218+0000", 2024-09-11T09:52:00.194 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.7795e-05 2024-09-11T09:52:00.194 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.194 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.194 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:00.195 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.154253+0000", 2024-09-11T09:52:00.195 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.5314000000000001e-05 2024-09-11T09:52:00.195 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:00.195 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:00.195 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:00.195 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.195 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.195 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4694.0:54 2.3 e53/15 2:c44f0253:ns::rbd_data.1256a0c02879.000000000000000b:head v 53'6, mlcod=53'4)", 2024-09-11T09:52:00.195 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.154926+0000", 2024-09-11T09:52:00.196 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 36.98763417, 2024-09-11T09:52:00.196 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0089497099999999996, 2024-09-11T09:52:00.196 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:00.196 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:00.196 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:00.196 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:00.196 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.196 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:00.196 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.154926+0000", 2024-09-11T09:52:00.196 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:00.196 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.196 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.196 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:00.197 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.154926+0000", 2024-09-11T09:52:00.197 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:00.197 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.197 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.197 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:00.197 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.154927+0000", 2024-09-11T09:52:00.197 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.7000000000000003e-07 2024-09-11T09:52:00.197 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.197 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.197 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:00.197 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.158712+0000", 2024-09-11T09:52:00.197 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0037855469999999998 2024-09-11T09:52:00.198 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.198 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.198 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:00.198 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.158726+0000", 2024-09-11T09:52:00.198 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3621000000000001e-05 2024-09-11T09:52:00.198 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.198 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.198 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:00.198 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.158732+0000", 2024-09-11T09:52:00.198 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.9089999999999997e-06 2024-09-11T09:52:00.198 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.198 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.198 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:00.198 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.158781+0000", 2024-09-11T09:52:00.199 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.8900000000000003e-05 2024-09-11T09:52:00.199 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.199 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.199 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:00.199 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.158805+0000", 2024-09-11T09:52:00.199 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.44e-05 2024-09-11T09:52:00.199 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.199 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.199 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:00.199 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.163826+0000", 2024-09-11T09:52:00.199 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0050206399999999998 2024-09-11T09:52:00.199 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.199 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.200 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:00.200 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.163875+0000", 2024-09-11T09:52:00.200 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.9722999999999998e-05 2024-09-11T09:52:00.200 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:00.200 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:00.200 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:00.200 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.200 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.200 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4694.0:56 2.3 e53/15 2:c462f029:ns::rbd_data.1256a0c02879.000000000000000c:head v 53'7, mlcod=53'4)", 2024-09-11T09:52:00.200 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.159549+0000", 2024-09-11T09:52:00.200 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 36.983010464000003, 2024-09-11T09:52:00.200 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.015591666000000001, 2024-09-11T09:52:00.200 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:00.201 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:00.201 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:00.201 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:00.201 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.201 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:00.201 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.159549+0000", 2024-09-11T09:52:00.201 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:00.201 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.201 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.201 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:00.201 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.159549+0000", 2024-09-11T09:52:00.201 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:00.201 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.202 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.202 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:00.202 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.159550+0000", 2024-09-11T09:52:00.202 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.1500000000000004e-07 2024-09-11T09:52:00.202 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.202 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.202 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:00.202 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.167834+0000", 2024-09-11T09:52:00.202 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0082840220000000003 2024-09-11T09:52:00.202 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.202 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.202 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:00.202 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.167857+0000", 2024-09-11T09:52:00.203 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.3337e-05 2024-09-11T09:52:00.203 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.203 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.203 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:00.203 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.167869+0000", 2024-09-11T09:52:00.203 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1556000000000001e-05 2024-09-11T09:52:00.203 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.203 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.203 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:00.203 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.167939+0000", 2024-09-11T09:52:00.203 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.0674999999999997e-05 2024-09-11T09:52:00.203 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.203 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.204 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:00.204 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.167967+0000", 2024-09-11T09:52:00.204 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.7322999999999999e-05 2024-09-11T09:52:00.204 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.204 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.204 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:00.204 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.175079+0000", 2024-09-11T09:52:00.204 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0071126150000000001 2024-09-11T09:52:00.204 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.204 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.204 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:00.204 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.175141+0000", 2024-09-11T09:52:00.204 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.1523e-05 2024-09-11T09:52:00.205 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:00.205 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:00.205 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:00.205 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.205 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.205 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.5314.0:31 2.1 2:8654721b:::rbd_data.1250f5e25b80.0000000000000000:head [write 393216~131072] snapc 3=[3] ondisk+write+known_if_redirected+supports_pool_eio e68)", 2024-09-11T09:52:00.205 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:50.230850+0000", 2024-09-11T09:52:00.205 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 9.9117097029999996, 2024-09-11T09:52:00.205 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0073772539999999998, 2024-09-11T09:52:00.206 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:00.206 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:00.206 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:00.206 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:00.206 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.5314", 2024-09-11T09:52:00.206 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3853995732", 2024-09-11T09:52:00.206 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 31 2024-09-11T09:52:00.206 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.206 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:00.206 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.206 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:00.206 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.230850+0000", 2024-09-11T09:52:00.206 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:00.207 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.207 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.207 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:00.207 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.230850+0000", 2024-09-11T09:52:00.207 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:00.207 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.207 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.207 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:00.207 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.230850+0000", 2024-09-11T09:52:00.207 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.32e-07 2024-09-11T09:52:00.207 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.207 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.208 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:00.208 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.230908+0000", 2024-09-11T09:52:00.208 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.7698000000000002e-05 2024-09-11T09:52:00.208 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.208 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.208 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:00.208 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.230915+0000", 2024-09-11T09:52:00.208 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.81e-06 2024-09-11T09:52:00.208 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.208 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.208 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:00.209 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.230920+0000", 2024-09-11T09:52:00.209 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.9660000000000004e-06 2024-09-11T09:52:00.209 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.209 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.209 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:00.209 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.235055+0000", 2024-09-11T09:52:00.209 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0041354629999999998 2024-09-11T09:52:00.209 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.209 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.209 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:00.209 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.235145+0000", 2024-09-11T09:52:00.209 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.9362000000000003e-05 2024-09-11T09:52:00.209 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.209 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.210 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1", 2024-09-11T09:52:00.210 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.235282+0000", 2024-09-11T09:52:00.210 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00013710999999999999 2024-09-11T09:52:00.210 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.210 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.210 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:00.210 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.237335+0000", 2024-09-11T09:52:00.210 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.002053165 2024-09-11T09:52:00.210 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.210 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.210 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:00.210 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.238106+0000", 2024-09-11T09:52:00.210 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000771015 2024-09-11T09:52:00.211 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.211 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.211 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:00.211 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.238165+0000", 2024-09-11T09:52:00.211 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.8588000000000002e-05 2024-09-11T09:52:00.211 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.211 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.211 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:00.211 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.238227+0000", 2024-09-11T09:52:00.211 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.2644999999999994e-05 2024-09-11T09:52:00.211 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:00.211 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:00.211 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:00.212 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.212 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.212 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.5314.0:32 2.1 2:8654721b:::rbd_data.1250f5e25b80.0000000000000000:head [write 524288~131072] snapc 3=[3] ondisk+write+known_if_redirected+supports_pool_eio e68)", 2024-09-11T09:52:00.212 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:50.230932+0000", 2024-09-11T09:52:00.212 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 9.9116280779999997, 2024-09-11T09:52:00.212 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0074882129999999996, 2024-09-11T09:52:00.212 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:00.212 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:00.212 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:00.212 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:00.212 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.5314", 2024-09-11T09:52:00.212 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3853995732", 2024-09-11T09:52:00.212 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 32 2024-09-11T09:52:00.213 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.213 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:00.213 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.213 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:00.213 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.230932+0000", 2024-09-11T09:52:00.213 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:00.213 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.213 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.213 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:00.213 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.230932+0000", 2024-09-11T09:52:00.213 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:00.213 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.213 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.214 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:00.214 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.230932+0000", 2024-09-11T09:52:00.214 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.9999999999999999e-07 2024-09-11T09:52:00.214 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.214 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.214 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:00.214 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.231022+0000", 2024-09-11T09:52:00.214 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.0439999999999995e-05 2024-09-11T09:52:00.214 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.214 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.214 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:00.214 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.231030+0000", 2024-09-11T09:52:00.214 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.3350000000000001e-06 2024-09-11T09:52:00.214 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.215 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.215 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:00.215 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.231034+0000", 2024-09-11T09:52:00.215 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.9759999999999997e-06 2024-09-11T09:52:00.215 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.215 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.215 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:00.215 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.235837+0000", 2024-09-11T09:52:00.215 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0048031910000000001 2024-09-11T09:52:00.215 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.215 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.215 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:00.215 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.235904+0000", 2024-09-11T09:52:00.215 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.7181999999999995e-05 2024-09-11T09:52:00.216 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.216 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.216 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1", 2024-09-11T09:52:00.216 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.236040+0000", 2024-09-11T09:52:00.216 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00013545199999999999 2024-09-11T09:52:00.216 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.216 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.216 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:00.216 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.238242+0000", 2024-09-11T09:52:00.216 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.002202809 2024-09-11T09:52:00.216 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.216 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.216 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:00.217 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.238301+0000", 2024-09-11T09:52:00.217 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.8573999999999999e-05 2024-09-11T09:52:00.217 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.217 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.217 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:00.217 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.238378+0000", 2024-09-11T09:52:00.217 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.7323999999999994e-05 2024-09-11T09:52:00.217 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.217 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.217 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:00.217 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.238420+0000", 2024-09-11T09:52:00.217 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.163e-05 2024-09-11T09:52:00.217 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:00.218 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:00.218 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:00.218 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.218 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.218 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.5314.0:33 2.1 2:8654721b:::rbd_data.1250f5e25b80.0000000000000000:head [write 655360~131072] snapc 3=[3] ondisk+write+known_if_redirected+supports_pool_eio e68)", 2024-09-11T09:52:00.218 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:50.231041+0000", 2024-09-11T09:52:00.218 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 9.9115189190000006, 2024-09-11T09:52:00.218 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0075627139999999999, 2024-09-11T09:52:00.218 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:00.218 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:00.218 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:00.218 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:00.219 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.5314", 2024-09-11T09:52:00.219 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3853995732", 2024-09-11T09:52:00.219 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 33 2024-09-11T09:52:00.219 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.219 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:00.219 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.219 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:00.219 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.231041+0000", 2024-09-11T09:52:00.219 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:00.219 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.219 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.219 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:00.220 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.231041+0000", 2024-09-11T09:52:00.220 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:00.220 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.220 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.220 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:00.220 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.231041+0000", 2024-09-11T09:52:00.220 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.9499999999999998e-07 2024-09-11T09:52:00.220 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.220 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.220 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:00.220 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.231111+0000", 2024-09-11T09:52:00.220 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.0166999999999994e-05 2024-09-11T09:52:00.220 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.221 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.221 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:00.221 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.231122+0000", 2024-09-11T09:52:00.221 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0518e-05 2024-09-11T09:52:00.221 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.221 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.221 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:00.221 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.231128+0000", 2024-09-11T09:52:00.221 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.3509999999999998e-06 2024-09-11T09:52:00.221 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.221 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.221 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:00.221 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.236622+0000", 2024-09-11T09:52:00.222 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0054934989999999998 2024-09-11T09:52:00.222 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.222 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.222 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:00.222 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.236719+0000", 2024-09-11T09:52:00.222 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.7497999999999994e-05 2024-09-11T09:52:00.222 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.222 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.222 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1", 2024-09-11T09:52:00.222 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.236854+0000", 2024-09-11T09:52:00.222 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000134535 2024-09-11T09:52:00.222 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.222 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.222 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:00.223 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.238491+0000", 2024-09-11T09:52:00.223 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0016368610000000001 2024-09-11T09:52:00.223 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.223 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.223 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:00.223 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.238504+0000", 2024-09-11T09:52:00.223 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3927999999999999e-05 2024-09-11T09:52:00.223 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.223 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.223 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:00.223 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.238561+0000", 2024-09-11T09:52:00.223 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.6029999999999997e-05 2024-09-11T09:52:00.223 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.224 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.224 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:00.224 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.238604+0000", 2024-09-11T09:52:00.224 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.3031999999999997e-05 2024-09-11T09:52:00.224 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:00.224 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:00.224 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:00.224 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.224 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.224 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.5314.0:34 2.1 2:8654721b:::rbd_data.1250f5e25b80.0000000000000000:head [write 786432~131072] snapc 3=[3] ondisk+write+known_if_redirected+supports_pool_eio e68)", 2024-09-11T09:52:00.224 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:50.231139+0000", 2024-09-11T09:52:00.224 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 9.9114203159999992, 2024-09-11T09:52:00.224 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0083689690000000004, 2024-09-11T09:52:00.225 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:00.225 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:00.225 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:00.225 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:00.225 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.5314", 2024-09-11T09:52:00.225 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3853995732", 2024-09-11T09:52:00.225 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 34 2024-09-11T09:52:00.225 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.225 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:00.225 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.225 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:00.225 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.231139+0000", 2024-09-11T09:52:00.225 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:00.226 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.226 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.226 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:00.226 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.231139+0000", 2024-09-11T09:52:00.226 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:00.226 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.226 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.226 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:00.226 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.231140+0000", 2024-09-11T09:52:00.226 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.1699999999999998e-07 2024-09-11T09:52:00.226 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.226 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.226 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:00.227 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.231247+0000", 2024-09-11T09:52:00.227 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000107088 2024-09-11T09:52:00.227 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.227 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.227 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:00.227 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.231260+0000", 2024-09-11T09:52:00.227 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.289e-05 2024-09-11T09:52:00.227 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.227 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.227 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:00.227 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.231272+0000", 2024-09-11T09:52:00.227 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2371999999999999e-05 2024-09-11T09:52:00.227 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.227 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.228 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:00.228 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.237380+0000", 2024-09-11T09:52:00.228 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.006107683 2024-09-11T09:52:00.228 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.228 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.228 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:00.228 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.237434+0000", 2024-09-11T09:52:00.228 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.4143e-05 2024-09-11T09:52:00.228 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.228 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.228 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1", 2024-09-11T09:52:00.228 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.237541+0000", 2024-09-11T09:52:00.228 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00010676800000000001 2024-09-11T09:52:00.228 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.229 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.229 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:00.229 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.239390+0000", 2024-09-11T09:52:00.229 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.001848861 2024-09-11T09:52:00.229 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.229 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.229 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:00.229 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.239409+0000", 2024-09-11T09:52:00.229 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.9505e-05 2024-09-11T09:52:00.229 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.229 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.229 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:00.229 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.239463+0000", 2024-09-11T09:52:00.229 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.3838000000000002e-05 2024-09-11T09:52:00.230 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.230 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.230 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:00.230 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.239508+0000", 2024-09-11T09:52:00.230 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.5303999999999999e-05 2024-09-11T09:52:00.230 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:00.230 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:00.230 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:00.230 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.230 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.230 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.5314.0:35 2.1 2:8654721b:::rbd_data.1250f5e25b80.0000000000000000:head [write 917504~131072] snapc 3=[3] ondisk+write+known_if_redirected+supports_pool_eio e68)", 2024-09-11T09:52:00.230 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:50.231287+0000", 2024-09-11T09:52:00.231 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 9.9112729819999998, 2024-09-11T09:52:00.231 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0086930730000000008, 2024-09-11T09:52:00.231 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:00.231 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:00.231 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:00.231 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:00.231 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.5314", 2024-09-11T09:52:00.231 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3853995732", 2024-09-11T09:52:00.231 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 35 2024-09-11T09:52:00.231 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.231 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:00.231 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.231 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:00.231 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.231287+0000", 2024-09-11T09:52:00.232 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:00.232 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.232 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.232 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:00.232 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.231287+0000", 2024-09-11T09:52:00.232 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:00.232 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.232 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.232 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:00.232 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.231287+0000", 2024-09-11T09:52:00.232 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.7700000000000004e-07 2024-09-11T09:52:00.232 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.232 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.233 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:00.233 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.231354+0000", 2024-09-11T09:52:00.233 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.6304999999999996e-05 2024-09-11T09:52:00.233 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.233 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.233 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:00.233 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.231367+0000", 2024-09-11T09:52:00.233 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2952999999999999e-05 2024-09-11T09:52:00.233 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.233 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.233 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:00.233 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.231374+0000", 2024-09-11T09:52:00.233 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.0469999999999996e-06 2024-09-11T09:52:00.233 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.234 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.234 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:00.234 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.238650+0000", 2024-09-11T09:52:00.234 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0072760280000000004 2024-09-11T09:52:00.234 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.234 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.234 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:00.234 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.238732+0000", 2024-09-11T09:52:00.234 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.2194000000000006e-05 2024-09-11T09:52:00.234 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.234 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.234 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1", 2024-09-11T09:52:00.234 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.238876+0000", 2024-09-11T09:52:00.234 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00014458 2024-09-11T09:52:00.235 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.235 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.235 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:00.235 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.239739+0000", 2024-09-11T09:52:00.235 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00086244499999999999 2024-09-11T09:52:00.235 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.235 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.235 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:00.235 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.239917+0000", 2024-09-11T09:52:00.235 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00017777299999999999 2024-09-11T09:52:00.235 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.235 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.235 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:00.236 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.239954+0000", 2024-09-11T09:52:00.236 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.7379999999999998e-05 2024-09-11T09:52:00.236 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.236 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.236 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:00.236 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.239980+0000", 2024-09-11T09:52:00.236 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.5791000000000001e-05 2024-09-11T09:52:00.236 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:00.236 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:00.236 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:00.236 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.236 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.236 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.5314.0:70 2.1 2:8654721b:::rbd_data.1250f5e25b80.0000000000000000:head [write 524288~131072] snapc 5=[5,3] ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-09-11T09:52:00.237 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:50.824556+0000", 2024-09-11T09:52:00.237 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 9.3180041809999992, 2024-09-11T09:52:00.237 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0072794469999999997, 2024-09-11T09:52:00.237 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:00.237 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:00.237 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:00.237 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:00.237 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.5314", 2024-09-11T09:52:00.237 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3853995732", 2024-09-11T09:52:00.237 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 70 2024-09-11T09:52:00.237 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.237 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:00.237 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.237 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:00.238 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.824556+0000", 2024-09-11T09:52:00.238 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:00.238 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.238 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.238 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:00.238 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.824556+0000", 2024-09-11T09:52:00.238 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:00.238 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.238 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.238 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:00.238 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.824564+0000", 2024-09-11T09:52:00.238 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.2409999999999995e-06 2024-09-11T09:52:00.238 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.238 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.238 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:00.239 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.824613+0000", 2024-09-11T09:52:00.239 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.9579000000000001e-05 2024-09-11T09:52:00.239 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.239 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.239 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:00.239 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.824638+0000", 2024-09-11T09:52:00.239 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.4510000000000001e-05 2024-09-11T09:52:00.239 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.239 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.239 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:00.239 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.824653+0000", 2024-09-11T09:52:00.239 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.4831000000000001e-05 2024-09-11T09:52:00.239 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.239 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.240 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:00.240 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.829072+0000", 2024-09-11T09:52:00.240 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0044195700000000003 2024-09-11T09:52:00.240 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.240 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.240 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:00.240 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.829166+0000", 2024-09-11T09:52:00.240 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.3212000000000001e-05 2024-09-11T09:52:00.240 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.240 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.240 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1", 2024-09-11T09:52:00.240 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.829309+0000", 2024-09-11T09:52:00.240 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00014301900000000001 2024-09-11T09:52:00.241 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.241 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.241 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:00.241 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.831074+0000", 2024-09-11T09:52:00.241 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.001765305 2024-09-11T09:52:00.241 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.241 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.241 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:00.241 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.831751+0000", 2024-09-11T09:52:00.241 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00067694000000000001 2024-09-11T09:52:00.241 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.241 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.242 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:00.242 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.831803+0000", 2024-09-11T09:52:00.242 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.2441000000000002e-05 2024-09-11T09:52:00.242 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.242 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.242 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:00.242 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.831835+0000", 2024-09-11T09:52:00.242 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.1798999999999998e-05 2024-09-11T09:52:00.242 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:00.242 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:00.242 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:00.242 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.243 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.243 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.5314.0:71 2.1 2:8654721b:::rbd_data.1250f5e25b80.0000000000000000:head [write 655360~131072] snapc 5=[5,3] ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-09-11T09:52:00.243 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:50.824662+0000", 2024-09-11T09:52:00.243 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 9.3178980649999996, 2024-09-11T09:52:00.243 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0082172219999999997, 2024-09-11T09:52:00.243 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:00.243 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:00.243 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:00.243 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:00.243 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.5314", 2024-09-11T09:52:00.243 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3853995732", 2024-09-11T09:52:00.243 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 71 2024-09-11T09:52:00.243 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.243 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:00.244 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.244 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:00.244 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.824662+0000", 2024-09-11T09:52:00.244 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:00.244 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.244 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.244 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:00.244 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.824662+0000", 2024-09-11T09:52:00.244 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:00.244 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.244 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.244 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:00.244 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.824662+0000", 2024-09-11T09:52:00.245 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.0699999999999998e-07 2024-09-11T09:52:00.245 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.245 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.245 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:00.245 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.824859+0000", 2024-09-11T09:52:00.245 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00019725799999999999 2024-09-11T09:52:00.245 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.245 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.245 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:00.245 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.824906+0000", 2024-09-11T09:52:00.245 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.7162e-05 2024-09-11T09:52:00.245 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.245 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.246 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:00.246 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.824926+0000", 2024-09-11T09:52:00.246 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.0038999999999999e-05 2024-09-11T09:52:00.246 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.246 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.246 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:00.246 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.830313+0000", 2024-09-11T09:52:00.246 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0053868409999999999 2024-09-11T09:52:00.246 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.246 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.246 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:00.246 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.830359+0000", 2024-09-11T09:52:00.246 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.5370000000000001e-05 2024-09-11T09:52:00.246 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.247 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.247 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1", 2024-09-11T09:52:00.247 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.830460+0000", 2024-09-11T09:52:00.247 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00010171299999999999 2024-09-11T09:52:00.247 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.247 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.247 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:00.247 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.831843+0000", 2024-09-11T09:52:00.247 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.001382501 2024-09-11T09:52:00.247 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.247 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.247 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:00.247 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.832769+0000", 2024-09-11T09:52:00.247 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00092638099999999999 2024-09-11T09:52:00.248 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.248 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.248 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:00.248 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.832835+0000", 2024-09-11T09:52:00.248 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.6106999999999996e-05 2024-09-11T09:52:00.248 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.248 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.248 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:00.248 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.832879+0000", 2024-09-11T09:52:00.248 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.3543e-05 2024-09-11T09:52:00.248 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:00.248 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:00.248 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:00.249 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.249 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.249 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.5314.0:72 2.1 2:8654721b:::rbd_data.1250f5e25b80.0000000000000000:head [write 786432~131072] snapc 5=[5,3] ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-09-11T09:52:00.249 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:50.824938+0000", 2024-09-11T09:52:00.249 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 9.317621763, 2024-09-11T09:52:00.249 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0082389799999999999, 2024-09-11T09:52:00.249 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:00.249 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:00.249 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:00.249 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:00.249 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.5314", 2024-09-11T09:52:00.249 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3853995732", 2024-09-11T09:52:00.249 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 72 2024-09-11T09:52:00.250 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.250 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:00.250 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.250 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:00.250 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.824938+0000", 2024-09-11T09:52:00.250 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:00.250 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.250 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.250 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:00.250 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.824938+0000", 2024-09-11T09:52:00.250 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:00.250 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.250 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.250 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:00.251 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.824939+0000", 2024-09-11T09:52:00.251 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.6199999999999997e-07 2024-09-11T09:52:00.251 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.251 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.251 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:00.251 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.825028+0000", 2024-09-11T09:52:00.251 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.9105999999999997e-05 2024-09-11T09:52:00.251 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.251 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.251 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:00.251 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.825042+0000", 2024-09-11T09:52:00.251 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3760999999999999e-05 2024-09-11T09:52:00.251 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.252 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.252 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:00.252 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.825061+0000", 2024-09-11T09:52:00.252 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.9088000000000001e-05 2024-09-11T09:52:00.252 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.252 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.252 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:00.252 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.831122+0000", 2024-09-11T09:52:00.252 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.006061299 2024-09-11T09:52:00.252 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.252 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.252 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:00.252 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.831195+0000", 2024-09-11T09:52:00.252 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.2957e-05 2024-09-11T09:52:00.252 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.253 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.253 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1", 2024-09-11T09:52:00.253 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.831301+0000", 2024-09-11T09:52:00.253 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000106049 2024-09-11T09:52:00.253 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.253 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.253 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:00.253 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.832893+0000", 2024-09-11T09:52:00.253 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0015923770000000001 2024-09-11T09:52:00.253 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.253 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.253 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:00.253 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.833076+0000", 2024-09-11T09:52:00.253 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000182855 2024-09-11T09:52:00.254 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.254 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.254 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:00.255 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.833137+0000", 2024-09-11T09:52:00.255 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.1193999999999997e-05 2024-09-11T09:52:00.256 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.256 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.256 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:00.256 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.833177+0000", 2024-09-11T09:52:00.256 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.9632000000000003e-05 2024-09-11T09:52:00.256 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:00.256 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:00.256 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:00.256 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.256 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.256 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.5314.0:73 2.1 2:8654721b:::rbd_data.1250f5e25b80.0000000000000000:head [write 917504~131072] snapc 5=[5,3] ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-09-11T09:52:00.256 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:50.825071+0000", 2024-09-11T09:52:00.257 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 9.3174882560000007, 2024-09-11T09:52:00.257 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0084039160000000009, 2024-09-11T09:52:00.257 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:00.257 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:00.257 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:00.257 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:00.257 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.5314", 2024-09-11T09:52:00.257 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3853995732", 2024-09-11T09:52:00.257 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 73 2024-09-11T09:52:00.257 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.257 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:00.257 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.257 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:00.257 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.825071+0000", 2024-09-11T09:52:00.258 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:00.258 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.258 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.258 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:00.258 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.825071+0000", 2024-09-11T09:52:00.258 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:00.258 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.258 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.258 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:00.258 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.825072+0000", 2024-09-11T09:52:00.258 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.5499999999999998e-07 2024-09-11T09:52:00.258 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.258 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.258 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:00.258 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.825124+0000", 2024-09-11T09:52:00.259 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.2197000000000002e-05 2024-09-11T09:52:00.259 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.259 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.259 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:00.259 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.825132+0000", 2024-09-11T09:52:00.259 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.0290000000000005e-06 2024-09-11T09:52:00.259 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.259 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.259 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:00.259 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.825136+0000", 2024-09-11T09:52:00.259 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.9380000000000002e-06 2024-09-11T09:52:00.259 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.259 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.259 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:00.260 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.832001+0000", 2024-09-11T09:52:00.260 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0068650949999999999 2024-09-11T09:52:00.260 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.260 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.260 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:00.260 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.832091+0000", 2024-09-11T09:52:00.260 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.9524999999999994e-05 2024-09-11T09:52:00.260 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.260 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.260 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1", 2024-09-11T09:52:00.260 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.832234+0000", 2024-09-11T09:52:00.260 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000142783 2024-09-11T09:52:00.260 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.261 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.261 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:00.261 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.833198+0000", 2024-09-11T09:52:00.261 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00096476899999999998 2024-09-11T09:52:00.261 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.261 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.261 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:00.261 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.833386+0000", 2024-09-11T09:52:00.261 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00018772 2024-09-11T09:52:00.261 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.261 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.261 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:00.261 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.833440+0000", 2024-09-11T09:52:00.261 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.4228999999999999e-05 2024-09-11T09:52:00.262 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.262 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.262 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:00.262 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.833475+0000", 2024-09-11T09:52:00.262 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.5076000000000001e-05 2024-09-11T09:52:00.262 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:00.262 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:00.262 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:00.262 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:00.262 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:00.263 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:52:00.264 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-11T09:52:00.265 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-11T09:52:00.265 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:52:00.265 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 74, 2024-09-11T09:52:00.265 INFO:teuthology.orchestra.run.smithi017.stdout: "fsid": "a2f1daec-d3b0-4bf5-82c8-72ecbb93d4e4", 2024-09-11T09:52:00.265 INFO:teuthology.orchestra.run.smithi017.stdout: "created": "2024-09-11T09:50:00.165804+0000", 2024-09-11T09:52:00.265 INFO:teuthology.orchestra.run.smithi017.stdout: "modified": "2024-09-11T09:51:58.805905+0000", 2024-09-11T09:52:00.266 INFO:teuthology.orchestra.run.smithi017.stdout: "last_up_change": "2024-09-11T09:51:44.764990+0000", 2024-09-11T09:52:00.266 INFO:teuthology.orchestra.run.smithi017.stdout: "last_in_change": "2024-09-11T09:50:45.581301+0000", 2024-09-11T09:52:00.266 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": "nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-11T09:52:00.266 INFO:teuthology.orchestra.run.smithi017.stdout: "flags_num": 5804032, 2024-09-11T09:52:00.266 INFO:teuthology.orchestra.run.smithi017.stdout: "flags_set": [ 2024-09-11T09:52:00.266 INFO:teuthology.orchestra.run.smithi017.stdout: "nodeep-scrub", 2024-09-11T09:52:00.266 INFO:teuthology.orchestra.run.smithi017.stdout: "pglog_hardlimit", 2024-09-11T09:52:00.266 INFO:teuthology.orchestra.run.smithi017.stdout: "purged_snapdirs", 2024-09-11T09:52:00.266 INFO:teuthology.orchestra.run.smithi017.stdout: "recovery_deletes", 2024-09-11T09:52:00.266 INFO:teuthology.orchestra.run.smithi017.stdout: "sortbitwise" 2024-09-11T09:52:00.266 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-11T09:52:00.266 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_version": 6, 2024-09-11T09:52:00.266 INFO:teuthology.orchestra.run.smithi017.stdout: "full_ratio": 0.94999998807907104, 2024-09-11T09:52:00.267 INFO:teuthology.orchestra.run.smithi017.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-11T09:52:00.267 INFO:teuthology.orchestra.run.smithi017.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-11T09:52:00.267 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_snapshot": "", 2024-09-11T09:52:00.267 INFO:teuthology.orchestra.run.smithi017.stdout: "pool_max": 3, 2024-09-11T09:52:00.267 INFO:teuthology.orchestra.run.smithi017.stdout: "max_osd": 6, 2024-09-11T09:52:00.267 INFO:teuthology.orchestra.run.smithi017.stdout: "require_min_compat_client": "luminous", 2024-09-11T09:52:00.267 INFO:teuthology.orchestra.run.smithi017.stdout: "min_compat_client": "luminous", 2024-09-11T09:52:00.267 INFO:teuthology.orchestra.run.smithi017.stdout: "require_osd_release": "squid", 2024-09-11T09:52:00.267 INFO:teuthology.orchestra.run.smithi017.stdout: "allow_crimson": false, 2024-09-11T09:52:00.267 INFO:teuthology.orchestra.run.smithi017.stdout: "pools": [ 2024-09-11T09:52:00.267 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.267 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 1, 2024-09-11T09:52:00.267 INFO:teuthology.orchestra.run.smithi017.stdout: "pool_name": ".mgr", 2024-09-11T09:52:00.268 INFO:teuthology.orchestra.run.smithi017.stdout: "create_time": "2024-09-11T09:50:10.457184+0000", 2024-09-11T09:52:00.268 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 1, 2024-09-11T09:52:00.268 INFO:teuthology.orchestra.run.smithi017.stdout: "flags_names": "hashpspool", 2024-09-11T09:52:00.268 INFO:teuthology.orchestra.run.smithi017.stdout: "type": 1, 2024-09-11T09:52:00.268 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 2, 2024-09-11T09:52:00.268 INFO:teuthology.orchestra.run.smithi017.stdout: "min_size": 1, 2024-09-11T09:52:00.268 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_rule": 0, 2024-09-11T09:52:00.268 INFO:teuthology.orchestra.run.smithi017.stdout: "peering_crush_bucket_count": 0, 2024-09-11T09:52:00.268 INFO:teuthology.orchestra.run.smithi017.stdout: "peering_crush_bucket_target": 0, 2024-09-11T09:52:00.268 INFO:teuthology.orchestra.run.smithi017.stdout: "peering_crush_bucket_barrier": 0, 2024-09-11T09:52:00.268 INFO:teuthology.orchestra.run.smithi017.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-11T09:52:00.268 INFO:teuthology.orchestra.run.smithi017.stdout: "is_stretch_pool": false, 2024-09-11T09:52:00.268 INFO:teuthology.orchestra.run.smithi017.stdout: "object_hash": 2, 2024-09-11T09:52:00.269 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_autoscale_mode": "off", 2024-09-11T09:52:00.269 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_num": 1, 2024-09-11T09:52:00.269 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_placement_num": 1, 2024-09-11T09:52:00.269 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_placement_num_target": 1, 2024-09-11T09:52:00.269 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_num_target": 1, 2024-09-11T09:52:00.269 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_num_pending": 1, 2024-09-11T09:52:00.269 INFO:teuthology.orchestra.run.smithi017.stdout: "last_pg_merge_meta": { 2024-09-11T09:52:00.269 INFO:teuthology.orchestra.run.smithi017.stdout: "source_pgid": "0.0", 2024-09-11T09:52:00.269 INFO:teuthology.orchestra.run.smithi017.stdout: "ready_epoch": 0, 2024-09-11T09:52:00.269 INFO:teuthology.orchestra.run.smithi017.stdout: "last_epoch_started": 0, 2024-09-11T09:52:00.269 INFO:teuthology.orchestra.run.smithi017.stdout: "last_epoch_clean": 0, 2024-09-11T09:52:00.269 INFO:teuthology.orchestra.run.smithi017.stdout: "source_version": "0'0", 2024-09-11T09:52:00.269 INFO:teuthology.orchestra.run.smithi017.stdout: "target_version": "0'0" 2024-09-11T09:52:00.269 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.270 INFO:teuthology.orchestra.run.smithi017.stdout: "last_change": "14", 2024-09-11T09:52:00.270 INFO:teuthology.orchestra.run.smithi017.stdout: "last_force_op_resend": "0", 2024-09-11T09:52:00.270 INFO:teuthology.orchestra.run.smithi017.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-11T09:52:00.270 INFO:teuthology.orchestra.run.smithi017.stdout: "last_force_op_resend_preluminous": "0", 2024-09-11T09:52:00.270 INFO:teuthology.orchestra.run.smithi017.stdout: "auid": 0, 2024-09-11T09:52:00.270 INFO:teuthology.orchestra.run.smithi017.stdout: "snap_mode": "selfmanaged", 2024-09-11T09:52:00.270 INFO:teuthology.orchestra.run.smithi017.stdout: "snap_seq": 0, 2024-09-11T09:52:00.270 INFO:teuthology.orchestra.run.smithi017.stdout: "snap_epoch": 0, 2024-09-11T09:52:00.270 INFO:teuthology.orchestra.run.smithi017.stdout: "pool_snaps": [], 2024-09-11T09:52:00.270 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_snaps": "[]", 2024-09-11T09:52:00.270 INFO:teuthology.orchestra.run.smithi017.stdout: "quota_max_bytes": 0, 2024-09-11T09:52:00.270 INFO:teuthology.orchestra.run.smithi017.stdout: "quota_max_objects": 0, 2024-09-11T09:52:00.270 INFO:teuthology.orchestra.run.smithi017.stdout: "tiers": [], 2024-09-11T09:52:00.270 INFO:teuthology.orchestra.run.smithi017.stdout: "tier_of": -1, 2024-09-11T09:52:00.271 INFO:teuthology.orchestra.run.smithi017.stdout: "read_tier": -1, 2024-09-11T09:52:00.271 INFO:teuthology.orchestra.run.smithi017.stdout: "write_tier": -1, 2024-09-11T09:52:00.271 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_mode": "none", 2024-09-11T09:52:00.271 INFO:teuthology.orchestra.run.smithi017.stdout: "target_max_bytes": 0, 2024-09-11T09:52:00.271 INFO:teuthology.orchestra.run.smithi017.stdout: "target_max_objects": 0, 2024-09-11T09:52:00.271 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-11T09:52:00.271 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-11T09:52:00.271 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-11T09:52:00.271 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_min_flush_age": 0, 2024-09-11T09:52:00.271 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_min_evict_age": 0, 2024-09-11T09:52:00.271 INFO:teuthology.orchestra.run.smithi017.stdout: "erasure_code_profile": "", 2024-09-11T09:52:00.271 INFO:teuthology.orchestra.run.smithi017.stdout: "hit_set_params": { 2024-09-11T09:52:00.271 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "none" 2024-09-11T09:52:00.271 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.272 INFO:teuthology.orchestra.run.smithi017.stdout: "hit_set_period": 0, 2024-09-11T09:52:00.272 INFO:teuthology.orchestra.run.smithi017.stdout: "hit_set_count": 0, 2024-09-11T09:52:00.272 INFO:teuthology.orchestra.run.smithi017.stdout: "use_gmt_hitset": true, 2024-09-11T09:52:00.272 INFO:teuthology.orchestra.run.smithi017.stdout: "min_read_recency_for_promote": 0, 2024-09-11T09:52:00.272 INFO:teuthology.orchestra.run.smithi017.stdout: "min_write_recency_for_promote": 0, 2024-09-11T09:52:00.272 INFO:teuthology.orchestra.run.smithi017.stdout: "hit_set_grade_decay_rate": 0, 2024-09-11T09:52:00.272 INFO:teuthology.orchestra.run.smithi017.stdout: "hit_set_search_last_n": 0, 2024-09-11T09:52:00.272 INFO:teuthology.orchestra.run.smithi017.stdout: "grade_table": [], 2024-09-11T09:52:00.272 INFO:teuthology.orchestra.run.smithi017.stdout: "stripe_width": 0, 2024-09-11T09:52:00.272 INFO:teuthology.orchestra.run.smithi017.stdout: "expected_num_objects": 0, 2024-09-11T09:52:00.272 INFO:teuthology.orchestra.run.smithi017.stdout: "fast_read": false, 2024-09-11T09:52:00.272 INFO:teuthology.orchestra.run.smithi017.stdout: "options": { 2024-09-11T09:52:00.272 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_num_max": 32, 2024-09-11T09:52:00.273 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_num_min": 1 2024-09-11T09:52:00.273 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.273 INFO:teuthology.orchestra.run.smithi017.stdout: "application_metadata": { 2024-09-11T09:52:00.273 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr": {} 2024-09-11T09:52:00.273 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.273 INFO:teuthology.orchestra.run.smithi017.stdout: "read_balance": { 2024-09-11T09:52:00.273 INFO:teuthology.orchestra.run.smithi017.stdout: "score_type": "Fair distribution", 2024-09-11T09:52:00.273 INFO:teuthology.orchestra.run.smithi017.stdout: "score_acting": 6.059999942779541, 2024-09-11T09:52:00.273 INFO:teuthology.orchestra.run.smithi017.stdout: "score_stable": 6.059999942779541, 2024-09-11T09:52:00.273 INFO:teuthology.orchestra.run.smithi017.stdout: "optimal_score": 0.33000001311302185, 2024-09-11T09:52:00.273 INFO:teuthology.orchestra.run.smithi017.stdout: "raw_score_acting": 2, 2024-09-11T09:52:00.273 INFO:teuthology.orchestra.run.smithi017.stdout: "raw_score_stable": 2, 2024-09-11T09:52:00.273 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity_weighted": 1, 2024-09-11T09:52:00.274 INFO:teuthology.orchestra.run.smithi017.stdout: "average_primary_affinity": 1, 2024-09-11T09:52:00.274 INFO:teuthology.orchestra.run.smithi017.stdout: "average_primary_affinity_weighted": 1 2024-09-11T09:52:00.274 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:00.274 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.274 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.274 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 2, 2024-09-11T09:52:00.274 INFO:teuthology.orchestra.run.smithi017.stdout: "pool_name": "rbd", 2024-09-11T09:52:00.274 INFO:teuthology.orchestra.run.smithi017.stdout: "create_time": "2024-09-11T09:50:16.993202+0000", 2024-09-11T09:52:00.274 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 8193, 2024-09-11T09:52:00.274 INFO:teuthology.orchestra.run.smithi017.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-11T09:52:00.274 INFO:teuthology.orchestra.run.smithi017.stdout: "type": 1, 2024-09-11T09:52:00.274 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 2, 2024-09-11T09:52:00.274 INFO:teuthology.orchestra.run.smithi017.stdout: "min_size": 1, 2024-09-11T09:52:00.275 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_rule": 0, 2024-09-11T09:52:00.275 INFO:teuthology.orchestra.run.smithi017.stdout: "peering_crush_bucket_count": 0, 2024-09-11T09:52:00.275 INFO:teuthology.orchestra.run.smithi017.stdout: "peering_crush_bucket_target": 0, 2024-09-11T09:52:00.275 INFO:teuthology.orchestra.run.smithi017.stdout: "peering_crush_bucket_barrier": 0, 2024-09-11T09:52:00.275 INFO:teuthology.orchestra.run.smithi017.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-11T09:52:00.275 INFO:teuthology.orchestra.run.smithi017.stdout: "is_stretch_pool": false, 2024-09-11T09:52:00.275 INFO:teuthology.orchestra.run.smithi017.stdout: "object_hash": 2, 2024-09-11T09:52:00.275 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_autoscale_mode": "off", 2024-09-11T09:52:00.275 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_num": 8, 2024-09-11T09:52:00.275 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_placement_num": 8, 2024-09-11T09:52:00.275 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_placement_num_target": 8, 2024-09-11T09:52:00.275 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_num_target": 8, 2024-09-11T09:52:00.276 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_num_pending": 8, 2024-09-11T09:52:00.276 INFO:teuthology.orchestra.run.smithi017.stdout: "last_pg_merge_meta": { 2024-09-11T09:52:00.276 INFO:teuthology.orchestra.run.smithi017.stdout: "source_pgid": "0.0", 2024-09-11T09:52:00.276 INFO:teuthology.orchestra.run.smithi017.stdout: "ready_epoch": 0, 2024-09-11T09:52:00.276 INFO:teuthology.orchestra.run.smithi017.stdout: "last_epoch_started": 0, 2024-09-11T09:52:00.276 INFO:teuthology.orchestra.run.smithi017.stdout: "last_epoch_clean": 0, 2024-09-11T09:52:00.276 INFO:teuthology.orchestra.run.smithi017.stdout: "source_version": "0'0", 2024-09-11T09:52:00.276 INFO:teuthology.orchestra.run.smithi017.stdout: "target_version": "0'0" 2024-09-11T09:52:00.276 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.276 INFO:teuthology.orchestra.run.smithi017.stdout: "last_change": "74", 2024-09-11T09:52:00.276 INFO:teuthology.orchestra.run.smithi017.stdout: "last_force_op_resend": "0", 2024-09-11T09:52:00.276 INFO:teuthology.orchestra.run.smithi017.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-11T09:52:00.276 INFO:teuthology.orchestra.run.smithi017.stdout: "last_force_op_resend_preluminous": "0", 2024-09-11T09:52:00.277 INFO:teuthology.orchestra.run.smithi017.stdout: "auid": 0, 2024-09-11T09:52:00.277 INFO:teuthology.orchestra.run.smithi017.stdout: "snap_mode": "selfmanaged", 2024-09-11T09:52:00.277 INFO:teuthology.orchestra.run.smithi017.stdout: "snap_seq": 6, 2024-09-11T09:52:00.277 INFO:teuthology.orchestra.run.smithi017.stdout: "snap_epoch": 74, 2024-09-11T09:52:00.277 INFO:teuthology.orchestra.run.smithi017.stdout: "pool_snaps": [], 2024-09-11T09:52:00.277 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_snaps": "[]", 2024-09-11T09:52:00.277 INFO:teuthology.orchestra.run.smithi017.stdout: "quota_max_bytes": 0, 2024-09-11T09:52:00.277 INFO:teuthology.orchestra.run.smithi017.stdout: "quota_max_objects": 0, 2024-09-11T09:52:00.277 INFO:teuthology.orchestra.run.smithi017.stdout: "tiers": [], 2024-09-11T09:52:00.277 INFO:teuthology.orchestra.run.smithi017.stdout: "tier_of": -1, 2024-09-11T09:52:00.277 INFO:teuthology.orchestra.run.smithi017.stdout: "read_tier": -1, 2024-09-11T09:52:00.278 INFO:teuthology.orchestra.run.smithi017.stdout: "write_tier": -1, 2024-09-11T09:52:00.278 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_mode": "none", 2024-09-11T09:52:00.278 INFO:teuthology.orchestra.run.smithi017.stdout: "target_max_bytes": 0, 2024-09-11T09:52:00.278 INFO:teuthology.orchestra.run.smithi017.stdout: "target_max_objects": 0, 2024-09-11T09:52:00.278 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-11T09:52:00.278 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-11T09:52:00.278 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-11T09:52:00.278 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_min_flush_age": 0, 2024-09-11T09:52:00.278 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_min_evict_age": 0, 2024-09-11T09:52:00.278 INFO:teuthology.orchestra.run.smithi017.stdout: "erasure_code_profile": "", 2024-09-11T09:52:00.278 INFO:teuthology.orchestra.run.smithi017.stdout: "hit_set_params": { 2024-09-11T09:52:00.278 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "none" 2024-09-11T09:52:00.278 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.278 INFO:teuthology.orchestra.run.smithi017.stdout: "hit_set_period": 0, 2024-09-11T09:52:00.279 INFO:teuthology.orchestra.run.smithi017.stdout: "hit_set_count": 0, 2024-09-11T09:52:00.279 INFO:teuthology.orchestra.run.smithi017.stdout: "use_gmt_hitset": true, 2024-09-11T09:52:00.279 INFO:teuthology.orchestra.run.smithi017.stdout: "min_read_recency_for_promote": 0, 2024-09-11T09:52:00.279 INFO:teuthology.orchestra.run.smithi017.stdout: "min_write_recency_for_promote": 0, 2024-09-11T09:52:00.279 INFO:teuthology.orchestra.run.smithi017.stdout: "hit_set_grade_decay_rate": 0, 2024-09-11T09:52:00.279 INFO:teuthology.orchestra.run.smithi017.stdout: "hit_set_search_last_n": 0, 2024-09-11T09:52:00.279 INFO:teuthology.orchestra.run.smithi017.stdout: "grade_table": [], 2024-09-11T09:52:00.279 INFO:teuthology.orchestra.run.smithi017.stdout: "stripe_width": 0, 2024-09-11T09:52:00.279 INFO:teuthology.orchestra.run.smithi017.stdout: "expected_num_objects": 0, 2024-09-11T09:52:00.279 INFO:teuthology.orchestra.run.smithi017.stdout: "fast_read": false, 2024-09-11T09:52:00.279 INFO:teuthology.orchestra.run.smithi017.stdout: "options": {}, 2024-09-11T09:52:00.279 INFO:teuthology.orchestra.run.smithi017.stdout: "application_metadata": { 2024-09-11T09:52:00.279 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd": {} 2024-09-11T09:52:00.279 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.280 INFO:teuthology.orchestra.run.smithi017.stdout: "read_balance": { 2024-09-11T09:52:00.280 INFO:teuthology.orchestra.run.smithi017.stdout: "score_type": "Fair distribution", 2024-09-11T09:52:00.280 INFO:teuthology.orchestra.run.smithi017.stdout: "score_acting": 1.5099999904632568, 2024-09-11T09:52:00.280 INFO:teuthology.orchestra.run.smithi017.stdout: "score_stable": 1.5099999904632568, 2024-09-11T09:52:00.280 INFO:teuthology.orchestra.run.smithi017.stdout: "optimal_score": 0.82999998331069946, 2024-09-11T09:52:00.280 INFO:teuthology.orchestra.run.smithi017.stdout: "raw_score_acting": 1.25, 2024-09-11T09:52:00.280 INFO:teuthology.orchestra.run.smithi017.stdout: "raw_score_stable": 1.25, 2024-09-11T09:52:00.280 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity_weighted": 1, 2024-09-11T09:52:00.280 INFO:teuthology.orchestra.run.smithi017.stdout: "average_primary_affinity": 1, 2024-09-11T09:52:00.280 INFO:teuthology.orchestra.run.smithi017.stdout: "average_primary_affinity_weighted": 1 2024-09-11T09:52:00.280 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:00.280 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.280 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.281 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 3, 2024-09-11T09:52:00.281 INFO:teuthology.orchestra.run.smithi017.stdout: "pool_name": "new_default_pool27121", 2024-09-11T09:52:00.281 INFO:teuthology.orchestra.run.smithi017.stdout: "create_time": "2024-09-11T09:51:23.493016+0000", 2024-09-11T09:52:00.281 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 8193, 2024-09-11T09:52:00.281 INFO:teuthology.orchestra.run.smithi017.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-11T09:52:00.281 INFO:teuthology.orchestra.run.smithi017.stdout: "type": 1, 2024-09-11T09:52:00.281 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 2, 2024-09-11T09:52:00.281 INFO:teuthology.orchestra.run.smithi017.stdout: "min_size": 1, 2024-09-11T09:52:00.281 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_rule": 0, 2024-09-11T09:52:00.281 INFO:teuthology.orchestra.run.smithi017.stdout: "peering_crush_bucket_count": 0, 2024-09-11T09:52:00.281 INFO:teuthology.orchestra.run.smithi017.stdout: "peering_crush_bucket_target": 0, 2024-09-11T09:52:00.281 INFO:teuthology.orchestra.run.smithi017.stdout: "peering_crush_bucket_barrier": 0, 2024-09-11T09:52:00.281 INFO:teuthology.orchestra.run.smithi017.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-11T09:52:00.282 INFO:teuthology.orchestra.run.smithi017.stdout: "is_stretch_pool": false, 2024-09-11T09:52:00.282 INFO:teuthology.orchestra.run.smithi017.stdout: "object_hash": 2, 2024-09-11T09:52:00.282 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_autoscale_mode": "off", 2024-09-11T09:52:00.282 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_num": 8, 2024-09-11T09:52:00.282 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_placement_num": 8, 2024-09-11T09:52:00.282 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_placement_num_target": 8, 2024-09-11T09:52:00.282 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_num_target": 8, 2024-09-11T09:52:00.282 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_num_pending": 8, 2024-09-11T09:52:00.282 INFO:teuthology.orchestra.run.smithi017.stdout: "last_pg_merge_meta": { 2024-09-11T09:52:00.282 INFO:teuthology.orchestra.run.smithi017.stdout: "source_pgid": "0.0", 2024-09-11T09:52:00.282 INFO:teuthology.orchestra.run.smithi017.stdout: "ready_epoch": 0, 2024-09-11T09:52:00.282 INFO:teuthology.orchestra.run.smithi017.stdout: "last_epoch_started": 0, 2024-09-11T09:52:00.282 INFO:teuthology.orchestra.run.smithi017.stdout: "last_epoch_clean": 0, 2024-09-11T09:52:00.283 INFO:teuthology.orchestra.run.smithi017.stdout: "source_version": "0'0", 2024-09-11T09:52:00.283 INFO:teuthology.orchestra.run.smithi017.stdout: "target_version": "0'0" 2024-09-11T09:52:00.283 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.283 INFO:teuthology.orchestra.run.smithi017.stdout: "last_change": "58", 2024-09-11T09:52:00.283 INFO:teuthology.orchestra.run.smithi017.stdout: "last_force_op_resend": "0", 2024-09-11T09:52:00.283 INFO:teuthology.orchestra.run.smithi017.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-11T09:52:00.283 INFO:teuthology.orchestra.run.smithi017.stdout: "last_force_op_resend_preluminous": "0", 2024-09-11T09:52:00.283 INFO:teuthology.orchestra.run.smithi017.stdout: "auid": 0, 2024-09-11T09:52:00.283 INFO:teuthology.orchestra.run.smithi017.stdout: "snap_mode": "selfmanaged", 2024-09-11T09:52:00.283 INFO:teuthology.orchestra.run.smithi017.stdout: "snap_seq": 2, 2024-09-11T09:52:00.283 INFO:teuthology.orchestra.run.smithi017.stdout: "snap_epoch": 58, 2024-09-11T09:52:00.283 INFO:teuthology.orchestra.run.smithi017.stdout: "pool_snaps": [], 2024-09-11T09:52:00.284 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_snaps": "[]", 2024-09-11T09:52:00.284 INFO:teuthology.orchestra.run.smithi017.stdout: "quota_max_bytes": 0, 2024-09-11T09:52:00.284 INFO:teuthology.orchestra.run.smithi017.stdout: "quota_max_objects": 0, 2024-09-11T09:52:00.284 INFO:teuthology.orchestra.run.smithi017.stdout: "tiers": [], 2024-09-11T09:52:00.284 INFO:teuthology.orchestra.run.smithi017.stdout: "tier_of": -1, 2024-09-11T09:52:00.284 INFO:teuthology.orchestra.run.smithi017.stdout: "read_tier": -1, 2024-09-11T09:52:00.284 INFO:teuthology.orchestra.run.smithi017.stdout: "write_tier": -1, 2024-09-11T09:52:00.284 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_mode": "none", 2024-09-11T09:52:00.284 INFO:teuthology.orchestra.run.smithi017.stdout: "target_max_bytes": 0, 2024-09-11T09:52:00.284 INFO:teuthology.orchestra.run.smithi017.stdout: "target_max_objects": 0, 2024-09-11T09:52:00.284 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-11T09:52:00.284 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-11T09:52:00.284 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-11T09:52:00.285 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_min_flush_age": 0, 2024-09-11T09:52:00.285 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_min_evict_age": 0, 2024-09-11T09:52:00.285 INFO:teuthology.orchestra.run.smithi017.stdout: "erasure_code_profile": "", 2024-09-11T09:52:00.285 INFO:teuthology.orchestra.run.smithi017.stdout: "hit_set_params": { 2024-09-11T09:52:00.285 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "none" 2024-09-11T09:52:00.285 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.285 INFO:teuthology.orchestra.run.smithi017.stdout: "hit_set_period": 0, 2024-09-11T09:52:00.285 INFO:teuthology.orchestra.run.smithi017.stdout: "hit_set_count": 0, 2024-09-11T09:52:00.285 INFO:teuthology.orchestra.run.smithi017.stdout: "use_gmt_hitset": true, 2024-09-11T09:52:00.285 INFO:teuthology.orchestra.run.smithi017.stdout: "min_read_recency_for_promote": 0, 2024-09-11T09:52:00.285 INFO:teuthology.orchestra.run.smithi017.stdout: "min_write_recency_for_promote": 0, 2024-09-11T09:52:00.285 INFO:teuthology.orchestra.run.smithi017.stdout: "hit_set_grade_decay_rate": 0, 2024-09-11T09:52:00.285 INFO:teuthology.orchestra.run.smithi017.stdout: "hit_set_search_last_n": 0, 2024-09-11T09:52:00.286 INFO:teuthology.orchestra.run.smithi017.stdout: "grade_table": [], 2024-09-11T09:52:00.286 INFO:teuthology.orchestra.run.smithi017.stdout: "stripe_width": 0, 2024-09-11T09:52:00.286 INFO:teuthology.orchestra.run.smithi017.stdout: "expected_num_objects": 0, 2024-09-11T09:52:00.286 INFO:teuthology.orchestra.run.smithi017.stdout: "fast_read": false, 2024-09-11T09:52:00.286 INFO:teuthology.orchestra.run.smithi017.stdout: "options": {}, 2024-09-11T09:52:00.286 INFO:teuthology.orchestra.run.smithi017.stdout: "application_metadata": { 2024-09-11T09:52:00.286 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd": {} 2024-09-11T09:52:00.286 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.286 INFO:teuthology.orchestra.run.smithi017.stdout: "read_balance": { 2024-09-11T09:52:00.286 INFO:teuthology.orchestra.run.smithi017.stdout: "score_type": "Fair distribution", 2024-09-11T09:52:00.286 INFO:teuthology.orchestra.run.smithi017.stdout: "score_acting": 2.2699999809265137, 2024-09-11T09:52:00.286 INFO:teuthology.orchestra.run.smithi017.stdout: "score_stable": 2.2699999809265137, 2024-09-11T09:52:00.286 INFO:teuthology.orchestra.run.smithi017.stdout: "optimal_score": 0.82999998331069946, 2024-09-11T09:52:00.287 INFO:teuthology.orchestra.run.smithi017.stdout: "raw_score_acting": 1.8799999952316284, 2024-09-11T09:52:00.287 INFO:teuthology.orchestra.run.smithi017.stdout: "raw_score_stable": 1.8799999952316284, 2024-09-11T09:52:00.287 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity_weighted": 1, 2024-09-11T09:52:00.287 INFO:teuthology.orchestra.run.smithi017.stdout: "average_primary_affinity": 1, 2024-09-11T09:52:00.287 INFO:teuthology.orchestra.run.smithi017.stdout: "average_primary_affinity_weighted": 1 2024-09-11T09:52:00.287 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:00.287 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:00.287 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-11T09:52:00.287 INFO:teuthology.orchestra.run.smithi017.stdout: "osds": [ 2024-09-11T09:52:00.287 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.287 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 0, 2024-09-11T09:52:00.287 INFO:teuthology.orchestra.run.smithi017.stdout: "uuid": "4829b7a1-d607-4a56-b6d2-55036cb8acf8", 2024-09-11T09:52:00.288 INFO:teuthology.orchestra.run.smithi017.stdout: "up": 1, 2024-09-11T09:52:00.288 INFO:teuthology.orchestra.run.smithi017.stdout: "in": 1, 2024-09-11T09:52:00.288 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 1, 2024-09-11T09:52:00.288 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity": 1, 2024-09-11T09:52:00.288 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_begin": 0, 2024-09-11T09:52:00.288 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_end": 0, 2024-09-11T09:52:00.288 INFO:teuthology.orchestra.run.smithi017.stdout: "up_from": 12, 2024-09-11T09:52:00.288 INFO:teuthology.orchestra.run.smithi017.stdout: "up_thru": 66, 2024-09-11T09:52:00.288 INFO:teuthology.orchestra.run.smithi017.stdout: "down_at": 0, 2024-09-11T09:52:00.288 INFO:teuthology.orchestra.run.smithi017.stdout: "lost_at": 0, 2024-09-11T09:52:00.288 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-11T09:52:00.288 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-11T09:52:00.288 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.289 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-11T09:52:00.289 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6808", 2024-09-11T09:52:00.289 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 4154449929 2024-09-11T09:52:00.289 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.289 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.289 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-11T09:52:00.289 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6809", 2024-09-11T09:52:00.289 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 4154449929 2024-09-11T09:52:00.289 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:00.289 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:00.289 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.289 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addrs": { 2024-09-11T09:52:00.290 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-11T09:52:00.290 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.290 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-11T09:52:00.290 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6810", 2024-09-11T09:52:00.290 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 4154449929 2024-09-11T09:52:00.290 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.290 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.290 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-11T09:52:00.290 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6811", 2024-09-11T09:52:00.290 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 4154449929 2024-09-11T09:52:00.290 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:00.290 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:00.291 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.291 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addrs": { 2024-09-11T09:52:00.291 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-11T09:52:00.291 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.291 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-11T09:52:00.291 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6814", 2024-09-11T09:52:00.291 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 4154449929 2024-09-11T09:52:00.291 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.291 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.291 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-11T09:52:00.291 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6815", 2024-09-11T09:52:00.291 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 4154449929 2024-09-11T09:52:00.292 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:00.292 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:00.292 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.292 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addrs": { 2024-09-11T09:52:00.292 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-11T09:52:00.292 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.292 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-11T09:52:00.292 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6812", 2024-09-11T09:52:00.292 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 4154449929 2024-09-11T09:52:00.292 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.292 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.292 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-11T09:52:00.292 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6813", 2024-09-11T09:52:00.293 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 4154449929 2024-09-11T09:52:00.293 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:00.293 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:00.293 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.293 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6809/4154449929", 2024-09-11T09:52:00.293 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addr": "172.21.15.17:6811/4154449929", 2024-09-11T09:52:00.293 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addr": "172.21.15.17:6815/4154449929", 2024-09-11T09:52:00.293 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addr": "172.21.15.17:6813/4154449929", 2024-09-11T09:52:00.293 INFO:teuthology.orchestra.run.smithi017.stdout: "state": [ 2024-09-11T09:52:00.293 INFO:teuthology.orchestra.run.smithi017.stdout: "exists", 2024-09-11T09:52:00.293 INFO:teuthology.orchestra.run.smithi017.stdout: "up" 2024-09-11T09:52:00.293 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:00.294 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.294 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.294 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 1, 2024-09-11T09:52:00.294 INFO:teuthology.orchestra.run.smithi017.stdout: "uuid": "ab1d5df0-3c0e-44cd-88a0-96b07964148b", 2024-09-11T09:52:00.294 INFO:teuthology.orchestra.run.smithi017.stdout: "up": 1, 2024-09-11T09:52:00.294 INFO:teuthology.orchestra.run.smithi017.stdout: "in": 1, 2024-09-11T09:52:00.294 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 1, 2024-09-11T09:52:00.294 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity": 1, 2024-09-11T09:52:00.294 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_begin": 0, 2024-09-11T09:52:00.294 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_end": 0, 2024-09-11T09:52:00.294 INFO:teuthology.orchestra.run.smithi017.stdout: "up_from": 11, 2024-09-11T09:52:00.294 INFO:teuthology.orchestra.run.smithi017.stdout: "up_thru": 54, 2024-09-11T09:52:00.294 INFO:teuthology.orchestra.run.smithi017.stdout: "down_at": 0, 2024-09-11T09:52:00.295 INFO:teuthology.orchestra.run.smithi017.stdout: "lost_at": 0, 2024-09-11T09:52:00.295 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-11T09:52:00.295 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-11T09:52:00.295 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.295 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-11T09:52:00.295 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6800", 2024-09-11T09:52:00.295 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 4139511644 2024-09-11T09:52:00.295 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.295 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.295 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-11T09:52:00.295 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6801", 2024-09-11T09:52:00.295 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 4139511644 2024-09-11T09:52:00.296 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:00.296 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:00.296 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.296 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addrs": { 2024-09-11T09:52:00.296 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-11T09:52:00.296 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.296 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-11T09:52:00.296 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6802", 2024-09-11T09:52:00.296 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 4139511644 2024-09-11T09:52:00.296 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.296 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.296 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-11T09:52:00.296 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6803", 2024-09-11T09:52:00.297 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 4139511644 2024-09-11T09:52:00.297 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:00.297 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:00.297 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.297 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addrs": { 2024-09-11T09:52:00.297 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-11T09:52:00.297 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.297 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-11T09:52:00.297 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6806", 2024-09-11T09:52:00.297 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 4139511644 2024-09-11T09:52:00.297 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.297 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.297 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-11T09:52:00.298 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6807", 2024-09-11T09:52:00.298 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 4139511644 2024-09-11T09:52:00.298 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:00.298 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:00.298 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.298 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addrs": { 2024-09-11T09:52:00.298 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-11T09:52:00.298 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.298 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-11T09:52:00.298 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6804", 2024-09-11T09:52:00.298 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 4139511644 2024-09-11T09:52:00.298 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.298 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.298 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-11T09:52:00.299 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6805", 2024-09-11T09:52:00.299 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 4139511644 2024-09-11T09:52:00.299 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:00.299 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:00.299 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.299 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6801/4139511644", 2024-09-11T09:52:00.299 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addr": "172.21.15.17:6803/4139511644", 2024-09-11T09:52:00.299 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addr": "172.21.15.17:6807/4139511644", 2024-09-11T09:52:00.299 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addr": "172.21.15.17:6805/4139511644", 2024-09-11T09:52:00.299 INFO:teuthology.orchestra.run.smithi017.stdout: "state": [ 2024-09-11T09:52:00.299 INFO:teuthology.orchestra.run.smithi017.stdout: "exists", 2024-09-11T09:52:00.299 INFO:teuthology.orchestra.run.smithi017.stdout: "up" 2024-09-11T09:52:00.299 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:00.299 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.300 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.300 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 2, 2024-09-11T09:52:00.300 INFO:teuthology.orchestra.run.smithi017.stdout: "uuid": "4c8e50db-b49c-4d44-a913-47b5b30f8699", 2024-09-11T09:52:00.300 INFO:teuthology.orchestra.run.smithi017.stdout: "up": 1, 2024-09-11T09:52:00.300 INFO:teuthology.orchestra.run.smithi017.stdout: "in": 1, 2024-09-11T09:52:00.300 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 1, 2024-09-11T09:52:00.300 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity": 1, 2024-09-11T09:52:00.300 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_begin": 0, 2024-09-11T09:52:00.300 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_end": 0, 2024-09-11T09:52:00.300 INFO:teuthology.orchestra.run.smithi017.stdout: "up_from": 11, 2024-09-11T09:52:00.300 INFO:teuthology.orchestra.run.smithi017.stdout: "up_thru": 54, 2024-09-11T09:52:00.300 INFO:teuthology.orchestra.run.smithi017.stdout: "down_at": 0, 2024-09-11T09:52:00.300 INFO:teuthology.orchestra.run.smithi017.stdout: "lost_at": 0, 2024-09-11T09:52:00.301 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-11T09:52:00.301 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-11T09:52:00.301 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.301 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-11T09:52:00.301 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6816", 2024-09-11T09:52:00.301 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1151282506 2024-09-11T09:52:00.301 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.301 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.301 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-11T09:52:00.301 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6817", 2024-09-11T09:52:00.301 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1151282506 2024-09-11T09:52:00.301 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:00.301 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:00.301 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.302 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addrs": { 2024-09-11T09:52:00.302 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-11T09:52:00.302 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.302 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-11T09:52:00.302 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6818", 2024-09-11T09:52:00.302 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1151282506 2024-09-11T09:52:00.302 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.302 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.302 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-11T09:52:00.302 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6819", 2024-09-11T09:52:00.302 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1151282506 2024-09-11T09:52:00.302 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:00.302 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:00.303 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.303 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addrs": { 2024-09-11T09:52:00.303 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-11T09:52:00.303 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.303 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-11T09:52:00.303 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6822", 2024-09-11T09:52:00.303 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1151282506 2024-09-11T09:52:00.303 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.303 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.303 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-11T09:52:00.303 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6823", 2024-09-11T09:52:00.304 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1151282506 2024-09-11T09:52:00.304 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:00.304 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:00.304 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.304 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addrs": { 2024-09-11T09:52:00.304 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-11T09:52:00.304 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.304 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-11T09:52:00.304 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6820", 2024-09-11T09:52:00.304 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1151282506 2024-09-11T09:52:00.304 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.304 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.304 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-11T09:52:00.305 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6821", 2024-09-11T09:52:00.305 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1151282506 2024-09-11T09:52:00.305 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:00.305 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:00.305 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.305 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6817/1151282506", 2024-09-11T09:52:00.305 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addr": "172.21.15.17:6819/1151282506", 2024-09-11T09:52:00.305 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addr": "172.21.15.17:6823/1151282506", 2024-09-11T09:52:00.305 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addr": "172.21.15.17:6821/1151282506", 2024-09-11T09:52:00.305 INFO:teuthology.orchestra.run.smithi017.stdout: "state": [ 2024-09-11T09:52:00.305 INFO:teuthology.orchestra.run.smithi017.stdout: "exists", 2024-09-11T09:52:00.305 INFO:teuthology.orchestra.run.smithi017.stdout: "up" 2024-09-11T09:52:00.306 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:00.306 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.306 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.306 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 3, 2024-09-11T09:52:00.306 INFO:teuthology.orchestra.run.smithi017.stdout: "uuid": "37abc55b-a1eb-49d5-b1ff-b507ee2cb5a6", 2024-09-11T09:52:00.306 INFO:teuthology.orchestra.run.smithi017.stdout: "up": 1, 2024-09-11T09:52:00.306 INFO:teuthology.orchestra.run.smithi017.stdout: "in": 1, 2024-09-11T09:52:00.306 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 1, 2024-09-11T09:52:00.306 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity": 1, 2024-09-11T09:52:00.306 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_begin": 0, 2024-09-11T09:52:00.306 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_end": 0, 2024-09-11T09:52:00.306 INFO:teuthology.orchestra.run.smithi017.stdout: "up_from": 12, 2024-09-11T09:52:00.306 INFO:teuthology.orchestra.run.smithi017.stdout: "up_thru": 35, 2024-09-11T09:52:00.306 INFO:teuthology.orchestra.run.smithi017.stdout: "down_at": 0, 2024-09-11T09:52:00.307 INFO:teuthology.orchestra.run.smithi017.stdout: "lost_at": 0, 2024-09-11T09:52:00.307 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-11T09:52:00.307 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-11T09:52:00.307 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.307 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-11T09:52:00.307 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.96:6816", 2024-09-11T09:52:00.307 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 332854807 2024-09-11T09:52:00.307 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.307 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.307 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-11T09:52:00.307 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.96:6817", 2024-09-11T09:52:00.307 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 332854807 2024-09-11T09:52:00.307 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:00.308 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:00.308 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.308 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addrs": { 2024-09-11T09:52:00.308 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-11T09:52:00.308 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.308 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-11T09:52:00.308 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.96:6818", 2024-09-11T09:52:00.308 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 332854807 2024-09-11T09:52:00.308 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.308 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.308 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-11T09:52:00.308 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.96:6819", 2024-09-11T09:52:00.308 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 332854807 2024-09-11T09:52:00.309 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:00.309 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:00.309 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.309 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addrs": { 2024-09-11T09:52:00.309 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-11T09:52:00.309 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.309 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-11T09:52:00.309 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.96:6822", 2024-09-11T09:52:00.309 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 332854807 2024-09-11T09:52:00.309 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.309 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.309 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-11T09:52:00.309 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.96:6823", 2024-09-11T09:52:00.310 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 332854807 2024-09-11T09:52:00.310 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:00.310 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:00.310 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.310 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addrs": { 2024-09-11T09:52:00.310 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-11T09:52:00.310 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.310 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-11T09:52:00.310 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.96:6820", 2024-09-11T09:52:00.310 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 332854807 2024-09-11T09:52:00.310 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.310 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.310 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-11T09:52:00.311 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.96:6821", 2024-09-11T09:52:00.311 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 332854807 2024-09-11T09:52:00.311 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:00.311 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:00.311 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.311 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.96:6817/332854807", 2024-09-11T09:52:00.311 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addr": "172.21.15.96:6819/332854807", 2024-09-11T09:52:00.311 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addr": "172.21.15.96:6823/332854807", 2024-09-11T09:52:00.311 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addr": "172.21.15.96:6821/332854807", 2024-09-11T09:52:00.311 INFO:teuthology.orchestra.run.smithi017.stdout: "state": [ 2024-09-11T09:52:00.311 INFO:teuthology.orchestra.run.smithi017.stdout: "exists", 2024-09-11T09:52:00.311 INFO:teuthology.orchestra.run.smithi017.stdout: "up" 2024-09-11T09:52:00.311 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:00.312 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.312 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.312 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 4, 2024-09-11T09:52:00.312 INFO:teuthology.orchestra.run.smithi017.stdout: "uuid": "067d9586-4f5a-4d17-96c5-86a7b8bcd114", 2024-09-11T09:52:00.312 INFO:teuthology.orchestra.run.smithi017.stdout: "up": 1, 2024-09-11T09:52:00.312 INFO:teuthology.orchestra.run.smithi017.stdout: "in": 1, 2024-09-11T09:52:00.312 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 1, 2024-09-11T09:52:00.312 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity": 1, 2024-09-11T09:52:00.312 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_begin": 11, 2024-09-11T09:52:00.312 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_end": 54, 2024-09-11T09:52:00.312 INFO:teuthology.orchestra.run.smithi017.stdout: "up_from": 66, 2024-09-11T09:52:00.312 INFO:teuthology.orchestra.run.smithi017.stdout: "up_thru": 66, 2024-09-11T09:52:00.312 INFO:teuthology.orchestra.run.smithi017.stdout: "down_at": 55, 2024-09-11T09:52:00.312 INFO:teuthology.orchestra.run.smithi017.stdout: "lost_at": 0, 2024-09-11T09:52:00.313 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-11T09:52:00.313 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-11T09:52:00.313 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.313 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-11T09:52:00.313 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.96:6800", 2024-09-11T09:52:00.313 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3374675329 2024-09-11T09:52:00.313 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.313 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.313 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-11T09:52:00.313 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.96:6801", 2024-09-11T09:52:00.313 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3374675329 2024-09-11T09:52:00.313 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:00.313 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:00.314 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.314 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addrs": { 2024-09-11T09:52:00.314 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-11T09:52:00.314 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.314 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-11T09:52:00.314 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.96:6802", 2024-09-11T09:52:00.314 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3374675329 2024-09-11T09:52:00.314 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.314 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.314 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-11T09:52:00.314 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.96:6803", 2024-09-11T09:52:00.314 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3374675329 2024-09-11T09:52:00.314 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:00.315 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:00.315 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.315 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addrs": { 2024-09-11T09:52:00.315 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-11T09:52:00.315 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.315 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-11T09:52:00.315 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.96:6806", 2024-09-11T09:52:00.315 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3374675329 2024-09-11T09:52:00.315 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.315 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.315 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-11T09:52:00.315 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.96:6807", 2024-09-11T09:52:00.315 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3374675329 2024-09-11T09:52:00.316 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:00.316 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:00.316 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.316 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addrs": { 2024-09-11T09:52:00.316 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-11T09:52:00.316 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.316 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-11T09:52:00.316 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.96:6804", 2024-09-11T09:52:00.316 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3374675329 2024-09-11T09:52:00.316 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.316 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.316 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-11T09:52:00.317 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.96:6805", 2024-09-11T09:52:00.317 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3374675329 2024-09-11T09:52:00.317 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:00.317 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:00.317 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.317 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.96:6801/3374675329", 2024-09-11T09:52:00.317 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addr": "172.21.15.96:6803/3374675329", 2024-09-11T09:52:00.317 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addr": "172.21.15.96:6807/3374675329", 2024-09-11T09:52:00.317 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addr": "172.21.15.96:6805/3374675329", 2024-09-11T09:52:00.317 INFO:teuthology.orchestra.run.smithi017.stdout: "state": [ 2024-09-11T09:52:00.317 INFO:teuthology.orchestra.run.smithi017.stdout: "exists", 2024-09-11T09:52:00.317 INFO:teuthology.orchestra.run.smithi017.stdout: "up" 2024-09-11T09:52:00.318 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:00.318 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.318 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.318 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 5, 2024-09-11T09:52:00.318 INFO:teuthology.orchestra.run.smithi017.stdout: "uuid": "8ff7a724-0a3a-4aa2-baf1-fd52f88d476c", 2024-09-11T09:52:00.318 INFO:teuthology.orchestra.run.smithi017.stdout: "up": 1, 2024-09-11T09:52:00.318 INFO:teuthology.orchestra.run.smithi017.stdout: "in": 1, 2024-09-11T09:52:00.318 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 1, 2024-09-11T09:52:00.318 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity": 1, 2024-09-11T09:52:00.318 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_begin": 0, 2024-09-11T09:52:00.318 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_end": 0, 2024-09-11T09:52:00.318 INFO:teuthology.orchestra.run.smithi017.stdout: "up_from": 11, 2024-09-11T09:52:00.318 INFO:teuthology.orchestra.run.smithi017.stdout: "up_thru": 54, 2024-09-11T09:52:00.319 INFO:teuthology.orchestra.run.smithi017.stdout: "down_at": 0, 2024-09-11T09:52:00.319 INFO:teuthology.orchestra.run.smithi017.stdout: "lost_at": 0, 2024-09-11T09:52:00.319 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-11T09:52:00.319 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-11T09:52:00.319 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.319 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-11T09:52:00.319 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.96:6808", 2024-09-11T09:52:00.319 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 509935370 2024-09-11T09:52:00.319 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.319 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.319 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-11T09:52:00.319 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.96:6809", 2024-09-11T09:52:00.320 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 509935370 2024-09-11T09:52:00.320 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:00.320 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:00.320 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.320 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addrs": { 2024-09-11T09:52:00.320 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-11T09:52:00.320 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.320 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-11T09:52:00.320 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.96:6810", 2024-09-11T09:52:00.320 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 509935370 2024-09-11T09:52:00.320 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.320 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.320 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-11T09:52:00.321 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.96:6811", 2024-09-11T09:52:00.321 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 509935370 2024-09-11T09:52:00.321 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:00.321 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:00.321 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.321 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addrs": { 2024-09-11T09:52:00.321 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-11T09:52:00.321 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.321 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-11T09:52:00.321 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.96:6814", 2024-09-11T09:52:00.321 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 509935370 2024-09-11T09:52:00.321 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.321 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.322 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-11T09:52:00.322 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.96:6815", 2024-09-11T09:52:00.322 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 509935370 2024-09-11T09:52:00.322 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:00.322 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:00.322 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.322 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addrs": { 2024-09-11T09:52:00.322 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-11T09:52:00.322 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.322 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-11T09:52:00.322 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.96:6812", 2024-09-11T09:52:00.322 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 509935370 2024-09-11T09:52:00.322 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.323 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.323 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-11T09:52:00.323 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.96:6813", 2024-09-11T09:52:00.323 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 509935370 2024-09-11T09:52:00.323 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:00.323 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:00.323 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.323 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.96:6809/509935370", 2024-09-11T09:52:00.323 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addr": "172.21.15.96:6811/509935370", 2024-09-11T09:52:00.323 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addr": "172.21.15.96:6815/509935370", 2024-09-11T09:52:00.323 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addr": "172.21.15.96:6813/509935370", 2024-09-11T09:52:00.323 INFO:teuthology.orchestra.run.smithi017.stdout: "state": [ 2024-09-11T09:52:00.324 INFO:teuthology.orchestra.run.smithi017.stdout: "exists", 2024-09-11T09:52:00.324 INFO:teuthology.orchestra.run.smithi017.stdout: "up" 2024-09-11T09:52:00.324 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:00.324 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:00.324 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-11T09:52:00.324 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_xinfo": [ 2024-09-11T09:52:00.324 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.324 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 0, 2024-09-11T09:52:00.324 INFO:teuthology.orchestra.run.smithi017.stdout: "down_stamp": "0.000000", 2024-09-11T09:52:00.324 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_probability": 0, 2024-09-11T09:52:00.324 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_interval": 0, 2024-09-11T09:52:00.324 INFO:teuthology.orchestra.run.smithi017.stdout: "features": 4540701547738038271, 2024-09-11T09:52:00.325 INFO:teuthology.orchestra.run.smithi017.stdout: "old_weight": 0, 2024-09-11T09:52:00.325 INFO:teuthology.orchestra.run.smithi017.stdout: "last_purged_snaps_scrub": "2024-09-11T09:50:08.513270+0000", 2024-09-11T09:52:00.325 INFO:teuthology.orchestra.run.smithi017.stdout: "dead_epoch": 0 2024-09-11T09:52:00.325 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.325 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.325 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 1, 2024-09-11T09:52:00.325 INFO:teuthology.orchestra.run.smithi017.stdout: "down_stamp": "0.000000", 2024-09-11T09:52:00.325 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_probability": 0, 2024-09-11T09:52:00.325 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_interval": 0, 2024-09-11T09:52:00.325 INFO:teuthology.orchestra.run.smithi017.stdout: "features": 4540701547738038271, 2024-09-11T09:52:00.325 INFO:teuthology.orchestra.run.smithi017.stdout: "old_weight": 0, 2024-09-11T09:52:00.325 INFO:teuthology.orchestra.run.smithi017.stdout: "last_purged_snaps_scrub": "2024-09-11T09:50:07.522863+0000", 2024-09-11T09:52:00.325 INFO:teuthology.orchestra.run.smithi017.stdout: "dead_epoch": 0 2024-09-11T09:52:00.326 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.326 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.326 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 2, 2024-09-11T09:52:00.326 INFO:teuthology.orchestra.run.smithi017.stdout: "down_stamp": "0.000000", 2024-09-11T09:52:00.326 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_probability": 0, 2024-09-11T09:52:00.326 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_interval": 0, 2024-09-11T09:52:00.326 INFO:teuthology.orchestra.run.smithi017.stdout: "features": 4540701547738038271, 2024-09-11T09:52:00.326 INFO:teuthology.orchestra.run.smithi017.stdout: "old_weight": 0, 2024-09-11T09:52:00.326 INFO:teuthology.orchestra.run.smithi017.stdout: "last_purged_snaps_scrub": "2024-09-11T09:50:08.134364+0000", 2024-09-11T09:52:00.326 INFO:teuthology.orchestra.run.smithi017.stdout: "dead_epoch": 0 2024-09-11T09:52:00.326 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.326 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.326 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 3, 2024-09-11T09:52:00.327 INFO:teuthology.orchestra.run.smithi017.stdout: "down_stamp": "0.000000", 2024-09-11T09:52:00.327 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_probability": 0, 2024-09-11T09:52:00.327 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_interval": 0, 2024-09-11T09:52:00.327 INFO:teuthology.orchestra.run.smithi017.stdout: "features": 4540701547738038271, 2024-09-11T09:52:00.327 INFO:teuthology.orchestra.run.smithi017.stdout: "old_weight": 0, 2024-09-11T09:52:00.327 INFO:teuthology.orchestra.run.smithi017.stdout: "last_purged_snaps_scrub": "2024-09-11T09:50:08.622366+0000", 2024-09-11T09:52:00.327 INFO:teuthology.orchestra.run.smithi017.stdout: "dead_epoch": 0 2024-09-11T09:52:00.327 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.327 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.327 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 4, 2024-09-11T09:52:00.327 INFO:teuthology.orchestra.run.smithi017.stdout: "down_stamp": "0.000000", 2024-09-11T09:52:00.327 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_probability": 0, 2024-09-11T09:52:00.327 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_interval": 0, 2024-09-11T09:52:00.328 INFO:teuthology.orchestra.run.smithi017.stdout: "features": 4540701547738038271, 2024-09-11T09:52:00.328 INFO:teuthology.orchestra.run.smithi017.stdout: "old_weight": 0, 2024-09-11T09:52:00.328 INFO:teuthology.orchestra.run.smithi017.stdout: "last_purged_snaps_scrub": "2024-09-11T09:50:08.156142+0000", 2024-09-11T09:52:00.328 INFO:teuthology.orchestra.run.smithi017.stdout: "dead_epoch": 55 2024-09-11T09:52:00.328 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.328 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.328 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 5, 2024-09-11T09:52:00.328 INFO:teuthology.orchestra.run.smithi017.stdout: "down_stamp": "0.000000", 2024-09-11T09:52:00.328 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_probability": 0, 2024-09-11T09:52:00.328 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_interval": 0, 2024-09-11T09:52:00.328 INFO:teuthology.orchestra.run.smithi017.stdout: "features": 4540701547738038271, 2024-09-11T09:52:00.328 INFO:teuthology.orchestra.run.smithi017.stdout: "old_weight": 0, 2024-09-11T09:52:00.328 INFO:teuthology.orchestra.run.smithi017.stdout: "last_purged_snaps_scrub": "2024-09-11T09:50:08.108116+0000", 2024-09-11T09:52:00.329 INFO:teuthology.orchestra.run.smithi017.stdout: "dead_epoch": 0 2024-09-11T09:52:00.329 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:00.329 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-11T09:52:00.329 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_upmap": [], 2024-09-11T09:52:00.329 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_upmap_items": [ 2024-09-11T09:52:00.329 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.329 INFO:teuthology.orchestra.run.smithi017.stdout: "pgid": "2.7", 2024-09-11T09:52:00.329 INFO:teuthology.orchestra.run.smithi017.stdout: "mappings": [ 2024-09-11T09:52:00.329 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.329 INFO:teuthology.orchestra.run.smithi017.stdout: "from": 4, 2024-09-11T09:52:00.329 INFO:teuthology.orchestra.run.smithi017.stdout: "to": 5 2024-09-11T09:52:00.329 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.329 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:00.330 INFO:teuthology.orchestra.run.smithi017.stdout: "from": 3, 2024-09-11T09:52:00.330 INFO:teuthology.orchestra.run.smithi017.stdout: "to": 0 2024-09-11T09:52:00.330 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:00.330 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:00.330 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:00.330 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-11T09:52:00.330 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_upmap_primaries": [], 2024-09-11T09:52:00.330 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_temp": [], 2024-09-11T09:52:00.330 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_temp": [], 2024-09-11T09:52:00.330 INFO:teuthology.orchestra.run.smithi017.stdout: "blocklist": {}, 2024-09-11T09:52:00.330 INFO:teuthology.orchestra.run.smithi017.stdout: "range_blocklist": {}, 2024-09-11T09:52:00.330 INFO:teuthology.orchestra.run.smithi017.stdout: "erasure_code_profiles": { 2024-09-11T09:52:00.330 INFO:teuthology.orchestra.run.smithi017.stdout: "default": { 2024-09-11T09:52:00.330 INFO:teuthology.orchestra.run.smithi017.stdout: "crush-failure-domain": "osd", 2024-09-11T09:52:00.331 INFO:teuthology.orchestra.run.smithi017.stdout: "k": "2", 2024-09-11T09:52:00.331 INFO:teuthology.orchestra.run.smithi017.stdout: "m": "1", 2024-09-11T09:52:00.331 INFO:teuthology.orchestra.run.smithi017.stdout: "plugin": "jerasure", 2024-09-11T09:52:00.331 INFO:teuthology.orchestra.run.smithi017.stdout: "technique": "reed_sol_van" 2024-09-11T09:52:00.331 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:00.331 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:00.331 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_snaps_queue": [], 2024-09-11T09:52:00.331 INFO:teuthology.orchestra.run.smithi017.stdout: "new_removed_snaps": [], 2024-09-11T09:52:00.331 INFO:teuthology.orchestra.run.smithi017.stdout: "new_purged_snaps": [], 2024-09-11T09:52:00.331 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_node_flags": {}, 2024-09-11T09:52:00.331 INFO:teuthology.orchestra.run.smithi017.stdout: "device_class_flags": {}, 2024-09-11T09:52:00.331 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": { 2024-09-11T09:52:00.331 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode_enabled": false, 2024-09-11T09:52:00.332 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_bucket_count": 0, 2024-09-11T09:52:00.332 INFO:teuthology.orchestra.run.smithi017.stdout: "degraded_stretch_mode": 0, 2024-09-11T09:52:00.332 INFO:teuthology.orchestra.run.smithi017.stdout: "recovering_stretch_mode": 0, 2024-09-11T09:52:00.332 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode_bucket": 0 2024-09-11T09:52:00.332 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:00.332 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:52:00.332 INFO:tasks.ceph.osd.2.smithi017.stderr:2024-09-11T09:52:00.265+0000 7fd20b170640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31672) UID: 0 2024-09-11T09:52:00.334 INFO:tasks.thrashosds.thrasher:j is {'epoch': 74, 'fsid': 'a2f1daec-d3b0-4bf5-82c8-72ecbb93d4e4', 'created': '2024-09-11T09:50:00.165804+0000', 'modified': '2024-09-11T09:51:58.805905+0000', 'last_up_change': '2024-09-11T09:51:44.764990+0000', 'last_in_change': '2024-09-11T09:50:45.581301+0000', 'flags': 'nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5804032, 'flags_set': ['nodeep-scrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 6, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 3, 'max_osd': 6, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-11T09:50:10.457184+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '14', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 6.059999942779541, 'score_stable': 6.059999942779541, 'optimal_score': 0.33000001311302185, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2024-09-11T09:50:16.993202+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '74', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 6, 'snap_epoch': 74, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5099999904632568, 'score_stable': 1.5099999904632568, 'optimal_score': 0.8299999833106995, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'new_default_pool27121', 'create_time': '2024-09-11T09:51:23.493016+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '58', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 58, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.2699999809265137, 'score_stable': 2.2699999809265137, 'optimal_score': 0.8299999833106995, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}], 'osds': [{'osd': 0, 'uuid': '4829b7a1-d607-4a56-b6d2-55036cb8acf8', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 12, 'up_thru': 66, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6808', 'nonce': 4154449929}, {'type': 'v1', 'addr': '172.21.15.17:6809', 'nonce': 4154449929}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6810', 'nonce': 4154449929}, {'type': 'v1', 'addr': '172.21.15.17:6811', 'nonce': 4154449929}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6814', 'nonce': 4154449929}, {'type': 'v1', 'addr': '172.21.15.17:6815', 'nonce': 4154449929}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6812', 'nonce': 4154449929}, {'type': 'v1', 'addr': '172.21.15.17:6813', 'nonce': 4154449929}]}, 'public_addr': '172.21.15.17:6809/4154449929', 'cluster_addr': '172.21.15.17:6811/4154449929', 'heartbeat_back_addr': '172.21.15.17:6815/4154449929', 'heartbeat_front_addr': '172.21.15.17:6813/4154449929', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'ab1d5df0-3c0e-44cd-88a0-96b07964148b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 54, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6800', 'nonce': 4139511644}, {'type': 'v1', 'addr': '172.21.15.17:6801', 'nonce': 4139511644}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6802', 'nonce': 4139511644}, {'type': 'v1', 'addr': '172.21.15.17:6803', 'nonce': 4139511644}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6806', 'nonce': 4139511644}, {'type': 'v1', 'addr': '172.21.15.17:6807', 'nonce': 4139511644}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6804', 'nonce': 4139511644}, {'type': 'v1', 'addr': '172.21.15.17:6805', 'nonce': 4139511644}]}, 'public_addr': '172.21.15.17:6801/4139511644', 'cluster_addr': '172.21.15.17:6803/4139511644', 'heartbeat_back_addr': '172.21.15.17:6807/4139511644', 'heartbeat_front_addr': '172.21.15.17:6805/4139511644', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '4c8e50db-b49c-4d44-a913-47b5b30f8699', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 54, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6816', 'nonce': 1151282506}, {'type': 'v1', 'addr': '172.21.15.17:6817', 'nonce': 1151282506}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6818', 'nonce': 1151282506}, {'type': 'v1', 'addr': '172.21.15.17:6819', 'nonce': 1151282506}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6822', 'nonce': 1151282506}, {'type': 'v1', 'addr': '172.21.15.17:6823', 'nonce': 1151282506}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6820', 'nonce': 1151282506}, {'type': 'v1', 'addr': '172.21.15.17:6821', 'nonce': 1151282506}]}, 'public_addr': '172.21.15.17:6817/1151282506', 'cluster_addr': '172.21.15.17:6819/1151282506', 'heartbeat_back_addr': '172.21.15.17:6823/1151282506', 'heartbeat_front_addr': '172.21.15.17:6821/1151282506', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '37abc55b-a1eb-49d5-b1ff-b507ee2cb5a6', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 12, 'up_thru': 35, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.96:6816', 'nonce': 332854807}, {'type': 'v1', 'addr': '172.21.15.96:6817', 'nonce': 332854807}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.96:6818', 'nonce': 332854807}, {'type': 'v1', 'addr': '172.21.15.96:6819', 'nonce': 332854807}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.96:6822', 'nonce': 332854807}, {'type': 'v1', 'addr': '172.21.15.96:6823', 'nonce': 332854807}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.96:6820', 'nonce': 332854807}, {'type': 'v1', 'addr': '172.21.15.96:6821', 'nonce': 332854807}]}, 'public_addr': '172.21.15.96:6817/332854807', 'cluster_addr': '172.21.15.96:6819/332854807', 'heartbeat_back_addr': '172.21.15.96:6823/332854807', 'heartbeat_front_addr': '172.21.15.96:6821/332854807', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '067d9586-4f5a-4d17-96c5-86a7b8bcd114', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 11, 'last_clean_end': 54, 'up_from': 66, 'up_thru': 66, 'down_at': 55, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.96:6800', 'nonce': 3374675329}, {'type': 'v1', 'addr': '172.21.15.96:6801', 'nonce': 3374675329}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.96:6802', 'nonce': 3374675329}, {'type': 'v1', 'addr': '172.21.15.96:6803', 'nonce': 3374675329}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.96:6806', 'nonce': 3374675329}, {'type': 'v1', 'addr': '172.21.15.96:6807', 'nonce': 3374675329}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.96:6804', 'nonce': 3374675329}, {'type': 'v1', 'addr': '172.21.15.96:6805', 'nonce': 3374675329}]}, 'public_addr': '172.21.15.96:6801/3374675329', 'cluster_addr': '172.21.15.96:6803/3374675329', 'heartbeat_back_addr': '172.21.15.96:6807/3374675329', 'heartbeat_front_addr': '172.21.15.96:6805/3374675329', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '8ff7a724-0a3a-4aa2-baf1-fd52f88d476c', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 54, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.96:6808', 'nonce': 509935370}, {'type': 'v1', 'addr': '172.21.15.96:6809', 'nonce': 509935370}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.96:6810', 'nonce': 509935370}, {'type': 'v1', 'addr': '172.21.15.96:6811', 'nonce': 509935370}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.96:6814', 'nonce': 509935370}, {'type': 'v1', 'addr': '172.21.15.96:6815', 'nonce': 509935370}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.96:6812', 'nonce': 509935370}, {'type': 'v1', 'addr': '172.21.15.96:6813', 'nonce': 509935370}]}, 'public_addr': '172.21.15.96:6809/509935370', 'cluster_addr': '172.21.15.96:6811/509935370', 'heartbeat_back_addr': '172.21.15.96:6815/509935370', 'heartbeat_front_addr': '172.21.15.96:6813/509935370', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-11T09:50:08.513270+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-11T09:50:07.522863+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-11T09:50:08.134364+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-11T09:50:08.622366+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-11T09:50:08.156142+0000', 'dead_epoch': 55}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-11T09:50:08.108116+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [{'pgid': '2.7', 'mappings': [{'from': 4, 'to': 5}, {'from': 3, 'to': 0}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-11T09:52:00.334 INFO:tasks.thrashosds.thrasher:Clearing pg_upmap on 2.7 2024-09-11T09:52:00.334 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd rm-pg-upmap-items 2.7 2024-09-11T09:52:00.374 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:52:00.375+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:52:00.450 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-11T09:52:00.450 INFO:teuthology.orchestra.run.smithi096.stdout: "ops": [], 2024-09-11T09:52:00.450 INFO:teuthology.orchestra.run.smithi096.stdout: "num_ops": 0 2024-09-11T09:52:00.450 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-11T09:52:00.459 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-11T09:52:00.465 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:52:00.466+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:52:00.563 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-11T09:52:00.563 INFO:teuthology.orchestra.run.smithi096.stdout: "ops": [], 2024-09-11T09:52:00.563 INFO:teuthology.orchestra.run.smithi096.stdout: "complaint_time": 30, 2024-09-11T09:52:00.563 INFO:teuthology.orchestra.run.smithi096.stdout: "num_blocked_ops": 0 2024-09-11T09:52:00.563 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-11T09:52:00.565 INFO:tasks.ceph.osd.0.smithi017.stderr:2024-09-11T09:52:00.566+0000 7fb73c0f1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31665) UID: 0 2024-09-11T09:52:00.571 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-11T09:52:00.666 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:52:00.667+0000 7f3a7b90a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 42475) UID: 0 2024-09-11T09:52:00.676 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-11T09:52:00.676 INFO:teuthology.orchestra.run.smithi096.stdout: "size": 20, 2024-09-11T09:52:00.676 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 600, 2024-09-11T09:52:00.677 INFO:teuthology.orchestra.run.smithi096.stdout: "ops": [ 2024-09-11T09:52:00.677 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.677 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:111 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:52:00.677 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.711668+0000", 2024-09-11T09:52:00.677 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 108.96437734200001, 2024-09-11T09:52:00.677 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0077145570000000004, 2024-09-11T09:52:00.677 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:00.677 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:00.677 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:00.677 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:52:00.677 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:52:00.677 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:52:00.677 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 111 2024-09-11T09:52:00.678 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.678 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:00.678 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.678 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:00.678 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711668+0000", 2024-09-11T09:52:00.678 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:00.678 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.678 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.678 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:00.678 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711668+0000", 2024-09-11T09:52:00.678 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:00.678 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.678 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.679 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:00.679 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711669+0000", 2024-09-11T09:52:00.679 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.99e-07 2024-09-11T09:52:00.679 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.679 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.679 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:00.679 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711673+0000", 2024-09-11T09:52:00.679 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.7440000000000001e-06 2024-09-11T09:52:00.679 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.679 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.679 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:00.679 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711686+0000", 2024-09-11T09:52:00.679 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.3317e-05 2024-09-11T09:52:00.680 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.680 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.680 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:00.680 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711697+0000", 2024-09-11T09:52:00.680 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1022000000000001e-05 2024-09-11T09:52:00.680 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.680 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.680 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:00.680 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.717065+0000", 2024-09-11T09:52:00.680 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.005367531 2024-09-11T09:52:00.680 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.680 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.680 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:00.680 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.717114+0000", 2024-09-11T09:52:00.681 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.9707999999999999e-05 2024-09-11T09:52:00.681 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.681 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.681 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:52:00.681 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.717190+0000", 2024-09-11T09:52:00.681 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.5394000000000004e-05 2024-09-11T09:52:00.681 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.681 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.681 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:00.681 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718793+0000", 2024-09-11T09:52:00.681 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001603653 2024-09-11T09:52:00.681 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.681 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.681 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:52:00.682 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719329+0000", 2024-09-11T09:52:00.682 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00053543799999999995 2024-09-11T09:52:00.682 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.682 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.682 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:00.682 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719359+0000", 2024-09-11T09:52:00.682 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.0420999999999999e-05 2024-09-11T09:52:00.682 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.682 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.682 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:00.682 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719383+0000", 2024-09-11T09:52:00.682 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.3730000000000001e-05 2024-09-11T09:52:00.682 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:00.683 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:00.683 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:00.683 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.683 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.683 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:112 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:52:00.683 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.711712+0000", 2024-09-11T09:52:00.683 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 108.964333782, 2024-09-11T09:52:00.683 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.008162084, 2024-09-11T09:52:00.683 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:00.683 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:00.683 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:00.683 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:52:00.683 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:52:00.683 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:52:00.684 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 112 2024-09-11T09:52:00.684 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.684 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:00.684 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.684 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:00.684 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711712+0000", 2024-09-11T09:52:00.684 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:00.684 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.684 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.684 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:00.684 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711712+0000", 2024-09-11T09:52:00.684 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:00.685 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.685 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.685 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:00.685 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711712+0000", 2024-09-11T09:52:00.685 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.6000000000000004e-07 2024-09-11T09:52:00.685 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.685 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.685 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:00.685 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711808+0000", 2024-09-11T09:52:00.685 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.5977999999999995e-05 2024-09-11T09:52:00.685 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.685 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.686 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:00.686 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711819+0000", 2024-09-11T09:52:00.686 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.0755e-05 2024-09-11T09:52:00.686 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.686 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.686 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:00.686 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711826+0000", 2024-09-11T09:52:00.686 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.7970000000000003e-06 2024-09-11T09:52:00.686 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.686 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.686 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:00.686 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.717435+0000", 2024-09-11T09:52:00.686 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0056090239999999998 2024-09-11T09:52:00.686 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.687 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.687 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:00.687 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.717496+0000", 2024-09-11T09:52:00.687 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.0543e-05 2024-09-11T09:52:00.687 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.687 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.687 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:52:00.687 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.717591+0000", 2024-09-11T09:52:00.687 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.5667000000000004e-05 2024-09-11T09:52:00.687 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.687 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.687 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:00.687 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718831+0000", 2024-09-11T09:52:00.688 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001240204 2024-09-11T09:52:00.688 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.688 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.688 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:52:00.688 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719819+0000", 2024-09-11T09:52:00.688 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00098751500000000001 2024-09-11T09:52:00.688 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.688 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.688 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:00.688 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719850+0000", 2024-09-11T09:52:00.688 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.1535999999999998e-05 2024-09-11T09:52:00.688 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.688 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.689 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:00.689 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719874+0000", 2024-09-11T09:52:00.689 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.3504999999999999e-05 2024-09-11T09:52:00.689 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:00.689 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:00.689 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:00.689 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.689 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.689 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:113 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:52:00.689 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.711842+0000", 2024-09-11T09:52:00.689 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 108.964203283, 2024-09-11T09:52:00.689 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0081160099999999999, 2024-09-11T09:52:00.689 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:00.689 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:00.690 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:00.690 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:52:00.690 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:52:00.690 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:52:00.690 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 113 2024-09-11T09:52:00.690 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.690 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:00.690 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.690 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:00.690 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711842+0000", 2024-09-11T09:52:00.690 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:00.690 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.690 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.691 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:00.691 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711842+0000", 2024-09-11T09:52:00.691 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:00.691 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.691 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.691 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:00.691 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711843+0000", 2024-09-11T09:52:00.691 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.4399999999999999e-07 2024-09-11T09:52:00.691 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.691 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.691 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:00.691 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711848+0000", 2024-09-11T09:52:00.691 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.4159999999999997e-06 2024-09-11T09:52:00.692 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.692 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.692 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:00.692 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711856+0000", 2024-09-11T09:52:00.692 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.7870000000000007e-06 2024-09-11T09:52:00.692 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.692 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.692 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:00.692 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711862+0000", 2024-09-11T09:52:00.692 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.3689999999999998e-06 2024-09-11T09:52:00.692 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.692 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.692 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:00.693 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.717965+0000", 2024-09-11T09:52:00.693 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0061028899999999997 2024-09-11T09:52:00.693 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.693 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.693 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:00.693 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718023+0000", 2024-09-11T09:52:00.693 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.8313000000000002e-05 2024-09-11T09:52:00.693 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.693 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.693 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:52:00.693 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718109+0000", 2024-09-11T09:52:00.693 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.6205000000000006e-05 2024-09-11T09:52:00.693 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.694 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.694 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:00.694 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719808+0000", 2024-09-11T09:52:00.694 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0016990060000000001 2024-09-11T09:52:00.694 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.694 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.694 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:52:00.694 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719883+0000", 2024-09-11T09:52:00.694 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.4624999999999998e-05 2024-09-11T09:52:00.694 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.694 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.694 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:00.694 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719910+0000", 2024-09-11T09:52:00.694 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.7512e-05 2024-09-11T09:52:00.695 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.695 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.695 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:00.695 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719958+0000", 2024-09-11T09:52:00.695 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.8143000000000003e-05 2024-09-11T09:52:00.695 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:00.695 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:00.695 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:00.695 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.695 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.695 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:52:00.695 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.711887+0000", 2024-09-11T09:52:00.695 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 108.964158669, 2024-09-11T09:52:00.696 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0086063589999999992, 2024-09-11T09:52:00.696 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:00.696 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:00.696 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:00.696 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:52:00.696 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:52:00.696 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:52:00.696 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 114 2024-09-11T09:52:00.696 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.696 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:00.696 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.696 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:00.696 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711887+0000", 2024-09-11T09:52:00.697 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:00.697 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.697 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.697 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:00.697 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711887+0000", 2024-09-11T09:52:00.697 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:00.697 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.697 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.697 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:00.697 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711887+0000", 2024-09-11T09:52:00.697 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.1500000000000005e-07 2024-09-11T09:52:00.697 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.697 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.698 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:00.698 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711897+0000", 2024-09-11T09:52:00.698 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.1900000000000001e-06 2024-09-11T09:52:00.698 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.698 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.698 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:00.698 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711902+0000", 2024-09-11T09:52:00.698 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.2349999999999999e-06 2024-09-11T09:52:00.698 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.698 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.698 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:00.698 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711906+0000", 2024-09-11T09:52:00.698 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.7129999999999999e-06 2024-09-11T09:52:00.698 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.699 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.699 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:00.699 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718429+0000", 2024-09-11T09:52:00.699 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0065231960000000002 2024-09-11T09:52:00.699 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.699 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.699 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:00.699 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718480+0000", 2024-09-11T09:52:00.699 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.1161e-05 2024-09-11T09:52:00.699 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.700 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.700 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:52:00.700 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718558+0000", 2024-09-11T09:52:00.700 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.7519000000000001e-05 2024-09-11T09:52:00.700 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.700 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.700 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:00.700 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720038+0000", 2024-09-11T09:52:00.700 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001480016 2024-09-11T09:52:00.700 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.700 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.700 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:52:00.700 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720417+0000", 2024-09-11T09:52:00.700 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00037943699999999999 2024-09-11T09:52:00.701 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.701 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.701 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:00.701 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720452+0000", 2024-09-11T09:52:00.701 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.4595000000000003e-05 2024-09-11T09:52:00.701 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.701 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.701 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:00.701 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720493+0000", 2024-09-11T09:52:00.701 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.1782000000000001e-05 2024-09-11T09:52:00.701 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:00.701 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:00.701 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:00.702 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.702 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.702 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:52:00.702 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.711912+0000", 2024-09-11T09:52:00.702 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 108.964134, 2024-09-11T09:52:00.702 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0086864750000000008, 2024-09-11T09:52:00.702 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:00.702 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:00.702 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:00.702 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:52:00.702 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:52:00.702 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:52:00.702 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 115 2024-09-11T09:52:00.702 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.703 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:00.703 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.703 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:00.703 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711912+0000", 2024-09-11T09:52:00.703 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:00.703 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.703 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.703 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:00.703 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711912+0000", 2024-09-11T09:52:00.703 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:00.703 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.703 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.703 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:00.703 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711912+0000", 2024-09-11T09:52:00.704 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.3900000000000001e-07 2024-09-11T09:52:00.704 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.704 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.704 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:00.704 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711969+0000", 2024-09-11T09:52:00.704 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.7501999999999999e-05 2024-09-11T09:52:00.704 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.704 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.704 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:00.704 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711986+0000", 2024-09-11T09:52:00.704 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.6402999999999999e-05 2024-09-11T09:52:00.704 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.704 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.705 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:00.705 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711990+0000", 2024-09-11T09:52:00.705 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.929e-06 2024-09-11T09:52:00.705 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.705 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.705 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:00.705 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718861+0000", 2024-09-11T09:52:00.705 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0068710530000000002 2024-09-11T09:52:00.705 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.705 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.705 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:00.705 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718908+0000", 2024-09-11T09:52:00.705 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.7431e-05 2024-09-11T09:52:00.706 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.706 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.706 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:52:00.706 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718984+0000", 2024-09-11T09:52:00.706 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.5500999999999994e-05 2024-09-11T09:52:00.706 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.706 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.706 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:52:00.706 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720500+0000", 2024-09-11T09:52:00.706 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0015158820000000001 2024-09-11T09:52:00.706 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.706 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.706 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:00.706 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720536+0000", 2024-09-11T09:52:00.707 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.6118000000000002e-05 2024-09-11T09:52:00.707 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.707 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.707 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:00.707 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720570+0000", 2024-09-11T09:52:00.707 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.4387999999999997e-05 2024-09-11T09:52:00.707 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.707 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.707 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:00.707 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720598+0000", 2024-09-11T09:52:00.707 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.8028999999999999e-05 2024-09-11T09:52:00.707 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:00.707 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:00.708 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:00.708 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.708 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.708 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:52:00.708 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.711998+0000", 2024-09-11T09:52:00.708 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 108.96404782099999, 2024-09-11T09:52:00.708 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.010785813999999999, 2024-09-11T09:52:00.708 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:00.708 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:00.708 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:00.708 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:52:00.708 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:52:00.708 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:52:00.709 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 116 2024-09-11T09:52:00.709 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.709 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:00.709 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.709 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:00.709 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711998+0000", 2024-09-11T09:52:00.709 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:00.709 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.709 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.709 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:00.709 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711998+0000", 2024-09-11T09:52:00.709 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:00.709 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.709 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.710 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:00.710 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711998+0000", 2024-09-11T09:52:00.710 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.3300000000000003e-07 2024-09-11T09:52:00.710 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.710 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.710 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:00.710 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712001+0000", 2024-09-11T09:52:00.710 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.9349999999999999e-06 2024-09-11T09:52:00.710 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.710 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.710 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:00.710 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712006+0000", 2024-09-11T09:52:00.710 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.1630000000000004e-06 2024-09-11T09:52:00.711 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.711 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.711 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:00.711 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712010+0000", 2024-09-11T09:52:00.711 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.574e-06 2024-09-11T09:52:00.711 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.711 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.711 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:00.711 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719415+0000", 2024-09-11T09:52:00.711 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0074049099999999998 2024-09-11T09:52:00.711 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.711 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.711 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:00.712 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719465+0000", 2024-09-11T09:52:00.712 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.0129000000000001e-05 2024-09-11T09:52:00.712 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.712 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.712 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:52:00.712 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719545+0000", 2024-09-11T09:52:00.712 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.0253999999999994e-05 2024-09-11T09:52:00.712 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.712 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.712 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:00.712 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720636+0000", 2024-09-11T09:52:00.712 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0010909330000000001 2024-09-11T09:52:00.712 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.712 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.713 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:52:00.713 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.722707+0000", 2024-09-11T09:52:00.713 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0020713300000000001 2024-09-11T09:52:00.713 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.713 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.713 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:00.713 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.722743+0000", 2024-09-11T09:52:00.713 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.5351000000000001e-05 2024-09-11T09:52:00.713 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.713 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.713 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:00.713 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.722784+0000", 2024-09-11T09:52:00.713 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.0802e-05 2024-09-11T09:52:00.713 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:00.714 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:00.714 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:00.714 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.714 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.714 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:52:00.714 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.712014+0000", 2024-09-11T09:52:00.714 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 108.964031781, 2024-09-11T09:52:00.714 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.013420938, 2024-09-11T09:52:00.714 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:00.714 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:00.714 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:00.714 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:52:00.714 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:52:00.714 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:52:00.715 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 117 2024-09-11T09:52:00.715 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.715 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:00.715 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.715 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:00.715 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712014+0000", 2024-09-11T09:52:00.715 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:00.715 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.715 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.715 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:00.715 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712014+0000", 2024-09-11T09:52:00.715 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:00.716 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.716 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.716 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:00.716 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712014+0000", 2024-09-11T09:52:00.716 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.2000000000000001e-07 2024-09-11T09:52:00.716 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.716 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.716 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:00.716 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712016+0000", 2024-09-11T09:52:00.716 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.2309999999999999e-06 2024-09-11T09:52:00.716 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.716 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.716 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:00.717 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712020+0000", 2024-09-11T09:52:00.717 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.3830000000000001e-06 2024-09-11T09:52:00.717 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.717 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.717 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:00.717 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712023+0000", 2024-09-11T09:52:00.717 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.3340000000000002e-06 2024-09-11T09:52:00.717 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.717 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.717 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:00.717 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720083+0000", 2024-09-11T09:52:00.717 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0080601189999999993 2024-09-11T09:52:00.717 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.718 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.718 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:00.718 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720125+0000", 2024-09-11T09:52:00.718 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.2029e-05 2024-09-11T09:52:00.718 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.718 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.718 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:52:00.718 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720205+0000", 2024-09-11T09:52:00.718 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.0010000000000001e-05 2024-09-11T09:52:00.718 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.718 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.718 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:00.718 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720849+0000", 2024-09-11T09:52:00.718 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00064369899999999996 2024-09-11T09:52:00.719 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.719 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.719 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:52:00.719 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.725357+0000", 2024-09-11T09:52:00.719 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0045077379999999998 2024-09-11T09:52:00.719 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.719 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.719 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:00.719 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.725392+0000", 2024-09-11T09:52:00.719 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.5862999999999999e-05 2024-09-11T09:52:00.719 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.719 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.719 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:00.719 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.725435+0000", 2024-09-11T09:52:00.720 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.2311999999999997e-05 2024-09-11T09:52:00.720 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:00.720 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:00.720 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:00.720 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.720 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.720 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:159 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:52:00.720 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:12.452099+0000", 2024-09-11T09:52:00.720 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 108.223946193, 2024-09-11T09:52:00.720 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0069885950000000002, 2024-09-11T09:52:00.720 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:00.720 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:00.720 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:00.721 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:52:00.721 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:52:00.721 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:52:00.721 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 159 2024-09-11T09:52:00.721 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.721 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:00.721 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.721 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:00.721 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452099+0000", 2024-09-11T09:52:00.721 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:00.721 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.721 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.721 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:00.722 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452099+0000", 2024-09-11T09:52:00.722 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:00.722 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.722 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.722 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:00.722 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452100+0000", 2024-09-11T09:52:00.722 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.2800000000000003e-07 2024-09-11T09:52:00.722 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.722 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.722 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:00.722 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452102+0000", 2024-09-11T09:52:00.722 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.824e-06 2024-09-11T09:52:00.722 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.722 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.722 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:00.723 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452105+0000", 2024-09-11T09:52:00.723 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.0460000000000001e-06 2024-09-11T09:52:00.723 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.723 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.723 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:00.723 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452107+0000", 2024-09-11T09:52:00.723 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.543e-06 2024-09-11T09:52:00.723 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.723 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.723 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:00.723 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.455484+0000", 2024-09-11T09:52:00.723 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0033769759999999998 2024-09-11T09:52:00.723 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.723 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.723 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:00.724 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.455527+0000", 2024-09-11T09:52:00.724 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.2589999999999997e-05 2024-09-11T09:52:00.724 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.724 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.724 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:52:00.724 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.455594+0000", 2024-09-11T09:52:00.724 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.6775999999999999e-05 2024-09-11T09:52:00.724 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.724 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.724 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:00.724 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.456312+0000", 2024-09-11T09:52:00.724 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00071874700000000005 2024-09-11T09:52:00.724 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.724 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.725 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:52:00.725 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.459015+0000", 2024-09-11T09:52:00.725 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0027028540000000002 2024-09-11T09:52:00.725 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.725 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.725 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:00.725 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.459054+0000", 2024-09-11T09:52:00.725 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.8840000000000001e-05 2024-09-11T09:52:00.725 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.725 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.725 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:00.725 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.459088+0000", 2024-09-11T09:52:00.725 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.4071e-05 2024-09-11T09:52:00.726 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:00.726 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:00.726 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:00.726 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.726 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.726 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4688.0:38 2.5 2:abde2694:::rbd_data.1250f5e25b80.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:00.726 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:22.903926+0000", 2024-09-11T09:52:00.726 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 37.772119517, 2024-09-11T09:52:00.726 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.028303676, 2024-09-11T09:52:00.726 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:00.726 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:00.726 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:00.727 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:52:00.727 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4688", 2024-09-11T09:52:00.727 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:52:00.727 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 38 2024-09-11T09:52:00.727 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.727 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:00.727 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.727 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:00.727 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.903926+0000", 2024-09-11T09:52:00.727 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:00.727 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.727 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.727 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:00.728 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.903926+0000", 2024-09-11T09:52:00.728 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:00.728 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.728 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.728 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:00.728 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.903928+0000", 2024-09-11T09:52:00.728 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.4249999999999999e-06 2024-09-11T09:52:00.728 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.728 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.728 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:00.728 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.914774+0000", 2024-09-11T09:52:00.728 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.010846613 2024-09-11T09:52:00.728 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.728 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.729 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:00.729 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.914793+0000", 2024-09-11T09:52:00.729 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.8879999999999999e-05 2024-09-11T09:52:00.729 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.729 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.729 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:00.729 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.914801+0000", 2024-09-11T09:52:00.729 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.2230000000000007e-06 2024-09-11T09:52:00.729 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.729 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.729 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:00.729 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.914854+0000", 2024-09-11T09:52:00.729 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.2883999999999997e-05 2024-09-11T09:52:00.729 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.730 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.730 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:00.730 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.914951+0000", 2024-09-11T09:52:00.730 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.6731000000000006e-05 2024-09-11T09:52:00.730 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.730 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.730 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:52:00.730 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.915080+0000", 2024-09-11T09:52:00.730 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00012926999999999999 2024-09-11T09:52:00.730 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.730 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.730 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:00.730 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.931469+0000", 2024-09-11T09:52:00.730 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.016389107 2024-09-11T09:52:00.731 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.731 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.731 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:52:00.731 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.932118+0000", 2024-09-11T09:52:00.731 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00064915600000000002 2024-09-11T09:52:00.731 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.731 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.731 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:00.731 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.932177+0000", 2024-09-11T09:52:00.731 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.9032e-05 2024-09-11T09:52:00.731 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.731 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.732 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:00.732 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.932230+0000", 2024-09-11T09:52:00.732 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.2355000000000003e-05 2024-09-11T09:52:00.732 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:00.732 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:00.732 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:00.732 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.732 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.732 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_repop(client.4688.0:36 2.6 e53/15 2:61e1b442:::rbd_data.1250f5e25b80.0000000000000003:head v 53'2, mlcod=0'0)", 2024-09-11T09:52:00.732 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:22.915597+0000", 2024-09-11T09:52:00.732 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 37.760448967000002, 2024-09-11T09:52:00.732 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.020430798, 2024-09-11T09:52:00.732 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:00.732 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:00.733 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:00.733 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:00.733 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.733 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:00.733 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.915597+0000", 2024-09-11T09:52:00.733 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:00.733 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.733 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.733 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:00.733 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.915597+0000", 2024-09-11T09:52:00.733 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:00.733 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.733 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.733 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:00.734 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.915598+0000", 2024-09-11T09:52:00.734 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.7150000000000001e-06 2024-09-11T09:52:00.734 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.734 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.734 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:00.734 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921741+0000", 2024-09-11T09:52:00.734 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0061424339999999996 2024-09-11T09:52:00.734 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.734 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.734 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:00.734 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921757+0000", 2024-09-11T09:52:00.734 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.6144e-05 2024-09-11T09:52:00.734 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.735 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.735 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:00.735 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921765+0000", 2024-09-11T09:52:00.735 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.0809999999999994e-06 2024-09-11T09:52:00.735 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.735 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.735 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:00.735 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921826+0000", 2024-09-11T09:52:00.735 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.0878999999999998e-05 2024-09-11T09:52:00.735 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.735 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.735 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:00.735 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921849+0000", 2024-09-11T09:52:00.736 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.2966999999999999e-05 2024-09-11T09:52:00.736 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.736 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.736 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:00.736 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.935968+0000", 2024-09-11T09:52:00.736 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.014119191 2024-09-11T09:52:00.736 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.736 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.736 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:00.736 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.936027+0000", 2024-09-11T09:52:00.736 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.9386999999999999e-05 2024-09-11T09:52:00.736 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:00.736 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:00.737 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:00.737 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.737 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.737 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_repop(client.4688.0:39 2.6 e53/15 2:743e7616:::rbd_data.1250f5e25b80.0000000000000005:head v 53'3, mlcod=0'0)", 2024-09-11T09:52:00.737 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:22.921806+0000", 2024-09-11T09:52:00.737 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 37.754239917, 2024-09-11T09:52:00.737 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.015275018, 2024-09-11T09:52:00.737 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:00.737 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:00.737 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:00.737 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:00.737 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.737 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:00.737 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921806+0000", 2024-09-11T09:52:00.738 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:00.738 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.738 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.738 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:00.738 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921806+0000", 2024-09-11T09:52:00.738 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:00.738 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.738 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.739 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:00.739 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921806+0000", 2024-09-11T09:52:00.740 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.2200000000000003e-07 2024-09-11T09:52:00.740 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.740 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.740 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:00.740 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.925590+0000", 2024-09-11T09:52:00.740 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0037838949999999998 2024-09-11T09:52:00.740 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.740 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.740 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:00.740 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.925605+0000", 2024-09-11T09:52:00.740 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.4321999999999999e-05 2024-09-11T09:52:00.740 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.741 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.741 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:00.741 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.925613+0000", 2024-09-11T09:52:00.741 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.1980000000000001e-06 2024-09-11T09:52:00.741 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.741 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.741 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:00.741 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.925857+0000", 2024-09-11T09:52:00.741 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000244064 2024-09-11T09:52:00.741 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.741 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.741 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:00.742 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.925875+0000", 2024-09-11T09:52:00.742 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.838e-05 2024-09-11T09:52:00.742 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.742 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.742 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:00.742 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.937042+0000", 2024-09-11T09:52:00.742 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.011166416 2024-09-11T09:52:00.742 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.742 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.742 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:00.742 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.937081+0000", 2024-09-11T09:52:00.742 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.9020999999999998e-05 2024-09-11T09:52:00.742 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:00.743 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:00.743 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:00.743 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.743 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.743 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4688.0:49 2.5 2:b6dd9fc9:::rbd_data.1250f5e25b80.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:00.743 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:22.938548+0000", 2024-09-11T09:52:00.743 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 37.737497914000002, 2024-09-11T09:52:00.743 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.022766950000000001, 2024-09-11T09:52:00.743 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:00.743 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:00.743 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:00.743 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:52:00.743 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4688", 2024-09-11T09:52:00.744 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:52:00.744 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 49 2024-09-11T09:52:00.744 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.744 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:00.744 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.744 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:00.744 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.938548+0000", 2024-09-11T09:52:00.744 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:00.744 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.744 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.744 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:00.744 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.938548+0000", 2024-09-11T09:52:00.744 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:00.745 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.745 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.745 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:00.745 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.938550+0000", 2024-09-11T09:52:00.745 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.7710000000000001e-06 2024-09-11T09:52:00.745 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.745 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.745 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:00.745 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.944237+0000", 2024-09-11T09:52:00.745 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0056879790000000001 2024-09-11T09:52:00.745 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.745 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.745 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:00.745 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.944262+0000", 2024-09-11T09:52:00.746 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.4247e-05 2024-09-11T09:52:00.746 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.746 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.746 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:00.746 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.944274+0000", 2024-09-11T09:52:00.746 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.2639999999999999e-05 2024-09-11T09:52:00.746 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.746 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.746 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:00.746 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.944344+0000", 2024-09-11T09:52:00.746 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.9574999999999997e-05 2024-09-11T09:52:00.746 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.746 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.747 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:00.747 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.944474+0000", 2024-09-11T09:52:00.747 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000130464 2024-09-11T09:52:00.747 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.747 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.747 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:52:00.747 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.944585+0000", 2024-09-11T09:52:00.747 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000110096 2024-09-11T09:52:00.747 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.747 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.747 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:52:00.747 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.952724+0000", 2024-09-11T09:52:00.747 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0081398900000000003 2024-09-11T09:52:00.748 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.748 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.748 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:00.748 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.961219+0000", 2024-09-11T09:52:00.748 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0084949789999999997 2024-09-11T09:52:00.748 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.748 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.748 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:00.748 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.961280+0000", 2024-09-11T09:52:00.748 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.0404e-05 2024-09-11T09:52:00.748 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.748 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.748 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:00.749 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.961315+0000", 2024-09-11T09:52:00.749 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.4904999999999999e-05 2024-09-11T09:52:00.749 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:00.749 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:00.749 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:00.749 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.749 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.749 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_repop(client.4688.0:44 2.6 e53/15 2:67246a22:::rbd_data.1250f5e25b80.0000000000000007:head v 53'4, mlcod=53'2)", 2024-09-11T09:52:00.749 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:22.951721+0000", 2024-09-11T09:52:00.749 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 37.724324641999999, 2024-09-11T09:52:00.749 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.01080981, 2024-09-11T09:52:00.749 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:00.749 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:00.749 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:00.750 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:00.750 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.750 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:00.750 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.951721+0000", 2024-09-11T09:52:00.750 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:00.750 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.750 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.750 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:00.750 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.951721+0000", 2024-09-11T09:52:00.750 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:00.750 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.750 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.750 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:00.750 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.951722+0000", 2024-09-11T09:52:00.751 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.0279999999999999e-06 2024-09-11T09:52:00.751 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.751 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.751 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:00.751 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.955612+0000", 2024-09-11T09:52:00.751 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0038896920000000001 2024-09-11T09:52:00.751 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.751 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.751 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:00.751 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.955628+0000", 2024-09-11T09:52:00.751 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.5945000000000001e-05 2024-09-11T09:52:00.751 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.751 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.752 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:00.752 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.955635+0000", 2024-09-11T09:52:00.752 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.8080000000000005e-06 2024-09-11T09:52:00.752 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.752 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.752 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:00.752 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.955700+0000", 2024-09-11T09:52:00.752 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.4114000000000003e-05 2024-09-11T09:52:00.752 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.752 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.752 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:00.752 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.955724+0000", 2024-09-11T09:52:00.752 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.3989999999999999e-05 2024-09-11T09:52:00.753 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.753 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.753 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:00.753 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.962490+0000", 2024-09-11T09:52:00.753 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0067662599999999996 2024-09-11T09:52:00.753 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.753 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.753 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:00.753 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.962531+0000", 2024-09-11T09:52:00.753 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.0973000000000002e-05 2024-09-11T09:52:00.753 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:00.753 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:00.753 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:00.754 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.754 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.754 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_repop(client.4688.0:51 2.6 e53/15 2:6b5894fc:::rbd_data.1250f5e25b80.000000000000000a:head v 53'5, mlcod=53'2)", 2024-09-11T09:52:00.754 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:22.959030+0000", 2024-09-11T09:52:00.754 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 37.717015468, 2024-09-11T09:52:00.754 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.011503638, 2024-09-11T09:52:00.754 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:00.754 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:00.754 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:00.754 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:00.754 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.754 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:00.754 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.959030+0000", 2024-09-11T09:52:00.755 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:00.755 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.755 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.755 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:00.755 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.959030+0000", 2024-09-11T09:52:00.755 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:00.755 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.755 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.755 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:00.755 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.959032+0000", 2024-09-11T09:52:00.755 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.356e-06 2024-09-11T09:52:00.755 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.755 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.756 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:00.756 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.962899+0000", 2024-09-11T09:52:00.756 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.003867639 2024-09-11T09:52:00.756 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.756 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.756 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:00.756 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.962909+0000", 2024-09-11T09:52:00.756 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.0105999999999999e-05 2024-09-11T09:52:00.756 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.756 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.756 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:00.756 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.962915+0000", 2024-09-11T09:52:00.756 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.8869999999999997e-06 2024-09-11T09:52:00.757 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.757 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.757 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:00.757 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.962962+0000", 2024-09-11T09:52:00.757 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.6511999999999998e-05 2024-09-11T09:52:00.757 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.757 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.757 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:00.757 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.962989+0000", 2024-09-11T09:52:00.757 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.7702999999999999e-05 2024-09-11T09:52:00.757 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.757 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.757 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:00.757 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.970495+0000", 2024-09-11T09:52:00.758 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0075052449999999998 2024-09-11T09:52:00.758 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.758 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.758 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:00.758 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.970534+0000", 2024-09-11T09:52:00.758 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.9190000000000003e-05 2024-09-11T09:52:00.758 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:00.758 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:00.758 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:00.758 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.758 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.758 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_repop(client.4688.0:58 2.6 e53/15 2:6277c644:::rbd_data.1250f5e25b80.000000000000000e:head v 53'6, mlcod=53'4)", 2024-09-11T09:52:00.758 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:22.996199+0000", 2024-09-11T09:52:00.758 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 37.679846548, 2024-09-11T09:52:00.758 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.010145573999999999, 2024-09-11T09:52:00.759 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:00.759 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:00.759 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:00.759 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:00.759 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.759 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:00.759 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.996199+0000", 2024-09-11T09:52:00.759 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:00.759 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.759 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.759 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:00.759 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.996199+0000", 2024-09-11T09:52:00.759 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:00.760 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.760 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.760 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:00.760 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.996200+0000", 2024-09-11T09:52:00.760 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.85e-07 2024-09-11T09:52:00.760 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.760 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.760 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:00.760 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.999914+0000", 2024-09-11T09:52:00.760 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.003714406 2024-09-11T09:52:00.760 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.760 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.760 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:00.760 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.999928+0000", 2024-09-11T09:52:00.761 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.3149000000000001e-05 2024-09-11T09:52:00.761 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.761 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.761 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:00.761 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.999951+0000", 2024-09-11T09:52:00.761 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.3116e-05 2024-09-11T09:52:00.761 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.761 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.761 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:00.761 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.000003+0000", 2024-09-11T09:52:00.761 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.2169000000000001e-05 2024-09-11T09:52:00.761 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.761 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.762 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:00.762 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.000018+0000", 2024-09-11T09:52:00.762 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.5500000000000001e-05 2024-09-11T09:52:00.762 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.762 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.762 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:00.762 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.006303+0000", 2024-09-11T09:52:00.762 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0062846710000000004 2024-09-11T09:52:00.762 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.762 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.762 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:00.762 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.006345+0000", 2024-09-11T09:52:00.762 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.1678000000000003e-05 2024-09-11T09:52:00.763 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:00.763 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:00.763 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:00.763 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.763 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.763 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4694.0:42 2.5 2:a4fc00ec:ns::rbd_data.1256a0c02879.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:00.763 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.111570+0000", 2024-09-11T09:52:00.763 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 37.564475442000003, 2024-09-11T09:52:00.763 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.038002197000000001, 2024-09-11T09:52:00.763 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:00.763 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:00.763 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:00.764 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:52:00.764 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4694", 2024-09-11T09:52:00.764 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:52:00.764 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 42 2024-09-11T09:52:00.764 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.764 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:00.764 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.764 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:00.764 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.111570+0000", 2024-09-11T09:52:00.764 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:00.764 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.765 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.765 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:00.765 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.111570+0000", 2024-09-11T09:52:00.765 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:00.765 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.765 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.765 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:00.765 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.111572+0000", 2024-09-11T09:52:00.765 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.3909999999999999e-06 2024-09-11T09:52:00.765 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.765 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.765 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:00.766 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.135929+0000", 2024-09-11T09:52:00.766 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.024357410999999999 2024-09-11T09:52:00.766 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.766 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.766 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:00.766 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.135951+0000", 2024-09-11T09:52:00.766 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.1775000000000001e-05 2024-09-11T09:52:00.766 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.766 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.766 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:00.766 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.135962+0000", 2024-09-11T09:52:00.766 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.0903e-05 2024-09-11T09:52:00.766 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.766 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.766 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:00.767 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.136045+0000", 2024-09-11T09:52:00.767 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.3128999999999996e-05 2024-09-11T09:52:00.767 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.767 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.767 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:00.767 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.136229+0000", 2024-09-11T09:52:00.767 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00018393700000000001 2024-09-11T09:52:00.767 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.767 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.767 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:52:00.767 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.136394+0000", 2024-09-11T09:52:00.767 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000165527 2024-09-11T09:52:00.767 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.767 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.768 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:00.768 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.149320+0000", 2024-09-11T09:52:00.768 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.01292568 2024-09-11T09:52:00.768 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.768 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.768 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:52:00.768 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.149493+0000", 2024-09-11T09:52:00.768 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00017285999999999999 2024-09-11T09:52:00.768 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.768 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.768 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:00.768 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.149539+0000", 2024-09-11T09:52:00.768 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.6628000000000001e-05 2024-09-11T09:52:00.768 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.769 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.769 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:00.769 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.149572+0000", 2024-09-11T09:52:00.769 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.2956000000000001e-05 2024-09-11T09:52:00.769 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:00.769 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:00.769 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:00.769 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.769 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.769 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_repop(client.4694.0:45 2.6 e53/15 2:671fb830:ns::rbd_data.1256a0c02879.0000000000000007:head v 53'9, mlcod=53'7)", 2024-09-11T09:52:00.769 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.121833+0000", 2024-09-11T09:52:00.769 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 37.554213083999997, 2024-09-11T09:52:00.769 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.023876192000000001, 2024-09-11T09:52:00.770 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:00.770 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:00.770 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:00.770 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:00.770 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.770 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:00.770 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.121833+0000", 2024-09-11T09:52:00.770 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:00.770 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.770 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.770 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:00.770 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.121833+0000", 2024-09-11T09:52:00.770 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:00.770 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.771 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.771 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:00.771 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.121834+0000", 2024-09-11T09:52:00.771 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.471e-06 2024-09-11T09:52:00.771 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.771 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.771 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:00.771 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.128990+0000", 2024-09-11T09:52:00.771 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0071557879999999997 2024-09-11T09:52:00.771 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.771 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.771 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:00.771 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.129011+0000", 2024-09-11T09:52:00.772 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.0760000000000001e-05 2024-09-11T09:52:00.772 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.772 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.772 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:00.772 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.129022+0000", 2024-09-11T09:52:00.772 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1056000000000001e-05 2024-09-11T09:52:00.772 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.772 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.772 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:00.772 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.129097+0000", 2024-09-11T09:52:00.772 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.5082999999999999e-05 2024-09-11T09:52:00.772 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.772 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.773 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:00.773 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.129130+0000", 2024-09-11T09:52:00.773 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.3345e-05 2024-09-11T09:52:00.773 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.773 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.773 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:00.773 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.145657+0000", 2024-09-11T09:52:00.773 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.016527224 2024-09-11T09:52:00.773 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.773 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.773 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:00.773 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.145709+0000", 2024-09-11T09:52:00.774 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.1465000000000003e-05 2024-09-11T09:52:00.774 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:00.774 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:00.774 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:00.774 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.774 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.774 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4694.0:43 2.0 2:16c8ae84:ns::rbd_data.1256a0c02879.0000000000000005:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:00.774 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.135999+0000", 2024-09-11T09:52:00.774 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 37.540047133999998, 2024-09-11T09:52:00.774 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.039468309, 2024-09-11T09:52:00.774 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:00.774 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:00.774 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:00.775 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:52:00.775 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4694", 2024-09-11T09:52:00.775 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:52:00.775 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 43 2024-09-11T09:52:00.775 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.775 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:00.775 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.775 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:00.775 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.135999+0000", 2024-09-11T09:52:00.775 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:00.775 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.775 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.775 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:00.775 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.135999+0000", 2024-09-11T09:52:00.776 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:00.776 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.776 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.776 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:00.776 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.136000+0000", 2024-09-11T09:52:00.776 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.6330000000000001e-06 2024-09-11T09:52:00.776 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.776 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.776 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:00.776 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.157961+0000", 2024-09-11T09:52:00.776 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0219606 2024-09-11T09:52:00.776 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.777 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.777 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:00.777 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.157995+0000", 2024-09-11T09:52:00.777 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.4391999999999999e-05 2024-09-11T09:52:00.777 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.777 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.777 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:00.777 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.158011+0000", 2024-09-11T09:52:00.777 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.5794000000000002e-05 2024-09-11T09:52:00.777 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.777 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.777 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:00.777 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.158083+0000", 2024-09-11T09:52:00.778 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.2273000000000004e-05 2024-09-11T09:52:00.778 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.778 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.778 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:00.778 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.158239+0000", 2024-09-11T09:52:00.778 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000155372 2024-09-11T09:52:00.778 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.778 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.778 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 1", 2024-09-11T09:52:00.778 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.158403+0000", 2024-09-11T09:52:00.778 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00016408199999999999 2024-09-11T09:52:00.778 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.778 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.778 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:52:00.779 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.173860+0000", 2024-09-11T09:52:00.779 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.015457347 2024-09-11T09:52:00.779 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.779 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.779 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:00.779 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.175400+0000", 2024-09-11T09:52:00.779 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001540006 2024-09-11T09:52:00.779 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.779 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.779 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:00.779 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.175438+0000", 2024-09-11T09:52:00.779 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.8135e-05 2024-09-11T09:52:00.779 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.780 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.780 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:00.780 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.175467+0000", 2024-09-11T09:52:00.780 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.8674999999999999e-05 2024-09-11T09:52:00.780 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:00.780 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:00.780 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:00.780 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.780 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.780 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4694.0:57 2.0 2:04b5549c:ns::rbd_data.1256a0c02879.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:00.780 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.158037+0000", 2024-09-11T09:52:00.780 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 37.518008395999999, 2024-09-11T09:52:00.780 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.027748790999999998, 2024-09-11T09:52:00.781 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:00.781 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:00.781 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:00.781 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:52:00.781 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4694", 2024-09-11T09:52:00.781 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:52:00.781 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 57 2024-09-11T09:52:00.781 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.781 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:00.781 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.781 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:00.781 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.158037+0000", 2024-09-11T09:52:00.781 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:00.781 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.782 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.782 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:00.782 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.158037+0000", 2024-09-11T09:52:00.782 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:00.782 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.782 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.782 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:00.782 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.158039+0000", 2024-09-11T09:52:00.782 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.356e-06 2024-09-11T09:52:00.782 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.782 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.782 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:00.782 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.176367+0000", 2024-09-11T09:52:00.782 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.018328739 2024-09-11T09:52:00.783 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.783 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.783 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:00.783 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.176378+0000", 2024-09-11T09:52:00.783 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.0713999999999999e-05 2024-09-11T09:52:00.783 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.783 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.783 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:00.783 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.176383+0000", 2024-09-11T09:52:00.783 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.0209999999999999e-06 2024-09-11T09:52:00.783 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.783 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.783 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:00.784 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.176426+0000", 2024-09-11T09:52:00.784 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.2623999999999997e-05 2024-09-11T09:52:00.784 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.784 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.784 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:00.784 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.176552+0000", 2024-09-11T09:52:00.784 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000126076 2024-09-11T09:52:00.784 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.784 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.784 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 1", 2024-09-11T09:52:00.784 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.176665+0000", 2024-09-11T09:52:00.784 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000112995 2024-09-11T09:52:00.784 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.784 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.785 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:52:00.785 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.183639+0000", 2024-09-11T09:52:00.785 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0069740039999999998 2024-09-11T09:52:00.785 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.785 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.785 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:00.785 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.185689+0000", 2024-09-11T09:52:00.785 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.002049852 2024-09-11T09:52:00.785 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.785 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.785 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:00.785 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.185725+0000", 2024-09-11T09:52:00.785 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.6804999999999997e-05 2024-09-11T09:52:00.785 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.786 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.786 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:00.786 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.185786+0000", 2024-09-11T09:52:00.786 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.0605e-05 2024-09-11T09:52:00.786 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:00.786 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:00.786 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:00.786 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.786 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.786 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4694.0:60 2.0 2:054f2af4:ns::rbd_data.1256a0c02879.000000000000000f:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:00.786 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.186855+0000", 2024-09-11T09:52:00.786 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 37.489190379999997, 2024-09-11T09:52:00.786 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.013682913, 2024-09-11T09:52:00.787 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:00.787 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:00.787 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:00.787 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:52:00.787 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4694", 2024-09-11T09:52:00.787 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:52:00.787 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 60 2024-09-11T09:52:00.787 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.787 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:00.787 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.787 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:00.787 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.186855+0000", 2024-09-11T09:52:00.787 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:00.788 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.788 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.788 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:00.788 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.186855+0000", 2024-09-11T09:52:00.788 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:00.788 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.788 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.788 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:00.788 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.186856+0000", 2024-09-11T09:52:00.788 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.1200000000000001e-07 2024-09-11T09:52:00.788 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.788 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.788 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:00.788 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.191484+0000", 2024-09-11T09:52:00.789 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0046274159999999996 2024-09-11T09:52:00.789 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.789 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.789 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:00.789 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.191496+0000", 2024-09-11T09:52:00.789 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.2149e-05 2024-09-11T09:52:00.789 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.789 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.789 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:00.789 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.191502+0000", 2024-09-11T09:52:00.789 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.1890000000000004e-06 2024-09-11T09:52:00.789 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.790 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.790 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:00.790 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.191544+0000", 2024-09-11T09:52:00.790 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.1724999999999998e-05 2024-09-11T09:52:00.790 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.790 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.790 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:00.790 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.191641+0000", 2024-09-11T09:52:00.790 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.6939000000000001e-05 2024-09-11T09:52:00.790 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.790 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.790 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 1", 2024-09-11T09:52:00.790 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.191731+0000", 2024-09-11T09:52:00.790 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.0651999999999999e-05 2024-09-11T09:52:00.790 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.791 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.791 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:52:00.791 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.199852+0000", 2024-09-11T09:52:00.791 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0081204529999999997 2024-09-11T09:52:00.791 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.791 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.791 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:00.791 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.200461+0000", 2024-09-11T09:52:00.791 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00060900899999999998 2024-09-11T09:52:00.791 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.791 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.791 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:00.791 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.200497+0000", 2024-09-11T09:52:00.791 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.5800000000000003e-05 2024-09-11T09:52:00.792 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:00.792 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:00.792 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:00.792 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.200538+0000", 2024-09-11T09:52:00.792 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.1668999999999997e-05 2024-09-11T09:52:00.792 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:00.792 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:00.792 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:00.792 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:00.792 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:00.792 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-11T09:52:00.794 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-11T09:52:00.794 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:52:00.794+0000 7f3c96ec9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31909) UID: 0 2024-09-11T09:52:00.811 INFO:teuthology.orchestra.run.smithi017.stderr:clear 2.7 pg_upmap_items mapping 2024-09-11T09:52:00.812 INFO:teuthology.orchestra.run.smithi017.stderr:nodeep-scrub is unset 2024-09-11T09:52:00.895 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:52:00.896+0000 7f3c96ec9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31909) UID: 0 2024-09-11T09:52:00.937 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-11T09:52:00.938 INFO:teuthology.orchestra.run.smithi096.stdout: "ops": [], 2024-09-11T09:52:00.938 INFO:teuthology.orchestra.run.smithi096.stdout: "num_ops": 0 2024-09-11T09:52:00.938 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-11T09:52:00.946 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-11T09:52:00.996 INFO:tasks.ceph.osd.0.smithi017.stderr:2024-09-11T09:52:00.997+0000 7fb73c0f1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31665) UID: 0 2024-09-11T09:52:01.089 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-11T09:52:01.089 INFO:teuthology.orchestra.run.smithi096.stdout: "ops": [], 2024-09-11T09:52:01.089 INFO:teuthology.orchestra.run.smithi096.stdout: "complaint_time": 30, 2024-09-11T09:52:01.089 INFO:teuthology.orchestra.run.smithi096.stdout: "num_blocked_ops": 0 2024-09-11T09:52:01.089 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-11T09:52:01.096 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:52:01.097+0000 7f3a7b90a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 42475) UID: 0 2024-09-11T09:52:01.097 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-11T09:52:01.197 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:52:01.198+0000 7f3a7b90a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 42475) UID: 0 2024-09-11T09:52:01.242 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-11T09:52:01.242 INFO:teuthology.orchestra.run.smithi096.stdout: "size": 20, 2024-09-11T09:52:01.242 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 600, 2024-09-11T09:52:01.242 INFO:teuthology.orchestra.run.smithi096.stdout: "ops": [ 2024-09-11T09:52:01.242 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.242 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4230.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-11T09:52:01.242 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:18.490736+0000", 2024-09-11T09:52:01.243 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 102.75123408, 2024-09-11T09:52:01.243 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0059097569999999999, 2024-09-11T09:52:01.243 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:01.243 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:01.243 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:01.243 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:52:01.243 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4230", 2024-09-11T09:52:01.243 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.17:0/4101109576", 2024-09-11T09:52:01.243 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 1 2024-09-11T09:52:01.243 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.243 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:01.243 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.243 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:01.243 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.490736+0000", 2024-09-11T09:52:01.244 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:01.244 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.244 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.244 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:01.244 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.490736+0000", 2024-09-11T09:52:01.244 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:01.244 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.244 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.244 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:01.244 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.490742+0000", 2024-09-11T09:52:01.244 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.48e-06 2024-09-11T09:52:01.244 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.244 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.245 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:01.245 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.490749+0000", 2024-09-11T09:52:01.245 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.4569999999999999e-06 2024-09-11T09:52:01.245 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.245 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.245 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:01.245 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.490762+0000", 2024-09-11T09:52:01.245 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.2721000000000001e-05 2024-09-11T09:52:01.245 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.245 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.245 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:01.245 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.490774+0000", 2024-09-11T09:52:01.245 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1823e-05 2024-09-11T09:52:01.245 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.246 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.246 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:01.246 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.490846+0000", 2024-09-11T09:52:01.246 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.1878000000000005e-05 2024-09-11T09:52:01.246 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.246 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.246 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for peered", 2024-09-11T09:52:01.246 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.490857+0000", 2024-09-11T09:52:01.246 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1017e-05 2024-09-11T09:52:01.246 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.246 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.246 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:01.246 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.493856+0000", 2024-09-11T09:52:01.247 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00299885 2024-09-11T09:52:01.247 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.247 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.247 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:01.247 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.494020+0000", 2024-09-11T09:52:01.247 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00016436599999999999 2024-09-11T09:52:01.247 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.247 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.247 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 2", 2024-09-11T09:52:01.247 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.494186+0000", 2024-09-11T09:52:01.247 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00016562799999999999 2024-09-11T09:52:01.247 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.247 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.247 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:01.248 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.495407+0000", 2024-09-11T09:52:01.248 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0012217420000000001 2024-09-11T09:52:01.248 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.248 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.248 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:52:01.248 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.496579+0000", 2024-09-11T09:52:01.248 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001171844 2024-09-11T09:52:01.248 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.248 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.248 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:01.248 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.496620+0000", 2024-09-11T09:52:01.248 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.0479000000000003e-05 2024-09-11T09:52:01.248 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.249 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.249 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:01.249 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.496646+0000", 2024-09-11T09:52:01.249 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.6472e-05 2024-09-11T09:52:01.249 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:01.249 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:01.249 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:01.249 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.249 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.249 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4230.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-09-11T09:52:01.249 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:20.498634+0000", 2024-09-11T09:52:01.249 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 100.743336059, 2024-09-11T09:52:01.249 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0050648539999999997, 2024-09-11T09:52:01.250 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:01.250 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:01.250 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:01.250 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:52:01.250 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4230", 2024-09-11T09:52:01.250 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.17:0/4101109576", 2024-09-11T09:52:01.250 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 7 2024-09-11T09:52:01.250 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.250 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:01.250 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.250 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:01.250 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.498634+0000", 2024-09-11T09:52:01.250 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:01.251 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.251 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.251 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:01.251 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.498634+0000", 2024-09-11T09:52:01.251 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:01.251 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.251 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.251 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:01.251 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.498637+0000", 2024-09-11T09:52:01.251 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.1919999999999999e-06 2024-09-11T09:52:01.251 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.251 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.251 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:01.252 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.498648+0000", 2024-09-11T09:52:01.252 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1116e-05 2024-09-11T09:52:01.252 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.252 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.252 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:01.252 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.498666+0000", 2024-09-11T09:52:01.252 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.8411999999999999e-05 2024-09-11T09:52:01.252 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.252 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.252 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:01.252 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.498675+0000", 2024-09-11T09:52:01.252 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.4059999999999994e-06 2024-09-11T09:52:01.253 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.253 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.253 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:01.253 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.498745+0000", 2024-09-11T09:52:01.253 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.9764000000000005e-05 2024-09-11T09:52:01.253 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.253 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.253 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op must wait for map", 2024-09-11T09:52:01.253 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.498754+0000", 2024-09-11T09:52:01.253 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.5969999999999992e-06 2024-09-11T09:52:01.253 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.253 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.253 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:01.254 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.500385+0000", 2024-09-11T09:52:01.254 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0016308360000000001 2024-09-11T09:52:01.254 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.254 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.254 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:01.254 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.500473+0000", 2024-09-11T09:52:01.254 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.7966000000000004e-05 2024-09-11T09:52:01.254 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.254 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.254 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 1", 2024-09-11T09:52:01.254 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.500591+0000", 2024-09-11T09:52:01.254 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000117895 2024-09-11T09:52:01.255 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.255 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.255 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:01.255 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.501685+0000", 2024-09-11T09:52:01.255 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001094886 2024-09-11T09:52:01.255 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.255 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.255 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:52:01.255 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.503627+0000", 2024-09-11T09:52:01.255 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001941199 2024-09-11T09:52:01.255 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.255 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.255 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:01.256 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.503667+0000", 2024-09-11T09:52:01.256 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.0645000000000001e-05 2024-09-11T09:52:01.256 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.256 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.256 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:01.256 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.503699+0000", 2024-09-11T09:52:01.256 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.1940000000000003e-05 2024-09-11T09:52:01.256 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:01.256 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:01.256 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:01.256 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.256 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.256 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4688.0:32 2.2 2:45941fd2:::rbd_data.1250f5e25b80.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:01.257 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:22.896607+0000", 2024-09-11T09:52:01.257 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 38.345363456000001, 2024-09-11T09:52:01.257 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.038655179999999997, 2024-09-11T09:52:01.257 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:01.257 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:01.257 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:01.257 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:52:01.258 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4688", 2024-09-11T09:52:01.258 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:52:01.258 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 32 2024-09-11T09:52:01.258 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.258 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:01.258 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.258 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:01.258 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.896607+0000", 2024-09-11T09:52:01.258 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:01.258 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.258 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.258 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:01.258 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.896607+0000", 2024-09-11T09:52:01.259 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:01.259 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.259 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.259 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:01.259 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.896608+0000", 2024-09-11T09:52:01.259 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.0690000000000001e-06 2024-09-11T09:52:01.259 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.259 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.259 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:01.259 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.914461+0000", 2024-09-11T09:52:01.259 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.017852903 2024-09-11T09:52:01.259 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.259 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.260 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:01.260 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.914495+0000", 2024-09-11T09:52:01.260 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.4301000000000002e-05 2024-09-11T09:52:01.260 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.260 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.260 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:01.260 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.914522+0000", 2024-09-11T09:52:01.260 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.7127e-05 2024-09-11T09:52:01.260 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.260 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.260 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:01.260 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.914583+0000", 2024-09-11T09:52:01.260 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.0730999999999998e-05 2024-09-11T09:52:01.261 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.261 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.261 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:01.261 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.914709+0000", 2024-09-11T09:52:01.261 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00012590400000000001 2024-09-11T09:52:01.261 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.261 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.261 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 1", 2024-09-11T09:52:01.261 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.914809+0000", 2024-09-11T09:52:01.261 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.9882000000000004e-05 2024-09-11T09:52:01.261 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.261 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.261 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:01.261 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.932551+0000", 2024-09-11T09:52:01.262 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.017742048999999999 2024-09-11T09:52:01.262 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.262 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.262 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:52:01.262 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.935144+0000", 2024-09-11T09:52:01.262 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0025927630000000001 2024-09-11T09:52:01.262 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.262 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.262 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:01.262 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.935205+0000", 2024-09-11T09:52:01.262 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.1634e-05 2024-09-11T09:52:01.262 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.263 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.263 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:01.263 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.935262+0000", 2024-09-11T09:52:01.263 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.6817000000000002e-05 2024-09-11T09:52:01.263 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:01.263 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:01.263 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:01.263 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.263 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.263 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_repop(client.4688.0:40 2.7 e53/38 2:fe41411c:::rbd_object_map.1250f5e25b80:head v 53'9, mlcod=53'7)", 2024-09-11T09:52:01.263 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:22.920457+0000", 2024-09-11T09:52:01.263 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 38.321513365000001, 2024-09-11T09:52:01.264 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.011132817, 2024-09-11T09:52:01.264 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:01.264 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:01.264 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:01.264 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:01.264 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.264 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:01.264 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.920457+0000", 2024-09-11T09:52:01.264 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:01.264 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.264 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.264 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:01.264 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.920457+0000", 2024-09-11T09:52:01.265 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:01.265 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.265 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.265 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:01.265 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.920459+0000", 2024-09-11T09:52:01.265 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.6729999999999999e-06 2024-09-11T09:52:01.265 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.265 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.265 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:01.266 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.920469+0000", 2024-09-11T09:52:01.266 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.944e-06 2024-09-11T09:52:01.266 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.266 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.266 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:01.266 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.920495+0000", 2024-09-11T09:52:01.266 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.6401999999999999e-05 2024-09-11T09:52:01.266 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.266 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.266 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:01.266 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.920506+0000", 2024-09-11T09:52:01.266 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.133e-05 2024-09-11T09:52:01.267 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.267 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.267 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:01.267 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.920581+0000", 2024-09-11T09:52:01.267 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.4932999999999996e-05 2024-09-11T09:52:01.267 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.267 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.267 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:01.267 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.920612+0000", 2024-09-11T09:52:01.267 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.0537999999999998e-05 2024-09-11T09:52:01.267 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.267 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.267 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:01.268 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.931537+0000", 2024-09-11T09:52:01.268 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.010925239 2024-09-11T09:52:01.268 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.268 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.268 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:01.268 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.931590+0000", 2024-09-11T09:52:01.268 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.2757999999999999e-05 2024-09-11T09:52:01.268 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:01.268 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:01.268 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:01.268 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.268 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.268 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_repop(client.4688.0:41 2.7 e53/38 2:fe41411c:::rbd_object_map.1250f5e25b80:head v 53'10, mlcod=53'7)", 2024-09-11T09:52:01.269 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:22.921422+0000", 2024-09-11T09:52:01.269 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 38.320547949000002, 2024-09-11T09:52:01.269 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.013734942, 2024-09-11T09:52:01.269 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:01.269 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:01.269 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:01.269 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:01.269 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.269 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:01.269 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921422+0000", 2024-09-11T09:52:01.269 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:01.269 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.270 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.270 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:01.270 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921422+0000", 2024-09-11T09:52:01.270 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:01.270 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.270 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.270 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:01.270 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921426+0000", 2024-09-11T09:52:01.270 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.6150000000000001e-06 2024-09-11T09:52:01.270 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.270 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.270 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:01.270 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921435+0000", 2024-09-11T09:52:01.270 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.9840000000000007e-06 2024-09-11T09:52:01.271 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.271 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.271 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:01.271 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921444+0000", 2024-09-11T09:52:01.271 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.6000000000000007e-06 2024-09-11T09:52:01.271 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.271 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.271 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:01.271 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921450+0000", 2024-09-11T09:52:01.271 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.4690000000000002e-06 2024-09-11T09:52:01.271 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.271 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.271 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:01.271 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921511+0000", 2024-09-11T09:52:01.272 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.0779999999999997e-05 2024-09-11T09:52:01.272 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.272 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.272 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:01.272 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921531+0000", 2024-09-11T09:52:01.272 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.0302999999999999e-05 2024-09-11T09:52:01.272 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.272 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.272 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:01.272 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.935115+0000", 2024-09-11T09:52:01.272 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.013583471 2024-09-11T09:52:01.272 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.272 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.272 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:01.273 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.935157+0000", 2024-09-11T09:52:01.273 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.2719999999999998e-05 2024-09-11T09:52:01.273 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:01.273 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:01.273 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:01.273 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.273 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.273 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4688.0:42 2.3 2:c54742d8:::rbd_data.1250f5e25b80.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:01.273 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:22.932560+0000", 2024-09-11T09:52:01.273 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 38.309410206000003, 2024-09-11T09:52:01.273 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.017798198000000001, 2024-09-11T09:52:01.273 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:01.274 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:01.274 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:01.274 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:52:01.274 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4688", 2024-09-11T09:52:01.274 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:52:01.274 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 42 2024-09-11T09:52:01.274 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.274 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:01.274 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.274 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:01.274 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.932560+0000", 2024-09-11T09:52:01.274 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:01.274 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.275 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.275 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:01.275 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.932560+0000", 2024-09-11T09:52:01.275 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:01.275 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.275 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.275 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:01.275 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.932562+0000", 2024-09-11T09:52:01.275 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.17e-06 2024-09-11T09:52:01.275 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.275 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.275 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:01.275 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.938851+0000", 2024-09-11T09:52:01.275 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.006288971 2024-09-11T09:52:01.276 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.276 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.276 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:01.276 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.938865+0000", 2024-09-11T09:52:01.276 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.365e-05 2024-09-11T09:52:01.276 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.276 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.276 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:01.276 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.938871+0000", 2024-09-11T09:52:01.276 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.8850000000000004e-06 2024-09-11T09:52:01.276 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.276 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.276 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:01.277 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.938926+0000", 2024-09-11T09:52:01.277 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.4691999999999997e-05 2024-09-11T09:52:01.277 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.277 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.277 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:01.277 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.939059+0000", 2024-09-11T09:52:01.277 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00013310899999999999 2024-09-11T09:52:01.277 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.277 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.277 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 2", 2024-09-11T09:52:01.277 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.939171+0000", 2024-09-11T09:52:01.277 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000112718 2024-09-11T09:52:01.277 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.278 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.278 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:01.278 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.950262+0000", 2024-09-11T09:52:01.278 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.011091063 2024-09-11T09:52:01.278 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.278 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.278 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:52:01.278 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.950274+0000", 2024-09-11T09:52:01.278 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1963e-05 2024-09-11T09:52:01.278 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.278 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.278 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:01.278 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.950318+0000", 2024-09-11T09:52:01.279 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.3519999999999997e-05 2024-09-11T09:52:01.279 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.279 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.279 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:01.279 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.950358+0000", 2024-09-11T09:52:01.279 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.0457000000000002e-05 2024-09-11T09:52:01.279 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:01.279 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:01.279 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:01.280 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.280 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.280 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_repop(client.4694.0:34 2.7 e53/38 2:f80bfc0c:ns::rbd_data.1256a0c02879.0000000000000002:head v 53'25, mlcod=53'23)", 2024-09-11T09:52:01.280 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.094944+0000", 2024-09-11T09:52:01.280 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 38.147026814, 2024-09-11T09:52:01.280 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.010896785000000001, 2024-09-11T09:52:01.280 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:01.280 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:01.280 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:01.280 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:01.280 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.280 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:01.280 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.094944+0000", 2024-09-11T09:52:01.280 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:01.281 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.281 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.281 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:01.281 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.094944+0000", 2024-09-11T09:52:01.281 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:01.281 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.281 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.281 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:01.281 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.094944+0000", 2024-09-11T09:52:01.281 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.0699999999999996e-07 2024-09-11T09:52:01.281 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.281 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.281 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:01.282 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.099060+0000", 2024-09-11T09:52:01.282 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0041155580000000001 2024-09-11T09:52:01.282 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.282 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.282 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:01.282 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.099082+0000", 2024-09-11T09:52:01.282 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.2323999999999998e-05 2024-09-11T09:52:01.282 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.282 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.282 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:01.282 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.099094+0000", 2024-09-11T09:52:01.282 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1535000000000001e-05 2024-09-11T09:52:01.282 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.283 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.283 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:01.283 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.099160+0000", 2024-09-11T09:52:01.283 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.6449e-05 2024-09-11T09:52:01.283 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.283 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.283 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:01.283 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.099183+0000", 2024-09-11T09:52:01.283 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.3108e-05 2024-09-11T09:52:01.283 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.283 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.283 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:01.283 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.105776+0000", 2024-09-11T09:52:01.283 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0065922760000000002 2024-09-11T09:52:01.284 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.284 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.284 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:01.284 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.105840+0000", 2024-09-11T09:52:01.284 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.4727999999999994e-05 2024-09-11T09:52:01.284 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:01.284 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:01.284 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:01.284 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.284 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.284 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_repop(client.4694.0:35 2.7 e53/38 2:e8766325:ns::rbd_object_map.1256a0c02879:head v 53'26, mlcod=53'23)", 2024-09-11T09:52:01.284 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.100964+0000", 2024-09-11T09:52:01.284 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 38.141006286, 2024-09-11T09:52:01.285 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0049649359999999997, 2024-09-11T09:52:01.285 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:01.285 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:01.285 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:01.285 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:01.285 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.285 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:01.285 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.100964+0000", 2024-09-11T09:52:01.285 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:01.285 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.285 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.285 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:01.285 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.100964+0000", 2024-09-11T09:52:01.286 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:01.286 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.286 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.286 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:01.286 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.100966+0000", 2024-09-11T09:52:01.286 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.6330000000000001e-06 2024-09-11T09:52:01.286 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.286 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.286 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:01.286 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.100979+0000", 2024-09-11T09:52:01.286 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.2813000000000001e-05 2024-09-11T09:52:01.286 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.286 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.286 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:01.287 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.100992+0000", 2024-09-11T09:52:01.287 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.3414999999999999e-05 2024-09-11T09:52:01.287 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.287 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.287 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:01.287 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.101002+0000", 2024-09-11T09:52:01.287 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.0159e-05 2024-09-11T09:52:01.287 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.287 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.287 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:01.287 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.102365+0000", 2024-09-11T09:52:01.287 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001362912 2024-09-11T09:52:01.288 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.288 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.288 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:01.288 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.102382+0000", 2024-09-11T09:52:01.288 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.7433000000000001e-05 2024-09-11T09:52:01.288 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.288 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.288 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:01.288 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.105901+0000", 2024-09-11T09:52:01.288 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.003518476 2024-09-11T09:52:01.288 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.288 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.288 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:01.289 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.105929+0000", 2024-09-11T09:52:01.289 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.8095000000000001e-05 2024-09-11T09:52:01.289 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:01.289 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:01.289 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:01.289 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.289 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.289 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4694.0:41 2.2 2:5e2505c4:ns::rbd_data.1256a0c02879.0000000000000003:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:01.289 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.107511+0000", 2024-09-11T09:52:01.289 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 38.134459133, 2024-09-11T09:52:01.289 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.018571066000000001, 2024-09-11T09:52:01.289 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:01.289 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:01.290 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:01.290 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:52:01.290 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4694", 2024-09-11T09:52:01.290 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:52:01.290 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 41 2024-09-11T09:52:01.290 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.290 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:01.290 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.290 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:01.290 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.107511+0000", 2024-09-11T09:52:01.290 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:01.290 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.290 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.290 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:01.291 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.107511+0000", 2024-09-11T09:52:01.291 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:01.291 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.291 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.291 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:01.291 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.107513+0000", 2024-09-11T09:52:01.291 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.3489999999999999e-06 2024-09-11T09:52:01.291 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.291 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.291 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:01.291 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.112704+0000", 2024-09-11T09:52:01.291 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0051918509999999999 2024-09-11T09:52:01.291 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.291 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.292 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:01.292 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.112718+0000", 2024-09-11T09:52:01.292 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.359e-05 2024-09-11T09:52:01.292 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.292 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.292 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:01.292 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.112725+0000", 2024-09-11T09:52:01.292 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.9940000000000003e-06 2024-09-11T09:52:01.292 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.292 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.292 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:01.292 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.112788+0000", 2024-09-11T09:52:01.292 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.2447999999999996e-05 2024-09-11T09:52:01.292 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.293 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.293 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:01.293 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.112937+0000", 2024-09-11T09:52:01.293 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00014921499999999999 2024-09-11T09:52:01.293 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.293 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.293 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 1", 2024-09-11T09:52:01.293 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.113085+0000", 2024-09-11T09:52:01.293 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000148085 2024-09-11T09:52:01.293 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.293 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.293 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:52:01.293 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.124318+0000", 2024-09-11T09:52:01.294 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.011233158 2024-09-11T09:52:01.294 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.294 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.294 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:01.294 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.125957+0000", 2024-09-11T09:52:01.294 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001638701 2024-09-11T09:52:01.294 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.294 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.294 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:01.294 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.126029+0000", 2024-09-11T09:52:01.294 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.2125000000000005e-05 2024-09-11T09:52:01.294 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.294 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.295 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:01.295 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.126082+0000", 2024-09-11T09:52:01.295 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.3550000000000001e-05 2024-09-11T09:52:01.295 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:01.295 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:01.295 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:01.295 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.295 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.295 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4694.0:44 2.3 2:cd1a7d1c:ns::rbd_data.1256a0c02879.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:01.295 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.112744+0000", 2024-09-11T09:52:01.295 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 38.129226566, 2024-09-11T09:52:01.295 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.021672212999999999, 2024-09-11T09:52:01.295 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:01.296 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:01.296 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:01.296 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:52:01.296 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4694", 2024-09-11T09:52:01.296 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:52:01.296 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 44 2024-09-11T09:52:01.296 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.296 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:01.296 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.296 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:01.296 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.112744+0000", 2024-09-11T09:52:01.296 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:01.296 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.297 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.297 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:01.297 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.112744+0000", 2024-09-11T09:52:01.297 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:01.297 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.297 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.297 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:01.297 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.112745+0000", 2024-09-11T09:52:01.297 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.2900000000000002e-07 2024-09-11T09:52:01.297 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.297 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.297 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:01.297 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.119929+0000", 2024-09-11T09:52:01.297 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0071846230000000002 2024-09-11T09:52:01.298 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.298 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.298 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:01.298 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.119949+0000", 2024-09-11T09:52:01.298 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.9264000000000001e-05 2024-09-11T09:52:01.298 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.298 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.298 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:01.298 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.119959+0000", 2024-09-11T09:52:01.298 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.0465999999999999e-05 2024-09-11T09:52:01.298 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.298 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.298 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:01.298 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.120028+0000", 2024-09-11T09:52:01.299 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.9318000000000003e-05 2024-09-11T09:52:01.299 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.299 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.299 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:01.299 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.120160+0000", 2024-09-11T09:52:01.299 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000132015 2024-09-11T09:52:01.299 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.299 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.299 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 2", 2024-09-11T09:52:01.299 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.120260+0000", 2024-09-11T09:52:01.299 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.9397999999999999e-05 2024-09-11T09:52:01.299 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.300 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.300 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:52:01.300 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.134246+0000", 2024-09-11T09:52:01.300 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.013986721000000001 2024-09-11T09:52:01.300 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.300 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.300 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:01.300 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.134295+0000", 2024-09-11T09:52:01.300 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.8649000000000001e-05 2024-09-11T09:52:01.300 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.300 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.300 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:01.300 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.134361+0000", 2024-09-11T09:52:01.301 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.6081999999999995e-05 2024-09-11T09:52:01.301 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.301 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.301 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:01.301 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.134416+0000", 2024-09-11T09:52:01.301 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.4848000000000001e-05 2024-09-11T09:52:01.301 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:01.301 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:01.301 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:01.301 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.301 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.301 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4694.0:46 2.3 2:ce0e1469:ns::rbd_data.1256a0c02879.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:01.302 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.119993+0000", 2024-09-11T09:52:01.302 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 38.121977090000001, 2024-09-11T09:52:01.302 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.025840610999999999, 2024-09-11T09:52:01.302 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:01.302 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:01.302 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:01.302 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:52:01.302 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4694", 2024-09-11T09:52:01.302 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:52:01.302 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 46 2024-09-11T09:52:01.302 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.302 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:01.302 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.302 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:01.303 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.119993+0000", 2024-09-11T09:52:01.303 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:01.303 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.303 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.303 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:01.303 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.119993+0000", 2024-09-11T09:52:01.303 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:01.303 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.303 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.303 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:01.303 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.119995+0000", 2024-09-11T09:52:01.303 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1930000000000001e-06 2024-09-11T09:52:01.303 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.303 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.304 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:01.304 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.128458+0000", 2024-09-11T09:52:01.304 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0084632630000000004 2024-09-11T09:52:01.304 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.304 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.304 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:01.304 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.128478+0000", 2024-09-11T09:52:01.304 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.0353e-05 2024-09-11T09:52:01.304 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.304 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.304 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:01.304 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.128491+0000", 2024-09-11T09:52:01.304 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.2954e-05 2024-09-11T09:52:01.305 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.305 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.305 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:01.305 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.128558+0000", 2024-09-11T09:52:01.305 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.7076999999999995e-05 2024-09-11T09:52:01.305 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.305 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.305 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:01.305 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.128708+0000", 2024-09-11T09:52:01.305 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00014939599999999999 2024-09-11T09:52:01.305 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.305 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.305 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 2", 2024-09-11T09:52:01.306 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.128866+0000", 2024-09-11T09:52:01.306 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000158145 2024-09-11T09:52:01.306 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.306 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.306 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:01.306 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.141045+0000", 2024-09-11T09:52:01.306 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.012179331999999999 2024-09-11T09:52:01.306 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.306 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.306 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:52:01.306 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.145742+0000", 2024-09-11T09:52:01.306 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0046972050000000003 2024-09-11T09:52:01.306 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.307 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.307 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:01.307 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.145779+0000", 2024-09-11T09:52:01.307 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.6544000000000001e-05 2024-09-11T09:52:01.307 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.307 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.307 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:01.307 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.145834+0000", 2024-09-11T09:52:01.307 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.5148999999999997e-05 2024-09-11T09:52:01.307 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:01.307 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:01.307 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:01.307 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.308 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.308 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4694.0:48 2.3 2:d2090b70:ns::rbd_data.1256a0c02879.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:01.308 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.130540+0000", 2024-09-11T09:52:01.308 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 38.111430824000003, 2024-09-11T09:52:01.308 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.017167789999999999, 2024-09-11T09:52:01.308 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:01.308 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:01.308 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:01.308 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:52:01.308 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4694", 2024-09-11T09:52:01.308 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:52:01.308 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 48 2024-09-11T09:52:01.309 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.309 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:01.309 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.309 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:01.309 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.130540+0000", 2024-09-11T09:52:01.309 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:01.309 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.309 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.309 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:01.309 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.130540+0000", 2024-09-11T09:52:01.309 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:01.309 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.310 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.310 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:01.310 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.130542+0000", 2024-09-11T09:52:01.310 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.9209999999999999e-06 2024-09-11T09:52:01.310 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.310 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.310 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:01.310 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.134691+0000", 2024-09-11T09:52:01.310 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0041498739999999996 2024-09-11T09:52:01.310 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.310 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.310 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:01.311 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.134705+0000", 2024-09-11T09:52:01.311 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.3669000000000001e-05 2024-09-11T09:52:01.311 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.311 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.311 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:01.311 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.134712+0000", 2024-09-11T09:52:01.311 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.8410000000000002e-06 2024-09-11T09:52:01.311 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.311 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.311 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:01.311 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.134754+0000", 2024-09-11T09:52:01.311 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.2524999999999997e-05 2024-09-11T09:52:01.312 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.312 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.312 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:01.312 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.134875+0000", 2024-09-11T09:52:01.312 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000120332 2024-09-11T09:52:01.312 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.312 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.312 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 2", 2024-09-11T09:52:01.312 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.135005+0000", 2024-09-11T09:52:01.312 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00013024900000000001 2024-09-11T09:52:01.312 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.312 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.312 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:01.313 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.145971+0000", 2024-09-11T09:52:01.313 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.010966203000000001 2024-09-11T09:52:01.313 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.313 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.313 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:52:01.313 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.147600+0000", 2024-09-11T09:52:01.313 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001628382 2024-09-11T09:52:01.313 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.313 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.313 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:01.313 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.147635+0000", 2024-09-11T09:52:01.313 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.5420000000000003e-05 2024-09-11T09:52:01.313 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.314 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.314 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:01.314 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.147707+0000", 2024-09-11T09:52:01.314 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.2373999999999995e-05 2024-09-11T09:52:01.314 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:01.314 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:01.314 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:01.314 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.314 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.314 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_repop(client.4694.0:49 2.7 e53/38 2:e8766325:ns::rbd_object_map.1256a0c02879:head v 53'33, mlcod=53'31)", 2024-09-11T09:52:01.314 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.135391+0000", 2024-09-11T09:52:01.314 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 38.106579259, 2024-09-11T09:52:01.314 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0051189859999999998, 2024-09-11T09:52:01.315 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:01.315 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:01.315 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:01.315 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:01.315 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.315 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:01.315 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.135391+0000", 2024-09-11T09:52:01.315 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:01.315 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.315 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.315 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:01.315 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.135391+0000", 2024-09-11T09:52:01.315 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:01.316 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.316 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.316 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:01.316 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.135393+0000", 2024-09-11T09:52:01.316 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.4020000000000001e-06 2024-09-11T09:52:01.316 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.316 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.316 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:01.316 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.135400+0000", 2024-09-11T09:52:01.316 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.5129999999999999e-06 2024-09-11T09:52:01.316 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.316 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.316 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:01.317 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.135411+0000", 2024-09-11T09:52:01.317 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1038e-05 2024-09-11T09:52:01.317 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.317 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.317 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:01.317 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.135418+0000", 2024-09-11T09:52:01.317 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.1489999999999997e-06 2024-09-11T09:52:01.317 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.317 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.317 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:01.317 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.135470+0000", 2024-09-11T09:52:01.317 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.2021999999999998e-05 2024-09-11T09:52:01.317 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.318 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.318 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:01.318 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.135501+0000", 2024-09-11T09:52:01.318 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.0349999999999999e-05 2024-09-11T09:52:01.318 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.318 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.318 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:01.318 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.140453+0000", 2024-09-11T09:52:01.318 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0049522409999999996 2024-09-11T09:52:01.318 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.318 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.318 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:01.318 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.140510+0000", 2024-09-11T09:52:01.318 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.7271000000000001e-05 2024-09-11T09:52:01.319 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:01.319 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:01.319 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:01.319 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.319 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.319 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4694.0:54 2.3 2:c44f0253:ns::rbd_data.1256a0c02879.000000000000000b:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:01.319 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.150151+0000", 2024-09-11T09:52:01.319 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 38.091819407999999, 2024-09-11T09:52:01.319 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.015103484, 2024-09-11T09:52:01.319 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:01.319 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:01.319 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:01.319 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:52:01.319 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4694", 2024-09-11T09:52:01.320 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:52:01.320 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 54 2024-09-11T09:52:01.320 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.320 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:01.320 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.320 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:01.320 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.150151+0000", 2024-09-11T09:52:01.320 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:01.320 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.320 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.320 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:01.320 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.150151+0000", 2024-09-11T09:52:01.320 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:01.321 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.321 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.321 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:01.321 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.150152+0000", 2024-09-11T09:52:01.321 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.2550000000000001e-06 2024-09-11T09:52:01.321 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.321 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.321 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:01.321 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.154528+0000", 2024-09-11T09:52:01.321 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0043758219999999997 2024-09-11T09:52:01.321 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.321 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.321 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:01.321 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.154541+0000", 2024-09-11T09:52:01.322 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.3414999999999999e-05 2024-09-11T09:52:01.322 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.322 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.322 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:01.322 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.154551+0000", 2024-09-11T09:52:01.322 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.4350000000000003e-06 2024-09-11T09:52:01.322 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.322 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.322 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:01.322 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.154615+0000", 2024-09-11T09:52:01.322 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.3627000000000006e-05 2024-09-11T09:52:01.322 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.323 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.323 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:01.323 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.154717+0000", 2024-09-11T09:52:01.323 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000102729 2024-09-11T09:52:01.323 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.323 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.323 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 2", 2024-09-11T09:52:01.323 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.154826+0000", 2024-09-11T09:52:01.323 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000108409 2024-09-11T09:52:01.323 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.323 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.323 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:52:01.323 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.165094+0000", 2024-09-11T09:52:01.323 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.010268210999999999 2024-09-11T09:52:01.324 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.324 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.324 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:01.324 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.165138+0000", 2024-09-11T09:52:01.324 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.4078e-05 2024-09-11T09:52:01.324 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.324 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.324 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:01.324 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.165204+0000", 2024-09-11T09:52:01.324 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.5716000000000006e-05 2024-09-11T09:52:01.324 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.324 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.324 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:01.324 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.165254+0000", 2024-09-11T09:52:01.325 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.0787e-05 2024-09-11T09:52:01.325 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:01.325 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:01.325 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:01.325 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.325 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.325 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_repop(client.4694.0:55 2.7 e53/38 2:e8766325:ns::rbd_object_map.1256a0c02879:head v 53'37, mlcod=53'34)", 2024-09-11T09:52:01.325 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.150221+0000", 2024-09-11T09:52:01.325 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 38.091749542999999, 2024-09-11T09:52:01.325 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00530015, 2024-09-11T09:52:01.325 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:01.325 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:01.325 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:01.326 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:01.326 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.326 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:01.326 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.150221+0000", 2024-09-11T09:52:01.326 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:01.326 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.326 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.326 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:01.326 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.150221+0000", 2024-09-11T09:52:01.326 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:01.326 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.326 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.326 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:01.326 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.150221+0000", 2024-09-11T09:52:01.327 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.9100000000000004e-07 2024-09-11T09:52:01.327 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.327 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.327 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:01.327 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.150226+0000", 2024-09-11T09:52:01.327 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.2250000000000002e-06 2024-09-11T09:52:01.327 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.327 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.327 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:01.327 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.150251+0000", 2024-09-11T09:52:01.327 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.4987e-05 2024-09-11T09:52:01.327 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.327 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.327 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:01.328 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.150258+0000", 2024-09-11T09:52:01.328 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.0790000000000003e-06 2024-09-11T09:52:01.328 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.328 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.328 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:01.328 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.150305+0000", 2024-09-11T09:52:01.328 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.6879000000000003e-05 2024-09-11T09:52:01.328 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.328 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.328 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:01.328 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.150320+0000", 2024-09-11T09:52:01.328 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.5268999999999999e-05 2024-09-11T09:52:01.328 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.329 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.329 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:01.329 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.155493+0000", 2024-09-11T09:52:01.329 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0051731390000000002 2024-09-11T09:52:01.329 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.329 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.329 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:01.329 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.155521+0000", 2024-09-11T09:52:01.329 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.8081000000000001e-05 2024-09-11T09:52:01.329 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:01.329 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:01.329 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:01.329 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.330 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.330 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4694.0:56 2.3 2:c462f029:ns::rbd_data.1256a0c02879.000000000000000c:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:01.330 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.154576+0000", 2024-09-11T09:52:01.330 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 38.087394566999997, 2024-09-11T09:52:01.330 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.020781621, 2024-09-11T09:52:01.330 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:01.330 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:01.330 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:01.330 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:52:01.330 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4694", 2024-09-11T09:52:01.330 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:52:01.330 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 56 2024-09-11T09:52:01.331 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.331 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:01.331 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.331 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:01.331 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.154576+0000", 2024-09-11T09:52:01.331 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:01.331 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.331 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.331 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:01.331 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.154576+0000", 2024-09-11T09:52:01.331 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:01.331 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.331 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.331 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:01.332 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.154577+0000", 2024-09-11T09:52:01.332 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.2249999999999999e-06 2024-09-11T09:52:01.332 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.332 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.332 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:01.332 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.158783+0000", 2024-09-11T09:52:01.332 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0042063889999999996 2024-09-11T09:52:01.332 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.332 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.332 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:01.332 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.158803+0000", 2024-09-11T09:52:01.332 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.9576999999999999e-05 2024-09-11T09:52:01.333 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.333 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.333 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:01.333 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.158815+0000", 2024-09-11T09:52:01.333 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.2221999999999999e-05 2024-09-11T09:52:01.333 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.333 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.333 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:01.333 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.158902+0000", 2024-09-11T09:52:01.333 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.6390000000000005e-05 2024-09-11T09:52:01.333 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.333 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.333 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:01.334 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.159141+0000", 2024-09-11T09:52:01.334 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000238886 2024-09-11T09:52:01.334 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.334 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.334 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 2", 2024-09-11T09:52:01.334 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.159279+0000", 2024-09-11T09:52:01.334 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000138073 2024-09-11T09:52:01.334 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.334 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.334 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:52:01.334 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.173297+0000", 2024-09-11T09:52:01.334 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.014018113 2024-09-11T09:52:01.334 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.334 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.335 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:01.335 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.175278+0000", 2024-09-11T09:52:01.335 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0019817390000000002 2024-09-11T09:52:01.335 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.335 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.335 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:01.335 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.175316+0000", 2024-09-11T09:52:01.335 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.7829e-05 2024-09-11T09:52:01.335 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.335 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.335 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:01.335 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.175357+0000", 2024-09-11T09:52:01.335 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.1177999999999998e-05 2024-09-11T09:52:01.335 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:01.335 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:01.336 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:01.336 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.336 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.336 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4694.0:59 2.2 2:41adefc9:ns::rbd_data.1256a0c02879.000000000000000e:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:01.336 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.158843+0000", 2024-09-11T09:52:01.336 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 38.083127001000001, 2024-09-11T09:52:01.336 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.017874207, 2024-09-11T09:52:01.336 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:01.336 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:01.336 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:01.336 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:52:01.336 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4694", 2024-09-11T09:52:01.336 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:52:01.337 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 59 2024-09-11T09:52:01.337 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.337 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:01.337 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.337 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:01.337 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.158843+0000", 2024-09-11T09:52:01.337 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:01.337 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.337 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.337 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:01.337 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.158843+0000", 2024-09-11T09:52:01.337 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:01.337 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.338 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.338 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:01.338 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.158845+0000", 2024-09-11T09:52:01.338 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.7439999999999999e-06 2024-09-11T09:52:01.338 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.338 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.338 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:01.338 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.162713+0000", 2024-09-11T09:52:01.338 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0038678829999999999 2024-09-11T09:52:01.338 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.338 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.338 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:01.338 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.162728+0000", 2024-09-11T09:52:01.338 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.5114e-05 2024-09-11T09:52:01.339 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.339 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.339 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:01.339 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.162737+0000", 2024-09-11T09:52:01.339 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.1099999999999992e-06 2024-09-11T09:52:01.339 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.339 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.339 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:01.339 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.162799+0000", 2024-09-11T09:52:01.339 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.1934999999999996e-05 2024-09-11T09:52:01.339 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.339 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.339 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:01.339 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.162911+0000", 2024-09-11T09:52:01.339 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000111867 2024-09-11T09:52:01.340 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.340 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.340 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 1", 2024-09-11T09:52:01.340 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.163030+0000", 2024-09-11T09:52:01.340 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000119108 2024-09-11T09:52:01.340 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.340 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.340 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:52:01.340 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.174086+0000", 2024-09-11T09:52:01.340 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.01105543 2024-09-11T09:52:01.340 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.340 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.341 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:01.341 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.176614+0000", 2024-09-11T09:52:01.341 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0025284940000000001 2024-09-11T09:52:01.341 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.341 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.341 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:01.341 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.176672+0000", 2024-09-11T09:52:01.341 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.7892000000000001e-05 2024-09-11T09:52:01.341 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.341 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.341 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:01.341 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.176718+0000", 2024-09-11T09:52:01.341 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.5630000000000002e-05 2024-09-11T09:52:01.341 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:01.342 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:01.342 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:01.342 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.342 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.342 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.5002.0:89 2.3 2:c54742d8:::rbd_data.1250f5e25b80.0000000000000006:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-09-11T09:52:01.342 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:38.546016+0000", 2024-09-11T09:52:01.342 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 22.695954454999999, 2024-09-11T09:52:01.342 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0049098550000000003, 2024-09-11T09:52:01.342 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:01.342 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:01.342 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:01.342 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:52:01.342 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.5002", 2024-09-11T09:52:01.342 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/2204440692", 2024-09-11T09:52:01.343 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 89 2024-09-11T09:52:01.343 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.343 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:01.343 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.343 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:01.343 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.546016+0000", 2024-09-11T09:52:01.343 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:01.343 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.343 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.343 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:01.343 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.546016+0000", 2024-09-11T09:52:01.343 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:01.343 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.344 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.344 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:01.344 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.546017+0000", 2024-09-11T09:52:01.344 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.3399999999999998e-07 2024-09-11T09:52:01.344 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.344 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.344 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:01.344 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.546839+0000", 2024-09-11T09:52:01.344 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00082222800000000004 2024-09-11T09:52:01.344 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.344 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.344 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:01.344 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.546851+0000", 2024-09-11T09:52:01.344 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.2075999999999999e-05 2024-09-11T09:52:01.345 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.345 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.345 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:01.345 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.546856+0000", 2024-09-11T09:52:01.345 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.4009999999999996e-06 2024-09-11T09:52:01.345 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.345 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.345 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:01.345 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.546897+0000", 2024-09-11T09:52:01.345 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.0113e-05 2024-09-11T09:52:01.345 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.345 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.345 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:01.345 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.546954+0000", 2024-09-11T09:52:01.346 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.7825000000000003e-05 2024-09-11T09:52:01.346 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.346 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.346 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 2", 2024-09-11T09:52:01.346 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.547055+0000", 2024-09-11T09:52:01.346 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000100269 2024-09-11T09:52:01.346 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.346 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.346 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:01.346 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.549675+0000", 2024-09-11T09:52:01.346 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0026206609999999998 2024-09-11T09:52:01.346 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.346 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.346 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:52:01.347 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.550858+0000", 2024-09-11T09:52:01.347 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0011825270000000001 2024-09-11T09:52:01.347 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.347 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.347 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:01.347 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.550894+0000", 2024-09-11T09:52:01.347 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.5896999999999999e-05 2024-09-11T09:52:01.347 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.347 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.347 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:01.347 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.550926+0000", 2024-09-11T09:52:01.347 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.2023999999999997e-05 2024-09-11T09:52:01.347 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:01.348 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:01.348 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:01.348 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.348 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.348 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.5002.0:91 2.3 2:c54742d8:::rbd_data.1250f5e25b80.0000000000000006:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-09-11T09:52:01.348 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:38.553743+0000", 2024-09-11T09:52:01.348 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 22.688227412, 2024-09-11T09:52:01.348 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0051633629999999998, 2024-09-11T09:52:01.348 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:01.348 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:01.348 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:01.348 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:52:01.348 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.5002", 2024-09-11T09:52:01.349 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/2204440692", 2024-09-11T09:52:01.349 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 91 2024-09-11T09:52:01.349 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.349 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:01.349 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.349 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:01.349 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.553743+0000", 2024-09-11T09:52:01.349 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:01.349 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.349 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.349 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:01.349 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.553743+0000", 2024-09-11T09:52:01.349 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:01.349 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.350 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.350 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:01.350 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.553745+0000", 2024-09-11T09:52:01.350 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.5120000000000001e-06 2024-09-11T09:52:01.350 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.350 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.350 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:01.350 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.554588+0000", 2024-09-11T09:52:01.350 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00084352299999999999 2024-09-11T09:52:01.350 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.350 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.350 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:01.350 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.554604+0000", 2024-09-11T09:52:01.351 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.643e-05 2024-09-11T09:52:01.351 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.351 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.351 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:01.351 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.554613+0000", 2024-09-11T09:52:01.351 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.5669999999999995e-06 2024-09-11T09:52:01.351 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.351 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.351 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:01.351 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.554662+0000", 2024-09-11T09:52:01.351 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.8510000000000001e-05 2024-09-11T09:52:01.351 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.351 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.352 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:01.352 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.554732+0000", 2024-09-11T09:52:01.352 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.0297999999999997e-05 2024-09-11T09:52:01.352 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.352 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.352 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 2", 2024-09-11T09:52:01.352 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.554870+0000", 2024-09-11T09:52:01.352 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00013856599999999999 2024-09-11T09:52:01.352 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.352 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.352 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:01.352 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.557875+0000", 2024-09-11T09:52:01.352 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0030042760000000002 2024-09-11T09:52:01.353 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.353 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.353 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:52:01.353 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.558835+0000", 2024-09-11T09:52:01.353 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00096057900000000003 2024-09-11T09:52:01.353 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.353 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.353 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:01.353 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.558874+0000", 2024-09-11T09:52:01.353 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.8992999999999997e-05 2024-09-11T09:52:01.353 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.353 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.353 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:01.354 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.558906+0000", 2024-09-11T09:52:01.354 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.2109e-05 2024-09-11T09:52:01.354 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:01.354 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:01.354 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:01.354 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.354 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.354 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.5002.0:93 2.3 2:c54742d8:::rbd_data.1250f5e25b80.0000000000000006:head [write 2228224~917504] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e61)", 2024-09-11T09:52:01.354 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:38.562067+0000", 2024-09-11T09:52:01.354 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 22.679903815999999, 2024-09-11T09:52:01.354 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0050551099999999998, 2024-09-11T09:52:01.354 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:01.355 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:01.355 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:01.355 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:52:01.355 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.5002", 2024-09-11T09:52:01.355 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/2204440692", 2024-09-11T09:52:01.355 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 93 2024-09-11T09:52:01.355 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.355 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:01.355 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.355 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:01.355 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.562067+0000", 2024-09-11T09:52:01.355 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:01.355 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.356 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.356 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:01.356 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.562067+0000", 2024-09-11T09:52:01.356 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:01.356 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.356 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.356 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:01.356 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.562068+0000", 2024-09-11T09:52:01.356 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.0669999999999999e-06 2024-09-11T09:52:01.356 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.356 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.356 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:01.356 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.563170+0000", 2024-09-11T09:52:01.357 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00110256 2024-09-11T09:52:01.357 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.357 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.357 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:01.357 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.563185+0000", 2024-09-11T09:52:01.357 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.4873e-05 2024-09-11T09:52:01.357 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.357 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.357 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:01.357 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.563193+0000", 2024-09-11T09:52:01.357 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.1319999999999994e-06 2024-09-11T09:52:01.357 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.357 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.358 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:01.358 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.563238+0000", 2024-09-11T09:52:01.358 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.4773e-05 2024-09-11T09:52:01.358 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.358 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.358 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:01.358 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.563290+0000", 2024-09-11T09:52:01.358 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.1690000000000001e-05 2024-09-11T09:52:01.358 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.358 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.358 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 2", 2024-09-11T09:52:01.358 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.563365+0000", 2024-09-11T09:52:01.359 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.5023000000000003e-05 2024-09-11T09:52:01.359 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.359 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.359 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:01.359 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.566552+0000", 2024-09-11T09:52:01.359 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0031869960000000001 2024-09-11T09:52:01.359 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.359 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.359 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:52:01.359 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.567048+0000", 2024-09-11T09:52:01.359 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00049605099999999996 2024-09-11T09:52:01.359 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.359 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.359 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:01.360 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.567095+0000", 2024-09-11T09:52:01.360 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.6823999999999998e-05 2024-09-11T09:52:01.360 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:01.360 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:01.360 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:01.360 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.567122+0000", 2024-09-11T09:52:01.360 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.7121e-05 2024-09-11T09:52:01.360 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:01.360 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:01.360 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:01.360 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:01.360 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:01.360 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-11T09:52:01.361 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T09:52:01.362 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-11T09:52:01.363 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:52:01.363+0000 7f3c96ec9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31909) UID: 0 2024-09-11T09:52:01.463 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:52:01.464+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:52:01.494 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-11T09:52:01.494 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T09:52:01.506 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-11T09:52:01.506 INFO:teuthology.orchestra.run.smithi096.stdout: "ops": [], 2024-09-11T09:52:01.506 INFO:teuthology.orchestra.run.smithi096.stdout: "num_ops": 0 2024-09-11T09:52:01.506 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-11T09:52:01.511 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-11T09:52:01.511 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T09:52:01.515 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-11T09:52:01.528 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-11T09:52:01.528 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T09:52:01.545 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-11T09:52:01.545 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T09:52:01.562 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-11T09:52:01.562 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T09:52:01.564 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-09-11T09:52:01.565+0000 7f7b6c7d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31904) UID: 0 2024-09-11T09:52:01.579 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-11T09:52:01.579 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T09:52:01.618 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:52:01.618 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:52:01.627 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-11T09:52:01.665 INFO:tasks.ceph.osd.2.smithi017.stderr:2024-09-11T09:52:01.666+0000 7fd20b170640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31672) UID: 0 2024-09-11T09:52:01.766 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:52:01.767+0000 7f3c96ec9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31909) UID: 0 2024-09-11T09:52:01.769 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:52:01.769 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:52:01.779 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-11T09:52:01.866 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:52:01.867+0000 7f3c96ec9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31909) UID: 0 2024-09-11T09:52:01.891 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:52:01.891 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:52:01.899 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-11T09:52:01.968 INFO:tasks.ceph.osd.2.smithi017.stderr:2024-09-11T09:52:01.969+0000 7fd20b170640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31672) UID: 0 2024-09-11T09:52:02.006 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:52:02.007 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:52:02.015 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-11T09:52:02.069 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:52:02.070+0000 7f3a7b90a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 42475) UID: 0 2024-09-11T09:52:02.162 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:52:02.162 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:52:02.170 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:52:02.170+0000 7f3a7b90a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 42475) UID: 0 2024-09-11T09:52:02.171 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-11T09:52:02.270 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:52:02.271+0000 7f3c96ec9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31909) UID: 0 2024-09-11T09:52:02.277 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:52:02.278 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:52:02.286 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-11T09:52:02.371 INFO:tasks.ceph.osd.2.smithi017.stderr:2024-09-11T09:52:02.372+0000 7fd20b170640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31672) UID: 0 2024-09-11T09:52:02.393 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:52:02.394 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:52:02.402 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-11T09:52:02.473 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-09-11T09:52:02.474+0000 7f7b6c7d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31904) UID: 0 2024-09-11T09:52:02.551 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:52:02.551 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:52:02.560 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-11T09:52:02.574 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:52:02.575+0000 7f3c96ec9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31909) UID: 0 2024-09-11T09:52:02.667 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:52:02.668 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:52:02.675 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:52:02.676+0000 7f3a7b90a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 42475) UID: 0 2024-09-11T09:52:02.677 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-11T09:52:02.775 INFO:tasks.ceph.osd.0.smithi017.stderr:2024-09-11T09:52:02.776+0000 7fb73c0f1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31665) UID: 0 2024-09-11T09:52:02.783 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:52:02.784 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:52:02.792 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-11T09:52:02.827 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-11T09:52:02.877 INFO:tasks.ceph.osd.2.smithi017.stderr:2024-09-11T09:52:02.878+0000 7fd20b170640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31672) UID: 0 2024-09-11T09:52:02.943 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:52:02.943 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:52:02.955 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-11T09:52:02.996 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:52:02.997+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:52:03.067 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:52:03.067 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:52:03.076 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-11T09:52:03.077 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-09-11T09:52:03.078+0000 7f7b6c7d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31904) UID: 0 2024-09-11T09:52:03.179 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:52:03.180+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:52:03.187 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:52:03.187 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:52:03.196 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-11T09:52:03.280 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:52:03.281+0000 7f3c96ec9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31909) UID: 0 2024-09-11T09:52:03.300 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:52:03.300 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:52:03.309 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-11T09:52:03.381 INFO:tasks.ceph.osd.0.smithi017.stderr:2024-09-11T09:52:03.382+0000 7fb73c0f1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31665) UID: 0 2024-09-11T09:52:03.453 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:52:03.453 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:52:03.461 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-11T09:52:03.482 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:52:03.483+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:52:03.563 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:52:03.563 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:52:03.572 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-11T09:52:03.583 INFO:tasks.ceph.osd.0.smithi017.stderr:2024-09-11T09:52:03.583+0000 7fb73c0f1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31665) UID: 0 2024-09-11T09:52:03.590 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T09:52:03.674 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:52:03.674 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:52:03.682 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-11T09:52:03.683 INFO:tasks.ceph.osd.2.smithi017.stderr:2024-09-11T09:52:03.684+0000 7fd20b170640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31672) UID: 0 2024-09-11T09:52:03.721 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-11T09:52:03.721 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T09:52:03.738 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-11T09:52:03.738 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T09:52:03.754 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-11T09:52:03.754 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T09:52:03.770 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-11T09:52:03.771 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T09:52:03.785 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:52:03.786+0000 7f3a7b90a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 42475) UID: 0 2024-09-11T09:52:03.787 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-11T09:52:03.787 INFO:teuthology.orchestra.run.smithi096.stdout: "ops": [], 2024-09-11T09:52:03.787 INFO:teuthology.orchestra.run.smithi096.stdout: "num_ops": 0 2024-09-11T09:52:03.787 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-11T09:52:03.787 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-11T09:52:03.788 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T09:52:03.796 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-11T09:52:03.805 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-11T09:52:03.805 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T09:52:03.820 INFO:teuthology.orchestra.run.smithi017.stderr:noscrub is set 2024-09-11T09:52:03.886 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:52:03.887+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:52:03.940 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-11T09:52:03.940 INFO:teuthology.orchestra.run.smithi096.stdout: "ops": [], 2024-09-11T09:52:03.940 INFO:teuthology.orchestra.run.smithi096.stdout: "complaint_time": 30, 2024-09-11T09:52:03.940 INFO:teuthology.orchestra.run.smithi096.stdout: "num_blocked_ops": 0 2024-09-11T09:52:03.940 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-11T09:52:03.948 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-11T09:52:03.992 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-09-11T09:52:03.993+0000 7f7b6c7d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31904) UID: 0 2024-09-11T09:52:04.052 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-11T09:52:04.052 INFO:teuthology.orchestra.run.smithi096.stdout: "size": 20, 2024-09-11T09:52:04.052 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 600, 2024-09-11T09:52:04.052 INFO:teuthology.orchestra.run.smithi096.stdout: "ops": [ 2024-09-11T09:52:04.052 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:04.052 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "MOSDPGPush(3.3 67/66 [PushOp(3:c4c92e5a:::rbd_trash:head, version: 56'1, data_included: [], data_size: 0, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:c4c92e5a:::rbd_trash:head@56'1, size: 0, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 0, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-11T09:52:04.053 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:45.784642+0000", 2024-09-11T09:52:04.053 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 18.268840987000001, 2024-09-11T09:52:04.053 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00049292200000000002, 2024-09-11T09:52:04.053 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:04.053 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:04.053 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "started", 2024-09-11T09:52:04.053 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:04.053 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:04.053 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:04.053 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:45.784642+0000", 2024-09-11T09:52:04.053 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:04.053 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:04.053 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:04.054 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:04.054 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:45.784642+0000", 2024-09-11T09:52:04.054 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:04.054 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:04.054 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:04.054 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:04.054 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:45.784643+0000", 2024-09-11T09:52:04.054 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.2599999999999998e-07 2024-09-11T09:52:04.054 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:04.054 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:04.054 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:04.054 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:45.784652+0000", 2024-09-11T09:52:04.054 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.064e-06 2024-09-11T09:52:04.055 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:04.055 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:04.055 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:04.055 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:45.784664+0000", 2024-09-11T09:52:04.055 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1831000000000001e-05 2024-09-11T09:52:04.055 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:04.055 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:04.055 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:04.055 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:45.784676+0000", 2024-09-11T09:52:04.055 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1996e-05 2024-09-11T09:52:04.055 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:04.055 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:04.055 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:04.055 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:45.784744+0000", 2024-09-11T09:52:04.056 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.8310000000000002e-05 2024-09-11T09:52:04.056 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:04.056 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:04.056 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:04.056 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:45.784758+0000", 2024-09-11T09:52:04.056 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.3998999999999999e-05 2024-09-11T09:52:04.056 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:04.056 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:04.056 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:04.056 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:45.785135+0000", 2024-09-11T09:52:04.056 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000377296 2024-09-11T09:52:04.056 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:04.056 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:04.056 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:04.057 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:04.057 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:04.057 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-11T09:52:04.061 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-11T09:52:04.105 INFO:tasks.ceph.osd.2.smithi017.stderr:2024-09-11T09:52:04.106+0000 7fd20b170640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31672) UID: 0 2024-09-11T09:52:04.169 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:52:04.169 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [], 2024-09-11T09:52:04.169 INFO:teuthology.orchestra.run.smithi017.stdout: "num_ops": 0 2024-09-11T09:52:04.169 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:52:04.178 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-11T09:52:04.189 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-09-11T09:52:04.189+0000 7f7b6c7d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31904) UID: 0 2024-09-11T09:52:04.289 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:52:04.290+0000 7f3c96ec9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31909) UID: 0 2024-09-11T09:52:04.325 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:52:04.325 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [], 2024-09-11T09:52:04.326 INFO:teuthology.orchestra.run.smithi017.stdout: "complaint_time": 30, 2024-09-11T09:52:04.326 INFO:teuthology.orchestra.run.smithi017.stdout: "num_blocked_ops": 0 2024-09-11T09:52:04.326 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:52:04.334 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-11T09:52:04.390 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:52:04.391+0000 7f3c96ec9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31909) UID: 0 2024-09-11T09:52:04.443 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:52:04.444 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 20, 2024-09-11T09:52:04.444 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 600, 2024-09-11T09:52:04.444 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [ 2024-09-11T09:52:04.444 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.444 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4688.0:34 2.4 e53/32 2:2367385a:::rbd_data.1250f5e25b80.0000000000000002:head v 53'6, mlcod=53'4)", 2024-09-11T09:52:04.444 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.907415+0000", 2024-09-11T09:52:04.444 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 41.536101588999998, 2024-09-11T09:52:04.444 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.011285778999999999, 2024-09-11T09:52:04.444 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:04.444 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:04.444 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:04.445 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:04.445 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.445 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:04.445 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.907415+0000", 2024-09-11T09:52:04.445 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:04.445 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.445 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.445 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:04.445 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.907415+0000", 2024-09-11T09:52:04.445 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:04.445 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.445 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.445 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:04.445 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.907417+0000", 2024-09-11T09:52:04.446 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.2910000000000002e-06 2024-09-11T09:52:04.446 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.446 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.446 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:04.446 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.910477+0000", 2024-09-11T09:52:04.446 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0030597829999999999 2024-09-11T09:52:04.446 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.446 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.446 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:04.446 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.910501+0000", 2024-09-11T09:52:04.446 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.3332e-05 2024-09-11T09:52:04.446 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.446 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.446 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:04.447 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.910514+0000", 2024-09-11T09:52:04.447 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3957e-05 2024-09-11T09:52:04.447 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.447 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.447 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:04.447 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.910612+0000", 2024-09-11T09:52:04.447 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.7075000000000001e-05 2024-09-11T09:52:04.447 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.447 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.447 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:04.447 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.910640+0000", 2024-09-11T09:52:04.447 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.8768999999999998e-05 2024-09-11T09:52:04.447 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.447 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.448 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:04.448 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.918640+0000", 2024-09-11T09:52:04.448 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0079991899999999998 2024-09-11T09:52:04.448 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.448 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.448 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:04.448 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.918701+0000", 2024-09-11T09:52:04.448 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.1382000000000003e-05 2024-09-11T09:52:04.448 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:04.448 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:04.448 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:04.448 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.448 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.449 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4688.0:38 2.5 e53/32 2:abde2694:::rbd_data.1250f5e25b80.0000000000000004:head v 53'1, mlcod=0'0)", 2024-09-11T09:52:04.449 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.915432+0000", 2024-09-11T09:52:04.449 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 41.528085208999997, 2024-09-11T09:52:04.449 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.015849652999999998, 2024-09-11T09:52:04.449 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:04.449 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:04.449 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:04.449 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:04.449 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.449 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:04.449 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.915432+0000", 2024-09-11T09:52:04.449 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:04.449 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.449 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.450 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:04.450 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.915432+0000", 2024-09-11T09:52:04.450 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:04.450 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.450 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.450 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:04.450 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.915432+0000", 2024-09-11T09:52:04.450 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.91e-07 2024-09-11T09:52:04.450 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.450 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.450 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:04.450 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.924943+0000", 2024-09-11T09:52:04.450 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0095110630000000002 2024-09-11T09:52:04.451 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.451 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.451 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:04.451 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.924960+0000", 2024-09-11T09:52:04.451 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6915999999999999e-05 2024-09-11T09:52:04.451 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.451 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.451 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:04.451 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.924968+0000", 2024-09-11T09:52:04.451 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.6999999999999997e-06 2024-09-11T09:52:04.451 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.451 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.451 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:04.451 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.925025+0000", 2024-09-11T09:52:04.452 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.6949e-05 2024-09-11T09:52:04.452 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.452 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.452 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:04.452 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.925043+0000", 2024-09-11T09:52:04.452 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7784000000000001e-05 2024-09-11T09:52:04.452 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.452 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.452 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:04.452 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.931237+0000", 2024-09-11T09:52:04.452 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0061935389999999996 2024-09-11T09:52:04.452 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.453 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.453 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:04.453 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.931281+0000", 2024-09-11T09:52:04.453 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.4511000000000001e-05 2024-09-11T09:52:04.453 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:04.453 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:04.453 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:04.453 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.453 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.453 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:40 2.7 2:fe41411c:::rbd_object_map.1250f5e25b80:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:04.453 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.919652+0000", 2024-09-11T09:52:04.453 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 41.523864541999998, 2024-09-11T09:52:04.454 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.012187073, 2024-09-11T09:52:04.454 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:04.454 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:04.454 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:04.454 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:04.454 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:52:04.454 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:52:04.454 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 40 2024-09-11T09:52:04.454 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.454 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:04.454 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.454 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:04.454 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919652+0000", 2024-09-11T09:52:04.454 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:04.455 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.455 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.455 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:04.455 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919652+0000", 2024-09-11T09:52:04.455 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:04.455 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.455 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.455 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:04.455 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919654+0000", 2024-09-11T09:52:04.455 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.88e-06 2024-09-11T09:52:04.455 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.455 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.455 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:04.455 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919669+0000", 2024-09-11T09:52:04.456 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.5271e-05 2024-09-11T09:52:04.456 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.456 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.456 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:04.456 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919687+0000", 2024-09-11T09:52:04.456 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7448999999999998e-05 2024-09-11T09:52:04.456 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.456 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.456 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:04.456 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919695+0000", 2024-09-11T09:52:04.456 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.1149999999999994e-06 2024-09-11T09:52:04.456 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.456 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.456 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:04.456 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919757+0000", 2024-09-11T09:52:04.457 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.2348999999999995e-05 2024-09-11T09:52:04.457 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.457 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.457 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:04.457 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919875+0000", 2024-09-11T09:52:04.457 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00011750499999999999 2024-09-11T09:52:04.457 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.457 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.457 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 5", 2024-09-11T09:52:04.457 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.920201+0000", 2024-09-11T09:52:04.457 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00032628099999999999 2024-09-11T09:52:04.457 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.457 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.457 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:04.457 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.922089+0000", 2024-09-11T09:52:04.457 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.001887978 2024-09-11T09:52:04.458 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.458 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.458 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:04.458 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.931752+0000", 2024-09-11T09:52:04.458 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0096626309999999997 2024-09-11T09:52:04.458 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.458 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.458 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:04.458 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.931794+0000", 2024-09-11T09:52:04.458 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.2849999999999998e-05 2024-09-11T09:52:04.458 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.458 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.458 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:04.458 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.931839+0000", 2024-09-11T09:52:04.459 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.4764e-05 2024-09-11T09:52:04.459 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:04.459 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:04.459 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:04.459 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.459 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.459 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:41 2.7 2:fe41411c:::rbd_object_map.1250f5e25b80:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:04.459 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.919716+0000", 2024-09-11T09:52:04.459 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 41.523800698000002, 2024-09-11T09:52:04.459 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.015641605999999999, 2024-09-11T09:52:04.459 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:04.459 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:04.459 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:04.459 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:04.460 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:52:04.460 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:52:04.460 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 41 2024-09-11T09:52:04.460 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.460 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:04.460 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.460 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:04.460 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919716+0000", 2024-09-11T09:52:04.460 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:04.460 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.460 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.460 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:04.460 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919716+0000", 2024-09-11T09:52:04.461 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:04.461 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.461 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.461 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:04.461 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919716+0000", 2024-09-11T09:52:04.461 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.9999999999999998e-07 2024-09-11T09:52:04.461 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.461 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.461 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:04.461 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919720+0000", 2024-09-11T09:52:04.461 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.6440000000000002e-06 2024-09-11T09:52:04.461 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.461 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.462 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:04.462 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919729+0000", 2024-09-11T09:52:04.462 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.1759999999999999e-06 2024-09-11T09:52:04.462 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.462 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.462 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:04.462 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919736+0000", 2024-09-11T09:52:04.462 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.5390000000000004e-06 2024-09-11T09:52:04.462 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.462 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.462 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:04.462 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.920669+0000", 2024-09-11T09:52:04.462 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00093275599999999997 2024-09-11T09:52:04.462 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.463 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.463 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:04.463 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.920797+0000", 2024-09-11T09:52:04.463 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00012893400000000001 2024-09-11T09:52:04.463 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.463 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.463 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 5", 2024-09-11T09:52:04.463 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.921269+0000", 2024-09-11T09:52:04.463 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00047128600000000001 2024-09-11T09:52:04.463 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.463 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.463 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:04.463 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.923198+0000", 2024-09-11T09:52:04.464 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0019289369999999999 2024-09-11T09:52:04.464 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.464 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.464 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:04.464 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.935300+0000", 2024-09-11T09:52:04.464 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.012102392 2024-09-11T09:52:04.464 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.464 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.464 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:04.464 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.935332+0000", 2024-09-11T09:52:04.464 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.2292000000000002e-05 2024-09-11T09:52:04.464 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.465 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.465 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:04.465 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.935358+0000", 2024-09-11T09:52:04.465 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.525e-05 2024-09-11T09:52:04.465 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:04.465 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:04.465 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:04.465 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.465 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.465 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:48 2.7 2:fe41411c:::rbd_object_map.1250f5e25b80:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:04.465 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.938799+0000", 2024-09-11T09:52:04.465 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 41.504717589000002, 2024-09-11T09:52:04.465 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0041642340000000002, 2024-09-11T09:52:04.465 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:04.466 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:04.466 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:04.466 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:04.466 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:52:04.466 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:52:04.466 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 48 2024-09-11T09:52:04.466 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.466 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:04.466 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.466 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:04.466 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.938799+0000", 2024-09-11T09:52:04.466 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:04.466 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.467 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.467 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:04.467 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.938799+0000", 2024-09-11T09:52:04.467 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:04.467 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.467 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.467 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:04.467 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.938800+0000", 2024-09-11T09:52:04.467 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.71e-07 2024-09-11T09:52:04.467 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.467 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.467 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:04.467 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.938805+0000", 2024-09-11T09:52:04.467 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.3140000000000003e-06 2024-09-11T09:52:04.468 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.468 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.468 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:04.468 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.938816+0000", 2024-09-11T09:52:04.468 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0231000000000001e-05 2024-09-11T09:52:04.468 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.468 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.468 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:04.468 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.938822+0000", 2024-09-11T09:52:04.468 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.9550000000000003e-06 2024-09-11T09:52:04.468 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.468 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.468 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:04.469 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.938868+0000", 2024-09-11T09:52:04.469 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.5562999999999997e-05 2024-09-11T09:52:04.469 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.469 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.469 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:04.469 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.938918+0000", 2024-09-11T09:52:04.469 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.0317e-05 2024-09-11T09:52:04.469 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.469 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.469 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 5", 2024-09-11T09:52:04.469 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.939114+0000", 2024-09-11T09:52:04.469 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00019538300000000001 2024-09-11T09:52:04.469 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.469 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.470 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:04.470 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.939663+0000", 2024-09-11T09:52:04.470 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00054932199999999998 2024-09-11T09:52:04.470 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.470 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.470 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:04.470 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.942910+0000", 2024-09-11T09:52:04.470 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0032473839999999999 2024-09-11T09:52:04.470 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.470 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.470 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:04.470 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.942940+0000", 2024-09-11T09:52:04.470 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.9306999999999999e-05 2024-09-11T09:52:04.470 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.471 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.471 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:04.471 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.942963+0000", 2024-09-11T09:52:04.471 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.3587e-05 2024-09-11T09:52:04.471 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:04.471 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:04.471 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:04.471 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.471 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.471 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4688.0:49 2.5 e53/32 2:b6dd9fc9:::rbd_data.1250f5e25b80.0000000000000009:head v 53'2, mlcod=0'0)", 2024-09-11T09:52:04.471 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.944712+0000", 2024-09-11T09:52:04.471 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 41.498804749999998, 2024-09-11T09:52:04.471 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.016185069999999999, 2024-09-11T09:52:04.471 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:04.472 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:04.472 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:04.472 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:04.472 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.472 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:04.472 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.944712+0000", 2024-09-11T09:52:04.472 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:04.472 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.472 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.472 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:04.472 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.944712+0000", 2024-09-11T09:52:04.473 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:04.473 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.473 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.473 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:04.473 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.944713+0000", 2024-09-11T09:52:04.473 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0139999999999999e-06 2024-09-11T09:52:04.473 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.473 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.473 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:04.473 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.955518+0000", 2024-09-11T09:52:04.473 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.01080542 2024-09-11T09:52:04.473 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.474 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.474 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:04.474 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.955536+0000", 2024-09-11T09:52:04.474 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7818000000000001e-05 2024-09-11T09:52:04.474 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.474 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.474 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:04.474 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.955544+0000", 2024-09-11T09:52:04.474 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.1699999999999997e-06 2024-09-11T09:52:04.474 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.475 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.475 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:04.475 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.955607+0000", 2024-09-11T09:52:04.475 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.2451000000000002e-05 2024-09-11T09:52:04.475 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.475 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.475 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:04.475 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.955632+0000", 2024-09-11T09:52:04.475 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.5335000000000001e-05 2024-09-11T09:52:04.475 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.475 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.475 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:04.475 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.960826+0000", 2024-09-11T09:52:04.476 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0051938560000000002 2024-09-11T09:52:04.476 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.476 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.476 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:04.476 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.960897+0000", 2024-09-11T09:52:04.476 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.1006000000000004e-05 2024-09-11T09:52:04.476 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:04.476 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:04.476 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:04.476 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.476 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.477 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:53 2.7 2:fe41411c:::rbd_object_map.1250f5e25b80:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:04.477 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.951291+0000", 2024-09-11T09:52:04.477 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 41.492225759999997, 2024-09-11T09:52:04.477 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0044964999999999996, 2024-09-11T09:52:04.477 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:04.477 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:04.477 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:04.477 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:04.477 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:52:04.478 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:52:04.478 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 53 2024-09-11T09:52:04.478 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.478 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:04.478 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.478 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:04.478 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951291+0000", 2024-09-11T09:52:04.478 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:04.478 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.478 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.478 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:04.478 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951291+0000", 2024-09-11T09:52:04.479 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:04.479 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.479 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.479 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:04.479 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951293+0000", 2024-09-11T09:52:04.479 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7889999999999999e-06 2024-09-11T09:52:04.479 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.479 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.479 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:04.479 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951302+0000", 2024-09-11T09:52:04.479 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.9819999999999997e-06 2024-09-11T09:52:04.479 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.479 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.480 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:04.480 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951324+0000", 2024-09-11T09:52:04.480 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.2141e-05 2024-09-11T09:52:04.480 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.480 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.480 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:04.480 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951338+0000", 2024-09-11T09:52:04.480 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3750999999999999e-05 2024-09-11T09:52:04.480 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.480 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.480 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:04.480 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951409+0000", 2024-09-11T09:52:04.480 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.1658999999999999e-05 2024-09-11T09:52:04.480 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.481 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.481 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:04.481 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951517+0000", 2024-09-11T09:52:04.481 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00010746799999999999 2024-09-11T09:52:04.481 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.481 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.481 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 5", 2024-09-11T09:52:04.481 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951942+0000", 2024-09-11T09:52:04.481 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00042524300000000001 2024-09-11T09:52:04.481 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.481 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.481 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:04.481 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.952872+0000", 2024-09-11T09:52:04.482 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00092976300000000001 2024-09-11T09:52:04.482 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.482 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.482 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:04.482 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.955682+0000", 2024-09-11T09:52:04.482 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0028099459999999998 2024-09-11T09:52:04.482 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.482 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.482 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:04.482 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.955746+0000", 2024-09-11T09:52:04.482 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.4084999999999994e-05 2024-09-11T09:52:04.482 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.482 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.482 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:04.483 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.955787+0000", 2024-09-11T09:52:04.483 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.1672999999999999e-05 2024-09-11T09:52:04.483 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:04.483 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:04.483 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:04.483 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.483 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.483 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4688.0:52 2.4 e53/32 2:26d5c2f0:::rbd_data.1250f5e25b80.000000000000000b:head v 53'8, mlcod=53'6)", 2024-09-11T09:52:04.483 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.963677+0000", 2024-09-11T09:52:04.483 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 41.479840205000002, 2024-09-11T09:52:04.483 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0089437969999999999, 2024-09-11T09:52:04.483 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:04.483 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:04.483 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:04.484 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:04.484 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.484 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:04.484 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.963677+0000", 2024-09-11T09:52:04.484 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:04.484 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.484 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.484 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:04.484 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.963677+0000", 2024-09-11T09:52:04.484 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:04.484 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.484 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.484 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:04.485 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.963677+0000", 2024-09-11T09:52:04.485 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.7199999999999998e-07 2024-09-11T09:52:04.485 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.485 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.485 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:04.485 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.965506+0000", 2024-09-11T09:52:04.485 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.001829064 2024-09-11T09:52:04.485 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.485 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.485 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:04.485 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.965538+0000", 2024-09-11T09:52:04.485 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.1306999999999997e-05 2024-09-11T09:52:04.485 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.485 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.486 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:04.486 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.965553+0000", 2024-09-11T09:52:04.486 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.4924e-05 2024-09-11T09:52:04.486 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.486 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.486 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:04.486 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.965663+0000", 2024-09-11T09:52:04.486 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000110315 2024-09-11T09:52:04.486 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.486 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.486 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:04.486 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.965692+0000", 2024-09-11T09:52:04.487 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.934e-05 2024-09-11T09:52:04.487 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.487 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.487 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:04.487 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.972537+0000", 2024-09-11T09:52:04.487 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0068444969999999997 2024-09-11T09:52:04.487 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.487 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.487 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:04.487 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.972620+0000", 2024-09-11T09:52:04.487 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.3578000000000005e-05 2024-09-11T09:52:04.487 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:04.487 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:04.488 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:04.488 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.488 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.488 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4688.0:54 2.4 e53/32 2:239e915b:::rbd_data.1250f5e25b80.000000000000000c:head v 53'9, mlcod=53'7)", 2024-09-11T09:52:04.488 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.987028+0000", 2024-09-11T09:52:04.488 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 41.456488288000003, 2024-09-11T09:52:04.488 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0094155650000000007, 2024-09-11T09:52:04.488 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:04.488 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:04.488 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:04.488 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:04.488 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.488 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:04.489 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.987028+0000", 2024-09-11T09:52:04.489 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:04.489 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.489 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.489 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:04.489 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.987028+0000", 2024-09-11T09:52:04.489 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:04.489 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.489 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.489 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:04.489 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.987030+0000", 2024-09-11T09:52:04.489 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6130000000000001e-06 2024-09-11T09:52:04.489 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.489 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.490 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:04.490 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.988575+0000", 2024-09-11T09:52:04.490 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.001545193 2024-09-11T09:52:04.490 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.490 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.490 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:04.490 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.988604+0000", 2024-09-11T09:52:04.490 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.9173e-05 2024-09-11T09:52:04.490 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.490 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.490 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:04.490 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.988620+0000", 2024-09-11T09:52:04.490 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.5517999999999999e-05 2024-09-11T09:52:04.491 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.491 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.491 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:04.491 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.988704+0000", 2024-09-11T09:52:04.491 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.3588e-05 2024-09-11T09:52:04.491 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.491 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.491 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:04.491 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.988735+0000", 2024-09-11T09:52:04.491 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.1643000000000002e-05 2024-09-11T09:52:04.491 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.491 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.491 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:04.492 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.996383+0000", 2024-09-11T09:52:04.492 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0076478099999999997 2024-09-11T09:52:04.492 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.492 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.492 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:04.492 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.996444+0000", 2024-09-11T09:52:04.492 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.1026999999999997e-05 2024-09-11T09:52:04.492 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:04.492 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:04.492 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:04.492 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.492 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.493 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4688.0:56 2.4 e53/32 2:2167012f:::rbd_data.1250f5e25b80.000000000000000d:head v 53'10, mlcod=53'7)", 2024-09-11T09:52:04.493 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.993894+0000", 2024-09-11T09:52:04.493 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 41.449622746000003, 2024-09-11T09:52:04.493 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0099268559999999995, 2024-09-11T09:52:04.493 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:04.493 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:04.493 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:04.493 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:04.493 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.493 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:04.493 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.993894+0000", 2024-09-11T09:52:04.494 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:04.494 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.494 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.494 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:04.494 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.993894+0000", 2024-09-11T09:52:04.494 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:04.494 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.494 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.494 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:04.494 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.993896+0000", 2024-09-11T09:52:04.494 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6249999999999999e-06 2024-09-11T09:52:04.494 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.495 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.495 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:04.495 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.997606+0000", 2024-09-11T09:52:04.495 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.003710737 2024-09-11T09:52:04.495 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.495 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.495 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:04.495 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.997628+0000", 2024-09-11T09:52:04.495 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.1971e-05 2024-09-11T09:52:04.495 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.495 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.495 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:04.495 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.997640+0000", 2024-09-11T09:52:04.495 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1481000000000001e-05 2024-09-11T09:52:04.496 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.496 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.496 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:04.496 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.997712+0000", 2024-09-11T09:52:04.496 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.2255000000000005e-05 2024-09-11T09:52:04.496 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.496 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.496 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:04.496 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.997738+0000", 2024-09-11T09:52:04.496 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.6050999999999998e-05 2024-09-11T09:52:04.497 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.497 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.497 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:04.497 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.003779+0000", 2024-09-11T09:52:04.497 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0060410960000000001 2024-09-11T09:52:04.497 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.497 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.497 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:04.497 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.003821+0000", 2024-09-11T09:52:04.497 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.1640000000000001e-05 2024-09-11T09:52:04.497 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:04.497 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:04.497 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:04.497 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.498 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.498 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "pg_update_log_missing(2.4 epoch 53/32 rep_tid 16 entries 53'11 (0'0) error 2:2906e6a7:::rbd_header.1250f5e25b80:head by client.4688.0:59 0.000000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 53'7)", 2024-09-11T09:52:04.498 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.998956+0000", 2024-09-11T09:52:04.498 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 41.444560576999997, 2024-09-11T09:52:04.498 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.004934127, 2024-09-11T09:52:04.498 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:04.498 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:04.498 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "reached pg", 2024-09-11T09:52:04.498 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:04.498 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.498 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:04.498 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.998956+0000", 2024-09-11T09:52:04.498 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:04.498 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.499 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.499 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:04.499 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.998956+0000", 2024-09-11T09:52:04.499 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:04.499 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.499 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.499 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:04.499 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.998957+0000", 2024-09-11T09:52:04.499 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.139e-06 2024-09-11T09:52:04.499 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.499 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.499 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:04.499 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.998985+0000", 2024-09-11T09:52:04.499 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.7912e-05 2024-09-11T09:52:04.500 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.500 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.500 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:04.500 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.999011+0000", 2024-09-11T09:52:04.500 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.6061e-05 2024-09-11T09:52:04.500 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.500 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.500 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:04.500 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.999028+0000", 2024-09-11T09:52:04.500 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7065999999999999e-05 2024-09-11T09:52:04.500 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.500 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.500 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:04.500 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.001919+0000", 2024-09-11T09:52:04.501 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.002891167 2024-09-11T09:52:04.501 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.501 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.501 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:04.501 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.003890+0000", 2024-09-11T09:52:04.501 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0019707819999999999 2024-09-11T09:52:04.501 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:04.501 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:04.501 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:04.501 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.501 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.501 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4694.0:34 2.7 2:f80bfc0c:ns::rbd_data.1256a0c02879.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:04.501 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.085183+0000", 2024-09-11T09:52:04.502 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 41.358334063999997, 2024-09-11T09:52:04.502 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.020908599, 2024-09-11T09:52:04.502 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:04.502 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:04.502 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:04.502 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:04.502 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4694", 2024-09-11T09:52:04.502 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:52:04.502 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 34 2024-09-11T09:52:04.502 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.502 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:04.502 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.502 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:04.503 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.085183+0000", 2024-09-11T09:52:04.503 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:04.503 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.503 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.503 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:04.503 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.085183+0000", 2024-09-11T09:52:04.503 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:04.503 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.503 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.503 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:04.503 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.085183+0000", 2024-09-11T09:52:04.503 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.4300000000000002e-07 2024-09-11T09:52:04.503 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.503 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.504 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:04.504 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094337+0000", 2024-09-11T09:52:04.504 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0091532769999999996 2024-09-11T09:52:04.504 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.504 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.504 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:04.504 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094362+0000", 2024-09-11T09:52:04.504 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.5287999999999999e-05 2024-09-11T09:52:04.504 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.504 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.504 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:04.504 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094375+0000", 2024-09-11T09:52:04.504 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2446000000000001e-05 2024-09-11T09:52:04.505 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.505 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.505 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:04.505 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094455+0000", 2024-09-11T09:52:04.505 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.0092999999999994e-05 2024-09-11T09:52:04.505 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.505 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.505 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:04.505 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094618+0000", 2024-09-11T09:52:04.505 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000163747 2024-09-11T09:52:04.505 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.505 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.505 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 5", 2024-09-11T09:52:04.506 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094783+0000", 2024-09-11T09:52:04.506 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000165038 2024-09-11T09:52:04.506 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.506 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.506 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:04.506 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.105853+0000", 2024-09-11T09:52:04.506 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.011069294 2024-09-11T09:52:04.506 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.506 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.506 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:04.506 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.106004+0000", 2024-09-11T09:52:04.506 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00015123900000000001 2024-09-11T09:52:04.506 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.507 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.507 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:04.507 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.106062+0000", 2024-09-11T09:52:04.507 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.8205000000000003e-05 2024-09-11T09:52:04.507 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.507 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.507 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:04.507 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.106091+0000", 2024-09-11T09:52:04.507 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.9128999999999998e-05 2024-09-11T09:52:04.507 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:04.507 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:04.507 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:04.507 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.507 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.508 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4694.0:35 2.7 2:e8766325:ns::rbd_object_map.1256a0c02879:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:04.508 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.094405+0000", 2024-09-11T09:52:04.508 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 41.349111428000001, 2024-09-11T09:52:04.508 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.012623564, 2024-09-11T09:52:04.508 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:04.508 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:04.508 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:04.508 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:04.508 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4694", 2024-09-11T09:52:04.508 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:52:04.508 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 35 2024-09-11T09:52:04.508 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.508 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:04.509 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.509 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:04.509 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094405+0000", 2024-09-11T09:52:04.509 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:04.509 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.509 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.509 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:04.509 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094405+0000", 2024-09-11T09:52:04.509 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:04.509 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.509 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.509 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:04.509 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094407+0000", 2024-09-11T09:52:04.509 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.739e-06 2024-09-11T09:52:04.510 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.510 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.510 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:04.510 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094411+0000", 2024-09-11T09:52:04.510 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.3340000000000003e-06 2024-09-11T09:52:04.510 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.510 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.510 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:04.510 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094419+0000", 2024-09-11T09:52:04.510 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.8909999999999995e-06 2024-09-11T09:52:04.510 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.510 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.510 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:04.511 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094424+0000", 2024-09-11T09:52:04.511 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.7639999999999996e-06 2024-09-11T09:52:04.511 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.511 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.511 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:04.511 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.100209+0000", 2024-09-11T09:52:04.511 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0057850489999999996 2024-09-11T09:52:04.511 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.511 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.511 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:04.511 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.100341+0000", 2024-09-11T09:52:04.511 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000131438 2024-09-11T09:52:04.511 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.511 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.512 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 5", 2024-09-11T09:52:04.512 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.100805+0000", 2024-09-11T09:52:04.512 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00046471499999999998 2024-09-11T09:52:04.512 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.512 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.512 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:04.512 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.106148+0000", 2024-09-11T09:52:04.512 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0053426460000000004 2024-09-11T09:52:04.512 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.512 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.512 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:04.512 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.106929+0000", 2024-09-11T09:52:04.512 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00078161300000000001 2024-09-11T09:52:04.513 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.513 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.513 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:04.513 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.106985+0000", 2024-09-11T09:52:04.513 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.5223999999999998e-05 2024-09-11T09:52:04.513 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.513 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.513 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:04.513 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.107029+0000", 2024-09-11T09:52:04.513 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.4150999999999998e-05 2024-09-11T09:52:04.513 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:04.513 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:04.513 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:04.513 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.514 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.514 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4694.0:36 2.7 2:e8766325:ns::rbd_object_map.1256a0c02879:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:04.514 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.094452+0000", 2024-09-11T09:52:04.514 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 41.349064417999998, 2024-09-11T09:52:04.514 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.012680383, 2024-09-11T09:52:04.514 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:04.514 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:04.514 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:04.514 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:04.514 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4694", 2024-09-11T09:52:04.514 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:52:04.514 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 36 2024-09-11T09:52:04.515 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.515 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:04.515 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.515 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:04.515 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094452+0000", 2024-09-11T09:52:04.515 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:04.515 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.515 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.515 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:04.515 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094452+0000", 2024-09-11T09:52:04.515 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:04.515 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.515 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.515 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:04.516 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094453+0000", 2024-09-11T09:52:04.516 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.1800000000000002e-07 2024-09-11T09:52:04.516 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.516 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.516 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:04.516 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094455+0000", 2024-09-11T09:52:04.516 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.7020000000000001e-06 2024-09-11T09:52:04.516 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.516 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.516 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:04.516 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094463+0000", 2024-09-11T09:52:04.516 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.1999999999999997e-06 2024-09-11T09:52:04.516 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.517 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.517 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:04.517 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094467+0000", 2024-09-11T09:52:04.517 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.729e-06 2024-09-11T09:52:04.517 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.517 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.517 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:04.517 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.101269+0000", 2024-09-11T09:52:04.517 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0068018799999999997 2024-09-11T09:52:04.517 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.517 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.517 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:04.517 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.101353+0000", 2024-09-11T09:52:04.518 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.3827999999999998e-05 2024-09-11T09:52:04.518 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.518 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.518 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 5", 2024-09-11T09:52:04.518 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.101588+0000", 2024-09-11T09:52:04.518 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00023467099999999999 2024-09-11T09:52:04.518 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.518 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.518 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:04.518 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.106182+0000", 2024-09-11T09:52:04.518 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0045939489999999999 2024-09-11T09:52:04.518 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.518 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.518 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:04.519 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.107041+0000", 2024-09-11T09:52:04.519 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00085965800000000002 2024-09-11T09:52:04.519 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.519 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.519 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:04.519 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.107084+0000", 2024-09-11T09:52:04.519 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.261e-05 2024-09-11T09:52:04.519 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.519 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.519 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:04.519 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.107133+0000", 2024-09-11T09:52:04.519 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.8838000000000002e-05 2024-09-11T09:52:04.519 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:04.520 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:04.520 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:04.520 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.520 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.520 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4694.0:37 2.7 2:e8766325:ns::rbd_object_map.1256a0c02879:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:04.520 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.094473+0000", 2024-09-11T09:52:04.520 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 41.349043266999999, 2024-09-11T09:52:04.520 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.012778484999999999, 2024-09-11T09:52:04.520 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:04.520 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:04.520 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:04.520 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:04.520 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4694", 2024-09-11T09:52:04.521 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:52:04.521 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 37 2024-09-11T09:52:04.521 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.521 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:04.521 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.521 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:04.521 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094473+0000", 2024-09-11T09:52:04.521 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:04.521 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.521 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.521 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:04.521 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094473+0000", 2024-09-11T09:52:04.521 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:04.521 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.522 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.522 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:04.522 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094474+0000", 2024-09-11T09:52:04.522 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.9699999999999996e-07 2024-09-11T09:52:04.522 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.522 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.522 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:04.522 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094477+0000", 2024-09-11T09:52:04.522 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.151e-06 2024-09-11T09:52:04.522 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.522 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.522 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:04.523 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094483+0000", 2024-09-11T09:52:04.523 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.6330000000000001e-06 2024-09-11T09:52:04.523 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.523 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.523 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:04.523 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094487+0000", 2024-09-11T09:52:04.523 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.5399999999999997e-06 2024-09-11T09:52:04.523 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.523 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.523 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:04.523 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.101866+0000", 2024-09-11T09:52:04.524 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.007378768 2024-09-11T09:52:04.524 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.524 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.524 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:04.524 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.101960+0000", 2024-09-11T09:52:04.524 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.3861000000000001e-05 2024-09-11T09:52:04.524 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.524 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.524 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 5", 2024-09-11T09:52:04.524 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.102197+0000", 2024-09-11T09:52:04.524 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00023723200000000001 2024-09-11T09:52:04.524 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.524 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.525 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:04.525 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.106222+0000", 2024-09-11T09:52:04.525 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0040249689999999998 2024-09-11T09:52:04.525 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.525 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.525 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:04.525 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.107141+0000", 2024-09-11T09:52:04.525 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00091913900000000002 2024-09-11T09:52:04.525 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.525 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.525 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:04.525 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.107198+0000", 2024-09-11T09:52:04.525 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.6564999999999998e-05 2024-09-11T09:52:04.526 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.526 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.526 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:04.526 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.107252+0000", 2024-09-11T09:52:04.526 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.4030000000000003e-05 2024-09-11T09:52:04.526 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:04.526 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:04.526 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:04.526 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.526 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.526 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4694.0:33 2.4 e53/32 2:3bfc818c:ns::rbd_data.1256a0c02879.0000000000000001:head v 53'14, mlcod=53'12)", 2024-09-11T09:52:04.526 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.096496+0000", 2024-09-11T09:52:04.527 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 41.347020694000001, 2024-09-11T09:52:04.527 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0099139310000000008, 2024-09-11T09:52:04.527 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:04.527 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:04.527 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:04.527 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:04.527 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.527 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:04.527 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.096496+0000", 2024-09-11T09:52:04.527 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:04.527 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.527 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.527 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:04.528 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.096496+0000", 2024-09-11T09:52:04.528 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:04.528 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.528 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.528 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:04.528 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.096498+0000", 2024-09-11T09:52:04.528 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6309999999999999e-06 2024-09-11T09:52:04.528 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.528 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.528 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:04.529 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.098087+0000", 2024-09-11T09:52:04.529 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0015889019999999999 2024-09-11T09:52:04.529 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.529 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.529 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:04.529 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.098112+0000", 2024-09-11T09:52:04.529 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.5369000000000001e-05 2024-09-11T09:52:04.529 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.529 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.529 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:04.529 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.098127+0000", 2024-09-11T09:52:04.529 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.5218999999999999e-05 2024-09-11T09:52:04.529 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.530 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.530 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:04.530 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.098209+0000", 2024-09-11T09:52:04.530 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.2120999999999995e-05 2024-09-11T09:52:04.530 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.530 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.530 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:04.530 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.098237+0000", 2024-09-11T09:52:04.530 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.7657999999999998e-05 2024-09-11T09:52:04.530 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.530 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.530 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:04.530 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.106376+0000", 2024-09-11T09:52:04.531 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0081386749999999997 2024-09-11T09:52:04.531 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.531 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.531 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:04.531 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.106410+0000", 2024-09-11T09:52:04.531 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.4356000000000001e-05 2024-09-11T09:52:04.531 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:04.531 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:04.531 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:04.531 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.531 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.531 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4694.0:49 2.7 2:e8766325:ns::rbd_object_map.1256a0c02879:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:04.531 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.134806+0000", 2024-09-11T09:52:04.531 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 41.308710625000003, 2024-09-11T09:52:04.532 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0061629049999999998, 2024-09-11T09:52:04.532 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:04.532 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:04.532 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:04.532 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:04.532 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4694", 2024-09-11T09:52:04.532 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:52:04.532 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 49 2024-09-11T09:52:04.532 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.532 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:04.532 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.532 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:04.532 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.134806+0000", 2024-09-11T09:52:04.533 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:04.533 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.533 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.533 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:04.533 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.134806+0000", 2024-09-11T09:52:04.533 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:04.533 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.533 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.533 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:04.533 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.134808+0000", 2024-09-11T09:52:04.533 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.9870000000000002e-06 2024-09-11T09:52:04.533 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.533 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.533 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:04.534 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.134818+0000", 2024-09-11T09:52:04.534 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.5599999999999999e-06 2024-09-11T09:52:04.534 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.534 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.534 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:04.534 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.134835+0000", 2024-09-11T09:52:04.534 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7399000000000001e-05 2024-09-11T09:52:04.534 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.534 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.534 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:04.534 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.134845+0000", 2024-09-11T09:52:04.534 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0285000000000001e-05 2024-09-11T09:52:04.534 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.535 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.535 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:04.535 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.134911+0000", 2024-09-11T09:52:04.535 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.5654999999999994e-05 2024-09-11T09:52:04.535 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.535 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.535 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:04.535 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.135023+0000", 2024-09-11T09:52:04.535 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00011186799999999999 2024-09-11T09:52:04.535 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.535 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.535 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 5", 2024-09-11T09:52:04.535 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.135290+0000", 2024-09-11T09:52:04.536 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00026725300000000001 2024-09-11T09:52:04.536 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.536 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.536 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:04.536 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.136506+0000", 2024-09-11T09:52:04.536 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0012161139999999999 2024-09-11T09:52:04.536 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.536 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.536 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:04.536 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.140884+0000", 2024-09-11T09:52:04.536 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0043781990000000002 2024-09-11T09:52:04.536 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.536 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.537 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:04.537 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.140923+0000", 2024-09-11T09:52:04.537 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.8813999999999998e-05 2024-09-11T09:52:04.537 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.537 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.537 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:04.537 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.140969+0000", 2024-09-11T09:52:04.537 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.5770999999999999e-05 2024-09-11T09:52:04.537 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:04.537 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:04.537 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:04.537 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.537 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.537 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4694.0:42 2.5 e53/32 2:a4fc00ec:ns::rbd_data.1256a0c02879.0000000000000004:head v 53'3, mlcod=53'1)", 2024-09-11T09:52:04.538 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.136646+0000", 2024-09-11T09:52:04.538 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 41.306870318999998, 2024-09-11T09:52:04.538 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.012545501000000001, 2024-09-11T09:52:04.538 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:04.538 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:04.538 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:04.538 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:04.538 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.538 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:04.538 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.136646+0000", 2024-09-11T09:52:04.538 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:04.538 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.538 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.538 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:04.539 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.136646+0000", 2024-09-11T09:52:04.539 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:04.539 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.539 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.539 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:04.539 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.136647+0000", 2024-09-11T09:52:04.539 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.2900000000000002e-07 2024-09-11T09:52:04.539 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.539 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.539 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:04.539 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.143620+0000", 2024-09-11T09:52:04.539 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0069727440000000003 2024-09-11T09:52:04.539 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.540 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.540 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:04.540 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.143642+0000", 2024-09-11T09:52:04.540 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.1846000000000001e-05 2024-09-11T09:52:04.540 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.540 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.540 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:04.540 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.143654+0000", 2024-09-11T09:52:04.540 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2081e-05 2024-09-11T09:52:04.540 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.540 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.541 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:04.541 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.143717+0000", 2024-09-11T09:52:04.541 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.3443999999999998e-05 2024-09-11T09:52:04.541 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.541 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.541 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:04.541 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.143740+0000", 2024-09-11T09:52:04.541 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.2626999999999999e-05 2024-09-11T09:52:04.541 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.541 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.542 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:04.542 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.149142+0000", 2024-09-11T09:52:04.542 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0054017630000000004 2024-09-11T09:52:04.542 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.542 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.542 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:04.542 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.149192+0000", 2024-09-11T09:52:04.542 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.0067e-05 2024-09-11T09:52:04.542 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:04.542 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:04.542 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:04.542 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.542 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.543 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4694.0:53 2.7 2:e8766325:ns::rbd_object_map.1256a0c02879:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:04.543 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.147978+0000", 2024-09-11T09:52:04.543 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 41.295538835000002, 2024-09-11T09:52:04.543 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0044549769999999997, 2024-09-11T09:52:04.543 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:04.543 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:04.543 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:04.543 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:04.543 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4694", 2024-09-11T09:52:04.543 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:52:04.543 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 53 2024-09-11T09:52:04.543 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.543 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:04.544 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.544 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:04.544 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.147978+0000", 2024-09-11T09:52:04.544 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:04.544 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.544 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.544 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:04.544 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.147978+0000", 2024-09-11T09:52:04.544 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:04.544 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.544 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.544 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:04.545 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.147980+0000", 2024-09-11T09:52:04.545 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.8309999999999999e-06 2024-09-11T09:52:04.545 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.545 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.545 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:04.545 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.147989+0000", 2024-09-11T09:52:04.545 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.9639999999999992e-06 2024-09-11T09:52:04.545 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.545 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.545 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:04.545 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.148007+0000", 2024-09-11T09:52:04.545 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7884999999999999e-05 2024-09-11T09:52:04.545 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.545 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.546 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:04.546 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.148017+0000", 2024-09-11T09:52:04.546 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0895e-05 2024-09-11T09:52:04.546 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.546 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.546 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:04.546 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.148091+0000", 2024-09-11T09:52:04.546 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.3709999999999997e-05 2024-09-11T09:52:04.546 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.546 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.546 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:04.546 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.148189+0000", 2024-09-11T09:52:04.546 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.7403000000000002e-05 2024-09-11T09:52:04.547 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.547 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.547 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 5", 2024-09-11T09:52:04.547 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.148504+0000", 2024-09-11T09:52:04.547 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00031589900000000002 2024-09-11T09:52:04.547 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.547 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.547 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:04.547 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.149514+0000", 2024-09-11T09:52:04.547 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0010096 2024-09-11T09:52:04.547 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.547 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.547 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:04.548 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.152367+0000", 2024-09-11T09:52:04.548 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0028532309999999999 2024-09-11T09:52:04.548 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.548 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.548 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:04.548 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.152402+0000", 2024-09-11T09:52:04.548 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.4232e-05 2024-09-11T09:52:04.548 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.548 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.548 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:04.548 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.152433+0000", 2024-09-11T09:52:04.548 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.1327e-05 2024-09-11T09:52:04.549 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:04.549 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:04.549 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:04.549 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.549 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.549 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4694.0:55 2.7 2:e8766325:ns::rbd_object_map.1256a0c02879:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:04.549 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.149838+0000", 2024-09-11T09:52:04.549 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 41.293679101999999, 2024-09-11T09:52:04.549 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0060695159999999996, 2024-09-11T09:52:04.549 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:04.549 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:04.549 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:04.549 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:04.550 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4694", 2024-09-11T09:52:04.550 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:52:04.550 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 55 2024-09-11T09:52:04.550 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.550 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:04.550 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.550 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:04.550 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.149838+0000", 2024-09-11T09:52:04.550 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:04.550 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.550 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.550 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:04.550 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.149838+0000", 2024-09-11T09:52:04.551 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:04.551 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.551 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.551 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:04.551 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.149839+0000", 2024-09-11T09:52:04.551 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.119e-06 2024-09-11T09:52:04.551 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.551 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.551 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:04.551 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.149845+0000", 2024-09-11T09:52:04.551 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.7810000000000002e-06 2024-09-11T09:52:04.551 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.552 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.552 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:04.552 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.149856+0000", 2024-09-11T09:52:04.552 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1199999999999999e-05 2024-09-11T09:52:04.552 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.552 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.552 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:04.552 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.149863+0000", 2024-09-11T09:52:04.552 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.4109999999999999e-06 2024-09-11T09:52:04.552 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.552 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.552 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:04.552 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.149908+0000", 2024-09-11T09:52:04.553 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.5184e-05 2024-09-11T09:52:04.553 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.553 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.553 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:04.553 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.149964+0000", 2024-09-11T09:52:04.553 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.5395000000000001e-05 2024-09-11T09:52:04.553 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.553 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.553 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 5", 2024-09-11T09:52:04.553 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.150157+0000", 2024-09-11T09:52:04.553 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000193489 2024-09-11T09:52:04.553 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.553 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.554 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:04.554 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.152238+0000", 2024-09-11T09:52:04.554 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0020803480000000001 2024-09-11T09:52:04.554 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.554 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.554 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:04.554 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.155828+0000", 2024-09-11T09:52:04.554 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0035902210000000002 2024-09-11T09:52:04.554 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.554 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.554 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:04.554 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.155873+0000", 2024-09-11T09:52:04.554 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.5145999999999998e-05 2024-09-11T09:52:04.555 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.555 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.555 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:04.555 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.155907+0000", 2024-09-11T09:52:04.555 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.4221999999999999e-05 2024-09-11T09:52:04.555 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:04.555 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:04.555 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:04.555 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:04.555 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:04.555 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:52:04.557 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-11T09:52:04.557 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:52:04.557+0000 7f3a7b90a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 42475) UID: 0 2024-09-11T09:52:04.658 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:52:04.659+0000 7f3c96ec9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31909) UID: 0 2024-09-11T09:52:04.666 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:52:04.666 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [], 2024-09-11T09:52:04.666 INFO:teuthology.orchestra.run.smithi017.stdout: "num_ops": 0 2024-09-11T09:52:04.666 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:52:04.675 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-11T09:52:04.759 INFO:tasks.ceph.osd.2.smithi017.stderr:2024-09-11T09:52:04.760+0000 7fd20b170640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31672) UID: 0 2024-09-11T09:52:04.782 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:52:04.782 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [], 2024-09-11T09:52:04.783 INFO:teuthology.orchestra.run.smithi017.stdout: "complaint_time": 30, 2024-09-11T09:52:04.783 INFO:teuthology.orchestra.run.smithi017.stdout: "num_blocked_ops": 0 2024-09-11T09:52:04.783 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:52:04.791 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-11T09:52:04.875 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:52:04.876+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:52:04.940 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:52:04.940 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 20, 2024-09-11T09:52:04.940 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 600, 2024-09-11T09:52:04.940 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [ 2024-09-11T09:52:04.940 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.941 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:34 2.4 2:2367385a:::rbd_data.1250f5e25b80.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:04.941 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.897717+0000", 2024-09-11T09:52:04.941 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 42.042402070999998, 2024-09-11T09:52:04.941 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.021323661000000001, 2024-09-11T09:52:04.941 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:04.941 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:04.941 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:04.941 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:04.941 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:52:04.941 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:52:04.941 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 34 2024-09-11T09:52:04.941 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.942 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:04.942 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.942 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:04.942 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.897717+0000", 2024-09-11T09:52:04.942 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:04.942 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.942 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.942 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:04.942 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.897717+0000", 2024-09-11T09:52:04.942 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:04.942 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.942 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.942 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:04.942 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.897718+0000", 2024-09-11T09:52:04.943 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.1800000000000004e-07 2024-09-11T09:52:04.943 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.943 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.943 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:04.943 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.905339+0000", 2024-09-11T09:52:04.943 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0076210239999999997 2024-09-11T09:52:04.943 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.943 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.943 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:04.943 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.905366+0000", 2024-09-11T09:52:04.944 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.6959000000000001e-05 2024-09-11T09:52:04.944 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.944 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.944 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:04.944 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.905375+0000", 2024-09-11T09:52:04.944 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.516e-06 2024-09-11T09:52:04.944 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.944 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.944 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:04.944 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.905475+0000", 2024-09-11T09:52:04.944 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.9738e-05 2024-09-11T09:52:04.944 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.944 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.945 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:04.945 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.905660+0000", 2024-09-11T09:52:04.945 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00018554199999999999 2024-09-11T09:52:04.945 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.945 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.945 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 0", 2024-09-11T09:52:04.945 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.905808+0000", 2024-09-11T09:52:04.945 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000147386 2024-09-11T09:52:04.945 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.945 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.945 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:04.945 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.918230+0000", 2024-09-11T09:52:04.946 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.012421690000000001 2024-09-11T09:52:04.946 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.946 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.946 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:04.946 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.918932+0000", 2024-09-11T09:52:04.946 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00070267300000000001 2024-09-11T09:52:04.946 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.946 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.946 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:04.946 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.918989+0000", 2024-09-11T09:52:04.946 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.7268999999999997e-05 2024-09-11T09:52:04.946 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.946 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.946 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:04.947 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919040+0000", 2024-09-11T09:52:04.947 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.0946000000000003e-05 2024-09-11T09:52:04.947 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:04.947 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:04.947 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:04.947 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.947 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.947 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4688.0:31 2.1 e53/15 2:8654721b:::rbd_data.1250f5e25b80.0000000000000000:head v 53'2, mlcod=0'0)", 2024-09-11T09:52:04.947 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.904661+0000", 2024-09-11T09:52:04.947 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 42.035457581999999, 2024-09-11T09:52:04.947 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.014152804999999999, 2024-09-11T09:52:04.947 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:04.947 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:04.948 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:04.948 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:04.948 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.948 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:04.948 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.904661+0000", 2024-09-11T09:52:04.948 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:04.948 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.948 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.948 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:04.948 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.904661+0000", 2024-09-11T09:52:04.948 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:04.948 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.948 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.949 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:04.949 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.904662+0000", 2024-09-11T09:52:04.949 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0759999999999999e-06 2024-09-11T09:52:04.949 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.949 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.949 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:04.949 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.907298+0000", 2024-09-11T09:52:04.949 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0026361240000000001 2024-09-11T09:52:04.949 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.949 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.949 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:04.949 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.907320+0000", 2024-09-11T09:52:04.949 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.1268999999999999e-05 2024-09-11T09:52:04.949 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.950 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.950 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:04.950 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.907331+0000", 2024-09-11T09:52:04.950 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1031e-05 2024-09-11T09:52:04.950 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.951 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.951 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:04.951 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.907403+0000", 2024-09-11T09:52:04.951 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.2571999999999996e-05 2024-09-11T09:52:04.951 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.952 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.952 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:04.952 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.907428+0000", 2024-09-11T09:52:04.952 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.4380999999999999e-05 2024-09-11T09:52:04.952 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.952 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.952 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:04.952 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.918750+0000", 2024-09-11T09:52:04.952 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.011322211 2024-09-11T09:52:04.952 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.953 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.953 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:04.953 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.918814+0000", 2024-09-11T09:52:04.953 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.4140999999999995e-05 2024-09-11T09:52:04.953 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:04.953 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:04.953 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:04.953 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.953 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.953 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:36 2.6 2:61e1b442:::rbd_data.1250f5e25b80.0000000000000003:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:04.953 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.905404+0000", 2024-09-11T09:52:04.953 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 42.034714710999999, 2024-09-11T09:52:04.954 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.030912479999999999, 2024-09-11T09:52:04.954 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:04.954 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:04.954 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:04.954 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:04.954 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:52:04.954 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:52:04.954 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 36 2024-09-11T09:52:04.954 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.954 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:04.954 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.954 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:04.955 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.905404+0000", 2024-09-11T09:52:04.955 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:04.955 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.955 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.955 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:04.955 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.905404+0000", 2024-09-11T09:52:04.955 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:04.955 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.955 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.955 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:04.955 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.905405+0000", 2024-09-11T09:52:04.955 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2270000000000001e-06 2024-09-11T09:52:04.956 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.956 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.956 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:04.956 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.914859+0000", 2024-09-11T09:52:04.956 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0094541190000000004 2024-09-11T09:52:04.956 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.956 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.956 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:04.956 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.914879+0000", 2024-09-11T09:52:04.956 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.0052999999999999e-05 2024-09-11T09:52:04.956 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.956 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.956 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:04.957 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.914897+0000", 2024-09-11T09:52:04.957 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7331000000000001e-05 2024-09-11T09:52:04.957 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.957 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.957 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:04.957 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.915019+0000", 2024-09-11T09:52:04.957 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00012168 2024-09-11T09:52:04.957 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.957 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.957 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:04.957 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.915269+0000", 2024-09-11T09:52:04.957 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00025054899999999998 2024-09-11T09:52:04.957 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.958 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.958 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 3", 2024-09-11T09:52:04.958 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.915479+0000", 2024-09-11T09:52:04.958 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00021017799999999999 2024-09-11T09:52:04.958 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.958 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.958 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:04.958 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.926070+0000", 2024-09-11T09:52:04.958 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.010590624999999999 2024-09-11T09:52:04.958 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.958 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.958 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:04.958 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.936198+0000", 2024-09-11T09:52:04.958 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.010128194 2024-09-11T09:52:04.959 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.959 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.959 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:04.959 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.936263+0000", 2024-09-11T09:52:04.959 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.5366999999999999e-05 2024-09-11T09:52:04.959 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.959 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.959 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:04.959 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.936317+0000", 2024-09-11T09:52:04.959 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.3156999999999999e-05 2024-09-11T09:52:04.959 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:04.959 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:04.959 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:04.959 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.959 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.960 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:39 2.6 2:743e7616:::rbd_data.1250f5e25b80.0000000000000005:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:04.960 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.914960+0000", 2024-09-11T09:52:04.960 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 42.025158408999999, 2024-09-11T09:52:04.960 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.022371518, 2024-09-11T09:52:04.960 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:04.960 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:04.960 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:04.960 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:04.960 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:52:04.960 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:52:04.960 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 39 2024-09-11T09:52:04.960 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.960 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:04.961 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.961 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:04.961 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.914960+0000", 2024-09-11T09:52:04.961 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:04.961 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.961 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.961 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:04.961 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.914960+0000", 2024-09-11T09:52:04.961 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:04.961 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.961 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.961 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:04.961 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.914963+0000", 2024-09-11T09:52:04.962 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.1119999999999999e-06 2024-09-11T09:52:04.962 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.962 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.962 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:04.962 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.920417+0000", 2024-09-11T09:52:04.962 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0054543969999999997 2024-09-11T09:52:04.962 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.962 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.962 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:04.962 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.920430+0000", 2024-09-11T09:52:04.962 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3414000000000001e-05 2024-09-11T09:52:04.962 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.962 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.963 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:04.963 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.920437+0000", 2024-09-11T09:52:04.963 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.6159999999999998e-06 2024-09-11T09:52:04.963 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.963 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.963 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:04.963 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.921325+0000", 2024-09-11T09:52:04.963 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000887731 2024-09-11T09:52:04.963 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.963 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.963 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:04.963 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.921422+0000", 2024-09-11T09:52:04.963 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.7201999999999995e-05 2024-09-11T09:52:04.963 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.963 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.964 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 3", 2024-09-11T09:52:04.964 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.921511+0000", 2024-09-11T09:52:04.964 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.8880000000000003e-05 2024-09-11T09:52:04.964 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.964 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.964 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:04.964 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.928092+0000", 2024-09-11T09:52:04.964 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0065816290000000003 2024-09-11T09:52:04.964 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.964 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.964 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:04.964 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.937240+0000", 2024-09-11T09:52:04.964 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0091481070000000008 2024-09-11T09:52:04.965 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.965 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.965 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:04.965 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.937294+0000", 2024-09-11T09:52:04.965 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.3100000000000003e-05 2024-09-11T09:52:04.965 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.965 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.965 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:04.965 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.937332+0000", 2024-09-11T09:52:04.965 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.8330000000000001e-05 2024-09-11T09:52:04.965 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:04.965 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:04.965 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:04.965 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.966 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.966 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4688.0:32 2.2 e53/15 2:45941fd2:::rbd_data.1250f5e25b80.0000000000000001:head v 53'3, mlcod=17'1)", 2024-09-11T09:52:04.966 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.916509+0000", 2024-09-11T09:52:04.966 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 42.023609327000003, 2024-09-11T09:52:04.966 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.015885318999999998, 2024-09-11T09:52:04.966 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:04.966 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:04.966 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:04.966 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:04.966 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.966 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:04.966 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.916509+0000", 2024-09-11T09:52:04.966 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:04.966 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.967 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.967 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:04.967 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.916509+0000", 2024-09-11T09:52:04.967 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:04.967 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.967 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.967 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:04.967 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.916511+0000", 2024-09-11T09:52:04.967 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.052e-06 2024-09-11T09:52:04.967 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.967 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.967 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:04.967 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.927019+0000", 2024-09-11T09:52:04.967 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.010508224 2024-09-11T09:52:04.968 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.968 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.968 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:04.968 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.927038+0000", 2024-09-11T09:52:04.968 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.9590999999999999e-05 2024-09-11T09:52:04.968 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.968 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.968 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:04.968 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.927050+0000", 2024-09-11T09:52:04.968 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1439e-05 2024-09-11T09:52:04.968 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.968 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.968 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:04.968 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.927115+0000", 2024-09-11T09:52:04.969 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.5301000000000004e-05 2024-09-11T09:52:04.969 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.969 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.969 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:04.969 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.927132+0000", 2024-09-11T09:52:04.969 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7297000000000001e-05 2024-09-11T09:52:04.969 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.969 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.969 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:04.969 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.932354+0000", 2024-09-11T09:52:04.969 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0052211870000000004 2024-09-11T09:52:04.969 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.969 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.970 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:04.970 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.932395+0000", 2024-09-11T09:52:04.970 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.1227999999999999e-05 2024-09-11T09:52:04.970 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:04.970 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:04.970 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:04.970 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.970 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.970 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:44 2.6 2:67246a22:::rbd_data.1250f5e25b80.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:04.970 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.943181+0000", 2024-09-11T09:52:04.970 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 41.996938116000003, 2024-09-11T09:52:04.970 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.022629371999999998, 2024-09-11T09:52:04.970 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:04.970 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:04.971 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:04.971 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:04.971 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:52:04.971 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:52:04.971 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 44 2024-09-11T09:52:04.971 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.971 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:04.971 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.971 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:04.971 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.943181+0000", 2024-09-11T09:52:04.971 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:04.971 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.971 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.972 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:04.972 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.943181+0000", 2024-09-11T09:52:04.972 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:04.972 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.972 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.972 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:04.972 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.943182+0000", 2024-09-11T09:52:04.972 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.0500000000000002e-07 2024-09-11T09:52:04.972 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.972 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.972 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:04.972 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951240+0000", 2024-09-11T09:52:04.972 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.008058529 2024-09-11T09:52:04.972 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.973 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.973 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:04.973 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951260+0000", 2024-09-11T09:52:04.973 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.9570000000000001e-05 2024-09-11T09:52:04.973 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.973 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.973 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:04.973 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951270+0000", 2024-09-11T09:52:04.973 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0018e-05 2024-09-11T09:52:04.973 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.973 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.973 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:04.973 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951349+0000", 2024-09-11T09:52:04.973 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.8801000000000007e-05 2024-09-11T09:52:04.974 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.974 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.974 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:04.974 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951483+0000", 2024-09-11T09:52:04.974 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00013427 2024-09-11T09:52:04.974 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.974 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.974 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 3", 2024-09-11T09:52:04.974 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951593+0000", 2024-09-11T09:52:04.974 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00011031000000000001 2024-09-11T09:52:04.975 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.975 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.975 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:04.975 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.957303+0000", 2024-09-11T09:52:04.975 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0057097959999999996 2024-09-11T09:52:04.975 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.975 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.975 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:04.975 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.965690+0000", 2024-09-11T09:52:04.975 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0083867420000000008 2024-09-11T09:52:04.975 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.975 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.975 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:04.975 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.965749+0000", 2024-09-11T09:52:04.976 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.8981999999999999e-05 2024-09-11T09:52:04.976 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.976 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.976 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:04.976 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.965810+0000", 2024-09-11T09:52:04.976 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.1449000000000001e-05 2024-09-11T09:52:04.976 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:04.976 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:04.976 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:04.976 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.977 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.977 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:51 2.6 2:6b5894fc:::rbd_data.1250f5e25b80.000000000000000a:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:04.977 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.951324+0000", 2024-09-11T09:52:04.977 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 41.988795043000003, 2024-09-11T09:52:04.977 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.021375273, 2024-09-11T09:52:04.977 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:04.977 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:04.977 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:04.977 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:04.977 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:52:04.978 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:52:04.978 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 51 2024-09-11T09:52:04.978 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.978 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:04.978 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.978 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:04.979 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951324+0000", 2024-09-11T09:52:04.979 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:04.979 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.979 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.979 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:04.979 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951324+0000", 2024-09-11T09:52:04.979 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:04.979 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.980 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.980 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:04.980 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951324+0000", 2024-09-11T09:52:04.980 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.39e-07 2024-09-11T09:52:04.980 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.980 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.980 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:04.981 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.958537+0000", 2024-09-11T09:52:04.981 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0072126769999999998 2024-09-11T09:52:04.981 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.981 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.981 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:04.981 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.958554+0000", 2024-09-11T09:52:04.981 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7232e-05 2024-09-11T09:52:04.981 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.981 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.981 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:04.981 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.958570+0000", 2024-09-11T09:52:04.981 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.5747999999999999e-05 2024-09-11T09:52:04.982 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.982 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.982 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:04.982 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.958638+0000", 2024-09-11T09:52:04.982 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.7810000000000003e-05 2024-09-11T09:52:04.982 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.982 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.982 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:04.982 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.958758+0000", 2024-09-11T09:52:04.982 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000120689 2024-09-11T09:52:04.982 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.982 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.982 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 3", 2024-09-11T09:52:04.983 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.958865+0000", 2024-09-11T09:52:04.983 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00010692 2024-09-11T09:52:04.983 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.983 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.983 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:04.983 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.970712+0000", 2024-09-11T09:52:04.983 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.011846327 2024-09-11T09:52:04.983 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.983 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.983 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:04.983 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.972603+0000", 2024-09-11T09:52:04.983 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0018918610000000001 2024-09-11T09:52:04.983 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.983 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.984 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:04.984 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.972653+0000", 2024-09-11T09:52:04.984 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.9762999999999998e-05 2024-09-11T09:52:04.984 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.984 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.984 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:04.984 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.972699+0000", 2024-09-11T09:52:04.984 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.5806999999999997e-05 2024-09-11T09:52:04.984 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:04.984 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:04.984 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:04.984 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.985 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.985 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:52 2.4 2:26d5c2f0:::rbd_data.1250f5e25b80.000000000000000b:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:04.985 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.958596+0000", 2024-09-11T09:52:04.985 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 41.981522847999997, 2024-09-11T09:52:04.985 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.016477849999999999, 2024-09-11T09:52:04.985 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:04.985 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:04.985 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:04.985 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:04.985 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:52:04.985 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:52:04.985 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 52 2024-09-11T09:52:04.985 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.986 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:04.986 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.986 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:04.986 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.958596+0000", 2024-09-11T09:52:04.986 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:04.986 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.986 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.986 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:04.986 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.958596+0000", 2024-09-11T09:52:04.986 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:04.986 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.986 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.986 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:04.986 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.958597+0000", 2024-09-11T09:52:04.987 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1960000000000001e-06 2024-09-11T09:52:04.987 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.987 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.987 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:04.987 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.963157+0000", 2024-09-11T09:52:04.987 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0045603029999999999 2024-09-11T09:52:04.987 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.987 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.987 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:04.987 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.963171+0000", 2024-09-11T09:52:04.987 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3777999999999999e-05 2024-09-11T09:52:04.987 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.987 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.988 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:04.988 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.963181+0000", 2024-09-11T09:52:04.988 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.9010000000000006e-06 2024-09-11T09:52:04.988 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.988 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.988 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:04.988 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.963246+0000", 2024-09-11T09:52:04.988 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.5351000000000005e-05 2024-09-11T09:52:04.988 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.988 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.988 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:04.988 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.963419+0000", 2024-09-11T09:52:04.989 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00017299700000000001 2024-09-11T09:52:04.989 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.989 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.989 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 0", 2024-09-11T09:52:04.989 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.963570+0000", 2024-09-11T09:52:04.989 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00015066999999999999 2024-09-11T09:52:04.989 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.989 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.989 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:04.989 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.973881+0000", 2024-09-11T09:52:04.989 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.010311268 2024-09-11T09:52:04.989 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.989 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.989 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:04.990 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.974963+0000", 2024-09-11T09:52:04.990 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.001081493 2024-09-11T09:52:04.990 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.990 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.990 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:04.990 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.975012+0000", 2024-09-11T09:52:04.990 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.9471000000000001e-05 2024-09-11T09:52:04.990 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.990 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.990 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:04.991 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.975074+0000", 2024-09-11T09:52:04.991 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.1421999999999996e-05 2024-09-11T09:52:04.991 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:04.991 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:04.991 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:04.991 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.991 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.991 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4688.0:47 2.1 e53/15 2:95f53b3e:::rbd_data.1250f5e25b80.0000000000000008:head v 53'3, mlcod=53'1)", 2024-09-11T09:52:04.991 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.960861+0000", 2024-09-11T09:52:04.991 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 41.979257429999997, 2024-09-11T09:52:04.991 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.012346107, 2024-09-11T09:52:04.991 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:04.991 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:04.991 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:04.992 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:04.992 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.992 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:04.992 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.960861+0000", 2024-09-11T09:52:04.992 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:04.992 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.992 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.992 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:04.992 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.960861+0000", 2024-09-11T09:52:04.992 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:04.992 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.992 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.992 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:04.993 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.960863+0000", 2024-09-11T09:52:04.993 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7150000000000001e-06 2024-09-11T09:52:04.993 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.993 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.993 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:04.993 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.962428+0000", 2024-09-11T09:52:04.993 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.001564668 2024-09-11T09:52:04.993 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.993 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.993 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:04.993 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.962446+0000", 2024-09-11T09:52:04.993 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.8476999999999999e-05 2024-09-11T09:52:04.993 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.994 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.994 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:04.994 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.962458+0000", 2024-09-11T09:52:04.994 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1263999999999999e-05 2024-09-11T09:52:04.994 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.994 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.994 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:04.994 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.962575+0000", 2024-09-11T09:52:04.994 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000117025 2024-09-11T09:52:04.994 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.994 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.994 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:04.994 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.962601+0000", 2024-09-11T09:52:04.995 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.5984999999999999e-05 2024-09-11T09:52:04.995 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.995 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.995 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:04.995 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.973065+0000", 2024-09-11T09:52:04.995 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.010464753 2024-09-11T09:52:04.995 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.995 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.995 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:04.995 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.973207+0000", 2024-09-11T09:52:04.995 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00014222 2024-09-11T09:52:04.995 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:04.995 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:04.996 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:04.996 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.996 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.996 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:54 2.4 2.da8979c4 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:04.996 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.963207+0000", 2024-09-11T09:52:04.996 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 41.976911991999998, 2024-09-11T09:52:04.996 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.010641575, 2024-09-11T09:52:04.996 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:04.996 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:04.996 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "reached pg", 2024-09-11T09:52:04.996 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:04.996 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:52:04.997 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:52:04.997 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 54 2024-09-11T09:52:04.997 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.997 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:04.997 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.997 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:04.997 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.963207+0000", 2024-09-11T09:52:04.997 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:04.997 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.997 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.997 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:04.997 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.963207+0000", 2024-09-11T09:52:04.998 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:04.998 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.998 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.998 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:04.998 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.963208+0000", 2024-09-11T09:52:04.998 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2979999999999999e-06 2024-09-11T09:52:04.998 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.998 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.998 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:04.998 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.967419+0000", 2024-09-11T09:52:04.998 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0042111750000000002 2024-09-11T09:52:04.998 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.998 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.999 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:04.999 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.967439+0000", 2024-09-11T09:52:04.999 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.0129999999999999e-05 2024-09-11T09:52:04.999 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.999 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.999 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:04.999 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.967453+0000", 2024-09-11T09:52:04.999 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3781000000000001e-05 2024-09-11T09:52:04.999 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:04.999 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:04.999 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:05.000 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.973840+0000", 2024-09-11T09:52:05.000 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0063872269999999997 2024-09-11T09:52:05.000 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.000 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.000 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:05.000 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.973848+0000", 2024-09-11T09:52:05.000 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.9640000000000003e-06 2024-09-11T09:52:05.000 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:05.000 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:05.000 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:05.000 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.000 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.000 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:54 2.4 2:239e915b:::rbd_data.1250f5e25b80.000000000000000c:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:05.000 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.982401+0000", 2024-09-11T09:52:05.001 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 41.957718065000002, 2024-09-11T09:52:05.001 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.016817347, 2024-09-11T09:52:05.001 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:05.001 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:05.001 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:05.001 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:05.001 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:52:05.001 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:52:05.001 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 54 2024-09-11T09:52:05.001 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.001 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:05.001 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.001 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:05.001 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.982401+0000", 2024-09-11T09:52:05.002 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:05.002 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.002 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.002 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:05.002 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.982401+0000", 2024-09-11T09:52:05.002 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:05.002 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.002 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.002 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:05.002 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.982402+0000", 2024-09-11T09:52:05.002 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.8999999999999995e-07 2024-09-11T09:52:05.002 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.002 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.003 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:05.003 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.986535+0000", 2024-09-11T09:52:05.003 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0041333689999999996 2024-09-11T09:52:05.003 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.003 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.003 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:05.003 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.986555+0000", 2024-09-11T09:52:05.003 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.9851e-05 2024-09-11T09:52:05.004 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.004 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.004 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:05.004 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.986569+0000", 2024-09-11T09:52:05.004 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3784000000000001e-05 2024-09-11T09:52:05.004 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.005 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.005 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:05.005 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.986683+0000", 2024-09-11T09:52:05.005 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000114474 2024-09-11T09:52:05.005 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.005 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.005 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:05.005 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.986815+0000", 2024-09-11T09:52:05.005 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00013217799999999999 2024-09-11T09:52:05.005 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.005 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.005 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 0", 2024-09-11T09:52:05.006 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.986918+0000", 2024-09-11T09:52:05.006 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00010235599999999999 2024-09-11T09:52:05.006 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.006 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.006 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:05.006 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.998983+0000", 2024-09-11T09:52:05.006 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.012065352 2024-09-11T09:52:05.006 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.006 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.006 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:05.006 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.999081+0000", 2024-09-11T09:52:05.006 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.8091000000000007e-05 2024-09-11T09:52:05.006 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.006 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.007 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:05.007 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.999159+0000", 2024-09-11T09:52:05.007 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.7823000000000004e-05 2024-09-11T09:52:05.007 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.007 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.007 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:05.007 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.999218+0000", 2024-09-11T09:52:05.007 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.9178999999999997e-05 2024-09-11T09:52:05.007 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:05.007 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:05.007 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:05.007 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.007 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.007 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:56 2.4 2:2167012f:::rbd_data.1250f5e25b80.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:05.008 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.986661+0000", 2024-09-11T09:52:05.008 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 41.953457612000001, 2024-09-11T09:52:05.008 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.017353539000000001, 2024-09-11T09:52:05.008 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:05.008 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:05.008 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:05.008 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:05.008 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:52:05.008 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:52:05.008 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 56 2024-09-11T09:52:05.008 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.008 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:05.008 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.009 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:05.009 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.986661+0000", 2024-09-11T09:52:05.009 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:05.009 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.009 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.009 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:05.009 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.986661+0000", 2024-09-11T09:52:05.010 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:05.010 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.010 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.010 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:05.010 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.986663+0000", 2024-09-11T09:52:05.010 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.8989999999999999e-06 2024-09-11T09:52:05.011 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.011 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.011 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:05.011 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.990444+0000", 2024-09-11T09:52:05.011 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0037808569999999999 2024-09-11T09:52:05.011 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.012 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.012 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:05.012 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.990462+0000", 2024-09-11T09:52:05.012 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.8450000000000001e-05 2024-09-11T09:52:05.012 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.012 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.013 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:05.013 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.990476+0000", 2024-09-11T09:52:05.013 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3565e-05 2024-09-11T09:52:05.013 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.013 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.013 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:05.014 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.993582+0000", 2024-09-11T09:52:05.014 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0031059109999999998 2024-09-11T09:52:05.014 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.014 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.014 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:05.014 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.993686+0000", 2024-09-11T09:52:05.015 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000104424 2024-09-11T09:52:05.015 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.015 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.015 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 0", 2024-09-11T09:52:05.015 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.993799+0000", 2024-09-11T09:52:05.015 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000112985 2024-09-11T09:52:05.015 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.016 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.016 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:05.016 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.002449+0000", 2024-09-11T09:52:05.016 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.008650151 2024-09-11T09:52:05.016 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.017 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.017 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:05.017 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.003948+0000", 2024-09-11T09:52:05.017 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.001498636 2024-09-11T09:52:05.017 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.017 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.018 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:05.018 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.003985+0000", 2024-09-11T09:52:05.018 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.7054999999999997e-05 2024-09-11T09:52:05.018 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.018 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.018 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:05.019 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.004015+0000", 2024-09-11T09:52:05.019 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.9606000000000001e-05 2024-09-11T09:52:05.019 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:05.019 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:05.019 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:05.019 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.020 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.020 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:58 2.6 2:6277c644:::rbd_data.1250f5e25b80.000000000000000e:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:05.020 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.990540+0000", 2024-09-11T09:52:05.020 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 41.949578817000003, 2024-09-11T09:52:05.020 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.016232068999999998, 2024-09-11T09:52:05.021 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:05.021 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:05.021 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:05.021 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:05.021 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:52:05.021 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:52:05.021 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 58 2024-09-11T09:52:05.022 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.022 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:05.022 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.022 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:05.022 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.990540+0000", 2024-09-11T09:52:05.023 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:05.023 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.023 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.023 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:05.023 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.990540+0000", 2024-09-11T09:52:05.023 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:05.023 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.024 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.024 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:05.024 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.990541+0000", 2024-09-11T09:52:05.024 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2359999999999999e-06 2024-09-11T09:52:05.024 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.025 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.025 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:05.025 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.994626+0000", 2024-09-11T09:52:05.025 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.004084865 2024-09-11T09:52:05.025 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.025 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.026 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:05.026 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.994644+0000", 2024-09-11T09:52:05.026 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7866000000000001e-05 2024-09-11T09:52:05.026 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.026 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.026 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:05.027 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.994653+0000", 2024-09-11T09:52:05.027 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.2159999999999995e-06 2024-09-11T09:52:05.027 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.027 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.027 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:05.027 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.994732+0000", 2024-09-11T09:52:05.028 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.8954000000000003e-05 2024-09-11T09:52:05.028 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.028 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.028 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:05.028 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.994906+0000", 2024-09-11T09:52:05.029 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0001741 2024-09-11T09:52:05.029 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.029 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.029 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 3", 2024-09-11T09:52:05.029 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.995083+0000", 2024-09-11T09:52:05.029 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00017637100000000001 2024-09-11T09:52:05.030 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.030 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.030 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:05.030 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.003066+0000", 2024-09-11T09:52:05.030 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0079833360000000006 2024-09-11T09:52:05.031 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.031 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.031 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:05.031 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.006634+0000", 2024-09-11T09:52:05.031 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.003568309 2024-09-11T09:52:05.031 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.032 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.032 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:05.032 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.006716+0000", 2024-09-11T09:52:05.032 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.1377000000000004e-05 2024-09-11T09:52:05.032 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.032 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.033 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:05.033 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.006772+0000", 2024-09-11T09:52:05.033 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.6439e-05 2024-09-11T09:52:05.033 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:05.033 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:05.034 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:05.034 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.034 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.034 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:59 2.4 2:2906e6a7:::rbd_header.1250f5e25b80:head [watch reconnect cookie 140424358411408 gen 2] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:05.034 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.994684+0000", 2024-09-11T09:52:05.035 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 41.945435185999997, 2024-09-11T09:52:05.035 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0094134349999999995, 2024-09-11T09:52:05.035 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:05.035 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:05.035 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "started", 2024-09-11T09:52:05.035 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:05.036 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:52:05.036 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:52:05.036 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 59 2024-09-11T09:52:05.036 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.036 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:05.037 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.037 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:05.037 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.994684+0000", 2024-09-11T09:52:05.037 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:05.037 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.037 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.038 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:05.038 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.994684+0000", 2024-09-11T09:52:05.038 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:05.038 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.038 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.039 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:05.039 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.994686+0000", 2024-09-11T09:52:05.039 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.8159999999999998e-06 2024-09-11T09:52:05.039 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.039 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.040 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:05.040 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.994691+0000", 2024-09-11T09:52:05.040 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.5009999999999998e-06 2024-09-11T09:52:05.040 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.040 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.040 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:05.041 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.994698+0000", 2024-09-11T09:52:05.041 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.3039999999999999e-06 2024-09-11T09:52:05.041 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.041 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.041 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:05.042 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.994703+0000", 2024-09-11T09:52:05.042 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.5190000000000003e-06 2024-09-11T09:52:05.042 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.042 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.042 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:05.042 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.998678+0000", 2024-09-11T09:52:05.043 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0039752340000000002 2024-09-11T09:52:05.043 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.043 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.043 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:05.043 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.998748+0000", 2024-09-11T09:52:05.043 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.0114e-05 2024-09-11T09:52:05.043 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.043 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.043 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:05.043 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.004097+0000", 2024-09-11T09:52:05.043 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0053489469999999997 2024-09-11T09:52:05.043 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:05.043 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:05.044 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:05.044 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.044 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.044 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4694.0:33 2.4 2:3bfc818c:ns::rbd_data.1256a0c02879.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:05.044 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.084537+0000", 2024-09-11T09:52:05.044 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 41.855581758, 2024-09-11T09:52:05.044 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.022075773999999999, 2024-09-11T09:52:05.044 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:05.044 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:05.044 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:05.044 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:05.044 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4694", 2024-09-11T09:52:05.044 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:52:05.045 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 33 2024-09-11T09:52:05.045 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.045 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:05.045 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.045 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:05.045 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.084537+0000", 2024-09-11T09:52:05.045 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:05.045 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.045 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.045 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:05.045 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.084537+0000", 2024-09-11T09:52:05.045 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:05.045 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.046 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.046 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:05.046 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.084538+0000", 2024-09-11T09:52:05.046 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.5000000000000001e-07 2024-09-11T09:52:05.046 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.046 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.046 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:05.046 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.095984+0000", 2024-09-11T09:52:05.046 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.011446543 2024-09-11T09:52:05.046 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.046 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.046 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:05.046 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.096014+0000", 2024-09-11T09:52:05.046 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.9745000000000001e-05 2024-09-11T09:52:05.047 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.047 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.047 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:05.047 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.096030+0000", 2024-09-11T09:52:05.047 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6152999999999999e-05 2024-09-11T09:52:05.047 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.047 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.047 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:05.047 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.096113+0000", 2024-09-11T09:52:05.047 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.2731999999999993e-05 2024-09-11T09:52:05.047 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.047 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.047 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:05.047 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.096273+0000", 2024-09-11T09:52:05.048 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00015973699999999999 2024-09-11T09:52:05.048 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.048 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.048 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 0", 2024-09-11T09:52:05.048 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.096394+0000", 2024-09-11T09:52:05.048 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000120883 2024-09-11T09:52:05.048 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.048 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.048 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:05.048 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.106048+0000", 2024-09-11T09:52:05.048 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0096548020000000005 2024-09-11T09:52:05.048 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.048 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.048 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:05.049 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.106531+0000", 2024-09-11T09:52:05.049 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00048264299999999999 2024-09-11T09:52:05.049 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.049 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.049 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:05.049 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.106565+0000", 2024-09-11T09:52:05.049 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.4343e-05 2024-09-11T09:52:05.049 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.049 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.049 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:05.049 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.106613+0000", 2024-09-11T09:52:05.049 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.7342999999999997e-05 2024-09-11T09:52:05.049 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:05.049 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:05.050 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:05.050 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.050 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.050 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4694.0:32 2.1 e53/15 2:8a862b48:ns::rbd_data.1256a0c02879.0000000000000000:head v 53'8, mlcod=53'6)", 2024-09-11T09:52:05.050 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.091418+0000", 2024-09-11T09:52:05.050 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 41.848700589000003, 2024-09-11T09:52:05.050 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.013027372000000001, 2024-09-11T09:52:05.050 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:05.050 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:05.050 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:05.050 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:05.050 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.050 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:05.050 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.091418+0000", 2024-09-11T09:52:05.051 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:05.051 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.051 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.051 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:05.051 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.091418+0000", 2024-09-11T09:52:05.051 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:05.051 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.051 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.051 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:05.051 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.091420+0000", 2024-09-11T09:52:05.051 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6899999999999999e-06 2024-09-11T09:52:05.051 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.051 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.051 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:05.052 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.093086+0000", 2024-09-11T09:52:05.052 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0016662510000000001 2024-09-11T09:52:05.052 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.052 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.052 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:05.052 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.093107+0000", 2024-09-11T09:52:05.052 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.0999e-05 2024-09-11T09:52:05.052 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.052 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.052 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:05.052 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.093120+0000", 2024-09-11T09:52:05.052 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2544999999999999e-05 2024-09-11T09:52:05.052 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.052 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.053 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:05.053 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.093186+0000", 2024-09-11T09:52:05.053 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.6595999999999997e-05 2024-09-11T09:52:05.053 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.053 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.053 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:05.053 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.093208+0000", 2024-09-11T09:52:05.053 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.1624000000000002e-05 2024-09-11T09:52:05.053 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.053 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.053 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:05.053 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.104392+0000", 2024-09-11T09:52:05.053 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.011184265000000001 2024-09-11T09:52:05.054 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.054 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.054 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:05.054 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.104446+0000", 2024-09-11T09:52:05.054 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.3402000000000001e-05 2024-09-11T09:52:05.054 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:05.054 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:05.054 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:05.054 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.054 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.054 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4694.0:45 2.6 2:671fb830:ns::rbd_data.1256a0c02879.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:05.054 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.109629+0000", 2024-09-11T09:52:05.054 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 41.830489747999998, 2024-09-11T09:52:05.054 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.036403143999999998, 2024-09-11T09:52:05.055 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:05.055 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:05.055 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:05.055 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:05.055 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4694", 2024-09-11T09:52:05.055 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:52:05.055 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 45 2024-09-11T09:52:05.055 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.055 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:05.055 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.055 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:05.055 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.109629+0000", 2024-09-11T09:52:05.055 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:05.056 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.056 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.056 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:05.056 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.109629+0000", 2024-09-11T09:52:05.056 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:05.056 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.056 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.056 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:05.056 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.109630+0000", 2024-09-11T09:52:05.056 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1200000000000001e-06 2024-09-11T09:52:05.056 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.056 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.056 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:05.057 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.121370+0000", 2024-09-11T09:52:05.057 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.011739571000000001 2024-09-11T09:52:05.057 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.057 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.057 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:05.057 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.121391+0000", 2024-09-11T09:52:05.057 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.1175e-05 2024-09-11T09:52:05.057 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.057 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.057 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:05.057 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.121404+0000", 2024-09-11T09:52:05.057 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2807e-05 2024-09-11T09:52:05.057 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.057 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.058 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:05.058 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.121498+0000", 2024-09-11T09:52:05.058 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.3770999999999994e-05 2024-09-11T09:52:05.058 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.058 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.058 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:05.058 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.121612+0000", 2024-09-11T09:52:05.058 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000114666 2024-09-11T09:52:05.058 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.058 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.058 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 3", 2024-09-11T09:52:05.058 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.121711+0000", 2024-09-11T09:52:05.058 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.8647999999999995e-05 2024-09-11T09:52:05.059 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.059 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.059 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:05.059 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.126558+0000", 2024-09-11T09:52:05.059 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0048473769999999999 2024-09-11T09:52:05.059 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.059 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.059 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:05.059 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.145912+0000", 2024-09-11T09:52:05.059 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.019354281000000001 2024-09-11T09:52:05.059 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.059 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.059 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:05.059 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.145977+0000", 2024-09-11T09:52:05.060 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.4135999999999997e-05 2024-09-11T09:52:05.060 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.060 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.060 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:05.060 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.146032+0000", 2024-09-11T09:52:05.060 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.5591999999999999e-05 2024-09-11T09:52:05.060 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:05.060 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:05.060 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:05.060 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.060 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.060 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4694.0:41 2.2 e53/15 2:5e2505c4:ns::rbd_data.1256a0c02879.0000000000000003:head v 53'4, mlcod=18'2)", 2024-09-11T09:52:05.060 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.113199+0000", 2024-09-11T09:52:05.060 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 41.826919338000003, 2024-09-11T09:52:05.061 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.011404512, 2024-09-11T09:52:05.061 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:05.061 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:05.061 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:05.061 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:05.061 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.061 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:05.061 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.113199+0000", 2024-09-11T09:52:05.061 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:05.061 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.061 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.061 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:05.061 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.113199+0000", 2024-09-11T09:52:05.062 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:05.062 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.062 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.062 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:05.062 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.113200+0000", 2024-09-11T09:52:05.062 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.2799999999999995e-07 2024-09-11T09:52:05.062 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.062 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.062 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:05.062 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.118872+0000", 2024-09-11T09:52:05.062 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0056718089999999999 2024-09-11T09:52:05.062 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.062 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.062 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:05.063 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.118889+0000", 2024-09-11T09:52:05.063 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7428999999999999e-05 2024-09-11T09:52:05.063 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.063 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.063 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:05.063 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.118900+0000", 2024-09-11T09:52:05.063 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0243e-05 2024-09-11T09:52:05.063 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.063 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.063 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:05.063 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.118959+0000", 2024-09-11T09:52:05.063 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.9364000000000003e-05 2024-09-11T09:52:05.063 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.063 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.064 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:05.064 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.118976+0000", 2024-09-11T09:52:05.064 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7124000000000001e-05 2024-09-11T09:52:05.064 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.064 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.064 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:05.064 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.124557+0000", 2024-09-11T09:52:05.064 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0055812659999999997 2024-09-11T09:52:05.064 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.064 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.064 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:05.064 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.124604+0000", 2024-09-11T09:52:05.064 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.6548999999999998e-05 2024-09-11T09:52:05.065 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:05.065 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:05.065 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:05.065 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.065 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.065 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4694.0:43 2.0 e53/15 2:16c8ae84:ns::rbd_data.1256a0c02879.0000000000000005:head v 53'2, mlcod=0'0)", 2024-09-11T09:52:05.065 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.158559+0000", 2024-09-11T09:52:05.065 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 41.781560012, 2024-09-11T09:52:05.065 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.016713677999999999, 2024-09-11T09:52:05.065 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:05.065 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:05.065 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:05.065 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:05.066 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.066 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:05.066 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.158559+0000", 2024-09-11T09:52:05.066 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:05.066 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.066 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.066 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:05.066 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.158559+0000", 2024-09-11T09:52:05.066 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:05.066 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.066 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.066 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:05.066 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.158560+0000", 2024-09-11T09:52:05.066 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1459999999999999e-06 2024-09-11T09:52:05.067 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.067 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.067 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:05.067 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.166465+0000", 2024-09-11T09:52:05.067 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.007904767 2024-09-11T09:52:05.067 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.067 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.067 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:05.067 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.166491+0000", 2024-09-11T09:52:05.067 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.6308e-05 2024-09-11T09:52:05.067 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.067 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.067 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:05.068 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.166502+0000", 2024-09-11T09:52:05.068 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0736e-05 2024-09-11T09:52:05.068 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.068 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.068 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:05.068 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.166564+0000", 2024-09-11T09:52:05.068 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.2578999999999998e-05 2024-09-11T09:52:05.068 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.068 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.068 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:05.068 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.166585+0000", 2024-09-11T09:52:05.068 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.0395999999999999e-05 2024-09-11T09:52:05.068 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.068 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.069 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:05.069 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.175227+0000", 2024-09-11T09:52:05.069 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0086426420000000007 2024-09-11T09:52:05.069 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.069 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.069 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:05.069 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.175272+0000", 2024-09-11T09:52:05.069 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.5104e-05 2024-09-11T09:52:05.069 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:05.069 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:05.069 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:05.069 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.069 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.069 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4694.0:59 2.2 e53/15 2:41adefc9:ns::rbd_data.1256a0c02879.000000000000000e:head v 53'5, mlcod=53'3)", 2024-09-11T09:52:05.070 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.163389+0000", 2024-09-11T09:52:05.070 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 41.776730036000004, 2024-09-11T09:52:05.070 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.013095719, 2024-09-11T09:52:05.070 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:05.070 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:05.070 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:05.070 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:05.070 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.070 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:05.070 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.163389+0000", 2024-09-11T09:52:05.070 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:05.070 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.070 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.070 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:05.071 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.163389+0000", 2024-09-11T09:52:05.071 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:05.071 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.071 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.071 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:05.071 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.163390+0000", 2024-09-11T09:52:05.071 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.336e-06 2024-09-11T09:52:05.071 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.071 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.071 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:05.071 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.170335+0000", 2024-09-11T09:52:05.071 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0069451080000000002 2024-09-11T09:52:05.071 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.072 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.072 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:05.072 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.170354+0000", 2024-09-11T09:52:05.072 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.8423e-05 2024-09-11T09:52:05.072 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.072 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.072 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:05.072 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.170366+0000", 2024-09-11T09:52:05.072 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2235999999999999e-05 2024-09-11T09:52:05.072 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.072 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.072 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:05.072 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.170439+0000", 2024-09-11T09:52:05.072 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.2989000000000002e-05 2024-09-11T09:52:05.073 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.073 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.073 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:05.073 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.170460+0000", 2024-09-11T09:52:05.073 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.1211000000000001e-05 2024-09-11T09:52:05.073 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.073 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.073 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:05.073 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.176442+0000", 2024-09-11T09:52:05.073 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0059821969999999999 2024-09-11T09:52:05.073 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.073 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.073 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:05.073 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.176484+0000", 2024-09-11T09:52:05.074 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.2218999999999997e-05 2024-09-11T09:52:05.074 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:05.074 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:05.074 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:05.074 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:05.074 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:05.074 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:52:05.075 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-11T09:52:05.076 INFO:tasks.ceph.osd.0.smithi017.stderr:2024-09-11T09:52:05.076+0000 7fb73c0f1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31665) UID: 0 2024-09-11T09:52:05.176 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:52:05.177+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:52:05.226 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:52:05.226 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [], 2024-09-11T09:52:05.226 INFO:teuthology.orchestra.run.smithi017.stdout: "num_ops": 0 2024-09-11T09:52:05.226 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:52:05.234 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-11T09:52:05.277 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-09-11T09:52:05.278+0000 7f7b6c7d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31904) UID: 0 2024-09-11T09:52:05.378 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-09-11T09:52:05.378+0000 7f7b6c7d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31904) UID: 0 2024-09-11T09:52:05.383 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:52:05.383 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [], 2024-09-11T09:52:05.383 INFO:teuthology.orchestra.run.smithi017.stdout: "complaint_time": 30, 2024-09-11T09:52:05.383 INFO:teuthology.orchestra.run.smithi017.stdout: "num_blocked_ops": 0 2024-09-11T09:52:05.383 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:52:05.392 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-11T09:52:05.478 INFO:tasks.ceph.osd.2.smithi017.stderr:2024-09-11T09:52:05.479+0000 7fd20b170640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31672) UID: 0 2024-09-11T09:52:05.501 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:52:05.501 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 20, 2024-09-11T09:52:05.501 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 600, 2024-09-11T09:52:05.501 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [ 2024-09-11T09:52:05.502 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.502 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:31 2.1 2:8654721b:::rbd_data.1250f5e25b80.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:05.502 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.896024+0000", 2024-09-11T09:52:05.502 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 42.604458653999998, 2024-09-11T09:52:05.502 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.023148401999999998, 2024-09-11T09:52:05.502 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:05.502 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:05.503 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:05.503 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:05.503 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:52:05.503 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:52:05.503 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 31 2024-09-11T09:52:05.503 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.503 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:05.503 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.504 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:05.504 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.896024+0000", 2024-09-11T09:52:05.504 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:05.504 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.504 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.504 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:05.504 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.896024+0000", 2024-09-11T09:52:05.504 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:05.505 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.505 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.505 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:05.505 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.896025+0000", 2024-09-11T09:52:05.505 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.54e-07 2024-09-11T09:52:05.505 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.505 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.505 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:05.505 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.904287+0000", 2024-09-11T09:52:05.505 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0082624729999999993 2024-09-11T09:52:05.506 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.506 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.506 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:05.506 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.904305+0000", 2024-09-11T09:52:05.506 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7912999999999999e-05 2024-09-11T09:52:05.506 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.506 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.506 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:05.506 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.904312+0000", 2024-09-11T09:52:05.506 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.0230000000000004e-06 2024-09-11T09:52:05.506 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.506 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.506 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:05.506 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.904395+0000", 2024-09-11T09:52:05.507 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.2355999999999996e-05 2024-09-11T09:52:05.507 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.507 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.507 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:05.507 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.904499+0000", 2024-09-11T09:52:05.507 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000104672 2024-09-11T09:52:05.507 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.507 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.507 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1", 2024-09-11T09:52:05.507 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.904594+0000", 2024-09-11T09:52:05.507 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.5129000000000003e-05 2024-09-11T09:52:05.507 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.507 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.507 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:05.508 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.918183+0000", 2024-09-11T09:52:05.508 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.013588355999999999 2024-09-11T09:52:05.508 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.508 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.508 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:05.508 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919053+0000", 2024-09-11T09:52:05.508 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00087066800000000001 2024-09-11T09:52:05.508 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.508 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.508 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:05.508 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919125+0000", 2024-09-11T09:52:05.508 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.1333000000000003e-05 2024-09-11T09:52:05.508 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.509 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.509 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:05.509 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919172+0000", 2024-09-11T09:52:05.509 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.7725000000000001e-05 2024-09-11T09:52:05.509 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:05.509 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:05.509 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:05.509 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.509 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.509 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4688.0:42 2.3 e53/15 2:c54742d8:::rbd_data.1250f5e25b80.0000000000000006:head v 53'2, mlcod=0'0)", 2024-09-11T09:52:05.510 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.939272+0000", 2024-09-11T09:52:05.510 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 42.561210670000001, 2024-09-11T09:52:05.510 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.010821958, 2024-09-11T09:52:05.510 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:05.510 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:05.510 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:05.510 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:05.510 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.510 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:05.510 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.939272+0000", 2024-09-11T09:52:05.510 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:05.510 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.511 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.511 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:05.511 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.939272+0000", 2024-09-11T09:52:05.511 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:05.511 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.511 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.511 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:05.511 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.939273+0000", 2024-09-11T09:52:05.511 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.7899999999999997e-07 2024-09-11T09:52:05.511 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.512 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.512 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:05.512 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.944403+0000", 2024-09-11T09:52:05.512 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0051302129999999998 2024-09-11T09:52:05.512 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.512 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.512 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:05.512 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.944413+0000", 2024-09-11T09:52:05.512 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0182e-05 2024-09-11T09:52:05.512 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.512 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.512 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:05.512 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.944420+0000", 2024-09-11T09:52:05.512 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.7000000000000002e-06 2024-09-11T09:52:05.513 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.513 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.513 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:05.513 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.944482+0000", 2024-09-11T09:52:05.513 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.1911999999999999e-05 2024-09-11T09:52:05.513 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.513 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.513 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:05.513 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.944508+0000", 2024-09-11T09:52:05.513 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.6582000000000001e-05 2024-09-11T09:52:05.513 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.513 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.513 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:05.514 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.950050+0000", 2024-09-11T09:52:05.514 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0055414080000000003 2024-09-11T09:52:05.514 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.514 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.514 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:05.514 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.950094+0000", 2024-09-11T09:52:05.514 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.4082000000000002e-05 2024-09-11T09:52:05.514 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:05.514 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:05.514 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:05.514 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.514 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.514 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:47 2.1 2:95f53b3e:::rbd_data.1250f5e25b80.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:05.515 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.948330+0000", 2024-09-11T09:52:05.515 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 42.552152325999998, 2024-09-11T09:52:05.515 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.025156603999999999, 2024-09-11T09:52:05.515 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:05.515 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:05.515 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:05.515 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:05.515 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:52:05.515 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:52:05.515 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 47 2024-09-11T09:52:05.515 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.515 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:05.516 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.516 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:05.516 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.948330+0000", 2024-09-11T09:52:05.516 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:05.516 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.516 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.516 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:05.516 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.948330+0000", 2024-09-11T09:52:05.516 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:05.516 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.516 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.516 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:05.516 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.948333+0000", 2024-09-11T09:52:05.517 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.481e-06 2024-09-11T09:52:05.517 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.517 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.517 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:05.517 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.960263+0000", 2024-09-11T09:52:05.517 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.011930671 2024-09-11T09:52:05.517 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.517 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.517 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:05.517 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.960289+0000", 2024-09-11T09:52:05.517 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.5573000000000001e-05 2024-09-11T09:52:05.517 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.517 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.518 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:05.518 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.960304+0000", 2024-09-11T09:52:05.518 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.4647000000000001e-05 2024-09-11T09:52:05.518 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.518 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.518 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:05.518 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.960390+0000", 2024-09-11T09:52:05.518 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.6088999999999995e-05 2024-09-11T09:52:05.518 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.518 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.518 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:05.518 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.960562+0000", 2024-09-11T09:52:05.518 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00017237400000000001 2024-09-11T09:52:05.519 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.519 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.519 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1", 2024-09-11T09:52:05.519 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.960725+0000", 2024-09-11T09:52:05.519 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00016321599999999999 2024-09-11T09:52:05.519 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.519 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.519 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:05.519 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.969465+0000", 2024-09-11T09:52:05.519 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0087396899999999996 2024-09-11T09:52:05.519 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.519 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.519 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:05.519 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.973393+0000", 2024-09-11T09:52:05.520 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0039274669999999996 2024-09-11T09:52:05.520 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.520 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.520 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:05.520 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.973443+0000", 2024-09-11T09:52:05.520 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.0340000000000003e-05 2024-09-11T09:52:05.520 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.520 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.520 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:05.520 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.973487+0000", 2024-09-11T09:52:05.520 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.4056e-05 2024-09-11T09:52:05.520 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:05.520 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:05.521 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:05.521 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.521 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.521 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:61 2.1 2:9af038b1:::rbd_data.1250f5e25b80.000000000000000f:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:05.521 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.975764+0000", 2024-09-11T09:52:05.521 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 42.524718802000002, 2024-09-11T09:52:05.521 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.016158891000000002, 2024-09-11T09:52:05.521 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:05.521 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:05.521 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:05.521 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:05.521 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:52:05.521 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:52:05.522 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 61 2024-09-11T09:52:05.522 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.522 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:05.522 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.522 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:05.522 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.975764+0000", 2024-09-11T09:52:05.522 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:05.522 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.522 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.522 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:05.522 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.975764+0000", 2024-09-11T09:52:05.522 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:05.522 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.522 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.523 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:05.523 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.975766+0000", 2024-09-11T09:52:05.523 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.767e-06 2024-09-11T09:52:05.523 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.523 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.523 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:05.523 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.983204+0000", 2024-09-11T09:52:05.523 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0074388400000000004 2024-09-11T09:52:05.523 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.523 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.523 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:05.523 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.983226+0000", 2024-09-11T09:52:05.523 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.1279000000000001e-05 2024-09-11T09:52:05.523 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.524 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.524 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:05.524 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.983238+0000", 2024-09-11T09:52:05.524 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2563e-05 2024-09-11T09:52:05.524 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.524 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.524 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:05.524 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.983318+0000", 2024-09-11T09:52:05.524 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.0075999999999997e-05 2024-09-11T09:52:05.524 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.524 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.524 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:05.524 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.983429+0000", 2024-09-11T09:52:05.525 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000110618 2024-09-11T09:52:05.525 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.525 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.525 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1", 2024-09-11T09:52:05.525 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.983525+0000", 2024-09-11T09:52:05.525 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.603e-05 2024-09-11T09:52:05.525 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.525 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.525 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:05.525 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.989004+0000", 2024-09-11T09:52:05.525 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0054785429999999998 2024-09-11T09:52:05.525 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.525 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.526 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:05.526 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.991788+0000", 2024-09-11T09:52:05.526 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0027845729999999998 2024-09-11T09:52:05.526 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.526 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.526 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:05.526 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.991870+0000", 2024-09-11T09:52:05.526 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.1567e-05 2024-09-11T09:52:05.526 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.526 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.526 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:05.526 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.991923+0000", 2024-09-11T09:52:05.526 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.3035000000000003e-05 2024-09-11T09:52:05.526 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:05.527 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:05.527 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:05.527 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.527 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.527 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4694.0:32 2.1 2:8a862b48:ns::rbd_data.1256a0c02879.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:05.527 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.084089+0000", 2024-09-11T09:52:05.527 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 42.416393327999998, 2024-09-11T09:52:05.527 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.020630553999999999, 2024-09-11T09:52:05.527 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:05.527 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:05.527 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:05.527 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:05.527 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4694", 2024-09-11T09:52:05.527 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:52:05.528 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 32 2024-09-11T09:52:05.528 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.528 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:05.528 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.528 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:05.528 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.084089+0000", 2024-09-11T09:52:05.528 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:05.528 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.528 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.528 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:05.528 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.084089+0000", 2024-09-11T09:52:05.528 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:05.528 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.529 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.529 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:05.529 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.084090+0000", 2024-09-11T09:52:05.529 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.2900000000000003e-07 2024-09-11T09:52:05.529 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.529 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.529 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:05.529 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.090866+0000", 2024-09-11T09:52:05.529 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0067756609999999997 2024-09-11T09:52:05.529 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.529 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.529 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:05.530 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.090891+0000", 2024-09-11T09:52:05.530 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.5119000000000001e-05 2024-09-11T09:52:05.530 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.530 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.530 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:05.530 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.090900+0000", 2024-09-11T09:52:05.530 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.4110000000000002e-06 2024-09-11T09:52:05.530 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.530 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.530 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:05.530 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.090968+0000", 2024-09-11T09:52:05.530 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.8041999999999997e-05 2024-09-11T09:52:05.530 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.530 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.531 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:05.531 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.091115+0000", 2024-09-11T09:52:05.531 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00014633499999999999 2024-09-11T09:52:05.531 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.531 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.531 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1", 2024-09-11T09:52:05.531 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.091284+0000", 2024-09-11T09:52:05.531 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000169836 2024-09-11T09:52:05.531 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.531 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.531 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:05.531 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.100896+0000", 2024-09-11T09:52:05.531 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0096118639999999995 2024-09-11T09:52:05.532 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.532 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.532 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:05.532 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.104604+0000", 2024-09-11T09:52:05.532 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0037074619999999999 2024-09-11T09:52:05.532 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.532 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.532 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:05.532 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.104669+0000", 2024-09-11T09:52:05.532 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.4836000000000001e-05 2024-09-11T09:52:05.532 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.532 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.532 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:05.533 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.104720+0000", 2024-09-11T09:52:05.533 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.1258999999999998e-05 2024-09-11T09:52:05.533 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:05.533 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:05.533 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:05.533 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.533 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.533 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4694.0:44 2.3 e53/15 2:cd1a7d1c:ns::rbd_data.1256a0c02879.0000000000000006:head v 53'3, mlcod=16'1)", 2024-09-11T09:52:05.533 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.120387+0000", 2024-09-11T09:52:05.533 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 42.380095675, 2024-09-11T09:52:05.533 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.011988468, 2024-09-11T09:52:05.533 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:05.533 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:05.533 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:05.534 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:05.534 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.534 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:05.534 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.120387+0000", 2024-09-11T09:52:05.534 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:05.534 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.534 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.534 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:05.534 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.120387+0000", 2024-09-11T09:52:05.534 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:05.534 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.534 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.534 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:05.535 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.120389+0000", 2024-09-11T09:52:05.535 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.018e-06 2024-09-11T09:52:05.535 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.535 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.535 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:05.535 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.127403+0000", 2024-09-11T09:52:05.535 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0070144769999999999 2024-09-11T09:52:05.535 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.535 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.535 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:05.535 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.127414+0000", 2024-09-11T09:52:05.536 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0991e-05 2024-09-11T09:52:05.536 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.536 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.536 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:05.536 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.127423+0000", 2024-09-11T09:52:05.536 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.8580000000000006e-06 2024-09-11T09:52:05.536 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.536 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.536 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:05.536 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.127468+0000", 2024-09-11T09:52:05.536 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.4830999999999998e-05 2024-09-11T09:52:05.536 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.536 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.536 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:05.537 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.127485+0000", 2024-09-11T09:52:05.537 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6838999999999999e-05 2024-09-11T09:52:05.537 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.537 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.537 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:05.537 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.132341+0000", 2024-09-11T09:52:05.537 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0048557959999999999 2024-09-11T09:52:05.537 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.537 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.537 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:05.538 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.132375+0000", 2024-09-11T09:52:05.538 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.4657999999999999e-05 2024-09-11T09:52:05.538 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:05.538 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:05.538 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:05.538 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.538 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.538 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4694.0:46 2.3 e53/15 2:ce0e1469:ns::rbd_data.1256a0c02879.0000000000000008:head v 53'4, mlcod=16'1)", 2024-09-11T09:52:05.538 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.129008+0000", 2024-09-11T09:52:05.538 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 42.371475089, 2024-09-11T09:52:05.538 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0087151220000000005, 2024-09-11T09:52:05.538 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:05.538 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:05.539 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:05.539 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:05.539 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.539 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:05.539 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.129008+0000", 2024-09-11T09:52:05.539 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:05.539 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.539 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.539 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:05.539 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.129008+0000", 2024-09-11T09:52:05.539 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:05.539 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.539 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.539 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:05.540 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.129009+0000", 2024-09-11T09:52:05.540 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.367e-06 2024-09-11T09:52:05.540 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.540 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.540 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:05.540 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.132839+0000", 2024-09-11T09:52:05.540 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0038297610000000001 2024-09-11T09:52:05.540 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.540 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.540 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:05.540 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.132849+0000", 2024-09-11T09:52:05.541 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0672e-05 2024-09-11T09:52:05.541 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.541 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.541 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:05.541 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.132856+0000", 2024-09-11T09:52:05.541 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.1380000000000004e-06 2024-09-11T09:52:05.541 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.541 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.541 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:05.541 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.132891+0000", 2024-09-11T09:52:05.541 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.5712e-05 2024-09-11T09:52:05.541 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.541 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.542 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:05.542 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.132907+0000", 2024-09-11T09:52:05.542 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6166000000000001e-05 2024-09-11T09:52:05.542 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.542 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.542 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:05.542 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.137681+0000", 2024-09-11T09:52:05.542 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0047735299999999998 2024-09-11T09:52:05.542 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.542 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.542 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:05.542 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.137723+0000", 2024-09-11T09:52:05.542 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.1776000000000001e-05 2024-09-11T09:52:05.542 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:05.543 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:05.543 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:05.543 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.543 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.543 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4694.0:48 2.3 e53/15 2:d2090b70:ns::rbd_data.1256a0c02879.0000000000000009:head v 53'5, mlcod=53'2)", 2024-09-11T09:52:05.543 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.135108+0000", 2024-09-11T09:52:05.543 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 42.365374191999997, 2024-09-11T09:52:05.543 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.010710679000000001, 2024-09-11T09:52:05.543 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:05.543 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:05.543 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:05.543 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:05.543 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.543 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:05.544 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.135108+0000", 2024-09-11T09:52:05.544 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:05.544 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.544 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.544 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:05.544 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.135108+0000", 2024-09-11T09:52:05.544 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:05.544 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.544 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.544 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:05.544 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.135110+0000", 2024-09-11T09:52:05.544 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.122e-06 2024-09-11T09:52:05.544 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.545 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.545 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:05.545 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.140876+0000", 2024-09-11T09:52:05.545 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.005766162 2024-09-11T09:52:05.545 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.545 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.545 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:05.545 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.140888+0000", 2024-09-11T09:52:05.545 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.201e-05 2024-09-11T09:52:05.545 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.545 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.545 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:05.545 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.140894+0000", 2024-09-11T09:52:05.546 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.1739999999999997e-06 2024-09-11T09:52:05.546 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.546 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.546 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:05.546 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.140971+0000", 2024-09-11T09:52:05.546 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.7495000000000003e-05 2024-09-11T09:52:05.546 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.546 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.546 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:05.546 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.140995+0000", 2024-09-11T09:52:05.546 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.3204999999999999e-05 2024-09-11T09:52:05.546 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.546 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.546 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:05.547 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.145757+0000", 2024-09-11T09:52:05.547 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0047627190000000003 2024-09-11T09:52:05.547 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.547 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.547 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:05.547 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.145819+0000", 2024-09-11T09:52:05.547 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.1791999999999994e-05 2024-09-11T09:52:05.547 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:05.547 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:05.547 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:05.547 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.547 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.547 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4694.0:50 2.1 2:83078786:ns::rbd_data.1256a0c02879.000000000000000a:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:05.548 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.141467+0000", 2024-09-11T09:52:05.548 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 42.359015468000003, 2024-09-11T09:52:05.548 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.012785889, 2024-09-11T09:52:05.548 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:05.548 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:05.548 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:05.548 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:05.548 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4694", 2024-09-11T09:52:05.548 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:52:05.548 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 50 2024-09-11T09:52:05.548 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.548 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:05.548 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.549 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:05.549 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.141467+0000", 2024-09-11T09:52:05.549 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:05.549 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.549 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.549 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:05.549 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.141467+0000", 2024-09-11T09:52:05.549 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:05.549 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.549 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.549 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:05.549 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.141468+0000", 2024-09-11T09:52:05.549 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1009999999999999e-06 2024-09-11T09:52:05.550 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.550 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.550 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:05.550 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.146432+0000", 2024-09-11T09:52:05.550 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0049640539999999999 2024-09-11T09:52:05.550 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.550 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.550 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:05.550 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.146446+0000", 2024-09-11T09:52:05.550 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3254e-05 2024-09-11T09:52:05.550 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.550 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.550 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:05.551 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.146452+0000", 2024-09-11T09:52:05.551 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.207e-06 2024-09-11T09:52:05.551 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.551 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.551 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:05.551 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.146510+0000", 2024-09-11T09:52:05.551 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.8322999999999997e-05 2024-09-11T09:52:05.551 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.551 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.551 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:05.551 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.146650+0000", 2024-09-11T09:52:05.551 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00013947100000000001 2024-09-11T09:52:05.551 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.551 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.552 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1", 2024-09-11T09:52:05.552 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.146779+0000", 2024-09-11T09:52:05.552 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000129059 2024-09-11T09:52:05.552 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.552 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.552 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:05.552 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.153778+0000", 2024-09-11T09:52:05.552 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0069997130000000003 2024-09-11T09:52:05.552 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.552 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.552 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:05.552 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.154180+0000", 2024-09-11T09:52:05.552 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00040159800000000003 2024-09-11T09:52:05.552 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.553 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.553 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:05.553 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.154218+0000", 2024-09-11T09:52:05.553 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.7795e-05 2024-09-11T09:52:05.553 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.553 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.553 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:05.553 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.154253+0000", 2024-09-11T09:52:05.553 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.5314000000000001e-05 2024-09-11T09:52:05.553 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:05.553 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:05.553 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:05.554 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.554 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.554 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4694.0:54 2.3 e53/15 2:c44f0253:ns::rbd_data.1256a0c02879.000000000000000b:head v 53'6, mlcod=53'4)", 2024-09-11T09:52:05.554 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.154926+0000", 2024-09-11T09:52:05.554 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 42.345557092999996, 2024-09-11T09:52:05.554 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0089497099999999996, 2024-09-11T09:52:05.554 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:05.554 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:05.554 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:05.554 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:05.554 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.554 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:05.554 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.154926+0000", 2024-09-11T09:52:05.555 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:05.555 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.555 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.555 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:05.555 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.154926+0000", 2024-09-11T09:52:05.555 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:05.555 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.555 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.555 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:05.555 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.154927+0000", 2024-09-11T09:52:05.555 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.7000000000000003e-07 2024-09-11T09:52:05.555 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.555 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.555 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:05.556 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.158712+0000", 2024-09-11T09:52:05.556 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0037855469999999998 2024-09-11T09:52:05.556 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.556 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.556 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:05.556 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.158726+0000", 2024-09-11T09:52:05.556 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3621000000000001e-05 2024-09-11T09:52:05.556 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.556 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.556 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:05.556 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.158732+0000", 2024-09-11T09:52:05.556 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.9089999999999997e-06 2024-09-11T09:52:05.556 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.557 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.557 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:05.557 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.158781+0000", 2024-09-11T09:52:05.557 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.8900000000000003e-05 2024-09-11T09:52:05.557 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.557 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.557 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:05.557 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.158805+0000", 2024-09-11T09:52:05.557 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.44e-05 2024-09-11T09:52:05.557 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.557 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.557 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:05.557 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.163826+0000", 2024-09-11T09:52:05.557 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0050206399999999998 2024-09-11T09:52:05.558 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.558 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.558 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:05.558 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.163875+0000", 2024-09-11T09:52:05.558 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.9722999999999998e-05 2024-09-11T09:52:05.558 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:05.558 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:05.558 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:05.558 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.558 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.559 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4694.0:56 2.3 e53/15 2:c462f029:ns::rbd_data.1256a0c02879.000000000000000c:head v 53'7, mlcod=53'4)", 2024-09-11T09:52:05.559 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.159549+0000", 2024-09-11T09:52:05.559 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 42.340933387, 2024-09-11T09:52:05.559 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.015591666000000001, 2024-09-11T09:52:05.559 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:05.559 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:05.559 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:05.559 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:05.559 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.559 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:05.559 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.159549+0000", 2024-09-11T09:52:05.559 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:05.559 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.559 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.560 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:05.560 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.159549+0000", 2024-09-11T09:52:05.560 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:05.560 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.560 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.560 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:05.560 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.159550+0000", 2024-09-11T09:52:05.560 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.1500000000000004e-07 2024-09-11T09:52:05.560 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.560 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.560 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:05.560 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.167834+0000", 2024-09-11T09:52:05.560 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0082840220000000003 2024-09-11T09:52:05.560 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.561 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.561 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:05.561 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.167857+0000", 2024-09-11T09:52:05.561 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.3337e-05 2024-09-11T09:52:05.561 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.561 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.561 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:05.561 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.167869+0000", 2024-09-11T09:52:05.561 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1556000000000001e-05 2024-09-11T09:52:05.561 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.561 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.561 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:05.561 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.167939+0000", 2024-09-11T09:52:05.562 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.0674999999999997e-05 2024-09-11T09:52:05.562 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.562 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.562 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:05.562 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.167967+0000", 2024-09-11T09:52:05.562 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.7322999999999999e-05 2024-09-11T09:52:05.562 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.563 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.563 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:05.564 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.175079+0000", 2024-09-11T09:52:05.564 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0071126150000000001 2024-09-11T09:52:05.564 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.564 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.564 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:05.564 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.175141+0000", 2024-09-11T09:52:05.564 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.1523e-05 2024-09-11T09:52:05.564 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:05.564 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:05.564 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:05.564 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.564 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.564 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.5314.0:31 2.1 2:8654721b:::rbd_data.1250f5e25b80.0000000000000000:head [write 393216~131072] snapc 3=[3] ondisk+write+known_if_redirected+supports_pool_eio e68)", 2024-09-11T09:52:05.564 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:50.230850+0000", 2024-09-11T09:52:05.565 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 15.269632626, 2024-09-11T09:52:05.565 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0073772539999999998, 2024-09-11T09:52:05.565 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:05.565 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:05.565 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:05.565 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:05.565 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.5314", 2024-09-11T09:52:05.565 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3853995732", 2024-09-11T09:52:05.565 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 31 2024-09-11T09:52:05.565 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.565 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:05.566 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.566 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:05.566 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.230850+0000", 2024-09-11T09:52:05.566 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:05.566 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.566 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.566 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:05.566 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.230850+0000", 2024-09-11T09:52:05.566 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:05.566 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.566 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.566 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:05.566 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.230850+0000", 2024-09-11T09:52:05.567 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.32e-07 2024-09-11T09:52:05.567 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.567 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.567 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:05.567 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.230908+0000", 2024-09-11T09:52:05.567 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.7698000000000002e-05 2024-09-11T09:52:05.567 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.567 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.567 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:05.567 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.230915+0000", 2024-09-11T09:52:05.567 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.81e-06 2024-09-11T09:52:05.567 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.567 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.567 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:05.568 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.230920+0000", 2024-09-11T09:52:05.568 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.9660000000000004e-06 2024-09-11T09:52:05.568 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.568 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.568 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:05.568 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.235055+0000", 2024-09-11T09:52:05.568 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0041354629999999998 2024-09-11T09:52:05.568 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.568 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.568 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:05.568 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.235145+0000", 2024-09-11T09:52:05.568 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.9362000000000003e-05 2024-09-11T09:52:05.568 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.568 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.569 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1", 2024-09-11T09:52:05.569 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.235282+0000", 2024-09-11T09:52:05.569 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00013710999999999999 2024-09-11T09:52:05.569 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.569 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.569 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:05.569 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.237335+0000", 2024-09-11T09:52:05.569 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.002053165 2024-09-11T09:52:05.569 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.569 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.569 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:05.569 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.238106+0000", 2024-09-11T09:52:05.569 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000771015 2024-09-11T09:52:05.570 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.570 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.570 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:05.570 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.238165+0000", 2024-09-11T09:52:05.570 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.8588000000000002e-05 2024-09-11T09:52:05.570 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.570 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.570 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:05.571 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.238227+0000", 2024-09-11T09:52:05.571 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.2644999999999994e-05 2024-09-11T09:52:05.571 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:05.571 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:05.571 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:05.571 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.571 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.571 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.5314.0:32 2.1 2:8654721b:::rbd_data.1250f5e25b80.0000000000000000:head [write 524288~131072] snapc 3=[3] ondisk+write+known_if_redirected+supports_pool_eio e68)", 2024-09-11T09:52:05.571 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:50.230932+0000", 2024-09-11T09:52:05.571 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 15.269551001, 2024-09-11T09:52:05.572 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0074882129999999996, 2024-09-11T09:52:05.572 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:05.572 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:05.572 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:05.572 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:05.572 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.5314", 2024-09-11T09:52:05.572 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3853995732", 2024-09-11T09:52:05.572 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 32 2024-09-11T09:52:05.572 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.573 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:05.573 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.573 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:05.573 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.230932+0000", 2024-09-11T09:52:05.573 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:05.573 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.573 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.573 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:05.573 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.230932+0000", 2024-09-11T09:52:05.573 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:05.573 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.574 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.574 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:05.574 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.230932+0000", 2024-09-11T09:52:05.574 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.9999999999999999e-07 2024-09-11T09:52:05.574 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.574 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.574 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:05.574 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.231022+0000", 2024-09-11T09:52:05.574 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.0439999999999995e-05 2024-09-11T09:52:05.574 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.574 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.574 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:05.574 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.231030+0000", 2024-09-11T09:52:05.575 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.3350000000000001e-06 2024-09-11T09:52:05.575 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.575 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.575 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:05.575 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.231034+0000", 2024-09-11T09:52:05.575 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.9759999999999997e-06 2024-09-11T09:52:05.575 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.575 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.575 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:05.575 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.235837+0000", 2024-09-11T09:52:05.575 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0048031910000000001 2024-09-11T09:52:05.575 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.575 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.575 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:05.575 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.235904+0000", 2024-09-11T09:52:05.576 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.7181999999999995e-05 2024-09-11T09:52:05.576 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.576 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.576 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1", 2024-09-11T09:52:05.576 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.236040+0000", 2024-09-11T09:52:05.576 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00013545199999999999 2024-09-11T09:52:05.576 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.576 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.576 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:05.576 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.238242+0000", 2024-09-11T09:52:05.576 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.002202809 2024-09-11T09:52:05.576 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.576 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.577 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:05.577 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.238301+0000", 2024-09-11T09:52:05.577 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.8573999999999999e-05 2024-09-11T09:52:05.577 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.577 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.577 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:05.577 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.238378+0000", 2024-09-11T09:52:05.577 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.7323999999999994e-05 2024-09-11T09:52:05.577 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.577 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.577 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:05.577 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.238420+0000", 2024-09-11T09:52:05.577 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.163e-05 2024-09-11T09:52:05.578 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:05.578 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:05.578 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:05.578 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.578 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.578 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.5314.0:33 2.1 2:8654721b:::rbd_data.1250f5e25b80.0000000000000000:head [write 655360~131072] snapc 3=[3] ondisk+write+known_if_redirected+supports_pool_eio e68)", 2024-09-11T09:52:05.578 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:50.231041+0000", 2024-09-11T09:52:05.578 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 15.269441841999999, 2024-09-11T09:52:05.578 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0075627139999999999, 2024-09-11T09:52:05.578 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:05.578 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:05.578 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:05.578 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:05.579 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.5314", 2024-09-11T09:52:05.579 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3853995732", 2024-09-11T09:52:05.579 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 33 2024-09-11T09:52:05.579 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.579 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:05.579 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.579 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:05.579 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.231041+0000", 2024-09-11T09:52:05.579 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:05.579 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.579 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.579 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:05.579 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.231041+0000", 2024-09-11T09:52:05.579 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:05.580 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.580 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.580 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:05.580 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.231041+0000", 2024-09-11T09:52:05.580 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.9499999999999998e-07 2024-09-11T09:52:05.580 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.580 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.580 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:05.580 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.231111+0000", 2024-09-11T09:52:05.581 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.0166999999999994e-05 2024-09-11T09:52:05.581 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.581 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.581 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:05.581 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.231122+0000", 2024-09-11T09:52:05.581 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0518e-05 2024-09-11T09:52:05.581 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.581 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.581 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:05.581 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.231128+0000", 2024-09-11T09:52:05.581 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.3509999999999998e-06 2024-09-11T09:52:05.581 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.582 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.582 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:05.582 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.236622+0000", 2024-09-11T09:52:05.582 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0054934989999999998 2024-09-11T09:52:05.582 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.582 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.582 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:05.582 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.236719+0000", 2024-09-11T09:52:05.582 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.7497999999999994e-05 2024-09-11T09:52:05.582 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.582 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.582 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1", 2024-09-11T09:52:05.582 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.236854+0000", 2024-09-11T09:52:05.583 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000134535 2024-09-11T09:52:05.583 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.583 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.583 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:05.583 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.238491+0000", 2024-09-11T09:52:05.583 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0016368610000000001 2024-09-11T09:52:05.583 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.583 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.583 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:05.583 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.238504+0000", 2024-09-11T09:52:05.583 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3927999999999999e-05 2024-09-11T09:52:05.583 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.583 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.584 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:05.584 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.238561+0000", 2024-09-11T09:52:05.584 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.6029999999999997e-05 2024-09-11T09:52:05.584 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.584 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.584 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:05.584 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.238604+0000", 2024-09-11T09:52:05.584 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.3031999999999997e-05 2024-09-11T09:52:05.584 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:05.584 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:05.584 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:05.585 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.585 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.585 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.5314.0:34 2.1 2:8654721b:::rbd_data.1250f5e25b80.0000000000000000:head [write 786432~131072] snapc 3=[3] ondisk+write+known_if_redirected+supports_pool_eio e68)", 2024-09-11T09:52:05.585 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:50.231139+0000", 2024-09-11T09:52:05.585 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 15.269343238999999, 2024-09-11T09:52:05.585 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0083689690000000004, 2024-09-11T09:52:05.585 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:05.585 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:05.585 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:05.585 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:05.585 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.5314", 2024-09-11T09:52:05.585 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3853995732", 2024-09-11T09:52:05.585 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 34 2024-09-11T09:52:05.586 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.586 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:05.586 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.586 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:05.586 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.231139+0000", 2024-09-11T09:52:05.586 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:05.586 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.586 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.586 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:05.586 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.231139+0000", 2024-09-11T09:52:05.586 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:05.586 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.586 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.586 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:05.587 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.231140+0000", 2024-09-11T09:52:05.587 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.1699999999999998e-07 2024-09-11T09:52:05.587 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.587 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.587 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:05.587 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.231247+0000", 2024-09-11T09:52:05.587 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000107088 2024-09-11T09:52:05.587 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.587 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.587 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:05.587 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.231260+0000", 2024-09-11T09:52:05.587 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.289e-05 2024-09-11T09:52:05.587 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.587 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.588 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:05.588 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.231272+0000", 2024-09-11T09:52:05.588 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2371999999999999e-05 2024-09-11T09:52:05.588 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.588 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.588 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:05.588 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.237380+0000", 2024-09-11T09:52:05.588 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.006107683 2024-09-11T09:52:05.588 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.588 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.588 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:05.588 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.237434+0000", 2024-09-11T09:52:05.588 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.4143e-05 2024-09-11T09:52:05.588 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.589 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.589 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1", 2024-09-11T09:52:05.589 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.237541+0000", 2024-09-11T09:52:05.589 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00010676800000000001 2024-09-11T09:52:05.589 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.589 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.589 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:05.589 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.239390+0000", 2024-09-11T09:52:05.589 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.001848861 2024-09-11T09:52:05.589 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.589 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.590 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:05.590 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.239409+0000", 2024-09-11T09:52:05.590 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.9505e-05 2024-09-11T09:52:05.590 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.590 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.590 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:05.590 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.239463+0000", 2024-09-11T09:52:05.590 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.3838000000000002e-05 2024-09-11T09:52:05.590 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.590 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.590 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:05.591 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.239508+0000", 2024-09-11T09:52:05.591 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.5303999999999999e-05 2024-09-11T09:52:05.591 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:05.591 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:05.591 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:05.591 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.591 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.591 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.5314.0:35 2.1 2:8654721b:::rbd_data.1250f5e25b80.0000000000000000:head [write 917504~131072] snapc 3=[3] ondisk+write+known_if_redirected+supports_pool_eio e68)", 2024-09-11T09:52:05.591 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:50.231287+0000", 2024-09-11T09:52:05.591 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 15.269195905, 2024-09-11T09:52:05.592 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0086930730000000008, 2024-09-11T09:52:05.592 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:05.592 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:05.592 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:05.592 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:05.592 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.5314", 2024-09-11T09:52:05.592 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3853995732", 2024-09-11T09:52:05.592 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 35 2024-09-11T09:52:05.592 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.592 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:05.592 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.592 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:05.592 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.231287+0000", 2024-09-11T09:52:05.593 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:05.593 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.593 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.593 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:05.593 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.231287+0000", 2024-09-11T09:52:05.593 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:05.593 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.593 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.593 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:05.593 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.231287+0000", 2024-09-11T09:52:05.593 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.7700000000000004e-07 2024-09-11T09:52:05.594 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.594 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.594 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:05.594 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.231354+0000", 2024-09-11T09:52:05.594 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.6304999999999996e-05 2024-09-11T09:52:05.594 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.594 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.594 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:05.594 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.231367+0000", 2024-09-11T09:52:05.594 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2952999999999999e-05 2024-09-11T09:52:05.594 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.595 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.595 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:05.595 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.231374+0000", 2024-09-11T09:52:05.595 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.0469999999999996e-06 2024-09-11T09:52:05.595 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.595 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.595 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:05.595 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.238650+0000", 2024-09-11T09:52:05.595 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0072760280000000004 2024-09-11T09:52:05.595 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.595 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.595 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:05.595 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.238732+0000", 2024-09-11T09:52:05.595 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.2194000000000006e-05 2024-09-11T09:52:05.596 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.596 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.596 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1", 2024-09-11T09:52:05.596 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.238876+0000", 2024-09-11T09:52:05.596 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00014458 2024-09-11T09:52:05.596 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.596 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.596 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:05.596 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.239739+0000", 2024-09-11T09:52:05.596 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00086244499999999999 2024-09-11T09:52:05.596 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.596 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.596 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:05.597 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.239917+0000", 2024-09-11T09:52:05.597 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00017777299999999999 2024-09-11T09:52:05.597 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.597 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.597 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:05.597 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.239954+0000", 2024-09-11T09:52:05.597 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.7379999999999998e-05 2024-09-11T09:52:05.597 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.598 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.598 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:05.598 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.239980+0000", 2024-09-11T09:52:05.598 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.5791000000000001e-05 2024-09-11T09:52:05.598 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:05.598 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:05.598 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:05.598 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.598 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.598 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.5314.0:70 2.1 2:8654721b:::rbd_data.1250f5e25b80.0000000000000000:head [write 524288~131072] snapc 5=[5,3] ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-09-11T09:52:05.598 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:50.824556+0000", 2024-09-11T09:52:05.598 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 14.675927103999999, 2024-09-11T09:52:05.599 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0072794469999999997, 2024-09-11T09:52:05.599 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:05.599 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:05.599 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:05.599 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:05.599 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.5314", 2024-09-11T09:52:05.599 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3853995732", 2024-09-11T09:52:05.599 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 70 2024-09-11T09:52:05.599 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.599 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:05.599 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.599 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:05.599 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.824556+0000", 2024-09-11T09:52:05.600 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:05.600 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.600 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.600 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:05.600 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.824556+0000", 2024-09-11T09:52:05.600 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:05.600 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.600 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.600 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:05.600 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.824564+0000", 2024-09-11T09:52:05.600 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.2409999999999995e-06 2024-09-11T09:52:05.600 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.601 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.601 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:05.601 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.824613+0000", 2024-09-11T09:52:05.601 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.9579000000000001e-05 2024-09-11T09:52:05.601 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.601 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.601 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:05.601 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.824638+0000", 2024-09-11T09:52:05.601 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.4510000000000001e-05 2024-09-11T09:52:05.601 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.601 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.602 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:05.602 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.824653+0000", 2024-09-11T09:52:05.602 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.4831000000000001e-05 2024-09-11T09:52:05.602 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.602 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.602 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:05.602 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.829072+0000", 2024-09-11T09:52:05.602 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0044195700000000003 2024-09-11T09:52:05.602 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.602 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.602 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:05.602 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.829166+0000", 2024-09-11T09:52:05.603 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.3212000000000001e-05 2024-09-11T09:52:05.603 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.603 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.603 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1", 2024-09-11T09:52:05.603 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.829309+0000", 2024-09-11T09:52:05.603 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00014301900000000001 2024-09-11T09:52:05.603 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.603 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.603 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:05.603 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.831074+0000", 2024-09-11T09:52:05.603 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.001765305 2024-09-11T09:52:05.603 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.604 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.604 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:05.604 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.831751+0000", 2024-09-11T09:52:05.604 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00067694000000000001 2024-09-11T09:52:05.604 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.604 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.604 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:05.604 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.831803+0000", 2024-09-11T09:52:05.604 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.2441000000000002e-05 2024-09-11T09:52:05.604 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.604 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.604 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:05.605 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.831835+0000", 2024-09-11T09:52:05.605 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.1798999999999998e-05 2024-09-11T09:52:05.605 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:05.605 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:05.605 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:05.605 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.605 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.605 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.5314.0:71 2.1 2:8654721b:::rbd_data.1250f5e25b80.0000000000000000:head [write 655360~131072] snapc 5=[5,3] ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-09-11T09:52:05.605 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:50.824662+0000", 2024-09-11T09:52:05.605 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 14.675820988, 2024-09-11T09:52:05.605 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0082172219999999997, 2024-09-11T09:52:05.605 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:05.606 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:05.606 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:05.606 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:05.606 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.5314", 2024-09-11T09:52:05.606 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3853995732", 2024-09-11T09:52:05.606 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 71 2024-09-11T09:52:05.606 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.606 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:05.606 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.606 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:05.606 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.824662+0000", 2024-09-11T09:52:05.606 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:05.606 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.607 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.607 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:05.607 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.824662+0000", 2024-09-11T09:52:05.607 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:05.607 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.607 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.607 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:05.607 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.824662+0000", 2024-09-11T09:52:05.607 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.0699999999999998e-07 2024-09-11T09:52:05.607 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.607 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.607 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:05.608 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.824859+0000", 2024-09-11T09:52:05.608 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00019725799999999999 2024-09-11T09:52:05.608 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.608 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.608 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:05.608 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.824906+0000", 2024-09-11T09:52:05.608 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.7162e-05 2024-09-11T09:52:05.608 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.608 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.608 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:05.608 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.824926+0000", 2024-09-11T09:52:05.608 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.0038999999999999e-05 2024-09-11T09:52:05.608 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.609 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.609 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:05.609 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.830313+0000", 2024-09-11T09:52:05.609 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0053868409999999999 2024-09-11T09:52:05.609 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.609 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.609 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:05.609 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.830359+0000", 2024-09-11T09:52:05.609 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.5370000000000001e-05 2024-09-11T09:52:05.609 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.609 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.610 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1", 2024-09-11T09:52:05.610 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.830460+0000", 2024-09-11T09:52:05.610 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00010171299999999999 2024-09-11T09:52:05.610 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.610 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.610 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:05.610 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.831843+0000", 2024-09-11T09:52:05.610 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.001382501 2024-09-11T09:52:05.610 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.610 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.611 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:05.611 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.832769+0000", 2024-09-11T09:52:05.611 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00092638099999999999 2024-09-11T09:52:05.611 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.611 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.611 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:05.611 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.832835+0000", 2024-09-11T09:52:05.611 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.6106999999999996e-05 2024-09-11T09:52:05.611 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.611 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.611 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:05.611 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.832879+0000", 2024-09-11T09:52:05.612 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.3543e-05 2024-09-11T09:52:05.612 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:05.612 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:05.612 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:05.612 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.612 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.612 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.5314.0:72 2.1 2:8654721b:::rbd_data.1250f5e25b80.0000000000000000:head [write 786432~131072] snapc 5=[5,3] ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-09-11T09:52:05.612 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:50.824938+0000", 2024-09-11T09:52:05.612 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 14.675544686, 2024-09-11T09:52:05.612 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0082389799999999999, 2024-09-11T09:52:05.612 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:05.613 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:05.613 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:05.613 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:05.613 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.5314", 2024-09-11T09:52:05.613 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3853995732", 2024-09-11T09:52:05.613 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 72 2024-09-11T09:52:05.613 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.613 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:05.613 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.613 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:05.613 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.824938+0000", 2024-09-11T09:52:05.614 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:05.614 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.614 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.614 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:05.614 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.824938+0000", 2024-09-11T09:52:05.614 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:05.614 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.614 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.614 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:05.614 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.824939+0000", 2024-09-11T09:52:05.614 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.6199999999999997e-07 2024-09-11T09:52:05.614 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.615 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.615 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:05.615 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.825028+0000", 2024-09-11T09:52:05.615 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.9105999999999997e-05 2024-09-11T09:52:05.615 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.615 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.615 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:05.615 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.825042+0000", 2024-09-11T09:52:05.615 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3760999999999999e-05 2024-09-11T09:52:05.615 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.615 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.615 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:05.616 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.825061+0000", 2024-09-11T09:52:05.616 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.9088000000000001e-05 2024-09-11T09:52:05.616 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.616 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.616 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:05.616 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.831122+0000", 2024-09-11T09:52:05.616 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.006061299 2024-09-11T09:52:05.616 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.616 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.616 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:05.616 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.831195+0000", 2024-09-11T09:52:05.616 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.2957e-05 2024-09-11T09:52:05.616 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.617 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.617 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1", 2024-09-11T09:52:05.617 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.831301+0000", 2024-09-11T09:52:05.617 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000106049 2024-09-11T09:52:05.617 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.617 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.617 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:05.617 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.832893+0000", 2024-09-11T09:52:05.617 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0015923770000000001 2024-09-11T09:52:05.617 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.617 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.617 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:05.618 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.833076+0000", 2024-09-11T09:52:05.618 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000182855 2024-09-11T09:52:05.618 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.618 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.618 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:05.618 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.833137+0000", 2024-09-11T09:52:05.618 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.1193999999999997e-05 2024-09-11T09:52:05.618 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.618 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.618 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:05.618 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.833177+0000", 2024-09-11T09:52:05.618 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.9632000000000003e-05 2024-09-11T09:52:05.619 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:05.619 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:05.619 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:05.619 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.619 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.619 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.5314.0:73 2.1 2:8654721b:::rbd_data.1250f5e25b80.0000000000000000:head [write 917504~131072] snapc 5=[5,3] ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-09-11T09:52:05.619 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:50.825071+0000", 2024-09-11T09:52:05.619 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 14.675411178999999, 2024-09-11T09:52:05.619 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0084039160000000009, 2024-09-11T09:52:05.619 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:05.619 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:05.619 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:05.620 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:05.620 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.5314", 2024-09-11T09:52:05.620 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3853995732", 2024-09-11T09:52:05.620 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 73 2024-09-11T09:52:05.620 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.620 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:05.620 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.620 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:05.620 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.825071+0000", 2024-09-11T09:52:05.620 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:05.620 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.621 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.621 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:05.621 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.825071+0000", 2024-09-11T09:52:05.621 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:05.621 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.621 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.621 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:05.621 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.825072+0000", 2024-09-11T09:52:05.621 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.5499999999999998e-07 2024-09-11T09:52:05.621 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.621 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.622 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:05.622 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.825124+0000", 2024-09-11T09:52:05.622 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.2197000000000002e-05 2024-09-11T09:52:05.622 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.622 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.622 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:05.622 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.825132+0000", 2024-09-11T09:52:05.622 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.0290000000000005e-06 2024-09-11T09:52:05.622 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.622 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.622 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:05.623 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.825136+0000", 2024-09-11T09:52:05.623 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.9380000000000002e-06 2024-09-11T09:52:05.623 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.623 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.623 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:05.623 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.832001+0000", 2024-09-11T09:52:05.623 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0068650949999999999 2024-09-11T09:52:05.623 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.623 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.623 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:05.623 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.832091+0000", 2024-09-11T09:52:05.624 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.9524999999999994e-05 2024-09-11T09:52:05.624 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.624 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.624 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1", 2024-09-11T09:52:05.624 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.832234+0000", 2024-09-11T09:52:05.624 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000142783 2024-09-11T09:52:05.624 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.624 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.624 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:05.624 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.833198+0000", 2024-09-11T09:52:05.624 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00096476899999999998 2024-09-11T09:52:05.624 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.625 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.625 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:05.625 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.833386+0000", 2024-09-11T09:52:05.625 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00018772 2024-09-11T09:52:05.625 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.625 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.625 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:05.625 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.833440+0000", 2024-09-11T09:52:05.625 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.4228999999999999e-05 2024-09-11T09:52:05.625 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:05.625 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:05.626 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:05.626 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.833475+0000", 2024-09-11T09:52:05.626 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.5076000000000001e-05 2024-09-11T09:52:05.626 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:05.626 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:05.626 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:05.626 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:05.626 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:05.626 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:52:05.628 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-11T09:52:05.629 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-09-11T09:52:05.629+0000 7f7b6c7d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31904) UID: 0 2024-09-11T09:52:05.728 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-09-11T09:52:05.729+0000 7f7b6c7d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31904) UID: 0 2024-09-11T09:52:05.743 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-11T09:52:05.743 INFO:teuthology.orchestra.run.smithi096.stdout: "ops": [], 2024-09-11T09:52:05.743 INFO:teuthology.orchestra.run.smithi096.stdout: "num_ops": 0 2024-09-11T09:52:05.743 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-11T09:52:05.752 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-11T09:52:05.816 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T09:52:05.827 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 5, 0] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 5, 4] 2024-09-11T09:52:05.828 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-11T09:52:05.828 INFO:tasks.thrashosds.thrasher:Removing osd 0, in_osds are: [1, 2, 3, 4, 5, 0] 2024-09-11T09:52:05.828 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd out 0 2024-09-11T09:52:05.834 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-11T09:52:05.837 INFO:tasks.workunit.client.0.smithi107.stderr:+ rbd device --device-type nbd list 2024-09-11T09:52:05.837 INFO:tasks.workunit.client.0.smithi107.stderr:+ expect_false grep '^29367\b' 2024-09-11T09:52:05.837 INFO:tasks.workunit.client.0.smithi107.stderr:+ grep '^29367\b' 2024-09-11T09:52:05.861 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:52:05.862+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:52:05.897 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-11T09:52:05.897 INFO:teuthology.orchestra.run.smithi096.stdout: "ops": [], 2024-09-11T09:52:05.897 INFO:teuthology.orchestra.run.smithi096.stdout: "complaint_time": 30, 2024-09-11T09:52:05.897 INFO:teuthology.orchestra.run.smithi096.stdout: "num_blocked_ops": 0 2024-09-11T09:52:05.897 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-11T09:52:05.902 INFO:tasks.workunit.client.0.smithi107.stderr:+ return 0 2024-09-11T09:52:05.902 INFO:tasks.workunit.client.0.smithi107.stderr:+ ps -C rbd-nbd 2024-09-11T09:52:05.902 INFO:tasks.workunit.client.0.smithi107.stderr:+ expect_false grep '^ *29367\b' 2024-09-11T09:52:05.902 INFO:tasks.workunit.client.0.smithi107.stderr:+ grep '^ *29367\b' 2024-09-11T09:52:05.906 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-11T09:52:05.914 INFO:tasks.workunit.client.0.smithi107.stderr:+ return 0 2024-09-11T09:52:05.914 INFO:tasks.workunit.client.0.smithi107.stderr:+ sleep 0.5 2024-09-11T09:52:05.945 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-11T09:52:05.945 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T09:52:05.949 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:52:05.950+0000 7f3a7b90a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 42475) UID: 0 2024-09-11T09:52:05.973 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-11T09:52:05.974 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T09:52:06.000 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-11T09:52:06.001 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T09:52:06.012 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-11T09:52:06.012 INFO:teuthology.orchestra.run.smithi096.stdout: "size": 20, 2024-09-11T09:52:06.012 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 600, 2024-09-11T09:52:06.012 INFO:teuthology.orchestra.run.smithi096.stdout: "ops": [ 2024-09-11T09:52:06.013 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.013 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:111 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:52:06.013 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.711668+0000", 2024-09-11T09:52:06.013 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 114.300439457, 2024-09-11T09:52:06.013 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0077145570000000004, 2024-09-11T09:52:06.013 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:06.013 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:06.013 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:06.013 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:52:06.013 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:52:06.014 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:52:06.014 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 111 2024-09-11T09:52:06.014 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.014 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:06.014 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.014 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:06.014 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711668+0000", 2024-09-11T09:52:06.014 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:06.014 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.014 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.014 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:06.015 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711668+0000", 2024-09-11T09:52:06.015 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:06.015 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.015 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.015 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:06.015 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711669+0000", 2024-09-11T09:52:06.015 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.99e-07 2024-09-11T09:52:06.015 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.015 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.015 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:06.016 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711673+0000", 2024-09-11T09:52:06.016 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.7440000000000001e-06 2024-09-11T09:52:06.016 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.016 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.016 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:06.016 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711686+0000", 2024-09-11T09:52:06.016 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.3317e-05 2024-09-11T09:52:06.016 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.016 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.016 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:06.016 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711697+0000", 2024-09-11T09:52:06.016 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1022000000000001e-05 2024-09-11T09:52:06.017 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.017 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.017 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:06.017 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.717065+0000", 2024-09-11T09:52:06.017 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.005367531 2024-09-11T09:52:06.017 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.017 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.017 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:06.017 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.717114+0000", 2024-09-11T09:52:06.017 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.9707999999999999e-05 2024-09-11T09:52:06.018 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.018 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.018 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:52:06.018 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.717190+0000", 2024-09-11T09:52:06.018 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.5394000000000004e-05 2024-09-11T09:52:06.018 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.018 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.018 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:06.018 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718793+0000", 2024-09-11T09:52:06.018 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001603653 2024-09-11T09:52:06.018 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.018 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.019 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:52:06.019 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719329+0000", 2024-09-11T09:52:06.019 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00053543799999999995 2024-09-11T09:52:06.019 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.019 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.019 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:06.019 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719359+0000", 2024-09-11T09:52:06.019 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.0420999999999999e-05 2024-09-11T09:52:06.019 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.019 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.019 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:06.020 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719383+0000", 2024-09-11T09:52:06.020 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.3730000000000001e-05 2024-09-11T09:52:06.020 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:06.020 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:06.020 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:06.020 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.020 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.020 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:112 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:52:06.020 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.711712+0000", 2024-09-11T09:52:06.020 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 114.300395897, 2024-09-11T09:52:06.020 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.008162084, 2024-09-11T09:52:06.020 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:06.021 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:06.021 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:06.021 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:52:06.021 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:52:06.021 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:52:06.021 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 112 2024-09-11T09:52:06.021 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.021 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:06.021 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.021 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:06.021 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711712+0000", 2024-09-11T09:52:06.022 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:06.022 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.022 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.022 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:06.022 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711712+0000", 2024-09-11T09:52:06.022 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:06.022 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.022 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.022 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:06.022 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711712+0000", 2024-09-11T09:52:06.022 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.6000000000000004e-07 2024-09-11T09:52:06.023 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.023 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.023 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:06.023 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711808+0000", 2024-09-11T09:52:06.023 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.5977999999999995e-05 2024-09-11T09:52:06.023 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.023 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.023 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:06.023 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711819+0000", 2024-09-11T09:52:06.023 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.0755e-05 2024-09-11T09:52:06.023 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.024 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.024 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:06.024 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711826+0000", 2024-09-11T09:52:06.024 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.7970000000000003e-06 2024-09-11T09:52:06.024 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.024 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.024 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:06.024 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.717435+0000", 2024-09-11T09:52:06.024 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0056090239999999998 2024-09-11T09:52:06.024 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.024 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.024 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:06.025 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.717496+0000", 2024-09-11T09:52:06.025 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.0543e-05 2024-09-11T09:52:06.025 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.025 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.025 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:52:06.025 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.717591+0000", 2024-09-11T09:52:06.025 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.5667000000000004e-05 2024-09-11T09:52:06.025 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.025 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.025 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:06.025 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718831+0000", 2024-09-11T09:52:06.025 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001240204 2024-09-11T09:52:06.026 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.026 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.026 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:52:06.026 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719819+0000", 2024-09-11T09:52:06.026 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00098751500000000001 2024-09-11T09:52:06.026 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.026 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.026 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:06.026 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719850+0000", 2024-09-11T09:52:06.026 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.1535999999999998e-05 2024-09-11T09:52:06.026 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.026 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.027 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:06.027 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719874+0000", 2024-09-11T09:52:06.027 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.3504999999999999e-05 2024-09-11T09:52:06.027 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:06.027 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:06.027 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:06.027 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.027 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.027 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:113 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:52:06.027 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.711842+0000", 2024-09-11T09:52:06.027 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 114.30026539799999, 2024-09-11T09:52:06.027 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0081160099999999999, 2024-09-11T09:52:06.028 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:06.028 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:06.028 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:06.028 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:52:06.028 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:52:06.028 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:52:06.028 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 113 2024-09-11T09:52:06.028 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.028 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:06.028 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.028 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:06.029 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711842+0000", 2024-09-11T09:52:06.029 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:06.029 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.029 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.029 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:06.029 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711842+0000", 2024-09-11T09:52:06.029 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:06.029 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.029 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.029 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:06.029 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711843+0000", 2024-09-11T09:52:06.030 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.4399999999999999e-07 2024-09-11T09:52:06.030 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.030 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.030 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:06.030 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711848+0000", 2024-09-11T09:52:06.030 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.4159999999999997e-06 2024-09-11T09:52:06.030 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.030 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.030 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:06.030 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711856+0000", 2024-09-11T09:52:06.030 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.7870000000000007e-06 2024-09-11T09:52:06.030 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.031 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.031 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:06.031 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711862+0000", 2024-09-11T09:52:06.031 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.3689999999999998e-06 2024-09-11T09:52:06.031 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.031 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.031 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:06.031 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.717965+0000", 2024-09-11T09:52:06.031 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0061028899999999997 2024-09-11T09:52:06.031 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.031 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.031 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:06.032 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718023+0000", 2024-09-11T09:52:06.032 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.8313000000000002e-05 2024-09-11T09:52:06.032 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.032 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.032 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:52:06.032 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718109+0000", 2024-09-11T09:52:06.032 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.6205000000000006e-05 2024-09-11T09:52:06.032 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.032 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.032 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:06.033 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719808+0000", 2024-09-11T09:52:06.033 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0016990060000000001 2024-09-11T09:52:06.033 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.033 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.033 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:52:06.033 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719883+0000", 2024-09-11T09:52:06.033 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.4624999999999998e-05 2024-09-11T09:52:06.033 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.033 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.033 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:06.033 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719910+0000", 2024-09-11T09:52:06.033 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.7512e-05 2024-09-11T09:52:06.034 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.034 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.034 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:06.034 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719958+0000", 2024-09-11T09:52:06.034 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.8143000000000003e-05 2024-09-11T09:52:06.034 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:06.034 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:06.034 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:06.034 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.034 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.035 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:52:06.035 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.711887+0000", 2024-09-11T09:52:06.035 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 114.300220784, 2024-09-11T09:52:06.035 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0086063589999999992, 2024-09-11T09:52:06.035 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:06.035 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:06.035 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:06.035 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:52:06.035 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:52:06.035 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:52:06.035 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 114 2024-09-11T09:52:06.035 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.036 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:06.036 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.036 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:06.036 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711887+0000", 2024-09-11T09:52:06.036 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:06.036 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.036 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.036 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:06.036 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711887+0000", 2024-09-11T09:52:06.036 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:06.036 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.037 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.037 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:06.037 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711887+0000", 2024-09-11T09:52:06.037 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.1500000000000005e-07 2024-09-11T09:52:06.037 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.037 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.037 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:06.037 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711897+0000", 2024-09-11T09:52:06.037 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.1900000000000001e-06 2024-09-11T09:52:06.037 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.037 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.038 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:06.038 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711902+0000", 2024-09-11T09:52:06.038 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.2349999999999999e-06 2024-09-11T09:52:06.038 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.038 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.038 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:06.038 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711906+0000", 2024-09-11T09:52:06.038 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.7129999999999999e-06 2024-09-11T09:52:06.038 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.038 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.039 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:06.039 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718429+0000", 2024-09-11T09:52:06.039 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0065231960000000002 2024-09-11T09:52:06.039 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.039 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.039 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:06.039 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718480+0000", 2024-09-11T09:52:06.039 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.1161e-05 2024-09-11T09:52:06.039 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.039 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.039 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:52:06.040 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718558+0000", 2024-09-11T09:52:06.040 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.7519000000000001e-05 2024-09-11T09:52:06.040 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.040 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.040 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:06.040 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720038+0000", 2024-09-11T09:52:06.040 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001480016 2024-09-11T09:52:06.040 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.040 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.040 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:52:06.040 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720417+0000", 2024-09-11T09:52:06.041 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00037943699999999999 2024-09-11T09:52:06.041 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.041 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.041 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:06.041 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720452+0000", 2024-09-11T09:52:06.041 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.4595000000000003e-05 2024-09-11T09:52:06.041 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.041 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.041 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:06.041 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720493+0000", 2024-09-11T09:52:06.041 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.1782000000000001e-05 2024-09-11T09:52:06.042 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:06.042 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:06.042 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:06.042 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.042 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.042 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:52:06.042 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.711912+0000", 2024-09-11T09:52:06.042 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 114.30019611500001, 2024-09-11T09:52:06.042 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0086864750000000008, 2024-09-11T09:52:06.042 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:06.042 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:06.043 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:06.043 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:52:06.043 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:52:06.043 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:52:06.043 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 115 2024-09-11T09:52:06.043 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.043 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:06.043 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.043 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:06.043 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711912+0000", 2024-09-11T09:52:06.044 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:06.044 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.044 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.044 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:06.044 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711912+0000", 2024-09-11T09:52:06.044 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:06.044 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.044 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.044 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:06.044 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711912+0000", 2024-09-11T09:52:06.044 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.3900000000000001e-07 2024-09-11T09:52:06.045 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.045 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.045 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:06.045 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711969+0000", 2024-09-11T09:52:06.045 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.7501999999999999e-05 2024-09-11T09:52:06.045 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.045 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.045 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:06.045 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711986+0000", 2024-09-11T09:52:06.045 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.6402999999999999e-05 2024-09-11T09:52:06.045 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.045 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.046 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:06.046 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711990+0000", 2024-09-11T09:52:06.046 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.929e-06 2024-09-11T09:52:06.046 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.046 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.046 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:06.046 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718861+0000", 2024-09-11T09:52:06.046 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0068710530000000002 2024-09-11T09:52:06.046 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.046 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.046 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:06.047 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718908+0000", 2024-09-11T09:52:06.047 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.7431e-05 2024-09-11T09:52:06.047 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.047 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.047 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:52:06.047 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718984+0000", 2024-09-11T09:52:06.047 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.5500999999999994e-05 2024-09-11T09:52:06.047 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.047 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.047 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:52:06.047 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720500+0000", 2024-09-11T09:52:06.047 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0015158820000000001 2024-09-11T09:52:06.048 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.048 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.048 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:06.048 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720536+0000", 2024-09-11T09:52:06.048 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.6118000000000002e-05 2024-09-11T09:52:06.048 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.048 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.048 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:06.048 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720570+0000", 2024-09-11T09:52:06.048 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.4387999999999997e-05 2024-09-11T09:52:06.048 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.049 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.049 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:06.049 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720598+0000", 2024-09-11T09:52:06.049 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.8028999999999999e-05 2024-09-11T09:52:06.049 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:06.049 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:06.049 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:06.049 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.049 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.049 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:52:06.049 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.711998+0000", 2024-09-11T09:52:06.049 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 114.300109936, 2024-09-11T09:52:06.050 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.010785813999999999, 2024-09-11T09:52:06.050 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:06.050 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:06.050 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:06.050 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:52:06.050 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:52:06.050 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:52:06.050 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 116 2024-09-11T09:52:06.050 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.050 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:06.050 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.050 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:06.051 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711998+0000", 2024-09-11T09:52:06.051 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:06.051 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.051 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.051 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:06.051 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711998+0000", 2024-09-11T09:52:06.051 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:06.051 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.051 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.051 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:06.051 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711998+0000", 2024-09-11T09:52:06.051 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.3300000000000003e-07 2024-09-11T09:52:06.052 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.052 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.052 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:06.052 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712001+0000", 2024-09-11T09:52:06.052 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.9349999999999999e-06 2024-09-11T09:52:06.052 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.052 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.052 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:06.052 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712006+0000", 2024-09-11T09:52:06.052 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.1630000000000004e-06 2024-09-11T09:52:06.052 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.053 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.053 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:06.053 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712010+0000", 2024-09-11T09:52:06.053 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.574e-06 2024-09-11T09:52:06.053 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.053 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.053 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:06.053 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719415+0000", 2024-09-11T09:52:06.053 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0074049099999999998 2024-09-11T09:52:06.053 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.053 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.054 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:06.054 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719465+0000", 2024-09-11T09:52:06.054 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.0129000000000001e-05 2024-09-11T09:52:06.054 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.054 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.054 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:52:06.054 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719545+0000", 2024-09-11T09:52:06.054 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.0253999999999994e-05 2024-09-11T09:52:06.054 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.054 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.054 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:06.054 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720636+0000", 2024-09-11T09:52:06.055 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0010909330000000001 2024-09-11T09:52:06.055 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.055 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.055 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:52:06.055 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.722707+0000", 2024-09-11T09:52:06.055 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0020713300000000001 2024-09-11T09:52:06.055 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.055 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.055 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:06.055 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.722743+0000", 2024-09-11T09:52:06.055 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.5351000000000001e-05 2024-09-11T09:52:06.056 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.056 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.056 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:06.056 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.722784+0000", 2024-09-11T09:52:06.056 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.0802e-05 2024-09-11T09:52:06.056 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:06.056 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:06.056 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:06.056 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.056 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.056 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:52:06.056 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.712014+0000", 2024-09-11T09:52:06.057 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 114.30009389600001, 2024-09-11T09:52:06.057 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.013420938, 2024-09-11T09:52:06.057 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:06.057 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:06.057 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:06.057 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:52:06.057 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:52:06.057 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:52:06.057 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 117 2024-09-11T09:52:06.057 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.057 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:06.057 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.057 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:06.058 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712014+0000", 2024-09-11T09:52:06.058 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:06.058 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.058 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.058 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:06.058 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712014+0000", 2024-09-11T09:52:06.058 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:06.058 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.058 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.058 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:06.058 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712014+0000", 2024-09-11T09:52:06.058 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.2000000000000001e-07 2024-09-11T09:52:06.059 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.059 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.059 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:06.059 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712016+0000", 2024-09-11T09:52:06.059 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.2309999999999999e-06 2024-09-11T09:52:06.059 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.059 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.059 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:06.059 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712020+0000", 2024-09-11T09:52:06.059 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.3830000000000001e-06 2024-09-11T09:52:06.059 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.059 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.059 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:06.060 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712023+0000", 2024-09-11T09:52:06.060 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.3340000000000002e-06 2024-09-11T09:52:06.060 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.060 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.060 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:06.060 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720083+0000", 2024-09-11T09:52:06.060 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0080601189999999993 2024-09-11T09:52:06.060 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.060 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.060 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:06.060 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720125+0000", 2024-09-11T09:52:06.060 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.2029e-05 2024-09-11T09:52:06.060 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.061 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.061 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:52:06.061 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720205+0000", 2024-09-11T09:52:06.061 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.0010000000000001e-05 2024-09-11T09:52:06.061 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.061 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.061 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:06.061 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720849+0000", 2024-09-11T09:52:06.061 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00064369899999999996 2024-09-11T09:52:06.061 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.061 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.062 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:52:06.062 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.725357+0000", 2024-09-11T09:52:06.062 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0045077379999999998 2024-09-11T09:52:06.062 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.062 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.062 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:06.062 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.725392+0000", 2024-09-11T09:52:06.062 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.5862999999999999e-05 2024-09-11T09:52:06.062 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.062 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.062 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:06.063 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.725435+0000", 2024-09-11T09:52:06.063 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.2311999999999997e-05 2024-09-11T09:52:06.063 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:06.063 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:06.063 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:06.063 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.063 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.063 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:159 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:52:06.063 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:12.452099+0000", 2024-09-11T09:52:06.063 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 113.56000830799999, 2024-09-11T09:52:06.063 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0069885950000000002, 2024-09-11T09:52:06.064 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:06.064 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:06.064 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:06.064 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:52:06.064 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:52:06.064 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:52:06.064 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 159 2024-09-11T09:52:06.064 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.064 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:06.064 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.064 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:06.064 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452099+0000", 2024-09-11T09:52:06.064 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:06.065 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.065 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.065 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:06.065 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452099+0000", 2024-09-11T09:52:06.065 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:06.065 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.065 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.065 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:06.065 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452100+0000", 2024-09-11T09:52:06.065 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.2800000000000003e-07 2024-09-11T09:52:06.065 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.065 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.066 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:06.066 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452102+0000", 2024-09-11T09:52:06.066 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.824e-06 2024-09-11T09:52:06.066 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.066 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.066 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:06.066 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452105+0000", 2024-09-11T09:52:06.066 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.0460000000000001e-06 2024-09-11T09:52:06.066 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.066 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.066 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:06.066 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452107+0000", 2024-09-11T09:52:06.067 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.543e-06 2024-09-11T09:52:06.067 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.067 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.067 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:06.067 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.455484+0000", 2024-09-11T09:52:06.067 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0033769759999999998 2024-09-11T09:52:06.067 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.067 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.067 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:06.067 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.455527+0000", 2024-09-11T09:52:06.067 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.2589999999999997e-05 2024-09-11T09:52:06.067 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.067 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.068 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:52:06.068 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.455594+0000", 2024-09-11T09:52:06.068 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.6775999999999999e-05 2024-09-11T09:52:06.068 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.068 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.068 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:06.068 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.456312+0000", 2024-09-11T09:52:06.068 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00071874700000000005 2024-09-11T09:52:06.068 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.068 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.068 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:52:06.068 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.459015+0000", 2024-09-11T09:52:06.068 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0027028540000000002 2024-09-11T09:52:06.069 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.069 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.069 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:06.069 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.459054+0000", 2024-09-11T09:52:06.069 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.8840000000000001e-05 2024-09-11T09:52:06.069 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.069 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.069 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:06.069 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.459088+0000", 2024-09-11T09:52:06.069 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.4071e-05 2024-09-11T09:52:06.069 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:06.069 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:06.070 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:06.070 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.070 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.070 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4688.0:38 2.5 2:abde2694:::rbd_data.1250f5e25b80.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:06.070 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:22.903926+0000", 2024-09-11T09:52:06.070 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 43.108181631999997, 2024-09-11T09:52:06.070 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.028303676, 2024-09-11T09:52:06.070 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:06.070 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:06.070 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:06.070 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:52:06.070 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4688", 2024-09-11T09:52:06.070 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:52:06.071 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 38 2024-09-11T09:52:06.071 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.071 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:06.071 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.071 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:06.071 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.903926+0000", 2024-09-11T09:52:06.071 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:06.071 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.071 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.071 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:06.071 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.903926+0000", 2024-09-11T09:52:06.072 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:06.072 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.072 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.072 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:06.072 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.903928+0000", 2024-09-11T09:52:06.072 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.4249999999999999e-06 2024-09-11T09:52:06.072 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.072 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.072 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:06.072 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.914774+0000", 2024-09-11T09:52:06.072 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.010846613 2024-09-11T09:52:06.072 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.073 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.073 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:06.073 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.914793+0000", 2024-09-11T09:52:06.073 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.8879999999999999e-05 2024-09-11T09:52:06.073 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.073 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.073 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:06.073 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.914801+0000", 2024-09-11T09:52:06.073 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.2230000000000007e-06 2024-09-11T09:52:06.073 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.073 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.073 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:06.074 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.914854+0000", 2024-09-11T09:52:06.074 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.2883999999999997e-05 2024-09-11T09:52:06.074 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.074 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.074 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:06.074 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.914951+0000", 2024-09-11T09:52:06.074 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.6731000000000006e-05 2024-09-11T09:52:06.074 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.074 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.074 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:52:06.074 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.915080+0000", 2024-09-11T09:52:06.074 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00012926999999999999 2024-09-11T09:52:06.074 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.075 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.075 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:06.075 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.931469+0000", 2024-09-11T09:52:06.075 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.016389107 2024-09-11T09:52:06.075 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.075 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.075 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:52:06.075 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.932118+0000", 2024-09-11T09:52:06.075 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00064915600000000002 2024-09-11T09:52:06.075 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.075 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.075 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:06.075 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.932177+0000", 2024-09-11T09:52:06.076 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.9032e-05 2024-09-11T09:52:06.076 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.076 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.076 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:06.076 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.932230+0000", 2024-09-11T09:52:06.076 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.2355000000000003e-05 2024-09-11T09:52:06.076 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:06.076 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:06.076 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:06.076 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.076 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.076 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_repop(client.4688.0:36 2.6 e53/15 2:61e1b442:::rbd_data.1250f5e25b80.0000000000000003:head v 53'2, mlcod=0'0)", 2024-09-11T09:52:06.076 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:22.915597+0000", 2024-09-11T09:52:06.077 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 43.096511081999999, 2024-09-11T09:52:06.077 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.020430798, 2024-09-11T09:52:06.077 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:06.077 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:06.077 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:06.077 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:06.077 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.077 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:06.077 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.915597+0000", 2024-09-11T09:52:06.077 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:06.077 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.078 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.078 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:06.078 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.915597+0000", 2024-09-11T09:52:06.078 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:06.078 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.078 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.078 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:06.078 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.915598+0000", 2024-09-11T09:52:06.078 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.7150000000000001e-06 2024-09-11T09:52:06.078 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.079 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.079 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:06.079 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921741+0000", 2024-09-11T09:52:06.079 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0061424339999999996 2024-09-11T09:52:06.079 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.079 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.079 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:06.079 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921757+0000", 2024-09-11T09:52:06.079 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.6144e-05 2024-09-11T09:52:06.079 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.080 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.080 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:06.080 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921765+0000", 2024-09-11T09:52:06.080 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.0809999999999994e-06 2024-09-11T09:52:06.080 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.080 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.080 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:06.080 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921826+0000", 2024-09-11T09:52:06.080 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.0878999999999998e-05 2024-09-11T09:52:06.080 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.080 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.081 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:06.081 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921849+0000", 2024-09-11T09:52:06.081 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.2966999999999999e-05 2024-09-11T09:52:06.081 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.081 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.081 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:06.081 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.935968+0000", 2024-09-11T09:52:06.081 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.014119191 2024-09-11T09:52:06.081 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.081 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.082 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:06.082 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.936027+0000", 2024-09-11T09:52:06.082 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.9386999999999999e-05 2024-09-11T09:52:06.082 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:06.082 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:06.082 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:06.082 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.082 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.082 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_repop(client.4688.0:39 2.6 e53/15 2:743e7616:::rbd_data.1250f5e25b80.0000000000000005:head v 53'3, mlcod=0'0)", 2024-09-11T09:52:06.082 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:22.921806+0000", 2024-09-11T09:52:06.082 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 43.090302031999997, 2024-09-11T09:52:06.083 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.015275018, 2024-09-11T09:52:06.083 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:06.083 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:06.083 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:06.083 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:06.083 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.083 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:06.083 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921806+0000", 2024-09-11T09:52:06.083 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:06.083 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.084 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.084 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:06.084 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921806+0000", 2024-09-11T09:52:06.084 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:06.084 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.084 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.084 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:06.084 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921806+0000", 2024-09-11T09:52:06.084 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.2200000000000003e-07 2024-09-11T09:52:06.084 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.084 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.085 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:06.085 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.925590+0000", 2024-09-11T09:52:06.085 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0037838949999999998 2024-09-11T09:52:06.085 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.085 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.085 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:06.085 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.925605+0000", 2024-09-11T09:52:06.085 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.4321999999999999e-05 2024-09-11T09:52:06.085 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.085 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.085 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:06.086 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.925613+0000", 2024-09-11T09:52:06.086 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.1980000000000001e-06 2024-09-11T09:52:06.086 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.086 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.086 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:06.086 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.925857+0000", 2024-09-11T09:52:06.086 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000244064 2024-09-11T09:52:06.086 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.086 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.086 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:06.086 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.925875+0000", 2024-09-11T09:52:06.086 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.838e-05 2024-09-11T09:52:06.087 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.087 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.087 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:06.087 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.937042+0000", 2024-09-11T09:52:06.087 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.011166416 2024-09-11T09:52:06.087 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.087 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.087 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:06.087 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.937081+0000", 2024-09-11T09:52:06.087 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.9020999999999998e-05 2024-09-11T09:52:06.087 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:06.087 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:06.088 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:06.088 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.088 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.088 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4688.0:49 2.5 2:b6dd9fc9:::rbd_data.1250f5e25b80.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:06.088 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:22.938548+0000", 2024-09-11T09:52:06.088 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 43.073560028999999, 2024-09-11T09:52:06.088 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.022766950000000001, 2024-09-11T09:52:06.088 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:06.088 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:06.088 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:06.088 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:52:06.088 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4688", 2024-09-11T09:52:06.089 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:52:06.089 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 49 2024-09-11T09:52:06.089 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.089 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:06.089 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.089 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:06.089 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.938548+0000", 2024-09-11T09:52:06.089 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:06.089 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.089 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.089 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:06.090 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.938548+0000", 2024-09-11T09:52:06.090 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:06.090 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.090 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.090 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:06.090 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.938550+0000", 2024-09-11T09:52:06.090 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.7710000000000001e-06 2024-09-11T09:52:06.090 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.090 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.090 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:06.090 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.944237+0000", 2024-09-11T09:52:06.091 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0056879790000000001 2024-09-11T09:52:06.091 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.091 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.091 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:06.091 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.944262+0000", 2024-09-11T09:52:06.091 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.4247e-05 2024-09-11T09:52:06.091 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.091 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.091 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:06.091 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.944274+0000", 2024-09-11T09:52:06.091 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.2639999999999999e-05 2024-09-11T09:52:06.091 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.092 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.092 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:06.092 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.944344+0000", 2024-09-11T09:52:06.092 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.9574999999999997e-05 2024-09-11T09:52:06.092 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.092 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.092 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:06.092 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.944474+0000", 2024-09-11T09:52:06.092 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000130464 2024-09-11T09:52:06.092 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.093 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.093 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:52:06.093 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.944585+0000", 2024-09-11T09:52:06.093 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000110096 2024-09-11T09:52:06.093 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.093 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.093 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:52:06.093 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.952724+0000", 2024-09-11T09:52:06.093 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0081398900000000003 2024-09-11T09:52:06.093 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.093 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.094 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:06.094 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.961219+0000", 2024-09-11T09:52:06.094 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0084949789999999997 2024-09-11T09:52:06.094 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.094 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.094 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:06.094 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.961280+0000", 2024-09-11T09:52:06.094 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.0404e-05 2024-09-11T09:52:06.094 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.094 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.094 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:06.094 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.961315+0000", 2024-09-11T09:52:06.094 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.4904999999999999e-05 2024-09-11T09:52:06.095 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:06.095 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:06.095 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:06.095 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.095 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.095 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_repop(client.4688.0:44 2.6 e53/15 2:67246a22:::rbd_data.1250f5e25b80.0000000000000007:head v 53'4, mlcod=53'2)", 2024-09-11T09:52:06.095 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:22.951721+0000", 2024-09-11T09:52:06.095 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 43.060386757000003, 2024-09-11T09:52:06.095 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.01080981, 2024-09-11T09:52:06.095 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:06.095 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:06.095 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:06.096 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:06.096 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.096 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:06.096 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.951721+0000", 2024-09-11T09:52:06.096 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:06.096 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.096 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.096 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:06.096 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.951721+0000", 2024-09-11T09:52:06.096 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:06.096 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.097 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.097 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:06.097 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.951722+0000", 2024-09-11T09:52:06.097 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.0279999999999999e-06 2024-09-11T09:52:06.097 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.097 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.097 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:06.097 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.955612+0000", 2024-09-11T09:52:06.097 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0038896920000000001 2024-09-11T09:52:06.097 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.097 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.097 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:06.098 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.955628+0000", 2024-09-11T09:52:06.098 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.5945000000000001e-05 2024-09-11T09:52:06.098 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.098 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.098 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:06.098 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.955635+0000", 2024-09-11T09:52:06.098 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.8080000000000005e-06 2024-09-11T09:52:06.098 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.098 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.098 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:06.098 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.955700+0000", 2024-09-11T09:52:06.098 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.4114000000000003e-05 2024-09-11T09:52:06.099 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.099 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.099 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:06.099 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.955724+0000", 2024-09-11T09:52:06.099 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.3989999999999999e-05 2024-09-11T09:52:06.099 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.099 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.099 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:06.099 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.962490+0000", 2024-09-11T09:52:06.099 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0067662599999999996 2024-09-11T09:52:06.099 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.100 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.100 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:06.100 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.962531+0000", 2024-09-11T09:52:06.100 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.0973000000000002e-05 2024-09-11T09:52:06.100 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:06.100 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:06.100 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:06.100 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.100 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.100 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_repop(client.4688.0:51 2.6 e53/15 2:6b5894fc:::rbd_data.1250f5e25b80.000000000000000a:head v 53'5, mlcod=53'2)", 2024-09-11T09:52:06.100 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:22.959030+0000", 2024-09-11T09:52:06.100 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 43.053077582999997, 2024-09-11T09:52:06.101 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.011503638, 2024-09-11T09:52:06.101 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:06.101 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:06.101 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:06.101 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:06.101 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.101 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:06.101 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.959030+0000", 2024-09-11T09:52:06.101 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:06.101 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.101 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.102 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:06.102 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.959030+0000", 2024-09-11T09:52:06.102 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:06.102 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.102 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.102 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:06.102 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.959032+0000", 2024-09-11T09:52:06.102 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.356e-06 2024-09-11T09:52:06.102 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.102 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.102 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:06.103 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.962899+0000", 2024-09-11T09:52:06.103 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.003867639 2024-09-11T09:52:06.103 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.103 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.103 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:06.103 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.962909+0000", 2024-09-11T09:52:06.103 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.0105999999999999e-05 2024-09-11T09:52:06.103 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.103 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.103 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:06.103 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.962915+0000", 2024-09-11T09:52:06.103 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.8869999999999997e-06 2024-09-11T09:52:06.104 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.104 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.104 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:06.104 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.962962+0000", 2024-09-11T09:52:06.104 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.6511999999999998e-05 2024-09-11T09:52:06.104 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.104 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.104 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:06.104 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.962989+0000", 2024-09-11T09:52:06.104 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.7702999999999999e-05 2024-09-11T09:52:06.104 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.104 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.105 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:06.105 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.970495+0000", 2024-09-11T09:52:06.105 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0075052449999999998 2024-09-11T09:52:06.105 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.105 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.105 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:06.105 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.970534+0000", 2024-09-11T09:52:06.105 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.9190000000000003e-05 2024-09-11T09:52:06.105 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:06.105 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:06.105 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:06.105 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.106 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.106 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_repop(client.4688.0:58 2.6 e53/15 2:6277c644:::rbd_data.1250f5e25b80.000000000000000e:head v 53'6, mlcod=53'4)", 2024-09-11T09:52:06.106 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:22.996199+0000", 2024-09-11T09:52:06.106 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 43.015908662999998, 2024-09-11T09:52:06.106 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.010145573999999999, 2024-09-11T09:52:06.106 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:06.106 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:06.106 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:06.106 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:06.106 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.106 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:06.106 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.996199+0000", 2024-09-11T09:52:06.107 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:06.107 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.107 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.107 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:06.107 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.996199+0000", 2024-09-11T09:52:06.107 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:06.107 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.107 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.107 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:06.107 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.996200+0000", 2024-09-11T09:52:06.107 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.85e-07 2024-09-11T09:52:06.107 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.108 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.108 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:06.108 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.999914+0000", 2024-09-11T09:52:06.108 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.003714406 2024-09-11T09:52:06.108 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.108 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.108 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:06.108 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.999928+0000", 2024-09-11T09:52:06.108 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.3149000000000001e-05 2024-09-11T09:52:06.108 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.108 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.109 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:06.109 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.999951+0000", 2024-09-11T09:52:06.109 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.3116e-05 2024-09-11T09:52:06.109 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.109 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.109 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:06.109 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.000003+0000", 2024-09-11T09:52:06.109 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.2169000000000001e-05 2024-09-11T09:52:06.109 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.109 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.109 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:06.110 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.000018+0000", 2024-09-11T09:52:06.110 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.5500000000000001e-05 2024-09-11T09:52:06.110 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.110 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.110 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:06.110 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.006303+0000", 2024-09-11T09:52:06.110 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0062846710000000004 2024-09-11T09:52:06.110 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.110 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.110 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:06.110 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.006345+0000", 2024-09-11T09:52:06.110 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.1678000000000003e-05 2024-09-11T09:52:06.110 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:06.111 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:06.111 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:06.111 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.111 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.111 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4694.0:42 2.5 2:a4fc00ec:ns::rbd_data.1256a0c02879.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:06.111 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.111570+0000", 2024-09-11T09:52:06.111 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 42.900537557, 2024-09-11T09:52:06.111 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.038002197000000001, 2024-09-11T09:52:06.111 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:06.111 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:06.111 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:06.111 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:52:06.112 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4694", 2024-09-11T09:52:06.112 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:52:06.112 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 42 2024-09-11T09:52:06.112 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.112 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:06.112 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.112 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:06.112 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.111570+0000", 2024-09-11T09:52:06.112 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:06.112 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.112 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.113 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:06.113 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.111570+0000", 2024-09-11T09:52:06.113 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:06.113 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.113 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.113 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:06.113 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.111572+0000", 2024-09-11T09:52:06.113 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.3909999999999999e-06 2024-09-11T09:52:06.113 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.113 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.113 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:06.114 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.135929+0000", 2024-09-11T09:52:06.114 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.024357410999999999 2024-09-11T09:52:06.114 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.114 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.114 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:06.114 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.135951+0000", 2024-09-11T09:52:06.114 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.1775000000000001e-05 2024-09-11T09:52:06.114 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.114 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.114 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:06.114 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.135962+0000", 2024-09-11T09:52:06.114 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.0903e-05 2024-09-11T09:52:06.115 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.115 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.115 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:06.115 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.136045+0000", 2024-09-11T09:52:06.115 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.3128999999999996e-05 2024-09-11T09:52:06.115 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.115 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.115 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:06.115 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.136229+0000", 2024-09-11T09:52:06.115 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00018393700000000001 2024-09-11T09:52:06.115 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.116 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.116 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:52:06.116 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.136394+0000", 2024-09-11T09:52:06.116 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000165527 2024-09-11T09:52:06.116 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.116 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.116 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:06.116 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.149320+0000", 2024-09-11T09:52:06.116 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.01292568 2024-09-11T09:52:06.116 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.116 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.117 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:52:06.117 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.149493+0000", 2024-09-11T09:52:06.117 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00017285999999999999 2024-09-11T09:52:06.117 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.117 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.117 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:06.117 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.149539+0000", 2024-09-11T09:52:06.117 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.6628000000000001e-05 2024-09-11T09:52:06.117 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.117 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.117 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:06.118 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.149572+0000", 2024-09-11T09:52:06.118 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.2956000000000001e-05 2024-09-11T09:52:06.118 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:06.118 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:06.118 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:06.118 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.118 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.118 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_repop(client.4694.0:45 2.6 e53/15 2:671fb830:ns::rbd_data.1256a0c02879.0000000000000007:head v 53'9, mlcod=53'7)", 2024-09-11T09:52:06.118 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.121833+0000", 2024-09-11T09:52:06.118 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 42.890275199000001, 2024-09-11T09:52:06.118 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.023876192000000001, 2024-09-11T09:52:06.118 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:06.118 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:06.119 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:06.119 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:06.119 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.119 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:06.119 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.121833+0000", 2024-09-11T09:52:06.119 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:06.119 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.119 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.119 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:06.119 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.121833+0000", 2024-09-11T09:52:06.119 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:06.119 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.120 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.120 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:06.120 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.121834+0000", 2024-09-11T09:52:06.120 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.471e-06 2024-09-11T09:52:06.120 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.120 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.120 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:06.120 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.128990+0000", 2024-09-11T09:52:06.120 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0071557879999999997 2024-09-11T09:52:06.120 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.120 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.120 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:06.121 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.129011+0000", 2024-09-11T09:52:06.121 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.0760000000000001e-05 2024-09-11T09:52:06.121 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.121 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.121 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:06.121 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.129022+0000", 2024-09-11T09:52:06.121 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1056000000000001e-05 2024-09-11T09:52:06.121 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.121 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.121 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:06.121 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.129097+0000", 2024-09-11T09:52:06.121 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.5082999999999999e-05 2024-09-11T09:52:06.122 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.122 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.122 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:06.122 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.129130+0000", 2024-09-11T09:52:06.122 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.3345e-05 2024-09-11T09:52:06.122 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.122 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.122 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:06.122 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.145657+0000", 2024-09-11T09:52:06.122 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.016527224 2024-09-11T09:52:06.122 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.123 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.123 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:06.123 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.145709+0000", 2024-09-11T09:52:06.123 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.1465000000000003e-05 2024-09-11T09:52:06.123 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:06.123 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:06.123 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:06.123 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.123 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.123 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4694.0:43 2.0 2:16c8ae84:ns::rbd_data.1256a0c02879.0000000000000005:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:06.123 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.135999+0000", 2024-09-11T09:52:06.123 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 42.876109249000002, 2024-09-11T09:52:06.124 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.039468309, 2024-09-11T09:52:06.124 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:06.124 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:06.124 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:06.124 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:52:06.124 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4694", 2024-09-11T09:52:06.124 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:52:06.124 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 43 2024-09-11T09:52:06.124 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.124 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:06.124 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.124 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:06.125 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.135999+0000", 2024-09-11T09:52:06.125 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:06.125 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.125 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.125 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:06.125 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.135999+0000", 2024-09-11T09:52:06.125 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:06.125 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.125 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.125 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:06.125 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.136000+0000", 2024-09-11T09:52:06.125 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.6330000000000001e-06 2024-09-11T09:52:06.126 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.126 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.126 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:06.126 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.157961+0000", 2024-09-11T09:52:06.126 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0219606 2024-09-11T09:52:06.126 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.126 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.126 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:06.126 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.157995+0000", 2024-09-11T09:52:06.126 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.4391999999999999e-05 2024-09-11T09:52:06.126 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.126 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.126 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:06.127 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.158011+0000", 2024-09-11T09:52:06.127 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.5794000000000002e-05 2024-09-11T09:52:06.127 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.127 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.127 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:06.127 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.158083+0000", 2024-09-11T09:52:06.127 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.2273000000000004e-05 2024-09-11T09:52:06.127 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.127 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.127 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:06.127 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.158239+0000", 2024-09-11T09:52:06.127 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000155372 2024-09-11T09:52:06.128 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.128 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.128 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 1", 2024-09-11T09:52:06.128 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.158403+0000", 2024-09-11T09:52:06.128 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00016408199999999999 2024-09-11T09:52:06.128 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.128 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.128 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:52:06.128 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.173860+0000", 2024-09-11T09:52:06.128 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.015457347 2024-09-11T09:52:06.128 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.128 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.129 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:06.129 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.175400+0000", 2024-09-11T09:52:06.129 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001540006 2024-09-11T09:52:06.129 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.129 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.129 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:06.129 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.175438+0000", 2024-09-11T09:52:06.129 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.8135e-05 2024-09-11T09:52:06.129 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.129 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.129 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:06.129 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.175467+0000", 2024-09-11T09:52:06.129 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.8674999999999999e-05 2024-09-11T09:52:06.130 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:06.130 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:06.130 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:06.130 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.130 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.130 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4694.0:57 2.0 2:04b5549c:ns::rbd_data.1256a0c02879.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:06.130 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.158037+0000", 2024-09-11T09:52:06.130 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 42.854070511000003, 2024-09-11T09:52:06.130 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.027748790999999998, 2024-09-11T09:52:06.130 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:06.130 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:06.130 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:06.131 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:52:06.131 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4694", 2024-09-11T09:52:06.131 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:52:06.131 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 57 2024-09-11T09:52:06.131 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.131 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:06.131 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.131 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:06.131 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.158037+0000", 2024-09-11T09:52:06.131 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:06.131 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.131 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.131 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:06.132 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.158037+0000", 2024-09-11T09:52:06.132 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:06.132 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.132 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.132 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:06.132 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.158039+0000", 2024-09-11T09:52:06.132 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.356e-06 2024-09-11T09:52:06.132 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.132 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.132 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:06.132 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.176367+0000", 2024-09-11T09:52:06.133 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.018328739 2024-09-11T09:52:06.133 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.133 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.133 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:06.133 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.176378+0000", 2024-09-11T09:52:06.133 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.0713999999999999e-05 2024-09-11T09:52:06.133 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.133 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.133 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:06.133 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.176383+0000", 2024-09-11T09:52:06.133 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.0209999999999999e-06 2024-09-11T09:52:06.133 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.134 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.134 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:06.134 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.176426+0000", 2024-09-11T09:52:06.134 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.2623999999999997e-05 2024-09-11T09:52:06.134 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.134 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.134 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:06.134 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.176552+0000", 2024-09-11T09:52:06.134 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000126076 2024-09-11T09:52:06.134 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.134 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.134 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 1", 2024-09-11T09:52:06.135 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.176665+0000", 2024-09-11T09:52:06.135 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000112995 2024-09-11T09:52:06.135 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.135 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.135 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:52:06.135 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.183639+0000", 2024-09-11T09:52:06.135 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0069740039999999998 2024-09-11T09:52:06.135 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.135 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.135 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:06.135 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.185689+0000", 2024-09-11T09:52:06.135 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.002049852 2024-09-11T09:52:06.136 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.136 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.136 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:06.136 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.185725+0000", 2024-09-11T09:52:06.136 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.6804999999999997e-05 2024-09-11T09:52:06.136 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.136 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.136 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:06.136 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.185786+0000", 2024-09-11T09:52:06.136 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.0605e-05 2024-09-11T09:52:06.136 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:06.137 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:06.137 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:06.137 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.137 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.137 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4694.0:60 2.0 2:054f2af4:ns::rbd_data.1256a0c02879.000000000000000f:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:06.137 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.186855+0000", 2024-09-11T09:52:06.137 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 42.825252495000001, 2024-09-11T09:52:06.137 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.013682913, 2024-09-11T09:52:06.137 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:06.137 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:06.137 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:06.137 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:52:06.138 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4694", 2024-09-11T09:52:06.138 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:52:06.138 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 60 2024-09-11T09:52:06.138 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.138 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:06.138 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.138 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:06.138 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.186855+0000", 2024-09-11T09:52:06.138 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:06.138 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.138 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.138 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:06.139 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.186855+0000", 2024-09-11T09:52:06.139 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:06.139 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.139 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.139 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:06.139 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.186856+0000", 2024-09-11T09:52:06.139 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.1200000000000001e-07 2024-09-11T09:52:06.139 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.139 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.139 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:06.139 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.191484+0000", 2024-09-11T09:52:06.140 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0046274159999999996 2024-09-11T09:52:06.140 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.140 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.140 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:06.140 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.191496+0000", 2024-09-11T09:52:06.140 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.2149e-05 2024-09-11T09:52:06.140 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.140 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.140 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:06.140 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.191502+0000", 2024-09-11T09:52:06.140 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.1890000000000004e-06 2024-09-11T09:52:06.141 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.141 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.141 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:06.141 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.191544+0000", 2024-09-11T09:52:06.141 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.1724999999999998e-05 2024-09-11T09:52:06.141 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.141 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.141 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:06.141 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.191641+0000", 2024-09-11T09:52:06.141 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.6939000000000001e-05 2024-09-11T09:52:06.141 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.141 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.141 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 1", 2024-09-11T09:52:06.142 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.191731+0000", 2024-09-11T09:52:06.142 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.0651999999999999e-05 2024-09-11T09:52:06.142 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.142 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.142 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:52:06.142 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.199852+0000", 2024-09-11T09:52:06.142 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0081204529999999997 2024-09-11T09:52:06.142 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.142 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.142 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:06.142 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.200461+0000", 2024-09-11T09:52:06.143 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00060900899999999998 2024-09-11T09:52:06.143 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.143 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.143 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:06.143 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.200497+0000", 2024-09-11T09:52:06.143 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.5800000000000003e-05 2024-09-11T09:52:06.143 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:06.143 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:06.143 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:06.143 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.200538+0000", 2024-09-11T09:52:06.143 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.1668999999999997e-05 2024-09-11T09:52:06.143 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:06.144 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:06.144 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:06.144 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:06.144 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:06.144 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-11T09:52:06.145 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-11T09:52:06.145 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T09:52:06.146 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-11T09:52:06.146 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T09:52:06.147 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-11T09:52:06.147 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:52:06.146+0000 7f3a7b90a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 42475) UID: 0 2024-09-11T09:52:06.148 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-11T09:52:06.148 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T09:52:06.183 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-11T09:52:06.184+0000 7ff44db6e640 -1 mon.a@0(leader).osd e79 definitely_dead 0 2024-09-11T09:52:06.246 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:52:06.247+0000 7f3a7b90a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 42475) UID: 0 2024-09-11T09:52:06.290 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:52:06.291 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:52:06.299 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-11T09:52:06.347 INFO:tasks.ceph.osd.2.smithi017.stderr:2024-09-11T09:52:06.348+0000 7fd20b170640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31672) UID: 0 2024-09-11T09:52:06.416 INFO:tasks.workunit.client.0.smithi107.stderr:+ LOG_FILE=/tmp/tmp.rvrcDirNue/rbd-nbd.log 2024-09-11T09:52:06.416 INFO:tasks.workunit.client.0.smithi107.stderr:+ '[' -n '' ']' 2024-09-11T09:52:06.416 INFO:tasks.workunit.client.0.smithi107.stderr:++ _sudo rbd device --device-type nbd map --quiesce rbd/testrbdnbd27121 --log-file=/tmp/tmp.rvrcDirNue/rbd-nbd.log 2024-09-11T09:52:06.416 INFO:tasks.workunit.client.0.smithi107.stderr:++ local cmd 2024-09-11T09:52:06.416 INFO:tasks.workunit.client.0.smithi107.stderr:+++ id -u 2024-09-11T09:52:06.417 INFO:tasks.workunit.client.0.smithi107.stderr:++ '[' 1000 -eq 0 ']' 2024-09-11T09:52:06.417 INFO:tasks.workunit.client.0.smithi107.stderr:+++ which rbd 2024-09-11T09:52:06.419 INFO:tasks.workunit.client.0.smithi107.stderr:++ cmd=/usr/bin/rbd 2024-09-11T09:52:06.420 INFO:tasks.workunit.client.0.smithi107.stderr:++ shift 2024-09-11T09:52:06.420 INFO:tasks.workunit.client.0.smithi107.stderr:++ sudo -nE /usr/bin/rbd device --device-type nbd map --quiesce rbd/testrbdnbd27121 --log-file=/tmp/tmp.rvrcDirNue/rbd-nbd.log 2024-09-11T09:52:06.443 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:52:06.443 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:52:06.448 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:52:06.449+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:52:06.452 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-11T09:52:06.535 INFO:tasks.workunit.client.0.smithi107.stderr:+ DEV=/dev/nbd1 2024-09-11T09:52:06.535 INFO:tasks.workunit.client.0.smithi107.stderr:+ get_pid rbd 2024-09-11T09:52:06.535 INFO:tasks.workunit.client.0.smithi107.stderr:+ local pool=rbd 2024-09-11T09:52:06.535 INFO:tasks.workunit.client.0.smithi107.stderr:+ local ns= 2024-09-11T09:52:06.535 INFO:tasks.workunit.client.0.smithi107.stderr:++ rbd device --device-type nbd --format xml list 2024-09-11T09:52:06.535 INFO:tasks.workunit.client.0.smithi107.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd27121'\''][device='\''/dev/nbd1'\'']/id' 2024-09-11T09:52:06.549 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:52:06.550+0000 7f3c96ec9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31909) UID: 0 2024-09-11T09:52:06.556 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:52:06.556 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:52:06.565 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-11T09:52:06.621 INFO:tasks.workunit.client.0.smithi107.stderr:+ PID=29573 2024-09-11T09:52:06.621 INFO:tasks.workunit.client.0.smithi107.stderr:+ test -n 29573 2024-09-11T09:52:06.621 INFO:tasks.workunit.client.0.smithi107.stderr:+ ps -p 29573 -C rbd-nbd 2024-09-11T09:52:06.632 INFO:tasks.workunit.client.0.smithi107.stdout: PID TTY TIME CMD 2024-09-11T09:52:06.632 INFO:tasks.workunit.client.0.smithi107.stdout: 29573 ? 00:00:00 rbd-nbd 2024-09-11T09:52:06.633 INFO:tasks.workunit.client.0.smithi107.stderr:+ _sudo mkfs /dev/nbd1 2024-09-11T09:52:06.633 INFO:tasks.workunit.client.0.smithi107.stderr:+ local cmd 2024-09-11T09:52:06.633 INFO:tasks.workunit.client.0.smithi107.stderr:++ id -u 2024-09-11T09:52:06.633 INFO:tasks.workunit.client.0.smithi107.stderr:+ '[' 1000 -eq 0 ']' 2024-09-11T09:52:06.634 INFO:tasks.workunit.client.0.smithi107.stderr:++ which mkfs 2024-09-11T09:52:06.635 INFO:tasks.workunit.client.0.smithi107.stderr:+ cmd=/usr/sbin/mkfs 2024-09-11T09:52:06.636 INFO:tasks.workunit.client.0.smithi107.stderr:+ shift 2024-09-11T09:52:06.636 INFO:tasks.workunit.client.0.smithi107.stderr:+ sudo -nE /usr/sbin/mkfs /dev/nbd1 2024-09-11T09:52:06.650 INFO:tasks.ceph.osd.0.smithi017.stderr:2024-09-11T09:52:06.651+0000 7fb73c0f1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31665) UID: 0 2024-09-11T09:52:06.666 INFO:tasks.workunit.client.0.smithi107.stderr:mke2fs 1.46.5 (30-Dec-2021) 2024-09-11T09:52:06.701 INFO:tasks.workunit.client.0.smithi107.stdout:Discarding device blocks: 0/65536 done 2024-09-11T09:52:06.701 INFO:tasks.workunit.client.0.smithi107.stdout:Creating filesystem with 65536 1k blocks and 16384 inodes 2024-09-11T09:52:06.701 INFO:tasks.workunit.client.0.smithi107.stdout:Filesystem UUID: 79bf567d-6fd7-4e56-9650-aa64700d5562 2024-09-11T09:52:06.701 INFO:tasks.workunit.client.0.smithi107.stdout:Superblock backups stored on blocks: 2024-09-11T09:52:06.702 INFO:tasks.workunit.client.0.smithi107.stdout: 8193, 24577, 40961, 57345 2024-09-11T09:52:06.702 INFO:tasks.workunit.client.0.smithi107.stdout: 2024-09-11T09:52:06.702 INFO:tasks.workunit.client.0.smithi107.stdout:Allocating group tables: 0/8 done 2024-09-11T09:52:06.714 INFO:tasks.workunit.client.0.smithi107.stdout:Writing inode tables: 0/8 done 2024-09-11T09:52:06.725 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:52:06.726 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:52:06.736 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-11T09:52:06.751 INFO:tasks.ceph.osd.0.smithi017.stderr:2024-09-11T09:52:06.751+0000 7fb73c0f1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31665) UID: 0 2024-09-11T09:52:06.768 INFO:tasks.workunit.client.0.smithi107.stdout:Writing superblocks and filesystem accounting information: 0/8 done 2024-09-11T09:52:06.769 INFO:tasks.workunit.client.0.smithi107.stdout: 2024-09-11T09:52:06.770 INFO:tasks.workunit.client.0.smithi107.stderr:+ mkdir /tmp/tmp.rvrcDirNue/mnt 2024-09-11T09:52:06.771 INFO:tasks.workunit.client.0.smithi107.stderr:+ _sudo mount /dev/nbd1 /tmp/tmp.rvrcDirNue/mnt 2024-09-11T09:52:06.771 INFO:tasks.workunit.client.0.smithi107.stderr:+ local cmd 2024-09-11T09:52:06.771 INFO:tasks.workunit.client.0.smithi107.stderr:++ id -u 2024-09-11T09:52:06.772 INFO:tasks.workunit.client.0.smithi107.stderr:+ '[' 1000 -eq 0 ']' 2024-09-11T09:52:06.772 INFO:tasks.workunit.client.0.smithi107.stderr:++ which mount 2024-09-11T09:52:06.774 INFO:tasks.workunit.client.0.smithi107.stderr:+ cmd=/usr/bin/mount 2024-09-11T09:52:06.774 INFO:tasks.workunit.client.0.smithi107.stderr:+ shift 2024-09-11T09:52:06.774 INFO:tasks.workunit.client.0.smithi107.stderr:+ sudo -nE /usr/bin/mount /dev/nbd1 /tmp/tmp.rvrcDirNue/mnt 2024-09-11T09:52:06.806 INFO:tasks.workunit.client.0.smithi107.stderr:+ rbd snap create rbd/testrbdnbd27121@quiesce3 2024-09-11T09:52:06.835 INFO:teuthology.orchestra.run.smithi017.stderr:marked out osd.0. 2024-09-11T09:52:06.835 INFO:teuthology.orchestra.run.smithi017.stderr:nodeep-scrub is set 2024-09-11T09:52:06.841 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:52:06.842 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:52:06.851 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-11T09:52:06.853 INFO:tasks.ceph.osd.0.smithi017.stderr:2024-09-11T09:52:06.853+0000 7fb73c0f1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31665) UID: 0 2024-09-11T09:52:06.953 INFO:tasks.ceph.osd.2.smithi017.stderr:2024-09-11T09:52:06.954+0000 7fd20b170640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31672) UID: 0 2024-09-11T09:52:06.954 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:52:06.954 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:52:06.963 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-11T09:52:07.054 INFO:tasks.ceph.osd.2.smithi017.stderr:2024-09-11T09:52:07.055+0000 7fd20b170640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31672) UID: 0 2024-09-11T09:52:07.071 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:52:07.071 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:52:07.080 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-11T09:52:07.164 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:52:07.165+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:52:07.227 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:52:07.228 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:52:07.236 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-11T09:52:07.256 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-09-11T09:52:07.257+0000 7f7b6c7d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31904) UID: 0 2024-09-11T09:52:07.357 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:52:07.358+0000 7f3a7b90a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 42475) UID: 0 2024-09-11T09:52:07.384 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:52:07.385 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:52:07.393 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-11T09:52:07.458 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-09-11T09:52:07.459+0000 7f7b6c7d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31904) UID: 0 2024-09-11T09:52:07.500 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:52:07.501 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:52:07.510 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-11T09:52:07.559 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:52:07.560+0000 7f3c96ec9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31909) UID: 0 2024-09-11T09:52:07.617 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:52:07.617 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:52:07.626 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-11T09:52:07.660 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:52:07.661+0000 7f3c96ec9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31909) UID: 0 2024-09-11T09:52:07.734 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:52:07.735 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:52:07.744 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-11T09:52:07.762 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-09-11T09:52:07.763+0000 7f7b6c7d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31904) UID: 0 2024-09-11T09:52:07.856 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:52:07.856 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:52:07.862 INFO:tasks.ceph.osd.0.smithi017.stderr:2024-09-11T09:52:07.863+0000 7fb73c0f1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31665) UID: 0 2024-09-11T09:52:07.865 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-11T09:52:07.963 INFO:tasks.ceph.osd.2.smithi017.stderr:2024-09-11T09:52:07.964+0000 7fd20b170640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31672) UID: 0 2024-09-11T09:52:08.012 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:52:08.013 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:52:08.022 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-11T09:52:08.064 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:52:08.065+0000 7f3a7b90a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 42475) UID: 0 2024-09-11T09:52:08.150 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T09:52:08.171 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:52:08.171 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:52:08.171 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:52:08.172+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:52:08.181 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-11T09:52:08.265 INFO:tasks.ceph.osd.2.smithi017.stderr:2024-09-11T09:52:08.266+0000 7fd20b170640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31672) UID: 0 2024-09-11T09:52:08.273 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-11T09:52:08.274 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T09:52:08.290 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:52:08.291 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:52:08.291 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-11T09:52:08.291 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T09:52:08.299 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-11T09:52:08.307 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-11T09:52:08.307 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T09:52:08.324 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-11T09:52:08.324 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T09:52:08.340 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-11T09:52:08.341 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T09:52:08.357 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-11T09:52:08.358 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T09:52:08.366 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:52:08.367+0000 7f3c96ec9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31909) UID: 0 2024-09-11T09:52:08.403 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-11T09:52:08.403 INFO:teuthology.orchestra.run.smithi096.stdout: "ops": [], 2024-09-11T09:52:08.404 INFO:teuthology.orchestra.run.smithi096.stdout: "complaint_time": 30, 2024-09-11T09:52:08.404 INFO:teuthology.orchestra.run.smithi096.stdout: "num_blocked_ops": 0 2024-09-11T09:52:08.404 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-11T09:52:08.412 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-11T09:52:08.467 INFO:tasks.ceph.osd.2.smithi017.stderr:2024-09-11T09:52:08.468+0000 7fd20b170640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31672) UID: 0 2024-09-11T09:52:08.557 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-11T09:52:08.557 INFO:teuthology.orchestra.run.smithi096.stdout: "size": 20, 2024-09-11T09:52:08.557 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 600, 2024-09-11T09:52:08.557 INFO:teuthology.orchestra.run.smithi096.stdout: "ops": [ 2024-09-11T09:52:08.558 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.558 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:111 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:52:08.558 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.711668+0000", 2024-09-11T09:52:08.558 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 116.845418204, 2024-09-11T09:52:08.558 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0077145570000000004, 2024-09-11T09:52:08.558 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:08.558 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:08.558 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:08.558 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:52:08.558 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:52:08.559 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:52:08.559 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 111 2024-09-11T09:52:08.559 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.559 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:08.559 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.559 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:08.559 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711668+0000", 2024-09-11T09:52:08.559 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:08.559 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.559 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.559 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:08.559 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711668+0000", 2024-09-11T09:52:08.560 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:08.560 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.560 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.560 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:08.560 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711669+0000", 2024-09-11T09:52:08.560 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.99e-07 2024-09-11T09:52:08.560 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.560 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.560 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:08.560 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711673+0000", 2024-09-11T09:52:08.560 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.7440000000000001e-06 2024-09-11T09:52:08.560 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.561 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.561 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:08.561 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711686+0000", 2024-09-11T09:52:08.561 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.3317e-05 2024-09-11T09:52:08.561 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.561 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.561 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:08.561 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711697+0000", 2024-09-11T09:52:08.561 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1022000000000001e-05 2024-09-11T09:52:08.561 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.561 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.561 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:08.562 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.717065+0000", 2024-09-11T09:52:08.562 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.005367531 2024-09-11T09:52:08.562 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.562 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.562 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:08.562 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.717114+0000", 2024-09-11T09:52:08.562 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.9707999999999999e-05 2024-09-11T09:52:08.562 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.562 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.562 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:52:08.562 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.717190+0000", 2024-09-11T09:52:08.562 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.5394000000000004e-05 2024-09-11T09:52:08.562 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.563 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.563 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:08.563 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718793+0000", 2024-09-11T09:52:08.563 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001603653 2024-09-11T09:52:08.563 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.563 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.563 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:52:08.563 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719329+0000", 2024-09-11T09:52:08.563 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00053543799999999995 2024-09-11T09:52:08.563 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.563 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.563 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:08.564 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719359+0000", 2024-09-11T09:52:08.564 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.0420999999999999e-05 2024-09-11T09:52:08.564 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.564 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.564 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:08.564 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719383+0000", 2024-09-11T09:52:08.564 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.3730000000000001e-05 2024-09-11T09:52:08.564 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:08.564 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:08.564 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:08.564 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.565 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.565 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:112 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:52:08.565 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.711712+0000", 2024-09-11T09:52:08.565 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 116.845374644, 2024-09-11T09:52:08.565 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.008162084, 2024-09-11T09:52:08.565 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:08.565 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:08.565 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:08.565 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:52:08.565 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:52:08.565 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:52:08.565 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 112 2024-09-11T09:52:08.565 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.566 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:08.566 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.566 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:08.566 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711712+0000", 2024-09-11T09:52:08.566 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:08.566 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.566 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.566 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:08.566 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711712+0000", 2024-09-11T09:52:08.566 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:08.566 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.566 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.566 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:08.567 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711712+0000", 2024-09-11T09:52:08.567 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.6000000000000004e-07 2024-09-11T09:52:08.567 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.567 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.567 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:08.567 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711808+0000", 2024-09-11T09:52:08.567 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.5977999999999995e-05 2024-09-11T09:52:08.567 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.567 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.567 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:08.568 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711819+0000", 2024-09-11T09:52:08.568 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.0755e-05 2024-09-11T09:52:08.568 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.568 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.568 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:08.568 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711826+0000", 2024-09-11T09:52:08.568 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.7970000000000003e-06 2024-09-11T09:52:08.568 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.568 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.568 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:08.568 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.717435+0000", 2024-09-11T09:52:08.568 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0056090239999999998 2024-09-11T09:52:08.569 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.569 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.569 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:08.569 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.717496+0000", 2024-09-11T09:52:08.569 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.0543e-05 2024-09-11T09:52:08.569 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.569 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.569 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:52:08.569 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.717591+0000", 2024-09-11T09:52:08.569 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.5667000000000004e-05 2024-09-11T09:52:08.569 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.570 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.570 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:08.570 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718831+0000", 2024-09-11T09:52:08.570 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001240204 2024-09-11T09:52:08.570 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.570 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.570 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:52:08.570 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719819+0000", 2024-09-11T09:52:08.570 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00098751500000000001 2024-09-11T09:52:08.570 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.570 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.570 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:08.571 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719850+0000", 2024-09-11T09:52:08.571 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.1535999999999998e-05 2024-09-11T09:52:08.571 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.571 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.571 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:08.571 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719874+0000", 2024-09-11T09:52:08.571 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.3504999999999999e-05 2024-09-11T09:52:08.571 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:08.571 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:08.571 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:08.571 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.571 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.571 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:113 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:52:08.572 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.711842+0000", 2024-09-11T09:52:08.572 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 116.845244145, 2024-09-11T09:52:08.572 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0081160099999999999, 2024-09-11T09:52:08.572 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:08.572 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:08.572 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:08.572 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:52:08.572 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:52:08.572 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:52:08.573 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 113 2024-09-11T09:52:08.573 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.573 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:08.573 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.573 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:08.573 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711842+0000", 2024-09-11T09:52:08.573 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:08.573 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.573 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.573 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:08.573 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711842+0000", 2024-09-11T09:52:08.573 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:08.573 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.574 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.574 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:08.574 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711843+0000", 2024-09-11T09:52:08.574 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.4399999999999999e-07 2024-09-11T09:52:08.574 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.574 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.574 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:08.574 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711848+0000", 2024-09-11T09:52:08.574 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.4159999999999997e-06 2024-09-11T09:52:08.574 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.574 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.574 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:08.575 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711856+0000", 2024-09-11T09:52:08.575 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.7870000000000007e-06 2024-09-11T09:52:08.575 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.575 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.575 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:08.575 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711862+0000", 2024-09-11T09:52:08.575 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.3689999999999998e-06 2024-09-11T09:52:08.575 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.575 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.575 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:08.575 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.717965+0000", 2024-09-11T09:52:08.575 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0061028899999999997 2024-09-11T09:52:08.575 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.576 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.576 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:08.576 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718023+0000", 2024-09-11T09:52:08.576 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.8313000000000002e-05 2024-09-11T09:52:08.576 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.576 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.576 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:52:08.576 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718109+0000", 2024-09-11T09:52:08.576 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.6205000000000006e-05 2024-09-11T09:52:08.576 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.576 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.576 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:08.577 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719808+0000", 2024-09-11T09:52:08.577 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0016990060000000001 2024-09-11T09:52:08.577 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.577 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.577 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:52:08.577 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719883+0000", 2024-09-11T09:52:08.577 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.4624999999999998e-05 2024-09-11T09:52:08.577 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.577 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.577 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:08.577 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719910+0000", 2024-09-11T09:52:08.577 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.7512e-05 2024-09-11T09:52:08.577 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.578 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.578 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:08.578 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719958+0000", 2024-09-11T09:52:08.578 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.8143000000000003e-05 2024-09-11T09:52:08.578 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:08.578 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:08.578 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:08.578 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.578 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.578 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:52:08.578 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.711887+0000", 2024-09-11T09:52:08.578 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 116.84519953100001, 2024-09-11T09:52:08.578 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0086063589999999992, 2024-09-11T09:52:08.578 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:08.579 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:08.579 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:08.579 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:52:08.579 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:52:08.579 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:52:08.579 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 114 2024-09-11T09:52:08.579 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.579 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:08.579 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.579 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:08.579 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711887+0000", 2024-09-11T09:52:08.579 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:08.579 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.580 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.580 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:08.580 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711887+0000", 2024-09-11T09:52:08.580 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:08.580 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.580 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.580 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:08.580 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711887+0000", 2024-09-11T09:52:08.580 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.1500000000000005e-07 2024-09-11T09:52:08.580 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.580 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.580 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:08.581 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711897+0000", 2024-09-11T09:52:08.581 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.1900000000000001e-06 2024-09-11T09:52:08.581 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.581 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.581 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:08.581 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711902+0000", 2024-09-11T09:52:08.581 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.2349999999999999e-06 2024-09-11T09:52:08.581 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.581 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.581 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:08.581 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711906+0000", 2024-09-11T09:52:08.581 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.7129999999999999e-06 2024-09-11T09:52:08.581 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.582 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.582 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:08.582 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718429+0000", 2024-09-11T09:52:08.582 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0065231960000000002 2024-09-11T09:52:08.582 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.582 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.582 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:08.582 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718480+0000", 2024-09-11T09:52:08.582 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.1161e-05 2024-09-11T09:52:08.582 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.582 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.582 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:52:08.582 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718558+0000", 2024-09-11T09:52:08.583 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.7519000000000001e-05 2024-09-11T09:52:08.583 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.583 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.583 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:08.583 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720038+0000", 2024-09-11T09:52:08.583 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001480016 2024-09-11T09:52:08.583 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.583 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.583 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:52:08.583 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720417+0000", 2024-09-11T09:52:08.583 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00037943699999999999 2024-09-11T09:52:08.583 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.583 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.583 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:08.584 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720452+0000", 2024-09-11T09:52:08.584 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.4595000000000003e-05 2024-09-11T09:52:08.584 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.584 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.584 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:08.584 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720493+0000", 2024-09-11T09:52:08.584 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.1782000000000001e-05 2024-09-11T09:52:08.584 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:08.584 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:08.584 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:08.584 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.584 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.585 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:52:08.585 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.711912+0000", 2024-09-11T09:52:08.585 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 116.84517486199999, 2024-09-11T09:52:08.585 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0086864750000000008, 2024-09-11T09:52:08.585 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:08.585 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:08.585 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:08.585 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:52:08.585 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:52:08.585 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:52:08.586 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 115 2024-09-11T09:52:08.586 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.586 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:08.586 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.586 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:08.586 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711912+0000", 2024-09-11T09:52:08.586 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:08.586 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.586 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.586 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:08.586 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711912+0000", 2024-09-11T09:52:08.587 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:08.587 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.587 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.587 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:08.587 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711912+0000", 2024-09-11T09:52:08.587 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.3900000000000001e-07 2024-09-11T09:52:08.587 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.587 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.587 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:08.587 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711969+0000", 2024-09-11T09:52:08.587 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.7501999999999999e-05 2024-09-11T09:52:08.587 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.587 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.588 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:08.588 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711986+0000", 2024-09-11T09:52:08.588 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.6402999999999999e-05 2024-09-11T09:52:08.588 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.588 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.588 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:08.588 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711990+0000", 2024-09-11T09:52:08.588 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.929e-06 2024-09-11T09:52:08.588 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.589 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.589 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:08.589 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718861+0000", 2024-09-11T09:52:08.589 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0068710530000000002 2024-09-11T09:52:08.589 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.589 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.589 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:08.589 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718908+0000", 2024-09-11T09:52:08.589 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.7431e-05 2024-09-11T09:52:08.589 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.589 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.589 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:52:08.590 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718984+0000", 2024-09-11T09:52:08.590 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.5500999999999994e-05 2024-09-11T09:52:08.590 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.590 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.590 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:52:08.590 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720500+0000", 2024-09-11T09:52:08.590 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0015158820000000001 2024-09-11T09:52:08.590 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.590 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.590 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:08.590 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720536+0000", 2024-09-11T09:52:08.590 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.6118000000000002e-05 2024-09-11T09:52:08.591 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.591 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.591 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:08.591 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720570+0000", 2024-09-11T09:52:08.591 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.4387999999999997e-05 2024-09-11T09:52:08.591 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.591 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.591 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:08.591 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720598+0000", 2024-09-11T09:52:08.591 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.8028999999999999e-05 2024-09-11T09:52:08.591 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:08.592 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:08.592 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:08.592 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.592 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.592 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:52:08.592 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.711998+0000", 2024-09-11T09:52:08.592 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 116.845088683, 2024-09-11T09:52:08.592 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.010785813999999999, 2024-09-11T09:52:08.592 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:08.592 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:08.592 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:08.592 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:52:08.593 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:52:08.593 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:52:08.593 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 116 2024-09-11T09:52:08.593 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.593 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:08.593 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.593 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:08.593 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711998+0000", 2024-09-11T09:52:08.593 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:08.593 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.593 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.593 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:08.593 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711998+0000", 2024-09-11T09:52:08.594 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:08.594 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.594 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.594 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:08.594 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711998+0000", 2024-09-11T09:52:08.594 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.3300000000000003e-07 2024-09-11T09:52:08.594 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.594 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.594 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:08.594 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712001+0000", 2024-09-11T09:52:08.594 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.9349999999999999e-06 2024-09-11T09:52:08.594 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.594 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.595 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:08.595 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712006+0000", 2024-09-11T09:52:08.595 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.1630000000000004e-06 2024-09-11T09:52:08.595 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.595 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.595 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:08.595 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712010+0000", 2024-09-11T09:52:08.595 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.574e-06 2024-09-11T09:52:08.595 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.595 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.595 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:08.595 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719415+0000", 2024-09-11T09:52:08.596 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0074049099999999998 2024-09-11T09:52:08.596 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.596 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.596 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:08.596 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719465+0000", 2024-09-11T09:52:08.596 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.0129000000000001e-05 2024-09-11T09:52:08.596 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.596 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.596 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:52:08.596 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719545+0000", 2024-09-11T09:52:08.596 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.0253999999999994e-05 2024-09-11T09:52:08.596 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.596 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.597 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:08.597 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720636+0000", 2024-09-11T09:52:08.597 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0010909330000000001 2024-09-11T09:52:08.597 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.597 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.597 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:52:08.597 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.722707+0000", 2024-09-11T09:52:08.597 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0020713300000000001 2024-09-11T09:52:08.597 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.597 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.597 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:08.597 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.722743+0000", 2024-09-11T09:52:08.598 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.5351000000000001e-05 2024-09-11T09:52:08.598 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.598 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.598 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:08.598 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.722784+0000", 2024-09-11T09:52:08.598 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.0802e-05 2024-09-11T09:52:08.598 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:08.598 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:08.598 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:08.598 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.598 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.598 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:52:08.598 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.712014+0000", 2024-09-11T09:52:08.599 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 116.84507264299999, 2024-09-11T09:52:08.599 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.013420938, 2024-09-11T09:52:08.599 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:08.599 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:08.599 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:08.599 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:52:08.599 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:52:08.599 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:52:08.599 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 117 2024-09-11T09:52:08.599 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.599 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:08.599 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.599 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:08.600 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712014+0000", 2024-09-11T09:52:08.600 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:08.600 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.600 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.600 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:08.600 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712014+0000", 2024-09-11T09:52:08.600 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:08.600 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.600 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.600 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:08.600 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712014+0000", 2024-09-11T09:52:08.600 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.2000000000000001e-07 2024-09-11T09:52:08.600 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.601 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.601 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:08.601 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712016+0000", 2024-09-11T09:52:08.601 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.2309999999999999e-06 2024-09-11T09:52:08.601 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.601 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.601 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:08.601 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712020+0000", 2024-09-11T09:52:08.601 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.3830000000000001e-06 2024-09-11T09:52:08.601 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.601 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.601 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:08.602 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712023+0000", 2024-09-11T09:52:08.602 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.3340000000000002e-06 2024-09-11T09:52:08.602 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.602 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.602 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:08.602 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720083+0000", 2024-09-11T09:52:08.602 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0080601189999999993 2024-09-11T09:52:08.602 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.602 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.602 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:08.602 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720125+0000", 2024-09-11T09:52:08.602 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.2029e-05 2024-09-11T09:52:08.603 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.603 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.603 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:52:08.603 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720205+0000", 2024-09-11T09:52:08.603 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.0010000000000001e-05 2024-09-11T09:52:08.603 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.603 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.603 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:08.603 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720849+0000", 2024-09-11T09:52:08.603 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00064369899999999996 2024-09-11T09:52:08.603 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.603 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.604 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:52:08.604 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.725357+0000", 2024-09-11T09:52:08.604 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0045077379999999998 2024-09-11T09:52:08.604 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.604 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.604 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:08.604 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.725392+0000", 2024-09-11T09:52:08.604 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.5862999999999999e-05 2024-09-11T09:52:08.604 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.604 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.604 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:08.605 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.725435+0000", 2024-09-11T09:52:08.605 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.2311999999999997e-05 2024-09-11T09:52:08.605 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:08.605 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:08.605 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:08.605 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.605 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.605 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:159 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:52:08.605 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:12.452099+0000", 2024-09-11T09:52:08.605 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 116.104987055, 2024-09-11T09:52:08.605 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0069885950000000002, 2024-09-11T09:52:08.606 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:08.606 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:08.606 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:08.606 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:52:08.606 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:52:08.606 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:52:08.606 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 159 2024-09-11T09:52:08.606 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.606 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:08.606 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.606 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:08.607 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452099+0000", 2024-09-11T09:52:08.607 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:08.607 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.607 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.607 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:08.607 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452099+0000", 2024-09-11T09:52:08.607 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:08.607 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.607 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.607 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:08.607 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452100+0000", 2024-09-11T09:52:08.607 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.2800000000000003e-07 2024-09-11T09:52:08.608 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.608 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.608 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:08.608 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452102+0000", 2024-09-11T09:52:08.608 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.824e-06 2024-09-11T09:52:08.608 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.608 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.608 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:08.608 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452105+0000", 2024-09-11T09:52:08.608 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.0460000000000001e-06 2024-09-11T09:52:08.608 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.608 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.609 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:08.609 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452107+0000", 2024-09-11T09:52:08.609 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.543e-06 2024-09-11T09:52:08.609 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.609 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.609 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:08.609 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.455484+0000", 2024-09-11T09:52:08.609 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0033769759999999998 2024-09-11T09:52:08.609 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.609 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.609 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:08.609 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.455527+0000", 2024-09-11T09:52:08.610 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.2589999999999997e-05 2024-09-11T09:52:08.610 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.610 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.610 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:52:08.610 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.455594+0000", 2024-09-11T09:52:08.610 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.6775999999999999e-05 2024-09-11T09:52:08.610 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.610 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.610 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:08.610 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.456312+0000", 2024-09-11T09:52:08.610 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00071874700000000005 2024-09-11T09:52:08.610 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.610 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.611 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:52:08.611 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.459015+0000", 2024-09-11T09:52:08.611 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0027028540000000002 2024-09-11T09:52:08.611 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.611 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.611 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:08.611 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.459054+0000", 2024-09-11T09:52:08.611 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.8840000000000001e-05 2024-09-11T09:52:08.611 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.611 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.611 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:08.612 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.459088+0000", 2024-09-11T09:52:08.612 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.4071e-05 2024-09-11T09:52:08.612 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:08.612 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:08.612 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:08.612 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.612 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.612 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4688.0:38 2.5 2:abde2694:::rbd_data.1250f5e25b80.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:08.612 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:22.903926+0000", 2024-09-11T09:52:08.612 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 45.653160378999999, 2024-09-11T09:52:08.612 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.028303676, 2024-09-11T09:52:08.612 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:08.613 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:08.613 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:08.613 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:52:08.613 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4688", 2024-09-11T09:52:08.613 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:52:08.613 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 38 2024-09-11T09:52:08.613 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.613 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:08.613 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.613 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:08.613 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.903926+0000", 2024-09-11T09:52:08.614 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:08.614 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.614 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.614 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:08.614 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.903926+0000", 2024-09-11T09:52:08.614 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:08.614 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.614 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.614 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:08.614 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.903928+0000", 2024-09-11T09:52:08.614 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.4249999999999999e-06 2024-09-11T09:52:08.614 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.614 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.615 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:08.615 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.914774+0000", 2024-09-11T09:52:08.615 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.010846613 2024-09-11T09:52:08.615 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.615 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.615 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:08.615 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.914793+0000", 2024-09-11T09:52:08.615 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.8879999999999999e-05 2024-09-11T09:52:08.615 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.615 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.615 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:08.615 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.914801+0000", 2024-09-11T09:52:08.616 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.2230000000000007e-06 2024-09-11T09:52:08.616 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.616 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.616 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:08.616 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.914854+0000", 2024-09-11T09:52:08.616 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.2883999999999997e-05 2024-09-11T09:52:08.616 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.616 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.616 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:08.616 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.914951+0000", 2024-09-11T09:52:08.616 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.6731000000000006e-05 2024-09-11T09:52:08.616 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.616 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.617 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:52:08.617 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.915080+0000", 2024-09-11T09:52:08.617 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00012926999999999999 2024-09-11T09:52:08.617 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.617 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.617 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:08.617 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.931469+0000", 2024-09-11T09:52:08.617 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.016389107 2024-09-11T09:52:08.617 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.617 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.618 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:52:08.618 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.932118+0000", 2024-09-11T09:52:08.618 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00064915600000000002 2024-09-11T09:52:08.618 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.618 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.618 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:08.618 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.932177+0000", 2024-09-11T09:52:08.618 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.9032e-05 2024-09-11T09:52:08.618 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.618 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.618 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:08.618 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.932230+0000", 2024-09-11T09:52:08.618 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.2355000000000003e-05 2024-09-11T09:52:08.619 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:08.619 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:08.619 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:08.619 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.619 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.619 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_repop(client.4688.0:36 2.6 e53/15 2:61e1b442:::rbd_data.1250f5e25b80.0000000000000003:head v 53'2, mlcod=0'0)", 2024-09-11T09:52:08.619 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:22.915597+0000", 2024-09-11T09:52:08.619 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 45.641489829000001, 2024-09-11T09:52:08.619 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.020430798, 2024-09-11T09:52:08.619 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:08.619 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:08.619 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:08.619 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:08.620 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.620 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:08.620 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.915597+0000", 2024-09-11T09:52:08.620 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:08.620 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.620 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.620 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:08.620 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.915597+0000", 2024-09-11T09:52:08.620 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:08.620 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.620 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.620 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:08.620 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.915598+0000", 2024-09-11T09:52:08.621 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.7150000000000001e-06 2024-09-11T09:52:08.621 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.621 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.621 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:08.621 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921741+0000", 2024-09-11T09:52:08.621 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0061424339999999996 2024-09-11T09:52:08.621 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.621 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.621 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:08.621 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921757+0000", 2024-09-11T09:52:08.622 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.6144e-05 2024-09-11T09:52:08.622 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.622 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.622 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:08.622 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921765+0000", 2024-09-11T09:52:08.622 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.0809999999999994e-06 2024-09-11T09:52:08.622 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.622 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.622 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:08.622 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921826+0000", 2024-09-11T09:52:08.623 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.0878999999999998e-05 2024-09-11T09:52:08.623 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.623 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.623 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:08.623 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921849+0000", 2024-09-11T09:52:08.623 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.2966999999999999e-05 2024-09-11T09:52:08.623 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.623 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.623 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:08.623 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.935968+0000", 2024-09-11T09:52:08.623 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.014119191 2024-09-11T09:52:08.623 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.623 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.624 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:08.624 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.936027+0000", 2024-09-11T09:52:08.624 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.9386999999999999e-05 2024-09-11T09:52:08.624 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:08.624 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:08.624 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:08.624 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.624 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.624 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_repop(client.4688.0:39 2.6 e53/15 2:743e7616:::rbd_data.1250f5e25b80.0000000000000005:head v 53'3, mlcod=0'0)", 2024-09-11T09:52:08.624 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:22.921806+0000", 2024-09-11T09:52:08.625 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 45.635280778999999, 2024-09-11T09:52:08.625 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.015275018, 2024-09-11T09:52:08.625 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:08.625 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:08.625 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:08.625 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:08.625 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.625 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:08.625 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921806+0000", 2024-09-11T09:52:08.625 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:08.626 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.626 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.626 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:08.626 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921806+0000", 2024-09-11T09:52:08.626 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:08.626 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.626 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.626 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:08.626 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921806+0000", 2024-09-11T09:52:08.626 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.2200000000000003e-07 2024-09-11T09:52:08.626 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.626 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.626 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:08.627 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.925590+0000", 2024-09-11T09:52:08.627 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0037838949999999998 2024-09-11T09:52:08.627 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.627 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.627 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:08.627 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.925605+0000", 2024-09-11T09:52:08.627 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.4321999999999999e-05 2024-09-11T09:52:08.627 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.627 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.627 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:08.627 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.925613+0000", 2024-09-11T09:52:08.627 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.1980000000000001e-06 2024-09-11T09:52:08.627 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.628 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.628 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:08.628 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.925857+0000", 2024-09-11T09:52:08.628 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000244064 2024-09-11T09:52:08.628 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.628 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.628 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:08.628 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.925875+0000", 2024-09-11T09:52:08.628 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.838e-05 2024-09-11T09:52:08.628 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.628 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.628 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:08.629 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.937042+0000", 2024-09-11T09:52:08.629 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.011166416 2024-09-11T09:52:08.629 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.629 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.629 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:08.629 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.937081+0000", 2024-09-11T09:52:08.629 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.9020999999999998e-05 2024-09-11T09:52:08.629 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:08.629 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:08.629 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:08.629 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.629 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.629 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4688.0:49 2.5 2:b6dd9fc9:::rbd_data.1250f5e25b80.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:08.630 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:22.938548+0000", 2024-09-11T09:52:08.630 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 45.618538776000001, 2024-09-11T09:52:08.630 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.022766950000000001, 2024-09-11T09:52:08.630 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:08.630 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:08.630 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:08.631 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:52:08.631 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4688", 2024-09-11T09:52:08.632 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:52:08.632 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 49 2024-09-11T09:52:08.632 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.632 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:08.632 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.632 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:08.632 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.938548+0000", 2024-09-11T09:52:08.632 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:08.632 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.632 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.632 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:08.632 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.938548+0000", 2024-09-11T09:52:08.632 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:08.633 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.633 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.633 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:08.633 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.938550+0000", 2024-09-11T09:52:08.633 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.7710000000000001e-06 2024-09-11T09:52:08.633 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.633 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.633 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:08.633 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.944237+0000", 2024-09-11T09:52:08.633 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0056879790000000001 2024-09-11T09:52:08.633 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.633 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.633 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:08.634 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.944262+0000", 2024-09-11T09:52:08.634 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.4247e-05 2024-09-11T09:52:08.634 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.634 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.634 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:08.634 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.944274+0000", 2024-09-11T09:52:08.634 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.2639999999999999e-05 2024-09-11T09:52:08.634 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.634 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.634 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:08.634 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.944344+0000", 2024-09-11T09:52:08.634 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.9574999999999997e-05 2024-09-11T09:52:08.634 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.635 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.635 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:08.635 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.944474+0000", 2024-09-11T09:52:08.635 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000130464 2024-09-11T09:52:08.635 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.635 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.635 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:52:08.635 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.944585+0000", 2024-09-11T09:52:08.635 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000110096 2024-09-11T09:52:08.635 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.635 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.635 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:52:08.635 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.952724+0000", 2024-09-11T09:52:08.636 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0081398900000000003 2024-09-11T09:52:08.636 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.636 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.636 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:08.636 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.961219+0000", 2024-09-11T09:52:08.636 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0084949789999999997 2024-09-11T09:52:08.636 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.636 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.636 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:08.636 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.961280+0000", 2024-09-11T09:52:08.636 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.0404e-05 2024-09-11T09:52:08.636 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.637 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.637 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:08.637 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.961315+0000", 2024-09-11T09:52:08.637 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.4904999999999999e-05 2024-09-11T09:52:08.637 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:08.637 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:08.637 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:08.637 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.637 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.637 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_repop(client.4688.0:44 2.6 e53/15 2:67246a22:::rbd_data.1250f5e25b80.0000000000000007:head v 53'4, mlcod=53'2)", 2024-09-11T09:52:08.637 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:22.951721+0000", 2024-09-11T09:52:08.637 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 45.605365503999998, 2024-09-11T09:52:08.637 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.01080981, 2024-09-11T09:52:08.638 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:08.638 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:08.638 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:08.638 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:08.638 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.638 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:08.638 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.951721+0000", 2024-09-11T09:52:08.638 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:08.638 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.638 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.638 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:08.638 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.951721+0000", 2024-09-11T09:52:08.639 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:08.639 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.639 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.639 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:08.639 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.951722+0000", 2024-09-11T09:52:08.639 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.0279999999999999e-06 2024-09-11T09:52:08.639 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.639 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.639 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:08.639 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.955612+0000", 2024-09-11T09:52:08.640 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0038896920000000001 2024-09-11T09:52:08.640 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.640 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.640 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:08.640 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.955628+0000", 2024-09-11T09:52:08.640 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.5945000000000001e-05 2024-09-11T09:52:08.640 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.640 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.640 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:08.640 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.955635+0000", 2024-09-11T09:52:08.640 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.8080000000000005e-06 2024-09-11T09:52:08.640 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.641 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.641 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:08.641 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.955700+0000", 2024-09-11T09:52:08.641 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.4114000000000003e-05 2024-09-11T09:52:08.641 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.641 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.641 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:08.641 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.955724+0000", 2024-09-11T09:52:08.641 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.3989999999999999e-05 2024-09-11T09:52:08.641 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.641 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.641 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:08.641 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.962490+0000", 2024-09-11T09:52:08.642 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0067662599999999996 2024-09-11T09:52:08.642 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.642 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.642 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:08.642 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.962531+0000", 2024-09-11T09:52:08.642 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.0973000000000002e-05 2024-09-11T09:52:08.643 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:08.643 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:08.643 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:08.643 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.643 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.643 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_repop(client.4688.0:51 2.6 e53/15 2:6b5894fc:::rbd_data.1250f5e25b80.000000000000000a:head v 53'5, mlcod=53'2)", 2024-09-11T09:52:08.643 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:22.959030+0000", 2024-09-11T09:52:08.643 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 45.598056329999999, 2024-09-11T09:52:08.643 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.011503638, 2024-09-11T09:52:08.643 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:08.643 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:08.643 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:08.644 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:08.644 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.644 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:08.644 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.959030+0000", 2024-09-11T09:52:08.644 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:08.644 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.644 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.644 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:08.644 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.959030+0000", 2024-09-11T09:52:08.644 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:08.644 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.644 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.644 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:08.645 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.959032+0000", 2024-09-11T09:52:08.645 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.356e-06 2024-09-11T09:52:08.645 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.645 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.645 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:08.645 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.962899+0000", 2024-09-11T09:52:08.645 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.003867639 2024-09-11T09:52:08.645 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.645 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.645 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:08.645 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.962909+0000", 2024-09-11T09:52:08.646 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.0105999999999999e-05 2024-09-11T09:52:08.646 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.646 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.646 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:08.646 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.962915+0000", 2024-09-11T09:52:08.646 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.8869999999999997e-06 2024-09-11T09:52:08.646 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.646 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.646 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:08.646 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.962962+0000", 2024-09-11T09:52:08.646 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.6511999999999998e-05 2024-09-11T09:52:08.646 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.647 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.647 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:08.647 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.962989+0000", 2024-09-11T09:52:08.647 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.7702999999999999e-05 2024-09-11T09:52:08.647 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.647 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.647 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:08.647 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.970495+0000", 2024-09-11T09:52:08.647 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0075052449999999998 2024-09-11T09:52:08.647 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.647 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.647 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:08.647 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.970534+0000", 2024-09-11T09:52:08.647 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.9190000000000003e-05 2024-09-11T09:52:08.648 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:08.648 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:08.648 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:08.648 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.648 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.648 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_repop(client.4688.0:58 2.6 e53/15 2:6277c644:::rbd_data.1250f5e25b80.000000000000000e:head v 53'6, mlcod=53'4)", 2024-09-11T09:52:08.648 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:22.996199+0000", 2024-09-11T09:52:08.648 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 45.560887409999999, 2024-09-11T09:52:08.648 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.010145573999999999, 2024-09-11T09:52:08.648 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:08.648 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:08.648 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:08.648 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:08.649 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.649 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:08.649 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.996199+0000", 2024-09-11T09:52:08.649 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:08.649 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.649 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.649 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:08.649 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.996199+0000", 2024-09-11T09:52:08.649 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:08.649 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.649 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.649 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:08.649 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.996200+0000", 2024-09-11T09:52:08.650 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.85e-07 2024-09-11T09:52:08.650 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.650 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.650 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:08.650 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.999914+0000", 2024-09-11T09:52:08.650 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.003714406 2024-09-11T09:52:08.650 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.650 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.650 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:08.650 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.999928+0000", 2024-09-11T09:52:08.650 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.3149000000000001e-05 2024-09-11T09:52:08.650 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.650 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.651 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:08.651 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.999951+0000", 2024-09-11T09:52:08.651 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.3116e-05 2024-09-11T09:52:08.651 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.651 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.651 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:08.651 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.000003+0000", 2024-09-11T09:52:08.651 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.2169000000000001e-05 2024-09-11T09:52:08.651 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.651 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.651 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:08.651 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.000018+0000", 2024-09-11T09:52:08.651 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.5500000000000001e-05 2024-09-11T09:52:08.652 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.652 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.652 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:08.652 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.006303+0000", 2024-09-11T09:52:08.652 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0062846710000000004 2024-09-11T09:52:08.652 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.652 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.652 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:08.652 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.006345+0000", 2024-09-11T09:52:08.652 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.1678000000000003e-05 2024-09-11T09:52:08.652 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:08.652 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:08.653 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:08.653 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.653 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.653 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4694.0:42 2.5 2:a4fc00ec:ns::rbd_data.1256a0c02879.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:08.653 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.111570+0000", 2024-09-11T09:52:08.653 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 45.445516304000002, 2024-09-11T09:52:08.653 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.038002197000000001, 2024-09-11T09:52:08.653 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:08.653 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:08.653 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:08.653 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:52:08.653 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4694", 2024-09-11T09:52:08.653 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:52:08.654 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 42 2024-09-11T09:52:08.654 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.654 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:08.654 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.654 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:08.654 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.111570+0000", 2024-09-11T09:52:08.654 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:08.654 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.654 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.654 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:08.655 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.111570+0000", 2024-09-11T09:52:08.655 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:08.655 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.655 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.655 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:08.655 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.111572+0000", 2024-09-11T09:52:08.655 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.3909999999999999e-06 2024-09-11T09:52:08.655 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.655 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.655 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:08.655 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.135929+0000", 2024-09-11T09:52:08.655 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.024357410999999999 2024-09-11T09:52:08.655 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.656 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.656 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:08.656 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.135951+0000", 2024-09-11T09:52:08.656 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.1775000000000001e-05 2024-09-11T09:52:08.656 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.656 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.656 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:08.656 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.135962+0000", 2024-09-11T09:52:08.656 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.0903e-05 2024-09-11T09:52:08.656 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.656 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.657 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:08.657 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.136045+0000", 2024-09-11T09:52:08.657 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.3128999999999996e-05 2024-09-11T09:52:08.657 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.657 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.657 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:08.657 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.136229+0000", 2024-09-11T09:52:08.657 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00018393700000000001 2024-09-11T09:52:08.657 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.657 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.657 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:52:08.657 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.136394+0000", 2024-09-11T09:52:08.657 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000165527 2024-09-11T09:52:08.658 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.658 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.658 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:08.658 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.149320+0000", 2024-09-11T09:52:08.658 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.01292568 2024-09-11T09:52:08.658 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.658 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.658 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:52:08.658 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.149493+0000", 2024-09-11T09:52:08.658 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00017285999999999999 2024-09-11T09:52:08.659 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.659 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.659 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:08.659 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.149539+0000", 2024-09-11T09:52:08.659 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.6628000000000001e-05 2024-09-11T09:52:08.659 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.659 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.659 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:08.659 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.149572+0000", 2024-09-11T09:52:08.659 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.2956000000000001e-05 2024-09-11T09:52:08.659 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:08.659 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:08.659 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:08.660 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.660 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.660 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_repop(client.4694.0:45 2.6 e53/15 2:671fb830:ns::rbd_data.1256a0c02879.0000000000000007:head v 53'9, mlcod=53'7)", 2024-09-11T09:52:08.660 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.121833+0000", 2024-09-11T09:52:08.660 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 45.435253946000003, 2024-09-11T09:52:08.660 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.023876192000000001, 2024-09-11T09:52:08.660 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:08.660 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:08.660 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:08.660 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:08.660 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.660 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:08.661 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.121833+0000", 2024-09-11T09:52:08.661 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:08.661 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.661 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.661 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:08.661 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.121833+0000", 2024-09-11T09:52:08.661 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:08.661 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.661 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.661 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:08.661 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.121834+0000", 2024-09-11T09:52:08.661 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.471e-06 2024-09-11T09:52:08.661 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.662 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.662 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:08.662 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.128990+0000", 2024-09-11T09:52:08.662 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0071557879999999997 2024-09-11T09:52:08.662 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.662 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.662 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:08.662 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.129011+0000", 2024-09-11T09:52:08.662 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.0760000000000001e-05 2024-09-11T09:52:08.662 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.662 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.662 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:08.662 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.129022+0000", 2024-09-11T09:52:08.663 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1056000000000001e-05 2024-09-11T09:52:08.663 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.663 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.663 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:08.663 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.129097+0000", 2024-09-11T09:52:08.663 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.5082999999999999e-05 2024-09-11T09:52:08.663 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.663 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.663 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:08.663 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.129130+0000", 2024-09-11T09:52:08.663 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.3345e-05 2024-09-11T09:52:08.664 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.664 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.664 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:08.664 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.145657+0000", 2024-09-11T09:52:08.664 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.016527224 2024-09-11T09:52:08.664 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.664 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.664 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:08.664 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.145709+0000", 2024-09-11T09:52:08.664 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.1465000000000003e-05 2024-09-11T09:52:08.665 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:08.665 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:08.665 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:08.665 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.665 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.665 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4694.0:43 2.0 2:16c8ae84:ns::rbd_data.1256a0c02879.0000000000000005:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:08.665 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.135999+0000", 2024-09-11T09:52:08.665 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 45.421087995999997, 2024-09-11T09:52:08.665 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.039468309, 2024-09-11T09:52:08.665 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:08.665 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:08.665 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:08.666 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:52:08.666 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4694", 2024-09-11T09:52:08.666 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:52:08.666 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 43 2024-09-11T09:52:08.666 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.666 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:08.666 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.666 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:08.666 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.135999+0000", 2024-09-11T09:52:08.666 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:08.666 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.666 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.666 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:08.667 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.135999+0000", 2024-09-11T09:52:08.667 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:08.667 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.667 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.667 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:08.667 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.136000+0000", 2024-09-11T09:52:08.667 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.6330000000000001e-06 2024-09-11T09:52:08.667 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.667 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.667 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:08.667 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.157961+0000", 2024-09-11T09:52:08.667 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0219606 2024-09-11T09:52:08.667 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.668 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.668 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:08.668 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.157995+0000", 2024-09-11T09:52:08.668 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.4391999999999999e-05 2024-09-11T09:52:08.668 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.668 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.668 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:08.668 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.158011+0000", 2024-09-11T09:52:08.668 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.5794000000000002e-05 2024-09-11T09:52:08.668 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.668 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.668 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:08.668 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.158083+0000", 2024-09-11T09:52:08.669 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.2273000000000004e-05 2024-09-11T09:52:08.669 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.669 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.669 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:08.669 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.158239+0000", 2024-09-11T09:52:08.669 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000155372 2024-09-11T09:52:08.669 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.669 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.669 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 1", 2024-09-11T09:52:08.669 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.158403+0000", 2024-09-11T09:52:08.669 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00016408199999999999 2024-09-11T09:52:08.670 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.670 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.670 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:52:08.670 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.173860+0000", 2024-09-11T09:52:08.670 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.015457347 2024-09-11T09:52:08.670 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.670 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.670 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:08.670 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.175400+0000", 2024-09-11T09:52:08.670 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001540006 2024-09-11T09:52:08.670 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.670 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.671 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:08.671 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.175438+0000", 2024-09-11T09:52:08.671 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.8135e-05 2024-09-11T09:52:08.671 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.671 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.671 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:08.671 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.175467+0000", 2024-09-11T09:52:08.671 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.8674999999999999e-05 2024-09-11T09:52:08.671 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:08.671 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:08.671 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:08.671 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.671 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.672 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4694.0:57 2.0 2:04b5549c:ns::rbd_data.1256a0c02879.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:08.672 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.158037+0000", 2024-09-11T09:52:08.672 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 45.399049257999998, 2024-09-11T09:52:08.672 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.027748790999999998, 2024-09-11T09:52:08.672 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:08.672 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:08.672 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:08.672 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:52:08.672 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4694", 2024-09-11T09:52:08.673 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:52:08.673 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 57 2024-09-11T09:52:08.673 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.673 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:08.673 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.673 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:08.673 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.158037+0000", 2024-09-11T09:52:08.673 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:08.673 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.673 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.673 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:08.674 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.158037+0000", 2024-09-11T09:52:08.674 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:08.674 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.674 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.674 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:08.674 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.158039+0000", 2024-09-11T09:52:08.674 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.356e-06 2024-09-11T09:52:08.674 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.674 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.674 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:08.674 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.176367+0000", 2024-09-11T09:52:08.674 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.018328739 2024-09-11T09:52:08.674 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.675 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.675 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:08.675 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.176378+0000", 2024-09-11T09:52:08.675 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.0713999999999999e-05 2024-09-11T09:52:08.675 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.675 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.675 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:08.675 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.176383+0000", 2024-09-11T09:52:08.675 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.0209999999999999e-06 2024-09-11T09:52:08.675 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.675 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.675 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:08.675 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.176426+0000", 2024-09-11T09:52:08.676 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.2623999999999997e-05 2024-09-11T09:52:08.676 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.676 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.676 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:08.676 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.176552+0000", 2024-09-11T09:52:08.676 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000126076 2024-09-11T09:52:08.676 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.676 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.676 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 1", 2024-09-11T09:52:08.676 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.176665+0000", 2024-09-11T09:52:08.676 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000112995 2024-09-11T09:52:08.676 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.677 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.677 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:52:08.677 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.183639+0000", 2024-09-11T09:52:08.677 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0069740039999999998 2024-09-11T09:52:08.677 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.677 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.677 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:08.677 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.185689+0000", 2024-09-11T09:52:08.677 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.002049852 2024-09-11T09:52:08.677 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.677 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.677 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:08.677 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.185725+0000", 2024-09-11T09:52:08.678 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.6804999999999997e-05 2024-09-11T09:52:08.678 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.678 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.678 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:08.678 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.185786+0000", 2024-09-11T09:52:08.678 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.0605e-05 2024-09-11T09:52:08.678 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:08.678 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:08.678 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:08.678 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.678 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.679 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4694.0:60 2.0 2:054f2af4:ns::rbd_data.1256a0c02879.000000000000000f:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:08.679 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.186855+0000", 2024-09-11T09:52:08.679 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 45.370231242000003, 2024-09-11T09:52:08.679 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.013682913, 2024-09-11T09:52:08.679 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:08.679 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:08.679 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:08.679 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:52:08.679 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4694", 2024-09-11T09:52:08.679 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:52:08.679 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 60 2024-09-11T09:52:08.679 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.679 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:08.679 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.680 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:08.680 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.186855+0000", 2024-09-11T09:52:08.680 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:08.680 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.680 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.680 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:08.680 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.186855+0000", 2024-09-11T09:52:08.680 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:08.680 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.680 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.680 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:08.680 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.186856+0000", 2024-09-11T09:52:08.680 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.1200000000000001e-07 2024-09-11T09:52:08.681 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.681 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.681 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:08.681 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.191484+0000", 2024-09-11T09:52:08.681 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0046274159999999996 2024-09-11T09:52:08.681 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.681 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.681 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:08.681 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.191496+0000", 2024-09-11T09:52:08.681 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.2149e-05 2024-09-11T09:52:08.681 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.681 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.681 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:08.682 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.191502+0000", 2024-09-11T09:52:08.682 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.1890000000000004e-06 2024-09-11T09:52:08.682 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.682 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.682 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:08.682 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.191544+0000", 2024-09-11T09:52:08.682 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.1724999999999998e-05 2024-09-11T09:52:08.682 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.682 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.682 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:08.682 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.191641+0000", 2024-09-11T09:52:08.682 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.6939000000000001e-05 2024-09-11T09:52:08.682 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.683 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.683 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 1", 2024-09-11T09:52:08.683 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.191731+0000", 2024-09-11T09:52:08.683 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.0651999999999999e-05 2024-09-11T09:52:08.683 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.683 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.683 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:52:08.684 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.199852+0000", 2024-09-11T09:52:08.684 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0081204529999999997 2024-09-11T09:52:08.684 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.684 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.684 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:08.684 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.200461+0000", 2024-09-11T09:52:08.685 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00060900899999999998 2024-09-11T09:52:08.685 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.685 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.685 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:08.685 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.200497+0000", 2024-09-11T09:52:08.686 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.5800000000000003e-05 2024-09-11T09:52:08.686 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:08.686 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:08.686 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:08.686 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.200538+0000", 2024-09-11T09:52:08.686 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.1668999999999997e-05 2024-09-11T09:52:08.686 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:08.687 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:08.687 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:08.687 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:08.687 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:08.687 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-11T09:52:08.689 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:52:08.690+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:52:08.690 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-11T09:52:08.789 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:52:08.790+0000 7f3a7b90a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 42475) UID: 0 2024-09-11T09:52:08.793 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-11T09:52:08.793 INFO:teuthology.orchestra.run.smithi096.stdout: "ops": [], 2024-09-11T09:52:08.794 INFO:teuthology.orchestra.run.smithi096.stdout: "num_ops": 0 2024-09-11T09:52:08.794 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-11T09:52:08.803 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-11T09:52:08.854 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-11T09:52:08.898 INFO:tasks.ceph.osd.2.smithi017.stderr:2024-09-11T09:52:08.899+0000 7fd20b170640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31672) UID: 0 2024-09-11T09:52:08.948 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-11T09:52:08.948 INFO:teuthology.orchestra.run.smithi096.stdout: "ops": [], 2024-09-11T09:52:08.949 INFO:teuthology.orchestra.run.smithi096.stdout: "complaint_time": 30, 2024-09-11T09:52:08.949 INFO:teuthology.orchestra.run.smithi096.stdout: "num_blocked_ops": 0 2024-09-11T09:52:08.949 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-11T09:52:08.960 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-11T09:52:09.003 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:52:09.004+0000 7f3a7b90a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 42475) UID: 0 2024-09-11T09:52:09.067 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-11T09:52:09.068 INFO:teuthology.orchestra.run.smithi096.stdout: "size": 20, 2024-09-11T09:52:09.068 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 600, 2024-09-11T09:52:09.068 INFO:teuthology.orchestra.run.smithi096.stdout: "ops": [ 2024-09-11T09:52:09.068 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.068 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4230.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-11T09:52:09.068 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:18.490736+0000", 2024-09-11T09:52:09.068 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 110.57652267899999, 2024-09-11T09:52:09.068 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0059097569999999999, 2024-09-11T09:52:09.068 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:09.069 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:09.069 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:09.069 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:52:09.069 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4230", 2024-09-11T09:52:09.069 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.17:0/4101109576", 2024-09-11T09:52:09.069 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 1 2024-09-11T09:52:09.069 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.069 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:09.069 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.069 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:09.069 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.490736+0000", 2024-09-11T09:52:09.069 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:09.069 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.070 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.070 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:09.070 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.490736+0000", 2024-09-11T09:52:09.070 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:09.070 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.070 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.070 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:09.070 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.490742+0000", 2024-09-11T09:52:09.070 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.48e-06 2024-09-11T09:52:09.070 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.070 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.070 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:09.070 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.490749+0000", 2024-09-11T09:52:09.070 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.4569999999999999e-06 2024-09-11T09:52:09.071 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.071 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.071 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:09.071 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.490762+0000", 2024-09-11T09:52:09.071 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.2721000000000001e-05 2024-09-11T09:52:09.071 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.071 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.071 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:09.071 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.490774+0000", 2024-09-11T09:52:09.071 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1823e-05 2024-09-11T09:52:09.071 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.071 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.071 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:09.072 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.490846+0000", 2024-09-11T09:52:09.072 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.1878000000000005e-05 2024-09-11T09:52:09.072 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.072 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.072 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for peered", 2024-09-11T09:52:09.072 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.490857+0000", 2024-09-11T09:52:09.072 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1017e-05 2024-09-11T09:52:09.072 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.072 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.072 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:09.072 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.493856+0000", 2024-09-11T09:52:09.072 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00299885 2024-09-11T09:52:09.072 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.073 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.073 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:09.073 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.494020+0000", 2024-09-11T09:52:09.073 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00016436599999999999 2024-09-11T09:52:09.073 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.073 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.073 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 2", 2024-09-11T09:52:09.073 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.494186+0000", 2024-09-11T09:52:09.073 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00016562799999999999 2024-09-11T09:52:09.073 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.073 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.073 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:09.073 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.495407+0000", 2024-09-11T09:52:09.074 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0012217420000000001 2024-09-11T09:52:09.074 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.074 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.074 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:52:09.074 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.496579+0000", 2024-09-11T09:52:09.074 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001171844 2024-09-11T09:52:09.074 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.074 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.074 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:09.074 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.496620+0000", 2024-09-11T09:52:09.074 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.0479000000000003e-05 2024-09-11T09:52:09.074 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.075 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.075 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:09.075 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.496646+0000", 2024-09-11T09:52:09.075 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.6472e-05 2024-09-11T09:52:09.075 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:09.075 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:09.075 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:09.075 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.075 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.075 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4230.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-09-11T09:52:09.075 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:20.498634+0000", 2024-09-11T09:52:09.075 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 108.568624658, 2024-09-11T09:52:09.075 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0050648539999999997, 2024-09-11T09:52:09.076 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:09.076 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:09.076 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:09.076 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:52:09.076 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4230", 2024-09-11T09:52:09.076 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.17:0/4101109576", 2024-09-11T09:52:09.076 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 7 2024-09-11T09:52:09.076 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.076 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:09.076 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.076 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:09.076 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.498634+0000", 2024-09-11T09:52:09.076 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:09.076 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.077 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.077 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:09.077 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.498634+0000", 2024-09-11T09:52:09.077 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:09.077 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.077 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.077 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:09.077 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.498637+0000", 2024-09-11T09:52:09.077 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.1919999999999999e-06 2024-09-11T09:52:09.077 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.077 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.077 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:09.077 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.498648+0000", 2024-09-11T09:52:09.078 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1116e-05 2024-09-11T09:52:09.078 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.078 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.078 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:09.078 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.498666+0000", 2024-09-11T09:52:09.078 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.8411999999999999e-05 2024-09-11T09:52:09.078 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.078 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.078 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:09.078 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.498675+0000", 2024-09-11T09:52:09.078 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.4059999999999994e-06 2024-09-11T09:52:09.078 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.078 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.079 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:09.079 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.498745+0000", 2024-09-11T09:52:09.079 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.9764000000000005e-05 2024-09-11T09:52:09.079 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.079 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.079 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op must wait for map", 2024-09-11T09:52:09.079 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.498754+0000", 2024-09-11T09:52:09.079 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.5969999999999992e-06 2024-09-11T09:52:09.079 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.079 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.079 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:09.079 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.500385+0000", 2024-09-11T09:52:09.079 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0016308360000000001 2024-09-11T09:52:09.079 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.080 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.080 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:09.080 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.500473+0000", 2024-09-11T09:52:09.080 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.7966000000000004e-05 2024-09-11T09:52:09.080 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.080 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.080 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 1", 2024-09-11T09:52:09.080 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.500591+0000", 2024-09-11T09:52:09.080 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000117895 2024-09-11T09:52:09.080 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.080 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.080 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:09.080 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.501685+0000", 2024-09-11T09:52:09.080 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001094886 2024-09-11T09:52:09.080 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.081 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.081 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:52:09.081 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.503627+0000", 2024-09-11T09:52:09.081 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001941199 2024-09-11T09:52:09.081 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.081 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.081 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:09.081 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.503667+0000", 2024-09-11T09:52:09.081 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.0645000000000001e-05 2024-09-11T09:52:09.081 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.081 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.081 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:09.081 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.503699+0000", 2024-09-11T09:52:09.081 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.1940000000000003e-05 2024-09-11T09:52:09.082 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:09.082 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:09.082 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:09.082 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.082 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.082 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4688.0:32 2.2 2:45941fd2:::rbd_data.1250f5e25b80.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:09.082 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:22.896607+0000", 2024-09-11T09:52:09.082 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 46.170652054999998, 2024-09-11T09:52:09.082 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.038655179999999997, 2024-09-11T09:52:09.082 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:09.082 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:09.082 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:09.082 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:52:09.082 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4688", 2024-09-11T09:52:09.083 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:52:09.083 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 32 2024-09-11T09:52:09.083 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.083 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:09.083 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.083 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:09.083 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.896607+0000", 2024-09-11T09:52:09.083 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:09.083 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.083 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.083 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:09.083 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.896607+0000", 2024-09-11T09:52:09.083 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:09.083 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.084 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.084 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:09.084 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.896608+0000", 2024-09-11T09:52:09.084 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.0690000000000001e-06 2024-09-11T09:52:09.084 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.084 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.084 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:09.084 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.914461+0000", 2024-09-11T09:52:09.084 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.017852903 2024-09-11T09:52:09.084 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.084 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.084 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:09.084 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.914495+0000", 2024-09-11T09:52:09.084 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.4301000000000002e-05 2024-09-11T09:52:09.085 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.085 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.085 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:09.085 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.914522+0000", 2024-09-11T09:52:09.085 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.7127e-05 2024-09-11T09:52:09.085 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.085 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.085 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:09.085 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.914583+0000", 2024-09-11T09:52:09.085 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.0730999999999998e-05 2024-09-11T09:52:09.085 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.085 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.085 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:09.086 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.914709+0000", 2024-09-11T09:52:09.086 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00012590400000000001 2024-09-11T09:52:09.086 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.086 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.086 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 1", 2024-09-11T09:52:09.086 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.914809+0000", 2024-09-11T09:52:09.086 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.9882000000000004e-05 2024-09-11T09:52:09.086 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.086 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.086 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:09.086 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.932551+0000", 2024-09-11T09:52:09.086 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.017742048999999999 2024-09-11T09:52:09.086 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.086 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.087 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:52:09.087 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.935144+0000", 2024-09-11T09:52:09.087 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0025927630000000001 2024-09-11T09:52:09.087 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.087 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.087 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:09.087 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.935205+0000", 2024-09-11T09:52:09.087 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.1634e-05 2024-09-11T09:52:09.087 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.087 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.087 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:09.087 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.935262+0000", 2024-09-11T09:52:09.087 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.6817000000000002e-05 2024-09-11T09:52:09.087 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:09.088 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:09.088 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:09.088 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.088 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.088 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_repop(client.4688.0:40 2.7 e53/38 2:fe41411c:::rbd_object_map.1250f5e25b80:head v 53'9, mlcod=53'7)", 2024-09-11T09:52:09.088 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:22.920457+0000", 2024-09-11T09:52:09.088 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 46.146801963999998, 2024-09-11T09:52:09.088 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.011132817, 2024-09-11T09:52:09.088 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:09.088 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:09.088 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:09.088 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:09.089 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.089 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:09.089 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.920457+0000", 2024-09-11T09:52:09.089 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:09.089 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.089 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.089 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:09.089 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.920457+0000", 2024-09-11T09:52:09.089 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:09.089 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.089 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.089 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:09.089 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.920459+0000", 2024-09-11T09:52:09.090 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.6729999999999999e-06 2024-09-11T09:52:09.090 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.090 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.090 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:09.090 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.920469+0000", 2024-09-11T09:52:09.090 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.944e-06 2024-09-11T09:52:09.090 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.090 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.090 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:09.090 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.920495+0000", 2024-09-11T09:52:09.090 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.6401999999999999e-05 2024-09-11T09:52:09.090 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.090 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.091 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:09.091 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.920506+0000", 2024-09-11T09:52:09.091 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.133e-05 2024-09-11T09:52:09.091 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.091 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.091 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:09.091 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.920581+0000", 2024-09-11T09:52:09.091 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.4932999999999996e-05 2024-09-11T09:52:09.091 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.091 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.091 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:09.091 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.920612+0000", 2024-09-11T09:52:09.091 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.0537999999999998e-05 2024-09-11T09:52:09.091 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.092 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.092 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:09.092 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.931537+0000", 2024-09-11T09:52:09.092 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.010925239 2024-09-11T09:52:09.092 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.092 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.092 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:09.092 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.931590+0000", 2024-09-11T09:52:09.092 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.2757999999999999e-05 2024-09-11T09:52:09.092 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:09.092 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:09.092 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:09.092 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.092 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.093 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_repop(client.4688.0:41 2.7 e53/38 2:fe41411c:::rbd_object_map.1250f5e25b80:head v 53'10, mlcod=53'7)", 2024-09-11T09:52:09.093 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:22.921422+0000", 2024-09-11T09:52:09.093 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 46.145836547999998, 2024-09-11T09:52:09.093 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.013734942, 2024-09-11T09:52:09.093 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:09.093 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:09.093 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:09.093 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:09.093 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.093 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:09.093 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921422+0000", 2024-09-11T09:52:09.093 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:09.093 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.094 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.094 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:09.094 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921422+0000", 2024-09-11T09:52:09.094 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:09.094 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.094 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.094 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:09.094 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921426+0000", 2024-09-11T09:52:09.094 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.6150000000000001e-06 2024-09-11T09:52:09.094 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.094 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.094 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:09.094 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921435+0000", 2024-09-11T09:52:09.094 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.9840000000000007e-06 2024-09-11T09:52:09.095 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.095 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.095 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:09.095 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921444+0000", 2024-09-11T09:52:09.095 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.6000000000000007e-06 2024-09-11T09:52:09.095 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.095 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.095 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:09.095 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921450+0000", 2024-09-11T09:52:09.095 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.4690000000000002e-06 2024-09-11T09:52:09.095 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.095 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.095 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:09.095 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921511+0000", 2024-09-11T09:52:09.096 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.0779999999999997e-05 2024-09-11T09:52:09.096 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.096 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.096 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:09.096 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921531+0000", 2024-09-11T09:52:09.096 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.0302999999999999e-05 2024-09-11T09:52:09.096 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.096 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.096 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:09.096 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.935115+0000", 2024-09-11T09:52:09.096 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.013583471 2024-09-11T09:52:09.096 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.096 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.097 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:09.097 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.935157+0000", 2024-09-11T09:52:09.097 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.2719999999999998e-05 2024-09-11T09:52:09.097 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:09.097 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:09.097 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:09.097 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.097 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.097 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4688.0:42 2.3 2:c54742d8:::rbd_data.1250f5e25b80.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:09.097 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:22.932560+0000", 2024-09-11T09:52:09.097 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 46.134698804999999, 2024-09-11T09:52:09.097 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.017798198000000001, 2024-09-11T09:52:09.098 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:09.098 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:09.098 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:09.098 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:52:09.098 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4688", 2024-09-11T09:52:09.098 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:52:09.098 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 42 2024-09-11T09:52:09.098 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.098 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:09.098 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.098 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:09.098 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.932560+0000", 2024-09-11T09:52:09.098 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:09.098 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.098 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.099 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:09.099 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.932560+0000", 2024-09-11T09:52:09.099 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:09.099 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.099 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.099 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:09.099 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.932562+0000", 2024-09-11T09:52:09.099 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.17e-06 2024-09-11T09:52:09.099 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.099 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.099 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:09.099 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.938851+0000", 2024-09-11T09:52:09.099 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.006288971 2024-09-11T09:52:09.099 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.100 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.100 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:09.100 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.938865+0000", 2024-09-11T09:52:09.100 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.365e-05 2024-09-11T09:52:09.100 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.100 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.100 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:09.100 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.938871+0000", 2024-09-11T09:52:09.100 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.8850000000000004e-06 2024-09-11T09:52:09.100 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.100 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.100 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:09.100 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.938926+0000", 2024-09-11T09:52:09.100 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.4691999999999997e-05 2024-09-11T09:52:09.100 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.101 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.101 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:09.101 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.939059+0000", 2024-09-11T09:52:09.101 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00013310899999999999 2024-09-11T09:52:09.101 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.101 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.101 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 2", 2024-09-11T09:52:09.101 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.939171+0000", 2024-09-11T09:52:09.101 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000112718 2024-09-11T09:52:09.101 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.101 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.101 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:09.101 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.950262+0000", 2024-09-11T09:52:09.101 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.011091063 2024-09-11T09:52:09.101 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.102 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.102 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:52:09.102 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.950274+0000", 2024-09-11T09:52:09.102 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1963e-05 2024-09-11T09:52:09.102 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.102 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.102 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:09.102 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.950318+0000", 2024-09-11T09:52:09.102 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.3519999999999997e-05 2024-09-11T09:52:09.102 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.102 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.102 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:09.102 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.950358+0000", 2024-09-11T09:52:09.102 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.0457000000000002e-05 2024-09-11T09:52:09.102 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:09.103 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:09.103 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:09.103 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.103 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.103 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_repop(client.4694.0:34 2.7 e53/38 2:f80bfc0c:ns::rbd_data.1256a0c02879.0000000000000002:head v 53'25, mlcod=53'23)", 2024-09-11T09:52:09.103 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.094944+0000", 2024-09-11T09:52:09.103 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 45.972315412999997, 2024-09-11T09:52:09.103 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.010896785000000001, 2024-09-11T09:52:09.103 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:09.103 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:09.103 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:09.103 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:09.103 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.103 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:09.104 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.094944+0000", 2024-09-11T09:52:09.104 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:09.104 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.104 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.104 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:09.104 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.094944+0000", 2024-09-11T09:52:09.104 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:09.104 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.104 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.104 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:09.104 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.094944+0000", 2024-09-11T09:52:09.104 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.0699999999999996e-07 2024-09-11T09:52:09.104 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.104 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.105 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:09.105 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.099060+0000", 2024-09-11T09:52:09.105 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0041155580000000001 2024-09-11T09:52:09.105 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.105 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.105 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:09.105 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.099082+0000", 2024-09-11T09:52:09.105 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.2323999999999998e-05 2024-09-11T09:52:09.105 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.105 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.105 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:09.105 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.099094+0000", 2024-09-11T09:52:09.105 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1535000000000001e-05 2024-09-11T09:52:09.106 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.106 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.106 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:09.106 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.099160+0000", 2024-09-11T09:52:09.106 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.6449e-05 2024-09-11T09:52:09.106 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.106 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.106 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:09.106 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.099183+0000", 2024-09-11T09:52:09.106 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.3108e-05 2024-09-11T09:52:09.106 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.106 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.106 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:09.106 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.105776+0000", 2024-09-11T09:52:09.107 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0065922760000000002 2024-09-11T09:52:09.107 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.107 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.107 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:09.107 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.105840+0000", 2024-09-11T09:52:09.107 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.4727999999999994e-05 2024-09-11T09:52:09.107 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:09.107 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:09.107 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:09.107 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.107 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.107 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_repop(client.4694.0:35 2.7 e53/38 2:e8766325:ns::rbd_object_map.1256a0c02879:head v 53'26, mlcod=53'23)", 2024-09-11T09:52:09.107 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.100964+0000", 2024-09-11T09:52:09.108 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 45.966294885000003, 2024-09-11T09:52:09.108 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0049649359999999997, 2024-09-11T09:52:09.108 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:09.108 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:09.108 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:09.108 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:09.108 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.108 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:09.108 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.100964+0000", 2024-09-11T09:52:09.108 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:09.108 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.108 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.108 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:09.109 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.100964+0000", 2024-09-11T09:52:09.109 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:09.109 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.109 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.109 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:09.109 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.100966+0000", 2024-09-11T09:52:09.109 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.6330000000000001e-06 2024-09-11T09:52:09.109 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.109 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.109 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:09.109 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.100979+0000", 2024-09-11T09:52:09.109 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.2813000000000001e-05 2024-09-11T09:52:09.109 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.109 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.110 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:09.110 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.100992+0000", 2024-09-11T09:52:09.110 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.3414999999999999e-05 2024-09-11T09:52:09.110 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.110 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.110 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:09.110 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.101002+0000", 2024-09-11T09:52:09.110 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.0159e-05 2024-09-11T09:52:09.110 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.110 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.110 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:09.110 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.102365+0000", 2024-09-11T09:52:09.111 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001362912 2024-09-11T09:52:09.111 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.111 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.111 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:09.111 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.102382+0000", 2024-09-11T09:52:09.111 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.7433000000000001e-05 2024-09-11T09:52:09.111 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.111 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.111 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:09.111 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.105901+0000", 2024-09-11T09:52:09.111 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.003518476 2024-09-11T09:52:09.111 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.111 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.111 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:09.111 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.105929+0000", 2024-09-11T09:52:09.112 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.8095000000000001e-05 2024-09-11T09:52:09.112 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:09.112 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:09.112 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:09.112 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.112 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.112 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4694.0:41 2.2 2:5e2505c4:ns::rbd_data.1256a0c02879.0000000000000003:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:09.112 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.107511+0000", 2024-09-11T09:52:09.112 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 45.959747731999997, 2024-09-11T09:52:09.112 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.018571066000000001, 2024-09-11T09:52:09.112 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:09.112 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:09.112 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:09.112 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:52:09.113 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4694", 2024-09-11T09:52:09.113 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:52:09.113 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 41 2024-09-11T09:52:09.113 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.113 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:09.113 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.113 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:09.113 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.107511+0000", 2024-09-11T09:52:09.113 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:09.113 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.113 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.113 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:09.113 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.107511+0000", 2024-09-11T09:52:09.113 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:09.113 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.114 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.114 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:09.114 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.107513+0000", 2024-09-11T09:52:09.114 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.3489999999999999e-06 2024-09-11T09:52:09.114 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.114 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.114 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:09.114 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.112704+0000", 2024-09-11T09:52:09.114 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0051918509999999999 2024-09-11T09:52:09.114 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.114 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.114 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:09.115 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.112718+0000", 2024-09-11T09:52:09.115 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.359e-05 2024-09-11T09:52:09.115 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.115 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.115 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:09.115 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.112725+0000", 2024-09-11T09:52:09.115 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.9940000000000003e-06 2024-09-11T09:52:09.115 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.115 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.115 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:09.115 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.112788+0000", 2024-09-11T09:52:09.115 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.2447999999999996e-05 2024-09-11T09:52:09.115 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.116 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.116 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:09.116 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.112937+0000", 2024-09-11T09:52:09.116 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00014921499999999999 2024-09-11T09:52:09.116 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.116 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.116 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 1", 2024-09-11T09:52:09.116 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.113085+0000", 2024-09-11T09:52:09.116 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000148085 2024-09-11T09:52:09.116 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.116 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.116 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:52:09.116 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.124318+0000", 2024-09-11T09:52:09.116 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.011233158 2024-09-11T09:52:09.117 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.117 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.117 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:09.117 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.125957+0000", 2024-09-11T09:52:09.117 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001638701 2024-09-11T09:52:09.117 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.117 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.117 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:09.117 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.126029+0000", 2024-09-11T09:52:09.117 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.2125000000000005e-05 2024-09-11T09:52:09.117 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.117 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.117 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:09.117 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.126082+0000", 2024-09-11T09:52:09.118 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.3550000000000001e-05 2024-09-11T09:52:09.118 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:09.118 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:09.118 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:09.118 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.118 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.118 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4694.0:44 2.3 2:cd1a7d1c:ns::rbd_data.1256a0c02879.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:09.118 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.112744+0000", 2024-09-11T09:52:09.118 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 45.954515164999997, 2024-09-11T09:52:09.118 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.021672212999999999, 2024-09-11T09:52:09.118 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:09.118 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:09.118 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:09.119 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:52:09.119 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4694", 2024-09-11T09:52:09.119 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:52:09.119 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 44 2024-09-11T09:52:09.119 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.119 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:09.119 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.119 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:09.119 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.112744+0000", 2024-09-11T09:52:09.119 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:09.119 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.119 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.119 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:09.119 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.112744+0000", 2024-09-11T09:52:09.120 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:09.120 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.120 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.120 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:09.120 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.112745+0000", 2024-09-11T09:52:09.120 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.2900000000000002e-07 2024-09-11T09:52:09.120 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.120 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.120 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:09.120 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.119929+0000", 2024-09-11T09:52:09.120 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0071846230000000002 2024-09-11T09:52:09.120 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.120 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.121 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:09.121 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.119949+0000", 2024-09-11T09:52:09.121 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.9264000000000001e-05 2024-09-11T09:52:09.121 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.121 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.121 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:09.121 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.119959+0000", 2024-09-11T09:52:09.121 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.0465999999999999e-05 2024-09-11T09:52:09.121 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.121 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.121 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:09.121 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.120028+0000", 2024-09-11T09:52:09.121 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.9318000000000003e-05 2024-09-11T09:52:09.122 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.122 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.122 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:09.122 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.120160+0000", 2024-09-11T09:52:09.122 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000132015 2024-09-11T09:52:09.122 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.122 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.122 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 2", 2024-09-11T09:52:09.122 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.120260+0000", 2024-09-11T09:52:09.122 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.9397999999999999e-05 2024-09-11T09:52:09.122 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.122 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.122 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:52:09.122 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.134246+0000", 2024-09-11T09:52:09.123 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.013986721000000001 2024-09-11T09:52:09.123 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.123 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.123 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:09.123 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.134295+0000", 2024-09-11T09:52:09.123 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.8649000000000001e-05 2024-09-11T09:52:09.123 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.123 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.123 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:09.123 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.134361+0000", 2024-09-11T09:52:09.123 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.6081999999999995e-05 2024-09-11T09:52:09.123 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.123 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.123 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:09.124 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.134416+0000", 2024-09-11T09:52:09.124 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.4848000000000001e-05 2024-09-11T09:52:09.124 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:09.124 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:09.124 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:09.124 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.124 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.124 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4694.0:46 2.3 2:ce0e1469:ns::rbd_data.1256a0c02879.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:09.124 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.119993+0000", 2024-09-11T09:52:09.124 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 45.947265688999998, 2024-09-11T09:52:09.125 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.025840610999999999, 2024-09-11T09:52:09.125 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:09.125 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:09.125 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:09.125 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:52:09.125 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4694", 2024-09-11T09:52:09.125 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:52:09.125 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 46 2024-09-11T09:52:09.125 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.125 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:09.125 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.125 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:09.125 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.119993+0000", 2024-09-11T09:52:09.126 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:09.126 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.126 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.126 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:09.126 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.119993+0000", 2024-09-11T09:52:09.126 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:09.126 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.126 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.126 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:09.126 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.119995+0000", 2024-09-11T09:52:09.126 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1930000000000001e-06 2024-09-11T09:52:09.126 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.126 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.126 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:09.127 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.128458+0000", 2024-09-11T09:52:09.127 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0084632630000000004 2024-09-11T09:52:09.127 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.127 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.127 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:09.127 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.128478+0000", 2024-09-11T09:52:09.127 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.0353e-05 2024-09-11T09:52:09.127 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.127 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.127 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:09.127 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.128491+0000", 2024-09-11T09:52:09.127 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.2954e-05 2024-09-11T09:52:09.127 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.128 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.128 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:09.128 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.128558+0000", 2024-09-11T09:52:09.128 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.7076999999999995e-05 2024-09-11T09:52:09.128 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.128 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.128 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:09.128 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.128708+0000", 2024-09-11T09:52:09.128 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00014939599999999999 2024-09-11T09:52:09.128 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.128 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.128 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 2", 2024-09-11T09:52:09.128 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.128866+0000", 2024-09-11T09:52:09.129 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000158145 2024-09-11T09:52:09.129 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.129 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.129 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:09.129 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.141045+0000", 2024-09-11T09:52:09.129 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.012179331999999999 2024-09-11T09:52:09.129 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.129 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.129 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:52:09.129 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.145742+0000", 2024-09-11T09:52:09.129 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0046972050000000003 2024-09-11T09:52:09.129 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.129 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.129 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:09.130 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.145779+0000", 2024-09-11T09:52:09.130 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.6544000000000001e-05 2024-09-11T09:52:09.130 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.130 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.130 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:09.130 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.145834+0000", 2024-09-11T09:52:09.130 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.5148999999999997e-05 2024-09-11T09:52:09.130 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:09.130 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:09.130 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:09.130 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.130 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.130 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4694.0:48 2.3 2:d2090b70:ns::rbd_data.1256a0c02879.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:09.130 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.130540+0000", 2024-09-11T09:52:09.131 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 45.936719423, 2024-09-11T09:52:09.131 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.017167789999999999, 2024-09-11T09:52:09.131 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:09.131 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:09.131 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:09.131 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:52:09.131 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4694", 2024-09-11T09:52:09.131 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:52:09.131 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 48 2024-09-11T09:52:09.131 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.131 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:09.131 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.131 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:09.132 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.130540+0000", 2024-09-11T09:52:09.132 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:09.132 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.132 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.132 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:09.132 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.130540+0000", 2024-09-11T09:52:09.132 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:09.132 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.132 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.132 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:09.132 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.130542+0000", 2024-09-11T09:52:09.132 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.9209999999999999e-06 2024-09-11T09:52:09.132 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.132 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.133 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:09.133 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.134691+0000", 2024-09-11T09:52:09.133 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0041498739999999996 2024-09-11T09:52:09.133 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.133 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.133 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:09.133 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.134705+0000", 2024-09-11T09:52:09.133 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.3669000000000001e-05 2024-09-11T09:52:09.133 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.133 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.133 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:09.133 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.134712+0000", 2024-09-11T09:52:09.133 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.8410000000000002e-06 2024-09-11T09:52:09.134 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.134 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.134 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:09.134 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.134754+0000", 2024-09-11T09:52:09.134 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.2524999999999997e-05 2024-09-11T09:52:09.134 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.134 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.134 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:09.134 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.134875+0000", 2024-09-11T09:52:09.134 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000120332 2024-09-11T09:52:09.134 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.134 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.134 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 2", 2024-09-11T09:52:09.134 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.135005+0000", 2024-09-11T09:52:09.134 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00013024900000000001 2024-09-11T09:52:09.135 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.135 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.135 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:09.135 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.145971+0000", 2024-09-11T09:52:09.135 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.010966203000000001 2024-09-11T09:52:09.135 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.135 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.135 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:52:09.135 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.147600+0000", 2024-09-11T09:52:09.135 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001628382 2024-09-11T09:52:09.135 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.135 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.135 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:09.135 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.147635+0000", 2024-09-11T09:52:09.135 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.5420000000000003e-05 2024-09-11T09:52:09.136 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.136 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.136 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:09.136 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.147707+0000", 2024-09-11T09:52:09.136 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.2373999999999995e-05 2024-09-11T09:52:09.136 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:09.136 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:09.136 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:09.136 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.136 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.136 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_repop(client.4694.0:49 2.7 e53/38 2:e8766325:ns::rbd_object_map.1256a0c02879:head v 53'33, mlcod=53'31)", 2024-09-11T09:52:09.136 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.135391+0000", 2024-09-11T09:52:09.136 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 45.931867857999997, 2024-09-11T09:52:09.136 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0051189859999999998, 2024-09-11T09:52:09.136 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:09.137 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:09.137 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:09.137 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:09.137 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.137 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:09.137 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.135391+0000", 2024-09-11T09:52:09.137 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:09.137 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.137 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.137 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:09.137 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.135391+0000", 2024-09-11T09:52:09.137 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:09.137 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.138 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.138 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:09.138 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.135393+0000", 2024-09-11T09:52:09.138 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.4020000000000001e-06 2024-09-11T09:52:09.138 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.138 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.138 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:09.138 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.135400+0000", 2024-09-11T09:52:09.138 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.5129999999999999e-06 2024-09-11T09:52:09.138 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.138 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.138 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:09.138 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.135411+0000", 2024-09-11T09:52:09.138 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1038e-05 2024-09-11T09:52:09.139 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.139 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.139 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:09.139 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.135418+0000", 2024-09-11T09:52:09.139 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.1489999999999997e-06 2024-09-11T09:52:09.139 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.139 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.139 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:09.139 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.135470+0000", 2024-09-11T09:52:09.139 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.2021999999999998e-05 2024-09-11T09:52:09.139 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.139 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.139 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:09.139 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.135501+0000", 2024-09-11T09:52:09.140 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.0349999999999999e-05 2024-09-11T09:52:09.140 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.140 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.140 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:09.140 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.140453+0000", 2024-09-11T09:52:09.140 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0049522409999999996 2024-09-11T09:52:09.140 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.140 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.140 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:09.140 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.140510+0000", 2024-09-11T09:52:09.140 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.7271000000000001e-05 2024-09-11T09:52:09.140 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:09.140 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:09.141 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:09.141 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.141 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.141 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4694.0:54 2.3 2:c44f0253:ns::rbd_data.1256a0c02879.000000000000000b:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:09.141 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.150151+0000", 2024-09-11T09:52:09.141 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 45.917108007000003, 2024-09-11T09:52:09.141 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.015103484, 2024-09-11T09:52:09.141 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:09.141 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:09.141 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:09.141 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:52:09.141 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4694", 2024-09-11T09:52:09.141 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:52:09.141 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 54 2024-09-11T09:52:09.142 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.142 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:09.142 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.142 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:09.142 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.150151+0000", 2024-09-11T09:52:09.142 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:09.142 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.142 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.142 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:09.142 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.150151+0000", 2024-09-11T09:52:09.142 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:09.142 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.142 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.142 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:09.142 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.150152+0000", 2024-09-11T09:52:09.142 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.2550000000000001e-06 2024-09-11T09:52:09.143 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.143 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.143 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:09.143 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.154528+0000", 2024-09-11T09:52:09.143 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0043758219999999997 2024-09-11T09:52:09.143 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.143 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.143 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:09.143 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.154541+0000", 2024-09-11T09:52:09.143 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.3414999999999999e-05 2024-09-11T09:52:09.143 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.143 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.143 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:09.143 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.154551+0000", 2024-09-11T09:52:09.143 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.4350000000000003e-06 2024-09-11T09:52:09.144 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.144 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.144 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:09.144 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.154615+0000", 2024-09-11T09:52:09.144 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.3627000000000006e-05 2024-09-11T09:52:09.144 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.144 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.144 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:09.144 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.154717+0000", 2024-09-11T09:52:09.144 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000102729 2024-09-11T09:52:09.144 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.144 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.144 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 2", 2024-09-11T09:52:09.144 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.154826+0000", 2024-09-11T09:52:09.145 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000108409 2024-09-11T09:52:09.145 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.145 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.145 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:52:09.145 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.165094+0000", 2024-09-11T09:52:09.145 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.010268210999999999 2024-09-11T09:52:09.145 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.145 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.145 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:09.145 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.165138+0000", 2024-09-11T09:52:09.145 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.4078e-05 2024-09-11T09:52:09.145 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.145 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.145 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:09.146 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.165204+0000", 2024-09-11T09:52:09.146 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.5716000000000006e-05 2024-09-11T09:52:09.146 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.146 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.146 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:09.146 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.165254+0000", 2024-09-11T09:52:09.146 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.0787e-05 2024-09-11T09:52:09.146 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:09.146 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:09.146 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:09.146 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.146 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.146 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_repop(client.4694.0:55 2.7 e53/38 2:e8766325:ns::rbd_object_map.1256a0c02879:head v 53'37, mlcod=53'34)", 2024-09-11T09:52:09.146 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.150221+0000", 2024-09-11T09:52:09.147 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 45.917038142000003, 2024-09-11T09:52:09.147 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00530015, 2024-09-11T09:52:09.147 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:09.147 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:09.147 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:09.147 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:09.147 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.147 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:09.147 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.150221+0000", 2024-09-11T09:52:09.147 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:09.147 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.147 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.148 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:09.148 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.150221+0000", 2024-09-11T09:52:09.148 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:09.148 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.148 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.148 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:09.148 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.150221+0000", 2024-09-11T09:52:09.148 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.9100000000000004e-07 2024-09-11T09:52:09.148 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.148 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.148 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:09.148 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.150226+0000", 2024-09-11T09:52:09.148 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.2250000000000002e-06 2024-09-11T09:52:09.148 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.149 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.149 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:09.149 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.150251+0000", 2024-09-11T09:52:09.149 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.4987e-05 2024-09-11T09:52:09.149 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.149 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.149 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:09.149 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.150258+0000", 2024-09-11T09:52:09.149 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.0790000000000003e-06 2024-09-11T09:52:09.149 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.149 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.149 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:09.150 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.150305+0000", 2024-09-11T09:52:09.150 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.6879000000000003e-05 2024-09-11T09:52:09.150 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.150 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.150 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:09.150 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.150320+0000", 2024-09-11T09:52:09.150 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.5268999999999999e-05 2024-09-11T09:52:09.150 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.150 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.150 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:09.150 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.155493+0000", 2024-09-11T09:52:09.150 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0051731390000000002 2024-09-11T09:52:09.150 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.150 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.151 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:09.151 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.155521+0000", 2024-09-11T09:52:09.151 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.8081000000000001e-05 2024-09-11T09:52:09.151 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:09.151 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:09.151 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:09.151 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.151 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.151 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4694.0:56 2.3 2:c462f029:ns::rbd_data.1256a0c02879.000000000000000c:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:09.151 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.154576+0000", 2024-09-11T09:52:09.151 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 45.912683166000001, 2024-09-11T09:52:09.151 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.020781621, 2024-09-11T09:52:09.151 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:09.152 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:09.152 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:09.152 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:52:09.152 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4694", 2024-09-11T09:52:09.152 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:52:09.152 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 56 2024-09-11T09:52:09.152 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.152 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:09.152 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.152 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:09.152 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.154576+0000", 2024-09-11T09:52:09.152 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:09.152 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.152 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.153 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:09.153 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.154576+0000", 2024-09-11T09:52:09.153 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:09.153 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.153 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.153 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:09.153 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.154577+0000", 2024-09-11T09:52:09.153 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.2249999999999999e-06 2024-09-11T09:52:09.153 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.153 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.154 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:09.154 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.158783+0000", 2024-09-11T09:52:09.154 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0042063889999999996 2024-09-11T09:52:09.154 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.154 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.154 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:09.154 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.158803+0000", 2024-09-11T09:52:09.154 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.9576999999999999e-05 2024-09-11T09:52:09.154 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.154 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.154 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:09.154 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.158815+0000", 2024-09-11T09:52:09.154 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.2221999999999999e-05 2024-09-11T09:52:09.154 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.155 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.155 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:09.155 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.158902+0000", 2024-09-11T09:52:09.155 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.6390000000000005e-05 2024-09-11T09:52:09.155 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.155 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.155 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:09.155 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.159141+0000", 2024-09-11T09:52:09.155 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000238886 2024-09-11T09:52:09.155 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.155 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.155 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 2", 2024-09-11T09:52:09.155 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.159279+0000", 2024-09-11T09:52:09.155 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000138073 2024-09-11T09:52:09.155 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.156 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.156 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:52:09.156 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.173297+0000", 2024-09-11T09:52:09.156 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.014018113 2024-09-11T09:52:09.156 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.156 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.156 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:09.156 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.175278+0000", 2024-09-11T09:52:09.156 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0019817390000000002 2024-09-11T09:52:09.156 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.156 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.156 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:09.156 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.175316+0000", 2024-09-11T09:52:09.157 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.7829e-05 2024-09-11T09:52:09.157 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.157 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.157 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:09.157 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.175357+0000", 2024-09-11T09:52:09.157 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.1177999999999998e-05 2024-09-11T09:52:09.157 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:09.157 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:09.157 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:09.157 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.157 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.157 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4694.0:59 2.2 2:41adefc9:ns::rbd_data.1256a0c02879.000000000000000e:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:09.157 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.158843+0000", 2024-09-11T09:52:09.157 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 45.908415599999998, 2024-09-11T09:52:09.158 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.017874207, 2024-09-11T09:52:09.158 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:09.158 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:09.158 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:09.158 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:52:09.158 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4694", 2024-09-11T09:52:09.158 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:52:09.158 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 59 2024-09-11T09:52:09.158 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.158 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:09.158 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.158 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:09.158 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.158843+0000", 2024-09-11T09:52:09.159 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:09.159 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.159 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.159 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:09.159 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.158843+0000", 2024-09-11T09:52:09.159 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:09.159 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.159 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.159 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:09.159 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.158845+0000", 2024-09-11T09:52:09.159 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.7439999999999999e-06 2024-09-11T09:52:09.159 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.159 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.159 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:09.160 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.162713+0000", 2024-09-11T09:52:09.160 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0038678829999999999 2024-09-11T09:52:09.160 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.160 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.160 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:09.160 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.162728+0000", 2024-09-11T09:52:09.160 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.5114e-05 2024-09-11T09:52:09.160 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.160 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.160 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:09.160 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.162737+0000", 2024-09-11T09:52:09.160 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.1099999999999992e-06 2024-09-11T09:52:09.160 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.161 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.161 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:09.161 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.162799+0000", 2024-09-11T09:52:09.161 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.1934999999999996e-05 2024-09-11T09:52:09.161 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.161 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.161 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:09.161 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.162911+0000", 2024-09-11T09:52:09.161 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000111867 2024-09-11T09:52:09.161 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.161 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.161 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 1", 2024-09-11T09:52:09.161 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.163030+0000", 2024-09-11T09:52:09.161 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000119108 2024-09-11T09:52:09.162 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.162 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.162 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:52:09.162 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.174086+0000", 2024-09-11T09:52:09.162 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.01105543 2024-09-11T09:52:09.162 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.162 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.162 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:09.162 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.176614+0000", 2024-09-11T09:52:09.162 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0025284940000000001 2024-09-11T09:52:09.162 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.162 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.162 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:09.162 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.176672+0000", 2024-09-11T09:52:09.163 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.7892000000000001e-05 2024-09-11T09:52:09.163 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.163 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.163 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:09.163 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.176718+0000", 2024-09-11T09:52:09.163 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.5630000000000002e-05 2024-09-11T09:52:09.163 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:09.163 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:09.163 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:09.163 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.163 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.163 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.5002.0:89 2.3 2:c54742d8:::rbd_data.1250f5e25b80.0000000000000006:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-09-11T09:52:09.163 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:38.546016+0000", 2024-09-11T09:52:09.163 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 30.521243053999999, 2024-09-11T09:52:09.164 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0049098550000000003, 2024-09-11T09:52:09.164 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:09.164 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:09.164 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:09.164 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:52:09.164 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.5002", 2024-09-11T09:52:09.164 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/2204440692", 2024-09-11T09:52:09.164 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 89 2024-09-11T09:52:09.164 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.164 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:09.164 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.164 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:09.164 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.546016+0000", 2024-09-11T09:52:09.164 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:09.165 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.165 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.165 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:09.165 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.546016+0000", 2024-09-11T09:52:09.165 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:09.165 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.165 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.165 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:09.165 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.546017+0000", 2024-09-11T09:52:09.165 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.3399999999999998e-07 2024-09-11T09:52:09.165 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.165 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.165 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:09.165 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.546839+0000", 2024-09-11T09:52:09.165 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00082222800000000004 2024-09-11T09:52:09.166 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.166 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.166 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:09.166 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.546851+0000", 2024-09-11T09:52:09.166 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.2075999999999999e-05 2024-09-11T09:52:09.166 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.166 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.166 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:09.166 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.546856+0000", 2024-09-11T09:52:09.166 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.4009999999999996e-06 2024-09-11T09:52:09.166 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.166 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.166 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:09.166 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.546897+0000", 2024-09-11T09:52:09.167 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.0113e-05 2024-09-11T09:52:09.167 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.167 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.167 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:09.167 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.546954+0000", 2024-09-11T09:52:09.167 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.7825000000000003e-05 2024-09-11T09:52:09.167 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.167 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.167 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 2", 2024-09-11T09:52:09.167 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.547055+0000", 2024-09-11T09:52:09.167 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000100269 2024-09-11T09:52:09.167 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.167 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.168 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:09.168 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.549675+0000", 2024-09-11T09:52:09.168 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0026206609999999998 2024-09-11T09:52:09.168 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.168 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.168 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:52:09.168 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.550858+0000", 2024-09-11T09:52:09.168 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0011825270000000001 2024-09-11T09:52:09.168 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.168 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.168 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:09.168 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.550894+0000", 2024-09-11T09:52:09.168 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.5896999999999999e-05 2024-09-11T09:52:09.169 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.169 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.169 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:09.169 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.550926+0000", 2024-09-11T09:52:09.169 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.2023999999999997e-05 2024-09-11T09:52:09.169 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:09.169 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:09.169 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:09.169 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.169 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.169 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.5002.0:91 2.3 2:c54742d8:::rbd_data.1250f5e25b80.0000000000000006:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-09-11T09:52:09.169 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:38.553743+0000", 2024-09-11T09:52:09.169 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 30.513516011, 2024-09-11T09:52:09.170 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0051633629999999998, 2024-09-11T09:52:09.170 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:09.170 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:09.170 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:09.170 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:52:09.170 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.5002", 2024-09-11T09:52:09.170 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/2204440692", 2024-09-11T09:52:09.170 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 91 2024-09-11T09:52:09.170 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.170 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:09.170 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.170 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:09.170 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.553743+0000", 2024-09-11T09:52:09.170 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:09.170 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.171 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.171 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:09.171 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.553743+0000", 2024-09-11T09:52:09.171 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:09.171 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.171 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.171 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:09.171 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.553745+0000", 2024-09-11T09:52:09.171 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.5120000000000001e-06 2024-09-11T09:52:09.171 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.171 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.171 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:09.171 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.554588+0000", 2024-09-11T09:52:09.171 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00084352299999999999 2024-09-11T09:52:09.172 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.172 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.172 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:09.172 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.554604+0000", 2024-09-11T09:52:09.172 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.643e-05 2024-09-11T09:52:09.172 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.172 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.172 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:09.172 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.554613+0000", 2024-09-11T09:52:09.172 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.5669999999999995e-06 2024-09-11T09:52:09.172 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.172 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.172 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:09.173 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.554662+0000", 2024-09-11T09:52:09.173 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.8510000000000001e-05 2024-09-11T09:52:09.173 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.173 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.173 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:09.173 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.554732+0000", 2024-09-11T09:52:09.173 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.0297999999999997e-05 2024-09-11T09:52:09.173 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.173 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.173 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 2", 2024-09-11T09:52:09.173 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.554870+0000", 2024-09-11T09:52:09.173 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00013856599999999999 2024-09-11T09:52:09.173 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.173 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.174 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:09.174 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.557875+0000", 2024-09-11T09:52:09.174 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0030042760000000002 2024-09-11T09:52:09.174 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.174 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.174 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:52:09.174 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.558835+0000", 2024-09-11T09:52:09.174 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00096057900000000003 2024-09-11T09:52:09.174 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.174 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.174 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:09.174 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.558874+0000", 2024-09-11T09:52:09.174 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.8992999999999997e-05 2024-09-11T09:52:09.174 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.175 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.175 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:09.175 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.558906+0000", 2024-09-11T09:52:09.175 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.2109e-05 2024-09-11T09:52:09.175 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:09.175 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:09.175 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:09.175 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.175 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.175 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.5002.0:93 2.3 2:c54742d8:::rbd_data.1250f5e25b80.0000000000000006:head [write 2228224~917504] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e61)", 2024-09-11T09:52:09.175 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:38.562067+0000", 2024-09-11T09:52:09.175 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 30.505192415, 2024-09-11T09:52:09.175 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0050551099999999998, 2024-09-11T09:52:09.176 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:09.176 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:09.176 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:09.176 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:52:09.176 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.5002", 2024-09-11T09:52:09.176 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/2204440692", 2024-09-11T09:52:09.176 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 93 2024-09-11T09:52:09.176 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.176 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:09.176 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.176 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:09.176 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.562067+0000", 2024-09-11T09:52:09.176 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:09.177 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.177 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.177 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:09.177 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.562067+0000", 2024-09-11T09:52:09.177 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:09.177 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.177 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.177 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:09.177 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.562068+0000", 2024-09-11T09:52:09.177 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.0669999999999999e-06 2024-09-11T09:52:09.177 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.177 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.177 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:09.177 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.563170+0000", 2024-09-11T09:52:09.178 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00110256 2024-09-11T09:52:09.178 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.178 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.178 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:09.178 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.563185+0000", 2024-09-11T09:52:09.178 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.4873e-05 2024-09-11T09:52:09.178 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.178 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.178 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:09.178 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.563193+0000", 2024-09-11T09:52:09.178 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.1319999999999994e-06 2024-09-11T09:52:09.178 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.178 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.178 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:09.179 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.563238+0000", 2024-09-11T09:52:09.179 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.4773e-05 2024-09-11T09:52:09.179 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.179 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.179 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:09.179 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.563290+0000", 2024-09-11T09:52:09.179 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.1690000000000001e-05 2024-09-11T09:52:09.179 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.179 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.179 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 2", 2024-09-11T09:52:09.179 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.563365+0000", 2024-09-11T09:52:09.179 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.5023000000000003e-05 2024-09-11T09:52:09.179 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.179 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.180 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:09.180 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.566552+0000", 2024-09-11T09:52:09.180 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0031869960000000001 2024-09-11T09:52:09.180 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.180 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.180 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:52:09.180 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.567048+0000", 2024-09-11T09:52:09.180 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00049605099999999996 2024-09-11T09:52:09.180 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.180 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.180 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:09.180 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.567095+0000", 2024-09-11T09:52:09.180 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.6823999999999998e-05 2024-09-11T09:52:09.181 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.181 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.181 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:09.181 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.567122+0000", 2024-09-11T09:52:09.181 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.7121e-05 2024-09-11T09:52:09.181 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:09.181 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:09.181 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:09.181 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:09.181 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:09.181 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-11T09:52:09.183 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-11T09:52:09.183 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:52:09.183+0000 7f3a7b90a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 42475) UID: 0 2024-09-11T09:52:09.283 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:52:09.284+0000 7f3c96ec9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31909) UID: 0 2024-09-11T09:52:09.327 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-11T09:52:09.327 INFO:teuthology.orchestra.run.smithi096.stdout: "ops": [], 2024-09-11T09:52:09.327 INFO:teuthology.orchestra.run.smithi096.stdout: "num_ops": 0 2024-09-11T09:52:09.327 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-11T09:52:09.336 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-11T09:52:09.420 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:52:09.421+0000 7f3a7b90a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 42475) UID: 0 2024-09-11T09:52:09.479 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-11T09:52:09.480 INFO:teuthology.orchestra.run.smithi096.stdout: "ops": [], 2024-09-11T09:52:09.480 INFO:teuthology.orchestra.run.smithi096.stdout: "complaint_time": 30, 2024-09-11T09:52:09.480 INFO:teuthology.orchestra.run.smithi096.stdout: "num_blocked_ops": 0 2024-09-11T09:52:09.480 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-11T09:52:09.485 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-09-11T09:52:09.486+0000 7f7b6c7d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31904) UID: 0 2024-09-11T09:52:09.489 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-11T09:52:09.586 INFO:tasks.ceph.osd.2.smithi017.stderr:2024-09-11T09:52:09.587+0000 7fd20b170640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31672) UID: 0 2024-09-11T09:52:09.634 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-11T09:52:09.634 INFO:teuthology.orchestra.run.smithi096.stdout: "size": 20, 2024-09-11T09:52:09.634 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 600, 2024-09-11T09:52:09.634 INFO:teuthology.orchestra.run.smithi096.stdout: "ops": [ 2024-09-11T09:52:09.634 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.635 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "MOSDPGPush(2.4 82/81 [PushOp(2:3bfc818c:ns::rbd_data.1256a0c02879.0000000000000001:head, version: 53'14, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:3bfc818c:ns::rbd_data.1256a0c02879.0000000000000001:head@53'14, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-11T09:52:09.635 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:52:08.947926+0000", 2024-09-11T09:52:09.635 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 0.68616693799999995, 2024-09-11T09:52:09.635 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0020846179999999999, 2024-09-11T09:52:09.635 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:09.635 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:09.635 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "started", 2024-09-11T09:52:09.635 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:09.635 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.635 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:09.635 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.947926+0000", 2024-09-11T09:52:09.635 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:09.635 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.636 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.636 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:09.636 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.947926+0000", 2024-09-11T09:52:09.636 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:09.636 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.636 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.636 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:09.636 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.947927+0000", 2024-09-11T09:52:09.636 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.9599999999999999e-07 2024-09-11T09:52:09.636 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.636 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.636 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:09.636 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.949094+0000", 2024-09-11T09:52:09.636 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001167944 2024-09-11T09:52:09.637 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.637 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.637 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:09.637 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.949110+0000", 2024-09-11T09:52:09.637 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.5650000000000001e-05 2024-09-11T09:52:09.637 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.637 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.637 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:09.637 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.949138+0000", 2024-09-11T09:52:09.637 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.7515e-05 2024-09-11T09:52:09.637 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.637 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.637 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:09.638 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.949196+0000", 2024-09-11T09:52:09.638 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.8272e-05 2024-09-11T09:52:09.638 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.638 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.638 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:09.638 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.949204+0000", 2024-09-11T09:52:09.638 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.2150000000000001e-06 2024-09-11T09:52:09.638 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.638 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.638 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:09.638 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.950011+0000", 2024-09-11T09:52:09.638 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00080652600000000001 2024-09-11T09:52:09.638 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:09.639 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:09.639 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:09.639 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.639 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.639 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "MOSDPGPush(2.4 82/81 [PushOp(2:3bfc818c:ns::rbd_data.1256a0c02879.0000000000000001:head, version: 53'14, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:3bfc818c:ns::rbd_data.1256a0c02879.0000000000000001:head@53'14, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-11T09:52:09.639 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:52:08.951738+0000", 2024-09-11T09:52:09.639 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 0.68235542000000005, 2024-09-11T09:52:09.639 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0018266020000000001, 2024-09-11T09:52:09.639 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:09.639 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:09.639 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "started", 2024-09-11T09:52:09.639 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:09.639 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.640 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:09.640 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.951738+0000", 2024-09-11T09:52:09.640 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:09.640 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.640 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.640 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:09.640 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.951738+0000", 2024-09-11T09:52:09.640 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:09.640 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.640 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.640 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:09.640 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.951739+0000", 2024-09-11T09:52:09.640 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.004e-06 2024-09-11T09:52:09.641 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.641 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.641 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:09.641 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.952722+0000", 2024-09-11T09:52:09.641 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00098314800000000009 2024-09-11T09:52:09.641 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.641 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.641 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:09.641 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.952750+0000", 2024-09-11T09:52:09.642 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.8050000000000001e-05 2024-09-11T09:52:09.642 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.642 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.642 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:09.642 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.952771+0000", 2024-09-11T09:52:09.642 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.0891e-05 2024-09-11T09:52:09.643 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.643 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.643 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:09.643 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.952844+0000", 2024-09-11T09:52:09.643 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.3064000000000004e-05 2024-09-11T09:52:09.643 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.643 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.643 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:09.643 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.952853+0000", 2024-09-11T09:52:09.643 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.1160000000000005e-06 2024-09-11T09:52:09.643 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.644 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.644 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:09.644 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.953564+0000", 2024-09-11T09:52:09.644 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00071132900000000002 2024-09-11T09:52:09.644 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:09.644 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:09.644 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:09.644 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.644 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.644 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "MOSDPGPush(2.4 82/81 [PushOp(2:3bfc818c:ns::rbd_data.1256a0c02879.0000000000000001:head, version: 53'14, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:3bfc818c:ns::rbd_data.1256a0c02879.0000000000000001:head@53'14, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3145728, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-11T09:52:09.644 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:52:08.955761+0000", 2024-09-11T09:52:09.644 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 0.67833226300000005, 2024-09-11T09:52:09.645 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0019711659999999999, 2024-09-11T09:52:09.645 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:09.645 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:09.645 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "started", 2024-09-11T09:52:09.645 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:09.645 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.645 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:09.645 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.955761+0000", 2024-09-11T09:52:09.645 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:09.645 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.645 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.645 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:09.645 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.955761+0000", 2024-09-11T09:52:09.646 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:09.646 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.646 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.646 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:09.646 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.955761+0000", 2024-09-11T09:52:09.646 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.2399999999999997e-07 2024-09-11T09:52:09.646 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.646 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.646 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:09.646 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.956708+0000", 2024-09-11T09:52:09.646 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00094629899999999995 2024-09-11T09:52:09.646 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.647 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.647 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:09.647 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.956731+0000", 2024-09-11T09:52:09.647 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.3175000000000001e-05 2024-09-11T09:52:09.647 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.647 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.647 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:09.647 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.956750+0000", 2024-09-11T09:52:09.647 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.8859e-05 2024-09-11T09:52:09.647 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.648 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.648 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:09.648 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.956828+0000", 2024-09-11T09:52:09.648 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.8567999999999997e-05 2024-09-11T09:52:09.648 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.648 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.648 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:09.648 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.956841+0000", 2024-09-11T09:52:09.648 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.2489e-05 2024-09-11T09:52:09.648 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.649 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.649 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:09.649 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.957732+0000", 2024-09-11T09:52:09.649 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00089105200000000003 2024-09-11T09:52:09.649 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:09.649 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:09.649 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:09.649 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.649 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.649 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "MOSDPGPush(2.4 82/81 [PushOp(2:3bfc818c:ns::rbd_data.1256a0c02879.0000000000000001:head, version: 53'14, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:3bfc818c:ns::rbd_data.1256a0c02879.0000000000000001:head@53'14, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 4194304, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3145728, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-11T09:52:09.649 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:52:08.959483+0000", 2024-09-11T09:52:09.650 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 0.67460973300000004, 2024-09-11T09:52:09.650 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0021404520000000002, 2024-09-11T09:52:09.650 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:09.650 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:09.650 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "started", 2024-09-11T09:52:09.650 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:09.650 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.650 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:09.650 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.959483+0000", 2024-09-11T09:52:09.650 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:09.650 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.650 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.650 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:09.651 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.959483+0000", 2024-09-11T09:52:09.651 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:09.651 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.651 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.651 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:09.651 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.959484+0000", 2024-09-11T09:52:09.651 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.4900000000000005e-07 2024-09-11T09:52:09.651 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.651 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.651 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:09.651 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.960450+0000", 2024-09-11T09:52:09.651 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000966141 2024-09-11T09:52:09.651 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.652 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.652 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:09.652 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.960466+0000", 2024-09-11T09:52:09.652 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.6302000000000001e-05 2024-09-11T09:52:09.652 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.652 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.652 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:09.652 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.960477+0000", 2024-09-11T09:52:09.652 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.0406e-05 2024-09-11T09:52:09.652 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.652 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.652 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:09.653 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.960555+0000", 2024-09-11T09:52:09.653 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.7837999999999996e-05 2024-09-11T09:52:09.653 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.653 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.653 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:09.653 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.960568+0000", 2024-09-11T09:52:09.653 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.2904e-05 2024-09-11T09:52:09.653 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.653 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.653 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:09.653 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.961624+0000", 2024-09-11T09:52:09.653 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0010561119999999999 2024-09-11T09:52:09.653 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:09.654 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:09.654 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:09.654 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.654 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.654 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "MOSDPGPush(2.4 82/81 [PushOp(2:2367385a:::rbd_data.1250f5e25b80.0000000000000002:6, version: 79'169, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 1, recovery_info: ObjectRecoveryInfo(2:2367385a:::rbd_data.1250f5e25b80.0000000000000002:6@79'169, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 6=[]:{6: [6, 5, 3]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-11T09:52:09.654 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:52:08.964132+0000", 2024-09-11T09:52:09.654 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 0.66996125200000001, 2024-09-11T09:52:09.654 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0021074240000000001, 2024-09-11T09:52:09.654 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:09.654 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:09.654 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "started", 2024-09-11T09:52:09.654 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:09.654 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.654 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:09.655 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.964132+0000", 2024-09-11T09:52:09.655 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:09.655 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.655 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.655 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:09.655 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.964132+0000", 2024-09-11T09:52:09.655 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:09.655 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.655 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.655 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:09.655 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.964133+0000", 2024-09-11T09:52:09.655 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.02e-07 2024-09-11T09:52:09.655 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.656 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.656 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:09.656 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.965104+0000", 2024-09-11T09:52:09.656 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000971849 2024-09-11T09:52:09.656 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.656 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.656 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:09.656 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.965122+0000", 2024-09-11T09:52:09.656 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.7706999999999998e-05 2024-09-11T09:52:09.656 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.656 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.656 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:09.656 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.965134+0000", 2024-09-11T09:52:09.656 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1371e-05 2024-09-11T09:52:09.657 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.657 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.657 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:09.657 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.965221+0000", 2024-09-11T09:52:09.657 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.7037e-05 2024-09-11T09:52:09.657 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.657 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.657 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:09.657 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.965236+0000", 2024-09-11T09:52:09.657 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.5724999999999999e-05 2024-09-11T09:52:09.657 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.657 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.658 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:09.658 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.966239+0000", 2024-09-11T09:52:09.658 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0010028330000000001 2024-09-11T09:52:09.658 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:09.658 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:09.658 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:09.658 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.658 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.658 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "MOSDPGPush(2.4 82/81 [PushOp(2:2367385a:::rbd_data.1250f5e25b80.0000000000000002:6, version: 79'169, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:2367385a:::rbd_data.1250f5e25b80.0000000000000002:6@79'169, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 6=[]:{6: [6, 5, 3]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-11T09:52:09.658 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:52:08.968106+0000", 2024-09-11T09:52:09.658 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 0.66598678499999997, 2024-09-11T09:52:09.658 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.002036936, 2024-09-11T09:52:09.658 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:09.659 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:09.659 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "started", 2024-09-11T09:52:09.659 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:09.659 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.659 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:09.659 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.968106+0000", 2024-09-11T09:52:09.659 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:09.659 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.659 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.659 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:09.659 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.968106+0000", 2024-09-11T09:52:09.659 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:09.660 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.660 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.660 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:09.660 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.968107+0000", 2024-09-11T09:52:09.660 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.0100000000000001e-06 2024-09-11T09:52:09.660 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.660 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.660 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:09.660 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.969086+0000", 2024-09-11T09:52:09.660 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00097889099999999996 2024-09-11T09:52:09.660 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.660 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.661 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:09.661 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.969104+0000", 2024-09-11T09:52:09.661 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.7940000000000001e-05 2024-09-11T09:52:09.661 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.661 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.661 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:09.661 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.969117+0000", 2024-09-11T09:52:09.661 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.3207000000000001e-05 2024-09-11T09:52:09.661 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.661 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.661 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:09.661 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.969192+0000", 2024-09-11T09:52:09.661 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.4419999999999996e-05 2024-09-11T09:52:09.662 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.662 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.662 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:09.662 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.969201+0000", 2024-09-11T09:52:09.662 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.4960000000000002e-06 2024-09-11T09:52:09.662 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.662 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.662 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:09.662 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.970143+0000", 2024-09-11T09:52:09.662 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00094197200000000001 2024-09-11T09:52:09.662 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:09.662 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:09.662 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:09.663 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.663 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.663 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "MOSDPGPush(2.4 82/81 [PushOp(2:2367385a:::rbd_data.1250f5e25b80.0000000000000002:6, version: 79'169, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:2367385a:::rbd_data.1250f5e25b80.0000000000000002:6@79'169, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 6=[]:{6: [6, 5, 3]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3145728, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-11T09:52:09.663 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:52:08.971856+0000", 2024-09-11T09:52:09.663 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 0.66223676300000001, 2024-09-11T09:52:09.663 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0018957309999999999, 2024-09-11T09:52:09.663 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:09.663 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:09.663 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "started", 2024-09-11T09:52:09.663 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:09.663 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.663 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:09.663 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.971856+0000", 2024-09-11T09:52:09.663 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:09.664 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.664 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.664 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:09.664 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.971856+0000", 2024-09-11T09:52:09.664 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:09.664 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.664 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.664 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:09.664 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.971857+0000", 2024-09-11T09:52:09.664 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.4900000000000004e-07 2024-09-11T09:52:09.664 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.664 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.664 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:09.664 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.972795+0000", 2024-09-11T09:52:09.665 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00093767999999999998 2024-09-11T09:52:09.665 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.665 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.665 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:09.665 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.972813+0000", 2024-09-11T09:52:09.665 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.8448999999999999e-05 2024-09-11T09:52:09.665 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.665 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.665 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:09.665 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.972825+0000", 2024-09-11T09:52:09.665 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1837e-05 2024-09-11T09:52:09.665 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.665 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.666 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:09.666 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.972904+0000", 2024-09-11T09:52:09.666 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.8777999999999997e-05 2024-09-11T09:52:09.666 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.666 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.666 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:09.666 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.972913+0000", 2024-09-11T09:52:09.666 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.1139999999999995e-06 2024-09-11T09:52:09.666 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.666 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.666 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:09.666 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.973752+0000", 2024-09-11T09:52:09.666 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00083892400000000001 2024-09-11T09:52:09.667 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:09.667 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:09.667 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:09.667 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.667 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.667 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "MOSDPGPush(2.4 82/81 [PushOp(2:2367385a:::rbd_data.1250f5e25b80.0000000000000002:6, version: 79'169, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:2367385a:::rbd_data.1250f5e25b80.0000000000000002:6@79'169, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 6=[]:{6: [6, 5, 3]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 4194304, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3145728, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-11T09:52:09.667 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:52:08.975206+0000", 2024-09-11T09:52:09.667 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 0.65888707700000004, 2024-09-11T09:52:09.667 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0020632770000000001, 2024-09-11T09:52:09.667 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:09.668 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:09.668 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "started", 2024-09-11T09:52:09.668 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:09.668 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.668 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:09.668 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.975206+0000", 2024-09-11T09:52:09.668 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:09.668 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.668 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.668 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:09.668 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.975206+0000", 2024-09-11T09:52:09.668 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:09.668 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.668 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.669 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:09.669 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.975207+0000", 2024-09-11T09:52:09.669 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.2399999999999998e-07 2024-09-11T09:52:09.669 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.669 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.669 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:09.669 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.976173+0000", 2024-09-11T09:52:09.669 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00096649400000000003 2024-09-11T09:52:09.670 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.670 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.670 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:09.670 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.976189+0000", 2024-09-11T09:52:09.670 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.6223e-05 2024-09-11T09:52:09.670 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.671 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.671 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:09.671 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.976200+0000", 2024-09-11T09:52:09.671 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.0729e-05 2024-09-11T09:52:09.671 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.671 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.671 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:09.671 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.976265+0000", 2024-09-11T09:52:09.671 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.5326000000000004e-05 2024-09-11T09:52:09.671 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.671 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.672 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:09.672 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.976274+0000", 2024-09-11T09:52:09.672 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.8140000000000007e-06 2024-09-11T09:52:09.672 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.672 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.672 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:09.672 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.977269+0000", 2024-09-11T09:52:09.672 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00099506700000000009 2024-09-11T09:52:09.672 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:09.672 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:09.672 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:09.672 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.673 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.673 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "MOSDPGPush(2.4 82/81 [PushOp(2:26d5c2f0:::rbd_data.1250f5e25b80.000000000000000b:6, version: 79'171, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 1, recovery_info: ObjectRecoveryInfo(2:26d5c2f0:::rbd_data.1250f5e25b80.000000000000000b:6@79'171, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 6=[]:{6: [6, 5, 3]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-11T09:52:09.673 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:52:08.979123+0000", 2024-09-11T09:52:09.673 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 0.65496964800000002, 2024-09-11T09:52:09.673 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0019749350000000001, 2024-09-11T09:52:09.673 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:09.673 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:09.673 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "started", 2024-09-11T09:52:09.673 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:09.673 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.673 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:09.673 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.979123+0000", 2024-09-11T09:52:09.673 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:09.674 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.674 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.674 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:09.674 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.979123+0000", 2024-09-11T09:52:09.674 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:09.674 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.674 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.674 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:09.674 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.979124+0000", 2024-09-11T09:52:09.674 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.5799999999999999e-07 2024-09-11T09:52:09.674 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.674 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.675 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:09.675 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.980088+0000", 2024-09-11T09:52:09.675 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00096377699999999997 2024-09-11T09:52:09.675 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.675 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.675 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:09.675 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.980106+0000", 2024-09-11T09:52:09.675 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.7881e-05 2024-09-11T09:52:09.675 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.675 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.675 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:09.675 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.980117+0000", 2024-09-11T09:52:09.676 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1681e-05 2024-09-11T09:52:09.676 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.676 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.676 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:09.676 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.980189+0000", 2024-09-11T09:52:09.676 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.1365999999999994e-05 2024-09-11T09:52:09.676 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.676 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.676 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:09.676 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.980198+0000", 2024-09-11T09:52:09.676 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.3300000000000005e-06 2024-09-11T09:52:09.677 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.677 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.677 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:09.677 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.981098+0000", 2024-09-11T09:52:09.677 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00090024199999999999 2024-09-11T09:52:09.677 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:09.677 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:09.677 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:09.677 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.677 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.677 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "MOSDPGPush(2.4 82/81 [PushOp(2:26d5c2f0:::rbd_data.1250f5e25b80.000000000000000b:6, version: 79'171, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:26d5c2f0:::rbd_data.1250f5e25b80.000000000000000b:6@79'171, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 6=[]:{6: [6, 5, 3]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-11T09:52:09.677 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:52:08.982550+0000", 2024-09-11T09:52:09.677 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 0.65154320300000002, 2024-09-11T09:52:09.678 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0020071540000000001, 2024-09-11T09:52:09.678 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:09.678 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:09.678 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "started", 2024-09-11T09:52:09.678 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:09.678 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.678 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:09.678 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.982550+0000", 2024-09-11T09:52:09.678 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:09.678 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.678 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.678 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:09.678 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.982550+0000", 2024-09-11T09:52:09.679 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:09.679 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.679 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.679 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:09.679 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.982550+0000", 2024-09-11T09:52:09.679 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.89e-07 2024-09-11T09:52:09.679 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.679 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.679 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:09.679 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.983523+0000", 2024-09-11T09:52:09.679 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00097246400000000003 2024-09-11T09:52:09.679 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.680 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.680 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:09.680 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.983540+0000", 2024-09-11T09:52:09.680 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.7388999999999999e-05 2024-09-11T09:52:09.680 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.680 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.680 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:09.680 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.983553+0000", 2024-09-11T09:52:09.680 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.2876e-05 2024-09-11T09:52:09.680 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.680 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.680 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:09.681 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.983623+0000", 2024-09-11T09:52:09.681 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.0199999999999999e-05 2024-09-11T09:52:09.681 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.681 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.681 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:09.681 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.983632+0000", 2024-09-11T09:52:09.681 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.8859999999999993e-06 2024-09-11T09:52:09.681 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.681 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.681 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:09.682 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.984557+0000", 2024-09-11T09:52:09.682 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00092484999999999996 2024-09-11T09:52:09.682 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:09.682 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:09.682 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:09.682 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.682 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.682 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "MOSDPGPush(2.4 82/81 [PushOp(2:26d5c2f0:::rbd_data.1250f5e25b80.000000000000000b:6, version: 79'171, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:26d5c2f0:::rbd_data.1250f5e25b80.000000000000000b:6@79'171, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 6=[]:{6: [6, 5, 3]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3145728, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-11T09:52:09.682 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:52:08.985896+0000", 2024-09-11T09:52:09.682 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 0.64819661200000001, 2024-09-11T09:52:09.682 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0018739799999999999, 2024-09-11T09:52:09.682 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:09.682 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:09.683 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "started", 2024-09-11T09:52:09.683 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:09.683 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.683 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:09.683 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.985896+0000", 2024-09-11T09:52:09.683 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:09.683 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.683 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.683 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:09.683 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.985896+0000", 2024-09-11T09:52:09.683 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:09.683 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.683 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.683 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:09.684 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.985897+0000", 2024-09-11T09:52:09.684 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.3099999999999996e-07 2024-09-11T09:52:09.684 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.684 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.684 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:09.684 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.986883+0000", 2024-09-11T09:52:09.684 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00098564300000000002 2024-09-11T09:52:09.684 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.684 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.684 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:09.684 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.986900+0000", 2024-09-11T09:52:09.684 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.7475000000000001e-05 2024-09-11T09:52:09.685 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.685 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.685 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:09.685 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.986912+0000", 2024-09-11T09:52:09.685 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1924e-05 2024-09-11T09:52:09.685 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.685 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.685 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:09.685 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.986988+0000", 2024-09-11T09:52:09.685 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.6099000000000004e-05 2024-09-11T09:52:09.685 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.685 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.685 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:09.686 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.986998+0000", 2024-09-11T09:52:09.686 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.5710000000000004e-06 2024-09-11T09:52:09.686 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.686 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.686 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:09.686 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.987770+0000", 2024-09-11T09:52:09.686 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000772337 2024-09-11T09:52:09.686 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:09.686 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:09.686 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:09.686 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.686 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.686 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "MOSDPGPush(2.4 82/81 [PushOp(2:26d5c2f0:::rbd_data.1250f5e25b80.000000000000000b:6, version: 79'171, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:26d5c2f0:::rbd_data.1250f5e25b80.000000000000000b:6@79'171, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 6=[]:{6: [6, 5, 3]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 4194304, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3145728, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-11T09:52:09.687 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:52:08.989143+0000", 2024-09-11T09:52:09.687 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 0.64494951199999995, 2024-09-11T09:52:09.687 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0022141589999999998, 2024-09-11T09:52:09.687 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:09.687 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:09.687 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "started", 2024-09-11T09:52:09.687 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:09.687 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.687 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:09.687 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.989143+0000", 2024-09-11T09:52:09.687 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:09.687 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.687 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.687 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:09.688 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.989143+0000", 2024-09-11T09:52:09.688 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:09.688 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.688 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.688 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:09.688 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.989144+0000", 2024-09-11T09:52:09.688 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.1699999999999997e-07 2024-09-11T09:52:09.688 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.688 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.688 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:09.688 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.990124+0000", 2024-09-11T09:52:09.688 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00097941400000000007 2024-09-11T09:52:09.689 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.689 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.689 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:09.689 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.990140+0000", 2024-09-11T09:52:09.689 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.6583e-05 2024-09-11T09:52:09.689 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.689 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.689 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:09.689 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.990152+0000", 2024-09-11T09:52:09.689 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1331999999999999e-05 2024-09-11T09:52:09.689 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.689 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.690 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:09.690 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.990220+0000", 2024-09-11T09:52:09.690 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.8777999999999998e-05 2024-09-11T09:52:09.690 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.690 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.690 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:09.690 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.990229+0000", 2024-09-11T09:52:09.690 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.9590000000000001e-06 2024-09-11T09:52:09.690 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.690 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.690 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:09.690 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.991358+0000", 2024-09-11T09:52:09.690 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0011283759999999999 2024-09-11T09:52:09.690 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:09.691 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:09.691 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:09.691 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.691 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.691 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "MOSDPGPush(2.4 82/81 [PushOp(2:239e915b:::rbd_data.1250f5e25b80.000000000000000c:6, version: 79'173, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 1, recovery_info: ObjectRecoveryInfo(2:239e915b:::rbd_data.1250f5e25b80.000000000000000c:6@79'173, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 6=[]:{6: [6, 5, 3]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-11T09:52:09.691 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:52:08.994080+0000", 2024-09-11T09:52:09.691 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 0.640013375, 2024-09-11T09:52:09.691 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0019554189999999999, 2024-09-11T09:52:09.691 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:09.691 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:09.691 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "started", 2024-09-11T09:52:09.691 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:09.691 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.692 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:09.692 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.994080+0000", 2024-09-11T09:52:09.692 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:09.692 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.692 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.692 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:09.692 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.994080+0000", 2024-09-11T09:52:09.692 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:09.692 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.692 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.692 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:09.692 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.994080+0000", 2024-09-11T09:52:09.692 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.7599999999999997e-07 2024-09-11T09:52:09.693 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.693 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.693 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:09.693 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.995065+0000", 2024-09-11T09:52:09.693 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00098436199999999991 2024-09-11T09:52:09.693 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.693 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.693 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:09.693 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.995080+0000", 2024-09-11T09:52:09.693 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.5119e-05 2024-09-11T09:52:09.693 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.693 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.693 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:09.694 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.995090+0000", 2024-09-11T09:52:09.694 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.0715e-05 2024-09-11T09:52:09.694 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.694 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.694 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:09.694 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.995152+0000", 2024-09-11T09:52:09.694 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.2089999999999997e-05 2024-09-11T09:52:09.694 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.694 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.694 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:09.694 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.995159+0000", 2024-09-11T09:52:09.694 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.9210000000000003e-06 2024-09-11T09:52:09.694 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.695 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.695 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:09.695 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.996035+0000", 2024-09-11T09:52:09.695 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00087563599999999995 2024-09-11T09:52:09.695 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:09.695 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:09.695 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:09.695 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.695 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.696 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "MOSDPGPush(2.4 82/81 [PushOp(2:239e915b:::rbd_data.1250f5e25b80.000000000000000c:6, version: 79'173, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:239e915b:::rbd_data.1250f5e25b80.000000000000000c:6@79'173, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 6=[]:{6: [6, 5, 3]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-11T09:52:09.696 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:52:08.997853+0000", 2024-09-11T09:52:09.696 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 0.63624042800000002, 2024-09-11T09:52:09.696 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0018792979999999999, 2024-09-11T09:52:09.696 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:09.696 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:09.696 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "started", 2024-09-11T09:52:09.696 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:09.696 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.696 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:09.697 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.997853+0000", 2024-09-11T09:52:09.697 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:09.697 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.697 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.697 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:09.697 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.997853+0000", 2024-09-11T09:52:09.697 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:09.697 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.697 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.697 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:09.697 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.997853+0000", 2024-09-11T09:52:09.697 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.4000000000000002e-07 2024-09-11T09:52:09.697 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.698 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.698 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:09.698 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.998831+0000", 2024-09-11T09:52:09.698 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00097792100000000009 2024-09-11T09:52:09.698 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.698 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.698 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:09.698 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.998846+0000", 2024-09-11T09:52:09.698 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.4987000000000001e-05 2024-09-11T09:52:09.698 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.698 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.698 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:09.698 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.998856+0000", 2024-09-11T09:52:09.698 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.0448000000000001e-05 2024-09-11T09:52:09.699 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.699 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.699 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:09.699 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.998942+0000", 2024-09-11T09:52:09.699 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.5563999999999996e-05 2024-09-11T09:52:09.699 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.699 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.699 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:09.699 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.998952+0000", 2024-09-11T09:52:09.699 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.5170000000000005e-06 2024-09-11T09:52:09.699 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.699 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.699 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:09.700 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.999732+0000", 2024-09-11T09:52:09.700 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00078032100000000005 2024-09-11T09:52:09.700 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:09.700 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:09.700 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:09.700 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.700 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.700 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "MOSDPGPush(2.4 82/81 [PushOp(2:239e915b:::rbd_data.1250f5e25b80.000000000000000c:6, version: 79'173, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:239e915b:::rbd_data.1250f5e25b80.000000000000000c:6@79'173, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 6=[]:{6: [6, 5, 3]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3145728, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-11T09:52:09.700 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:52:09.001584+0000", 2024-09-11T09:52:09.700 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 0.63250850800000002, 2024-09-11T09:52:09.700 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0019129119999999999, 2024-09-11T09:52:09.700 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:09.700 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:09.701 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "started", 2024-09-11T09:52:09.701 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:09.701 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.701 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:09.701 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.001584+0000", 2024-09-11T09:52:09.701 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:09.701 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.701 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.701 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:09.701 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.001584+0000", 2024-09-11T09:52:09.701 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:09.701 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.701 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.702 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:09.702 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.001585+0000", 2024-09-11T09:52:09.702 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.8899999999999999e-07 2024-09-11T09:52:09.702 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.702 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.702 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:09.702 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.002541+0000", 2024-09-11T09:52:09.702 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00095588800000000001 2024-09-11T09:52:09.702 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.702 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.702 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:09.702 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.002558+0000", 2024-09-11T09:52:09.702 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.6452000000000001e-05 2024-09-11T09:52:09.703 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.703 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.703 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:09.703 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.002569+0000", 2024-09-11T09:52:09.703 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.148e-05 2024-09-11T09:52:09.703 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.703 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.703 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:09.703 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.002647+0000", 2024-09-11T09:52:09.703 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.7570000000000004e-05 2024-09-11T09:52:09.704 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.704 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.704 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:09.704 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.002655+0000", 2024-09-11T09:52:09.704 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.4610000000000008e-06 2024-09-11T09:52:09.704 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.704 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.704 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:09.704 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.003497+0000", 2024-09-11T09:52:09.704 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00084237199999999998 2024-09-11T09:52:09.704 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:09.704 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:09.704 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:09.705 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.705 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.705 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "MOSDPGPush(2.4 82/81 [PushOp(2:239e915b:::rbd_data.1250f5e25b80.000000000000000c:6, version: 79'173, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:239e915b:::rbd_data.1250f5e25b80.000000000000000c:6@79'173, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 6=[]:{6: [6, 5, 3]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 4194304, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3145728, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-11T09:52:09.705 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:52:09.005094+0000", 2024-09-11T09:52:09.705 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 0.62899851399999995, 2024-09-11T09:52:09.705 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0024127419999999998, 2024-09-11T09:52:09.705 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:09.705 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:09.705 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "started", 2024-09-11T09:52:09.705 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:09.705 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.705 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:09.705 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.005094+0000", 2024-09-11T09:52:09.706 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:09.706 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.706 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.706 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:09.706 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.005094+0000", 2024-09-11T09:52:09.706 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:09.706 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.706 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.706 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:09.706 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.005095+0000", 2024-09-11T09:52:09.706 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.8999999999999997e-07 2024-09-11T09:52:09.706 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.706 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.706 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:09.707 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.006059+0000", 2024-09-11T09:52:09.707 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00096429100000000004 2024-09-11T09:52:09.707 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.707 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.707 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:09.707 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.006075+0000", 2024-09-11T09:52:09.707 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.6036e-05 2024-09-11T09:52:09.707 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.707 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.707 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:09.707 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.006088+0000", 2024-09-11T09:52:09.707 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.2557e-05 2024-09-11T09:52:09.707 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.708 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.708 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:09.708 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.006167+0000", 2024-09-11T09:52:09.708 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.8843000000000004e-05 2024-09-11T09:52:09.708 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.708 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.708 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:09.708 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.006182+0000", 2024-09-11T09:52:09.708 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.4968e-05 2024-09-11T09:52:09.708 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.708 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.708 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:09.708 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.007507+0000", 2024-09-11T09:52:09.709 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001325557 2024-09-11T09:52:09.709 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:09.709 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:09.709 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:09.709 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.709 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.709 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "MOSDPGPush(2.4 82/81 [PushOp(2:2167012f:::rbd_data.1250f5e25b80.000000000000000d:6, version: 79'175, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 1, recovery_info: ObjectRecoveryInfo(2:2167012f:::rbd_data.1250f5e25b80.000000000000000d:6@79'175, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 6=[]:{6: [6, 5, 3]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-11T09:52:09.709 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:52:09.009862+0000", 2024-09-11T09:52:09.709 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 0.62423105499999998, 2024-09-11T09:52:09.709 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001845522, 2024-09-11T09:52:09.709 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:09.709 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:09.709 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "started", 2024-09-11T09:52:09.709 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:09.710 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.710 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:09.710 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.009862+0000", 2024-09-11T09:52:09.710 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:09.710 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.710 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.710 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:09.710 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.009862+0000", 2024-09-11T09:52:09.710 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:09.710 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.711 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.711 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:09.711 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.009863+0000", 2024-09-11T09:52:09.711 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.1200000000000002e-07 2024-09-11T09:52:09.711 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.711 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.711 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:09.711 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.010797+0000", 2024-09-11T09:52:09.711 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00093439599999999999 2024-09-11T09:52:09.711 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.711 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.712 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:09.712 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.010814+0000", 2024-09-11T09:52:09.712 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.7354e-05 2024-09-11T09:52:09.712 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.712 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.712 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:09.712 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.010827+0000", 2024-09-11T09:52:09.712 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.3096e-05 2024-09-11T09:52:09.712 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.712 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.712 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:09.713 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.010894+0000", 2024-09-11T09:52:09.713 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.6501999999999994e-05 2024-09-11T09:52:09.713 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.713 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.713 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:09.713 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.010903+0000", 2024-09-11T09:52:09.713 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.0899999999999994e-06 2024-09-11T09:52:09.713 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.713 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.713 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:09.713 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.011707+0000", 2024-09-11T09:52:09.713 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00080437200000000003 2024-09-11T09:52:09.713 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:09.714 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:09.714 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:09.714 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.714 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.714 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "MOSDPGPush(2.4 82/81 [PushOp(2:2167012f:::rbd_data.1250f5e25b80.000000000000000d:6, version: 79'175, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:2167012f:::rbd_data.1250f5e25b80.000000000000000d:6@79'175, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 6=[]:{6: [6, 5, 3]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-11T09:52:09.714 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:52:09.013431+0000", 2024-09-11T09:52:09.714 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 0.62066217999999995, 2024-09-11T09:52:09.714 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.002012247, 2024-09-11T09:52:09.714 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:09.714 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:09.714 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "started", 2024-09-11T09:52:09.714 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:09.714 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.715 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:09.715 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.013431+0000", 2024-09-11T09:52:09.715 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:09.715 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.715 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.715 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:09.715 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.013431+0000", 2024-09-11T09:52:09.715 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:09.715 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.715 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.715 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:09.715 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.013432+0000", 2024-09-11T09:52:09.715 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.1900000000000001e-07 2024-09-11T09:52:09.715 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.716 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.716 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:09.716 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.014359+0000", 2024-09-11T09:52:09.716 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00092687900000000003 2024-09-11T09:52:09.716 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.716 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.716 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:09.716 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.014377+0000", 2024-09-11T09:52:09.716 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.8085e-05 2024-09-11T09:52:09.716 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.716 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.716 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:09.716 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.014391+0000", 2024-09-11T09:52:09.717 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.4099e-05 2024-09-11T09:52:09.717 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.717 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.717 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:09.717 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.014476+0000", 2024-09-11T09:52:09.717 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.4827999999999995e-05 2024-09-11T09:52:09.717 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.717 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.717 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:09.717 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.014489+0000", 2024-09-11T09:52:09.717 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.3169e-05 2024-09-11T09:52:09.717 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.718 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.718 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:09.718 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.015443+0000", 2024-09-11T09:52:09.718 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00095436800000000003 2024-09-11T09:52:09.718 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:09.718 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:09.718 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:09.718 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.718 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.718 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "MOSDPGPush(2.4 82/81 [PushOp(2:2167012f:::rbd_data.1250f5e25b80.000000000000000d:6, version: 79'175, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:2167012f:::rbd_data.1250f5e25b80.000000000000000d:6@79'175, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 6=[]:{6: [6, 5, 3]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3145728, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-11T09:52:09.718 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:52:09.017158+0000", 2024-09-11T09:52:09.718 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 0.61693519200000002, 2024-09-11T09:52:09.718 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0020239889999999999, 2024-09-11T09:52:09.719 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:09.719 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:09.719 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "started", 2024-09-11T09:52:09.719 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:09.719 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.719 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:09.719 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.017158+0000", 2024-09-11T09:52:09.719 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:09.719 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.719 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.719 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:09.719 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.017158+0000", 2024-09-11T09:52:09.719 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:09.720 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.720 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.720 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:09.720 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.017158+0000", 2024-09-11T09:52:09.720 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.7000000000000004e-07 2024-09-11T09:52:09.720 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.720 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.720 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:09.720 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.018136+0000", 2024-09-11T09:52:09.720 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00097795 2024-09-11T09:52:09.720 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.720 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.720 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:09.721 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.018152+0000", 2024-09-11T09:52:09.721 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.5208000000000001e-05 2024-09-11T09:52:09.721 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.721 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.721 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:09.721 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.018164+0000", 2024-09-11T09:52:09.721 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.2157000000000001e-05 2024-09-11T09:52:09.721 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.722 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.722 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:09.722 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.018245+0000", 2024-09-11T09:52:09.722 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.1284000000000003e-05 2024-09-11T09:52:09.722 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.722 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.722 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:09.722 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.018257+0000", 2024-09-11T09:52:09.722 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.2160999999999999e-05 2024-09-11T09:52:09.722 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.722 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.722 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:09.722 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.019182+0000", 2024-09-11T09:52:09.722 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00092455900000000004 2024-09-11T09:52:09.723 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:09.723 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:09.723 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:09.723 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.723 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.723 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "MOSDPGPush(2.4 82/81 [PushOp(2:2167012f:::rbd_data.1250f5e25b80.000000000000000d:6, version: 79'175, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:2167012f:::rbd_data.1250f5e25b80.000000000000000d:6@79'175, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 6=[]:{6: [6, 5, 3]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 4194304, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3145728, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-11T09:52:09.723 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:52:09.020895+0000", 2024-09-11T09:52:09.723 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 0.61319813899999998, 2024-09-11T09:52:09.723 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.002075234, 2024-09-11T09:52:09.723 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:09.723 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:09.724 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "started", 2024-09-11T09:52:09.724 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:09.724 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.724 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:09.724 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.020895+0000", 2024-09-11T09:52:09.724 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:09.724 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.724 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.724 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:09.724 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.020895+0000", 2024-09-11T09:52:09.724 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:09.724 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.724 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.725 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:09.725 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.020895+0000", 2024-09-11T09:52:09.725 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.0999999999999999e-07 2024-09-11T09:52:09.725 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.725 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.725 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:09.725 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.021855+0000", 2024-09-11T09:52:09.725 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00095982700000000001 2024-09-11T09:52:09.725 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.725 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.725 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:09.725 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.021870+0000", 2024-09-11T09:52:09.725 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.4609e-05 2024-09-11T09:52:09.726 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.726 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.726 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:09.726 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.021881+0000", 2024-09-11T09:52:09.726 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.083e-05 2024-09-11T09:52:09.726 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.726 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.726 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:09.727 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.021943+0000", 2024-09-11T09:52:09.727 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.2780999999999994e-05 2024-09-11T09:52:09.727 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.727 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.727 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:09.727 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.021952+0000", 2024-09-11T09:52:09.727 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.8629999999999997e-06 2024-09-11T09:52:09.727 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:09.727 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:09.727 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:09.727 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.022970+0000", 2024-09-11T09:52:09.727 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0010178139999999999 2024-09-11T09:52:09.727 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:09.728 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:09.728 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:09.728 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:09.728 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:09.728 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-11T09:52:09.729 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-11T09:52:09.730 INFO:tasks.ceph.osd.2.smithi017.stderr:2024-09-11T09:52:09.730+0000 7fd20b170640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31672) UID: 0 2024-09-11T09:52:09.829 INFO:tasks.ceph.osd.2.smithi017.stderr:2024-09-11T09:52:09.830+0000 7fd20b170640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31672) UID: 0 2024-09-11T09:52:09.843 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:52:09.843 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [], 2024-09-11T09:52:09.843 INFO:teuthology.orchestra.run.smithi017.stdout: "num_ops": 0 2024-09-11T09:52:09.843 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:52:09.851 INFO:teuthology.orchestra.run.smithi017.stderr:noscrub is unset 2024-09-11T09:52:09.855 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-11T09:52:09.952 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:52:09.953+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:52:10.015 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:52:10.016 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [], 2024-09-11T09:52:10.016 INFO:teuthology.orchestra.run.smithi017.stdout: "complaint_time": 30, 2024-09-11T09:52:10.016 INFO:teuthology.orchestra.run.smithi017.stdout: "num_blocked_ops": 0 2024-09-11T09:52:10.016 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:52:10.026 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-11T09:52:10.031 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:52:10.032+0000 7f3c96ec9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31909) UID: 0 2024-09-11T09:52:10.132 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:52:10.133+0000 7f3c96ec9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31909) UID: 0 2024-09-11T09:52:10.176 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:52:10.176 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 20, 2024-09-11T09:52:10.176 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 600, 2024-09-11T09:52:10.176 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [ 2024-09-11T09:52:10.176 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.176 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4688.0:34 2.4 e53/32 2:2367385a:::rbd_data.1250f5e25b80.0000000000000002:head v 53'6, mlcod=53'4)", 2024-09-11T09:52:10.176 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.907415+0000", 2024-09-11T09:52:10.177 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 47.268400145999998, 2024-09-11T09:52:10.177 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.011285778999999999, 2024-09-11T09:52:10.177 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:10.177 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:10.177 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:10.177 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:10.177 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.177 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:10.177 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.907415+0000", 2024-09-11T09:52:10.177 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:10.177 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.178 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.178 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:10.178 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.907415+0000", 2024-09-11T09:52:10.178 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:10.178 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.178 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.178 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:10.178 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.907417+0000", 2024-09-11T09:52:10.178 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.2910000000000002e-06 2024-09-11T09:52:10.178 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.178 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.178 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:10.179 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.910477+0000", 2024-09-11T09:52:10.179 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0030597829999999999 2024-09-11T09:52:10.179 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.179 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.179 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:10.179 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.910501+0000", 2024-09-11T09:52:10.179 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.3332e-05 2024-09-11T09:52:10.179 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.179 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.179 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:10.179 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.910514+0000", 2024-09-11T09:52:10.180 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3957e-05 2024-09-11T09:52:10.180 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.180 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.180 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:10.180 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.910612+0000", 2024-09-11T09:52:10.180 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.7075000000000001e-05 2024-09-11T09:52:10.180 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.180 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.180 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:10.180 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.910640+0000", 2024-09-11T09:52:10.180 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.8768999999999998e-05 2024-09-11T09:52:10.180 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.181 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.181 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:10.181 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.918640+0000", 2024-09-11T09:52:10.181 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0079991899999999998 2024-09-11T09:52:10.181 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.181 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.181 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:10.181 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.918701+0000", 2024-09-11T09:52:10.181 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.1382000000000003e-05 2024-09-11T09:52:10.181 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:10.182 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:10.182 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:10.182 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.182 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.182 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4688.0:38 2.5 e53/32 2:abde2694:::rbd_data.1250f5e25b80.0000000000000004:head v 53'1, mlcod=0'0)", 2024-09-11T09:52:10.182 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.915432+0000", 2024-09-11T09:52:10.182 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 47.260383765999997, 2024-09-11T09:52:10.182 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.015849652999999998, 2024-09-11T09:52:10.182 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:10.182 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:10.182 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:10.183 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:10.183 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.183 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:10.183 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.915432+0000", 2024-09-11T09:52:10.183 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:10.183 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.183 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.183 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:10.183 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.915432+0000", 2024-09-11T09:52:10.183 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:10.183 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.184 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.184 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:10.184 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.915432+0000", 2024-09-11T09:52:10.184 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.91e-07 2024-09-11T09:52:10.184 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.184 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.184 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:10.184 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.924943+0000", 2024-09-11T09:52:10.184 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0095110630000000002 2024-09-11T09:52:10.184 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.184 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.185 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:10.185 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.924960+0000", 2024-09-11T09:52:10.185 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6915999999999999e-05 2024-09-11T09:52:10.185 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.185 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.185 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:10.185 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.924968+0000", 2024-09-11T09:52:10.185 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.6999999999999997e-06 2024-09-11T09:52:10.185 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.185 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.185 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:10.186 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.925025+0000", 2024-09-11T09:52:10.186 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.6949e-05 2024-09-11T09:52:10.186 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.186 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.186 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:10.186 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.925043+0000", 2024-09-11T09:52:10.186 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7784000000000001e-05 2024-09-11T09:52:10.186 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.186 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.186 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:10.187 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.931237+0000", 2024-09-11T09:52:10.187 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0061935389999999996 2024-09-11T09:52:10.187 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.187 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.187 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:10.187 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.931281+0000", 2024-09-11T09:52:10.187 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.4511000000000001e-05 2024-09-11T09:52:10.187 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:10.187 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:10.187 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:10.187 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.188 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.188 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:40 2.7 2:fe41411c:::rbd_object_map.1250f5e25b80:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:10.188 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.919652+0000", 2024-09-11T09:52:10.188 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 47.256163098999998, 2024-09-11T09:52:10.188 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.012187073, 2024-09-11T09:52:10.188 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:10.188 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:10.188 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:10.188 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:10.188 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:52:10.188 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:52:10.189 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 40 2024-09-11T09:52:10.189 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.189 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:10.189 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.189 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:10.189 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919652+0000", 2024-09-11T09:52:10.189 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:10.189 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.189 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.189 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:10.189 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919652+0000", 2024-09-11T09:52:10.189 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:10.190 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.190 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.190 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:10.190 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919654+0000", 2024-09-11T09:52:10.190 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.88e-06 2024-09-11T09:52:10.190 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.190 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.190 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:10.190 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919669+0000", 2024-09-11T09:52:10.190 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.5271e-05 2024-09-11T09:52:10.190 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.191 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.191 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:10.191 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919687+0000", 2024-09-11T09:52:10.191 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7448999999999998e-05 2024-09-11T09:52:10.191 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.191 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.191 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:10.191 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919695+0000", 2024-09-11T09:52:10.191 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.1149999999999994e-06 2024-09-11T09:52:10.191 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.191 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.192 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:10.192 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919757+0000", 2024-09-11T09:52:10.192 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.2348999999999995e-05 2024-09-11T09:52:10.192 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.192 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.192 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:10.192 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919875+0000", 2024-09-11T09:52:10.192 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00011750499999999999 2024-09-11T09:52:10.192 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.192 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.192 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 5", 2024-09-11T09:52:10.193 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.920201+0000", 2024-09-11T09:52:10.193 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00032628099999999999 2024-09-11T09:52:10.193 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.193 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.193 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:10.193 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.922089+0000", 2024-09-11T09:52:10.193 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.001887978 2024-09-11T09:52:10.193 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.193 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.193 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:10.193 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.931752+0000", 2024-09-11T09:52:10.193 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0096626309999999997 2024-09-11T09:52:10.194 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.194 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.194 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:10.194 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.931794+0000", 2024-09-11T09:52:10.194 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.2849999999999998e-05 2024-09-11T09:52:10.194 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.194 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.194 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:10.194 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.931839+0000", 2024-09-11T09:52:10.194 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.4764e-05 2024-09-11T09:52:10.194 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:10.195 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:10.195 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:10.195 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.195 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.195 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:41 2.7 2:fe41411c:::rbd_object_map.1250f5e25b80:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:10.195 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.919716+0000", 2024-09-11T09:52:10.195 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 47.256099255000002, 2024-09-11T09:52:10.195 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.015641605999999999, 2024-09-11T09:52:10.195 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:10.195 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:10.196 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:10.196 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:10.196 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:52:10.196 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:52:10.196 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 41 2024-09-11T09:52:10.196 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.196 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:10.196 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.196 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:10.196 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919716+0000", 2024-09-11T09:52:10.196 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:10.196 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.197 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.197 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:10.197 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919716+0000", 2024-09-11T09:52:10.197 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:10.197 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.197 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.197 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:10.197 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919716+0000", 2024-09-11T09:52:10.197 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.9999999999999998e-07 2024-09-11T09:52:10.197 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.197 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.198 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:10.198 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919720+0000", 2024-09-11T09:52:10.198 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.6440000000000002e-06 2024-09-11T09:52:10.198 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.198 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.198 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:10.198 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919729+0000", 2024-09-11T09:52:10.198 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.1759999999999999e-06 2024-09-11T09:52:10.198 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.198 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.198 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:10.199 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919736+0000", 2024-09-11T09:52:10.199 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.5390000000000004e-06 2024-09-11T09:52:10.199 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.199 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.199 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:10.199 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.920669+0000", 2024-09-11T09:52:10.199 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00093275599999999997 2024-09-11T09:52:10.199 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.199 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.199 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:10.199 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.920797+0000", 2024-09-11T09:52:10.200 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00012893400000000001 2024-09-11T09:52:10.200 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.200 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.200 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 5", 2024-09-11T09:52:10.200 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.921269+0000", 2024-09-11T09:52:10.200 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00047128600000000001 2024-09-11T09:52:10.200 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.200 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.200 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:10.200 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.923198+0000", 2024-09-11T09:52:10.200 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0019289369999999999 2024-09-11T09:52:10.201 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.201 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.201 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:10.201 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.935300+0000", 2024-09-11T09:52:10.201 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.012102392 2024-09-11T09:52:10.201 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.201 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.201 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:10.201 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.935332+0000", 2024-09-11T09:52:10.201 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.2292000000000002e-05 2024-09-11T09:52:10.201 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.202 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.202 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:10.202 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.935358+0000", 2024-09-11T09:52:10.202 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.525e-05 2024-09-11T09:52:10.202 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:10.202 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:10.202 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:10.202 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.202 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.202 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:48 2.7 2:fe41411c:::rbd_object_map.1250f5e25b80:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:10.203 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.938799+0000", 2024-09-11T09:52:10.203 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 47.237016146000002, 2024-09-11T09:52:10.203 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0041642340000000002, 2024-09-11T09:52:10.203 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:10.203 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:10.203 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:10.203 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:10.203 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:52:10.203 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:52:10.203 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 48 2024-09-11T09:52:10.203 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.204 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:10.204 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.204 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:10.204 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.938799+0000", 2024-09-11T09:52:10.204 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:10.204 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.204 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.204 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:10.204 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.938799+0000", 2024-09-11T09:52:10.204 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:10.204 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.204 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.205 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:10.205 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.938800+0000", 2024-09-11T09:52:10.205 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.71e-07 2024-09-11T09:52:10.205 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.205 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.205 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:10.205 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.938805+0000", 2024-09-11T09:52:10.205 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.3140000000000003e-06 2024-09-11T09:52:10.205 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.205 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.205 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:10.206 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.938816+0000", 2024-09-11T09:52:10.206 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0231000000000001e-05 2024-09-11T09:52:10.206 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.206 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.206 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:10.206 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.938822+0000", 2024-09-11T09:52:10.206 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.9550000000000003e-06 2024-09-11T09:52:10.206 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.206 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.206 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:10.206 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.938868+0000", 2024-09-11T09:52:10.206 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.5562999999999997e-05 2024-09-11T09:52:10.207 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.207 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.207 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:10.207 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.938918+0000", 2024-09-11T09:52:10.207 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.0317e-05 2024-09-11T09:52:10.207 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.207 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.207 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 5", 2024-09-11T09:52:10.207 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.939114+0000", 2024-09-11T09:52:10.207 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00019538300000000001 2024-09-11T09:52:10.207 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.207 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.208 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:10.208 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.939663+0000", 2024-09-11T09:52:10.208 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00054932199999999998 2024-09-11T09:52:10.208 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.208 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.208 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:10.208 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.942910+0000", 2024-09-11T09:52:10.208 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0032473839999999999 2024-09-11T09:52:10.208 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.208 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.209 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:10.209 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.942940+0000", 2024-09-11T09:52:10.209 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.9306999999999999e-05 2024-09-11T09:52:10.209 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.209 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.209 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:10.209 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.942963+0000", 2024-09-11T09:52:10.209 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.3587e-05 2024-09-11T09:52:10.209 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:10.209 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:10.209 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:10.209 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.210 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.210 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4688.0:49 2.5 e53/32 2:b6dd9fc9:::rbd_data.1250f5e25b80.0000000000000009:head v 53'2, mlcod=0'0)", 2024-09-11T09:52:10.210 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.944712+0000", 2024-09-11T09:52:10.210 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 47.231103306999998, 2024-09-11T09:52:10.210 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.016185069999999999, 2024-09-11T09:52:10.210 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:10.210 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:10.210 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:10.210 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:10.210 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.210 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:10.210 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.944712+0000", 2024-09-11T09:52:10.211 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:10.211 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.211 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.211 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:10.211 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.944712+0000", 2024-09-11T09:52:10.211 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:10.211 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.211 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.211 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:10.211 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.944713+0000", 2024-09-11T09:52:10.211 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0139999999999999e-06 2024-09-11T09:52:10.212 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.212 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.212 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:10.212 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.955518+0000", 2024-09-11T09:52:10.212 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.01080542 2024-09-11T09:52:10.212 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.212 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.212 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:10.212 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.955536+0000", 2024-09-11T09:52:10.213 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7818000000000001e-05 2024-09-11T09:52:10.213 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.213 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.213 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:10.213 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.955544+0000", 2024-09-11T09:52:10.213 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.1699999999999997e-06 2024-09-11T09:52:10.213 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.213 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.214 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:10.214 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.955607+0000", 2024-09-11T09:52:10.214 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.2451000000000002e-05 2024-09-11T09:52:10.214 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.214 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.214 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:10.214 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.955632+0000", 2024-09-11T09:52:10.214 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.5335000000000001e-05 2024-09-11T09:52:10.214 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.214 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.214 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:10.215 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.960826+0000", 2024-09-11T09:52:10.215 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0051938560000000002 2024-09-11T09:52:10.215 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.215 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.215 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:10.215 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.960897+0000", 2024-09-11T09:52:10.215 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.1006000000000004e-05 2024-09-11T09:52:10.215 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:10.215 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:10.215 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:10.215 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.216 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.216 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:53 2.7 2:fe41411c:::rbd_object_map.1250f5e25b80:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:10.216 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.951291+0000", 2024-09-11T09:52:10.216 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 47.224524316999997, 2024-09-11T09:52:10.216 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0044964999999999996, 2024-09-11T09:52:10.216 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:10.216 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:10.216 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:10.216 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:10.216 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:52:10.216 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:52:10.217 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 53 2024-09-11T09:52:10.217 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.217 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:10.217 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.217 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:10.217 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951291+0000", 2024-09-11T09:52:10.217 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:10.217 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.217 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.217 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:10.217 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951291+0000", 2024-09-11T09:52:10.218 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:10.218 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.218 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.218 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:10.218 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951293+0000", 2024-09-11T09:52:10.218 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7889999999999999e-06 2024-09-11T09:52:10.218 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.218 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.218 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:10.218 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951302+0000", 2024-09-11T09:52:10.218 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.9819999999999997e-06 2024-09-11T09:52:10.219 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.219 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.219 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:10.219 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951324+0000", 2024-09-11T09:52:10.219 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.2141e-05 2024-09-11T09:52:10.219 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.219 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.219 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:10.219 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951338+0000", 2024-09-11T09:52:10.219 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3750999999999999e-05 2024-09-11T09:52:10.219 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.219 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.220 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:10.220 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951409+0000", 2024-09-11T09:52:10.220 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.1658999999999999e-05 2024-09-11T09:52:10.220 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.220 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.220 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:10.220 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951517+0000", 2024-09-11T09:52:10.220 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00010746799999999999 2024-09-11T09:52:10.220 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.220 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.220 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 5", 2024-09-11T09:52:10.221 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951942+0000", 2024-09-11T09:52:10.221 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00042524300000000001 2024-09-11T09:52:10.221 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.221 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.221 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:10.221 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.952872+0000", 2024-09-11T09:52:10.221 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00092976300000000001 2024-09-11T09:52:10.221 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.221 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.221 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:10.221 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.955682+0000", 2024-09-11T09:52:10.221 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0028099459999999998 2024-09-11T09:52:10.222 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.222 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.222 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:10.222 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.955746+0000", 2024-09-11T09:52:10.222 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.4084999999999994e-05 2024-09-11T09:52:10.222 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.222 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.222 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:10.222 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.955787+0000", 2024-09-11T09:52:10.222 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.1672999999999999e-05 2024-09-11T09:52:10.223 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:10.223 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:10.223 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:10.223 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.223 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.223 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4688.0:52 2.4 e53/32 2:26d5c2f0:::rbd_data.1250f5e25b80.000000000000000b:head v 53'8, mlcod=53'6)", 2024-09-11T09:52:10.223 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.963677+0000", 2024-09-11T09:52:10.223 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 47.212138762000002, 2024-09-11T09:52:10.223 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0089437969999999999, 2024-09-11T09:52:10.223 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:10.223 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:10.223 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:10.224 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:10.224 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.224 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:10.224 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.963677+0000", 2024-09-11T09:52:10.224 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:10.224 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.224 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.224 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:10.224 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.963677+0000", 2024-09-11T09:52:10.224 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:10.224 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.225 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.225 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:10.225 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.963677+0000", 2024-09-11T09:52:10.225 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.7199999999999998e-07 2024-09-11T09:52:10.225 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.225 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.225 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:10.225 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.965506+0000", 2024-09-11T09:52:10.225 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.001829064 2024-09-11T09:52:10.225 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.226 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.226 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:10.226 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.965538+0000", 2024-09-11T09:52:10.226 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.1306999999999997e-05 2024-09-11T09:52:10.226 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.226 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.226 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:10.226 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.965553+0000", 2024-09-11T09:52:10.226 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.4924e-05 2024-09-11T09:52:10.226 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.226 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.227 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:10.227 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.965663+0000", 2024-09-11T09:52:10.227 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000110315 2024-09-11T09:52:10.227 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.227 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.227 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:10.227 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.965692+0000", 2024-09-11T09:52:10.227 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.934e-05 2024-09-11T09:52:10.227 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.227 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.227 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:10.228 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.972537+0000", 2024-09-11T09:52:10.228 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0068444969999999997 2024-09-11T09:52:10.228 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.228 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.228 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:10.228 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.972620+0000", 2024-09-11T09:52:10.228 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.3578000000000005e-05 2024-09-11T09:52:10.228 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:10.228 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:10.228 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:10.229 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.229 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.229 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4688.0:54 2.4 e53/32 2:239e915b:::rbd_data.1250f5e25b80.000000000000000c:head v 53'9, mlcod=53'7)", 2024-09-11T09:52:10.229 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.987028+0000", 2024-09-11T09:52:10.229 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 47.188786845000003, 2024-09-11T09:52:10.229 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0094155650000000007, 2024-09-11T09:52:10.229 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:10.229 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:10.229 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:10.229 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:10.230 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.230 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:10.230 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.987028+0000", 2024-09-11T09:52:10.230 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:10.230 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.230 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.230 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:10.230 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.987028+0000", 2024-09-11T09:52:10.230 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:10.230 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.230 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.231 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:10.231 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.987030+0000", 2024-09-11T09:52:10.231 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6130000000000001e-06 2024-09-11T09:52:10.231 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.231 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.231 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:10.231 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.988575+0000", 2024-09-11T09:52:10.231 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.001545193 2024-09-11T09:52:10.231 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.231 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.231 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:10.232 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.988604+0000", 2024-09-11T09:52:10.232 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.9173e-05 2024-09-11T09:52:10.232 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.232 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.232 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:10.232 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.988620+0000", 2024-09-11T09:52:10.232 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.5517999999999999e-05 2024-09-11T09:52:10.232 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.232 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.232 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:10.233 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.988704+0000", 2024-09-11T09:52:10.233 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.3588e-05 2024-09-11T09:52:10.233 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.233 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.233 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:10.233 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.988735+0000", 2024-09-11T09:52:10.233 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.1643000000000002e-05 2024-09-11T09:52:10.234 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.234 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.234 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:10.234 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.996383+0000", 2024-09-11T09:52:10.234 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0076478099999999997 2024-09-11T09:52:10.234 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.234 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.234 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:10.234 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.996444+0000", 2024-09-11T09:52:10.234 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.1026999999999997e-05 2024-09-11T09:52:10.235 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:10.235 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:10.235 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:10.235 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.235 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.235 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4688.0:56 2.4 e53/32 2:2167012f:::rbd_data.1250f5e25b80.000000000000000d:head v 53'10, mlcod=53'7)", 2024-09-11T09:52:10.235 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.993894+0000", 2024-09-11T09:52:10.235 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 47.181921303000003, 2024-09-11T09:52:10.235 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0099268559999999995, 2024-09-11T09:52:10.235 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:10.235 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:10.236 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:10.236 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:10.236 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.236 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:10.236 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.993894+0000", 2024-09-11T09:52:10.236 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:10.236 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.236 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.236 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:10.236 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.993894+0000", 2024-09-11T09:52:10.236 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:10.237 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.237 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.237 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:10.237 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.993896+0000", 2024-09-11T09:52:10.237 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6249999999999999e-06 2024-09-11T09:52:10.237 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.237 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.237 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:10.237 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.997606+0000", 2024-09-11T09:52:10.237 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.003710737 2024-09-11T09:52:10.237 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.237 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.238 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:10.238 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.997628+0000", 2024-09-11T09:52:10.238 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.1971e-05 2024-09-11T09:52:10.238 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.238 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.238 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:10.238 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.997640+0000", 2024-09-11T09:52:10.238 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1481000000000001e-05 2024-09-11T09:52:10.238 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.238 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.238 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:10.238 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.997712+0000", 2024-09-11T09:52:10.239 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.2255000000000005e-05 2024-09-11T09:52:10.239 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.239 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.239 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:10.239 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.997738+0000", 2024-09-11T09:52:10.239 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.6050999999999998e-05 2024-09-11T09:52:10.239 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.239 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.239 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:10.239 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.003779+0000", 2024-09-11T09:52:10.239 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0060410960000000001 2024-09-11T09:52:10.240 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.240 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.240 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:10.240 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.003821+0000", 2024-09-11T09:52:10.240 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.1640000000000001e-05 2024-09-11T09:52:10.240 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:10.240 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:10.240 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:10.240 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.240 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.240 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "pg_update_log_missing(2.4 epoch 53/32 rep_tid 16 entries 53'11 (0'0) error 2:2906e6a7:::rbd_header.1250f5e25b80:head by client.4688.0:59 0.000000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 53'7)", 2024-09-11T09:52:10.240 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.998956+0000", 2024-09-11T09:52:10.241 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 47.176859133999997, 2024-09-11T09:52:10.241 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.004934127, 2024-09-11T09:52:10.241 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:10.241 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:10.241 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "reached pg", 2024-09-11T09:52:10.241 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:10.241 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.241 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:10.241 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.998956+0000", 2024-09-11T09:52:10.241 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:10.241 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.242 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.242 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:10.242 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.998956+0000", 2024-09-11T09:52:10.242 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:10.242 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.242 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.242 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:10.242 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.998957+0000", 2024-09-11T09:52:10.242 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.139e-06 2024-09-11T09:52:10.242 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.242 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.243 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:10.243 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.998985+0000", 2024-09-11T09:52:10.243 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.7912e-05 2024-09-11T09:52:10.243 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.243 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.243 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:10.243 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.999011+0000", 2024-09-11T09:52:10.243 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.6061e-05 2024-09-11T09:52:10.243 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.243 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.243 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:10.243 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.999028+0000", 2024-09-11T09:52:10.244 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7065999999999999e-05 2024-09-11T09:52:10.244 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.244 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.244 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:10.244 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.001919+0000", 2024-09-11T09:52:10.244 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.002891167 2024-09-11T09:52:10.244 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.244 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.244 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:10.244 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.003890+0000", 2024-09-11T09:52:10.244 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0019707819999999999 2024-09-11T09:52:10.245 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:10.245 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:10.245 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:10.245 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.245 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.245 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4694.0:34 2.7 2:f80bfc0c:ns::rbd_data.1256a0c02879.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:10.245 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.085183+0000", 2024-09-11T09:52:10.245 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 47.090632620999997, 2024-09-11T09:52:10.245 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.020908599, 2024-09-11T09:52:10.245 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:10.245 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:10.245 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:10.246 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:10.246 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4694", 2024-09-11T09:52:10.246 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:52:10.246 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 34 2024-09-11T09:52:10.246 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.246 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:10.246 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.246 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:10.246 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.085183+0000", 2024-09-11T09:52:10.246 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:10.246 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.247 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.247 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:10.247 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.085183+0000", 2024-09-11T09:52:10.247 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:10.247 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.247 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.247 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:10.247 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.085183+0000", 2024-09-11T09:52:10.247 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.4300000000000002e-07 2024-09-11T09:52:10.247 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.247 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.248 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:10.248 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094337+0000", 2024-09-11T09:52:10.248 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0091532769999999996 2024-09-11T09:52:10.248 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.248 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.248 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:10.248 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094362+0000", 2024-09-11T09:52:10.248 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.5287999999999999e-05 2024-09-11T09:52:10.248 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.248 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.248 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:10.249 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094375+0000", 2024-09-11T09:52:10.249 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2446000000000001e-05 2024-09-11T09:52:10.249 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.249 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.249 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:10.249 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094455+0000", 2024-09-11T09:52:10.249 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.0092999999999994e-05 2024-09-11T09:52:10.249 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.249 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.249 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:10.249 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094618+0000", 2024-09-11T09:52:10.250 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000163747 2024-09-11T09:52:10.250 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.250 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.250 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 5", 2024-09-11T09:52:10.250 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094783+0000", 2024-09-11T09:52:10.250 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000165038 2024-09-11T09:52:10.250 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.250 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.250 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:10.250 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.105853+0000", 2024-09-11T09:52:10.250 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.011069294 2024-09-11T09:52:10.250 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.250 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.251 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:10.251 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.106004+0000", 2024-09-11T09:52:10.251 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00015123900000000001 2024-09-11T09:52:10.251 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.251 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.251 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:10.251 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.106062+0000", 2024-09-11T09:52:10.251 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.8205000000000003e-05 2024-09-11T09:52:10.251 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.251 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.251 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:10.252 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.106091+0000", 2024-09-11T09:52:10.252 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.9128999999999998e-05 2024-09-11T09:52:10.252 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:10.252 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:10.252 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:10.252 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.252 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.252 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4694.0:35 2.7 2:e8766325:ns::rbd_object_map.1256a0c02879:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:10.252 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.094405+0000", 2024-09-11T09:52:10.252 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 47.081409985000001, 2024-09-11T09:52:10.252 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.012623564, 2024-09-11T09:52:10.252 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:10.253 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:10.253 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:10.253 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:10.253 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4694", 2024-09-11T09:52:10.253 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:52:10.253 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 35 2024-09-11T09:52:10.253 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.253 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:10.253 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.253 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:10.254 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094405+0000", 2024-09-11T09:52:10.254 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:10.254 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.254 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.254 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:10.254 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094405+0000", 2024-09-11T09:52:10.254 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:10.254 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.254 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.254 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:10.254 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094407+0000", 2024-09-11T09:52:10.254 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.739e-06 2024-09-11T09:52:10.255 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.255 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.255 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:10.255 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094411+0000", 2024-09-11T09:52:10.255 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.3340000000000003e-06 2024-09-11T09:52:10.255 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.255 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.255 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:10.255 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094419+0000", 2024-09-11T09:52:10.255 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.8909999999999995e-06 2024-09-11T09:52:10.255 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.255 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.256 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:10.256 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094424+0000", 2024-09-11T09:52:10.256 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.7639999999999996e-06 2024-09-11T09:52:10.256 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.256 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.256 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:10.256 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.100209+0000", 2024-09-11T09:52:10.256 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0057850489999999996 2024-09-11T09:52:10.256 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.256 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.256 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:10.256 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.100341+0000", 2024-09-11T09:52:10.257 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000131438 2024-09-11T09:52:10.257 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.257 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.257 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 5", 2024-09-11T09:52:10.257 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.100805+0000", 2024-09-11T09:52:10.257 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00046471499999999998 2024-09-11T09:52:10.257 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.257 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.257 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:10.257 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.106148+0000", 2024-09-11T09:52:10.257 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0053426460000000004 2024-09-11T09:52:10.258 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.258 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.258 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:10.258 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.106929+0000", 2024-09-11T09:52:10.258 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00078161300000000001 2024-09-11T09:52:10.258 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.258 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.258 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:10.258 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.106985+0000", 2024-09-11T09:52:10.258 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.5223999999999998e-05 2024-09-11T09:52:10.258 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.259 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.259 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:10.259 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.107029+0000", 2024-09-11T09:52:10.259 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.4150999999999998e-05 2024-09-11T09:52:10.259 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:10.259 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:10.259 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:10.259 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.259 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.259 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4694.0:36 2.7 2:e8766325:ns::rbd_object_map.1256a0c02879:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:10.259 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.094452+0000", 2024-09-11T09:52:10.259 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 47.081362974999998, 2024-09-11T09:52:10.259 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.012680383, 2024-09-11T09:52:10.260 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:10.260 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:10.260 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:10.260 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:10.260 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4694", 2024-09-11T09:52:10.260 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:52:10.260 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 36 2024-09-11T09:52:10.260 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.260 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:10.261 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.261 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:10.261 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094452+0000", 2024-09-11T09:52:10.261 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:10.261 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.261 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.261 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:10.261 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094452+0000", 2024-09-11T09:52:10.261 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:10.261 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.262 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.262 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:10.262 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094453+0000", 2024-09-11T09:52:10.262 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.1800000000000002e-07 2024-09-11T09:52:10.262 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.262 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.262 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:10.262 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094455+0000", 2024-09-11T09:52:10.262 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.7020000000000001e-06 2024-09-11T09:52:10.262 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.262 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.263 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:10.263 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094463+0000", 2024-09-11T09:52:10.263 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.1999999999999997e-06 2024-09-11T09:52:10.263 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.263 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.263 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:10.263 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094467+0000", 2024-09-11T09:52:10.263 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.729e-06 2024-09-11T09:52:10.263 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.263 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.263 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:10.264 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.101269+0000", 2024-09-11T09:52:10.264 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0068018799999999997 2024-09-11T09:52:10.264 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.264 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.264 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:10.264 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.101353+0000", 2024-09-11T09:52:10.264 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.3827999999999998e-05 2024-09-11T09:52:10.264 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.264 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.264 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 5", 2024-09-11T09:52:10.264 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.101588+0000", 2024-09-11T09:52:10.265 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00023467099999999999 2024-09-11T09:52:10.265 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.265 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.265 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:10.265 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.106182+0000", 2024-09-11T09:52:10.265 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0045939489999999999 2024-09-11T09:52:10.265 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.265 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.265 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:10.265 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.107041+0000", 2024-09-11T09:52:10.265 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00085965800000000002 2024-09-11T09:52:10.265 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.266 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.266 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:10.266 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.107084+0000", 2024-09-11T09:52:10.266 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.261e-05 2024-09-11T09:52:10.266 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.266 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.266 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:10.266 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.107133+0000", 2024-09-11T09:52:10.266 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.8838000000000002e-05 2024-09-11T09:52:10.266 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:10.266 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:10.266 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:10.267 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.267 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.267 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4694.0:37 2.7 2:e8766325:ns::rbd_object_map.1256a0c02879:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:10.267 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.094473+0000", 2024-09-11T09:52:10.267 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 47.081341823999999, 2024-09-11T09:52:10.267 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.012778484999999999, 2024-09-11T09:52:10.267 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:10.267 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:10.267 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:10.267 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:10.267 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4694", 2024-09-11T09:52:10.268 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:52:10.268 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 37 2024-09-11T09:52:10.268 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.268 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:10.268 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.268 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:10.268 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094473+0000", 2024-09-11T09:52:10.268 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:10.268 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.268 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.268 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:10.269 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094473+0000", 2024-09-11T09:52:10.269 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:10.269 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.269 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.269 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:10.269 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094474+0000", 2024-09-11T09:52:10.269 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.9699999999999996e-07 2024-09-11T09:52:10.269 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.269 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.269 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:10.270 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094477+0000", 2024-09-11T09:52:10.270 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.151e-06 2024-09-11T09:52:10.270 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.270 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.270 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:10.270 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094483+0000", 2024-09-11T09:52:10.270 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.6330000000000001e-06 2024-09-11T09:52:10.270 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.270 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.270 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:10.270 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094487+0000", 2024-09-11T09:52:10.270 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.5399999999999997e-06 2024-09-11T09:52:10.271 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.271 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.271 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:10.271 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.101866+0000", 2024-09-11T09:52:10.271 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.007378768 2024-09-11T09:52:10.271 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.271 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.271 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:10.271 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.101960+0000", 2024-09-11T09:52:10.271 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.3861000000000001e-05 2024-09-11T09:52:10.271 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.272 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.272 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 5", 2024-09-11T09:52:10.272 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.102197+0000", 2024-09-11T09:52:10.272 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00023723200000000001 2024-09-11T09:52:10.272 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.272 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.272 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:10.272 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.106222+0000", 2024-09-11T09:52:10.272 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0040249689999999998 2024-09-11T09:52:10.272 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.272 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.273 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:10.273 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.107141+0000", 2024-09-11T09:52:10.273 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00091913900000000002 2024-09-11T09:52:10.273 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.273 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.273 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:10.273 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.107198+0000", 2024-09-11T09:52:10.273 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.6564999999999998e-05 2024-09-11T09:52:10.273 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.273 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.273 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:10.274 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.107252+0000", 2024-09-11T09:52:10.274 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.4030000000000003e-05 2024-09-11T09:52:10.274 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:10.274 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:10.274 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:10.274 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.274 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.274 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4694.0:33 2.4 e53/32 2:3bfc818c:ns::rbd_data.1256a0c02879.0000000000000001:head v 53'14, mlcod=53'12)", 2024-09-11T09:52:10.274 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.096496+0000", 2024-09-11T09:52:10.274 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 47.079319251000001, 2024-09-11T09:52:10.274 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0099139310000000008, 2024-09-11T09:52:10.274 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:10.275 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:10.276 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:10.276 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:10.276 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.276 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:10.276 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.096496+0000", 2024-09-11T09:52:10.277 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:10.277 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.277 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.277 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:10.277 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.096496+0000", 2024-09-11T09:52:10.277 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:10.277 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.277 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.277 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:10.277 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.096498+0000", 2024-09-11T09:52:10.277 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6309999999999999e-06 2024-09-11T09:52:10.277 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.278 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.278 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:10.278 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.098087+0000", 2024-09-11T09:52:10.278 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0015889019999999999 2024-09-11T09:52:10.278 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.278 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.278 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:10.278 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.098112+0000", 2024-09-11T09:52:10.278 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.5369000000000001e-05 2024-09-11T09:52:10.278 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.278 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.279 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:10.279 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.098127+0000", 2024-09-11T09:52:10.279 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.5218999999999999e-05 2024-09-11T09:52:10.279 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.279 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.279 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:10.279 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.098209+0000", 2024-09-11T09:52:10.279 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.2120999999999995e-05 2024-09-11T09:52:10.279 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.279 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.279 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:10.279 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.098237+0000", 2024-09-11T09:52:10.280 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.7657999999999998e-05 2024-09-11T09:52:10.280 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.280 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.280 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:10.280 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.106376+0000", 2024-09-11T09:52:10.280 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0081386749999999997 2024-09-11T09:52:10.280 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.280 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.280 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:10.280 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.106410+0000", 2024-09-11T09:52:10.280 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.4356000000000001e-05 2024-09-11T09:52:10.281 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:10.281 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:10.281 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:10.281 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.281 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.281 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4694.0:49 2.7 2:e8766325:ns::rbd_object_map.1256a0c02879:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:10.281 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.134806+0000", 2024-09-11T09:52:10.281 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 47.041009182000003, 2024-09-11T09:52:10.281 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0061629049999999998, 2024-09-11T09:52:10.281 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:10.282 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:10.282 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:10.282 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:10.282 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4694", 2024-09-11T09:52:10.282 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:52:10.282 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 49 2024-09-11T09:52:10.282 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.282 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:10.282 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.282 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:10.282 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.134806+0000", 2024-09-11T09:52:10.283 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:10.283 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.283 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.283 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:10.283 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.134806+0000", 2024-09-11T09:52:10.283 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:10.283 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.283 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.283 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:10.283 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.134808+0000", 2024-09-11T09:52:10.283 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.9870000000000002e-06 2024-09-11T09:52:10.284 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.284 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.284 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:10.284 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.134818+0000", 2024-09-11T09:52:10.284 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.5599999999999999e-06 2024-09-11T09:52:10.284 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.284 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.284 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:10.284 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.134835+0000", 2024-09-11T09:52:10.284 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7399000000000001e-05 2024-09-11T09:52:10.284 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.284 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.285 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:10.285 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.134845+0000", 2024-09-11T09:52:10.285 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0285000000000001e-05 2024-09-11T09:52:10.285 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.285 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.285 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:10.285 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.134911+0000", 2024-09-11T09:52:10.285 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.5654999999999994e-05 2024-09-11T09:52:10.285 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.285 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.285 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:10.286 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.135023+0000", 2024-09-11T09:52:10.286 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00011186799999999999 2024-09-11T09:52:10.286 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.286 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.286 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 5", 2024-09-11T09:52:10.286 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.135290+0000", 2024-09-11T09:52:10.286 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00026725300000000001 2024-09-11T09:52:10.286 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.286 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.286 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:10.286 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.136506+0000", 2024-09-11T09:52:10.287 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0012161139999999999 2024-09-11T09:52:10.287 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.287 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.287 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:10.287 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.140884+0000", 2024-09-11T09:52:10.287 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0043781990000000002 2024-09-11T09:52:10.287 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.287 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.287 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:10.287 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.140923+0000", 2024-09-11T09:52:10.288 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.8813999999999998e-05 2024-09-11T09:52:10.288 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.288 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.288 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:10.288 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.140969+0000", 2024-09-11T09:52:10.288 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.5770999999999999e-05 2024-09-11T09:52:10.288 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:10.288 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:10.288 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:10.288 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.288 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.289 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4694.0:42 2.5 e53/32 2:a4fc00ec:ns::rbd_data.1256a0c02879.0000000000000004:head v 53'3, mlcod=53'1)", 2024-09-11T09:52:10.289 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.136646+0000", 2024-09-11T09:52:10.289 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 47.039168875999998, 2024-09-11T09:52:10.289 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.012545501000000001, 2024-09-11T09:52:10.289 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:10.289 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:10.289 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:10.289 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:10.289 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.289 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:10.289 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.136646+0000", 2024-09-11T09:52:10.289 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:10.290 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.290 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.290 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:10.290 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.136646+0000", 2024-09-11T09:52:10.290 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:10.290 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.290 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.290 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:10.290 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.136647+0000", 2024-09-11T09:52:10.290 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.2900000000000002e-07 2024-09-11T09:52:10.290 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.290 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.291 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:10.291 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.143620+0000", 2024-09-11T09:52:10.291 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0069727440000000003 2024-09-11T09:52:10.291 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.291 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.291 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:10.291 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.143642+0000", 2024-09-11T09:52:10.291 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.1846000000000001e-05 2024-09-11T09:52:10.291 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.291 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.291 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:10.292 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.143654+0000", 2024-09-11T09:52:10.292 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2081e-05 2024-09-11T09:52:10.292 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.292 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.292 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:10.292 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.143717+0000", 2024-09-11T09:52:10.292 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.3443999999999998e-05 2024-09-11T09:52:10.292 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.292 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.292 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:10.292 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.143740+0000", 2024-09-11T09:52:10.293 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.2626999999999999e-05 2024-09-11T09:52:10.293 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.293 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.293 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:10.293 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.149142+0000", 2024-09-11T09:52:10.293 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0054017630000000004 2024-09-11T09:52:10.293 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.293 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.293 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:10.293 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.149192+0000", 2024-09-11T09:52:10.293 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.0067e-05 2024-09-11T09:52:10.294 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:10.294 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:10.294 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:10.294 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.294 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.294 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4694.0:53 2.7 2:e8766325:ns::rbd_object_map.1256a0c02879:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:10.294 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.147978+0000", 2024-09-11T09:52:10.294 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 47.027837392000002, 2024-09-11T09:52:10.294 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0044549769999999997, 2024-09-11T09:52:10.294 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:10.294 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:10.295 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:10.295 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:10.295 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4694", 2024-09-11T09:52:10.295 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:52:10.295 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 53 2024-09-11T09:52:10.295 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.295 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:10.295 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.295 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:10.295 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.147978+0000", 2024-09-11T09:52:10.295 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:10.295 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.296 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.296 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:10.296 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.147978+0000", 2024-09-11T09:52:10.296 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:10.296 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.296 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.296 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:10.296 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.147980+0000", 2024-09-11T09:52:10.296 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.8309999999999999e-06 2024-09-11T09:52:10.296 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.296 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.297 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:10.297 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.147989+0000", 2024-09-11T09:52:10.297 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.9639999999999992e-06 2024-09-11T09:52:10.297 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.297 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.297 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:10.297 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.148007+0000", 2024-09-11T09:52:10.297 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7884999999999999e-05 2024-09-11T09:52:10.297 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.297 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.297 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:10.297 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.148017+0000", 2024-09-11T09:52:10.298 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0895e-05 2024-09-11T09:52:10.298 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.298 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.298 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:10.298 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.148091+0000", 2024-09-11T09:52:10.298 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.3709999999999997e-05 2024-09-11T09:52:10.298 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.298 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.298 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:10.298 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.148189+0000", 2024-09-11T09:52:10.298 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.7403000000000002e-05 2024-09-11T09:52:10.298 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.299 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.299 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 5", 2024-09-11T09:52:10.299 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.148504+0000", 2024-09-11T09:52:10.299 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00031589900000000002 2024-09-11T09:52:10.299 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.299 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.299 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:10.299 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.149514+0000", 2024-09-11T09:52:10.299 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0010096 2024-09-11T09:52:10.299 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.299 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.299 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:10.300 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.152367+0000", 2024-09-11T09:52:10.300 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0028532309999999999 2024-09-11T09:52:10.300 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.300 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.300 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:10.300 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.152402+0000", 2024-09-11T09:52:10.300 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.4232e-05 2024-09-11T09:52:10.300 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.300 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.300 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:10.300 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.152433+0000", 2024-09-11T09:52:10.300 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.1327e-05 2024-09-11T09:52:10.301 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:10.301 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:10.301 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:10.301 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.301 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.301 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4694.0:55 2.7 2:e8766325:ns::rbd_object_map.1256a0c02879:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:10.301 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.149838+0000", 2024-09-11T09:52:10.301 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 47.025977658999999, 2024-09-11T09:52:10.301 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0060695159999999996, 2024-09-11T09:52:10.301 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:10.301 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:10.302 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:10.302 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:10.302 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4694", 2024-09-11T09:52:10.302 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:52:10.302 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 55 2024-09-11T09:52:10.302 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.302 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:10.302 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.302 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:10.302 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.149838+0000", 2024-09-11T09:52:10.302 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:10.303 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.303 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.303 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:10.303 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.149838+0000", 2024-09-11T09:52:10.303 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:10.303 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.303 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.303 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:10.303 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.149839+0000", 2024-09-11T09:52:10.303 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.119e-06 2024-09-11T09:52:10.303 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.303 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.304 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:10.304 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.149845+0000", 2024-09-11T09:52:10.304 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.7810000000000002e-06 2024-09-11T09:52:10.304 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.304 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.304 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:10.304 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.149856+0000", 2024-09-11T09:52:10.304 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1199999999999999e-05 2024-09-11T09:52:10.304 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.304 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.304 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:10.305 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.149863+0000", 2024-09-11T09:52:10.305 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.4109999999999999e-06 2024-09-11T09:52:10.305 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.305 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.305 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:10.305 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.149908+0000", 2024-09-11T09:52:10.305 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.5184e-05 2024-09-11T09:52:10.305 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.305 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.305 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:10.305 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.149964+0000", 2024-09-11T09:52:10.305 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.5395000000000001e-05 2024-09-11T09:52:10.306 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.306 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.306 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 5", 2024-09-11T09:52:10.306 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.150157+0000", 2024-09-11T09:52:10.306 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000193489 2024-09-11T09:52:10.306 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.306 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.306 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:10.306 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.152238+0000", 2024-09-11T09:52:10.306 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0020803480000000001 2024-09-11T09:52:10.306 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.306 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.307 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:10.307 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.155828+0000", 2024-09-11T09:52:10.307 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0035902210000000002 2024-09-11T09:52:10.307 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.307 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.307 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:10.307 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.155873+0000", 2024-09-11T09:52:10.307 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.5145999999999998e-05 2024-09-11T09:52:10.307 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.307 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.307 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:10.308 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.155907+0000", 2024-09-11T09:52:10.308 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.4221999999999999e-05 2024-09-11T09:52:10.308 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:10.308 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:10.308 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:10.308 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:10.308 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:10.308 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:52:10.310 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-11T09:52:10.311 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:52:10.311+0000 7f3c96ec9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31909) UID: 0 2024-09-11T09:52:10.369 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T09:52:10.410 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:52:10.411+0000 7f3c96ec9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31909) UID: 0 2024-09-11T09:52:10.419 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:52:10.419 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [ 2024-09-11T09:52:10.419 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.419 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.5455.0:259 2.4 2.e5676094 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e83)", 2024-09-11T09:52:10.419 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:52:09.853948+0000", 2024-09-11T09:52:10.419 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 0.56661847399999998, 2024-09-11T09:52:10.419 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.56664197299999997, 2024-09-11T09:52:10.419 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:10.419 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:10.420 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "waiting for peered", 2024-09-11T09:52:10.420 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:10.420 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.5455", 2024-09-11T09:52:10.420 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3911579706", 2024-09-11T09:52:10.420 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 259 2024-09-11T09:52:10.420 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.420 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:10.420 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.420 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:10.420 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:52:09.853948+0000", 2024-09-11T09:52:10.420 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:10.421 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.421 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.421 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:10.421 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:52:09.853948+0000", 2024-09-11T09:52:10.421 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:10.421 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.421 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.421 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:10.421 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:52:09.853949+0000", 2024-09-11T09:52:10.421 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.1099999999999995e-07 2024-09-11T09:52:10.421 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.422 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.422 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:10.422 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:52:09.853953+0000", 2024-09-11T09:52:10.422 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.8229999999999997e-06 2024-09-11T09:52:10.422 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.422 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.422 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:10.422 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:52:09.853978+0000", 2024-09-11T09:52:10.422 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.5528e-05 2024-09-11T09:52:10.422 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.422 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.423 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:10.423 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:52:09.853984+0000", 2024-09-11T09:52:10.423 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.8730000000000004e-06 2024-09-11T09:52:10.423 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.423 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.423 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:10.423 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:52:09.854092+0000", 2024-09-11T09:52:10.423 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00010733199999999999 2024-09-11T09:52:10.423 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.423 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.423 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for peered", 2024-09-11T09:52:10.424 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:52:09.854099+0000", 2024-09-11T09:52:10.424 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.1509999999999998e-06 2024-09-11T09:52:10.424 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:10.424 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:10.424 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:10.424 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.424 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.424 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.5455.0:258 2.4 2.e5676094 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e83)", 2024-09-11T09:52:10.424 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:52:09.853892+0000", 2024-09-11T09:52:10.424 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 0.56667458100000001, 2024-09-11T09:52:10.424 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.56674359399999996, 2024-09-11T09:52:10.425 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:10.425 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:10.425 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "waiting for peered", 2024-09-11T09:52:10.425 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:10.425 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.5455", 2024-09-11T09:52:10.425 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3911579706", 2024-09-11T09:52:10.425 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 258 2024-09-11T09:52:10.425 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.425 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:10.425 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.425 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:10.426 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:52:09.853892+0000", 2024-09-11T09:52:10.426 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:10.426 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.426 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.426 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:10.426 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:52:09.853892+0000", 2024-09-11T09:52:10.426 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:10.426 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.426 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.426 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:10.426 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:52:09.853893+0000", 2024-09-11T09:52:10.426 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1990000000000001e-06 2024-09-11T09:52:10.427 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.427 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.427 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:10.427 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:52:09.853901+0000", 2024-09-11T09:52:10.427 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.9610000000000005e-06 2024-09-11T09:52:10.427 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.427 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.427 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:10.427 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:52:09.853911+0000", 2024-09-11T09:52:10.427 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.2629999999999992e-06 2024-09-11T09:52:10.427 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.427 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.428 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:10.428 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:52:09.853921+0000", 2024-09-11T09:52:10.428 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0294e-05 2024-09-11T09:52:10.428 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.428 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.428 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:10.428 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:52:09.854052+0000", 2024-09-11T09:52:10.428 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00013145 2024-09-11T09:52:10.428 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:10.428 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:10.428 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for peered", 2024-09-11T09:52:10.428 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:52:09.854063+0000", 2024-09-11T09:52:10.428 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0631e-05 2024-09-11T09:52:10.429 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:10.429 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:10.429 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:10.429 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:10.429 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-11T09:52:10.429 INFO:teuthology.orchestra.run.smithi017.stdout: "num_ops": 2 2024-09-11T09:52:10.429 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:52:10.430 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-11T09:52:10.495 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-11T09:52:10.496 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T09:52:10.511 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:52:10.512+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:52:10.513 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-11T09:52:10.513 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T09:52:10.529 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-11T09:52:10.529 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T09:52:10.540 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:52:10.541 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:52:10.547 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-11T09:52:10.547 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T09:52:10.550 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-11T09:52:10.564 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-11T09:52:10.564 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T09:52:10.581 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-11T09:52:10.581 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T09:52:10.613 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:52:10.614+0000 7f3c96ec9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31909) UID: 0 2024-09-11T09:52:10.671 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:52:10.672 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:52:10.680 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-11T09:52:10.724 INFO:tasks.ceph.osd.2.smithi017.stderr:2024-09-11T09:52:10.725+0000 7fd20b170640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31672) UID: 0 2024-09-11T09:52:10.787 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:52:10.787 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:52:10.796 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-11T09:52:10.837 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:52:10.838+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:52:10.918 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-09-11T09:52:10.919+0000 7f7b6c7d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31904) UID: 0 2024-09-11T09:52:10.947 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:52:10.947 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:52:10.956 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-11T09:52:11.019 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:52:11.020+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:52:11.064 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:52:11.064 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:52:11.073 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-11T09:52:11.119 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:52:11.120+0000 7f3a7b90a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 42475) UID: 0 2024-09-11T09:52:11.188 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:52:11.188 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:52:11.197 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-11T09:52:11.221 INFO:tasks.ceph.osd.0.smithi017.stderr:2024-09-11T09:52:11.222+0000 7fb73c0f1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31665) UID: 0 2024-09-11T09:52:11.322 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:52:11.323+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:52:11.340 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:52:11.340 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:52:11.349 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-11T09:52:11.423 INFO:tasks.ceph.osd.0.smithi017.stderr:2024-09-11T09:52:11.424+0000 7fb73c0f1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31665) UID: 0 2024-09-11T09:52:11.452 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:52:11.452 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:52:11.461 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-11T09:52:11.524 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-09-11T09:52:11.524+0000 7f7b6c7d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31904) UID: 0 2024-09-11T09:52:11.563 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:52:11.563 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:52:11.572 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-11T09:52:11.657 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-09-11T09:52:11.658+0000 7f7b6c7d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31904) UID: 0 2024-09-11T09:52:11.715 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:52:11.715 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:52:11.723 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-11T09:52:11.725 INFO:tasks.ceph.osd.0.smithi017.stderr:2024-09-11T09:52:11.726+0000 7fb73c0f1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31665) UID: 0 2024-09-11T09:52:11.827 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:52:11.827+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:52:11.845 INFO:tasks.workunit.client.0.smithi107.stderr: Creating snap: 10% complete... Creating snap: 100% complete...done. 2024-09-11T09:52:11.851 INFO:tasks.workunit.client.0.smithi107.stderr:+ _sudo dd if=/tmp/tmp.rvrcDirNue/data of=/tmp/tmp.rvrcDirNue/mnt/test bs=1M count=1 oflag=direct 2024-09-11T09:52:11.851 INFO:tasks.workunit.client.0.smithi107.stderr:+ local cmd 2024-09-11T09:52:11.851 INFO:tasks.workunit.client.0.smithi107.stderr:++ id -u 2024-09-11T09:52:11.852 INFO:tasks.workunit.client.0.smithi107.stderr:+ '[' 1000 -eq 0 ']' 2024-09-11T09:52:11.852 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 5] out_osds: [0] dead_osds: [] live_osds: [0, 1, 2, 3, 5, 4] 2024-09-11T09:52:11.852 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-11T09:52:11.853 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-09-11T09:52:11.853 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump -f json-pretty 2024-09-11T09:52:11.853 INFO:tasks.workunit.client.0.smithi107.stderr:++ which dd 2024-09-11T09:52:11.854 INFO:tasks.workunit.client.0.smithi107.stderr:+ cmd=/usr/bin/dd 2024-09-11T09:52:11.855 INFO:tasks.workunit.client.0.smithi107.stderr:+ shift 2024-09-11T09:52:11.855 INFO:tasks.workunit.client.0.smithi107.stderr:+ sudo -nE /usr/bin/dd if=/tmp/tmp.rvrcDirNue/data of=/tmp/tmp.rvrcDirNue/mnt/test bs=1M count=1 oflag=direct 2024-09-11T09:52:11.867 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:52:11.867 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:52:11.869 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-11T09:52:11.869 INFO:tasks.workunit.client.0.smithi107.stderr:1+0 records in 2024-09-11T09:52:11.870 INFO:tasks.workunit.client.0.smithi107.stderr:1+0 records out 2024-09-11T09:52:11.870 INFO:tasks.workunit.client.0.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00143205 s, 732 MB/s 2024-09-11T09:52:11.871 INFO:tasks.workunit.client.0.smithi107.stderr:+ _sudo umount /tmp/tmp.rvrcDirNue/mnt 2024-09-11T09:52:11.871 INFO:tasks.workunit.client.0.smithi107.stderr:+ local cmd 2024-09-11T09:52:11.871 INFO:tasks.workunit.client.0.smithi107.stderr:++ id -u 2024-09-11T09:52:11.872 INFO:tasks.workunit.client.0.smithi107.stderr:+ '[' 1000 -eq 0 ']' 2024-09-11T09:52:11.872 INFO:tasks.workunit.client.0.smithi107.stderr:++ which umount 2024-09-11T09:52:11.874 INFO:tasks.workunit.client.0.smithi107.stderr:+ cmd=/usr/bin/umount 2024-09-11T09:52:11.874 INFO:tasks.workunit.client.0.smithi107.stderr:+ shift 2024-09-11T09:52:11.874 INFO:tasks.workunit.client.0.smithi107.stderr:+ sudo -nE /usr/bin/umount /tmp/tmp.rvrcDirNue/mnt 2024-09-11T09:52:11.876 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-11T09:52:11.900 INFO:tasks.workunit.client.0.smithi107.stderr:+ unmap_device /dev/nbd1 29573 2024-09-11T09:52:11.900 INFO:tasks.workunit.client.0.smithi107.stderr:+ local args=/dev/nbd1 2024-09-11T09:52:11.900 INFO:tasks.workunit.client.0.smithi107.stderr:+ local pid=29573 2024-09-11T09:52:11.900 INFO:tasks.workunit.client.0.smithi107.stderr:+ _sudo rbd device --device-type nbd unmap /dev/nbd1 2024-09-11T09:52:11.900 INFO:tasks.workunit.client.0.smithi107.stderr:+ local cmd 2024-09-11T09:52:11.900 INFO:tasks.workunit.client.0.smithi107.stderr:++ id -u 2024-09-11T09:52:11.901 INFO:tasks.workunit.client.0.smithi107.stderr:+ '[' 1000 -eq 0 ']' 2024-09-11T09:52:11.901 INFO:tasks.workunit.client.0.smithi107.stderr:++ which rbd 2024-09-11T09:52:11.903 INFO:tasks.workunit.client.0.smithi107.stderr:+ cmd=/usr/bin/rbd 2024-09-11T09:52:11.903 INFO:tasks.workunit.client.0.smithi107.stderr:+ shift 2024-09-11T09:52:11.903 INFO:tasks.workunit.client.0.smithi107.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd unmap /dev/nbd1 2024-09-11T09:52:11.927 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:52:11.927+0000 7f3a7b90a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 42475) UID: 0 2024-09-11T09:52:11.979 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:52:11.980 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:52:11.989 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-11T09:52:12.010 INFO:tasks.workunit.client.0.smithi107.stderr:+ rbd device --device-type nbd list 2024-09-11T09:52:12.010 INFO:tasks.workunit.client.0.smithi107.stderr:+ expect_false grep '^29573\b' 2024-09-11T09:52:12.010 INFO:tasks.workunit.client.0.smithi107.stderr:+ grep '^29573\b' 2024-09-11T09:52:12.028 INFO:tasks.ceph.osd.0.smithi017.stderr:2024-09-11T09:52:12.029+0000 7fb73c0f1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31665) UID: 0 2024-09-11T09:52:12.070 INFO:tasks.workunit.client.0.smithi107.stderr:+ return 0 2024-09-11T09:52:12.070 INFO:tasks.workunit.client.0.smithi107.stderr:+ ps -C rbd-nbd 2024-09-11T09:52:12.070 INFO:tasks.workunit.client.0.smithi107.stderr:+ expect_false grep '^ *29573\b' 2024-09-11T09:52:12.070 INFO:tasks.workunit.client.0.smithi107.stderr:+ grep '^ *29573\b' 2024-09-11T09:52:12.081 INFO:tasks.workunit.client.0.smithi107.stderr:+ return 0 2024-09-11T09:52:12.081 INFO:tasks.workunit.client.0.smithi107.stderr:+ sleep 0.5 2024-09-11T09:52:12.129 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:52:12.130+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:52:12.131 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:52:12.131 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:52:12.140 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-11T09:52:12.164 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-11T09:52:12.164 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:52:12.164 INFO:teuthology.orchestra.run.smithi017.stdout: "epoch": 84, 2024-09-11T09:52:12.164 INFO:teuthology.orchestra.run.smithi017.stdout: "fsid": "a2f1daec-d3b0-4bf5-82c8-72ecbb93d4e4", 2024-09-11T09:52:12.164 INFO:teuthology.orchestra.run.smithi017.stdout: "created": "2024-09-11T09:50:00.165804+0000", 2024-09-11T09:52:12.164 INFO:teuthology.orchestra.run.smithi017.stdout: "modified": "2024-09-11T09:52:10.851328+0000", 2024-09-11T09:52:12.164 INFO:teuthology.orchestra.run.smithi017.stdout: "last_up_change": "2024-09-11T09:51:44.764990+0000", 2024-09-11T09:52:12.164 INFO:teuthology.orchestra.run.smithi017.stdout: "last_in_change": "2024-09-11T09:52:06.832073+0000", 2024-09-11T09:52:12.165 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": "nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-11T09:52:12.165 INFO:teuthology.orchestra.run.smithi017.stdout: "flags_num": 5804032, 2024-09-11T09:52:12.165 INFO:teuthology.orchestra.run.smithi017.stdout: "flags_set": [ 2024-09-11T09:52:12.165 INFO:teuthology.orchestra.run.smithi017.stdout: "nodeep-scrub", 2024-09-11T09:52:12.165 INFO:teuthology.orchestra.run.smithi017.stdout: "pglog_hardlimit", 2024-09-11T09:52:12.165 INFO:teuthology.orchestra.run.smithi017.stdout: "purged_snapdirs", 2024-09-11T09:52:12.165 INFO:teuthology.orchestra.run.smithi017.stdout: "recovery_deletes", 2024-09-11T09:52:12.165 INFO:teuthology.orchestra.run.smithi017.stdout: "sortbitwise" 2024-09-11T09:52:12.165 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-11T09:52:12.165 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_version": 6, 2024-09-11T09:52:12.165 INFO:teuthology.orchestra.run.smithi017.stdout: "full_ratio": 0.94999998807907104, 2024-09-11T09:52:12.165 INFO:teuthology.orchestra.run.smithi017.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-11T09:52:12.165 INFO:teuthology.orchestra.run.smithi017.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-11T09:52:12.165 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_snapshot": "", 2024-09-11T09:52:12.166 INFO:teuthology.orchestra.run.smithi017.stdout: "pool_max": 3, 2024-09-11T09:52:12.166 INFO:teuthology.orchestra.run.smithi017.stdout: "max_osd": 6, 2024-09-11T09:52:12.166 INFO:teuthology.orchestra.run.smithi017.stdout: "require_min_compat_client": "luminous", 2024-09-11T09:52:12.166 INFO:teuthology.orchestra.run.smithi017.stdout: "min_compat_client": "jewel", 2024-09-11T09:52:12.166 INFO:teuthology.orchestra.run.smithi017.stdout: "require_osd_release": "squid", 2024-09-11T09:52:12.166 INFO:teuthology.orchestra.run.smithi017.stdout: "allow_crimson": false, 2024-09-11T09:52:12.166 INFO:teuthology.orchestra.run.smithi017.stdout: "pools": [ 2024-09-11T09:52:12.166 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:12.166 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 1, 2024-09-11T09:52:12.166 INFO:teuthology.orchestra.run.smithi017.stdout: "pool_name": ".mgr", 2024-09-11T09:52:12.166 INFO:teuthology.orchestra.run.smithi017.stdout: "create_time": "2024-09-11T09:50:10.457184+0000", 2024-09-11T09:52:12.166 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 1, 2024-09-11T09:52:12.167 INFO:teuthology.orchestra.run.smithi017.stdout: "flags_names": "hashpspool", 2024-09-11T09:52:12.167 INFO:teuthology.orchestra.run.smithi017.stdout: "type": 1, 2024-09-11T09:52:12.167 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 2, 2024-09-11T09:52:12.167 INFO:teuthology.orchestra.run.smithi017.stdout: "min_size": 1, 2024-09-11T09:52:12.167 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_rule": 0, 2024-09-11T09:52:12.167 INFO:teuthology.orchestra.run.smithi017.stdout: "peering_crush_bucket_count": 0, 2024-09-11T09:52:12.167 INFO:teuthology.orchestra.run.smithi017.stdout: "peering_crush_bucket_target": 0, 2024-09-11T09:52:12.167 INFO:teuthology.orchestra.run.smithi017.stdout: "peering_crush_bucket_barrier": 0, 2024-09-11T09:52:12.167 INFO:teuthology.orchestra.run.smithi017.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-11T09:52:12.167 INFO:teuthology.orchestra.run.smithi017.stdout: "is_stretch_pool": false, 2024-09-11T09:52:12.168 INFO:teuthology.orchestra.run.smithi017.stdout: "object_hash": 2, 2024-09-11T09:52:12.168 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_autoscale_mode": "off", 2024-09-11T09:52:12.168 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_num": 1, 2024-09-11T09:52:12.168 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_placement_num": 1, 2024-09-11T09:52:12.168 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_placement_num_target": 1, 2024-09-11T09:52:12.168 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_num_target": 1, 2024-09-11T09:52:12.168 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_num_pending": 1, 2024-09-11T09:52:12.168 INFO:teuthology.orchestra.run.smithi017.stdout: "last_pg_merge_meta": { 2024-09-11T09:52:12.168 INFO:teuthology.orchestra.run.smithi017.stdout: "source_pgid": "0.0", 2024-09-11T09:52:12.168 INFO:teuthology.orchestra.run.smithi017.stdout: "ready_epoch": 0, 2024-09-11T09:52:12.168 INFO:teuthology.orchestra.run.smithi017.stdout: "last_epoch_started": 0, 2024-09-11T09:52:12.168 INFO:teuthology.orchestra.run.smithi017.stdout: "last_epoch_clean": 0, 2024-09-11T09:52:12.168 INFO:teuthology.orchestra.run.smithi017.stdout: "source_version": "0'0", 2024-09-11T09:52:12.169 INFO:teuthology.orchestra.run.smithi017.stdout: "target_version": "0'0" 2024-09-11T09:52:12.169 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:12.169 INFO:teuthology.orchestra.run.smithi017.stdout: "last_change": "14", 2024-09-11T09:52:12.169 INFO:teuthology.orchestra.run.smithi017.stdout: "last_force_op_resend": "0", 2024-09-11T09:52:12.169 INFO:teuthology.orchestra.run.smithi017.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-11T09:52:12.169 INFO:teuthology.orchestra.run.smithi017.stdout: "last_force_op_resend_preluminous": "0", 2024-09-11T09:52:12.169 INFO:teuthology.orchestra.run.smithi017.stdout: "auid": 0, 2024-09-11T09:52:12.169 INFO:teuthology.orchestra.run.smithi017.stdout: "snap_mode": "selfmanaged", 2024-09-11T09:52:12.169 INFO:teuthology.orchestra.run.smithi017.stdout: "snap_seq": 0, 2024-09-11T09:52:12.169 INFO:teuthology.orchestra.run.smithi017.stdout: "snap_epoch": 0, 2024-09-11T09:52:12.169 INFO:teuthology.orchestra.run.smithi017.stdout: "pool_snaps": [], 2024-09-11T09:52:12.169 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_snaps": "[]", 2024-09-11T09:52:12.169 INFO:teuthology.orchestra.run.smithi017.stdout: "quota_max_bytes": 0, 2024-09-11T09:52:12.170 INFO:teuthology.orchestra.run.smithi017.stdout: "quota_max_objects": 0, 2024-09-11T09:52:12.170 INFO:teuthology.orchestra.run.smithi017.stdout: "tiers": [], 2024-09-11T09:52:12.170 INFO:teuthology.orchestra.run.smithi017.stdout: "tier_of": -1, 2024-09-11T09:52:12.170 INFO:teuthology.orchestra.run.smithi017.stdout: "read_tier": -1, 2024-09-11T09:52:12.170 INFO:teuthology.orchestra.run.smithi017.stdout: "write_tier": -1, 2024-09-11T09:52:12.170 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_mode": "none", 2024-09-11T09:52:12.170 INFO:teuthology.orchestra.run.smithi017.stdout: "target_max_bytes": 0, 2024-09-11T09:52:12.170 INFO:teuthology.orchestra.run.smithi017.stdout: "target_max_objects": 0, 2024-09-11T09:52:12.170 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-11T09:52:12.170 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-11T09:52:12.170 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-11T09:52:12.170 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_min_flush_age": 0, 2024-09-11T09:52:12.170 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_min_evict_age": 0, 2024-09-11T09:52:12.170 INFO:teuthology.orchestra.run.smithi017.stdout: "erasure_code_profile": "", 2024-09-11T09:52:12.171 INFO:teuthology.orchestra.run.smithi017.stdout: "hit_set_params": { 2024-09-11T09:52:12.171 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "none" 2024-09-11T09:52:12.171 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:12.171 INFO:teuthology.orchestra.run.smithi017.stdout: "hit_set_period": 0, 2024-09-11T09:52:12.171 INFO:teuthology.orchestra.run.smithi017.stdout: "hit_set_count": 0, 2024-09-11T09:52:12.171 INFO:teuthology.orchestra.run.smithi017.stdout: "use_gmt_hitset": true, 2024-09-11T09:52:12.171 INFO:teuthology.orchestra.run.smithi017.stdout: "min_read_recency_for_promote": 0, 2024-09-11T09:52:12.171 INFO:teuthology.orchestra.run.smithi017.stdout: "min_write_recency_for_promote": 0, 2024-09-11T09:52:12.171 INFO:teuthology.orchestra.run.smithi017.stdout: "hit_set_grade_decay_rate": 0, 2024-09-11T09:52:12.171 INFO:teuthology.orchestra.run.smithi017.stdout: "hit_set_search_last_n": 0, 2024-09-11T09:52:12.171 INFO:teuthology.orchestra.run.smithi017.stdout: "grade_table": [], 2024-09-11T09:52:12.171 INFO:teuthology.orchestra.run.smithi017.stdout: "stripe_width": 0, 2024-09-11T09:52:12.171 INFO:teuthology.orchestra.run.smithi017.stdout: "expected_num_objects": 0, 2024-09-11T09:52:12.172 INFO:teuthology.orchestra.run.smithi017.stdout: "fast_read": false, 2024-09-11T09:52:12.172 INFO:teuthology.orchestra.run.smithi017.stdout: "options": { 2024-09-11T09:52:12.172 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_num_max": 32, 2024-09-11T09:52:12.172 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_num_min": 1 2024-09-11T09:52:12.172 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:12.172 INFO:teuthology.orchestra.run.smithi017.stdout: "application_metadata": { 2024-09-11T09:52:12.172 INFO:teuthology.orchestra.run.smithi017.stdout: "mgr": {} 2024-09-11T09:52:12.172 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:12.172 INFO:teuthology.orchestra.run.smithi017.stdout: "read_balance": { 2024-09-11T09:52:12.172 INFO:teuthology.orchestra.run.smithi017.stdout: "score_type": "Fair distribution", 2024-09-11T09:52:12.172 INFO:teuthology.orchestra.run.smithi017.stdout: "score_acting": 6.059999942779541, 2024-09-11T09:52:12.172 INFO:teuthology.orchestra.run.smithi017.stdout: "score_stable": 6.059999942779541, 2024-09-11T09:52:12.173 INFO:teuthology.orchestra.run.smithi017.stdout: "optimal_score": 0.33000001311302185, 2024-09-11T09:52:12.173 INFO:teuthology.orchestra.run.smithi017.stdout: "raw_score_acting": 2, 2024-09-11T09:52:12.173 INFO:teuthology.orchestra.run.smithi017.stdout: "raw_score_stable": 2, 2024-09-11T09:52:12.173 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity_weighted": 1, 2024-09-11T09:52:12.173 INFO:teuthology.orchestra.run.smithi017.stdout: "average_primary_affinity": 1, 2024-09-11T09:52:12.173 INFO:teuthology.orchestra.run.smithi017.stdout: "average_primary_affinity_weighted": 1 2024-09-11T09:52:12.173 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:12.173 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:12.173 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:12.173 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 2, 2024-09-11T09:52:12.173 INFO:teuthology.orchestra.run.smithi017.stdout: "pool_name": "rbd", 2024-09-11T09:52:12.174 INFO:teuthology.orchestra.run.smithi017.stdout: "create_time": "2024-09-11T09:50:16.993202+0000", 2024-09-11T09:52:12.174 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 8193, 2024-09-11T09:52:12.174 INFO:teuthology.orchestra.run.smithi017.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-11T09:52:12.174 INFO:teuthology.orchestra.run.smithi017.stdout: "type": 1, 2024-09-11T09:52:12.174 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 2, 2024-09-11T09:52:12.174 INFO:teuthology.orchestra.run.smithi017.stdout: "min_size": 1, 2024-09-11T09:52:12.174 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_rule": 0, 2024-09-11T09:52:12.174 INFO:teuthology.orchestra.run.smithi017.stdout: "peering_crush_bucket_count": 0, 2024-09-11T09:52:12.174 INFO:teuthology.orchestra.run.smithi017.stdout: "peering_crush_bucket_target": 0, 2024-09-11T09:52:12.174 INFO:teuthology.orchestra.run.smithi017.stdout: "peering_crush_bucket_barrier": 0, 2024-09-11T09:52:12.174 INFO:teuthology.orchestra.run.smithi017.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-11T09:52:12.174 INFO:teuthology.orchestra.run.smithi017.stdout: "is_stretch_pool": false, 2024-09-11T09:52:12.174 INFO:teuthology.orchestra.run.smithi017.stdout: "object_hash": 2, 2024-09-11T09:52:12.174 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_autoscale_mode": "off", 2024-09-11T09:52:12.175 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_num": 8, 2024-09-11T09:52:12.175 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_placement_num": 8, 2024-09-11T09:52:12.175 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_placement_num_target": 8, 2024-09-11T09:52:12.175 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_num_target": 8, 2024-09-11T09:52:12.175 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_num_pending": 8, 2024-09-11T09:52:12.175 INFO:teuthology.orchestra.run.smithi017.stdout: "last_pg_merge_meta": { 2024-09-11T09:52:12.175 INFO:teuthology.orchestra.run.smithi017.stdout: "source_pgid": "0.0", 2024-09-11T09:52:12.175 INFO:teuthology.orchestra.run.smithi017.stdout: "ready_epoch": 0, 2024-09-11T09:52:12.175 INFO:teuthology.orchestra.run.smithi017.stdout: "last_epoch_started": 0, 2024-09-11T09:52:12.175 INFO:teuthology.orchestra.run.smithi017.stdout: "last_epoch_clean": 0, 2024-09-11T09:52:12.176 INFO:teuthology.orchestra.run.smithi017.stdout: "source_version": "0'0", 2024-09-11T09:52:12.176 INFO:teuthology.orchestra.run.smithi017.stdout: "target_version": "0'0" 2024-09-11T09:52:12.176 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:12.176 INFO:teuthology.orchestra.run.smithi017.stdout: "last_change": "83", 2024-09-11T09:52:12.176 INFO:teuthology.orchestra.run.smithi017.stdout: "last_force_op_resend": "0", 2024-09-11T09:52:12.176 INFO:teuthology.orchestra.run.smithi017.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-11T09:52:12.176 INFO:teuthology.orchestra.run.smithi017.stdout: "last_force_op_resend_preluminous": "0", 2024-09-11T09:52:12.176 INFO:teuthology.orchestra.run.smithi017.stdout: "auid": 0, 2024-09-11T09:52:12.176 INFO:teuthology.orchestra.run.smithi017.stdout: "snap_mode": "selfmanaged", 2024-09-11T09:52:12.176 INFO:teuthology.orchestra.run.smithi017.stdout: "snap_seq": 7, 2024-09-11T09:52:12.176 INFO:teuthology.orchestra.run.smithi017.stdout: "snap_epoch": 83, 2024-09-11T09:52:12.176 INFO:teuthology.orchestra.run.smithi017.stdout: "pool_snaps": [], 2024-09-11T09:52:12.177 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_snaps": "[]", 2024-09-11T09:52:12.177 INFO:teuthology.orchestra.run.smithi017.stdout: "quota_max_bytes": 0, 2024-09-11T09:52:12.177 INFO:teuthology.orchestra.run.smithi017.stdout: "quota_max_objects": 0, 2024-09-11T09:52:12.177 INFO:teuthology.orchestra.run.smithi017.stdout: "tiers": [], 2024-09-11T09:52:12.177 INFO:teuthology.orchestra.run.smithi017.stdout: "tier_of": -1, 2024-09-11T09:52:12.177 INFO:teuthology.orchestra.run.smithi017.stdout: "read_tier": -1, 2024-09-11T09:52:12.177 INFO:teuthology.orchestra.run.smithi017.stdout: "write_tier": -1, 2024-09-11T09:52:12.177 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_mode": "none", 2024-09-11T09:52:12.177 INFO:teuthology.orchestra.run.smithi017.stdout: "target_max_bytes": 0, 2024-09-11T09:52:12.177 INFO:teuthology.orchestra.run.smithi017.stdout: "target_max_objects": 0, 2024-09-11T09:52:12.177 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-11T09:52:12.177 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-11T09:52:12.178 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-11T09:52:12.178 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_min_flush_age": 0, 2024-09-11T09:52:12.178 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_min_evict_age": 0, 2024-09-11T09:52:12.178 INFO:teuthology.orchestra.run.smithi017.stdout: "erasure_code_profile": "", 2024-09-11T09:52:12.178 INFO:teuthology.orchestra.run.smithi017.stdout: "hit_set_params": { 2024-09-11T09:52:12.178 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "none" 2024-09-11T09:52:12.178 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:12.178 INFO:teuthology.orchestra.run.smithi017.stdout: "hit_set_period": 0, 2024-09-11T09:52:12.178 INFO:teuthology.orchestra.run.smithi017.stdout: "hit_set_count": 0, 2024-09-11T09:52:12.178 INFO:teuthology.orchestra.run.smithi017.stdout: "use_gmt_hitset": true, 2024-09-11T09:52:12.178 INFO:teuthology.orchestra.run.smithi017.stdout: "min_read_recency_for_promote": 0, 2024-09-11T09:52:12.179 INFO:teuthology.orchestra.run.smithi017.stdout: "min_write_recency_for_promote": 0, 2024-09-11T09:52:12.179 INFO:teuthology.orchestra.run.smithi017.stdout: "hit_set_grade_decay_rate": 0, 2024-09-11T09:52:12.179 INFO:teuthology.orchestra.run.smithi017.stdout: "hit_set_search_last_n": 0, 2024-09-11T09:52:12.179 INFO:teuthology.orchestra.run.smithi017.stdout: "grade_table": [], 2024-09-11T09:52:12.179 INFO:teuthology.orchestra.run.smithi017.stdout: "stripe_width": 0, 2024-09-11T09:52:12.179 INFO:teuthology.orchestra.run.smithi017.stdout: "expected_num_objects": 0, 2024-09-11T09:52:12.179 INFO:teuthology.orchestra.run.smithi017.stdout: "fast_read": false, 2024-09-11T09:52:12.179 INFO:teuthology.orchestra.run.smithi017.stdout: "options": {}, 2024-09-11T09:52:12.179 INFO:teuthology.orchestra.run.smithi017.stdout: "application_metadata": { 2024-09-11T09:52:12.179 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd": {} 2024-09-11T09:52:12.180 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:12.180 INFO:teuthology.orchestra.run.smithi017.stdout: "read_balance": { 2024-09-11T09:52:12.180 INFO:teuthology.orchestra.run.smithi017.stdout: "score_type": "Fair distribution", 2024-09-11T09:52:12.180 INFO:teuthology.orchestra.run.smithi017.stdout: "score_acting": 2.2699999809265137, 2024-09-11T09:52:12.180 INFO:teuthology.orchestra.run.smithi017.stdout: "score_stable": 2.2699999809265137, 2024-09-11T09:52:12.180 INFO:teuthology.orchestra.run.smithi017.stdout: "optimal_score": 0.82999998331069946, 2024-09-11T09:52:12.180 INFO:teuthology.orchestra.run.smithi017.stdout: "raw_score_acting": 1.8799999952316284, 2024-09-11T09:52:12.180 INFO:teuthology.orchestra.run.smithi017.stdout: "raw_score_stable": 1.8799999952316284, 2024-09-11T09:52:12.180 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity_weighted": 1, 2024-09-11T09:52:12.180 INFO:teuthology.orchestra.run.smithi017.stdout: "average_primary_affinity": 1, 2024-09-11T09:52:12.180 INFO:teuthology.orchestra.run.smithi017.stdout: "average_primary_affinity_weighted": 1 2024-09-11T09:52:12.180 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:12.180 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:12.181 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:12.181 INFO:teuthology.orchestra.run.smithi017.stdout: "pool": 3, 2024-09-11T09:52:12.181 INFO:teuthology.orchestra.run.smithi017.stdout: "pool_name": "new_default_pool27121", 2024-09-11T09:52:12.181 INFO:teuthology.orchestra.run.smithi017.stdout: "create_time": "2024-09-11T09:51:23.493016+0000", 2024-09-11T09:52:12.181 INFO:teuthology.orchestra.run.smithi017.stdout: "flags": 8193, 2024-09-11T09:52:12.181 INFO:teuthology.orchestra.run.smithi017.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-11T09:52:12.181 INFO:teuthology.orchestra.run.smithi017.stdout: "type": 1, 2024-09-11T09:52:12.181 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 2, 2024-09-11T09:52:12.181 INFO:teuthology.orchestra.run.smithi017.stdout: "min_size": 1, 2024-09-11T09:52:12.181 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_rule": 0, 2024-09-11T09:52:12.181 INFO:teuthology.orchestra.run.smithi017.stdout: "peering_crush_bucket_count": 0, 2024-09-11T09:52:12.181 INFO:teuthology.orchestra.run.smithi017.stdout: "peering_crush_bucket_target": 0, 2024-09-11T09:52:12.181 INFO:teuthology.orchestra.run.smithi017.stdout: "peering_crush_bucket_barrier": 0, 2024-09-11T09:52:12.181 INFO:teuthology.orchestra.run.smithi017.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-11T09:52:12.182 INFO:teuthology.orchestra.run.smithi017.stdout: "is_stretch_pool": false, 2024-09-11T09:52:12.182 INFO:teuthology.orchestra.run.smithi017.stdout: "object_hash": 2, 2024-09-11T09:52:12.182 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_autoscale_mode": "off", 2024-09-11T09:52:12.182 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_num": 8, 2024-09-11T09:52:12.182 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_placement_num": 8, 2024-09-11T09:52:12.182 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_placement_num_target": 8, 2024-09-11T09:52:12.182 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_num_target": 8, 2024-09-11T09:52:12.182 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_num_pending": 8, 2024-09-11T09:52:12.182 INFO:teuthology.orchestra.run.smithi017.stdout: "last_pg_merge_meta": { 2024-09-11T09:52:12.182 INFO:teuthology.orchestra.run.smithi017.stdout: "source_pgid": "0.0", 2024-09-11T09:52:12.182 INFO:teuthology.orchestra.run.smithi017.stdout: "ready_epoch": 0, 2024-09-11T09:52:12.182 INFO:teuthology.orchestra.run.smithi017.stdout: "last_epoch_started": 0, 2024-09-11T09:52:12.182 INFO:teuthology.orchestra.run.smithi017.stdout: "last_epoch_clean": 0, 2024-09-11T09:52:12.183 INFO:teuthology.orchestra.run.smithi017.stdout: "source_version": "0'0", 2024-09-11T09:52:12.183 INFO:teuthology.orchestra.run.smithi017.stdout: "target_version": "0'0" 2024-09-11T09:52:12.183 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:12.183 INFO:teuthology.orchestra.run.smithi017.stdout: "last_change": "58", 2024-09-11T09:52:12.183 INFO:teuthology.orchestra.run.smithi017.stdout: "last_force_op_resend": "0", 2024-09-11T09:52:12.183 INFO:teuthology.orchestra.run.smithi017.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-11T09:52:12.183 INFO:teuthology.orchestra.run.smithi017.stdout: "last_force_op_resend_preluminous": "0", 2024-09-11T09:52:12.183 INFO:teuthology.orchestra.run.smithi017.stdout: "auid": 0, 2024-09-11T09:52:12.183 INFO:teuthology.orchestra.run.smithi017.stdout: "snap_mode": "selfmanaged", 2024-09-11T09:52:12.183 INFO:teuthology.orchestra.run.smithi017.stdout: "snap_seq": 2, 2024-09-11T09:52:12.183 INFO:teuthology.orchestra.run.smithi017.stdout: "snap_epoch": 58, 2024-09-11T09:52:12.183 INFO:teuthology.orchestra.run.smithi017.stdout: "pool_snaps": [], 2024-09-11T09:52:12.184 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_snaps": "[]", 2024-09-11T09:52:12.184 INFO:teuthology.orchestra.run.smithi017.stdout: "quota_max_bytes": 0, 2024-09-11T09:52:12.184 INFO:teuthology.orchestra.run.smithi017.stdout: "quota_max_objects": 0, 2024-09-11T09:52:12.184 INFO:teuthology.orchestra.run.smithi017.stdout: "tiers": [], 2024-09-11T09:52:12.184 INFO:teuthology.orchestra.run.smithi017.stdout: "tier_of": -1, 2024-09-11T09:52:12.184 INFO:teuthology.orchestra.run.smithi017.stdout: "read_tier": -1, 2024-09-11T09:52:12.184 INFO:teuthology.orchestra.run.smithi017.stdout: "write_tier": -1, 2024-09-11T09:52:12.184 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_mode": "none", 2024-09-11T09:52:12.184 INFO:teuthology.orchestra.run.smithi017.stdout: "target_max_bytes": 0, 2024-09-11T09:52:12.185 INFO:teuthology.orchestra.run.smithi017.stdout: "target_max_objects": 0, 2024-09-11T09:52:12.185 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-11T09:52:12.185 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-11T09:52:12.185 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-11T09:52:12.185 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_min_flush_age": 0, 2024-09-11T09:52:12.185 INFO:teuthology.orchestra.run.smithi017.stdout: "cache_min_evict_age": 0, 2024-09-11T09:52:12.185 INFO:teuthology.orchestra.run.smithi017.stdout: "erasure_code_profile": "", 2024-09-11T09:52:12.185 INFO:teuthology.orchestra.run.smithi017.stdout: "hit_set_params": { 2024-09-11T09:52:12.185 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "none" 2024-09-11T09:52:12.185 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:12.185 INFO:teuthology.orchestra.run.smithi017.stdout: "hit_set_period": 0, 2024-09-11T09:52:12.186 INFO:teuthology.orchestra.run.smithi017.stdout: "hit_set_count": 0, 2024-09-11T09:52:12.186 INFO:teuthology.orchestra.run.smithi017.stdout: "use_gmt_hitset": true, 2024-09-11T09:52:12.186 INFO:teuthology.orchestra.run.smithi017.stdout: "min_read_recency_for_promote": 0, 2024-09-11T09:52:12.186 INFO:teuthology.orchestra.run.smithi017.stdout: "min_write_recency_for_promote": 0, 2024-09-11T09:52:12.186 INFO:teuthology.orchestra.run.smithi017.stdout: "hit_set_grade_decay_rate": 0, 2024-09-11T09:52:12.186 INFO:teuthology.orchestra.run.smithi017.stdout: "hit_set_search_last_n": 0, 2024-09-11T09:52:12.186 INFO:teuthology.orchestra.run.smithi017.stdout: "grade_table": [], 2024-09-11T09:52:12.186 INFO:teuthology.orchestra.run.smithi017.stdout: "stripe_width": 0, 2024-09-11T09:52:12.186 INFO:teuthology.orchestra.run.smithi017.stdout: "expected_num_objects": 0, 2024-09-11T09:52:12.186 INFO:teuthology.orchestra.run.smithi017.stdout: "fast_read": false, 2024-09-11T09:52:12.186 INFO:teuthology.orchestra.run.smithi017.stdout: "options": {}, 2024-09-11T09:52:12.186 INFO:teuthology.orchestra.run.smithi017.stdout: "application_metadata": { 2024-09-11T09:52:12.186 INFO:teuthology.orchestra.run.smithi017.stdout: "rbd": {} 2024-09-11T09:52:12.186 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:12.187 INFO:teuthology.orchestra.run.smithi017.stdout: "read_balance": { 2024-09-11T09:52:12.187 INFO:teuthology.orchestra.run.smithi017.stdout: "score_type": "Fair distribution", 2024-09-11T09:52:12.187 INFO:teuthology.orchestra.run.smithi017.stdout: "score_acting": 3.0099999904632568, 2024-09-11T09:52:12.187 INFO:teuthology.orchestra.run.smithi017.stdout: "score_stable": 3.0099999904632568, 2024-09-11T09:52:12.187 INFO:teuthology.orchestra.run.smithi017.stdout: "optimal_score": 0.82999998331069946, 2024-09-11T09:52:12.187 INFO:teuthology.orchestra.run.smithi017.stdout: "raw_score_acting": 2.5, 2024-09-11T09:52:12.187 INFO:teuthology.orchestra.run.smithi017.stdout: "raw_score_stable": 2.5, 2024-09-11T09:52:12.187 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity_weighted": 1, 2024-09-11T09:52:12.187 INFO:teuthology.orchestra.run.smithi017.stdout: "average_primary_affinity": 1, 2024-09-11T09:52:12.187 INFO:teuthology.orchestra.run.smithi017.stdout: "average_primary_affinity_weighted": 1 2024-09-11T09:52:12.187 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:12.188 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:12.188 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-11T09:52:12.188 INFO:teuthology.orchestra.run.smithi017.stdout: "osds": [ 2024-09-11T09:52:12.188 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:12.188 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 0, 2024-09-11T09:52:12.188 INFO:teuthology.orchestra.run.smithi017.stdout: "uuid": "4829b7a1-d607-4a56-b6d2-55036cb8acf8", 2024-09-11T09:52:12.188 INFO:teuthology.orchestra.run.smithi017.stdout: "up": 1, 2024-09-11T09:52:12.188 INFO:teuthology.orchestra.run.smithi017.stdout: "in": 0, 2024-09-11T09:52:12.188 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 0, 2024-09-11T09:52:12.188 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity": 1, 2024-09-11T09:52:12.189 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_begin": 0, 2024-09-11T09:52:12.189 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_end": 0, 2024-09-11T09:52:12.189 INFO:teuthology.orchestra.run.smithi017.stdout: "up_from": 12, 2024-09-11T09:52:12.189 INFO:teuthology.orchestra.run.smithi017.stdout: "up_thru": 77, 2024-09-11T09:52:12.189 INFO:teuthology.orchestra.run.smithi017.stdout: "down_at": 0, 2024-09-11T09:52:12.189 INFO:teuthology.orchestra.run.smithi017.stdout: "lost_at": 0, 2024-09-11T09:52:12.189 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-11T09:52:12.189 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-11T09:52:12.189 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:12.189 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-11T09:52:12.189 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6808", 2024-09-11T09:52:12.189 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 4154449929 2024-09-11T09:52:12.189 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:12.190 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:12.190 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-11T09:52:12.190 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6809", 2024-09-11T09:52:12.190 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 4154449929 2024-09-11T09:52:12.190 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:12.190 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:12.190 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:12.190 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addrs": { 2024-09-11T09:52:12.190 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-11T09:52:12.190 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:12.190 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-11T09:52:12.190 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6810", 2024-09-11T09:52:12.191 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 4154449929 2024-09-11T09:52:12.191 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:12.191 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:12.191 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-11T09:52:12.191 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6811", 2024-09-11T09:52:12.191 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 4154449929 2024-09-11T09:52:12.191 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:12.191 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:12.191 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:12.191 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addrs": { 2024-09-11T09:52:12.191 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-11T09:52:12.191 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:12.191 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-11T09:52:12.192 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6814", 2024-09-11T09:52:12.192 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 4154449929 2024-09-11T09:52:12.192 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:12.192 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:12.192 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-11T09:52:12.192 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6815", 2024-09-11T09:52:12.192 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 4154449929 2024-09-11T09:52:12.192 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:12.192 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:12.192 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:12.192 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addrs": { 2024-09-11T09:52:12.192 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-11T09:52:12.193 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:12.193 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-11T09:52:12.193 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6812", 2024-09-11T09:52:12.193 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 4154449929 2024-09-11T09:52:12.193 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:12.193 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:12.193 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-11T09:52:12.193 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6813", 2024-09-11T09:52:12.193 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 4154449929 2024-09-11T09:52:12.193 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:12.193 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:12.194 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:12.194 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6809/4154449929", 2024-09-11T09:52:12.194 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addr": "172.21.15.17:6811/4154449929", 2024-09-11T09:52:12.194 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addr": "172.21.15.17:6815/4154449929", 2024-09-11T09:52:12.194 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addr": "172.21.15.17:6813/4154449929", 2024-09-11T09:52:12.194 INFO:teuthology.orchestra.run.smithi017.stdout: "state": [ 2024-09-11T09:52:12.194 INFO:teuthology.orchestra.run.smithi017.stdout: "exists", 2024-09-11T09:52:12.194 INFO:teuthology.orchestra.run.smithi017.stdout: "up" 2024-09-11T09:52:12.194 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:12.194 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:12.194 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:12.194 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 1, 2024-09-11T09:52:12.194 INFO:teuthology.orchestra.run.smithi017.stdout: "uuid": "ab1d5df0-3c0e-44cd-88a0-96b07964148b", 2024-09-11T09:52:12.195 INFO:teuthology.orchestra.run.smithi017.stdout: "up": 1, 2024-09-11T09:52:12.195 INFO:teuthology.orchestra.run.smithi017.stdout: "in": 1, 2024-09-11T09:52:12.195 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 1, 2024-09-11T09:52:12.195 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity": 1, 2024-09-11T09:52:12.195 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_begin": 0, 2024-09-11T09:52:12.195 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_end": 0, 2024-09-11T09:52:12.195 INFO:teuthology.orchestra.run.smithi017.stdout: "up_from": 11, 2024-09-11T09:52:12.195 INFO:teuthology.orchestra.run.smithi017.stdout: "up_thru": 83, 2024-09-11T09:52:12.195 INFO:teuthology.orchestra.run.smithi017.stdout: "down_at": 0, 2024-09-11T09:52:12.195 INFO:teuthology.orchestra.run.smithi017.stdout: "lost_at": 0, 2024-09-11T09:52:12.195 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-11T09:52:12.195 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-11T09:52:12.195 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:12.195 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-11T09:52:12.196 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6800", 2024-09-11T09:52:12.196 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 4139511644 2024-09-11T09:52:12.196 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:12.196 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:12.196 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-11T09:52:12.196 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6801", 2024-09-11T09:52:12.196 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 4139511644 2024-09-11T09:52:12.196 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:12.196 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:12.196 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:12.196 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addrs": { 2024-09-11T09:52:12.196 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-11T09:52:12.196 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:12.197 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-11T09:52:12.197 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6802", 2024-09-11T09:52:12.197 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 4139511644 2024-09-11T09:52:12.197 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:12.197 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:12.197 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-11T09:52:12.197 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6803", 2024-09-11T09:52:12.197 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 4139511644 2024-09-11T09:52:12.197 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:12.197 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:12.197 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:12.197 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addrs": { 2024-09-11T09:52:12.197 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-11T09:52:12.198 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:12.198 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-11T09:52:12.198 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6806", 2024-09-11T09:52:12.198 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 4139511644 2024-09-11T09:52:12.198 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:12.198 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:12.198 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-11T09:52:12.198 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6807", 2024-09-11T09:52:12.198 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 4139511644 2024-09-11T09:52:12.198 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:12.198 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:12.199 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:12.199 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addrs": { 2024-09-11T09:52:12.199 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-11T09:52:12.199 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:12.199 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-11T09:52:12.199 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6804", 2024-09-11T09:52:12.199 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 4139511644 2024-09-11T09:52:12.199 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:12.199 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:12.199 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-11T09:52:12.199 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6805", 2024-09-11T09:52:12.199 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 4139511644 2024-09-11T09:52:12.199 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:12.200 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:12.200 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:12.200 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6801/4139511644", 2024-09-11T09:52:12.200 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addr": "172.21.15.17:6803/4139511644", 2024-09-11T09:52:12.200 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addr": "172.21.15.17:6807/4139511644", 2024-09-11T09:52:12.200 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addr": "172.21.15.17:6805/4139511644", 2024-09-11T09:52:12.200 INFO:teuthology.orchestra.run.smithi017.stdout: "state": [ 2024-09-11T09:52:12.200 INFO:teuthology.orchestra.run.smithi017.stdout: "exists", 2024-09-11T09:52:12.200 INFO:teuthology.orchestra.run.smithi017.stdout: "up" 2024-09-11T09:52:12.200 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:12.200 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:12.200 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:12.200 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 2, 2024-09-11T09:52:12.200 INFO:teuthology.orchestra.run.smithi017.stdout: "uuid": "4c8e50db-b49c-4d44-a913-47b5b30f8699", 2024-09-11T09:52:12.201 INFO:teuthology.orchestra.run.smithi017.stdout: "up": 1, 2024-09-11T09:52:12.201 INFO:teuthology.orchestra.run.smithi017.stdout: "in": 1, 2024-09-11T09:52:12.201 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 1, 2024-09-11T09:52:12.201 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity": 1, 2024-09-11T09:52:12.201 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_begin": 0, 2024-09-11T09:52:12.201 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_end": 0, 2024-09-11T09:52:12.201 INFO:teuthology.orchestra.run.smithi017.stdout: "up_from": 11, 2024-09-11T09:52:12.201 INFO:teuthology.orchestra.run.smithi017.stdout: "up_thru": 54, 2024-09-11T09:52:12.201 INFO:teuthology.orchestra.run.smithi017.stdout: "down_at": 0, 2024-09-11T09:52:12.201 INFO:teuthology.orchestra.run.smithi017.stdout: "lost_at": 0, 2024-09-11T09:52:12.201 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-11T09:52:12.201 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-11T09:52:12.201 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:12.202 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-11T09:52:12.202 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6816", 2024-09-11T09:52:12.202 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1151282506 2024-09-11T09:52:12.202 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:12.202 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:12.202 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-11T09:52:12.202 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6817", 2024-09-11T09:52:12.202 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1151282506 2024-09-11T09:52:12.202 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:12.202 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:12.202 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:12.202 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addrs": { 2024-09-11T09:52:12.202 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-11T09:52:12.203 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:12.203 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-11T09:52:12.203 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6818", 2024-09-11T09:52:12.203 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1151282506 2024-09-11T09:52:12.203 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:12.203 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:12.203 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-11T09:52:12.203 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6819", 2024-09-11T09:52:12.203 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1151282506 2024-09-11T09:52:12.203 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:12.203 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:12.203 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:12.203 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addrs": { 2024-09-11T09:52:12.204 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-11T09:52:12.204 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:12.204 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-11T09:52:12.204 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6822", 2024-09-11T09:52:12.204 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1151282506 2024-09-11T09:52:12.204 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:12.204 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:12.204 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-11T09:52:12.205 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6823", 2024-09-11T09:52:12.205 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1151282506 2024-09-11T09:52:12.205 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:12.205 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:12.205 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:12.205 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addrs": { 2024-09-11T09:52:12.205 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-11T09:52:12.205 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:12.205 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-11T09:52:12.205 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6820", 2024-09-11T09:52:12.206 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1151282506 2024-09-11T09:52:12.206 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:12.206 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:12.206 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-11T09:52:12.206 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.17:6821", 2024-09-11T09:52:12.206 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 1151282506 2024-09-11T09:52:12.206 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:12.206 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:12.206 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:12.206 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.17:6817/1151282506", 2024-09-11T09:52:12.206 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addr": "172.21.15.17:6819/1151282506", 2024-09-11T09:52:12.206 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addr": "172.21.15.17:6823/1151282506", 2024-09-11T09:52:12.206 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addr": "172.21.15.17:6821/1151282506", 2024-09-11T09:52:12.206 INFO:teuthology.orchestra.run.smithi017.stdout: "state": [ 2024-09-11T09:52:12.206 INFO:teuthology.orchestra.run.smithi017.stdout: "exists", 2024-09-11T09:52:12.207 INFO:teuthology.orchestra.run.smithi017.stdout: "up" 2024-09-11T09:52:12.207 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:12.207 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:12.207 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:12.207 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 3, 2024-09-11T09:52:12.207 INFO:teuthology.orchestra.run.smithi017.stdout: "uuid": "37abc55b-a1eb-49d5-b1ff-b507ee2cb5a6", 2024-09-11T09:52:12.207 INFO:teuthology.orchestra.run.smithi017.stdout: "up": 1, 2024-09-11T09:52:12.207 INFO:teuthology.orchestra.run.smithi017.stdout: "in": 1, 2024-09-11T09:52:12.207 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 1, 2024-09-11T09:52:12.207 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity": 1, 2024-09-11T09:52:12.208 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_begin": 0, 2024-09-11T09:52:12.208 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_end": 0, 2024-09-11T09:52:12.208 INFO:teuthology.orchestra.run.smithi017.stdout: "up_from": 12, 2024-09-11T09:52:12.208 INFO:teuthology.orchestra.run.smithi017.stdout: "up_thru": 83, 2024-09-11T09:52:12.208 INFO:teuthology.orchestra.run.smithi017.stdout: "down_at": 0, 2024-09-11T09:52:12.208 INFO:teuthology.orchestra.run.smithi017.stdout: "lost_at": 0, 2024-09-11T09:52:12.208 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-11T09:52:12.208 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-11T09:52:12.208 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:12.208 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-11T09:52:12.208 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.96:6816", 2024-09-11T09:52:12.208 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 332854807 2024-09-11T09:52:12.208 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:12.209 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:12.209 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-11T09:52:12.209 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.96:6817", 2024-09-11T09:52:12.209 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 332854807 2024-09-11T09:52:12.209 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:12.209 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:12.209 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:12.209 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addrs": { 2024-09-11T09:52:12.209 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-11T09:52:12.209 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:12.210 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-11T09:52:12.210 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.96:6818", 2024-09-11T09:52:12.210 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 332854807 2024-09-11T09:52:12.210 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:12.210 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:12.210 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-11T09:52:12.210 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.96:6819", 2024-09-11T09:52:12.210 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 332854807 2024-09-11T09:52:12.210 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:12.210 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:12.210 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:12.210 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addrs": { 2024-09-11T09:52:12.211 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-11T09:52:12.211 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:12.211 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-11T09:52:12.211 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.96:6822", 2024-09-11T09:52:12.211 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 332854807 2024-09-11T09:52:12.211 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:12.211 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:12.211 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-11T09:52:12.211 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.96:6823", 2024-09-11T09:52:12.211 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 332854807 2024-09-11T09:52:12.211 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:12.211 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:12.211 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:12.212 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addrs": { 2024-09-11T09:52:12.212 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-11T09:52:12.212 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:12.212 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-11T09:52:12.212 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.96:6820", 2024-09-11T09:52:12.212 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 332854807 2024-09-11T09:52:12.212 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:12.212 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:12.212 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-11T09:52:12.212 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.96:6821", 2024-09-11T09:52:12.212 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 332854807 2024-09-11T09:52:12.212 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:12.212 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:12.212 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:12.213 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.96:6817/332854807", 2024-09-11T09:52:12.213 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addr": "172.21.15.96:6819/332854807", 2024-09-11T09:52:12.213 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addr": "172.21.15.96:6823/332854807", 2024-09-11T09:52:12.213 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addr": "172.21.15.96:6821/332854807", 2024-09-11T09:52:12.213 INFO:teuthology.orchestra.run.smithi017.stdout: "state": [ 2024-09-11T09:52:12.213 INFO:teuthology.orchestra.run.smithi017.stdout: "exists", 2024-09-11T09:52:12.213 INFO:teuthology.orchestra.run.smithi017.stdout: "up" 2024-09-11T09:52:12.213 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:12.213 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:12.213 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:12.213 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 4, 2024-09-11T09:52:12.213 INFO:teuthology.orchestra.run.smithi017.stdout: "uuid": "067d9586-4f5a-4d17-96c5-86a7b8bcd114", 2024-09-11T09:52:12.213 INFO:teuthology.orchestra.run.smithi017.stdout: "up": 1, 2024-09-11T09:52:12.213 INFO:teuthology.orchestra.run.smithi017.stdout: "in": 1, 2024-09-11T09:52:12.214 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 1, 2024-09-11T09:52:12.214 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity": 1, 2024-09-11T09:52:12.214 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_begin": 11, 2024-09-11T09:52:12.214 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_end": 54, 2024-09-11T09:52:12.214 INFO:teuthology.orchestra.run.smithi017.stdout: "up_from": 66, 2024-09-11T09:52:12.214 INFO:teuthology.orchestra.run.smithi017.stdout: "up_thru": 80, 2024-09-11T09:52:12.214 INFO:teuthology.orchestra.run.smithi017.stdout: "down_at": 55, 2024-09-11T09:52:12.214 INFO:teuthology.orchestra.run.smithi017.stdout: "lost_at": 0, 2024-09-11T09:52:12.214 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-11T09:52:12.214 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-11T09:52:12.215 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:12.215 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-11T09:52:12.215 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.96:6800", 2024-09-11T09:52:12.215 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3374675329 2024-09-11T09:52:12.215 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:12.215 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:12.215 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-11T09:52:12.215 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.96:6801", 2024-09-11T09:52:12.215 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3374675329 2024-09-11T09:52:12.215 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:12.216 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:12.216 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:12.216 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addrs": { 2024-09-11T09:52:12.216 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-11T09:52:12.216 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:12.216 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-11T09:52:12.216 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.96:6802", 2024-09-11T09:52:12.217 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3374675329 2024-09-11T09:52:12.217 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:12.217 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:12.217 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-11T09:52:12.217 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.96:6803", 2024-09-11T09:52:12.217 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3374675329 2024-09-11T09:52:12.217 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:12.217 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:12.218 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:12.218 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addrs": { 2024-09-11T09:52:12.218 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-11T09:52:12.218 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:12.218 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-11T09:52:12.218 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.96:6806", 2024-09-11T09:52:12.218 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3374675329 2024-09-11T09:52:12.218 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:12.219 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:12.219 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-11T09:52:12.219 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.96:6807", 2024-09-11T09:52:12.219 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3374675329 2024-09-11T09:52:12.219 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:12.219 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:12.219 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:12.220 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addrs": { 2024-09-11T09:52:12.220 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-11T09:52:12.220 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:12.220 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-11T09:52:12.220 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.96:6804", 2024-09-11T09:52:12.220 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3374675329 2024-09-11T09:52:12.220 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:12.221 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:12.221 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-11T09:52:12.221 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.96:6805", 2024-09-11T09:52:12.221 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 3374675329 2024-09-11T09:52:12.221 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:12.221 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:12.221 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:12.221 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.96:6801/3374675329", 2024-09-11T09:52:12.221 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addr": "172.21.15.96:6803/3374675329", 2024-09-11T09:52:12.222 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addr": "172.21.15.96:6807/3374675329", 2024-09-11T09:52:12.222 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addr": "172.21.15.96:6805/3374675329", 2024-09-11T09:52:12.222 INFO:teuthology.orchestra.run.smithi017.stdout: "state": [ 2024-09-11T09:52:12.222 INFO:teuthology.orchestra.run.smithi017.stdout: "exists", 2024-09-11T09:52:12.222 INFO:teuthology.orchestra.run.smithi017.stdout: "up" 2024-09-11T09:52:12.222 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:12.222 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:12.223 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:12.223 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 5, 2024-09-11T09:52:12.223 INFO:teuthology.orchestra.run.smithi017.stdout: "uuid": "8ff7a724-0a3a-4aa2-baf1-fd52f88d476c", 2024-09-11T09:52:12.223 INFO:teuthology.orchestra.run.smithi017.stdout: "up": 1, 2024-09-11T09:52:12.223 INFO:teuthology.orchestra.run.smithi017.stdout: "in": 1, 2024-09-11T09:52:12.223 INFO:teuthology.orchestra.run.smithi017.stdout: "weight": 1, 2024-09-11T09:52:12.223 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_affinity": 1, 2024-09-11T09:52:12.223 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_begin": 0, 2024-09-11T09:52:12.223 INFO:teuthology.orchestra.run.smithi017.stdout: "last_clean_end": 0, 2024-09-11T09:52:12.224 INFO:teuthology.orchestra.run.smithi017.stdout: "up_from": 11, 2024-09-11T09:52:12.224 INFO:teuthology.orchestra.run.smithi017.stdout: "up_thru": 54, 2024-09-11T09:52:12.224 INFO:teuthology.orchestra.run.smithi017.stdout: "down_at": 0, 2024-09-11T09:52:12.224 INFO:teuthology.orchestra.run.smithi017.stdout: "lost_at": 0, 2024-09-11T09:52:12.224 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addrs": { 2024-09-11T09:52:12.224 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-11T09:52:12.224 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:12.224 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-11T09:52:12.225 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.96:6808", 2024-09-11T09:52:12.225 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 509935370 2024-09-11T09:52:12.225 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:12.225 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:12.225 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-11T09:52:12.225 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.96:6809", 2024-09-11T09:52:12.225 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 509935370 2024-09-11T09:52:12.225 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:12.226 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:12.226 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:12.226 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addrs": { 2024-09-11T09:52:12.226 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-11T09:52:12.226 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:12.226 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-11T09:52:12.226 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.96:6810", 2024-09-11T09:52:12.226 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 509935370 2024-09-11T09:52:12.226 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:12.227 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:12.227 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-11T09:52:12.227 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.96:6811", 2024-09-11T09:52:12.227 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 509935370 2024-09-11T09:52:12.227 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:12.227 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:12.227 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:12.227 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addrs": { 2024-09-11T09:52:12.227 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-11T09:52:12.227 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:12.227 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-11T09:52:12.227 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.96:6814", 2024-09-11T09:52:12.227 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 509935370 2024-09-11T09:52:12.228 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:12.228 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:12.228 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-11T09:52:12.228 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.96:6815", 2024-09-11T09:52:12.228 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 509935370 2024-09-11T09:52:12.228 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:12.228 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:12.228 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:12.228 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addrs": { 2024-09-11T09:52:12.228 INFO:teuthology.orchestra.run.smithi017.stdout: "addrvec": [ 2024-09-11T09:52:12.228 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:12.228 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v2", 2024-09-11T09:52:12.229 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.96:6812", 2024-09-11T09:52:12.229 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 509935370 2024-09-11T09:52:12.229 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:12.229 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:12.229 INFO:teuthology.orchestra.run.smithi017.stdout: "type": "v1", 2024-09-11T09:52:12.229 INFO:teuthology.orchestra.run.smithi017.stdout: "addr": "172.21.15.96:6813", 2024-09-11T09:52:12.229 INFO:teuthology.orchestra.run.smithi017.stdout: "nonce": 509935370 2024-09-11T09:52:12.229 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:12.229 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:12.229 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:12.229 INFO:teuthology.orchestra.run.smithi017.stdout: "public_addr": "172.21.15.96:6809/509935370", 2024-09-11T09:52:12.229 INFO:teuthology.orchestra.run.smithi017.stdout: "cluster_addr": "172.21.15.96:6811/509935370", 2024-09-11T09:52:12.230 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_back_addr": "172.21.15.96:6815/509935370", 2024-09-11T09:52:12.230 INFO:teuthology.orchestra.run.smithi017.stdout: "heartbeat_front_addr": "172.21.15.96:6813/509935370", 2024-09-11T09:52:12.230 INFO:teuthology.orchestra.run.smithi017.stdout: "state": [ 2024-09-11T09:52:12.230 INFO:teuthology.orchestra.run.smithi017.stdout: "exists", 2024-09-11T09:52:12.230 INFO:teuthology.orchestra.run.smithi017.stdout: "up" 2024-09-11T09:52:12.230 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:12.230 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:12.230 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-11T09:52:12.230 INFO:teuthology.orchestra.run.smithi017.stdout: "osd_xinfo": [ 2024-09-11T09:52:12.230 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:12.230 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 0, 2024-09-11T09:52:12.230 INFO:teuthology.orchestra.run.smithi017.stdout: "down_stamp": "0.000000", 2024-09-11T09:52:12.231 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_probability": 0, 2024-09-11T09:52:12.231 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_interval": 0, 2024-09-11T09:52:12.231 INFO:teuthology.orchestra.run.smithi017.stdout: "features": 4540701547738038271, 2024-09-11T09:52:12.231 INFO:teuthology.orchestra.run.smithi017.stdout: "old_weight": 65536, 2024-09-11T09:52:12.231 INFO:teuthology.orchestra.run.smithi017.stdout: "last_purged_snaps_scrub": "2024-09-11T09:50:08.513270+0000", 2024-09-11T09:52:12.231 INFO:teuthology.orchestra.run.smithi017.stdout: "dead_epoch": 0 2024-09-11T09:52:12.231 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:12.231 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:12.231 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 1, 2024-09-11T09:52:12.231 INFO:teuthology.orchestra.run.smithi017.stdout: "down_stamp": "0.000000", 2024-09-11T09:52:12.231 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_probability": 0, 2024-09-11T09:52:12.232 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_interval": 0, 2024-09-11T09:52:12.232 INFO:teuthology.orchestra.run.smithi017.stdout: "features": 4540701547738038271, 2024-09-11T09:52:12.232 INFO:teuthology.orchestra.run.smithi017.stdout: "old_weight": 0, 2024-09-11T09:52:12.232 INFO:teuthology.orchestra.run.smithi017.stdout: "last_purged_snaps_scrub": "2024-09-11T09:50:07.522863+0000", 2024-09-11T09:52:12.232 INFO:teuthology.orchestra.run.smithi017.stdout: "dead_epoch": 0 2024-09-11T09:52:12.232 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:12.232 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:12.232 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 2, 2024-09-11T09:52:12.232 INFO:teuthology.orchestra.run.smithi017.stdout: "down_stamp": "0.000000", 2024-09-11T09:52:12.232 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_probability": 0, 2024-09-11T09:52:12.232 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_interval": 0, 2024-09-11T09:52:12.232 INFO:teuthology.orchestra.run.smithi017.stdout: "features": 4540701547738038271, 2024-09-11T09:52:12.233 INFO:teuthology.orchestra.run.smithi017.stdout: "old_weight": 0, 2024-09-11T09:52:12.233 INFO:teuthology.orchestra.run.smithi017.stdout: "last_purged_snaps_scrub": "2024-09-11T09:50:08.134364+0000", 2024-09-11T09:52:12.233 INFO:teuthology.orchestra.run.smithi017.stdout: "dead_epoch": 0 2024-09-11T09:52:12.233 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:12.233 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:12.233 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 3, 2024-09-11T09:52:12.233 INFO:teuthology.orchestra.run.smithi017.stdout: "down_stamp": "0.000000", 2024-09-11T09:52:12.233 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_probability": 0, 2024-09-11T09:52:12.233 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_interval": 0, 2024-09-11T09:52:12.233 INFO:teuthology.orchestra.run.smithi017.stdout: "features": 4540701547738038271, 2024-09-11T09:52:12.233 INFO:teuthology.orchestra.run.smithi017.stdout: "old_weight": 0, 2024-09-11T09:52:12.233 INFO:teuthology.orchestra.run.smithi017.stdout: "last_purged_snaps_scrub": "2024-09-11T09:50:08.622366+0000", 2024-09-11T09:52:12.233 INFO:teuthology.orchestra.run.smithi017.stdout: "dead_epoch": 0 2024-09-11T09:52:12.234 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:12.234 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:12.234 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 4, 2024-09-11T09:52:12.234 INFO:teuthology.orchestra.run.smithi017.stdout: "down_stamp": "0.000000", 2024-09-11T09:52:12.234 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_probability": 0, 2024-09-11T09:52:12.234 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_interval": 0, 2024-09-11T09:52:12.234 INFO:teuthology.orchestra.run.smithi017.stdout: "features": 4540701547738038271, 2024-09-11T09:52:12.234 INFO:teuthology.orchestra.run.smithi017.stdout: "old_weight": 0, 2024-09-11T09:52:12.234 INFO:teuthology.orchestra.run.smithi017.stdout: "last_purged_snaps_scrub": "2024-09-11T09:50:08.156142+0000", 2024-09-11T09:52:12.234 INFO:teuthology.orchestra.run.smithi017.stdout: "dead_epoch": 55 2024-09-11T09:52:12.234 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:12.234 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:12.234 INFO:teuthology.orchestra.run.smithi017.stdout: "osd": 5, 2024-09-11T09:52:12.234 INFO:teuthology.orchestra.run.smithi017.stdout: "down_stamp": "0.000000", 2024-09-11T09:52:12.235 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_probability": 0, 2024-09-11T09:52:12.235 INFO:teuthology.orchestra.run.smithi017.stdout: "laggy_interval": 0, 2024-09-11T09:52:12.235 INFO:teuthology.orchestra.run.smithi017.stdout: "features": 4540701547738038271, 2024-09-11T09:52:12.235 INFO:teuthology.orchestra.run.smithi017.stdout: "old_weight": 0, 2024-09-11T09:52:12.235 INFO:teuthology.orchestra.run.smithi017.stdout: "last_purged_snaps_scrub": "2024-09-11T09:50:08.108116+0000", 2024-09-11T09:52:12.235 INFO:teuthology.orchestra.run.smithi017.stdout: "dead_epoch": 0 2024-09-11T09:52:12.235 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:12.235 INFO:teuthology.orchestra.run.smithi017.stdout: ], 2024-09-11T09:52:12.235 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_upmap": [], 2024-09-11T09:52:12.235 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_upmap_items": [], 2024-09-11T09:52:12.235 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_upmap_primaries": [], 2024-09-11T09:52:12.235 INFO:teuthology.orchestra.run.smithi017.stdout: "pg_temp": [], 2024-09-11T09:52:12.235 INFO:teuthology.orchestra.run.smithi017.stdout: "primary_temp": [], 2024-09-11T09:52:12.235 INFO:teuthology.orchestra.run.smithi017.stdout: "blocklist": {}, 2024-09-11T09:52:12.236 INFO:teuthology.orchestra.run.smithi017.stdout: "range_blocklist": {}, 2024-09-11T09:52:12.236 INFO:teuthology.orchestra.run.smithi017.stdout: "erasure_code_profiles": { 2024-09-11T09:52:12.236 INFO:teuthology.orchestra.run.smithi017.stdout: "default": { 2024-09-11T09:52:12.236 INFO:teuthology.orchestra.run.smithi017.stdout: "crush-failure-domain": "osd", 2024-09-11T09:52:12.236 INFO:teuthology.orchestra.run.smithi017.stdout: "k": "2", 2024-09-11T09:52:12.236 INFO:teuthology.orchestra.run.smithi017.stdout: "m": "1", 2024-09-11T09:52:12.236 INFO:teuthology.orchestra.run.smithi017.stdout: "plugin": "jerasure", 2024-09-11T09:52:12.236 INFO:teuthology.orchestra.run.smithi017.stdout: "technique": "reed_sol_van" 2024-09-11T09:52:12.236 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:12.236 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:12.236 INFO:teuthology.orchestra.run.smithi017.stdout: "removed_snaps_queue": [], 2024-09-11T09:52:12.236 INFO:teuthology.orchestra.run.smithi017.stdout: "new_removed_snaps": [], 2024-09-11T09:52:12.236 INFO:teuthology.orchestra.run.smithi017.stdout: "new_purged_snaps": [], 2024-09-11T09:52:12.237 INFO:teuthology.orchestra.run.smithi017.stdout: "crush_node_flags": {}, 2024-09-11T09:52:12.237 INFO:teuthology.orchestra.run.smithi017.stdout: "device_class_flags": {}, 2024-09-11T09:52:12.237 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode": { 2024-09-11T09:52:12.237 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode_enabled": false, 2024-09-11T09:52:12.237 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_bucket_count": 0, 2024-09-11T09:52:12.237 INFO:teuthology.orchestra.run.smithi017.stdout: "degraded_stretch_mode": 0, 2024-09-11T09:52:12.237 INFO:teuthology.orchestra.run.smithi017.stdout: "recovering_stretch_mode": 0, 2024-09-11T09:52:12.237 INFO:teuthology.orchestra.run.smithi017.stdout: "stretch_mode_bucket": 0 2024-09-11T09:52:12.237 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:12.237 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:52:12.239 INFO:tasks.thrashosds.thrasher:j is {'epoch': 84, 'fsid': 'a2f1daec-d3b0-4bf5-82c8-72ecbb93d4e4', 'created': '2024-09-11T09:50:00.165804+0000', 'modified': '2024-09-11T09:52:10.851328+0000', 'last_up_change': '2024-09-11T09:51:44.764990+0000', 'last_in_change': '2024-09-11T09:52:06.832073+0000', 'flags': 'nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5804032, 'flags_set': ['nodeep-scrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 6, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 3, 'max_osd': 6, 'require_min_compat_client': 'luminous', 'min_compat_client': 'jewel', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-11T09:50:10.457184+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '14', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 6.059999942779541, 'score_stable': 6.059999942779541, 'optimal_score': 0.33000001311302185, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2024-09-11T09:50:16.993202+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '83', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 7, 'snap_epoch': 83, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.2699999809265137, 'score_stable': 2.2699999809265137, 'optimal_score': 0.8299999833106995, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'new_default_pool27121', 'create_time': '2024-09-11T09:51:23.493016+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '58', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 58, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 3.009999990463257, 'score_stable': 3.009999990463257, 'optimal_score': 0.8299999833106995, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}], 'osds': [{'osd': 0, 'uuid': '4829b7a1-d607-4a56-b6d2-55036cb8acf8', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 12, 'up_thru': 77, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6808', 'nonce': 4154449929}, {'type': 'v1', 'addr': '172.21.15.17:6809', 'nonce': 4154449929}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6810', 'nonce': 4154449929}, {'type': 'v1', 'addr': '172.21.15.17:6811', 'nonce': 4154449929}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6814', 'nonce': 4154449929}, {'type': 'v1', 'addr': '172.21.15.17:6815', 'nonce': 4154449929}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6812', 'nonce': 4154449929}, {'type': 'v1', 'addr': '172.21.15.17:6813', 'nonce': 4154449929}]}, 'public_addr': '172.21.15.17:6809/4154449929', 'cluster_addr': '172.21.15.17:6811/4154449929', 'heartbeat_back_addr': '172.21.15.17:6815/4154449929', 'heartbeat_front_addr': '172.21.15.17:6813/4154449929', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': 'ab1d5df0-3c0e-44cd-88a0-96b07964148b', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 83, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6800', 'nonce': 4139511644}, {'type': 'v1', 'addr': '172.21.15.17:6801', 'nonce': 4139511644}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6802', 'nonce': 4139511644}, {'type': 'v1', 'addr': '172.21.15.17:6803', 'nonce': 4139511644}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6806', 'nonce': 4139511644}, {'type': 'v1', 'addr': '172.21.15.17:6807', 'nonce': 4139511644}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6804', 'nonce': 4139511644}, {'type': 'v1', 'addr': '172.21.15.17:6805', 'nonce': 4139511644}]}, 'public_addr': '172.21.15.17:6801/4139511644', 'cluster_addr': '172.21.15.17:6803/4139511644', 'heartbeat_back_addr': '172.21.15.17:6807/4139511644', 'heartbeat_front_addr': '172.21.15.17:6805/4139511644', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '4c8e50db-b49c-4d44-a913-47b5b30f8699', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 54, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6816', 'nonce': 1151282506}, {'type': 'v1', 'addr': '172.21.15.17:6817', 'nonce': 1151282506}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6818', 'nonce': 1151282506}, {'type': 'v1', 'addr': '172.21.15.17:6819', 'nonce': 1151282506}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6822', 'nonce': 1151282506}, {'type': 'v1', 'addr': '172.21.15.17:6823', 'nonce': 1151282506}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.17:6820', 'nonce': 1151282506}, {'type': 'v1', 'addr': '172.21.15.17:6821', 'nonce': 1151282506}]}, 'public_addr': '172.21.15.17:6817/1151282506', 'cluster_addr': '172.21.15.17:6819/1151282506', 'heartbeat_back_addr': '172.21.15.17:6823/1151282506', 'heartbeat_front_addr': '172.21.15.17:6821/1151282506', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '37abc55b-a1eb-49d5-b1ff-b507ee2cb5a6', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 12, 'up_thru': 83, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.96:6816', 'nonce': 332854807}, {'type': 'v1', 'addr': '172.21.15.96:6817', 'nonce': 332854807}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.96:6818', 'nonce': 332854807}, {'type': 'v1', 'addr': '172.21.15.96:6819', 'nonce': 332854807}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.96:6822', 'nonce': 332854807}, {'type': 'v1', 'addr': '172.21.15.96:6823', 'nonce': 332854807}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.96:6820', 'nonce': 332854807}, {'type': 'v1', 'addr': '172.21.15.96:6821', 'nonce': 332854807}]}, 'public_addr': '172.21.15.96:6817/332854807', 'cluster_addr': '172.21.15.96:6819/332854807', 'heartbeat_back_addr': '172.21.15.96:6823/332854807', 'heartbeat_front_addr': '172.21.15.96:6821/332854807', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '067d9586-4f5a-4d17-96c5-86a7b8bcd114', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 11, 'last_clean_end': 54, 'up_from': 66, 'up_thru': 80, 'down_at': 55, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.96:6800', 'nonce': 3374675329}, {'type': 'v1', 'addr': '172.21.15.96:6801', 'nonce': 3374675329}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.96:6802', 'nonce': 3374675329}, {'type': 'v1', 'addr': '172.21.15.96:6803', 'nonce': 3374675329}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.96:6806', 'nonce': 3374675329}, {'type': 'v1', 'addr': '172.21.15.96:6807', 'nonce': 3374675329}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.96:6804', 'nonce': 3374675329}, {'type': 'v1', 'addr': '172.21.15.96:6805', 'nonce': 3374675329}]}, 'public_addr': '172.21.15.96:6801/3374675329', 'cluster_addr': '172.21.15.96:6803/3374675329', 'heartbeat_back_addr': '172.21.15.96:6807/3374675329', 'heartbeat_front_addr': '172.21.15.96:6805/3374675329', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '8ff7a724-0a3a-4aa2-baf1-fd52f88d476c', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 11, 'up_thru': 54, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.96:6808', 'nonce': 509935370}, {'type': 'v1', 'addr': '172.21.15.96:6809', 'nonce': 509935370}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.96:6810', 'nonce': 509935370}, {'type': 'v1', 'addr': '172.21.15.96:6811', 'nonce': 509935370}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.96:6814', 'nonce': 509935370}, {'type': 'v1', 'addr': '172.21.15.96:6815', 'nonce': 509935370}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.96:6812', 'nonce': 509935370}, {'type': 'v1', 'addr': '172.21.15.96:6813', 'nonce': 509935370}]}, 'public_addr': '172.21.15.96:6809/509935370', 'cluster_addr': '172.21.15.96:6811/509935370', 'heartbeat_back_addr': '172.21.15.96:6815/509935370', 'heartbeat_front_addr': '172.21.15.96:6813/509935370', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-09-11T09:50:08.513270+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-11T09:50:07.522863+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-11T09:50:08.134364+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-11T09:50:08.622366+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-11T09:50:08.156142+0000', 'dead_epoch': 55}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-11T09:50:08.108116+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-11T09:52:12.240 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T09:52:12.240 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:52:12.239+0000 7f3c96ec9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31909) UID: 0 2024-09-11T09:52:12.244 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:52:12.244 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:52:12.253 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-11T09:52:12.340 INFO:tasks.ceph.osd.0.smithi017.stderr:2024-09-11T09:52:12.341+0000 7fb73c0f1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31665) UID: 0 2024-09-11T09:52:12.390 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:52:12.390 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:52:12.400 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-11T09:52:12.441 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:52:12.442+0000 7f3a7b90a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 42475) UID: 0 2024-09-11T09:52:12.542 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:52:12.543+0000 7f3c96ec9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31909) UID: 0 2024-09-11T09:52:12.548 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:52:12.548 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:52:12.556 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-11T09:52:12.570 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-11T09:52:12.570 INFO:teuthology.orchestra.run.smithi017.stderr:dumped all 2024-09-11T09:52:12.583 INFO:tasks.workunit.client.0.smithi107.stderr:+ DEV= 2024-09-11T09:52:12.583 INFO:tasks.workunit.client.0.smithi107.stderr:+ cat /tmp/tmp.rvrcDirNue/rbd-nbd.log 2024-09-11T09:52:12.583 INFO:teuthology.orchestra.run.smithi017.stdout:{"pg_ready":true,"pg_map":{"version":140,"stamp":"2024-09-11T09:52:12.435714+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":146651005,"num_objects":74,"num_object_clones":18,"num_object_copies":148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":74,"num_whiteouts":7,"num_read":4460,"num_read_kb":165142,"num_write":914,"num_write_kb":273437,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":41107170,"num_keys_recovered":14,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":851,"ondisk_log_size":851,"up":34,"acting":34,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":30,"num_osds":5,"num_per_pool_osds":5,"num_per_pool_omap_osds":5,"kb":471859200,"kb_used":412544,"kb_used_data":251348,"kb_used_omap":10,"kb_used_meta":161077,"kb_avail":471446656,"statfs":{"total":483183820800,"available":482761375744,"internally_reserved":0,"allocated":257380352,"data_stored":275390125,"data_compressed":452534,"data_compressed_allocated":606208,"data_compressed_original":9125888,"omap_allocated":10808,"internal_metadata":164943304},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":8696832,"num_objects":15,"num_object_clones":15,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":7,"num_read":481,"num_read_kb":1284,"num_write":155,"num_write_kb":4576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":35732016,"num_keys_recovered":14,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.595906"},"pg_stats":[{"pgid":"1.0","version":"13'117","reported_seq":340,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T09:52:10.859814+0000","last_change":"2024-09-11T09:52:10.859352+0000","last_active":"2024-09-11T09:52:10.859814+0000","last_peered":"2024-09-11T09:52:10.859814+0000","last_clean":"2024-09-11T09:52:10.859814+0000","last_became_active":"2024-09-11T09:52:10.858918+0000","last_became_peered":"2024-09-11T09:52:10.858918+0000","last_unstale":"2024-09-11T09:52:10.859814+0000","last_undegraded":"2024-09-11T09:52:10.859814+0000","last_fullsized":"2024-09-11T09:52:10.859814+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:50:10.457262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:50:10.457262+0000","last_clean_scrub_stamp":"2024-09-11T09:50:10.457262+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:32:35.326531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"79'32","reported_seq":228,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-11T09:52:09.884299+0000","last_change":"2024-09-11T09:50:20.500687+0000","last_active":"2024-09-11T09:52:09.884299+0000","last_peered":"2024-09-11T09:52:09.884299+0000","last_clean":"2024-09-11T09:52:09.884299+0000","last_became_active":"2024-09-11T09:50:18.493650+0000","last_became_peered":"2024-09-11T09:50:18.493650+0000","last_unstale":"2024-09-11T09:52:09.884299+0000","last_undegraded":"2024-09-11T09:52:09.884299+0000","last_fullsized":"2024-09-11T09:52:09.884299+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_clean_scrub_stamp":"2024-09-11T09:50:17.477890+0000","objects_scrubbed":0,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:44:39.593950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041261100000000001,"stat_sum":{"num_bytes":25956352,"num_objects":8,"num_object_clones":1,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":45,"num_read_kb":12304,"num_write":30,"num_write_kb":33288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"67'3","reported_seq":77,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-11T09:52:09.852686+0000","last_change":"2024-09-11T09:51:27.719936+0000","last_active":"2024-09-11T09:52:09.852686+0000","last_peered":"2024-09-11T09:52:09.852686+0000","last_clean":"2024-09-11T09:52:09.852686+0000","last_became_active":"2024-09-11T09:51:24.714496+0000","last_became_peered":"2024-09-11T09:51:24.714496+0000","last_unstale":"2024-09-11T09:52:09.852686+0000","last_undegraded":"2024-09-11T09:52:09.852686+0000","last_fullsized":"2024-09-11T09:52:09.852686+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:51:23.699749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:51:23.699749+0000","last_clean_scrub_stamp":"2024-09-11T09:51:23.699749+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T20:58:07.190652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00068421499999999995,"stat_sum":{"num_bytes":54,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":5,"num_read_kb":4,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"74'6","reported_seq":315,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T09:52:10.858185+0000","last_change":"2024-09-11T09:50:20.498284+0000","last_active":"2024-09-11T09:52:10.858185+0000","last_peered":"2024-09-11T09:52:10.858185+0000","last_clean":"2024-09-11T09:52:10.858185+0000","last_became_active":"2024-09-11T09:50:18.493210+0000","last_became_peered":"2024-09-11T09:50:18.493210+0000","last_unstale":"2024-09-11T09:52:10.858185+0000","last_undegraded":"2024-09-11T09:52:10.858185+0000","last_fullsized":"2024-09-11T09:52:10.858185+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_clean_scrub_stamp":"2024-09-11T09:50:17.477890+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:06:18.020045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079749499999999995,"stat_sum":{"num_bytes":12582988,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":157,"num_read_kb":2850,"num_write":5,"num_write_kb":12290,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"67'1","reported_seq":54,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T09:52:07.853215+0000","last_change":"2024-09-11T09:52:07.849281+0000","last_active":"2024-09-11T09:52:07.853215+0000","last_peered":"2024-09-11T09:52:07.853215+0000","last_clean":"2024-09-11T09:52:07.853215+0000","last_became_active":"2024-09-11T09:52:07.845118+0000","last_became_peered":"2024-09-11T09:52:07.845118+0000","last_unstale":"2024-09-11T09:52:07.853215+0000","last_undegraded":"2024-09-11T09:52:07.853215+0000","last_fullsized":"2024-09-11T09:52:07.853215+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:51:23.699749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:51:23.699749+0000","last_clean_scrub_stamp":"2024-09-11T09:51:23.699749+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T21:07:40.901843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041844700000000001,"stat_sum":{"num_bytes":16,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":16,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1","version":"82'149","reported_seq":1433,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-11T09:52:09.852975+0000","last_change":"2024-09-11T09:50:21.030566+0000","last_active":"2024-09-11T09:52:09.852975+0000","last_peered":"2024-09-11T09:52:09.852975+0000","last_clean":"2024-09-11T09:52:09.852975+0000","last_became_active":"2024-09-11T09:50:18.493109+0000","last_became_peered":"2024-09-11T09:50:18.493109+0000","last_unstale":"2024-09-11T09:52:09.852975+0000","last_undegraded":"2024-09-11T09:52:09.852975+0000","last_fullsized":"2024-09-11T09:52:09.852975+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_clean_scrub_stamp":"2024-09-11T09:50:17.477890+0000","objects_scrubbed":0,"log_size":149,"log_dups_size":0,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T20:07:51.753387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041008499999999997,"stat_sum":{"num_bytes":28597302,"num_objects":13,"num_object_clones":5,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":1344,"num_read_kb":41277,"num_write":130,"num_write_kb":50537,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":70,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-11T09:52:09.852667+0000","last_change":"2024-09-11T09:51:27.719437+0000","last_active":"2024-09-11T09:52:09.852667+0000","last_peered":"2024-09-11T09:52:09.852667+0000","last_clean":"2024-09-11T09:52:09.852667+0000","last_became_active":"2024-09-11T09:51:24.714263+0000","last_became_peered":"2024-09-11T09:51:24.714263+0000","last_unstale":"2024-09-11T09:52:09.852667+0000","last_undegraded":"2024-09-11T09:52:09.852667+0000","last_fullsized":"2024-09-11T09:52:09.852667+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:51:23.699749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:51:23.699749+0000","last_clean_scrub_stamp":"2024-09-11T09:51:23.699749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:06:36.882446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045627500000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"79'24","reported_seq":302,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-11T09:52:09.884280+0000","last_change":"2024-09-11T09:50:20.501218+0000","last_active":"2024-09-11T09:52:09.884280+0000","last_peered":"2024-09-11T09:52:09.884280+0000","last_clean":"2024-09-11T09:52:09.884280+0000","last_became_active":"2024-09-11T09:50:18.493165+0000","last_became_peered":"2024-09-11T09:50:18.493165+0000","last_unstale":"2024-09-11T09:52:09.884280+0000","last_undegraded":"2024-09-11T09:52:09.884280+0000","last_fullsized":"2024-09-11T09:52:09.884280+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_clean_scrub_stamp":"2024-09-11T09:50:17.477890+0000","objects_scrubbed":0,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T19:11:24.668941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00094463900000000005,"stat_sum":{"num_bytes":12582931,"num_objects":5,"num_object_clones":1,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":127,"num_read_kb":8321,"num_write":23,"num_write_kb":20482,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"56'1","reported_seq":56,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-11T09:52:07.851088+0000","last_change":"2024-09-11T09:52:07.851088+0000","last_active":"2024-09-11T09:52:07.851088+0000","last_peered":"2024-09-11T09:52:07.851088+0000","last_clean":"2024-09-11T09:52:07.851088+0000","last_became_active":"2024-09-11T09:52:07.848151+0000","last_became_peered":"2024-09-11T09:52:07.848151+0000","last_unstale":"2024-09-11T09:52:07.851088+0000","last_undegraded":"2024-09-11T09:52:07.851088+0000","last_fullsized":"2024-09-11T09:52:07.851088+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:51:23.699749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:51:23.699749+0000","last_clean_scrub_stamp":"2024-09-11T09:51:23.699749+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T20:26:55.098291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040505599999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":70,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-11T09:52:09.884254+0000","last_change":"2024-09-11T09:51:27.729396+0000","last_active":"2024-09-11T09:52:09.884254+0000","last_peered":"2024-09-11T09:52:09.884254+0000","last_clean":"2024-09-11T09:52:09.884254+0000","last_became_active":"2024-09-11T09:51:24.715777+0000","last_became_peered":"2024-09-11T09:51:24.715777+0000","last_unstale":"2024-09-11T09:52:09.884254+0000","last_undegraded":"2024-09-11T09:52:09.884254+0000","last_fullsized":"2024-09-11T09:52:09.884254+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:51:23.699749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:51:23.699749+0000","last_clean_scrub_stamp":"2024-09-11T09:51:23.699749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:56:41.450651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026452099999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"82'191","reported_seq":1084,"reported_epoch":82,"state":"peering","last_fresh":"2024-09-11T09:52:09.031752+0000","last_change":"2024-09-11T09:52:09.853799+0000","last_active":"2024-09-11T09:52:09.853799+0000","last_peered":"2024-09-11T09:52:09.031752+0000","last_clean":"2024-09-11T09:52:06.769360+0000","last_became_active":"2024-09-11T09:52:08.941037+0000","last_became_peered":"2024-09-11T09:52:08.941037+0000","last_unstale":"2024-09-11T09:52:09.031752+0000","last_undegraded":"2024-09-11T09:52:09.031752+0000","last_fullsized":"2024-09-11T09:52:08.849330+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_clean_scrub_stamp":"2024-09-11T09:50:17.477890+0000","objects_scrubbed":0,"log_size":191,"log_dups_size":0,"ondisk_log_size":191,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00050337500000000002,"stat_sum":{"num_bytes":22289408,"num_objects":12,"num_object_clones":4,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":2,"num_read":1513,"num_read_kb":34003,"num_write":120,"num_write_kb":54295,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":22031360,"num_keys_recovered":14,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,4],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"68'4","reported_seq":84,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-11T09:52:09.853317+0000","last_change":"2024-09-11T09:51:27.720403+0000","last_active":"2024-09-11T09:52:09.853317+0000","last_peered":"2024-09-11T09:52:09.853317+0000","last_clean":"2024-09-11T09:52:09.853317+0000","last_became_active":"2024-09-11T09:51:24.714345+0000","last_became_peered":"2024-09-11T09:51:24.714345+0000","last_unstale":"2024-09-11T09:52:09.853317+0000","last_undegraded":"2024-09-11T09:52:09.853317+0000","last_fullsized":"2024-09-11T09:52:09.853317+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:51:23.699749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:51:23.699749+0000","last_clean_scrub_stamp":"2024-09-11T09:51:23.699749+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T20:24:54.033307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040133200000000001,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":28,"num_read_kb":25,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"79'46","reported_seq":296,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T09:52:10.857825+0000","last_change":"2024-09-11T09:52:07.917418+0000","last_active":"2024-09-11T09:52:10.857825+0000","last_peered":"2024-09-11T09:52:10.857825+0000","last_clean":"2024-09-11T09:52:10.857825+0000","last_became_active":"2024-09-11T09:52:07.853491+0000","last_became_peered":"2024-09-11T09:52:07.853491+0000","last_unstale":"2024-09-11T09:52:10.857825+0000","last_undegraded":"2024-09-11T09:52:10.857825+0000","last_fullsized":"2024-09-11T09:52:10.857825+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_clean_scrub_stamp":"2024-09-11T09:50:17.477890+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T19:04:30.211623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045546699999999998,"stat_sum":{"num_bytes":13110272,"num_objects":5,"num_object_clones":2,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":71,"num_read_kb":24580,"num_write":44,"num_write_kb":29188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":13110272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":71,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-11T09:52:09.852842+0000","last_change":"2024-09-11T09:51:27.870068+0000","last_active":"2024-09-11T09:52:09.852842+0000","last_peered":"2024-09-11T09:52:09.852842+0000","last_clean":"2024-09-11T09:52:09.852842+0000","last_became_active":"2024-09-11T09:51:24.716325+0000","last_became_peered":"2024-09-11T09:51:24.716325+0000","last_unstale":"2024-09-11T09:52:09.852842+0000","last_undegraded":"2024-09-11T09:52:09.852842+0000","last_fullsized":"2024-09-11T09:52:09.852842+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:51:23.699749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:51:23.699749+0000","last_clean_scrub_stamp":"2024-09-11T09:51:23.699749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T19:44:13.062941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035292500000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.6","version":"79'116","reported_seq":458,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-11T09:52:09.853209+0000","last_change":"2024-09-11T09:50:20.497446+0000","last_active":"2024-09-11T09:52:09.853209+0000","last_peered":"2024-09-11T09:52:09.853209+0000","last_clean":"2024-09-11T09:52:09.853209+0000","last_became_active":"2024-09-11T09:50:18.497481+0000","last_became_peered":"2024-09-11T09:50:18.497481+0000","last_unstale":"2024-09-11T09:52:09.853209+0000","last_undegraded":"2024-09-11T09:52:09.853209+0000","last_fullsized":"2024-09-11T09:52:09.853209+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_clean_scrub_stamp":"2024-09-11T09:50:17.477890+0000","objects_scrubbed":0,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T18:30:25.298071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00050697600000000002,"stat_sum":{"num_bytes":26746896,"num_objects":13,"num_object_clones":5,"num_object_copies":26,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":3,"num_read":207,"num_read_kb":40990,"num_write":111,"num_write_kb":66579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":15,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-09-11T09:52:09.852905+0000","last_change":"2024-09-11T09:52:07.845599+0000","last_active":"2024-09-11T09:52:09.852905+0000","last_peered":"2024-09-11T09:52:09.852905+0000","last_clean":"2024-09-11T09:52:09.852905+0000","last_became_active":"2024-09-11T09:52:07.845316+0000","last_became_peered":"2024-09-11T09:52:07.845316+0000","last_unstale":"2024-09-11T09:52:09.852905+0000","last_undegraded":"2024-09-11T09:52:09.852905+0000","last_fullsized":"2024-09-11T09:52:09.852905+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:51:23.699749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:51:23.699749+0000","last_clean_scrub_stamp":"2024-09-11T09:51:23.699749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:25:27.357405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"84'161","reported_seq":293,"reported_epoch":84,"state":"active+clean","last_fresh":"2024-09-11T09:52:10.865587+0000","last_change":"2024-09-11T09:52:04.840883+0000","last_active":"2024-09-11T09:52:10.865587+0000","last_peered":"2024-09-11T09:52:10.865587+0000","last_clean":"2024-09-11T09:52:10.865587+0000","last_became_active":"2024-09-11T09:52:04.840466+0000","last_became_peered":"2024-09-11T09:52:04.840466+0000","last_unstale":"2024-09-11T09:52:10.865587+0000","last_undegraded":"2024-09-11T09:52:10.865587+0000","last_fullsized":"2024-09-11T09:52:10.865587+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_clean_scrub_stamp":"2024-09-11T09:50:17.477890+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:09:32.466344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00046094899999999999,"stat_sum":{"num_bytes":4194418,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":845,"num_read_kb":688,"num_write":286,"num_write_kb":4356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":4194418,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":8,"stat_sum":{"num_bytes":70,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":6,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":16,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":156,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":16,"acting":16,"num_store_stats":6},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":146060567,"num_objects":66,"num_object_clones":18,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66,"num_whiteouts":7,"num_read":4309,"num_read_kb":165013,"num_write":749,"num_write_kb":271015,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":39336050,"num_keys_recovered":14,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":285954048,"data_stored":305905198,"data_compressed":385906,"data_compressed_allocated":516096,"data_compressed_original":7962624,"omap_allocated":0,"internal_metadata":0},"log_size":725,"ondisk_log_size":725,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1771104,"data_compressed":100760,"data_compressed_allocated":135168,"data_compressed_original":1753088,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":12,"seq":51539607580,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":85620,"kb_used_data":54428,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94286220,"statfs":{"total":96636764160,"available":96549089280,"internally_reserved":0,"allocated":55734272,"data_stored":57086577,"data_compressed":105864,"data_compressed_allocated":143360,"data_compressed_original":2129920,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":0,"seq":51539607579,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":11,"seq":47244640284,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69052,"kb_used_data":37864,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94302788,"statfs":{"total":96636764160,"available":96566054912,"internally_reserved":0,"allocated":38772736,"data_stored":38654063,"data_compressed":24712,"data_compressed_allocated":32768,"data_compressed_original":524288,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":66,"seq":283467841542,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":45480,"kb_used_data":17360,"kb_used_omap":4,"kb_used_meta":28091,"kb_avail":94326360,"statfs":{"total":96636764160,"available":96590192640,"internally_reserved":0,"allocated":17776640,"data_stored":17667136,"data_compressed":24718,"data_compressed_allocated":32768,"data_compressed_original":524288,"omap_allocated":4432,"internal_metadata":28765872},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640283,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":83568,"kb_used_data":48220,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94288272,"statfs":{"total":96636764160,"available":96551190528,"internally_reserved":0,"allocated":49377280,"data_stored":56831084,"data_compressed":94870,"data_compressed_allocated":126976,"data_compressed_original":1900544,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640283,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":128824,"kb_used_data":93476,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94243016,"statfs":{"total":96636764160,"available":96504848384,"internally_reserved":0,"allocated":95719424,"data_stored":105151265,"data_compressed":202370,"data_compressed_allocated":270336,"data_compressed_original":4046848,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":34132,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":34132,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32496,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":33554432,"data_stored":33554432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":94691328,"data_stored":104188069,"data_compressed":168238,"data_compressed_allocated":225280,"data_compressed_original":3457024,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":48402432,"data_stored":56458294,"data_compressed":94870,"data_compressed_allocated":126976,"data_compressed_original":1900544,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54685696,"data_stored":56118478,"data_compressed":73368,"data_compressed_allocated":98304,"data_compressed_original":1556480,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16826368,"data_stored":17304690,"data_compressed":24718,"data_compressed_allocated":32768,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":37793792,"data_stored":38281235,"data_compressed":24712,"data_compressed_allocated":32768,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":16,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":54,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":16,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":16,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":54,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T09:52:12.585 INFO:tasks.thrashosds.thrasher:Setting 3.2 to [0, 3] 2024-09-11T09:52:12.585 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '3.2', '0', '3'] 2024-09-11T09:52:12.585 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pg-upmap 3.2 0 3 2024-09-11T09:52:12.585 INFO:tasks.workunit.client.0.smithi107.stdout:2024-09-11T09:52:06.499+0000 7fd94bc44880 0 ceph version 19.1.1-298-gb53b6d3e (b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc) squid (stable), process rbd-nbd, pid 29564 2024-09-11T09:52:12.586 INFO:tasks.workunit.client.0.smithi107.stdout:2024-09-11T09:52:06.499+0000 7fd94bc44880 0 pidfile_write: ignore empty --pid-file 2024-09-11T09:52:12.586 INFO:tasks.workunit.client.0.smithi107.stdout:2024-09-11T09:52:06.719+0000 7fd943fff640 0 -- 172.21.15.107:0/3911579706 >> [v2:172.21.15.17:6816/1151282506,v1:172.21.15.17:6817/1151282506] conn(0x5562ba13b5d0 msgr2=0x5562ba13d9c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-11T09:52:12.586 INFO:tasks.workunit.client.0.smithi107.stdout:2024-09-11T09:52:06.726+0000 7fd943fff640 0 -- 172.21.15.107:0/3911579706 >> [v2:172.21.15.17:6816/1151282506,v1:172.21.15.17:6817/1151282506] conn(0x7fd92c03dae0 msgr2=0x7fd92c03fed0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-11T09:52:12.586 INFO:tasks.workunit.client.0.smithi107.stdout:2024-09-11T09:52:11.995+0000 7fd929ffb640 0 rbd-nbd: disconnect request received 2024-09-11T09:52:12.587 INFO:tasks.workunit.client.0.smithi107.stderr:+ expect_false grep 'quiesce failed' /tmp/tmp.rvrcDirNue/rbd-nbd.log 2024-09-11T09:52:12.587 INFO:tasks.workunit.client.0.smithi107.stderr:+ grep 'quiesce failed' /tmp/tmp.rvrcDirNue/rbd-nbd.log 2024-09-11T09:52:12.587 INFO:tasks.workunit.client.0.smithi107.stderr:+ return 0 2024-09-11T09:52:12.587 INFO:tasks.workunit.client.0.smithi107.stderr:++ _sudo rbd device --device-type nbd --show-cookie map rbd/testrbdnbd27121 2024-09-11T09:52:12.587 INFO:tasks.workunit.client.0.smithi107.stderr:++ local cmd 2024-09-11T09:52:12.587 INFO:tasks.workunit.client.0.smithi107.stderr:+++ id -u 2024-09-11T09:52:12.587 INFO:tasks.workunit.client.0.smithi107.stderr:++ '[' 1000 -eq 0 ']' 2024-09-11T09:52:12.587 INFO:tasks.workunit.client.0.smithi107.stderr:+++ which rbd 2024-09-11T09:52:12.588 INFO:tasks.workunit.client.0.smithi107.stderr:++ cmd=/usr/bin/rbd 2024-09-11T09:52:12.588 INFO:tasks.workunit.client.0.smithi107.stderr:++ shift 2024-09-11T09:52:12.589 INFO:tasks.workunit.client.0.smithi107.stderr:++ sudo -nE /usr/bin/rbd device --device-type nbd --show-cookie map rbd/testrbdnbd27121 2024-09-11T09:52:12.593 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T09:52:12.666 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:52:12.667 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:52:12.668 INFO:tasks.ceph.osd.2.smithi017.stderr:2024-09-11T09:52:12.669+0000 7fd20b170640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31672) UID: 0 2024-09-11T09:52:12.678 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-11T09:52:12.692 INFO:tasks.workunit.client.0.smithi107.stderr:+ OUT='/dev/nbd1 eaab7410-3050-4b82-9298-7fac480d4de7' 2024-09-11T09:52:12.692 INFO:tasks.workunit.client.0.smithi107.stderr:+ read DEV COOKIE 2024-09-11T09:52:12.692 INFO:tasks.workunit.client.0.smithi107.stderr:+ get_pid rbd 2024-09-11T09:52:12.692 INFO:tasks.workunit.client.0.smithi107.stderr:+ local pool=rbd 2024-09-11T09:52:12.692 INFO:tasks.workunit.client.0.smithi107.stderr:+ local ns= 2024-09-11T09:52:12.692 INFO:tasks.workunit.client.0.smithi107.stderr:++ rbd device --device-type nbd --format xml list 2024-09-11T09:52:12.693 INFO:tasks.workunit.client.0.smithi107.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd27121'\''][device='\''/dev/nbd1'\'']/id' 2024-09-11T09:52:12.761 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-11T09:52:12.762 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T09:52:12.762 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:52:12.762+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:52:12.775 INFO:tasks.workunit.client.0.smithi107.stderr:+ PID=29700 2024-09-11T09:52:12.775 INFO:tasks.workunit.client.0.smithi107.stderr:+ test -n 29700 2024-09-11T09:52:12.775 INFO:tasks.workunit.client.0.smithi107.stderr:+ ps -p 29700 -C rbd-nbd 2024-09-11T09:52:12.783 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-11T09:52:12.783 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T09:52:12.786 INFO:tasks.workunit.client.0.smithi107.stdout: PID TTY TIME CMD 2024-09-11T09:52:12.786 INFO:tasks.workunit.client.0.smithi107.stdout: 29700 ? 00:00:00 rbd-nbd 2024-09-11T09:52:12.787 INFO:tasks.workunit.client.0.smithi107.stderr:+ _sudo mount /dev/nbd1 /tmp/tmp.rvrcDirNue/mnt 2024-09-11T09:52:12.787 INFO:tasks.workunit.client.0.smithi107.stderr:+ local cmd 2024-09-11T09:52:12.787 INFO:tasks.workunit.client.0.smithi107.stderr:++ id -u 2024-09-11T09:52:12.787 INFO:tasks.workunit.client.0.smithi107.stderr:+ '[' 1000 -eq 0 ']' 2024-09-11T09:52:12.788 INFO:tasks.workunit.client.0.smithi107.stderr:++ which mount 2024-09-11T09:52:12.789 INFO:tasks.workunit.client.0.smithi107.stderr:+ cmd=/usr/bin/mount 2024-09-11T09:52:12.789 INFO:tasks.workunit.client.0.smithi107.stderr:+ shift 2024-09-11T09:52:12.790 INFO:tasks.workunit.client.0.smithi107.stderr:+ sudo -nE /usr/bin/mount /dev/nbd1 /tmp/tmp.rvrcDirNue/mnt 2024-09-11T09:52:12.801 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-11T09:52:12.802 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T09:52:12.829 INFO:tasks.workunit.client.0.smithi107.stderr:+ _sudo rbd device detach rbd/testrbdnbd27121 --device-type nbd 2024-09-11T09:52:12.829 INFO:tasks.workunit.client.0.smithi107.stderr:+ local cmd 2024-09-11T09:52:12.831 INFO:tasks.workunit.client.0.smithi107.stderr:++ id -u 2024-09-11T09:52:12.832 INFO:tasks.workunit.client.0.smithi107.stderr:+ '[' 1000 -eq 0 ']' 2024-09-11T09:52:12.833 INFO:tasks.workunit.client.0.smithi107.stderr:++ which rbd 2024-09-11T09:52:12.833 INFO:tasks.workunit.client.0.smithi107.stderr:+ cmd=/usr/bin/rbd 2024-09-11T09:52:12.833 INFO:tasks.workunit.client.0.smithi107.stderr:+ shift 2024-09-11T09:52:12.833 INFO:tasks.workunit.client.0.smithi107.stderr:+ sudo -nE /usr/bin/rbd device detach rbd/testrbdnbd27121 --device-type nbd 2024-09-11T09:52:12.834 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-11T09:52:12.835 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T09:52:12.837 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:52:12.837 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [], 2024-09-11T09:52:12.837 INFO:teuthology.orchestra.run.smithi017.stdout: "num_ops": 0 2024-09-11T09:52:12.837 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:52:12.844 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-09-11T09:52:12.845+0000 7f7b6c7d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31904) UID: 0 2024-09-11T09:52:12.846 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-11T09:52:12.861 INFO:teuthology.orchestra.run.smithi017.stderr:nodeep-scrub is unset 2024-09-11T09:52:12.869 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-11T09:52:12.869 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T09:52:12.905 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-11T09:52:12.905 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T09:52:12.929 INFO:tasks.workunit.client.0.smithi107.stderr:+ expect_false get_pid rbd 2024-09-11T09:52:12.929 INFO:tasks.workunit.client.0.smithi107.stderr:+ get_pid rbd 2024-09-11T09:52:12.929 INFO:tasks.workunit.client.0.smithi107.stderr:+ local pool=rbd 2024-09-11T09:52:12.929 INFO:tasks.workunit.client.0.smithi107.stderr:+ local ns= 2024-09-11T09:52:12.930 INFO:tasks.workunit.client.0.smithi107.stderr:++ rbd device --device-type nbd --format xml list 2024-09-11T09:52:12.930 INFO:tasks.workunit.client.0.smithi107.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd27121'\''][device='\''/dev/nbd1'\'']/id' 2024-09-11T09:52:12.945 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:52:12.947+0000 7f3c96ec9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31909) UID: 0 2024-09-11T09:52:12.970 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:52:12.970 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [], 2024-09-11T09:52:12.970 INFO:teuthology.orchestra.run.smithi017.stdout: "complaint_time": 30, 2024-09-11T09:52:12.970 INFO:teuthology.orchestra.run.smithi017.stdout: "num_blocked_ops": 0 2024-09-11T09:52:12.970 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:52:12.979 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-11T09:52:12.996 INFO:tasks.workunit.client.0.smithi107.stderr:+ PID= 2024-09-11T09:52:12.996 INFO:tasks.workunit.client.0.smithi107.stderr:+ test -n '' 2024-09-11T09:52:12.997 INFO:tasks.workunit.client.0.smithi107.stderr:+ return 1 2024-09-11T09:52:12.997 INFO:tasks.workunit.client.0.smithi107.stderr:+ return 0 2024-09-11T09:52:12.997 INFO:tasks.workunit.client.0.smithi107.stderr:+ expect_false _sudo rbd device attach --device /dev/nbd1 rbd/testrbdnbd27121 --device-type nbd 2024-09-11T09:52:12.997 INFO:tasks.workunit.client.0.smithi107.stderr:+ _sudo rbd device attach --device /dev/nbd1 rbd/testrbdnbd27121 --device-type nbd 2024-09-11T09:52:12.997 INFO:tasks.workunit.client.0.smithi107.stderr:+ local cmd 2024-09-11T09:52:12.997 INFO:tasks.workunit.client.0.smithi107.stderr:++ id -u 2024-09-11T09:52:12.997 INFO:tasks.workunit.client.0.smithi107.stderr:+ '[' 1000 -eq 0 ']' 2024-09-11T09:52:12.998 INFO:tasks.workunit.client.0.smithi107.stderr:++ which rbd 2024-09-11T09:52:12.999 INFO:tasks.workunit.client.0.smithi107.stderr:+ cmd=/usr/bin/rbd 2024-09-11T09:52:12.999 INFO:tasks.workunit.client.0.smithi107.stderr:+ shift 2024-09-11T09:52:12.999 INFO:tasks.workunit.client.0.smithi107.stderr:+ sudo -nE /usr/bin/rbd device attach --device /dev/nbd1 rbd/testrbdnbd27121 --device-type nbd 2024-09-11T09:52:13.034 INFO:tasks.workunit.client.0.smithi107.stderr:rbd: could not validate attach request 2024-09-11T09:52:13.035 INFO:tasks.workunit.client.0.smithi107.stderr:rbd: mismatching the image and the device may lead to data corruption 2024-09-11T09:52:13.035 INFO:tasks.workunit.client.0.smithi107.stderr:rbd: must specify --cookie or --force to proceed 2024-09-11T09:52:13.037 INFO:tasks.workunit.client.0.smithi107.stderr:+ return 0 2024-09-11T09:52:13.037 INFO:tasks.workunit.client.0.smithi107.stderr:+ '[' -n eaab7410-3050-4b82-9298-7fac480d4de7 ']' 2024-09-11T09:52:13.037 INFO:tasks.workunit.client.0.smithi107.stderr:+ _sudo rbd device attach --device /dev/nbd1 --cookie eaab7410-3050-4b82-9298-7fac480d4de7 rbd/testrbdnbd27121 --device-type nbd 2024-09-11T09:52:13.038 INFO:tasks.workunit.client.0.smithi107.stderr:+ local cmd 2024-09-11T09:52:13.038 INFO:tasks.workunit.client.0.smithi107.stderr:++ id -u 2024-09-11T09:52:13.038 INFO:tasks.workunit.client.0.smithi107.stderr:+ '[' 1000 -eq 0 ']' 2024-09-11T09:52:13.038 INFO:tasks.workunit.client.0.smithi107.stderr:++ which rbd 2024-09-11T09:52:13.041 INFO:tasks.workunit.client.0.smithi107.stderr:+ cmd=/usr/bin/rbd 2024-09-11T09:52:13.041 INFO:tasks.workunit.client.0.smithi107.stderr:+ shift 2024-09-11T09:52:13.041 INFO:tasks.workunit.client.0.smithi107.stderr:+ sudo -nE /usr/bin/rbd device attach --device /dev/nbd1 --cookie eaab7410-3050-4b82-9298-7fac480d4de7 rbd/testrbdnbd27121 --device-type nbd 2024-09-11T09:52:13.046 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-09-11T09:52:13.048+0000 7f7b6c7d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31904) UID: 0 2024-09-11T09:52:13.089 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:52:13.089 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 20, 2024-09-11T09:52:13.089 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 600, 2024-09-11T09:52:13.090 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [ 2024-09-11T09:52:13.090 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.090 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:34 2.4 2:2367385a:::rbd_data.1250f5e25b80.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:13.090 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.897717+0000", 2024-09-11T09:52:13.090 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 50.191464885000002, 2024-09-11T09:52:13.090 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.021323661000000001, 2024-09-11T09:52:13.090 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:13.090 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:13.090 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:13.090 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:13.090 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:52:13.091 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:52:13.091 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 34 2024-09-11T09:52:13.091 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.091 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:13.091 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.091 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:13.091 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.897717+0000", 2024-09-11T09:52:13.091 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:13.091 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.091 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.091 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:13.092 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.897717+0000", 2024-09-11T09:52:13.092 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:13.092 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.092 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.092 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:13.092 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.897718+0000", 2024-09-11T09:52:13.092 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.1800000000000004e-07 2024-09-11T09:52:13.092 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.092 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.092 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:13.092 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.905339+0000", 2024-09-11T09:52:13.092 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0076210239999999997 2024-09-11T09:52:13.092 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.093 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.093 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:13.093 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.905366+0000", 2024-09-11T09:52:13.093 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.6959000000000001e-05 2024-09-11T09:52:13.093 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.093 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.093 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:13.093 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.905375+0000", 2024-09-11T09:52:13.093 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.516e-06 2024-09-11T09:52:13.093 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.093 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.094 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:13.094 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.905475+0000", 2024-09-11T09:52:13.094 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.9738e-05 2024-09-11T09:52:13.094 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.094 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.094 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:13.094 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.905660+0000", 2024-09-11T09:52:13.094 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00018554199999999999 2024-09-11T09:52:13.095 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.095 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.095 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 0", 2024-09-11T09:52:13.095 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.905808+0000", 2024-09-11T09:52:13.095 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000147386 2024-09-11T09:52:13.095 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.095 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.095 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:13.095 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.918230+0000", 2024-09-11T09:52:13.095 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.012421690000000001 2024-09-11T09:52:13.095 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.095 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.095 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:13.096 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.918932+0000", 2024-09-11T09:52:13.096 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00070267300000000001 2024-09-11T09:52:13.096 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.096 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.096 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:13.096 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.918989+0000", 2024-09-11T09:52:13.096 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.7268999999999997e-05 2024-09-11T09:52:13.096 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.096 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.097 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:13.097 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919040+0000", 2024-09-11T09:52:13.097 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.0946000000000003e-05 2024-09-11T09:52:13.097 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:13.097 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:13.097 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:13.097 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.097 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.097 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4688.0:31 2.1 e53/15 2:8654721b:::rbd_data.1250f5e25b80.0000000000000000:head v 53'2, mlcod=0'0)", 2024-09-11T09:52:13.098 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.904661+0000", 2024-09-11T09:52:13.098 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 50.184520396000003, 2024-09-11T09:52:13.098 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.014152804999999999, 2024-09-11T09:52:13.098 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:13.098 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:13.098 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:13.098 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:13.098 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.098 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:13.098 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.904661+0000", 2024-09-11T09:52:13.098 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:13.098 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.099 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.099 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:13.099 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.904661+0000", 2024-09-11T09:52:13.099 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:13.099 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.099 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.099 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:13.099 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.904662+0000", 2024-09-11T09:52:13.099 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0759999999999999e-06 2024-09-11T09:52:13.099 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.099 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.099 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:13.100 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.907298+0000", 2024-09-11T09:52:13.100 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0026361240000000001 2024-09-11T09:52:13.100 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.100 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.100 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:13.100 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.907320+0000", 2024-09-11T09:52:13.100 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.1268999999999999e-05 2024-09-11T09:52:13.100 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.100 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.100 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:13.100 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.907331+0000", 2024-09-11T09:52:13.100 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1031e-05 2024-09-11T09:52:13.100 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.101 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.101 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:13.101 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.907403+0000", 2024-09-11T09:52:13.101 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.2571999999999996e-05 2024-09-11T09:52:13.101 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.101 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.101 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:13.101 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.907428+0000", 2024-09-11T09:52:13.101 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.4380999999999999e-05 2024-09-11T09:52:13.101 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.101 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.101 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:13.101 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.918750+0000", 2024-09-11T09:52:13.101 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.011322211 2024-09-11T09:52:13.102 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.102 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.102 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:13.102 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.918814+0000", 2024-09-11T09:52:13.102 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.4140999999999995e-05 2024-09-11T09:52:13.102 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:13.102 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:13.102 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:13.102 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.102 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.102 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:36 2.6 2:61e1b442:::rbd_data.1250f5e25b80.0000000000000003:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:13.102 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.905404+0000", 2024-09-11T09:52:13.102 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 50.183777524999996, 2024-09-11T09:52:13.103 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.030912479999999999, 2024-09-11T09:52:13.103 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:13.103 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:13.103 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:13.103 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:13.103 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:52:13.103 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:52:13.103 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 36 2024-09-11T09:52:13.103 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.103 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:13.103 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.103 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:13.103 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.905404+0000", 2024-09-11T09:52:13.104 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:13.104 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.104 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.104 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:13.104 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.905404+0000", 2024-09-11T09:52:13.104 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:13.104 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.104 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.104 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:13.104 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.905405+0000", 2024-09-11T09:52:13.104 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2270000000000001e-06 2024-09-11T09:52:13.104 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.104 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.105 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:13.105 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.914859+0000", 2024-09-11T09:52:13.105 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0094541190000000004 2024-09-11T09:52:13.105 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.105 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.105 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:13.105 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.914879+0000", 2024-09-11T09:52:13.105 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.0052999999999999e-05 2024-09-11T09:52:13.105 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.105 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.105 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:13.105 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.914897+0000", 2024-09-11T09:52:13.106 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7331000000000001e-05 2024-09-11T09:52:13.106 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.106 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.106 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:13.106 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.915019+0000", 2024-09-11T09:52:13.106 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00012168 2024-09-11T09:52:13.106 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.106 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.106 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:13.106 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.915269+0000", 2024-09-11T09:52:13.106 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00025054899999999998 2024-09-11T09:52:13.106 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.106 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.107 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 3", 2024-09-11T09:52:13.107 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.915479+0000", 2024-09-11T09:52:13.107 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00021017799999999999 2024-09-11T09:52:13.107 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.107 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.107 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:13.107 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.926070+0000", 2024-09-11T09:52:13.107 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.010590624999999999 2024-09-11T09:52:13.107 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.108 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.108 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:13.108 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.936198+0000", 2024-09-11T09:52:13.108 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.010128194 2024-09-11T09:52:13.108 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.108 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.108 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:13.108 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.936263+0000", 2024-09-11T09:52:13.108 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.5366999999999999e-05 2024-09-11T09:52:13.108 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.108 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.108 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:13.108 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.936317+0000", 2024-09-11T09:52:13.109 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.3156999999999999e-05 2024-09-11T09:52:13.109 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:13.109 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:13.109 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:13.109 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.109 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.109 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:39 2.6 2:743e7616:::rbd_data.1250f5e25b80.0000000000000005:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:13.109 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.914960+0000", 2024-09-11T09:52:13.109 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 50.174221223000004, 2024-09-11T09:52:13.109 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.022371518, 2024-09-11T09:52:13.109 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:13.109 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:13.109 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:13.110 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:13.110 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:52:13.110 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:52:13.110 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 39 2024-09-11T09:52:13.110 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.110 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:13.110 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.110 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:13.110 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.914960+0000", 2024-09-11T09:52:13.110 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:13.110 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.110 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.110 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:13.110 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.914960+0000", 2024-09-11T09:52:13.111 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:13.111 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.111 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.111 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:13.111 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.914963+0000", 2024-09-11T09:52:13.111 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.1119999999999999e-06 2024-09-11T09:52:13.111 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.111 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.111 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:13.111 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.920417+0000", 2024-09-11T09:52:13.111 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0054543969999999997 2024-09-11T09:52:13.111 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.112 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.112 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:13.112 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.920430+0000", 2024-09-11T09:52:13.112 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3414000000000001e-05 2024-09-11T09:52:13.112 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.112 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.112 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:13.112 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.920437+0000", 2024-09-11T09:52:13.113 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.6159999999999998e-06 2024-09-11T09:52:13.113 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.113 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.113 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:13.113 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.921325+0000", 2024-09-11T09:52:13.113 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000887731 2024-09-11T09:52:13.113 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.113 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.113 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:13.113 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.921422+0000", 2024-09-11T09:52:13.114 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.7201999999999995e-05 2024-09-11T09:52:13.114 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.114 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.114 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 3", 2024-09-11T09:52:13.114 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.921511+0000", 2024-09-11T09:52:13.114 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.8880000000000003e-05 2024-09-11T09:52:13.114 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.114 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.114 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:13.114 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.928092+0000", 2024-09-11T09:52:13.114 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0065816290000000003 2024-09-11T09:52:13.114 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.114 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.114 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:13.115 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.937240+0000", 2024-09-11T09:52:13.115 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0091481070000000008 2024-09-11T09:52:13.115 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.115 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.115 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:13.115 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.937294+0000", 2024-09-11T09:52:13.115 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.3100000000000003e-05 2024-09-11T09:52:13.115 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.115 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.115 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:13.115 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.937332+0000", 2024-09-11T09:52:13.115 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.8330000000000001e-05 2024-09-11T09:52:13.115 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:13.115 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:13.116 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:13.116 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.116 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.116 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4688.0:32 2.2 e53/15 2:45941fd2:::rbd_data.1250f5e25b80.0000000000000001:head v 53'3, mlcod=17'1)", 2024-09-11T09:52:13.116 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.916509+0000", 2024-09-11T09:52:13.116 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 50.172672141, 2024-09-11T09:52:13.116 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.015885318999999998, 2024-09-11T09:52:13.116 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:13.116 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:13.116 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:13.117 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:13.117 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.117 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:13.117 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.916509+0000", 2024-09-11T09:52:13.117 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:13.117 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.117 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.117 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:13.117 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.916509+0000", 2024-09-11T09:52:13.117 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:13.117 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.117 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.117 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:13.118 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.916511+0000", 2024-09-11T09:52:13.118 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.052e-06 2024-09-11T09:52:13.118 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.118 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.118 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:13.118 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.927019+0000", 2024-09-11T09:52:13.118 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.010508224 2024-09-11T09:52:13.118 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.118 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.118 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:13.118 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.927038+0000", 2024-09-11T09:52:13.118 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.9590999999999999e-05 2024-09-11T09:52:13.118 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.118 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.119 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:13.119 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.927050+0000", 2024-09-11T09:52:13.119 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1439e-05 2024-09-11T09:52:13.119 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.119 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.119 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:13.119 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.927115+0000", 2024-09-11T09:52:13.119 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.5301000000000004e-05 2024-09-11T09:52:13.119 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.119 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.119 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:13.120 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.927132+0000", 2024-09-11T09:52:13.120 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7297000000000001e-05 2024-09-11T09:52:13.120 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.120 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.120 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:13.120 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.932354+0000", 2024-09-11T09:52:13.120 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0052211870000000004 2024-09-11T09:52:13.120 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.120 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.121 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:13.121 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.932395+0000", 2024-09-11T09:52:13.121 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.1227999999999999e-05 2024-09-11T09:52:13.121 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:13.121 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:13.121 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:13.121 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.121 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.121 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:44 2.6 2:67246a22:::rbd_data.1250f5e25b80.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:13.121 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.943181+0000", 2024-09-11T09:52:13.121 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 50.14600093, 2024-09-11T09:52:13.121 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.022629371999999998, 2024-09-11T09:52:13.122 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:13.122 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:13.122 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:13.122 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:13.122 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:52:13.122 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:52:13.122 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 44 2024-09-11T09:52:13.122 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.122 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:13.122 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.122 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:13.122 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.943181+0000", 2024-09-11T09:52:13.122 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:13.123 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.123 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.123 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:13.123 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.943181+0000", 2024-09-11T09:52:13.123 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:13.123 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.123 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.123 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:13.123 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.943182+0000", 2024-09-11T09:52:13.123 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.0500000000000002e-07 2024-09-11T09:52:13.124 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.124 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.124 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:13.124 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951240+0000", 2024-09-11T09:52:13.124 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.008058529 2024-09-11T09:52:13.124 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.124 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.124 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:13.125 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951260+0000", 2024-09-11T09:52:13.125 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.9570000000000001e-05 2024-09-11T09:52:13.125 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.125 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.125 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:13.125 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951270+0000", 2024-09-11T09:52:13.125 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0018e-05 2024-09-11T09:52:13.125 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.125 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.125 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:13.125 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951349+0000", 2024-09-11T09:52:13.125 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.8801000000000007e-05 2024-09-11T09:52:13.125 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.126 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.126 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:13.126 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951483+0000", 2024-09-11T09:52:13.126 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00013427 2024-09-11T09:52:13.126 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.126 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.126 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 3", 2024-09-11T09:52:13.126 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951593+0000", 2024-09-11T09:52:13.126 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00011031000000000001 2024-09-11T09:52:13.126 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.126 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.126 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:13.126 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.957303+0000", 2024-09-11T09:52:13.127 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0057097959999999996 2024-09-11T09:52:13.127 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.127 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.127 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:13.127 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.965690+0000", 2024-09-11T09:52:13.127 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0083867420000000008 2024-09-11T09:52:13.127 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.127 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.127 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:13.127 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.965749+0000", 2024-09-11T09:52:13.127 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.8981999999999999e-05 2024-09-11T09:52:13.127 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.127 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.128 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:13.128 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.965810+0000", 2024-09-11T09:52:13.128 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.1449000000000001e-05 2024-09-11T09:52:13.128 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:13.128 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:13.128 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:13.128 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.128 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.128 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:51 2.6 2:6b5894fc:::rbd_data.1250f5e25b80.000000000000000a:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:13.128 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.951324+0000", 2024-09-11T09:52:13.128 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 50.137857857, 2024-09-11T09:52:13.128 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.021375273, 2024-09-11T09:52:13.128 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:13.129 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:13.129 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:13.129 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:13.129 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:52:13.129 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:52:13.129 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 51 2024-09-11T09:52:13.129 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.129 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:13.129 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.129 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:13.129 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951324+0000", 2024-09-11T09:52:13.129 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:13.129 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.130 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.130 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:13.130 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951324+0000", 2024-09-11T09:52:13.130 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:13.130 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.130 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.130 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:13.130 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951324+0000", 2024-09-11T09:52:13.130 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.39e-07 2024-09-11T09:52:13.130 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.130 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.130 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:13.130 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.958537+0000", 2024-09-11T09:52:13.131 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0072126769999999998 2024-09-11T09:52:13.131 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.131 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.131 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:13.131 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.958554+0000", 2024-09-11T09:52:13.131 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7232e-05 2024-09-11T09:52:13.131 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.131 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.131 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:13.131 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.958570+0000", 2024-09-11T09:52:13.131 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.5747999999999999e-05 2024-09-11T09:52:13.131 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.131 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.131 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:13.132 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.958638+0000", 2024-09-11T09:52:13.132 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.7810000000000003e-05 2024-09-11T09:52:13.132 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.132 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.132 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:13.132 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.958758+0000", 2024-09-11T09:52:13.132 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000120689 2024-09-11T09:52:13.132 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.132 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.132 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 3", 2024-09-11T09:52:13.132 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.958865+0000", 2024-09-11T09:52:13.132 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00010692 2024-09-11T09:52:13.132 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.133 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.133 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:13.133 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.970712+0000", 2024-09-11T09:52:13.133 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.011846327 2024-09-11T09:52:13.133 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.133 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.133 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:13.133 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.972603+0000", 2024-09-11T09:52:13.133 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0018918610000000001 2024-09-11T09:52:13.133 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.133 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.133 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:13.133 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.972653+0000", 2024-09-11T09:52:13.134 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.9762999999999998e-05 2024-09-11T09:52:13.134 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.134 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.134 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:13.134 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.972699+0000", 2024-09-11T09:52:13.134 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.5806999999999997e-05 2024-09-11T09:52:13.134 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:13.134 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:13.135 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:13.135 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.135 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.135 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:52 2.4 2:26d5c2f0:::rbd_data.1250f5e25b80.000000000000000b:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:13.135 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.958596+0000", 2024-09-11T09:52:13.135 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 50.130585662000001, 2024-09-11T09:52:13.135 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.016477849999999999, 2024-09-11T09:52:13.135 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:13.135 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:13.135 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:13.135 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:13.135 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:52:13.135 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:52:13.135 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 52 2024-09-11T09:52:13.136 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.136 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:13.136 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.136 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:13.136 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.958596+0000", 2024-09-11T09:52:13.136 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:13.136 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.136 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.136 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:13.137 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.958596+0000", 2024-09-11T09:52:13.137 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:13.137 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.137 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.137 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:13.137 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.958597+0000", 2024-09-11T09:52:13.137 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1960000000000001e-06 2024-09-11T09:52:13.137 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.137 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.137 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:13.137 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.963157+0000", 2024-09-11T09:52:13.138 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0045603029999999999 2024-09-11T09:52:13.138 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.138 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.138 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:13.138 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.963171+0000", 2024-09-11T09:52:13.138 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3777999999999999e-05 2024-09-11T09:52:13.138 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.138 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.138 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:13.138 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.963181+0000", 2024-09-11T09:52:13.138 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.9010000000000006e-06 2024-09-11T09:52:13.138 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.139 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.139 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:13.139 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.963246+0000", 2024-09-11T09:52:13.139 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.5351000000000005e-05 2024-09-11T09:52:13.139 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.139 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.139 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:13.139 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.963419+0000", 2024-09-11T09:52:13.139 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00017299700000000001 2024-09-11T09:52:13.139 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.139 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.139 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 0", 2024-09-11T09:52:13.139 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.963570+0000", 2024-09-11T09:52:13.140 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00015066999999999999 2024-09-11T09:52:13.140 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.140 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.140 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:13.140 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.973881+0000", 2024-09-11T09:52:13.140 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.010311268 2024-09-11T09:52:13.140 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.140 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.140 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:13.141 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.974963+0000", 2024-09-11T09:52:13.141 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.001081493 2024-09-11T09:52:13.141 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.141 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.141 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:13.141 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.975012+0000", 2024-09-11T09:52:13.141 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.9471000000000001e-05 2024-09-11T09:52:13.141 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.141 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.141 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:13.141 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.975074+0000", 2024-09-11T09:52:13.141 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.1421999999999996e-05 2024-09-11T09:52:13.142 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:13.142 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:13.142 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:13.142 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.142 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.142 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4688.0:47 2.1 e53/15 2:95f53b3e:::rbd_data.1250f5e25b80.0000000000000008:head v 53'3, mlcod=53'1)", 2024-09-11T09:52:13.142 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.960861+0000", 2024-09-11T09:52:13.142 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 50.128320244000001, 2024-09-11T09:52:13.142 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.012346107, 2024-09-11T09:52:13.142 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:13.142 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:13.142 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:13.142 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:13.142 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.143 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:13.143 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.960861+0000", 2024-09-11T09:52:13.143 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:13.143 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.143 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.143 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:13.143 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.960861+0000", 2024-09-11T09:52:13.143 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:13.143 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.143 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.143 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:13.143 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.960863+0000", 2024-09-11T09:52:13.143 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7150000000000001e-06 2024-09-11T09:52:13.144 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.144 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.144 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:13.144 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.962428+0000", 2024-09-11T09:52:13.144 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.001564668 2024-09-11T09:52:13.144 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.144 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.144 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:13.144 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.962446+0000", 2024-09-11T09:52:13.144 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.8476999999999999e-05 2024-09-11T09:52:13.144 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.144 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.145 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:13.145 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.962458+0000", 2024-09-11T09:52:13.145 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1263999999999999e-05 2024-09-11T09:52:13.145 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.145 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.145 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:13.145 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.962575+0000", 2024-09-11T09:52:13.145 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000117025 2024-09-11T09:52:13.145 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.145 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.145 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:13.145 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.962601+0000", 2024-09-11T09:52:13.145 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.5984999999999999e-05 2024-09-11T09:52:13.145 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.146 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.146 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:13.146 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.973065+0000", 2024-09-11T09:52:13.146 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.010464753 2024-09-11T09:52:13.146 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.146 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.146 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:13.146 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.973207+0000", 2024-09-11T09:52:13.146 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00014222 2024-09-11T09:52:13.146 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:13.146 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:13.146 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:13.146 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.147 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.147 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:54 2.4 2.da8979c4 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:13.147 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.963207+0000", 2024-09-11T09:52:13.147 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 50.125974806000002, 2024-09-11T09:52:13.147 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.010641575, 2024-09-11T09:52:13.147 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:13.147 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:13.147 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "reached pg", 2024-09-11T09:52:13.147 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:13.147 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:52:13.147 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:52:13.147 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 54 2024-09-11T09:52:13.147 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.148 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:13.148 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.148 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:13.148 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.963207+0000", 2024-09-11T09:52:13.148 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:13.148 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.148 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.148 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:13.148 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.963207+0000", 2024-09-11T09:52:13.148 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:13.148 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.148 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.148 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:13.149 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.963208+0000", 2024-09-11T09:52:13.149 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2979999999999999e-06 2024-09-11T09:52:13.149 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.149 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.149 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:13.149 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.967419+0000", 2024-09-11T09:52:13.149 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0042111750000000002 2024-09-11T09:52:13.149 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.149 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.149 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:13.149 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.967439+0000", 2024-09-11T09:52:13.149 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.0129999999999999e-05 2024-09-11T09:52:13.149 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.150 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.150 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:13.150 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.967453+0000", 2024-09-11T09:52:13.150 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3781000000000001e-05 2024-09-11T09:52:13.150 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.150 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.150 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:13.150 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.973840+0000", 2024-09-11T09:52:13.150 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0063872269999999997 2024-09-11T09:52:13.150 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.150 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.150 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:13.151 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.973848+0000", 2024-09-11T09:52:13.151 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.9640000000000003e-06 2024-09-11T09:52:13.151 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:13.151 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:13.151 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:13.151 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.151 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.151 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:54 2.4 2:239e915b:::rbd_data.1250f5e25b80.000000000000000c:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:13.151 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.982401+0000", 2024-09-11T09:52:13.151 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 50.106780878999999, 2024-09-11T09:52:13.151 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.016817347, 2024-09-11T09:52:13.151 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:13.151 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:13.151 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:13.152 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:13.152 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:52:13.152 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:52:13.152 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 54 2024-09-11T09:52:13.152 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.152 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:13.152 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.152 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:13.152 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.982401+0000", 2024-09-11T09:52:13.152 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:13.152 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.152 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.152 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:13.153 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.982401+0000", 2024-09-11T09:52:13.153 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:13.153 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.153 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.153 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:13.153 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.982402+0000", 2024-09-11T09:52:13.153 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.8999999999999995e-07 2024-09-11T09:52:13.153 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.153 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.153 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:13.153 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.986535+0000", 2024-09-11T09:52:13.154 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0041333689999999996 2024-09-11T09:52:13.154 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.154 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.154 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:13.154 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.986555+0000", 2024-09-11T09:52:13.154 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.9851e-05 2024-09-11T09:52:13.154 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.154 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.154 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:13.154 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.986569+0000", 2024-09-11T09:52:13.154 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3784000000000001e-05 2024-09-11T09:52:13.155 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.155 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.155 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:13.155 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.986683+0000", 2024-09-11T09:52:13.155 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000114474 2024-09-11T09:52:13.155 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.155 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.155 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:13.155 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.986815+0000", 2024-09-11T09:52:13.155 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00013217799999999999 2024-09-11T09:52:13.155 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.155 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.155 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 0", 2024-09-11T09:52:13.156 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.986918+0000", 2024-09-11T09:52:13.156 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00010235599999999999 2024-09-11T09:52:13.156 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.156 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.156 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:13.156 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.998983+0000", 2024-09-11T09:52:13.156 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.012065352 2024-09-11T09:52:13.156 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.156 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.156 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:13.156 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.999081+0000", 2024-09-11T09:52:13.156 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.8091000000000007e-05 2024-09-11T09:52:13.156 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.157 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.157 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:13.157 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.999159+0000", 2024-09-11T09:52:13.157 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.7823000000000004e-05 2024-09-11T09:52:13.157 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.157 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.157 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:13.157 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.999218+0000", 2024-09-11T09:52:13.157 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.9178999999999997e-05 2024-09-11T09:52:13.157 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:13.157 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:13.158 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:13.158 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.158 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.158 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:56 2.4 2:2167012f:::rbd_data.1250f5e25b80.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:13.158 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.986661+0000", 2024-09-11T09:52:13.158 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 50.102520425999998, 2024-09-11T09:52:13.158 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.017353539000000001, 2024-09-11T09:52:13.158 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:13.158 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:13.158 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:13.158 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:13.159 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:52:13.159 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:52:13.159 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 56 2024-09-11T09:52:13.159 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.159 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:13.159 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.159 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:13.159 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.986661+0000", 2024-09-11T09:52:13.159 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:13.159 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.159 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.159 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:13.159 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.986661+0000", 2024-09-11T09:52:13.159 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:13.160 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.160 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.160 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:13.160 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.986663+0000", 2024-09-11T09:52:13.160 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.8989999999999999e-06 2024-09-11T09:52:13.160 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.160 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.160 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:13.160 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.990444+0000", 2024-09-11T09:52:13.160 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0037808569999999999 2024-09-11T09:52:13.160 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.160 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.160 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:13.161 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.990462+0000", 2024-09-11T09:52:13.161 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.8450000000000001e-05 2024-09-11T09:52:13.161 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.161 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.161 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:13.161 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.990476+0000", 2024-09-11T09:52:13.161 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3565e-05 2024-09-11T09:52:13.161 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.161 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.161 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:13.161 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.993582+0000", 2024-09-11T09:52:13.161 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0031059109999999998 2024-09-11T09:52:13.162 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.162 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.162 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:13.162 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.993686+0000", 2024-09-11T09:52:13.162 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000104424 2024-09-11T09:52:13.162 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.162 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.162 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 0", 2024-09-11T09:52:13.162 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.993799+0000", 2024-09-11T09:52:13.162 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000112985 2024-09-11T09:52:13.162 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.162 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.163 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:13.163 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.002449+0000", 2024-09-11T09:52:13.163 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.008650151 2024-09-11T09:52:13.163 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.163 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.163 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:13.163 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.003948+0000", 2024-09-11T09:52:13.163 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.001498636 2024-09-11T09:52:13.163 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.163 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.163 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:13.163 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.003985+0000", 2024-09-11T09:52:13.163 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.7054999999999997e-05 2024-09-11T09:52:13.164 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.164 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.164 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:13.164 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.004015+0000", 2024-09-11T09:52:13.164 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.9606000000000001e-05 2024-09-11T09:52:13.164 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:13.164 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:13.164 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:13.164 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.164 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.164 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:58 2.6 2:6277c644:::rbd_data.1250f5e25b80.000000000000000e:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:13.164 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.990540+0000", 2024-09-11T09:52:13.164 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 50.098641631, 2024-09-11T09:52:13.165 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.016232068999999998, 2024-09-11T09:52:13.165 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:13.165 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:13.165 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:13.165 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:13.165 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:52:13.165 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:52:13.165 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 58 2024-09-11T09:52:13.165 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.165 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:13.166 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.166 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:13.166 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.990540+0000", 2024-09-11T09:52:13.166 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:13.166 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.166 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.166 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:13.166 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.990540+0000", 2024-09-11T09:52:13.166 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:13.166 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.166 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.166 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:13.167 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.990541+0000", 2024-09-11T09:52:13.167 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2359999999999999e-06 2024-09-11T09:52:13.167 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.167 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.167 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:13.167 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.994626+0000", 2024-09-11T09:52:13.167 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.004084865 2024-09-11T09:52:13.167 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.167 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.167 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:13.167 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.994644+0000", 2024-09-11T09:52:13.167 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7866000000000001e-05 2024-09-11T09:52:13.167 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.167 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.168 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:13.168 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.994653+0000", 2024-09-11T09:52:13.168 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.2159999999999995e-06 2024-09-11T09:52:13.168 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.168 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.168 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:13.168 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.994732+0000", 2024-09-11T09:52:13.168 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.8954000000000003e-05 2024-09-11T09:52:13.168 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.168 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.168 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:13.168 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.994906+0000", 2024-09-11T09:52:13.169 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0001741 2024-09-11T09:52:13.169 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.169 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.169 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 3", 2024-09-11T09:52:13.169 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.995083+0000", 2024-09-11T09:52:13.169 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00017637100000000001 2024-09-11T09:52:13.169 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.170 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.170 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:13.170 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.003066+0000", 2024-09-11T09:52:13.170 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0079833360000000006 2024-09-11T09:52:13.170 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.170 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.170 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:13.170 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.006634+0000", 2024-09-11T09:52:13.170 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.003568309 2024-09-11T09:52:13.170 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.170 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.170 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:13.170 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.006716+0000", 2024-09-11T09:52:13.170 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.1377000000000004e-05 2024-09-11T09:52:13.171 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.171 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.171 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:13.171 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.006772+0000", 2024-09-11T09:52:13.171 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.6439e-05 2024-09-11T09:52:13.171 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:13.171 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:13.171 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:13.171 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.172 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.172 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:59 2.4 2:2906e6a7:::rbd_header.1250f5e25b80:head [watch reconnect cookie 140424358411408 gen 2] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:13.172 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.994684+0000", 2024-09-11T09:52:13.172 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 50.094498000000002, 2024-09-11T09:52:13.172 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0094134349999999995, 2024-09-11T09:52:13.172 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:13.172 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:13.172 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "started", 2024-09-11T09:52:13.172 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:13.172 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:52:13.172 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:52:13.172 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 59 2024-09-11T09:52:13.173 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.173 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:13.173 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.173 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:13.173 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.994684+0000", 2024-09-11T09:52:13.173 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:13.173 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.173 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.173 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:13.173 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.994684+0000", 2024-09-11T09:52:13.173 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:13.173 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.173 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.173 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:13.174 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.994686+0000", 2024-09-11T09:52:13.174 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.8159999999999998e-06 2024-09-11T09:52:13.174 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.174 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.174 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:13.174 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.994691+0000", 2024-09-11T09:52:13.174 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.5009999999999998e-06 2024-09-11T09:52:13.174 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.174 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.174 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:13.174 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.994698+0000", 2024-09-11T09:52:13.174 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.3039999999999999e-06 2024-09-11T09:52:13.175 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.175 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.175 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:13.175 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.994703+0000", 2024-09-11T09:52:13.175 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.5190000000000003e-06 2024-09-11T09:52:13.176 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.176 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.176 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:13.176 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.998678+0000", 2024-09-11T09:52:13.176 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0039752340000000002 2024-09-11T09:52:13.176 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.177 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.177 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:13.177 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.998748+0000", 2024-09-11T09:52:13.177 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.0114e-05 2024-09-11T09:52:13.177 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.177 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.178 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:13.178 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.004097+0000", 2024-09-11T09:52:13.178 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0053489469999999997 2024-09-11T09:52:13.178 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:13.178 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:13.178 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:13.178 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.179 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.179 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4694.0:33 2.4 2:3bfc818c:ns::rbd_data.1256a0c02879.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:13.179 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.084537+0000", 2024-09-11T09:52:13.179 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 50.004644571999997, 2024-09-11T09:52:13.179 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.022075773999999999, 2024-09-11T09:52:13.179 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:13.179 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:13.180 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:13.180 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:13.180 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4694", 2024-09-11T09:52:13.180 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:52:13.180 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 33 2024-09-11T09:52:13.181 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.181 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:13.181 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.181 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:13.181 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.084537+0000", 2024-09-11T09:52:13.181 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:13.181 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.182 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.182 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:13.182 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.084537+0000", 2024-09-11T09:52:13.182 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:13.182 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.182 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.183 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:13.183 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.084538+0000", 2024-09-11T09:52:13.183 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.5000000000000001e-07 2024-09-11T09:52:13.183 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.183 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.183 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:13.183 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.095984+0000", 2024-09-11T09:52:13.183 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.011446543 2024-09-11T09:52:13.183 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.183 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.183 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:13.184 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.096014+0000", 2024-09-11T09:52:13.184 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.9745000000000001e-05 2024-09-11T09:52:13.184 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.184 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.184 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:13.184 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.096030+0000", 2024-09-11T09:52:13.184 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6152999999999999e-05 2024-09-11T09:52:13.184 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.184 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.184 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:13.184 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.096113+0000", 2024-09-11T09:52:13.184 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.2731999999999993e-05 2024-09-11T09:52:13.185 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.185 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.185 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:13.185 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.096273+0000", 2024-09-11T09:52:13.185 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00015973699999999999 2024-09-11T09:52:13.185 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.185 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.185 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 0", 2024-09-11T09:52:13.185 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.096394+0000", 2024-09-11T09:52:13.185 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000120883 2024-09-11T09:52:13.185 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.185 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.185 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:13.186 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.106048+0000", 2024-09-11T09:52:13.186 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0096548020000000005 2024-09-11T09:52:13.186 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.186 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.186 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:13.186 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.106531+0000", 2024-09-11T09:52:13.186 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00048264299999999999 2024-09-11T09:52:13.186 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.186 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.186 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:13.186 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.106565+0000", 2024-09-11T09:52:13.186 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.4343e-05 2024-09-11T09:52:13.186 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.187 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.187 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:13.187 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.106613+0000", 2024-09-11T09:52:13.187 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.7342999999999997e-05 2024-09-11T09:52:13.187 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:13.187 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:13.187 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:13.187 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.187 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.187 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4694.0:32 2.1 e53/15 2:8a862b48:ns::rbd_data.1256a0c02879.0000000000000000:head v 53'8, mlcod=53'6)", 2024-09-11T09:52:13.187 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.091418+0000", 2024-09-11T09:52:13.187 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 49.997763403, 2024-09-11T09:52:13.187 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.013027372000000001, 2024-09-11T09:52:13.188 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:13.188 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:13.188 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:13.188 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:13.188 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.188 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:13.188 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.091418+0000", 2024-09-11T09:52:13.188 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:13.188 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.188 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.188 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:13.188 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.091418+0000", 2024-09-11T09:52:13.189 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:13.189 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.189 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.189 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:13.189 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.091420+0000", 2024-09-11T09:52:13.189 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6899999999999999e-06 2024-09-11T09:52:13.189 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.189 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.189 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:13.189 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.093086+0000", 2024-09-11T09:52:13.189 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0016662510000000001 2024-09-11T09:52:13.189 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.189 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.190 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:13.190 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.093107+0000", 2024-09-11T09:52:13.190 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.0999e-05 2024-09-11T09:52:13.190 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.190 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.190 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:13.190 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.093120+0000", 2024-09-11T09:52:13.190 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2544999999999999e-05 2024-09-11T09:52:13.190 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.190 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.190 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:13.190 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.093186+0000", 2024-09-11T09:52:13.191 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.6595999999999997e-05 2024-09-11T09:52:13.191 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.191 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.191 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:13.191 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.093208+0000", 2024-09-11T09:52:13.191 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.1624000000000002e-05 2024-09-11T09:52:13.191 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.191 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.191 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:13.191 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.104392+0000", 2024-09-11T09:52:13.191 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.011184265000000001 2024-09-11T09:52:13.192 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.192 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.192 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:13.192 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.104446+0000", 2024-09-11T09:52:13.192 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.3402000000000001e-05 2024-09-11T09:52:13.192 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:13.192 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:13.192 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:13.192 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.192 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.192 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4694.0:45 2.6 2:671fb830:ns::rbd_data.1256a0c02879.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:13.193 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.109629+0000", 2024-09-11T09:52:13.193 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 49.979552562000002, 2024-09-11T09:52:13.193 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.036403143999999998, 2024-09-11T09:52:13.193 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:13.193 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:13.193 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:13.193 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:13.193 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4694", 2024-09-11T09:52:13.193 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:52:13.193 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 45 2024-09-11T09:52:13.193 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.193 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:13.193 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.194 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:13.194 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.109629+0000", 2024-09-11T09:52:13.194 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:13.194 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.194 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.194 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:13.194 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.109629+0000", 2024-09-11T09:52:13.194 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:13.194 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.194 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.194 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:13.194 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.109630+0000", 2024-09-11T09:52:13.195 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1200000000000001e-06 2024-09-11T09:52:13.195 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.195 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.195 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:13.195 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.121370+0000", 2024-09-11T09:52:13.195 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.011739571000000001 2024-09-11T09:52:13.195 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.195 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.195 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:13.195 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.121391+0000", 2024-09-11T09:52:13.195 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.1175e-05 2024-09-11T09:52:13.195 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.196 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.196 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:13.196 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.121404+0000", 2024-09-11T09:52:13.196 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2807e-05 2024-09-11T09:52:13.196 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.196 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.196 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:13.196 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.121498+0000", 2024-09-11T09:52:13.196 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.3770999999999994e-05 2024-09-11T09:52:13.196 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.196 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.196 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:13.197 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.121612+0000", 2024-09-11T09:52:13.197 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000114666 2024-09-11T09:52:13.197 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.197 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.197 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 3", 2024-09-11T09:52:13.197 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.121711+0000", 2024-09-11T09:52:13.197 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.8647999999999995e-05 2024-09-11T09:52:13.197 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.197 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.197 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:13.197 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.126558+0000", 2024-09-11T09:52:13.198 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0048473769999999999 2024-09-11T09:52:13.198 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.198 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.198 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:13.198 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.145912+0000", 2024-09-11T09:52:13.198 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.019354281000000001 2024-09-11T09:52:13.198 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.198 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.198 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:13.198 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.145977+0000", 2024-09-11T09:52:13.198 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.4135999999999997e-05 2024-09-11T09:52:13.199 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.199 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.199 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:13.199 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.146032+0000", 2024-09-11T09:52:13.199 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.5591999999999999e-05 2024-09-11T09:52:13.199 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:13.199 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:13.199 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:13.199 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.199 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.199 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4694.0:41 2.2 e53/15 2:5e2505c4:ns::rbd_data.1256a0c02879.0000000000000003:head v 53'4, mlcod=18'2)", 2024-09-11T09:52:13.199 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.113199+0000", 2024-09-11T09:52:13.199 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 49.975982152, 2024-09-11T09:52:13.200 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.011404512, 2024-09-11T09:52:13.200 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:13.200 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:13.200 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:13.200 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:13.200 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.200 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:13.200 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.113199+0000", 2024-09-11T09:52:13.200 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:13.200 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.200 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.200 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:13.200 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.113199+0000", 2024-09-11T09:52:13.201 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:13.201 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.201 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.201 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:13.201 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.113200+0000", 2024-09-11T09:52:13.201 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.2799999999999995e-07 2024-09-11T09:52:13.201 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.201 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.201 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:13.201 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.118872+0000", 2024-09-11T09:52:13.201 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0056718089999999999 2024-09-11T09:52:13.201 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.201 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.202 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:13.202 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.118889+0000", 2024-09-11T09:52:13.202 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7428999999999999e-05 2024-09-11T09:52:13.202 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.202 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.202 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:13.202 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.118900+0000", 2024-09-11T09:52:13.202 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0243e-05 2024-09-11T09:52:13.202 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.202 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.202 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:13.202 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.118959+0000", 2024-09-11T09:52:13.203 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.9364000000000003e-05 2024-09-11T09:52:13.203 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.203 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.203 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:13.203 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.118976+0000", 2024-09-11T09:52:13.203 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7124000000000001e-05 2024-09-11T09:52:13.203 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.203 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.203 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:13.203 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.124557+0000", 2024-09-11T09:52:13.203 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0055812659999999997 2024-09-11T09:52:13.203 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.203 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.204 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:13.204 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.124604+0000", 2024-09-11T09:52:13.204 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.6548999999999998e-05 2024-09-11T09:52:13.204 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:13.204 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:13.204 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:13.204 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.204 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.204 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4694.0:43 2.0 e53/15 2:16c8ae84:ns::rbd_data.1256a0c02879.0000000000000005:head v 53'2, mlcod=0'0)", 2024-09-11T09:52:13.204 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.158559+0000", 2024-09-11T09:52:13.204 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 49.930622825999997, 2024-09-11T09:52:13.204 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.016713677999999999, 2024-09-11T09:52:13.205 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:13.205 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:13.205 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:13.205 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:13.205 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.205 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:13.205 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.158559+0000", 2024-09-11T09:52:13.205 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:13.205 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.205 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.205 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:13.205 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.158559+0000", 2024-09-11T09:52:13.206 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:13.206 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.206 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.206 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:13.206 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.158560+0000", 2024-09-11T09:52:13.206 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1459999999999999e-06 2024-09-11T09:52:13.206 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.206 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.206 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:13.206 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.166465+0000", 2024-09-11T09:52:13.206 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.007904767 2024-09-11T09:52:13.206 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.206 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.207 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:13.207 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.166491+0000", 2024-09-11T09:52:13.207 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.6308e-05 2024-09-11T09:52:13.207 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.207 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.207 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:13.207 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.166502+0000", 2024-09-11T09:52:13.207 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0736e-05 2024-09-11T09:52:13.207 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.207 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.207 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:13.208 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.166564+0000", 2024-09-11T09:52:13.208 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.2578999999999998e-05 2024-09-11T09:52:13.208 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.208 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.208 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:13.208 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.166585+0000", 2024-09-11T09:52:13.208 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.0395999999999999e-05 2024-09-11T09:52:13.208 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.208 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.208 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:13.209 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.175227+0000", 2024-09-11T09:52:13.209 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0086426420000000007 2024-09-11T09:52:13.209 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.209 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.209 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:13.209 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.175272+0000", 2024-09-11T09:52:13.209 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.5104e-05 2024-09-11T09:52:13.209 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:13.209 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:13.209 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:13.209 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.209 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.209 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4694.0:59 2.2 e53/15 2:41adefc9:ns::rbd_data.1256a0c02879.000000000000000e:head v 53'5, mlcod=53'3)", 2024-09-11T09:52:13.210 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.163389+0000", 2024-09-11T09:52:13.210 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 49.925792850000001, 2024-09-11T09:52:13.210 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.013095719, 2024-09-11T09:52:13.210 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:13.210 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:13.210 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:13.210 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:13.210 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.210 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:13.210 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.163389+0000", 2024-09-11T09:52:13.210 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:13.210 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.210 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.210 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:13.211 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.163389+0000", 2024-09-11T09:52:13.211 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:13.211 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.211 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.211 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:13.211 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.163390+0000", 2024-09-11T09:52:13.211 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.336e-06 2024-09-11T09:52:13.211 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.211 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.211 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:13.211 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.170335+0000", 2024-09-11T09:52:13.211 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0069451080000000002 2024-09-11T09:52:13.211 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.212 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.212 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:13.212 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.170354+0000", 2024-09-11T09:52:13.212 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.8423e-05 2024-09-11T09:52:13.212 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.212 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.212 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:13.212 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.170366+0000", 2024-09-11T09:52:13.212 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2235999999999999e-05 2024-09-11T09:52:13.213 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.213 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.213 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:13.213 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.170439+0000", 2024-09-11T09:52:13.213 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.2989000000000002e-05 2024-09-11T09:52:13.213 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.213 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.213 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:13.213 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.170460+0000", 2024-09-11T09:52:13.213 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.1211000000000001e-05 2024-09-11T09:52:13.214 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.214 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.214 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:13.214 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.176442+0000", 2024-09-11T09:52:13.214 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0059821969999999999 2024-09-11T09:52:13.214 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.214 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.214 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:13.214 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.176484+0000", 2024-09-11T09:52:13.214 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.2218999999999997e-05 2024-09-11T09:52:13.214 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:13.215 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:13.215 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:13.215 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:13.215 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:13.215 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:52:13.216 INFO:tasks.workunit.client.0.smithi107.stdout:/dev/nbd1 2024-09-11T09:52:13.217 INFO:tasks.workunit.client.0.smithi107.stderr:+ get_pid rbd 2024-09-11T09:52:13.217 INFO:tasks.workunit.client.0.smithi107.stderr:+ local pool=rbd 2024-09-11T09:52:13.217 INFO:tasks.workunit.client.0.smithi107.stderr:+ local ns= 2024-09-11T09:52:13.217 INFO:tasks.workunit.client.0.smithi107.stderr:++ rbd device --device-type nbd --format xml list 2024-09-11T09:52:13.217 INFO:tasks.workunit.client.0.smithi107.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd27121'\''][device='\''/dev/nbd1'\'']/id' 2024-09-11T09:52:13.218 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-11T09:52:13.218 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:52:13.217+0000 7f3a7b90a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 42475) UID: 0 2024-09-11T09:52:13.241 INFO:tasks.workunit.client.0.smithi107.stderr:+ PID=29778 2024-09-11T09:52:13.241 INFO:tasks.workunit.client.0.smithi107.stderr:+ test -n 29778 2024-09-11T09:52:13.241 INFO:tasks.workunit.client.0.smithi107.stderr:+ ps -p 29778 -C rbd-nbd 2024-09-11T09:52:13.251 INFO:tasks.workunit.client.0.smithi107.stdout: PID TTY TIME CMD 2024-09-11T09:52:13.252 INFO:tasks.workunit.client.0.smithi107.stdout: 29778 ? 00:00:00 rbd-nbd 2024-09-11T09:52:13.252 INFO:tasks.workunit.client.0.smithi107.stderr:+ _sudo rbd device detach /dev/nbd1 --device-type nbd 2024-09-11T09:52:13.252 INFO:tasks.workunit.client.0.smithi107.stderr:+ local cmd 2024-09-11T09:52:13.252 INFO:tasks.workunit.client.0.smithi107.stderr:++ id -u 2024-09-11T09:52:13.253 INFO:tasks.workunit.client.0.smithi107.stderr:+ '[' 1000 -eq 0 ']' 2024-09-11T09:52:13.253 INFO:tasks.workunit.client.0.smithi107.stderr:++ which rbd 2024-09-11T09:52:13.255 INFO:tasks.workunit.client.0.smithi107.stderr:+ cmd=/usr/bin/rbd 2024-09-11T09:52:13.255 INFO:tasks.workunit.client.0.smithi107.stderr:+ shift 2024-09-11T09:52:13.255 INFO:tasks.workunit.client.0.smithi107.stderr:+ sudo -nE /usr/bin/rbd device detach /dev/nbd1 --device-type nbd 2024-09-11T09:52:13.316 INFO:tasks.ceph.osd.2.smithi017.stderr:2024-09-11T09:52:13.317+0000 7fd20b170640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31672) UID: 0 2024-09-11T09:52:13.325 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:52:13.325 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [], 2024-09-11T09:52:13.325 INFO:teuthology.orchestra.run.smithi017.stdout: "num_ops": 0 2024-09-11T09:52:13.325 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:52:13.334 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-11T09:52:13.345 INFO:tasks.workunit.client.0.smithi107.stderr:+ expect_false get_pid rbd 2024-09-11T09:52:13.346 INFO:tasks.workunit.client.0.smithi107.stderr:+ get_pid rbd 2024-09-11T09:52:13.346 INFO:tasks.workunit.client.0.smithi107.stderr:+ local pool=rbd 2024-09-11T09:52:13.346 INFO:tasks.workunit.client.0.smithi107.stderr:+ local ns= 2024-09-11T09:52:13.348 INFO:tasks.workunit.client.0.smithi107.stderr:++ rbd device --device-type nbd --format xml list 2024-09-11T09:52:13.348 INFO:tasks.workunit.client.0.smithi107.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd27121'\''][device='\''/dev/nbd1'\'']/id' 2024-09-11T09:52:13.413 INFO:tasks.workunit.client.0.smithi107.stderr:+ PID= 2024-09-11T09:52:13.413 INFO:tasks.workunit.client.0.smithi107.stderr:+ test -n '' 2024-09-11T09:52:13.414 INFO:tasks.workunit.client.0.smithi107.stderr:+ return 1 2024-09-11T09:52:13.414 INFO:tasks.workunit.client.0.smithi107.stderr:+ return 0 2024-09-11T09:52:13.414 INFO:tasks.workunit.client.0.smithi107.stderr:+ '[' -n eaab7410-3050-4b82-9298-7fac480d4de7 ']' 2024-09-11T09:52:13.414 INFO:tasks.workunit.client.0.smithi107.stderr:+ _sudo rbd device attach --device /dev/nbd1 --cookie eaab7410-3050-4b82-9298-7fac480d4de7 rbd/testrbdnbd27121 --device-type nbd 2024-09-11T09:52:13.414 INFO:tasks.workunit.client.0.smithi107.stderr:+ local cmd 2024-09-11T09:52:13.414 INFO:tasks.workunit.client.0.smithi107.stderr:++ id -u 2024-09-11T09:52:13.415 INFO:tasks.workunit.client.0.smithi107.stderr:+ '[' 1000 -eq 0 ']' 2024-09-11T09:52:13.415 INFO:tasks.workunit.client.0.smithi107.stderr:++ which rbd 2024-09-11T09:52:13.416 INFO:tasks.workunit.client.0.smithi107.stderr:+ cmd=/usr/bin/rbd 2024-09-11T09:52:13.416 INFO:tasks.workunit.client.0.smithi107.stderr:+ shift 2024-09-11T09:52:13.416 INFO:tasks.workunit.client.0.smithi107.stderr:+ sudo -nE /usr/bin/rbd device attach --device /dev/nbd1 --cookie eaab7410-3050-4b82-9298-7fac480d4de7 rbd/testrbdnbd27121 --device-type nbd 2024-09-11T09:52:13.419 INFO:tasks.ceph.osd.0.smithi017.stderr:2024-09-11T09:52:13.420+0000 7fb73c0f1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31665) UID: 0 2024-09-11T09:52:13.482 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:52:13.482 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [], 2024-09-11T09:52:13.482 INFO:teuthology.orchestra.run.smithi017.stdout: "complaint_time": 30, 2024-09-11T09:52:13.482 INFO:teuthology.orchestra.run.smithi017.stdout: "num_blocked_ops": 0 2024-09-11T09:52:13.482 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:52:13.490 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-11T09:52:13.531 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:52:13.532+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:52:13.533 INFO:tasks.workunit.client.0.smithi107.stdout:/dev/nbd1 2024-09-11T09:52:13.539 INFO:tasks.workunit.client.0.smithi107.stderr:+ get_pid rbd 2024-09-11T09:52:13.539 INFO:tasks.workunit.client.0.smithi107.stderr:+ local pool=rbd 2024-09-11T09:52:13.539 INFO:tasks.workunit.client.0.smithi107.stderr:+ local ns= 2024-09-11T09:52:13.539 INFO:tasks.workunit.client.0.smithi107.stderr:++ rbd device --device-type nbd --format xml list 2024-09-11T09:52:13.539 INFO:tasks.workunit.client.0.smithi107.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd27121'\''][device='\''/dev/nbd1'\'']/id' 2024-09-11T09:52:13.618 INFO:tasks.workunit.client.0.smithi107.stderr:+ PID=29840 2024-09-11T09:52:13.618 INFO:tasks.workunit.client.0.smithi107.stderr:+ test -n 29840 2024-09-11T09:52:13.618 INFO:tasks.workunit.client.0.smithi107.stderr:+ ps -p 29840 -C rbd-nbd 2024-09-11T09:52:13.618 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:52:13.619+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:52:13.629 INFO:tasks.workunit.client.0.smithi107.stdout: PID TTY TIME CMD 2024-09-11T09:52:13.629 INFO:tasks.workunit.client.0.smithi107.stdout: 29840 ? 00:00:00 rbd-nbd 2024-09-11T09:52:13.629 INFO:tasks.workunit.client.0.smithi107.stderr:+ ls /tmp/tmp.rvrcDirNue/mnt/ 2024-09-11T09:52:13.631 INFO:tasks.workunit.client.0.smithi107.stdout:lost+found 2024-09-11T09:52:13.631 INFO:tasks.workunit.client.0.smithi107.stdout:test 2024-09-11T09:52:13.631 INFO:tasks.workunit.client.0.smithi107.stderr:+ dd if=/tmp/tmp.rvrcDirNue/mnt/test of=/dev/null bs=1M count=1 2024-09-11T09:52:13.638 INFO:tasks.workunit.client.0.smithi107.stderr:1+0 records in 2024-09-11T09:52:13.638 INFO:tasks.workunit.client.0.smithi107.stderr:1+0 records out 2024-09-11T09:52:13.638 INFO:tasks.workunit.client.0.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00573082 s, 183 MB/s 2024-09-11T09:52:13.638 INFO:tasks.workunit.client.0.smithi107.stderr:+ _sudo dd if=/tmp/tmp.rvrcDirNue/data of=/tmp/tmp.rvrcDirNue/mnt/test1 bs=1M count=1 oflag=direct 2024-09-11T09:52:13.638 INFO:tasks.workunit.client.0.smithi107.stderr:+ local cmd 2024-09-11T09:52:13.638 INFO:tasks.workunit.client.0.smithi107.stderr:++ id -u 2024-09-11T09:52:13.639 INFO:tasks.workunit.client.0.smithi107.stderr:+ '[' 1000 -eq 0 ']' 2024-09-11T09:52:13.639 INFO:tasks.workunit.client.0.smithi107.stderr:++ which dd 2024-09-11T09:52:13.640 INFO:tasks.workunit.client.0.smithi107.stderr:+ cmd=/usr/bin/dd 2024-09-11T09:52:13.641 INFO:tasks.workunit.client.0.smithi107.stderr:+ shift 2024-09-11T09:52:13.641 INFO:tasks.workunit.client.0.smithi107.stderr:+ sudo -nE /usr/bin/dd if=/tmp/tmp.rvrcDirNue/data of=/tmp/tmp.rvrcDirNue/mnt/test1 bs=1M count=1 oflag=direct 2024-09-11T09:52:13.643 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:52:13.643 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 20, 2024-09-11T09:52:13.643 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 600, 2024-09-11T09:52:13.643 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [ 2024-09-11T09:52:13.643 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.644 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:31 2.1 2:8654721b:::rbd_data.1250f5e25b80.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:13.644 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.896024+0000", 2024-09-11T09:52:13.644 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 50.745451627000001, 2024-09-11T09:52:13.644 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.023148401999999998, 2024-09-11T09:52:13.644 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:13.644 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:13.645 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:13.645 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:13.645 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:52:13.645 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:52:13.645 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 31 2024-09-11T09:52:13.645 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.646 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:13.646 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.646 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:13.647 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.896024+0000", 2024-09-11T09:52:13.647 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:13.647 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.647 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.647 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:13.647 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.896024+0000", 2024-09-11T09:52:13.647 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:13.647 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.647 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.648 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:13.648 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.896025+0000", 2024-09-11T09:52:13.648 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.54e-07 2024-09-11T09:52:13.648 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.648 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.648 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:13.648 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.904287+0000", 2024-09-11T09:52:13.648 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0082624729999999993 2024-09-11T09:52:13.648 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.648 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.648 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:13.648 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.904305+0000", 2024-09-11T09:52:13.648 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7912999999999999e-05 2024-09-11T09:52:13.648 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.649 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.649 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:13.649 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.904312+0000", 2024-09-11T09:52:13.649 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.0230000000000004e-06 2024-09-11T09:52:13.649 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.649 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.649 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:13.649 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.904395+0000", 2024-09-11T09:52:13.649 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.2355999999999996e-05 2024-09-11T09:52:13.649 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.649 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.649 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:13.649 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.904499+0000", 2024-09-11T09:52:13.650 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000104672 2024-09-11T09:52:13.650 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.650 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.650 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1", 2024-09-11T09:52:13.650 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.904594+0000", 2024-09-11T09:52:13.650 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.5129000000000003e-05 2024-09-11T09:52:13.650 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.650 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.650 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:13.650 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.918183+0000", 2024-09-11T09:52:13.650 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.013588355999999999 2024-09-11T09:52:13.651 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.651 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.651 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:13.651 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919053+0000", 2024-09-11T09:52:13.651 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00087066800000000001 2024-09-11T09:52:13.651 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.651 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.651 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:13.651 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919125+0000", 2024-09-11T09:52:13.651 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.1333000000000003e-05 2024-09-11T09:52:13.651 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.651 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.651 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:13.652 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919172+0000", 2024-09-11T09:52:13.652 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.7725000000000001e-05 2024-09-11T09:52:13.652 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:13.652 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:13.652 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:13.652 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.652 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.652 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4688.0:42 2.3 e53/15 2:c54742d8:::rbd_data.1250f5e25b80.0000000000000006:head v 53'2, mlcod=0'0)", 2024-09-11T09:52:13.652 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.939272+0000", 2024-09-11T09:52:13.653 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 50.702203642999997, 2024-09-11T09:52:13.653 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.010821958, 2024-09-11T09:52:13.653 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:13.653 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:13.653 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:13.653 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:13.653 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.653 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:13.653 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.939272+0000", 2024-09-11T09:52:13.653 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:13.653 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.653 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.653 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:13.654 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.939272+0000", 2024-09-11T09:52:13.654 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:13.654 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.654 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.654 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:13.654 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.939273+0000", 2024-09-11T09:52:13.654 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.7899999999999997e-07 2024-09-11T09:52:13.654 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.654 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.654 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:13.654 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.944403+0000", 2024-09-11T09:52:13.654 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0051302129999999998 2024-09-11T09:52:13.654 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.654 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.655 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:13.655 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.944413+0000", 2024-09-11T09:52:13.655 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0182e-05 2024-09-11T09:52:13.655 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.655 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.655 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:13.655 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.944420+0000", 2024-09-11T09:52:13.655 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.7000000000000002e-06 2024-09-11T09:52:13.655 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.655 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.655 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:13.655 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.944482+0000", 2024-09-11T09:52:13.655 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.1911999999999999e-05 2024-09-11T09:52:13.656 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.656 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.656 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:13.656 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.944508+0000", 2024-09-11T09:52:13.656 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.6582000000000001e-05 2024-09-11T09:52:13.656 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.656 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.656 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:13.656 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.950050+0000", 2024-09-11T09:52:13.656 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0055414080000000003 2024-09-11T09:52:13.656 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.657 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.657 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:13.657 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.950094+0000", 2024-09-11T09:52:13.657 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.4082000000000002e-05 2024-09-11T09:52:13.657 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:13.657 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:13.657 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:13.657 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.657 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.658 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:47 2.1 2:95f53b3e:::rbd_data.1250f5e25b80.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:13.658 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.948330+0000", 2024-09-11T09:52:13.658 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 50.693145299000001, 2024-09-11T09:52:13.658 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.025156603999999999, 2024-09-11T09:52:13.658 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:13.658 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:13.659 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:13.659 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:13.659 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:52:13.659 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:52:13.659 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 47 2024-09-11T09:52:13.660 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.660 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:13.660 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.660 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:13.660 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.948330+0000", 2024-09-11T09:52:13.660 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:13.661 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.661 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.661 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:13.661 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.948330+0000", 2024-09-11T09:52:13.661 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:13.661 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.662 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.662 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:13.662 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.948333+0000", 2024-09-11T09:52:13.662 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.481e-06 2024-09-11T09:52:13.662 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.662 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.663 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:13.663 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.960263+0000", 2024-09-11T09:52:13.663 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.011930671 2024-09-11T09:52:13.663 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.663 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.663 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:13.664 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.960289+0000", 2024-09-11T09:52:13.664 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.5573000000000001e-05 2024-09-11T09:52:13.664 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.664 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.664 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:13.664 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.960304+0000", 2024-09-11T09:52:13.665 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.4647000000000001e-05 2024-09-11T09:52:13.665 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.665 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.665 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:13.665 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.960390+0000", 2024-09-11T09:52:13.665 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.6088999999999995e-05 2024-09-11T09:52:13.666 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.666 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.666 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:13.666 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.960562+0000", 2024-09-11T09:52:13.666 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00017237400000000001 2024-09-11T09:52:13.666 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.667 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.667 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1", 2024-09-11T09:52:13.667 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.960725+0000", 2024-09-11T09:52:13.667 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00016321599999999999 2024-09-11T09:52:13.667 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.667 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.668 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:13.668 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.969465+0000", 2024-09-11T09:52:13.668 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0087396899999999996 2024-09-11T09:52:13.668 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.668 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.668 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:13.668 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.973393+0000", 2024-09-11T09:52:13.669 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0039274669999999996 2024-09-11T09:52:13.669 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.669 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.669 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:13.669 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.973443+0000", 2024-09-11T09:52:13.669 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.0340000000000003e-05 2024-09-11T09:52:13.669 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.669 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.669 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:13.669 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.973487+0000", 2024-09-11T09:52:13.669 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.4056e-05 2024-09-11T09:52:13.670 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:13.670 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:13.670 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:13.670 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.670 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.670 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:61 2.1 2:9af038b1:::rbd_data.1250f5e25b80.000000000000000f:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:13.670 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.975764+0000", 2024-09-11T09:52:13.670 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 50.665711774999998, 2024-09-11T09:52:13.670 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.016158891000000002, 2024-09-11T09:52:13.670 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:13.670 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:13.670 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:13.670 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:13.671 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:52:13.671 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:52:13.671 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 61 2024-09-11T09:52:13.671 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.671 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:13.671 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.671 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:13.671 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.975764+0000", 2024-09-11T09:52:13.671 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:13.671 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.671 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.671 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:13.672 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.975764+0000", 2024-09-11T09:52:13.672 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:13.672 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.672 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.672 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:13.672 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.975766+0000", 2024-09-11T09:52:13.672 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.767e-06 2024-09-11T09:52:13.672 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.672 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.672 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:13.672 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.983204+0000", 2024-09-11T09:52:13.672 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0074388400000000004 2024-09-11T09:52:13.673 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.673 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.673 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:13.673 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.983226+0000", 2024-09-11T09:52:13.673 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.1279000000000001e-05 2024-09-11T09:52:13.673 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.673 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.673 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:13.673 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.983238+0000", 2024-09-11T09:52:13.673 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2563e-05 2024-09-11T09:52:13.673 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.674 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.674 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:13.674 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.983318+0000", 2024-09-11T09:52:13.674 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.0075999999999997e-05 2024-09-11T09:52:13.674 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.674 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.674 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:13.674 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.983429+0000", 2024-09-11T09:52:13.674 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000110618 2024-09-11T09:52:13.674 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.674 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.674 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1", 2024-09-11T09:52:13.674 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.983525+0000", 2024-09-11T09:52:13.675 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.603e-05 2024-09-11T09:52:13.675 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.675 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.675 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:13.675 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.989004+0000", 2024-09-11T09:52:13.675 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0054785429999999998 2024-09-11T09:52:13.675 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.675 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.675 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:13.675 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.991788+0000", 2024-09-11T09:52:13.675 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0027845729999999998 2024-09-11T09:52:13.675 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.675 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.675 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:13.676 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.991870+0000", 2024-09-11T09:52:13.676 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.1567e-05 2024-09-11T09:52:13.676 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.676 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.676 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:13.676 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.991923+0000", 2024-09-11T09:52:13.676 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.3035000000000003e-05 2024-09-11T09:52:13.676 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:13.676 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:13.676 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:13.676 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.676 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.676 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4694.0:32 2.1 2:8a862b48:ns::rbd_data.1256a0c02879.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:13.677 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.084089+0000", 2024-09-11T09:52:13.677 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 50.557386301000001, 2024-09-11T09:52:13.677 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.020630553999999999, 2024-09-11T09:52:13.677 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:13.677 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:13.677 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:13.677 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:13.677 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4694", 2024-09-11T09:52:13.677 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:52:13.677 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 32 2024-09-11T09:52:13.677 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.677 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:13.677 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.678 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:13.678 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.084089+0000", 2024-09-11T09:52:13.678 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:13.678 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.678 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.678 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:13.678 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.084089+0000", 2024-09-11T09:52:13.678 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:13.678 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.678 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.678 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:13.678 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.084090+0000", 2024-09-11T09:52:13.679 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.2900000000000003e-07 2024-09-11T09:52:13.679 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.679 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.679 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:13.679 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.090866+0000", 2024-09-11T09:52:13.679 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0067756609999999997 2024-09-11T09:52:13.679 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.679 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.679 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:13.679 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.090891+0000", 2024-09-11T09:52:13.679 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.5119000000000001e-05 2024-09-11T09:52:13.679 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.679 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.679 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:13.680 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.090900+0000", 2024-09-11T09:52:13.680 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.4110000000000002e-06 2024-09-11T09:52:13.680 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.680 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.680 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:13.680 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.090968+0000", 2024-09-11T09:52:13.680 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.8041999999999997e-05 2024-09-11T09:52:13.680 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.680 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.680 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:13.680 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.091115+0000", 2024-09-11T09:52:13.680 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00014633499999999999 2024-09-11T09:52:13.680 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.681 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.681 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1", 2024-09-11T09:52:13.681 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.091284+0000", 2024-09-11T09:52:13.681 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000169836 2024-09-11T09:52:13.681 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.681 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.681 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:13.681 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.100896+0000", 2024-09-11T09:52:13.681 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0096118639999999995 2024-09-11T09:52:13.681 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.681 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.681 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:13.682 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.104604+0000", 2024-09-11T09:52:13.682 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0037074619999999999 2024-09-11T09:52:13.682 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.682 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.682 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:13.682 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.104669+0000", 2024-09-11T09:52:13.682 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.4836000000000001e-05 2024-09-11T09:52:13.682 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.682 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.682 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:13.683 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.104720+0000", 2024-09-11T09:52:13.683 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.1258999999999998e-05 2024-09-11T09:52:13.683 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:13.683 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:13.683 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:13.683 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.683 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.683 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4694.0:44 2.3 e53/15 2:cd1a7d1c:ns::rbd_data.1256a0c02879.0000000000000006:head v 53'3, mlcod=16'1)", 2024-09-11T09:52:13.683 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.120387+0000", 2024-09-11T09:52:13.683 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 50.521088648000003, 2024-09-11T09:52:13.683 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.011988468, 2024-09-11T09:52:13.684 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:13.684 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:13.684 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:13.684 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:13.684 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.684 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:13.684 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.120387+0000", 2024-09-11T09:52:13.684 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:13.684 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.684 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.684 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:13.684 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.120387+0000", 2024-09-11T09:52:13.684 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:13.685 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.685 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.685 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:13.685 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.120389+0000", 2024-09-11T09:52:13.685 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.018e-06 2024-09-11T09:52:13.685 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.685 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.685 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:13.685 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.127403+0000", 2024-09-11T09:52:13.685 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0070144769999999999 2024-09-11T09:52:13.685 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.685 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.685 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:13.686 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.127414+0000", 2024-09-11T09:52:13.686 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0991e-05 2024-09-11T09:52:13.686 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.686 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.686 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:13.686 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.127423+0000", 2024-09-11T09:52:13.686 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.8580000000000006e-06 2024-09-11T09:52:13.686 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.686 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.686 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:13.686 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.127468+0000", 2024-09-11T09:52:13.686 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.4830999999999998e-05 2024-09-11T09:52:13.686 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.686 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.687 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:13.687 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.127485+0000", 2024-09-11T09:52:13.687 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6838999999999999e-05 2024-09-11T09:52:13.687 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.687 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.687 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:13.687 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.132341+0000", 2024-09-11T09:52:13.687 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0048557959999999999 2024-09-11T09:52:13.687 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.687 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.687 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:13.687 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.132375+0000", 2024-09-11T09:52:13.687 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.4657999999999999e-05 2024-09-11T09:52:13.688 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:13.688 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:13.688 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:13.688 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.688 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.688 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4694.0:46 2.3 e53/15 2:ce0e1469:ns::rbd_data.1256a0c02879.0000000000000008:head v 53'4, mlcod=16'1)", 2024-09-11T09:52:13.688 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.129008+0000", 2024-09-11T09:52:13.688 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 50.512468061999996, 2024-09-11T09:52:13.688 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0087151220000000005, 2024-09-11T09:52:13.688 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:13.688 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:13.688 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:13.688 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:13.689 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.689 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:13.689 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.129008+0000", 2024-09-11T09:52:13.689 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:13.689 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.689 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.689 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:13.689 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.129008+0000", 2024-09-11T09:52:13.689 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:13.689 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.689 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.689 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:13.689 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.129009+0000", 2024-09-11T09:52:13.690 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.367e-06 2024-09-11T09:52:13.690 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.690 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.690 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:13.690 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.132839+0000", 2024-09-11T09:52:13.690 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0038297610000000001 2024-09-11T09:52:13.690 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.690 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.690 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:13.690 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.132849+0000", 2024-09-11T09:52:13.690 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0672e-05 2024-09-11T09:52:13.690 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.690 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.691 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:13.691 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.132856+0000", 2024-09-11T09:52:13.691 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.1380000000000004e-06 2024-09-11T09:52:13.691 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.691 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.691 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:13.691 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.132891+0000", 2024-09-11T09:52:13.691 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.5712e-05 2024-09-11T09:52:13.691 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.691 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.691 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:13.691 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.132907+0000", 2024-09-11T09:52:13.691 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6166000000000001e-05 2024-09-11T09:52:13.691 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.692 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.692 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:13.692 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.137681+0000", 2024-09-11T09:52:13.692 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0047735299999999998 2024-09-11T09:52:13.692 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.692 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.692 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:13.692 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.137723+0000", 2024-09-11T09:52:13.692 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.1776000000000001e-05 2024-09-11T09:52:13.692 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:13.692 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:13.692 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:13.692 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.693 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.693 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4694.0:48 2.3 e53/15 2:d2090b70:ns::rbd_data.1256a0c02879.0000000000000009:head v 53'5, mlcod=53'2)", 2024-09-11T09:52:13.693 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.135108+0000", 2024-09-11T09:52:13.693 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 50.506367165, 2024-09-11T09:52:13.693 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.010710679000000001, 2024-09-11T09:52:13.693 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:13.693 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:13.693 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:13.693 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:13.693 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.693 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:13.693 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.135108+0000", 2024-09-11T09:52:13.693 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:13.694 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.694 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.694 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:13.694 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.135108+0000", 2024-09-11T09:52:13.694 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:13.694 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.694 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.694 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:13.694 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.135110+0000", 2024-09-11T09:52:13.694 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.122e-06 2024-09-11T09:52:13.694 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.694 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.694 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:13.694 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.140876+0000", 2024-09-11T09:52:13.695 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.005766162 2024-09-11T09:52:13.695 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.695 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.695 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:13.695 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.140888+0000", 2024-09-11T09:52:13.695 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.201e-05 2024-09-11T09:52:13.695 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.695 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.695 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:13.695 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.140894+0000", 2024-09-11T09:52:13.696 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.1739999999999997e-06 2024-09-11T09:52:13.696 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.696 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.696 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:13.696 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.140971+0000", 2024-09-11T09:52:13.696 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.7495000000000003e-05 2024-09-11T09:52:13.696 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.696 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.696 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:13.696 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.140995+0000", 2024-09-11T09:52:13.696 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.3204999999999999e-05 2024-09-11T09:52:13.696 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.696 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.697 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:13.697 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.145757+0000", 2024-09-11T09:52:13.697 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0047627190000000003 2024-09-11T09:52:13.697 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.697 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.697 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:13.697 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.145819+0000", 2024-09-11T09:52:13.697 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.1791999999999994e-05 2024-09-11T09:52:13.697 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:13.697 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:13.697 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:13.697 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.697 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.698 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4694.0:50 2.1 2:83078786:ns::rbd_data.1256a0c02879.000000000000000a:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:13.698 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.141467+0000", 2024-09-11T09:52:13.698 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 50.500008440999999, 2024-09-11T09:52:13.698 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.012785889, 2024-09-11T09:52:13.698 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:13.698 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:13.698 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:13.698 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:13.698 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4694", 2024-09-11T09:52:13.698 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:52:13.698 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 50 2024-09-11T09:52:13.698 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.698 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:13.698 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.699 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:13.699 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.141467+0000", 2024-09-11T09:52:13.699 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:13.699 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.699 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.699 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:13.699 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.141467+0000", 2024-09-11T09:52:13.699 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:13.699 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.699 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.699 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:13.699 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.141468+0000", 2024-09-11T09:52:13.699 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1009999999999999e-06 2024-09-11T09:52:13.700 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.700 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.700 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:13.700 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.146432+0000", 2024-09-11T09:52:13.700 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0049640539999999999 2024-09-11T09:52:13.700 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.700 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.700 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:13.700 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.146446+0000", 2024-09-11T09:52:13.700 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3254e-05 2024-09-11T09:52:13.701 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.701 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.701 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:13.701 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.146452+0000", 2024-09-11T09:52:13.701 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.207e-06 2024-09-11T09:52:13.701 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.701 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.701 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:13.701 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.146510+0000", 2024-09-11T09:52:13.701 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.8322999999999997e-05 2024-09-11T09:52:13.701 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.701 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.701 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:13.702 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.146650+0000", 2024-09-11T09:52:13.702 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00013947100000000001 2024-09-11T09:52:13.702 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.702 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.702 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1", 2024-09-11T09:52:13.702 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.146779+0000", 2024-09-11T09:52:13.702 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000129059 2024-09-11T09:52:13.702 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.702 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.702 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:13.702 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.153778+0000", 2024-09-11T09:52:13.702 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0069997130000000003 2024-09-11T09:52:13.702 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.703 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.703 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:13.703 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.154180+0000", 2024-09-11T09:52:13.703 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00040159800000000003 2024-09-11T09:52:13.703 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.703 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.703 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:13.703 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.154218+0000", 2024-09-11T09:52:13.703 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.7795e-05 2024-09-11T09:52:13.703 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.703 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.703 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:13.703 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.154253+0000", 2024-09-11T09:52:13.703 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.5314000000000001e-05 2024-09-11T09:52:13.704 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:13.704 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:13.704 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:13.704 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.704 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.704 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4694.0:54 2.3 e53/15 2:c44f0253:ns::rbd_data.1256a0c02879.000000000000000b:head v 53'6, mlcod=53'4)", 2024-09-11T09:52:13.704 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.154926+0000", 2024-09-11T09:52:13.704 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 50.486550066, 2024-09-11T09:52:13.704 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0089497099999999996, 2024-09-11T09:52:13.704 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:13.704 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:13.704 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:13.704 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:13.705 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.705 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:13.705 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.154926+0000", 2024-09-11T09:52:13.705 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:13.705 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.705 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.705 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:13.705 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.154926+0000", 2024-09-11T09:52:13.705 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:13.705 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.705 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.705 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:13.705 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.154927+0000", 2024-09-11T09:52:13.705 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.7000000000000003e-07 2024-09-11T09:52:13.706 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.706 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.706 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:13.706 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.158712+0000", 2024-09-11T09:52:13.706 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0037855469999999998 2024-09-11T09:52:13.706 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.706 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.706 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:13.706 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.158726+0000", 2024-09-11T09:52:13.706 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3621000000000001e-05 2024-09-11T09:52:13.706 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.706 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.706 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:13.707 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.158732+0000", 2024-09-11T09:52:13.707 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.9089999999999997e-06 2024-09-11T09:52:13.707 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.707 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.707 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:13.707 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.158781+0000", 2024-09-11T09:52:13.707 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.8900000000000003e-05 2024-09-11T09:52:13.707 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.707 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.707 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:13.707 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.158805+0000", 2024-09-11T09:52:13.708 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.44e-05 2024-09-11T09:52:13.708 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.708 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.708 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:13.708 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.163826+0000", 2024-09-11T09:52:13.708 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0050206399999999998 2024-09-11T09:52:13.708 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.708 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.708 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:13.708 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.163875+0000", 2024-09-11T09:52:13.708 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.9722999999999998e-05 2024-09-11T09:52:13.708 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:13.708 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:13.709 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:13.709 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.709 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.709 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4694.0:56 2.3 e53/15 2:c462f029:ns::rbd_data.1256a0c02879.000000000000000c:head v 53'7, mlcod=53'4)", 2024-09-11T09:52:13.709 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.159549+0000", 2024-09-11T09:52:13.709 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 50.481926360000003, 2024-09-11T09:52:13.709 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.015591666000000001, 2024-09-11T09:52:13.709 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:13.709 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:13.709 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:13.709 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:13.710 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.710 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:13.710 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.159549+0000", 2024-09-11T09:52:13.710 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:13.710 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.710 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.710 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:13.710 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.159549+0000", 2024-09-11T09:52:13.710 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:13.710 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.710 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.710 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:13.710 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.159550+0000", 2024-09-11T09:52:13.711 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.1500000000000004e-07 2024-09-11T09:52:13.711 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.711 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.711 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:13.711 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.167834+0000", 2024-09-11T09:52:13.711 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0082840220000000003 2024-09-11T09:52:13.711 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.711 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.711 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:13.711 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.167857+0000", 2024-09-11T09:52:13.711 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.3337e-05 2024-09-11T09:52:13.711 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.711 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.712 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:13.712 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.167869+0000", 2024-09-11T09:52:13.712 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1556000000000001e-05 2024-09-11T09:52:13.712 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.712 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.712 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:13.712 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.167939+0000", 2024-09-11T09:52:13.712 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.0674999999999997e-05 2024-09-11T09:52:13.712 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.712 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.712 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:13.712 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.167967+0000", 2024-09-11T09:52:13.712 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.7322999999999999e-05 2024-09-11T09:52:13.713 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.713 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.713 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:13.713 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.175079+0000", 2024-09-11T09:52:13.713 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0071126150000000001 2024-09-11T09:52:13.713 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.713 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.713 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:13.713 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.175141+0000", 2024-09-11T09:52:13.713 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.1523e-05 2024-09-11T09:52:13.713 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:13.713 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:13.713 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:13.714 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.714 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.714 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.5314.0:31 2.1 2:8654721b:::rbd_data.1250f5e25b80.0000000000000000:head [write 393216~131072] snapc 3=[3] ondisk+write+known_if_redirected+supports_pool_eio e68)", 2024-09-11T09:52:13.714 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:50.230850+0000", 2024-09-11T09:52:13.714 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 23.410625598999999, 2024-09-11T09:52:13.714 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0073772539999999998, 2024-09-11T09:52:13.714 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:13.714 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:13.714 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:13.714 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:13.714 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.5314", 2024-09-11T09:52:13.714 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3853995732", 2024-09-11T09:52:13.714 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 31 2024-09-11T09:52:13.714 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.715 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:13.715 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.715 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:13.715 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.230850+0000", 2024-09-11T09:52:13.715 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:13.715 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.715 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.715 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:13.715 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.230850+0000", 2024-09-11T09:52:13.716 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:13.716 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.716 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.716 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:13.716 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.230850+0000", 2024-09-11T09:52:13.716 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.32e-07 2024-09-11T09:52:13.717 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.717 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.717 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:13.717 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.230908+0000", 2024-09-11T09:52:13.717 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.7698000000000002e-05 2024-09-11T09:52:13.717 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.717 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.717 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:13.717 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.230915+0000", 2024-09-11T09:52:13.717 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.81e-06 2024-09-11T09:52:13.717 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.718 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.718 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:13.718 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.230920+0000", 2024-09-11T09:52:13.718 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.9660000000000004e-06 2024-09-11T09:52:13.718 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.718 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.718 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:13.718 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.235055+0000", 2024-09-11T09:52:13.718 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0041354629999999998 2024-09-11T09:52:13.718 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.718 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.719 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:13.719 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.235145+0000", 2024-09-11T09:52:13.719 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.9362000000000003e-05 2024-09-11T09:52:13.719 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.719 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.719 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1", 2024-09-11T09:52:13.719 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.235282+0000", 2024-09-11T09:52:13.719 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00013710999999999999 2024-09-11T09:52:13.719 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.719 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.719 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:13.719 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.237335+0000", 2024-09-11T09:52:13.719 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.002053165 2024-09-11T09:52:13.720 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.720 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.720 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:13.720 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.238106+0000", 2024-09-11T09:52:13.720 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000771015 2024-09-11T09:52:13.720 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.720 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.720 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:13.720 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.238165+0000", 2024-09-11T09:52:13.720 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.8588000000000002e-05 2024-09-11T09:52:13.720 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.720 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.720 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:13.721 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.238227+0000", 2024-09-11T09:52:13.721 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.2644999999999994e-05 2024-09-11T09:52:13.721 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:13.721 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:13.721 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:13.721 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.721 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.721 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.5314.0:32 2.1 2:8654721b:::rbd_data.1250f5e25b80.0000000000000000:head [write 524288~131072] snapc 3=[3] ondisk+write+known_if_redirected+supports_pool_eio e68)", 2024-09-11T09:52:13.721 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:50.230932+0000", 2024-09-11T09:52:13.721 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 23.410543973999999, 2024-09-11T09:52:13.721 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0074882129999999996, 2024-09-11T09:52:13.721 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:13.721 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:13.722 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:13.722 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:13.722 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.5314", 2024-09-11T09:52:13.722 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3853995732", 2024-09-11T09:52:13.722 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 32 2024-09-11T09:52:13.722 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.722 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:13.722 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.722 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:13.722 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.230932+0000", 2024-09-11T09:52:13.722 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:13.723 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.723 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.723 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:13.723 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.230932+0000", 2024-09-11T09:52:13.723 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:13.723 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.723 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.723 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:13.723 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.230932+0000", 2024-09-11T09:52:13.723 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.9999999999999999e-07 2024-09-11T09:52:13.723 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.723 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.723 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:13.724 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.231022+0000", 2024-09-11T09:52:13.724 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.0439999999999995e-05 2024-09-11T09:52:13.724 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.724 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.724 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:13.724 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.231030+0000", 2024-09-11T09:52:13.724 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.3350000000000001e-06 2024-09-11T09:52:13.724 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.724 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.724 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:13.725 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.231034+0000", 2024-09-11T09:52:13.725 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.9759999999999997e-06 2024-09-11T09:52:13.725 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.725 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.725 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:13.725 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.235837+0000", 2024-09-11T09:52:13.725 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0048031910000000001 2024-09-11T09:52:13.725 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.725 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.725 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:13.725 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.235904+0000", 2024-09-11T09:52:13.725 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.7181999999999995e-05 2024-09-11T09:52:13.725 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.726 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.726 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1", 2024-09-11T09:52:13.726 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.236040+0000", 2024-09-11T09:52:13.726 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00013545199999999999 2024-09-11T09:52:13.726 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.726 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.726 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:13.726 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.238242+0000", 2024-09-11T09:52:13.726 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.002202809 2024-09-11T09:52:13.726 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.726 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.726 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:13.726 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.238301+0000", 2024-09-11T09:52:13.726 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.8573999999999999e-05 2024-09-11T09:52:13.727 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.727 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.727 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:13.727 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.238378+0000", 2024-09-11T09:52:13.727 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.7323999999999994e-05 2024-09-11T09:52:13.727 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.727 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.727 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:13.728 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.238420+0000", 2024-09-11T09:52:13.728 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.163e-05 2024-09-11T09:52:13.728 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:13.728 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:13.728 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:13.728 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.728 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.728 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.5314.0:33 2.1 2:8654721b:::rbd_data.1250f5e25b80.0000000000000000:head [write 655360~131072] snapc 3=[3] ondisk+write+known_if_redirected+supports_pool_eio e68)", 2024-09-11T09:52:13.728 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:50.231041+0000", 2024-09-11T09:52:13.728 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 23.410434814999999, 2024-09-11T09:52:13.728 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0075627139999999999, 2024-09-11T09:52:13.728 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:13.729 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:13.729 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:13.729 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:13.729 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.5314", 2024-09-11T09:52:13.729 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3853995732", 2024-09-11T09:52:13.729 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 33 2024-09-11T09:52:13.729 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.729 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:13.729 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.729 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:13.729 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.231041+0000", 2024-09-11T09:52:13.729 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:13.729 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.730 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.730 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:13.730 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.231041+0000", 2024-09-11T09:52:13.730 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:13.730 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.730 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.730 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:13.730 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.231041+0000", 2024-09-11T09:52:13.730 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.9499999999999998e-07 2024-09-11T09:52:13.730 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.730 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.730 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:13.730 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.231111+0000", 2024-09-11T09:52:13.730 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.0166999999999994e-05 2024-09-11T09:52:13.731 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.731 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.731 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:13.731 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.231122+0000", 2024-09-11T09:52:13.731 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0518e-05 2024-09-11T09:52:13.731 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.731 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.731 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:13.731 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.231128+0000", 2024-09-11T09:52:13.731 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.3509999999999998e-06 2024-09-11T09:52:13.731 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.731 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.731 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:13.732 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.236622+0000", 2024-09-11T09:52:13.732 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0054934989999999998 2024-09-11T09:52:13.732 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.732 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.732 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:13.732 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.236719+0000", 2024-09-11T09:52:13.732 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.7497999999999994e-05 2024-09-11T09:52:13.732 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.732 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.732 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1", 2024-09-11T09:52:13.732 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.236854+0000", 2024-09-11T09:52:13.732 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000134535 2024-09-11T09:52:13.732 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.732 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.733 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:13.733 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.238491+0000", 2024-09-11T09:52:13.733 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0016368610000000001 2024-09-11T09:52:13.733 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.733 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.733 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:13.733 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.238504+0000", 2024-09-11T09:52:13.733 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3927999999999999e-05 2024-09-11T09:52:13.733 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.733 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.733 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:13.733 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.238561+0000", 2024-09-11T09:52:13.733 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.6029999999999997e-05 2024-09-11T09:52:13.734 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.734 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.734 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:13.734 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.238604+0000", 2024-09-11T09:52:13.734 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.3031999999999997e-05 2024-09-11T09:52:13.734 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:13.734 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:13.734 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:13.734 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.734 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.734 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.5314.0:34 2.1 2:8654721b:::rbd_data.1250f5e25b80.0000000000000000:head [write 786432~131072] snapc 3=[3] ondisk+write+known_if_redirected+supports_pool_eio e68)", 2024-09-11T09:52:13.734 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:50.231139+0000", 2024-09-11T09:52:13.734 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 23.410336212000001, 2024-09-11T09:52:13.734 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0083689690000000004, 2024-09-11T09:52:13.735 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:13.735 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:13.735 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:13.735 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:13.735 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.5314", 2024-09-11T09:52:13.735 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3853995732", 2024-09-11T09:52:13.735 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 34 2024-09-11T09:52:13.735 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.735 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:13.735 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.735 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:13.735 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.231139+0000", 2024-09-11T09:52:13.735 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:13.736 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.736 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.736 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:13.736 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.231139+0000", 2024-09-11T09:52:13.736 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:13.736 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.736 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.736 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:13.736 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.231140+0000", 2024-09-11T09:52:13.736 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.1699999999999998e-07 2024-09-11T09:52:13.736 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.736 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.736 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:13.736 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.231247+0000", 2024-09-11T09:52:13.737 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000107088 2024-09-11T09:52:13.737 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.737 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.737 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:13.737 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.231260+0000", 2024-09-11T09:52:13.737 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.289e-05 2024-09-11T09:52:13.737 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.737 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.737 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:13.737 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.231272+0000", 2024-09-11T09:52:13.737 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2371999999999999e-05 2024-09-11T09:52:13.737 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.737 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.738 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:13.738 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.237380+0000", 2024-09-11T09:52:13.738 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.006107683 2024-09-11T09:52:13.738 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.738 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.738 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:13.738 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.237434+0000", 2024-09-11T09:52:13.738 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.4143e-05 2024-09-11T09:52:13.738 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.738 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.738 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1", 2024-09-11T09:52:13.738 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.237541+0000", 2024-09-11T09:52:13.739 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00010676800000000001 2024-09-11T09:52:13.739 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.739 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.739 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:13.739 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.239390+0000", 2024-09-11T09:52:13.739 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.001848861 2024-09-11T09:52:13.739 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.739 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.739 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:13.739 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.239409+0000", 2024-09-11T09:52:13.739 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.9505e-05 2024-09-11T09:52:13.740 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.740 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.740 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:13.740 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.239463+0000", 2024-09-11T09:52:13.740 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.3838000000000002e-05 2024-09-11T09:52:13.740 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.740 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.740 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:13.740 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.239508+0000", 2024-09-11T09:52:13.740 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.5303999999999999e-05 2024-09-11T09:52:13.740 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:13.740 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:13.741 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:13.741 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.741 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.741 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.5314.0:35 2.1 2:8654721b:::rbd_data.1250f5e25b80.0000000000000000:head [write 917504~131072] snapc 3=[3] ondisk+write+known_if_redirected+supports_pool_eio e68)", 2024-09-11T09:52:13.741 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:50.231287+0000", 2024-09-11T09:52:13.741 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 23.410188878, 2024-09-11T09:52:13.741 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0086930730000000008, 2024-09-11T09:52:13.741 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:13.741 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:13.742 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:13.742 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:13.742 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.5314", 2024-09-11T09:52:13.742 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3853995732", 2024-09-11T09:52:13.742 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 35 2024-09-11T09:52:13.742 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.742 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:13.742 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.742 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:13.742 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.231287+0000", 2024-09-11T09:52:13.742 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:13.743 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.743 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.743 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:13.743 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.231287+0000", 2024-09-11T09:52:13.743 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:13.743 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.743 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.743 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:13.743 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.231287+0000", 2024-09-11T09:52:13.743 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.7700000000000004e-07 2024-09-11T09:52:13.743 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.743 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.743 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:13.743 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.231354+0000", 2024-09-11T09:52:13.744 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.6304999999999996e-05 2024-09-11T09:52:13.744 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.744 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.744 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:13.744 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.231367+0000", 2024-09-11T09:52:13.744 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2952999999999999e-05 2024-09-11T09:52:13.744 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.744 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.744 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:13.744 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.231374+0000", 2024-09-11T09:52:13.744 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.0469999999999996e-06 2024-09-11T09:52:13.744 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.744 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.745 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:13.745 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.238650+0000", 2024-09-11T09:52:13.745 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0072760280000000004 2024-09-11T09:52:13.745 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.745 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.745 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:13.745 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.238732+0000", 2024-09-11T09:52:13.745 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.2194000000000006e-05 2024-09-11T09:52:13.745 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.745 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.745 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1", 2024-09-11T09:52:13.745 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.238876+0000", 2024-09-11T09:52:13.745 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00014458 2024-09-11T09:52:13.746 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.746 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.746 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:13.746 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.239739+0000", 2024-09-11T09:52:13.746 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00086244499999999999 2024-09-11T09:52:13.746 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.746 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.746 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:13.746 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.239917+0000", 2024-09-11T09:52:13.746 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00017777299999999999 2024-09-11T09:52:13.746 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.746 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.746 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:13.747 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.239954+0000", 2024-09-11T09:52:13.747 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.7379999999999998e-05 2024-09-11T09:52:13.747 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.747 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.747 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:13.747 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.239980+0000", 2024-09-11T09:52:13.747 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.5791000000000001e-05 2024-09-11T09:52:13.747 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:13.747 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:13.747 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:13.747 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.747 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.747 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.5314.0:70 2.1 2:8654721b:::rbd_data.1250f5e25b80.0000000000000000:head [write 524288~131072] snapc 5=[5,3] ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-09-11T09:52:13.747 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:50.824556+0000", 2024-09-11T09:52:13.748 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 22.816920076999999, 2024-09-11T09:52:13.748 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0072794469999999997, 2024-09-11T09:52:13.748 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:13.748 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:13.748 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:13.748 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:13.748 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.5314", 2024-09-11T09:52:13.748 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3853995732", 2024-09-11T09:52:13.748 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 70 2024-09-11T09:52:13.748 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.748 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:13.748 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.748 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:13.749 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.824556+0000", 2024-09-11T09:52:13.749 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:13.749 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.749 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.749 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:13.749 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.824556+0000", 2024-09-11T09:52:13.749 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:13.749 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.749 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.749 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:13.749 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.824564+0000", 2024-09-11T09:52:13.749 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.2409999999999995e-06 2024-09-11T09:52:13.749 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.749 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.750 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:13.750 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.824613+0000", 2024-09-11T09:52:13.750 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.9579000000000001e-05 2024-09-11T09:52:13.750 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.750 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.750 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:13.750 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.824638+0000", 2024-09-11T09:52:13.750 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.4510000000000001e-05 2024-09-11T09:52:13.750 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.750 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.750 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:13.751 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.824653+0000", 2024-09-11T09:52:13.751 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.4831000000000001e-05 2024-09-11T09:52:13.751 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.751 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.751 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:13.751 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.829072+0000", 2024-09-11T09:52:13.751 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0044195700000000003 2024-09-11T09:52:13.751 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.751 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.751 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:13.751 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.829166+0000", 2024-09-11T09:52:13.751 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.3212000000000001e-05 2024-09-11T09:52:13.751 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.752 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.752 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1", 2024-09-11T09:52:13.752 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.829309+0000", 2024-09-11T09:52:13.752 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00014301900000000001 2024-09-11T09:52:13.752 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.752 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.752 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:13.752 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.831074+0000", 2024-09-11T09:52:13.752 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.001765305 2024-09-11T09:52:13.752 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.752 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.752 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:13.752 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.831751+0000", 2024-09-11T09:52:13.752 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00067694000000000001 2024-09-11T09:52:13.753 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.753 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.753 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:13.753 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.831803+0000", 2024-09-11T09:52:13.753 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.2441000000000002e-05 2024-09-11T09:52:13.753 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.753 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.753 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:13.754 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.831835+0000", 2024-09-11T09:52:13.754 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.1798999999999998e-05 2024-09-11T09:52:13.754 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:13.754 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:13.754 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:13.754 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.754 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.754 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.5314.0:71 2.1 2:8654721b:::rbd_data.1250f5e25b80.0000000000000000:head [write 655360~131072] snapc 5=[5,3] ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-09-11T09:52:13.754 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:50.824662+0000", 2024-09-11T09:52:13.754 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 22.816813961000001, 2024-09-11T09:52:13.755 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0082172219999999997, 2024-09-11T09:52:13.755 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:13.755 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:13.755 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:13.755 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:13.757 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.5314", 2024-09-11T09:52:13.757 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3853995732", 2024-09-11T09:52:13.757 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 71 2024-09-11T09:52:13.757 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.757 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:13.757 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.758 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:13.758 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.824662+0000", 2024-09-11T09:52:13.758 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:13.758 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.758 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.758 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:13.758 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.824662+0000", 2024-09-11T09:52:13.758 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:13.758 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.758 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.758 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:13.758 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.824662+0000", 2024-09-11T09:52:13.758 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.0699999999999998e-07 2024-09-11T09:52:13.758 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.759 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.759 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:13.759 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.824859+0000", 2024-09-11T09:52:13.759 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00019725799999999999 2024-09-11T09:52:13.759 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.759 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.759 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:13.759 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.824906+0000", 2024-09-11T09:52:13.759 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.7162e-05 2024-09-11T09:52:13.759 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.759 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.759 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:13.759 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.824926+0000", 2024-09-11T09:52:13.759 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.0038999999999999e-05 2024-09-11T09:52:13.760 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.760 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.760 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:13.760 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.830313+0000", 2024-09-11T09:52:13.760 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0053868409999999999 2024-09-11T09:52:13.760 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.760 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.760 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:13.760 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.830359+0000", 2024-09-11T09:52:13.760 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.5370000000000001e-05 2024-09-11T09:52:13.760 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.760 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.760 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1", 2024-09-11T09:52:13.761 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.830460+0000", 2024-09-11T09:52:13.761 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00010171299999999999 2024-09-11T09:52:13.761 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.761 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.761 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:13.761 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.831843+0000", 2024-09-11T09:52:13.761 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.001382501 2024-09-11T09:52:13.761 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.761 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.761 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:13.761 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.832769+0000", 2024-09-11T09:52:13.762 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00092638099999999999 2024-09-11T09:52:13.762 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.762 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.762 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:13.762 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.832835+0000", 2024-09-11T09:52:13.762 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.6106999999999996e-05 2024-09-11T09:52:13.762 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.762 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.762 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:13.762 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.832879+0000", 2024-09-11T09:52:13.762 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.3543e-05 2024-09-11T09:52:13.762 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:13.763 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:13.763 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:13.763 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.763 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.763 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.5314.0:72 2.1 2:8654721b:::rbd_data.1250f5e25b80.0000000000000000:head [write 786432~131072] snapc 5=[5,3] ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-09-11T09:52:13.763 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:50.824938+0000", 2024-09-11T09:52:13.763 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 22.816537659000002, 2024-09-11T09:52:13.763 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0082389799999999999, 2024-09-11T09:52:13.763 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:13.763 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:13.763 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:13.763 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:13.763 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.5314", 2024-09-11T09:52:13.764 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3853995732", 2024-09-11T09:52:13.764 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 72 2024-09-11T09:52:13.764 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.764 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:13.764 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.764 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:13.764 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.824938+0000", 2024-09-11T09:52:13.764 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:13.764 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.764 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.764 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:13.764 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.824938+0000", 2024-09-11T09:52:13.764 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:13.765 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.765 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.765 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:13.765 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.824939+0000", 2024-09-11T09:52:13.765 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.6199999999999997e-07 2024-09-11T09:52:13.765 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.765 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.765 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:13.765 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.825028+0000", 2024-09-11T09:52:13.765 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.9105999999999997e-05 2024-09-11T09:52:13.765 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.765 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.765 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:13.766 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.825042+0000", 2024-09-11T09:52:13.766 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3760999999999999e-05 2024-09-11T09:52:13.766 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.766 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.766 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:13.766 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.825061+0000", 2024-09-11T09:52:13.766 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.9088000000000001e-05 2024-09-11T09:52:13.766 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.766 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.766 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:13.766 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.831122+0000", 2024-09-11T09:52:13.766 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.006061299 2024-09-11T09:52:13.766 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.767 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.767 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:13.767 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.831195+0000", 2024-09-11T09:52:13.767 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.2957e-05 2024-09-11T09:52:13.767 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.767 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.767 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1", 2024-09-11T09:52:13.767 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.831301+0000", 2024-09-11T09:52:13.767 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000106049 2024-09-11T09:52:13.767 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.767 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.767 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:13.767 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.832893+0000", 2024-09-11T09:52:13.767 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0015923770000000001 2024-09-11T09:52:13.768 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.768 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.768 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:13.768 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.833076+0000", 2024-09-11T09:52:13.768 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000182855 2024-09-11T09:52:13.768 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.768 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.768 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:13.768 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.833137+0000", 2024-09-11T09:52:13.768 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.1193999999999997e-05 2024-09-11T09:52:13.768 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.768 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.768 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:13.769 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.833177+0000", 2024-09-11T09:52:13.769 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.9632000000000003e-05 2024-09-11T09:52:13.769 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:13.769 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:13.769 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:13.769 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.769 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.769 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.5314.0:73 2.1 2:8654721b:::rbd_data.1250f5e25b80.0000000000000000:head [write 917504~131072] snapc 5=[5,3] ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-09-11T09:52:13.769 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:50.825071+0000", 2024-09-11T09:52:13.769 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 22.816404152, 2024-09-11T09:52:13.769 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0084039160000000009, 2024-09-11T09:52:13.770 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:13.770 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:13.770 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:13.770 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:13.770 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.5314", 2024-09-11T09:52:13.770 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3853995732", 2024-09-11T09:52:13.770 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 73 2024-09-11T09:52:13.770 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.770 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:13.770 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.770 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:13.770 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.825071+0000", 2024-09-11T09:52:13.770 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:13.771 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.771 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.771 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:13.771 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.825071+0000", 2024-09-11T09:52:13.771 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:13.771 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.771 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.771 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:13.771 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.825072+0000", 2024-09-11T09:52:13.771 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.5499999999999998e-07 2024-09-11T09:52:13.771 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.771 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.771 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:13.771 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.825124+0000", 2024-09-11T09:52:13.772 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.2197000000000002e-05 2024-09-11T09:52:13.772 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.772 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.772 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:13.772 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.825132+0000", 2024-09-11T09:52:13.772 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.0290000000000005e-06 2024-09-11T09:52:13.772 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.772 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.772 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:13.772 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.825136+0000", 2024-09-11T09:52:13.772 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.9380000000000002e-06 2024-09-11T09:52:13.772 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.772 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.773 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:13.773 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.832001+0000", 2024-09-11T09:52:13.773 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0068650949999999999 2024-09-11T09:52:13.773 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.773 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.773 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:13.773 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.832091+0000", 2024-09-11T09:52:13.773 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.9524999999999994e-05 2024-09-11T09:52:13.773 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.773 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.773 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1", 2024-09-11T09:52:13.774 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.832234+0000", 2024-09-11T09:52:13.774 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000142783 2024-09-11T09:52:13.774 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.774 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.774 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:13.774 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.833198+0000", 2024-09-11T09:52:13.774 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00096476899999999998 2024-09-11T09:52:13.774 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.774 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.774 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:13.774 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.833386+0000", 2024-09-11T09:52:13.774 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00018772 2024-09-11T09:52:13.774 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.775 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.775 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:13.775 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.833440+0000", 2024-09-11T09:52:13.775 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.4228999999999999e-05 2024-09-11T09:52:13.775 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:13.775 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:13.775 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:13.775 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.833475+0000", 2024-09-11T09:52:13.775 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.5076000000000001e-05 2024-09-11T09:52:13.775 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:13.775 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:13.775 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:13.775 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:13.775 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:13.776 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:52:13.777 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:52:13.777+0000 7f3a7b90a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 42475) UID: 0 2024-09-11T09:52:13.778 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-11T09:52:13.780 INFO:tasks.workunit.client.0.smithi107.stderr:1+0 records in 2024-09-11T09:52:13.780 INFO:tasks.workunit.client.0.smithi107.stderr:1+0 records out 2024-09-11T09:52:13.780 INFO:tasks.workunit.client.0.smithi107.stderr:1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0124691 s, 84.1 MB/s 2024-09-11T09:52:13.780 INFO:tasks.workunit.client.0.smithi107.stderr:+ _sudo umount /tmp/tmp.rvrcDirNue/mnt 2024-09-11T09:52:13.780 INFO:tasks.workunit.client.0.smithi107.stderr:+ local cmd 2024-09-11T09:52:13.780 INFO:tasks.workunit.client.0.smithi107.stderr:++ id -u 2024-09-11T09:52:13.780 INFO:tasks.workunit.client.0.smithi107.stderr:+ '[' 1000 -eq 0 ']' 2024-09-11T09:52:13.780 INFO:tasks.workunit.client.0.smithi107.stderr:++ which umount 2024-09-11T09:52:13.780 INFO:tasks.workunit.client.0.smithi107.stderr:+ cmd=/usr/bin/umount 2024-09-11T09:52:13.780 INFO:tasks.workunit.client.0.smithi107.stderr:+ shift 2024-09-11T09:52:13.780 INFO:tasks.workunit.client.0.smithi107.stderr:+ sudo -nE /usr/bin/umount /tmp/tmp.rvrcDirNue/mnt 2024-09-11T09:52:13.781 INFO:tasks.workunit.client.0.smithi107.stderr:+ unmap_device /dev/nbd1 29840 2024-09-11T09:52:13.781 INFO:tasks.workunit.client.0.smithi107.stderr:+ local args=/dev/nbd1 2024-09-11T09:52:13.781 INFO:tasks.workunit.client.0.smithi107.stderr:+ local pid=29840 2024-09-11T09:52:13.781 INFO:tasks.workunit.client.0.smithi107.stderr:+ _sudo rbd device --device-type nbd unmap /dev/nbd1 2024-09-11T09:52:13.781 INFO:tasks.workunit.client.0.smithi107.stderr:+ local cmd 2024-09-11T09:52:13.781 INFO:tasks.workunit.client.0.smithi107.stderr:++ id -u 2024-09-11T09:52:13.781 INFO:tasks.workunit.client.0.smithi107.stderr:+ '[' 1000 -eq 0 ']' 2024-09-11T09:52:13.781 INFO:tasks.workunit.client.0.smithi107.stderr:++ which rbd 2024-09-11T09:52:13.781 INFO:tasks.workunit.client.0.smithi107.stderr:+ cmd=/usr/bin/rbd 2024-09-11T09:52:13.781 INFO:tasks.workunit.client.0.smithi107.stderr:+ shift 2024-09-11T09:52:13.781 INFO:tasks.workunit.client.0.smithi107.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd unmap /dev/nbd1 2024-09-11T09:52:13.786 INFO:tasks.workunit.client.0.smithi107.stderr:+ rbd device --device-type nbd list 2024-09-11T09:52:13.786 INFO:tasks.workunit.client.0.smithi107.stderr:+ expect_false grep '^29840\b' 2024-09-11T09:52:13.787 INFO:tasks.workunit.client.0.smithi107.stderr:+ grep '^29840\b' 2024-09-11T09:52:13.845 INFO:tasks.workunit.client.0.smithi107.stderr:+ return 0 2024-09-11T09:52:13.846 INFO:tasks.workunit.client.0.smithi107.stderr:+ ps -C rbd-nbd 2024-09-11T09:52:13.846 INFO:tasks.workunit.client.0.smithi107.stderr:+ expect_false grep '^ *29840\b' 2024-09-11T09:52:13.846 INFO:tasks.workunit.client.0.smithi107.stderr:+ grep '^ *29840\b' 2024-09-11T09:52:13.857 INFO:tasks.workunit.client.0.smithi107.stderr:+ return 0 2024-09-11T09:52:13.857 INFO:tasks.workunit.client.0.smithi107.stderr:+ sleep 0.5 2024-09-11T09:52:13.866 INFO:teuthology.orchestra.run.smithi017.stderr:set 3.2 pg_upmap mapping to [0,3] 2024-09-11T09:52:13.877 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-09-11T09:52:13.878+0000 7f7b6c7d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31904) UID: 0 2024-09-11T09:52:13.898 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-11T09:52:13.898 INFO:teuthology.orchestra.run.smithi096.stdout: "ops": [], 2024-09-11T09:52:13.899 INFO:teuthology.orchestra.run.smithi096.stdout: "num_ops": 0 2024-09-11T09:52:13.899 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-11T09:52:13.908 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-11T09:52:13.978 INFO:tasks.ceph.osd.0.smithi017.stderr:2024-09-11T09:52:13.979+0000 7fb73c0f1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31665) UID: 0 2024-09-11T09:52:14.052 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-11T09:52:14.052 INFO:teuthology.orchestra.run.smithi096.stdout: "ops": [], 2024-09-11T09:52:14.052 INFO:teuthology.orchestra.run.smithi096.stdout: "complaint_time": 30, 2024-09-11T09:52:14.052 INFO:teuthology.orchestra.run.smithi096.stdout: "num_blocked_ops": 0 2024-09-11T09:52:14.052 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-11T09:52:14.060 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-11T09:52:14.104 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:52:14.105+0000 7f3a7b90a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 42475) UID: 0 2024-09-11T09:52:14.165 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-11T09:52:14.166 INFO:teuthology.orchestra.run.smithi096.stdout: "size": 20, 2024-09-11T09:52:14.166 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 600, 2024-09-11T09:52:14.166 INFO:teuthology.orchestra.run.smithi096.stdout: "ops": [ 2024-09-11T09:52:14.166 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.169 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:111 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:52:14.169 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.711668+0000", 2024-09-11T09:52:14.169 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 122.45324713700001, 2024-09-11T09:52:14.169 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0077145570000000004, 2024-09-11T09:52:14.170 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:14.170 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:14.170 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:14.170 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:52:14.170 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:52:14.170 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:52:14.170 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 111 2024-09-11T09:52:14.170 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.170 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:14.170 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.170 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:14.171 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711668+0000", 2024-09-11T09:52:14.171 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:14.171 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.171 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.171 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:14.171 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711668+0000", 2024-09-11T09:52:14.171 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:14.171 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.171 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.171 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:14.171 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711669+0000", 2024-09-11T09:52:14.171 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.99e-07 2024-09-11T09:52:14.171 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.172 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.172 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:14.172 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711673+0000", 2024-09-11T09:52:14.172 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.7440000000000001e-06 2024-09-11T09:52:14.172 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.172 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.172 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:14.172 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711686+0000", 2024-09-11T09:52:14.172 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.3317e-05 2024-09-11T09:52:14.172 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.173 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.173 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:14.173 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711697+0000", 2024-09-11T09:52:14.173 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1022000000000001e-05 2024-09-11T09:52:14.173 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.173 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.173 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:14.173 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.717065+0000", 2024-09-11T09:52:14.173 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.005367531 2024-09-11T09:52:14.173 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.173 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.173 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:14.173 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.717114+0000", 2024-09-11T09:52:14.174 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.9707999999999999e-05 2024-09-11T09:52:14.174 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.174 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.174 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:52:14.174 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.717190+0000", 2024-09-11T09:52:14.174 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.5394000000000004e-05 2024-09-11T09:52:14.174 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.174 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.174 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:14.174 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718793+0000", 2024-09-11T09:52:14.174 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001603653 2024-09-11T09:52:14.174 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.174 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.175 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:52:14.175 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719329+0000", 2024-09-11T09:52:14.175 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00053543799999999995 2024-09-11T09:52:14.175 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.175 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.175 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:14.175 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719359+0000", 2024-09-11T09:52:14.175 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.0420999999999999e-05 2024-09-11T09:52:14.175 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.175 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.175 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:14.175 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719383+0000", 2024-09-11T09:52:14.176 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.3730000000000001e-05 2024-09-11T09:52:14.176 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:14.176 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:14.176 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:14.176 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.176 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.176 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:112 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:52:14.176 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.711712+0000", 2024-09-11T09:52:14.176 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 122.453203577, 2024-09-11T09:52:14.176 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.008162084, 2024-09-11T09:52:14.176 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:14.176 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:14.177 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:14.177 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:52:14.177 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:52:14.177 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:52:14.177 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 112 2024-09-11T09:52:14.177 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.177 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:14.177 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.177 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:14.177 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711712+0000", 2024-09-11T09:52:14.177 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:14.177 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.177 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.177 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:14.178 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711712+0000", 2024-09-11T09:52:14.178 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:14.178 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.178 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.178 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:14.178 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711712+0000", 2024-09-11T09:52:14.178 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.6000000000000004e-07 2024-09-11T09:52:14.178 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.178 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.178 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:14.178 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711808+0000", 2024-09-11T09:52:14.178 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.5977999999999995e-05 2024-09-11T09:52:14.178 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.179 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.179 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:14.179 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711819+0000", 2024-09-11T09:52:14.179 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.0755e-05 2024-09-11T09:52:14.179 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.179 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.179 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:14.179 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711826+0000", 2024-09-11T09:52:14.179 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.7970000000000003e-06 2024-09-11T09:52:14.179 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.179 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.179 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:14.179 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.717435+0000", 2024-09-11T09:52:14.180 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0056090239999999998 2024-09-11T09:52:14.180 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.180 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.180 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:14.180 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.717496+0000", 2024-09-11T09:52:14.180 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.0543e-05 2024-09-11T09:52:14.180 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.180 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.180 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:52:14.180 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.717591+0000", 2024-09-11T09:52:14.180 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.5667000000000004e-05 2024-09-11T09:52:14.180 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.180 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.181 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:14.181 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718831+0000", 2024-09-11T09:52:14.181 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001240204 2024-09-11T09:52:14.181 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.181 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.181 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:52:14.181 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719819+0000", 2024-09-11T09:52:14.181 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00098751500000000001 2024-09-11T09:52:14.181 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.181 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.181 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:14.181 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719850+0000", 2024-09-11T09:52:14.181 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.1535999999999998e-05 2024-09-11T09:52:14.182 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.182 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.182 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:14.182 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719874+0000", 2024-09-11T09:52:14.182 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.3504999999999999e-05 2024-09-11T09:52:14.182 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:14.182 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:14.182 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:14.182 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.182 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.182 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:113 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:52:14.182 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.711842+0000", 2024-09-11T09:52:14.182 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 122.453073078, 2024-09-11T09:52:14.183 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0081160099999999999, 2024-09-11T09:52:14.183 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:14.183 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:14.183 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:14.183 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:52:14.183 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:52:14.183 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:52:14.183 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 113 2024-09-11T09:52:14.183 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.183 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:14.183 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.183 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:14.183 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711842+0000", 2024-09-11T09:52:14.184 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:14.184 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.184 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.184 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:14.184 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711842+0000", 2024-09-11T09:52:14.184 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:14.184 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.184 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.184 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:14.184 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711843+0000", 2024-09-11T09:52:14.184 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.4399999999999999e-07 2024-09-11T09:52:14.185 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.185 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.185 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:14.185 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711848+0000", 2024-09-11T09:52:14.185 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.4159999999999997e-06 2024-09-11T09:52:14.185 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.185 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.185 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:14.185 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711856+0000", 2024-09-11T09:52:14.185 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.7870000000000007e-06 2024-09-11T09:52:14.185 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.185 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.186 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:14.186 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711862+0000", 2024-09-11T09:52:14.186 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.3689999999999998e-06 2024-09-11T09:52:14.186 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.186 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.186 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:14.186 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.717965+0000", 2024-09-11T09:52:14.186 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0061028899999999997 2024-09-11T09:52:14.186 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.186 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.186 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:14.187 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718023+0000", 2024-09-11T09:52:14.187 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.8313000000000002e-05 2024-09-11T09:52:14.187 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.187 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.187 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:52:14.187 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718109+0000", 2024-09-11T09:52:14.187 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.6205000000000006e-05 2024-09-11T09:52:14.187 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.187 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.187 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:14.187 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719808+0000", 2024-09-11T09:52:14.187 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0016990060000000001 2024-09-11T09:52:14.187 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.188 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.188 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:52:14.188 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719883+0000", 2024-09-11T09:52:14.188 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.4624999999999998e-05 2024-09-11T09:52:14.188 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.188 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.189 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:14.189 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719910+0000", 2024-09-11T09:52:14.189 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.7512e-05 2024-09-11T09:52:14.189 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.189 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.189 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:14.190 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719958+0000", 2024-09-11T09:52:14.190 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.8143000000000003e-05 2024-09-11T09:52:14.190 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:14.190 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:14.190 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:14.191 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.191 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.191 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:52:14.191 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.711887+0000", 2024-09-11T09:52:14.191 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 122.453028464, 2024-09-11T09:52:14.192 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0086063589999999992, 2024-09-11T09:52:14.192 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:14.192 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:14.192 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:14.192 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:52:14.193 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:52:14.193 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:52:14.193 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 114 2024-09-11T09:52:14.193 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.193 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:14.193 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.194 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:14.194 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711887+0000", 2024-09-11T09:52:14.194 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:14.194 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.194 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.195 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:14.195 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711887+0000", 2024-09-11T09:52:14.195 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:14.195 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.195 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.196 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:14.196 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711887+0000", 2024-09-11T09:52:14.196 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.1500000000000005e-07 2024-09-11T09:52:14.196 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.196 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.197 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:14.197 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711897+0000", 2024-09-11T09:52:14.197 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.1900000000000001e-06 2024-09-11T09:52:14.197 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.197 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.198 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:14.198 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711902+0000", 2024-09-11T09:52:14.198 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.2349999999999999e-06 2024-09-11T09:52:14.198 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.198 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.199 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:14.199 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711906+0000", 2024-09-11T09:52:14.199 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.7129999999999999e-06 2024-09-11T09:52:14.199 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.199 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.199 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:14.200 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718429+0000", 2024-09-11T09:52:14.200 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0065231960000000002 2024-09-11T09:52:14.200 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.200 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.200 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:14.201 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718480+0000", 2024-09-11T09:52:14.201 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.1161e-05 2024-09-11T09:52:14.201 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.201 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.201 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:52:14.202 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718558+0000", 2024-09-11T09:52:14.202 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.7519000000000001e-05 2024-09-11T09:52:14.202 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.202 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.202 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:14.202 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720038+0000", 2024-09-11T09:52:14.203 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001480016 2024-09-11T09:52:14.203 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.203 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.203 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:52:14.203 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720417+0000", 2024-09-11T09:52:14.204 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00037943699999999999 2024-09-11T09:52:14.204 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.204 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.204 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:14.204 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720452+0000", 2024-09-11T09:52:14.205 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.4595000000000003e-05 2024-09-11T09:52:14.205 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.205 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.205 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:14.205 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720493+0000", 2024-09-11T09:52:14.205 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.1782000000000001e-05 2024-09-11T09:52:14.206 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:14.206 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:14.206 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:14.206 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.206 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.207 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:52:14.207 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.711912+0000", 2024-09-11T09:52:14.207 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 122.453003795, 2024-09-11T09:52:14.207 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0086864750000000008, 2024-09-11T09:52:14.207 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:14.208 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:14.208 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:14.208 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:52:14.208 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:52:14.208 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:52:14.208 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 115 2024-09-11T09:52:14.209 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.209 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:14.209 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.209 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:14.209 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711912+0000", 2024-09-11T09:52:14.210 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:14.210 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.210 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.210 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:14.210 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711912+0000", 2024-09-11T09:52:14.211 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:14.211 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.211 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.211 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:14.211 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711912+0000", 2024-09-11T09:52:14.212 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.3900000000000001e-07 2024-09-11T09:52:14.212 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.212 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.212 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:14.212 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711969+0000", 2024-09-11T09:52:14.212 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.7501999999999999e-05 2024-09-11T09:52:14.213 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.213 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.213 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:14.213 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711986+0000", 2024-09-11T09:52:14.213 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.6402999999999999e-05 2024-09-11T09:52:14.214 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.214 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.214 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:14.214 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711990+0000", 2024-09-11T09:52:14.214 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.929e-06 2024-09-11T09:52:14.215 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.215 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.215 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:14.215 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718861+0000", 2024-09-11T09:52:14.215 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0068710530000000002 2024-09-11T09:52:14.216 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.216 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.216 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:14.216 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718908+0000", 2024-09-11T09:52:14.216 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.7431e-05 2024-09-11T09:52:14.217 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.217 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.217 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:52:14.217 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718984+0000", 2024-09-11T09:52:14.218 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.5500999999999994e-05 2024-09-11T09:52:14.218 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.218 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.218 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:52:14.218 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720500+0000", 2024-09-11T09:52:14.219 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0015158820000000001 2024-09-11T09:52:14.219 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.219 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.219 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:14.219 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720536+0000", 2024-09-11T09:52:14.220 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.6118000000000002e-05 2024-09-11T09:52:14.220 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.220 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.220 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:14.220 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720570+0000", 2024-09-11T09:52:14.221 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.4387999999999997e-05 2024-09-11T09:52:14.221 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.221 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.221 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:14.221 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720598+0000", 2024-09-11T09:52:14.221 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.8028999999999999e-05 2024-09-11T09:52:14.222 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:14.222 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:14.222 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:14.222 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.222 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.223 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:52:14.223 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.711998+0000", 2024-09-11T09:52:14.223 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 122.45291761599999, 2024-09-11T09:52:14.223 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.010785813999999999, 2024-09-11T09:52:14.223 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:14.224 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:14.224 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:14.224 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:52:14.224 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:52:14.224 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:52:14.225 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 116 2024-09-11T09:52:14.225 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.225 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:14.225 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.225 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:14.226 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711998+0000", 2024-09-11T09:52:14.226 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:14.226 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.226 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.226 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:14.226 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711998+0000", 2024-09-11T09:52:14.227 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:14.227 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.227 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.227 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:14.227 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711998+0000", 2024-09-11T09:52:14.228 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.3300000000000003e-07 2024-09-11T09:52:14.228 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.228 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.228 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:14.228 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712001+0000", 2024-09-11T09:52:14.229 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.9349999999999999e-06 2024-09-11T09:52:14.229 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.229 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.229 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:14.229 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712006+0000", 2024-09-11T09:52:14.229 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.1630000000000004e-06 2024-09-11T09:52:14.229 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.229 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.229 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:14.229 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712010+0000", 2024-09-11T09:52:14.229 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.574e-06 2024-09-11T09:52:14.229 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.230 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.230 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:14.230 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719415+0000", 2024-09-11T09:52:14.230 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0074049099999999998 2024-09-11T09:52:14.230 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.230 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.230 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:14.230 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719465+0000", 2024-09-11T09:52:14.230 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.0129000000000001e-05 2024-09-11T09:52:14.230 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.230 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.230 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:52:14.230 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719545+0000", 2024-09-11T09:52:14.230 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.0253999999999994e-05 2024-09-11T09:52:14.231 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.231 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.231 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:14.231 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720636+0000", 2024-09-11T09:52:14.231 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0010909330000000001 2024-09-11T09:52:14.231 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.231 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.231 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:52:14.231 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.722707+0000", 2024-09-11T09:52:14.231 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0020713300000000001 2024-09-11T09:52:14.231 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.231 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.231 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:14.231 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.722743+0000", 2024-09-11T09:52:14.232 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.5351000000000001e-05 2024-09-11T09:52:14.232 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.232 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.232 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:14.232 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.722784+0000", 2024-09-11T09:52:14.232 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.0802e-05 2024-09-11T09:52:14.232 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:14.232 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:14.232 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:14.232 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.232 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.232 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:52:14.232 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.712014+0000", 2024-09-11T09:52:14.233 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 122.452901576, 2024-09-11T09:52:14.233 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.013420938, 2024-09-11T09:52:14.233 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:14.233 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:14.233 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:14.233 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:52:14.233 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:52:14.233 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:52:14.233 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 117 2024-09-11T09:52:14.233 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.233 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:14.233 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.233 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:14.233 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712014+0000", 2024-09-11T09:52:14.234 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:14.234 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.234 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.234 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:14.234 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712014+0000", 2024-09-11T09:52:14.234 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:14.234 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.234 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.234 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:14.234 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712014+0000", 2024-09-11T09:52:14.234 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.2000000000000001e-07 2024-09-11T09:52:14.234 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.234 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.234 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:14.235 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712016+0000", 2024-09-11T09:52:14.235 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.2309999999999999e-06 2024-09-11T09:52:14.235 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.235 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.235 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:14.235 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712020+0000", 2024-09-11T09:52:14.235 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.3830000000000001e-06 2024-09-11T09:52:14.235 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.235 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.235 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:14.235 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712023+0000", 2024-09-11T09:52:14.235 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.3340000000000002e-06 2024-09-11T09:52:14.235 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.236 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.236 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:14.236 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720083+0000", 2024-09-11T09:52:14.236 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0080601189999999993 2024-09-11T09:52:14.236 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.236 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.236 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:14.236 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720125+0000", 2024-09-11T09:52:14.236 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.2029e-05 2024-09-11T09:52:14.236 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.236 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.236 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:52:14.236 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720205+0000", 2024-09-11T09:52:14.237 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.0010000000000001e-05 2024-09-11T09:52:14.237 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.237 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.237 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:14.237 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720849+0000", 2024-09-11T09:52:14.237 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00064369899999999996 2024-09-11T09:52:14.237 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.237 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.237 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:52:14.237 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.725357+0000", 2024-09-11T09:52:14.237 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0045077379999999998 2024-09-11T09:52:14.237 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.237 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.238 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:14.238 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.725392+0000", 2024-09-11T09:52:14.238 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.5862999999999999e-05 2024-09-11T09:52:14.238 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.238 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.238 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:14.238 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.725435+0000", 2024-09-11T09:52:14.238 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.2311999999999997e-05 2024-09-11T09:52:14.238 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:14.238 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:14.238 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:14.238 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.238 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.238 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:159 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:52:14.239 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:12.452099+0000", 2024-09-11T09:52:14.239 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 121.712815988, 2024-09-11T09:52:14.239 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0069885950000000002, 2024-09-11T09:52:14.239 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:14.239 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:14.239 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:14.239 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:52:14.239 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:52:14.239 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:52:14.239 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 159 2024-09-11T09:52:14.239 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.239 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:14.239 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.240 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:14.240 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452099+0000", 2024-09-11T09:52:14.240 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:14.240 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.240 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.240 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:14.240 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452099+0000", 2024-09-11T09:52:14.240 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:14.240 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.240 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.240 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:14.240 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452100+0000", 2024-09-11T09:52:14.241 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.2800000000000003e-07 2024-09-11T09:52:14.241 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.241 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.241 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:14.241 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452102+0000", 2024-09-11T09:52:14.241 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.824e-06 2024-09-11T09:52:14.241 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.241 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.241 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:14.241 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452105+0000", 2024-09-11T09:52:14.241 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.0460000000000001e-06 2024-09-11T09:52:14.241 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.241 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.242 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:14.242 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452107+0000", 2024-09-11T09:52:14.242 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.543e-06 2024-09-11T09:52:14.242 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.242 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.242 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:14.242 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.455484+0000", 2024-09-11T09:52:14.242 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0033769759999999998 2024-09-11T09:52:14.242 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.242 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.242 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:14.242 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.455527+0000", 2024-09-11T09:52:14.242 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.2589999999999997e-05 2024-09-11T09:52:14.242 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.243 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.243 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:52:14.243 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.455594+0000", 2024-09-11T09:52:14.243 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.6775999999999999e-05 2024-09-11T09:52:14.243 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.243 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.243 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:14.243 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.456312+0000", 2024-09-11T09:52:14.243 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00071874700000000005 2024-09-11T09:52:14.243 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.243 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.243 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:52:14.243 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.459015+0000", 2024-09-11T09:52:14.244 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0027028540000000002 2024-09-11T09:52:14.244 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.244 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.244 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:14.244 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.459054+0000", 2024-09-11T09:52:14.244 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.8840000000000001e-05 2024-09-11T09:52:14.244 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.244 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.244 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:14.244 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.459088+0000", 2024-09-11T09:52:14.244 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.4071e-05 2024-09-11T09:52:14.244 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:14.244 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:14.245 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:14.245 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.245 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.245 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4688.0:38 2.5 2:abde2694:::rbd_data.1250f5e25b80.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:14.245 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:22.903926+0000", 2024-09-11T09:52:14.245 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 51.260989312, 2024-09-11T09:52:14.245 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.028303676, 2024-09-11T09:52:14.245 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:14.245 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:14.245 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:14.245 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:52:14.245 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4688", 2024-09-11T09:52:14.245 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:52:14.246 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 38 2024-09-11T09:52:14.246 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.246 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:14.246 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.246 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:14.246 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.903926+0000", 2024-09-11T09:52:14.246 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:14.246 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.246 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.246 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:14.246 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.903926+0000", 2024-09-11T09:52:14.246 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:14.246 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.246 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.247 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:14.247 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.903928+0000", 2024-09-11T09:52:14.247 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.4249999999999999e-06 2024-09-11T09:52:14.247 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.247 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.247 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:14.247 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.914774+0000", 2024-09-11T09:52:14.247 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.010846613 2024-09-11T09:52:14.247 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.247 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.247 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:14.247 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.914793+0000", 2024-09-11T09:52:14.247 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.8879999999999999e-05 2024-09-11T09:52:14.248 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.248 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.248 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:14.248 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.914801+0000", 2024-09-11T09:52:14.248 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.2230000000000007e-06 2024-09-11T09:52:14.248 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.248 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.248 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:14.248 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.914854+0000", 2024-09-11T09:52:14.248 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.2883999999999997e-05 2024-09-11T09:52:14.248 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.248 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.248 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:14.249 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.914951+0000", 2024-09-11T09:52:14.249 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.6731000000000006e-05 2024-09-11T09:52:14.249 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.249 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.249 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:52:14.249 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.915080+0000", 2024-09-11T09:52:14.249 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00012926999999999999 2024-09-11T09:52:14.249 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.249 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.249 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:14.249 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.931469+0000", 2024-09-11T09:52:14.249 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.016389107 2024-09-11T09:52:14.249 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.250 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.250 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:52:14.250 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.932118+0000", 2024-09-11T09:52:14.250 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00064915600000000002 2024-09-11T09:52:14.250 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.250 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.250 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:14.250 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.932177+0000", 2024-09-11T09:52:14.250 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.9032e-05 2024-09-11T09:52:14.250 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.250 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.250 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:14.250 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.932230+0000", 2024-09-11T09:52:14.250 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.2355000000000003e-05 2024-09-11T09:52:14.251 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:14.251 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:14.251 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:14.251 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.251 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.251 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_repop(client.4688.0:36 2.6 e53/15 2:61e1b442:::rbd_data.1250f5e25b80.0000000000000003:head v 53'2, mlcod=0'0)", 2024-09-11T09:52:14.251 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:22.915597+0000", 2024-09-11T09:52:14.251 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 51.249318762000001, 2024-09-11T09:52:14.251 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.020430798, 2024-09-11T09:52:14.251 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:14.251 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:14.251 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:14.251 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:14.252 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.252 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:14.252 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.915597+0000", 2024-09-11T09:52:14.252 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:14.252 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.252 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.252 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:14.252 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.915597+0000", 2024-09-11T09:52:14.252 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:14.252 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.252 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.252 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:14.252 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.915598+0000", 2024-09-11T09:52:14.253 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.7150000000000001e-06 2024-09-11T09:52:14.253 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.253 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.253 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:14.253 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921741+0000", 2024-09-11T09:52:14.253 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0061424339999999996 2024-09-11T09:52:14.253 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.253 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.253 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:14.253 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921757+0000", 2024-09-11T09:52:14.253 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.6144e-05 2024-09-11T09:52:14.253 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.253 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.253 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:14.254 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921765+0000", 2024-09-11T09:52:14.254 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.0809999999999994e-06 2024-09-11T09:52:14.254 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.254 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.254 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:14.254 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921826+0000", 2024-09-11T09:52:14.254 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.0878999999999998e-05 2024-09-11T09:52:14.254 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.254 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.254 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:14.254 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921849+0000", 2024-09-11T09:52:14.254 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.2966999999999999e-05 2024-09-11T09:52:14.254 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.255 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.255 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:14.255 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.935968+0000", 2024-09-11T09:52:14.255 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.014119191 2024-09-11T09:52:14.255 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.255 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.255 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:14.255 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.936027+0000", 2024-09-11T09:52:14.255 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.9386999999999999e-05 2024-09-11T09:52:14.255 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:14.255 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:14.255 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:14.255 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.255 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.256 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_repop(client.4688.0:39 2.6 e53/15 2:743e7616:::rbd_data.1250f5e25b80.0000000000000005:head v 53'3, mlcod=0'0)", 2024-09-11T09:52:14.256 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:22.921806+0000", 2024-09-11T09:52:14.256 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 51.243109711999999, 2024-09-11T09:52:14.256 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.015275018, 2024-09-11T09:52:14.256 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:14.256 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:14.256 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:14.256 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:14.256 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.256 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:14.256 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921806+0000", 2024-09-11T09:52:14.256 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:14.256 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.257 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.257 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:14.257 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921806+0000", 2024-09-11T09:52:14.257 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:14.257 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.257 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.257 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:14.257 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921806+0000", 2024-09-11T09:52:14.257 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.2200000000000003e-07 2024-09-11T09:52:14.257 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.257 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.257 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:14.257 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.925590+0000", 2024-09-11T09:52:14.258 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0037838949999999998 2024-09-11T09:52:14.258 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.258 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.258 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:14.258 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.925605+0000", 2024-09-11T09:52:14.258 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.4321999999999999e-05 2024-09-11T09:52:14.258 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.258 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.258 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:14.258 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.925613+0000", 2024-09-11T09:52:14.258 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.1980000000000001e-06 2024-09-11T09:52:14.258 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.258 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.259 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:14.259 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.925857+0000", 2024-09-11T09:52:14.259 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000244064 2024-09-11T09:52:14.259 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.259 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.259 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:14.259 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.925875+0000", 2024-09-11T09:52:14.259 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.838e-05 2024-09-11T09:52:14.259 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.259 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.259 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:14.259 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.937042+0000", 2024-09-11T09:52:14.259 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.011166416 2024-09-11T09:52:14.260 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.260 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.260 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:14.260 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.937081+0000", 2024-09-11T09:52:14.260 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.9020999999999998e-05 2024-09-11T09:52:14.260 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:14.260 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:14.260 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:14.260 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.260 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.260 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4688.0:49 2.5 2:b6dd9fc9:::rbd_data.1250f5e25b80.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:14.260 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:22.938548+0000", 2024-09-11T09:52:14.260 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 51.226367709000002, 2024-09-11T09:52:14.261 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.022766950000000001, 2024-09-11T09:52:14.261 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:14.261 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:14.261 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:14.261 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:52:14.261 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4688", 2024-09-11T09:52:14.261 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:52:14.261 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 49 2024-09-11T09:52:14.261 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.261 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:14.261 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.261 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:14.261 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.938548+0000", 2024-09-11T09:52:14.262 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:14.262 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.262 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.262 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:14.262 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.938548+0000", 2024-09-11T09:52:14.262 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:14.262 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.262 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.262 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:14.262 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.938550+0000", 2024-09-11T09:52:14.262 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.7710000000000001e-06 2024-09-11T09:52:14.262 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.262 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.262 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:14.263 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.944237+0000", 2024-09-11T09:52:14.263 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0056879790000000001 2024-09-11T09:52:14.263 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.263 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.263 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:14.263 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.944262+0000", 2024-09-11T09:52:14.263 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.4247e-05 2024-09-11T09:52:14.263 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.263 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.263 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:14.263 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.944274+0000", 2024-09-11T09:52:14.263 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.2639999999999999e-05 2024-09-11T09:52:14.263 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.264 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.264 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:14.264 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.944344+0000", 2024-09-11T09:52:14.264 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.9574999999999997e-05 2024-09-11T09:52:14.264 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.264 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.264 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:14.264 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.944474+0000", 2024-09-11T09:52:14.264 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000130464 2024-09-11T09:52:14.264 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.264 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.264 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:52:14.264 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.944585+0000", 2024-09-11T09:52:14.265 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000110096 2024-09-11T09:52:14.265 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.265 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.265 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:52:14.265 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.952724+0000", 2024-09-11T09:52:14.265 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0081398900000000003 2024-09-11T09:52:14.265 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.265 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.265 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:14.265 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.961219+0000", 2024-09-11T09:52:14.265 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0084949789999999997 2024-09-11T09:52:14.265 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.265 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.265 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:14.266 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.961280+0000", 2024-09-11T09:52:14.266 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.0404e-05 2024-09-11T09:52:14.266 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.266 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.266 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:14.266 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.961315+0000", 2024-09-11T09:52:14.266 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.4904999999999999e-05 2024-09-11T09:52:14.266 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:14.266 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:14.266 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:14.266 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.266 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.266 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_repop(client.4688.0:44 2.6 e53/15 2:67246a22:::rbd_data.1250f5e25b80.0000000000000007:head v 53'4, mlcod=53'2)", 2024-09-11T09:52:14.266 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:22.951721+0000", 2024-09-11T09:52:14.267 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 51.213194436999999, 2024-09-11T09:52:14.267 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.01080981, 2024-09-11T09:52:14.267 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:14.267 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:14.267 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:14.267 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:14.267 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.267 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:14.267 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.951721+0000", 2024-09-11T09:52:14.267 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:14.267 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.267 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.267 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:14.268 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.951721+0000", 2024-09-11T09:52:14.268 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:14.268 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.268 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.268 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:14.268 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.951722+0000", 2024-09-11T09:52:14.268 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.0279999999999999e-06 2024-09-11T09:52:14.268 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.268 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.268 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:14.268 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.955612+0000", 2024-09-11T09:52:14.268 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0038896920000000001 2024-09-11T09:52:14.268 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.269 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.269 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:14.269 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.955628+0000", 2024-09-11T09:52:14.269 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.5945000000000001e-05 2024-09-11T09:52:14.269 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.269 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.269 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:14.269 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.955635+0000", 2024-09-11T09:52:14.269 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.8080000000000005e-06 2024-09-11T09:52:14.269 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.269 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.269 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:14.269 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.955700+0000", 2024-09-11T09:52:14.269 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.4114000000000003e-05 2024-09-11T09:52:14.270 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.270 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.270 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:14.270 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.955724+0000", 2024-09-11T09:52:14.270 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.3989999999999999e-05 2024-09-11T09:52:14.270 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.270 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.270 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:14.270 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.962490+0000", 2024-09-11T09:52:14.270 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0067662599999999996 2024-09-11T09:52:14.270 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.270 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.270 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:14.271 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.962531+0000", 2024-09-11T09:52:14.271 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.0973000000000002e-05 2024-09-11T09:52:14.271 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:14.271 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:14.271 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:14.271 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.271 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.271 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_repop(client.4688.0:51 2.6 e53/15 2:6b5894fc:::rbd_data.1250f5e25b80.000000000000000a:head v 53'5, mlcod=53'2)", 2024-09-11T09:52:14.271 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:22.959030+0000", 2024-09-11T09:52:14.271 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 51.205885262999999, 2024-09-11T09:52:14.271 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.011503638, 2024-09-11T09:52:14.271 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:14.271 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:14.272 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:14.272 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:14.272 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.272 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:14.272 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.959030+0000", 2024-09-11T09:52:14.272 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:14.272 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.272 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.272 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:14.272 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.959030+0000", 2024-09-11T09:52:14.272 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:14.272 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.272 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.273 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:14.273 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.959032+0000", 2024-09-11T09:52:14.273 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.356e-06 2024-09-11T09:52:14.273 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.273 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.273 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:14.273 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.962899+0000", 2024-09-11T09:52:14.273 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.003867639 2024-09-11T09:52:14.273 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.273 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.273 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:14.273 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.962909+0000", 2024-09-11T09:52:14.273 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.0105999999999999e-05 2024-09-11T09:52:14.274 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.274 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.274 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:14.274 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.962915+0000", 2024-09-11T09:52:14.274 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.8869999999999997e-06 2024-09-11T09:52:14.274 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.274 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.274 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:14.274 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.962962+0000", 2024-09-11T09:52:14.274 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.6511999999999998e-05 2024-09-11T09:52:14.274 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.274 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.274 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:14.274 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.962989+0000", 2024-09-11T09:52:14.275 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.7702999999999999e-05 2024-09-11T09:52:14.275 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.275 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.275 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:14.275 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.970495+0000", 2024-09-11T09:52:14.275 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0075052449999999998 2024-09-11T09:52:14.275 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.275 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.275 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:14.275 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.970534+0000", 2024-09-11T09:52:14.275 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.9190000000000003e-05 2024-09-11T09:52:14.275 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:14.275 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:14.275 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:14.276 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.276 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.276 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_repop(client.4688.0:58 2.6 e53/15 2:6277c644:::rbd_data.1250f5e25b80.000000000000000e:head v 53'6, mlcod=53'4)", 2024-09-11T09:52:14.276 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:22.996199+0000", 2024-09-11T09:52:14.276 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 51.168716343, 2024-09-11T09:52:14.276 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.010145573999999999, 2024-09-11T09:52:14.276 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:14.276 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:14.276 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:14.276 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:14.276 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.276 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:14.276 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.996199+0000", 2024-09-11T09:52:14.277 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:14.277 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.277 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.277 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:14.277 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.996199+0000", 2024-09-11T09:52:14.277 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:14.277 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.277 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.277 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:14.277 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.996200+0000", 2024-09-11T09:52:14.277 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.85e-07 2024-09-11T09:52:14.277 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.277 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.278 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:14.278 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.999914+0000", 2024-09-11T09:52:14.278 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.003714406 2024-09-11T09:52:14.278 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.278 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.278 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:14.278 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.999928+0000", 2024-09-11T09:52:14.278 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.3149000000000001e-05 2024-09-11T09:52:14.278 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.278 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.278 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:14.278 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.999951+0000", 2024-09-11T09:52:14.278 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.3116e-05 2024-09-11T09:52:14.278 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.279 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.279 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:14.279 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.000003+0000", 2024-09-11T09:52:14.279 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.2169000000000001e-05 2024-09-11T09:52:14.279 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.279 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.279 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:14.279 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.000018+0000", 2024-09-11T09:52:14.279 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.5500000000000001e-05 2024-09-11T09:52:14.279 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.279 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.279 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:14.279 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.006303+0000", 2024-09-11T09:52:14.280 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0062846710000000004 2024-09-11T09:52:14.280 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.280 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.280 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:14.280 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.006345+0000", 2024-09-11T09:52:14.280 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.1678000000000003e-05 2024-09-11T09:52:14.280 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:14.280 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:14.280 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:14.280 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.280 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.280 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4694.0:42 2.5 2:a4fc00ec:ns::rbd_data.1256a0c02879.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:14.280 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.111570+0000", 2024-09-11T09:52:14.280 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 51.053345237000002, 2024-09-11T09:52:14.281 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.038002197000000001, 2024-09-11T09:52:14.281 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:14.281 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:14.281 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:14.281 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:52:14.281 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4694", 2024-09-11T09:52:14.281 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:52:14.281 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 42 2024-09-11T09:52:14.281 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.281 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:14.281 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.281 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:14.282 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.111570+0000", 2024-09-11T09:52:14.282 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:14.282 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.282 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.282 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:14.282 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.111570+0000", 2024-09-11T09:52:14.282 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:14.282 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.282 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.282 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:14.282 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.111572+0000", 2024-09-11T09:52:14.282 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.3909999999999999e-06 2024-09-11T09:52:14.282 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.282 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.283 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:14.283 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.135929+0000", 2024-09-11T09:52:14.283 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.024357410999999999 2024-09-11T09:52:14.283 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.283 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.283 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:14.283 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.135951+0000", 2024-09-11T09:52:14.283 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.1775000000000001e-05 2024-09-11T09:52:14.283 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.283 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.283 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:14.283 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.135962+0000", 2024-09-11T09:52:14.283 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.0903e-05 2024-09-11T09:52:14.284 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.284 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.284 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:14.284 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.136045+0000", 2024-09-11T09:52:14.284 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.3128999999999996e-05 2024-09-11T09:52:14.284 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.284 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.284 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:14.284 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.136229+0000", 2024-09-11T09:52:14.284 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00018393700000000001 2024-09-11T09:52:14.285 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.285 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.285 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:52:14.285 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.136394+0000", 2024-09-11T09:52:14.285 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000165527 2024-09-11T09:52:14.285 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.285 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.285 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:14.285 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.149320+0000", 2024-09-11T09:52:14.285 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.01292568 2024-09-11T09:52:14.285 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.285 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.285 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:52:14.285 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.149493+0000", 2024-09-11T09:52:14.286 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00017285999999999999 2024-09-11T09:52:14.286 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.286 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.286 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:14.286 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.149539+0000", 2024-09-11T09:52:14.286 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.6628000000000001e-05 2024-09-11T09:52:14.286 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.286 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.286 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:14.286 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.149572+0000", 2024-09-11T09:52:14.286 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.2956000000000001e-05 2024-09-11T09:52:14.286 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:14.286 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:14.287 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:14.287 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.287 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.287 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_repop(client.4694.0:45 2.6 e53/15 2:671fb830:ns::rbd_data.1256a0c02879.0000000000000007:head v 53'9, mlcod=53'7)", 2024-09-11T09:52:14.287 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.121833+0000", 2024-09-11T09:52:14.287 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 51.043082879000004, 2024-09-11T09:52:14.287 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.023876192000000001, 2024-09-11T09:52:14.287 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:14.287 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:14.287 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:14.287 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:14.287 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.287 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:14.288 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.121833+0000", 2024-09-11T09:52:14.288 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:14.288 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.288 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.288 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:14.288 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.121833+0000", 2024-09-11T09:52:14.288 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:14.288 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.288 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.288 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:14.288 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.121834+0000", 2024-09-11T09:52:14.288 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.471e-06 2024-09-11T09:52:14.288 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.289 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.289 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:14.289 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.128990+0000", 2024-09-11T09:52:14.289 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0071557879999999997 2024-09-11T09:52:14.289 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.289 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.289 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:14.289 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.129011+0000", 2024-09-11T09:52:14.289 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.0760000000000001e-05 2024-09-11T09:52:14.289 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.289 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.289 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:14.289 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.129022+0000", 2024-09-11T09:52:14.289 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1056000000000001e-05 2024-09-11T09:52:14.290 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.290 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.290 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:14.290 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.129097+0000", 2024-09-11T09:52:14.290 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.5082999999999999e-05 2024-09-11T09:52:14.290 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.290 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.290 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:14.290 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.129130+0000", 2024-09-11T09:52:14.290 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.3345e-05 2024-09-11T09:52:14.290 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.290 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.290 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:14.291 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.145657+0000", 2024-09-11T09:52:14.291 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.016527224 2024-09-11T09:52:14.291 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.291 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.291 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:14.291 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.145709+0000", 2024-09-11T09:52:14.291 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.1465000000000003e-05 2024-09-11T09:52:14.291 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:14.291 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:14.291 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:14.291 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.291 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.291 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4694.0:43 2.0 2:16c8ae84:ns::rbd_data.1256a0c02879.0000000000000005:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:14.292 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.135999+0000", 2024-09-11T09:52:14.292 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 51.028916928999998, 2024-09-11T09:52:14.292 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.039468309, 2024-09-11T09:52:14.292 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:14.292 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:14.292 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:14.292 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:52:14.292 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4694", 2024-09-11T09:52:14.292 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:52:14.292 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 43 2024-09-11T09:52:14.292 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.292 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:14.292 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.293 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:14.293 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.135999+0000", 2024-09-11T09:52:14.293 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:14.293 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.293 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.293 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:14.293 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.135999+0000", 2024-09-11T09:52:14.293 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:14.293 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.293 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.293 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:14.293 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.136000+0000", 2024-09-11T09:52:14.293 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.6330000000000001e-06 2024-09-11T09:52:14.293 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.294 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.294 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:14.294 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.157961+0000", 2024-09-11T09:52:14.294 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0219606 2024-09-11T09:52:14.294 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.294 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.294 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:14.294 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.157995+0000", 2024-09-11T09:52:14.294 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.4391999999999999e-05 2024-09-11T09:52:14.294 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.294 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.294 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:14.294 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.158011+0000", 2024-09-11T09:52:14.295 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.5794000000000002e-05 2024-09-11T09:52:14.295 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.295 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.295 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:14.295 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.158083+0000", 2024-09-11T09:52:14.295 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.2273000000000004e-05 2024-09-11T09:52:14.295 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.295 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.295 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:14.295 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.158239+0000", 2024-09-11T09:52:14.295 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000155372 2024-09-11T09:52:14.295 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.295 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.295 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 1", 2024-09-11T09:52:14.296 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.158403+0000", 2024-09-11T09:52:14.296 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00016408199999999999 2024-09-11T09:52:14.296 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.296 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.296 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:52:14.296 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.173860+0000", 2024-09-11T09:52:14.296 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.015457347 2024-09-11T09:52:14.296 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.296 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.296 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:14.296 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.175400+0000", 2024-09-11T09:52:14.296 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001540006 2024-09-11T09:52:14.297 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.297 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.297 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:14.297 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.175438+0000", 2024-09-11T09:52:14.297 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.8135e-05 2024-09-11T09:52:14.297 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.297 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.297 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:14.297 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.175467+0000", 2024-09-11T09:52:14.297 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.8674999999999999e-05 2024-09-11T09:52:14.297 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:14.297 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:14.297 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:14.297 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.298 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.298 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4694.0:57 2.0 2:04b5549c:ns::rbd_data.1256a0c02879.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:14.298 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.158037+0000", 2024-09-11T09:52:14.298 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 51.006878190999998, 2024-09-11T09:52:14.298 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.027748790999999998, 2024-09-11T09:52:14.298 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:14.298 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:14.298 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:14.298 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:52:14.298 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4694", 2024-09-11T09:52:14.298 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:52:14.298 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 57 2024-09-11T09:52:14.298 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.299 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:14.299 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.299 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:14.299 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.158037+0000", 2024-09-11T09:52:14.299 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:14.299 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.299 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.299 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:14.299 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.158037+0000", 2024-09-11T09:52:14.299 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:14.299 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.299 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.299 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:14.299 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.158039+0000", 2024-09-11T09:52:14.300 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.356e-06 2024-09-11T09:52:14.300 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.300 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.300 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:14.300 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.176367+0000", 2024-09-11T09:52:14.300 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.018328739 2024-09-11T09:52:14.300 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.300 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.300 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:14.300 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.176378+0000", 2024-09-11T09:52:14.300 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.0713999999999999e-05 2024-09-11T09:52:14.300 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.300 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.301 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:14.301 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.176383+0000", 2024-09-11T09:52:14.301 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.0209999999999999e-06 2024-09-11T09:52:14.301 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.301 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.301 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:14.301 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.176426+0000", 2024-09-11T09:52:14.301 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.2623999999999997e-05 2024-09-11T09:52:14.301 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.301 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.301 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:14.301 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.176552+0000", 2024-09-11T09:52:14.301 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000126076 2024-09-11T09:52:14.302 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.302 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.302 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 1", 2024-09-11T09:52:14.302 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.176665+0000", 2024-09-11T09:52:14.302 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000112995 2024-09-11T09:52:14.302 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.302 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.302 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:52:14.302 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.183639+0000", 2024-09-11T09:52:14.302 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0069740039999999998 2024-09-11T09:52:14.302 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.302 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.302 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:14.303 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.185689+0000", 2024-09-11T09:52:14.303 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.002049852 2024-09-11T09:52:14.303 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.303 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.303 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:14.303 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.185725+0000", 2024-09-11T09:52:14.303 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.6804999999999997e-05 2024-09-11T09:52:14.303 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.303 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.303 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:14.303 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.185786+0000", 2024-09-11T09:52:14.303 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.0605e-05 2024-09-11T09:52:14.303 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:14.304 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:14.304 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:14.304 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.304 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.304 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4694.0:60 2.0 2:054f2af4:ns::rbd_data.1256a0c02879.000000000000000f:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:14.304 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.186855+0000", 2024-09-11T09:52:14.304 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 50.978060175000003, 2024-09-11T09:52:14.304 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.013682913, 2024-09-11T09:52:14.304 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:14.304 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:14.304 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:14.304 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:52:14.304 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4694", 2024-09-11T09:52:14.304 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:52:14.305 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 60 2024-09-11T09:52:14.305 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.305 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:14.305 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.305 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:14.305 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.186855+0000", 2024-09-11T09:52:14.305 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:14.305 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.305 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.305 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:14.305 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.186855+0000", 2024-09-11T09:52:14.305 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:14.305 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.306 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.306 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:14.306 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.186856+0000", 2024-09-11T09:52:14.306 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.1200000000000001e-07 2024-09-11T09:52:14.306 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.306 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.306 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:14.306 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.191484+0000", 2024-09-11T09:52:14.306 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0046274159999999996 2024-09-11T09:52:14.306 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.306 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.306 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:14.306 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.191496+0000", 2024-09-11T09:52:14.306 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.2149e-05 2024-09-11T09:52:14.307 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.307 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.307 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:14.307 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.191502+0000", 2024-09-11T09:52:14.307 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.1890000000000004e-06 2024-09-11T09:52:14.307 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.307 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.307 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:14.307 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.191544+0000", 2024-09-11T09:52:14.307 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.1724999999999998e-05 2024-09-11T09:52:14.307 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.307 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.307 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:14.307 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.191641+0000", 2024-09-11T09:52:14.308 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.6939000000000001e-05 2024-09-11T09:52:14.308 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.308 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.308 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 1", 2024-09-11T09:52:14.308 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.191731+0000", 2024-09-11T09:52:14.308 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.0651999999999999e-05 2024-09-11T09:52:14.308 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.308 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.308 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:52:14.308 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.199852+0000", 2024-09-11T09:52:14.308 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0081204529999999997 2024-09-11T09:52:14.308 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.308 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.309 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:14.309 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.200461+0000", 2024-09-11T09:52:14.309 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00060900899999999998 2024-09-11T09:52:14.309 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.309 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.309 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:14.309 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.200497+0000", 2024-09-11T09:52:14.309 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.5800000000000003e-05 2024-09-11T09:52:14.309 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.309 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.309 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:14.309 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.200538+0000", 2024-09-11T09:52:14.309 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.1668999999999997e-05 2024-09-11T09:52:14.309 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:14.310 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:14.310 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:14.310 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:14.310 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:14.310 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-11T09:52:14.311 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-11T09:52:14.312 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:52:14.312+0000 7f3a7b90a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 42475) UID: 0 2024-09-11T09:52:14.358 INFO:tasks.workunit.client.0.smithi107.stderr:+ '[' -n eaab7410-3050-4b82-9298-7fac480d4de7 ']' 2024-09-11T09:52:14.358 INFO:tasks.workunit.client.0.smithi107.stderr:++ _sudo rbd device --device-type nbd --show-cookie --cookie 'abc de' map rbd/testrbdnbd27121 2024-09-11T09:52:14.359 INFO:tasks.workunit.client.0.smithi107.stderr:++ local cmd 2024-09-11T09:52:14.359 INFO:tasks.workunit.client.0.smithi107.stderr:+++ id -u 2024-09-11T09:52:14.359 INFO:tasks.workunit.client.0.smithi107.stderr:++ '[' 1000 -eq 0 ']' 2024-09-11T09:52:14.360 INFO:tasks.workunit.client.0.smithi107.stderr:+++ which rbd 2024-09-11T09:52:14.361 INFO:tasks.workunit.client.0.smithi107.stderr:++ cmd=/usr/bin/rbd 2024-09-11T09:52:14.362 INFO:tasks.workunit.client.0.smithi107.stderr:++ shift 2024-09-11T09:52:14.362 INFO:tasks.workunit.client.0.smithi107.stderr:++ sudo -nE /usr/bin/rbd device --device-type nbd --show-cookie --cookie 'abc de' map rbd/testrbdnbd27121 2024-09-11T09:52:14.412 INFO:tasks.ceph.osd.2.smithi017.stderr:2024-09-11T09:52:14.413+0000 7fd20b170640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31672) UID: 0 2024-09-11T09:52:14.456 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-11T09:52:14.456 INFO:teuthology.orchestra.run.smithi096.stdout: "ops": [], 2024-09-11T09:52:14.456 INFO:teuthology.orchestra.run.smithi096.stdout: "num_ops": 0 2024-09-11T09:52:14.456 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-11T09:52:14.465 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-11T09:52:14.474 INFO:tasks.workunit.client.0.smithi107.stderr:+ OUT='/dev/nbd1 abc de' 2024-09-11T09:52:14.474 INFO:tasks.workunit.client.0.smithi107.stderr:+ read DEV ANOTHER_COOKIE 2024-09-11T09:52:14.474 INFO:tasks.workunit.client.0.smithi107.stderr:+ get_pid rbd 2024-09-11T09:52:14.474 INFO:tasks.workunit.client.0.smithi107.stderr:+ local pool=rbd 2024-09-11T09:52:14.474 INFO:tasks.workunit.client.0.smithi107.stderr:+ local ns= 2024-09-11T09:52:14.474 INFO:tasks.workunit.client.0.smithi107.stderr:++ rbd device --device-type nbd --format xml list 2024-09-11T09:52:14.474 INFO:tasks.workunit.client.0.smithi107.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd27121'\''][device='\''/dev/nbd1'\'']/id' 2024-09-11T09:52:14.513 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:52:14.513+0000 7f3a7b90a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 42475) UID: 0 2024-09-11T09:52:14.563 INFO:tasks.workunit.client.0.smithi107.stderr:+ PID=29922 2024-09-11T09:52:14.563 INFO:tasks.workunit.client.0.smithi107.stderr:+ test -n 29922 2024-09-11T09:52:14.563 INFO:tasks.workunit.client.0.smithi107.stderr:+ ps -p 29922 -C rbd-nbd 2024-09-11T09:52:14.567 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-11T09:52:14.567 INFO:teuthology.orchestra.run.smithi096.stdout: "ops": [], 2024-09-11T09:52:14.567 INFO:teuthology.orchestra.run.smithi096.stdout: "complaint_time": 30, 2024-09-11T09:52:14.567 INFO:teuthology.orchestra.run.smithi096.stdout: "num_blocked_ops": 0 2024-09-11T09:52:14.567 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-11T09:52:14.574 INFO:tasks.workunit.client.0.smithi107.stdout: PID TTY TIME CMD 2024-09-11T09:52:14.574 INFO:tasks.workunit.client.0.smithi107.stdout: 29922 ? 00:00:00 rbd-nbd 2024-09-11T09:52:14.575 INFO:tasks.workunit.client.0.smithi107.stderr:+ test 'abc de' = 'abc de' 2024-09-11T09:52:14.575 INFO:tasks.workunit.client.0.smithi107.stderr:+ unmap_device /dev/nbd1 29922 2024-09-11T09:52:14.575 INFO:tasks.workunit.client.0.smithi107.stderr:+ local args=/dev/nbd1 2024-09-11T09:52:14.575 INFO:tasks.workunit.client.0.smithi107.stderr:+ local pid=29922 2024-09-11T09:52:14.575 INFO:tasks.workunit.client.0.smithi107.stderr:+ _sudo rbd device --device-type nbd unmap /dev/nbd1 2024-09-11T09:52:14.575 INFO:tasks.workunit.client.0.smithi107.stderr:+ local cmd 2024-09-11T09:52:14.575 INFO:tasks.workunit.client.0.smithi107.stderr:++ id -u 2024-09-11T09:52:14.576 INFO:tasks.workunit.client.0.smithi107.stderr:+ '[' 1000 -eq 0 ']' 2024-09-11T09:52:14.576 INFO:tasks.workunit.client.0.smithi107.stderr:++ which rbd 2024-09-11T09:52:14.577 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-11T09:52:14.577 INFO:tasks.workunit.client.0.smithi107.stderr:+ cmd=/usr/bin/rbd 2024-09-11T09:52:14.577 INFO:tasks.workunit.client.0.smithi107.stderr:+ shift 2024-09-11T09:52:14.578 INFO:tasks.workunit.client.0.smithi107.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd unmap /dev/nbd1 2024-09-11T09:52:14.614 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:52:14.615+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:52:14.681 INFO:tasks.workunit.client.0.smithi107.stderr:+ rbd device --device-type nbd list 2024-09-11T09:52:14.681 INFO:tasks.workunit.client.0.smithi107.stderr:+ expect_false grep '^29922\b' 2024-09-11T09:52:14.681 INFO:tasks.workunit.client.0.smithi107.stderr:+ grep '^29922\b' 2024-09-11T09:52:14.715 INFO:tasks.ceph.osd.0.smithi017.stderr:2024-09-11T09:52:14.715+0000 7fb73c0f1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31665) UID: 0 2024-09-11T09:52:14.722 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-11T09:52:14.722 INFO:teuthology.orchestra.run.smithi096.stdout: "size": 20, 2024-09-11T09:52:14.722 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 600, 2024-09-11T09:52:14.722 INFO:teuthology.orchestra.run.smithi096.stdout: "ops": [ 2024-09-11T09:52:14.722 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.723 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4230.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-11T09:52:14.723 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:18.490736+0000", 2024-09-11T09:52:14.723 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 116.231212689, 2024-09-11T09:52:14.723 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0059097569999999999, 2024-09-11T09:52:14.723 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:14.723 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:14.723 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:14.723 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:52:14.723 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4230", 2024-09-11T09:52:14.723 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.17:0/4101109576", 2024-09-11T09:52:14.723 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 1 2024-09-11T09:52:14.723 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.723 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:14.724 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.724 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:14.724 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.490736+0000", 2024-09-11T09:52:14.724 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:14.724 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.724 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.724 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:14.724 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.490736+0000", 2024-09-11T09:52:14.724 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:14.724 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.724 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.724 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:14.724 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.490742+0000", 2024-09-11T09:52:14.724 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.48e-06 2024-09-11T09:52:14.725 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.725 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.725 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:14.725 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.490749+0000", 2024-09-11T09:52:14.725 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.4569999999999999e-06 2024-09-11T09:52:14.725 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.725 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.725 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:14.726 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.490762+0000", 2024-09-11T09:52:14.726 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.2721000000000001e-05 2024-09-11T09:52:14.726 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.726 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.726 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:14.726 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.490774+0000", 2024-09-11T09:52:14.726 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1823e-05 2024-09-11T09:52:14.726 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.726 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.726 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:14.726 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.490846+0000", 2024-09-11T09:52:14.726 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.1878000000000005e-05 2024-09-11T09:52:14.726 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.727 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.727 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for peered", 2024-09-11T09:52:14.727 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.490857+0000", 2024-09-11T09:52:14.727 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1017e-05 2024-09-11T09:52:14.727 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.727 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.727 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:14.727 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.493856+0000", 2024-09-11T09:52:14.727 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00299885 2024-09-11T09:52:14.727 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.727 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.727 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:14.727 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.494020+0000", 2024-09-11T09:52:14.728 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00016436599999999999 2024-09-11T09:52:14.728 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.728 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.728 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 2", 2024-09-11T09:52:14.728 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.494186+0000", 2024-09-11T09:52:14.728 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00016562799999999999 2024-09-11T09:52:14.728 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.728 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.728 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:14.728 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.495407+0000", 2024-09-11T09:52:14.728 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0012217420000000001 2024-09-11T09:52:14.728 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.728 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.729 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:52:14.729 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.496579+0000", 2024-09-11T09:52:14.729 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001171844 2024-09-11T09:52:14.729 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.729 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.729 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:14.729 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.496620+0000", 2024-09-11T09:52:14.729 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.0479000000000003e-05 2024-09-11T09:52:14.729 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.729 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.729 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:14.729 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.496646+0000", 2024-09-11T09:52:14.729 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.6472e-05 2024-09-11T09:52:14.729 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:14.730 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:14.730 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:14.730 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.730 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.730 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4230.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-09-11T09:52:14.730 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:20.498634+0000", 2024-09-11T09:52:14.730 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 114.223314668, 2024-09-11T09:52:14.730 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0050648539999999997, 2024-09-11T09:52:14.730 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:14.730 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:14.730 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:14.730 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:52:14.730 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4230", 2024-09-11T09:52:14.731 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.17:0/4101109576", 2024-09-11T09:52:14.731 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 7 2024-09-11T09:52:14.731 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.731 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:14.731 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.731 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:14.731 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.498634+0000", 2024-09-11T09:52:14.731 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:14.731 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.731 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.731 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:14.731 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.498634+0000", 2024-09-11T09:52:14.731 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:14.731 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.732 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.732 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:14.732 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.498637+0000", 2024-09-11T09:52:14.732 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.1919999999999999e-06 2024-09-11T09:52:14.732 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.732 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.732 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:14.732 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.498648+0000", 2024-09-11T09:52:14.732 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1116e-05 2024-09-11T09:52:14.732 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.732 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.732 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:14.732 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.498666+0000", 2024-09-11T09:52:14.733 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.8411999999999999e-05 2024-09-11T09:52:14.733 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.733 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.733 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:14.733 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.498675+0000", 2024-09-11T09:52:14.733 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.4059999999999994e-06 2024-09-11T09:52:14.733 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.733 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.733 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:14.733 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.498745+0000", 2024-09-11T09:52:14.733 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.9764000000000005e-05 2024-09-11T09:52:14.733 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.733 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.734 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op must wait for map", 2024-09-11T09:52:14.734 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.498754+0000", 2024-09-11T09:52:14.734 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.5969999999999992e-06 2024-09-11T09:52:14.734 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.734 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.734 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:14.734 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.500385+0000", 2024-09-11T09:52:14.734 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0016308360000000001 2024-09-11T09:52:14.734 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.734 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.734 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:14.734 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.500473+0000", 2024-09-11T09:52:14.734 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.7966000000000004e-05 2024-09-11T09:52:14.734 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.735 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.735 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 1", 2024-09-11T09:52:14.735 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.500591+0000", 2024-09-11T09:52:14.735 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000117895 2024-09-11T09:52:14.735 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.735 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.735 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:14.735 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.501685+0000", 2024-09-11T09:52:14.735 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001094886 2024-09-11T09:52:14.735 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.735 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.735 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:52:14.735 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.503627+0000", 2024-09-11T09:52:14.736 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001941199 2024-09-11T09:52:14.736 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.736 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.736 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:14.736 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.503667+0000", 2024-09-11T09:52:14.736 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.0645000000000001e-05 2024-09-11T09:52:14.736 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.736 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.736 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:14.736 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.503699+0000", 2024-09-11T09:52:14.736 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.1940000000000003e-05 2024-09-11T09:52:14.736 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:14.736 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:14.736 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:14.737 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.737 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.737 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4688.0:32 2.2 2:45941fd2:::rbd_data.1250f5e25b80.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:14.737 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:22.896607+0000", 2024-09-11T09:52:14.737 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 51.825342065000001, 2024-09-11T09:52:14.737 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.038655179999999997, 2024-09-11T09:52:14.737 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:14.737 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:14.737 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:14.737 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:52:14.737 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4688", 2024-09-11T09:52:14.737 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:52:14.738 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 32 2024-09-11T09:52:14.738 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.738 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:14.738 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.738 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:14.738 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.896607+0000", 2024-09-11T09:52:14.738 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:14.738 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.738 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.738 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:14.738 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.896607+0000", 2024-09-11T09:52:14.738 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:14.738 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.739 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.739 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:14.739 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.896608+0000", 2024-09-11T09:52:14.739 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.0690000000000001e-06 2024-09-11T09:52:14.739 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.739 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.739 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:14.739 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.914461+0000", 2024-09-11T09:52:14.739 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.017852903 2024-09-11T09:52:14.739 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.739 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.739 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:14.739 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.914495+0000", 2024-09-11T09:52:14.740 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.4301000000000002e-05 2024-09-11T09:52:14.740 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.740 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.740 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:14.740 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.914522+0000", 2024-09-11T09:52:14.740 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.7127e-05 2024-09-11T09:52:14.740 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.740 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.740 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:14.740 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.914583+0000", 2024-09-11T09:52:14.740 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.0730999999999998e-05 2024-09-11T09:52:14.740 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.740 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.741 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:14.741 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.914709+0000", 2024-09-11T09:52:14.741 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00012590400000000001 2024-09-11T09:52:14.741 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.741 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.741 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 1", 2024-09-11T09:52:14.741 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.914809+0000", 2024-09-11T09:52:14.741 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.9882000000000004e-05 2024-09-11T09:52:14.741 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.741 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.741 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:14.741 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.932551+0000", 2024-09-11T09:52:14.741 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.017742048999999999 2024-09-11T09:52:14.742 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.742 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.742 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:52:14.742 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.935144+0000", 2024-09-11T09:52:14.742 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0025927630000000001 2024-09-11T09:52:14.742 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.742 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.742 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:14.742 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.935205+0000", 2024-09-11T09:52:14.742 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.1634e-05 2024-09-11T09:52:14.742 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.742 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.742 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:14.743 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.935262+0000", 2024-09-11T09:52:14.743 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.6817000000000002e-05 2024-09-11T09:52:14.743 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:14.743 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:14.743 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:14.743 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.743 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.743 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_repop(client.4688.0:40 2.7 e53/38 2:fe41411c:::rbd_object_map.1250f5e25b80:head v 53'9, mlcod=53'7)", 2024-09-11T09:52:14.743 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:22.920457+0000", 2024-09-11T09:52:14.743 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 51.801491974000001, 2024-09-11T09:52:14.743 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.011132817, 2024-09-11T09:52:14.743 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:14.743 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:14.744 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:14.744 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:14.744 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.744 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:14.744 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.920457+0000", 2024-09-11T09:52:14.744 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:14.744 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.744 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.744 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:14.744 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.920457+0000", 2024-09-11T09:52:14.744 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:14.744 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.744 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.745 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:14.745 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.920459+0000", 2024-09-11T09:52:14.745 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.6729999999999999e-06 2024-09-11T09:52:14.745 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.745 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.745 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:14.745 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.920469+0000", 2024-09-11T09:52:14.745 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.944e-06 2024-09-11T09:52:14.745 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.745 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.745 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:14.745 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.920495+0000", 2024-09-11T09:52:14.745 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.6401999999999999e-05 2024-09-11T09:52:14.746 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.746 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.746 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:14.746 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.920506+0000", 2024-09-11T09:52:14.746 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.133e-05 2024-09-11T09:52:14.746 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.746 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.746 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:14.746 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.920581+0000", 2024-09-11T09:52:14.746 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.4932999999999996e-05 2024-09-11T09:52:14.746 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.746 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.746 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:14.746 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.920612+0000", 2024-09-11T09:52:14.747 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.0537999999999998e-05 2024-09-11T09:52:14.747 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.747 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.747 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:14.747 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.931537+0000", 2024-09-11T09:52:14.747 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.010925239 2024-09-11T09:52:14.747 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.747 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.747 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:14.747 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.931590+0000", 2024-09-11T09:52:14.747 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.2757999999999999e-05 2024-09-11T09:52:14.747 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:14.747 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:14.748 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:14.748 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.748 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.748 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_repop(client.4688.0:41 2.7 e53/38 2:fe41411c:::rbd_object_map.1250f5e25b80:head v 53'10, mlcod=53'7)", 2024-09-11T09:52:14.748 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:22.921422+0000", 2024-09-11T09:52:14.748 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 51.800526558000001, 2024-09-11T09:52:14.748 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.013734942, 2024-09-11T09:52:14.748 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:14.748 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:14.748 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:14.748 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:14.748 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.748 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:14.749 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921422+0000", 2024-09-11T09:52:14.749 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:14.749 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.749 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.749 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:14.749 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921422+0000", 2024-09-11T09:52:14.749 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:14.749 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.749 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.749 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:14.749 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921426+0000", 2024-09-11T09:52:14.749 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.6150000000000001e-06 2024-09-11T09:52:14.749 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.750 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.750 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:14.750 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921435+0000", 2024-09-11T09:52:14.750 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.9840000000000007e-06 2024-09-11T09:52:14.750 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.750 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.750 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:14.750 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921444+0000", 2024-09-11T09:52:14.750 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.6000000000000007e-06 2024-09-11T09:52:14.750 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.750 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.750 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:14.750 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921450+0000", 2024-09-11T09:52:14.750 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.4690000000000002e-06 2024-09-11T09:52:14.751 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.751 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.751 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:14.751 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921511+0000", 2024-09-11T09:52:14.751 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.0779999999999997e-05 2024-09-11T09:52:14.751 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.751 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.751 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:14.751 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921531+0000", 2024-09-11T09:52:14.751 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.0302999999999999e-05 2024-09-11T09:52:14.751 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.751 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.751 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:14.752 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.935115+0000", 2024-09-11T09:52:14.752 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.013583471 2024-09-11T09:52:14.752 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.752 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.752 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:14.752 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.935157+0000", 2024-09-11T09:52:14.752 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.2719999999999998e-05 2024-09-11T09:52:14.752 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:14.752 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:14.752 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:14.752 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.752 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.752 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4688.0:42 2.3 2:c54742d8:::rbd_data.1250f5e25b80.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:14.753 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:22.932560+0000", 2024-09-11T09:52:14.753 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 51.789388815000002, 2024-09-11T09:52:14.753 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.017798198000000001, 2024-09-11T09:52:14.753 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:14.753 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:14.753 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:14.753 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:52:14.753 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4688", 2024-09-11T09:52:14.753 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:52:14.753 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 42 2024-09-11T09:52:14.753 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.753 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:14.753 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.753 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:14.754 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.932560+0000", 2024-09-11T09:52:14.754 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:14.754 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.754 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.754 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:14.754 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.932560+0000", 2024-09-11T09:52:14.754 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:14.754 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.754 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.754 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:14.754 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.932562+0000", 2024-09-11T09:52:14.754 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.17e-06 2024-09-11T09:52:14.755 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.755 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.755 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:14.755 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.938851+0000", 2024-09-11T09:52:14.755 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.006288971 2024-09-11T09:52:14.755 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.755 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.755 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:14.755 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.938865+0000", 2024-09-11T09:52:14.755 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.365e-05 2024-09-11T09:52:14.755 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.755 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.755 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:14.756 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.938871+0000", 2024-09-11T09:52:14.756 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.8850000000000004e-06 2024-09-11T09:52:14.756 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.756 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.756 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:14.756 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.938926+0000", 2024-09-11T09:52:14.756 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.4691999999999997e-05 2024-09-11T09:52:14.756 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.756 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.756 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:14.756 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.939059+0000", 2024-09-11T09:52:14.756 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00013310899999999999 2024-09-11T09:52:14.757 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.757 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.757 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 2", 2024-09-11T09:52:14.757 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.939171+0000", 2024-09-11T09:52:14.757 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000112718 2024-09-11T09:52:14.757 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.757 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.757 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:14.757 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.950262+0000", 2024-09-11T09:52:14.757 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.011091063 2024-09-11T09:52:14.757 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.757 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.757 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:52:14.758 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.950274+0000", 2024-09-11T09:52:14.758 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1963e-05 2024-09-11T09:52:14.758 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.758 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.758 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:14.758 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.950318+0000", 2024-09-11T09:52:14.758 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.3519999999999997e-05 2024-09-11T09:52:14.758 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.758 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.758 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:14.758 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.950358+0000", 2024-09-11T09:52:14.758 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.0457000000000002e-05 2024-09-11T09:52:14.758 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:14.758 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:14.759 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:14.759 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.759 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.759 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_repop(client.4694.0:34 2.7 e53/38 2:f80bfc0c:ns::rbd_data.1256a0c02879.0000000000000002:head v 53'25, mlcod=53'23)", 2024-09-11T09:52:14.759 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.094944+0000", 2024-09-11T09:52:14.759 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 51.627005423, 2024-09-11T09:52:14.759 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.010896785000000001, 2024-09-11T09:52:14.759 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:14.759 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:14.759 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:14.759 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:14.759 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.759 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:14.759 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.094944+0000", 2024-09-11T09:52:14.760 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:14.760 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.760 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.760 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:14.760 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.094944+0000", 2024-09-11T09:52:14.760 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:14.760 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.760 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.760 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:14.760 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.094944+0000", 2024-09-11T09:52:14.760 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.0699999999999996e-07 2024-09-11T09:52:14.760 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.760 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.761 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:14.761 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.099060+0000", 2024-09-11T09:52:14.761 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0041155580000000001 2024-09-11T09:52:14.761 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.761 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.761 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:14.761 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.099082+0000", 2024-09-11T09:52:14.761 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.2323999999999998e-05 2024-09-11T09:52:14.761 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.761 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.761 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:14.761 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.099094+0000", 2024-09-11T09:52:14.761 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1535000000000001e-05 2024-09-11T09:52:14.762 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.762 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.762 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:14.762 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.099160+0000", 2024-09-11T09:52:14.762 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.6449e-05 2024-09-11T09:52:14.762 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.762 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.762 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:14.762 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.099183+0000", 2024-09-11T09:52:14.762 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.3108e-05 2024-09-11T09:52:14.762 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.762 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.762 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:14.762 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.105776+0000", 2024-09-11T09:52:14.763 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0065922760000000002 2024-09-11T09:52:14.763 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.763 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.763 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:14.763 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.105840+0000", 2024-09-11T09:52:14.763 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.4727999999999994e-05 2024-09-11T09:52:14.763 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:14.763 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:14.763 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:14.763 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.763 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.763 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_repop(client.4694.0:35 2.7 e53/38 2:e8766325:ns::rbd_object_map.1256a0c02879:head v 53'26, mlcod=53'23)", 2024-09-11T09:52:14.763 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.100964+0000", 2024-09-11T09:52:14.764 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 51.620984894999999, 2024-09-11T09:52:14.764 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0049649359999999997, 2024-09-11T09:52:14.764 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:14.764 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:14.764 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:14.764 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:14.764 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.764 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:14.764 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.100964+0000", 2024-09-11T09:52:14.764 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:14.764 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.764 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.764 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:14.764 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.100964+0000", 2024-09-11T09:52:14.765 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:14.765 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.765 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.765 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:14.765 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.100966+0000", 2024-09-11T09:52:14.765 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.6330000000000001e-06 2024-09-11T09:52:14.765 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.765 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.765 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:14.765 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.100979+0000", 2024-09-11T09:52:14.765 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.2813000000000001e-05 2024-09-11T09:52:14.765 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.765 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.766 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:14.766 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.100992+0000", 2024-09-11T09:52:14.766 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.3414999999999999e-05 2024-09-11T09:52:14.766 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.766 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.766 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:14.766 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.101002+0000", 2024-09-11T09:52:14.766 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.0159e-05 2024-09-11T09:52:14.766 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.766 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.766 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:14.766 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.102365+0000", 2024-09-11T09:52:14.766 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001362912 2024-09-11T09:52:14.766 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.767 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.767 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:14.767 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.102382+0000", 2024-09-11T09:52:14.767 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.7433000000000001e-05 2024-09-11T09:52:14.767 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.767 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.767 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:14.767 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.105901+0000", 2024-09-11T09:52:14.767 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.003518476 2024-09-11T09:52:14.767 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.767 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.767 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:14.767 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.105929+0000", 2024-09-11T09:52:14.767 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.8095000000000001e-05 2024-09-11T09:52:14.768 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:14.768 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:14.768 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:14.768 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.768 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.768 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4694.0:41 2.2 2:5e2505c4:ns::rbd_data.1256a0c02879.0000000000000003:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:14.768 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.107511+0000", 2024-09-11T09:52:14.768 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 51.614437742, 2024-09-11T09:52:14.768 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.018571066000000001, 2024-09-11T09:52:14.768 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:14.768 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:14.768 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:14.769 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:52:14.769 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4694", 2024-09-11T09:52:14.769 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:52:14.769 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 41 2024-09-11T09:52:14.769 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.769 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:14.769 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.769 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:14.769 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.107511+0000", 2024-09-11T09:52:14.769 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:14.769 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.769 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.769 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:14.770 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.107511+0000", 2024-09-11T09:52:14.770 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:14.770 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.770 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.770 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:14.770 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.107513+0000", 2024-09-11T09:52:14.770 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.3489999999999999e-06 2024-09-11T09:52:14.770 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.770 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.770 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:14.770 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.112704+0000", 2024-09-11T09:52:14.770 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0051918509999999999 2024-09-11T09:52:14.770 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.770 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.771 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:14.771 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.112718+0000", 2024-09-11T09:52:14.771 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.359e-05 2024-09-11T09:52:14.771 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.771 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.771 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:14.771 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.112725+0000", 2024-09-11T09:52:14.771 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.9940000000000003e-06 2024-09-11T09:52:14.771 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.771 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.771 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:14.771 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.112788+0000", 2024-09-11T09:52:14.771 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.2447999999999996e-05 2024-09-11T09:52:14.772 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.772 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.772 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:14.772 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.112937+0000", 2024-09-11T09:52:14.772 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00014921499999999999 2024-09-11T09:52:14.772 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.772 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.772 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 1", 2024-09-11T09:52:14.772 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.113085+0000", 2024-09-11T09:52:14.772 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000148085 2024-09-11T09:52:14.772 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.772 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.772 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:52:14.773 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.124318+0000", 2024-09-11T09:52:14.773 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.011233158 2024-09-11T09:52:14.773 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.773 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.773 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:14.773 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.125957+0000", 2024-09-11T09:52:14.773 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001638701 2024-09-11T09:52:14.773 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.773 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.773 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:14.773 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.126029+0000", 2024-09-11T09:52:14.773 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.2125000000000005e-05 2024-09-11T09:52:14.773 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.773 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.774 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:14.774 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.126082+0000", 2024-09-11T09:52:14.774 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.3550000000000001e-05 2024-09-11T09:52:14.774 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:14.774 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:14.774 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:14.774 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.774 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.774 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4694.0:44 2.3 2:cd1a7d1c:ns::rbd_data.1256a0c02879.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:14.774 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.112744+0000", 2024-09-11T09:52:14.774 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 51.609205175, 2024-09-11T09:52:14.774 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.021672212999999999, 2024-09-11T09:52:14.774 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:14.775 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:14.775 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:14.775 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:52:14.775 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4694", 2024-09-11T09:52:14.775 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:52:14.775 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 44 2024-09-11T09:52:14.775 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.775 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:14.775 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.775 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:14.775 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.112744+0000", 2024-09-11T09:52:14.775 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:14.775 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.775 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.776 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:14.776 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.112744+0000", 2024-09-11T09:52:14.776 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:14.776 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.776 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.776 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:14.776 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.112745+0000", 2024-09-11T09:52:14.776 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.2900000000000002e-07 2024-09-11T09:52:14.776 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.776 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.776 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:14.776 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.119929+0000", 2024-09-11T09:52:14.776 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0071846230000000002 2024-09-11T09:52:14.777 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.777 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.777 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:14.779 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.119949+0000", 2024-09-11T09:52:14.779 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.9264000000000001e-05 2024-09-11T09:52:14.780 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.780 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.780 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:14.780 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.119959+0000", 2024-09-11T09:52:14.780 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.0465999999999999e-05 2024-09-11T09:52:14.780 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.780 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.780 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:14.780 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.120028+0000", 2024-09-11T09:52:14.780 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.9318000000000003e-05 2024-09-11T09:52:14.780 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.780 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.780 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:14.781 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.120160+0000", 2024-09-11T09:52:14.781 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000132015 2024-09-11T09:52:14.781 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.781 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.781 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 2", 2024-09-11T09:52:14.781 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.120260+0000", 2024-09-11T09:52:14.781 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.9397999999999999e-05 2024-09-11T09:52:14.781 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.781 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.781 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:52:14.781 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.134246+0000", 2024-09-11T09:52:14.781 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.013986721000000001 2024-09-11T09:52:14.781 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.781 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.782 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:14.782 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.134295+0000", 2024-09-11T09:52:14.782 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.8649000000000001e-05 2024-09-11T09:52:14.782 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.782 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.782 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:14.782 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.134361+0000", 2024-09-11T09:52:14.782 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.6081999999999995e-05 2024-09-11T09:52:14.782 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.782 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.782 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:14.782 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.134416+0000", 2024-09-11T09:52:14.782 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.4848000000000001e-05 2024-09-11T09:52:14.783 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:14.783 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:14.783 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:14.783 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.783 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.783 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4694.0:46 2.3 2:ce0e1469:ns::rbd_data.1256a0c02879.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:14.783 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.119993+0000", 2024-09-11T09:52:14.783 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 51.601955699000001, 2024-09-11T09:52:14.783 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.025840610999999999, 2024-09-11T09:52:14.783 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:14.783 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:14.783 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:14.783 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:52:14.784 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4694", 2024-09-11T09:52:14.784 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:52:14.784 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 46 2024-09-11T09:52:14.784 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.784 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:14.784 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.784 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:14.784 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.119993+0000", 2024-09-11T09:52:14.784 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:14.784 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.784 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.784 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:14.784 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.119993+0000", 2024-09-11T09:52:14.785 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:14.785 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.785 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.785 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:14.785 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.119995+0000", 2024-09-11T09:52:14.785 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1930000000000001e-06 2024-09-11T09:52:14.785 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.785 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.785 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:14.785 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.128458+0000", 2024-09-11T09:52:14.785 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0084632630000000004 2024-09-11T09:52:14.785 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.786 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.786 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:14.786 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.128478+0000", 2024-09-11T09:52:14.786 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.0353e-05 2024-09-11T09:52:14.786 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.786 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.786 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:14.786 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.128491+0000", 2024-09-11T09:52:14.786 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.2954e-05 2024-09-11T09:52:14.786 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.786 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.786 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:14.786 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.128558+0000", 2024-09-11T09:52:14.786 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.7076999999999995e-05 2024-09-11T09:52:14.787 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.787 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.787 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:14.787 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.128708+0000", 2024-09-11T09:52:14.787 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00014939599999999999 2024-09-11T09:52:14.787 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.787 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.787 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 2", 2024-09-11T09:52:14.787 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.128866+0000", 2024-09-11T09:52:14.787 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000158145 2024-09-11T09:52:14.787 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.787 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.787 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:14.788 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.141045+0000", 2024-09-11T09:52:14.788 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.012179331999999999 2024-09-11T09:52:14.788 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.788 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.788 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:52:14.788 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.145742+0000", 2024-09-11T09:52:14.788 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0046972050000000003 2024-09-11T09:52:14.788 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.788 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.788 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:14.788 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.145779+0000", 2024-09-11T09:52:14.788 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.6544000000000001e-05 2024-09-11T09:52:14.788 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.789 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.789 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:14.789 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.145834+0000", 2024-09-11T09:52:14.789 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.5148999999999997e-05 2024-09-11T09:52:14.789 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:14.789 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:14.789 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:14.789 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.789 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.789 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4694.0:48 2.3 2:d2090b70:ns::rbd_data.1256a0c02879.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:14.789 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.130540+0000", 2024-09-11T09:52:14.789 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 51.591409433000003, 2024-09-11T09:52:14.789 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.017167789999999999, 2024-09-11T09:52:14.790 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:14.790 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:14.790 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:14.790 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:52:14.790 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4694", 2024-09-11T09:52:14.790 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:52:14.790 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 48 2024-09-11T09:52:14.790 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.790 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:14.790 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.790 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:14.790 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.130540+0000", 2024-09-11T09:52:14.790 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:14.791 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.791 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.791 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:14.791 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.130540+0000", 2024-09-11T09:52:14.791 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:14.791 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.791 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.791 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:14.791 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.130542+0000", 2024-09-11T09:52:14.791 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.9209999999999999e-06 2024-09-11T09:52:14.791 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.791 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.791 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:14.791 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.134691+0000", 2024-09-11T09:52:14.792 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0041498739999999996 2024-09-11T09:52:14.792 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.792 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.792 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:14.792 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.134705+0000", 2024-09-11T09:52:14.792 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.3669000000000001e-05 2024-09-11T09:52:14.792 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.792 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.792 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:14.792 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.134712+0000", 2024-09-11T09:52:14.792 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.8410000000000002e-06 2024-09-11T09:52:14.792 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.792 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.793 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:14.793 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.134754+0000", 2024-09-11T09:52:14.793 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.2524999999999997e-05 2024-09-11T09:52:14.793 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.793 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.793 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:14.793 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.134875+0000", 2024-09-11T09:52:14.793 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000120332 2024-09-11T09:52:14.793 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.793 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.793 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 2", 2024-09-11T09:52:14.793 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.135005+0000", 2024-09-11T09:52:14.793 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00013024900000000001 2024-09-11T09:52:14.794 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.794 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.794 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:14.794 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.145971+0000", 2024-09-11T09:52:14.794 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.010966203000000001 2024-09-11T09:52:14.794 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.794 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.794 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:52:14.794 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.147600+0000", 2024-09-11T09:52:14.794 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001628382 2024-09-11T09:52:14.794 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.794 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.794 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:14.794 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.147635+0000", 2024-09-11T09:52:14.795 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.5420000000000003e-05 2024-09-11T09:52:14.795 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.795 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.795 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:14.795 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.147707+0000", 2024-09-11T09:52:14.795 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.2373999999999995e-05 2024-09-11T09:52:14.795 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:14.795 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:14.795 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:14.795 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.795 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.795 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_repop(client.4694.0:49 2.7 e53/38 2:e8766325:ns::rbd_object_map.1256a0c02879:head v 53'33, mlcod=53'31)", 2024-09-11T09:52:14.795 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.135391+0000", 2024-09-11T09:52:14.796 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 51.586557868, 2024-09-11T09:52:14.796 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0051189859999999998, 2024-09-11T09:52:14.796 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:14.796 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:14.796 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:14.796 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:14.796 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.796 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:14.796 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.135391+0000", 2024-09-11T09:52:14.796 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:14.796 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.796 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.796 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:14.797 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.135391+0000", 2024-09-11T09:52:14.797 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:14.797 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.797 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.797 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:14.797 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.135393+0000", 2024-09-11T09:52:14.797 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.4020000000000001e-06 2024-09-11T09:52:14.797 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.797 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.797 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:14.797 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.135400+0000", 2024-09-11T09:52:14.797 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.5129999999999999e-06 2024-09-11T09:52:14.797 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.798 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.798 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:14.798 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.135411+0000", 2024-09-11T09:52:14.798 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1038e-05 2024-09-11T09:52:14.798 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.798 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.798 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:14.798 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.135418+0000", 2024-09-11T09:52:14.798 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.1489999999999997e-06 2024-09-11T09:52:14.798 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.798 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.798 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:14.799 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.135470+0000", 2024-09-11T09:52:14.799 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.2021999999999998e-05 2024-09-11T09:52:14.799 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.799 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.799 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:14.799 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.135501+0000", 2024-09-11T09:52:14.799 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.0349999999999999e-05 2024-09-11T09:52:14.799 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.799 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.799 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:14.799 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.140453+0000", 2024-09-11T09:52:14.799 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0049522409999999996 2024-09-11T09:52:14.799 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.800 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.800 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:14.800 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.140510+0000", 2024-09-11T09:52:14.800 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.7271000000000001e-05 2024-09-11T09:52:14.800 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:14.800 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:14.800 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:14.800 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.800 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.800 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4694.0:54 2.3 2:c44f0253:ns::rbd_data.1256a0c02879.000000000000000b:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:14.800 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.150151+0000", 2024-09-11T09:52:14.800 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 51.571798016999999, 2024-09-11T09:52:14.800 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.015103484, 2024-09-11T09:52:14.801 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:14.801 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:14.801 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:14.801 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:52:14.801 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4694", 2024-09-11T09:52:14.801 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:52:14.801 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 54 2024-09-11T09:52:14.801 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.801 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:14.801 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.801 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:14.801 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.150151+0000", 2024-09-11T09:52:14.801 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:14.802 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.802 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.802 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:14.802 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.150151+0000", 2024-09-11T09:52:14.802 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:14.802 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.802 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.802 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:14.802 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.150152+0000", 2024-09-11T09:52:14.802 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.2550000000000001e-06 2024-09-11T09:52:14.802 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.802 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.802 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:14.803 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.154528+0000", 2024-09-11T09:52:14.803 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0043758219999999997 2024-09-11T09:52:14.803 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.803 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.803 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:14.803 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.154541+0000", 2024-09-11T09:52:14.803 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.3414999999999999e-05 2024-09-11T09:52:14.803 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.803 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.803 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:14.803 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.154551+0000", 2024-09-11T09:52:14.803 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.4350000000000003e-06 2024-09-11T09:52:14.803 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.804 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.804 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:14.804 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.154615+0000", 2024-09-11T09:52:14.804 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.3627000000000006e-05 2024-09-11T09:52:14.804 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.804 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.804 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:14.804 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.154717+0000", 2024-09-11T09:52:14.804 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000102729 2024-09-11T09:52:14.804 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.804 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.804 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 2", 2024-09-11T09:52:14.804 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.154826+0000", 2024-09-11T09:52:14.805 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000108409 2024-09-11T09:52:14.805 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.805 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.805 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:52:14.805 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.165094+0000", 2024-09-11T09:52:14.805 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.010268210999999999 2024-09-11T09:52:14.805 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.805 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.805 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:14.805 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.165138+0000", 2024-09-11T09:52:14.805 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.4078e-05 2024-09-11T09:52:14.805 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.805 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.805 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:14.806 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.165204+0000", 2024-09-11T09:52:14.806 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.5716000000000006e-05 2024-09-11T09:52:14.806 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.806 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.806 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:14.806 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.165254+0000", 2024-09-11T09:52:14.806 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.0787e-05 2024-09-11T09:52:14.806 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:14.806 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:14.806 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:14.806 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.806 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.806 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_repop(client.4694.0:55 2.7 e53/38 2:e8766325:ns::rbd_object_map.1256a0c02879:head v 53'37, mlcod=53'34)", 2024-09-11T09:52:14.807 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.150221+0000", 2024-09-11T09:52:14.807 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 51.571728151999999, 2024-09-11T09:52:14.807 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00530015, 2024-09-11T09:52:14.807 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:14.807 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:14.807 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:14.807 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:14.807 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.807 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:14.807 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.150221+0000", 2024-09-11T09:52:14.807 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:14.807 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.807 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.808 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:14.808 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.150221+0000", 2024-09-11T09:52:14.808 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:14.808 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.808 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.808 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:14.808 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.150221+0000", 2024-09-11T09:52:14.808 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.9100000000000004e-07 2024-09-11T09:52:14.808 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.808 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.808 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:14.808 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.150226+0000", 2024-09-11T09:52:14.808 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.2250000000000002e-06 2024-09-11T09:52:14.809 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.809 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.809 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:14.809 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.150251+0000", 2024-09-11T09:52:14.809 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.4987e-05 2024-09-11T09:52:14.809 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.809 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.809 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:14.809 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.150258+0000", 2024-09-11T09:52:14.809 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.0790000000000003e-06 2024-09-11T09:52:14.809 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.809 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.809 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:14.810 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.150305+0000", 2024-09-11T09:52:14.810 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.6879000000000003e-05 2024-09-11T09:52:14.810 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.810 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.810 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:14.810 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.150320+0000", 2024-09-11T09:52:14.810 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.5268999999999999e-05 2024-09-11T09:52:14.810 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.810 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.810 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:14.810 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.155493+0000", 2024-09-11T09:52:14.810 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0051731390000000002 2024-09-11T09:52:14.810 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.811 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.811 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:14.811 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.155521+0000", 2024-09-11T09:52:14.811 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.8081000000000001e-05 2024-09-11T09:52:14.811 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:14.811 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:14.811 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:14.811 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.811 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.811 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4694.0:56 2.3 2:c462f029:ns::rbd_data.1256a0c02879.000000000000000c:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:14.811 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.154576+0000", 2024-09-11T09:52:14.811 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 51.567373175999997, 2024-09-11T09:52:14.811 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.020781621, 2024-09-11T09:52:14.811 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:14.812 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:14.812 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:14.812 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:52:14.812 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4694", 2024-09-11T09:52:14.812 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:52:14.812 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 56 2024-09-11T09:52:14.812 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.812 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:14.812 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.812 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:14.812 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.154576+0000", 2024-09-11T09:52:14.812 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:14.813 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.813 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.813 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:14.813 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.154576+0000", 2024-09-11T09:52:14.813 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:14.813 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.813 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.813 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:14.813 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.154577+0000", 2024-09-11T09:52:14.813 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.2249999999999999e-06 2024-09-11T09:52:14.813 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.813 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.813 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:14.813 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.158783+0000", 2024-09-11T09:52:14.814 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0042063889999999996 2024-09-11T09:52:14.814 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.814 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.814 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:14.814 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.158803+0000", 2024-09-11T09:52:14.814 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.9576999999999999e-05 2024-09-11T09:52:14.814 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.814 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.814 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:14.814 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.158815+0000", 2024-09-11T09:52:14.814 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.2221999999999999e-05 2024-09-11T09:52:14.814 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.814 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.815 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:14.815 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.158902+0000", 2024-09-11T09:52:14.815 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.6390000000000005e-05 2024-09-11T09:52:14.815 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.815 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.815 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:14.815 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.159141+0000", 2024-09-11T09:52:14.815 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000238886 2024-09-11T09:52:14.815 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.815 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.815 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 2", 2024-09-11T09:52:14.815 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.159279+0000", 2024-09-11T09:52:14.815 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000138073 2024-09-11T09:52:14.815 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.816 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.816 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:52:14.816 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.173297+0000", 2024-09-11T09:52:14.816 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.014018113 2024-09-11T09:52:14.816 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.816 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.816 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:14.816 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.175278+0000", 2024-09-11T09:52:14.816 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0019817390000000002 2024-09-11T09:52:14.816 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.816 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.816 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:14.817 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.175316+0000", 2024-09-11T09:52:14.817 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.7829e-05 2024-09-11T09:52:14.817 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.817 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.817 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:14.817 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.175357+0000", 2024-09-11T09:52:14.817 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.1177999999999998e-05 2024-09-11T09:52:14.817 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:14.817 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:14.817 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:14.817 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.817 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.817 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4694.0:59 2.2 2:41adefc9:ns::rbd_data.1256a0c02879.000000000000000e:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:14.817 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.158843+0000", 2024-09-11T09:52:14.818 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 51.563105610000001, 2024-09-11T09:52:14.818 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.017874207, 2024-09-11T09:52:14.818 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:14.818 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:14.818 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:14.818 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:52:14.818 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4694", 2024-09-11T09:52:14.818 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:52:14.818 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 59 2024-09-11T09:52:14.818 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.818 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:14.818 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.818 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:14.819 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.158843+0000", 2024-09-11T09:52:14.819 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:14.819 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.819 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.819 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:14.819 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.158843+0000", 2024-09-11T09:52:14.819 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:14.819 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.819 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.819 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:14.819 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.158845+0000", 2024-09-11T09:52:14.819 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.7439999999999999e-06 2024-09-11T09:52:14.819 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.820 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.820 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:14.820 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.162713+0000", 2024-09-11T09:52:14.820 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0038678829999999999 2024-09-11T09:52:14.820 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.820 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.820 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:14.820 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.162728+0000", 2024-09-11T09:52:14.820 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.5114e-05 2024-09-11T09:52:14.820 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.820 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.820 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:14.821 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.162737+0000", 2024-09-11T09:52:14.821 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.1099999999999992e-06 2024-09-11T09:52:14.821 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.821 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.821 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:14.821 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.162799+0000", 2024-09-11T09:52:14.821 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.1934999999999996e-05 2024-09-11T09:52:14.821 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.821 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.821 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:14.821 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.162911+0000", 2024-09-11T09:52:14.821 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000111867 2024-09-11T09:52:14.821 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.822 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.822 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 1", 2024-09-11T09:52:14.822 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.163030+0000", 2024-09-11T09:52:14.822 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000119108 2024-09-11T09:52:14.822 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.822 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.822 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:52:14.822 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.174086+0000", 2024-09-11T09:52:14.822 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.01105543 2024-09-11T09:52:14.822 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.822 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.822 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:14.823 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.176614+0000", 2024-09-11T09:52:14.823 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0025284940000000001 2024-09-11T09:52:14.823 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.823 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.823 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:14.823 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.176672+0000", 2024-09-11T09:52:14.823 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.7892000000000001e-05 2024-09-11T09:52:14.823 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.823 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.823 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:14.823 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.176718+0000", 2024-09-11T09:52:14.823 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.5630000000000002e-05 2024-09-11T09:52:14.823 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:14.823 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:14.824 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:14.824 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.824 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.824 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.5002.0:89 2.3 2:c54742d8:::rbd_data.1250f5e25b80.0000000000000006:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-09-11T09:52:14.824 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:38.546016+0000", 2024-09-11T09:52:14.824 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 36.175933063999999, 2024-09-11T09:52:14.824 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0049098550000000003, 2024-09-11T09:52:14.824 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:14.824 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:14.824 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:14.824 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:52:14.824 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.5002", 2024-09-11T09:52:14.824 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/2204440692", 2024-09-11T09:52:14.825 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 89 2024-09-11T09:52:14.825 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.825 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:14.825 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.825 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:14.825 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.546016+0000", 2024-09-11T09:52:14.825 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:14.825 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.825 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.825 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:14.825 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.546016+0000", 2024-09-11T09:52:14.825 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:14.826 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.826 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.826 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:14.826 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.546017+0000", 2024-09-11T09:52:14.826 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.3399999999999998e-07 2024-09-11T09:52:14.826 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.826 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.826 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:14.826 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.546839+0000", 2024-09-11T09:52:14.826 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00082222800000000004 2024-09-11T09:52:14.826 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.826 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.826 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:14.826 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.546851+0000", 2024-09-11T09:52:14.827 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.2075999999999999e-05 2024-09-11T09:52:14.827 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.827 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.827 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:14.827 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.546856+0000", 2024-09-11T09:52:14.827 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.4009999999999996e-06 2024-09-11T09:52:14.827 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.827 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.827 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:14.827 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.546897+0000", 2024-09-11T09:52:14.827 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.0113e-05 2024-09-11T09:52:14.827 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.827 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.828 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:14.828 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.546954+0000", 2024-09-11T09:52:14.828 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.7825000000000003e-05 2024-09-11T09:52:14.828 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.828 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.828 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 2", 2024-09-11T09:52:14.828 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.547055+0000", 2024-09-11T09:52:14.828 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000100269 2024-09-11T09:52:14.828 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.828 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.828 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:14.828 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.549675+0000", 2024-09-11T09:52:14.828 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0026206609999999998 2024-09-11T09:52:14.829 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.829 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.829 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:52:14.829 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.550858+0000", 2024-09-11T09:52:14.829 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0011825270000000001 2024-09-11T09:52:14.829 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.829 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.829 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:14.829 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.550894+0000", 2024-09-11T09:52:14.829 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.5896999999999999e-05 2024-09-11T09:52:14.829 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.829 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.829 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:14.830 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.550926+0000", 2024-09-11T09:52:14.830 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.2023999999999997e-05 2024-09-11T09:52:14.830 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:14.830 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:14.830 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:14.830 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.830 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.830 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.5002.0:91 2.3 2:c54742d8:::rbd_data.1250f5e25b80.0000000000000006:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-09-11T09:52:14.830 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:38.553743+0000", 2024-09-11T09:52:14.830 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 36.168206021000003, 2024-09-11T09:52:14.830 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0051633629999999998, 2024-09-11T09:52:14.830 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:14.830 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:14.831 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:14.831 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:52:14.831 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.5002", 2024-09-11T09:52:14.831 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/2204440692", 2024-09-11T09:52:14.831 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 91 2024-09-11T09:52:14.831 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.831 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:14.831 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.831 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:14.831 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.553743+0000", 2024-09-11T09:52:14.831 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:14.831 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.831 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.832 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:14.832 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.553743+0000", 2024-09-11T09:52:14.832 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:14.832 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.832 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.832 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:14.832 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.553745+0000", 2024-09-11T09:52:14.832 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.5120000000000001e-06 2024-09-11T09:52:14.832 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.832 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.832 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:14.832 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.554588+0000", 2024-09-11T09:52:14.832 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00084352299999999999 2024-09-11T09:52:14.833 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.833 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.833 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:14.833 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.554604+0000", 2024-09-11T09:52:14.833 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.643e-05 2024-09-11T09:52:14.833 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.833 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.833 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:14.833 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.554613+0000", 2024-09-11T09:52:14.833 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.5669999999999995e-06 2024-09-11T09:52:14.833 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.833 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.833 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:14.834 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.554662+0000", 2024-09-11T09:52:14.834 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.8510000000000001e-05 2024-09-11T09:52:14.834 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.834 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.834 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:14.834 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.554732+0000", 2024-09-11T09:52:14.834 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.0297999999999997e-05 2024-09-11T09:52:14.834 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.834 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.834 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 2", 2024-09-11T09:52:14.834 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.554870+0000", 2024-09-11T09:52:14.834 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00013856599999999999 2024-09-11T09:52:14.834 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.835 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.835 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:14.835 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.557875+0000", 2024-09-11T09:52:14.835 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0030042760000000002 2024-09-11T09:52:14.835 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.835 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.835 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:52:14.835 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.558835+0000", 2024-09-11T09:52:14.835 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00096057900000000003 2024-09-11T09:52:14.835 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.835 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.835 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:14.835 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.558874+0000", 2024-09-11T09:52:14.835 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.8992999999999997e-05 2024-09-11T09:52:14.836 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.836 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.836 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:14.836 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.558906+0000", 2024-09-11T09:52:14.836 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.2109e-05 2024-09-11T09:52:14.836 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:14.836 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:14.836 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:14.836 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.836 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.836 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.5002.0:93 2.3 2:c54742d8:::rbd_data.1250f5e25b80.0000000000000006:head [write 2228224~917504] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e61)", 2024-09-11T09:52:14.836 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:38.562067+0000", 2024-09-11T09:52:14.836 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 36.159882424999999, 2024-09-11T09:52:14.837 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0050551099999999998, 2024-09-11T09:52:14.837 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:14.837 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:14.837 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:14.837 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:52:14.837 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.5002", 2024-09-11T09:52:14.837 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/2204440692", 2024-09-11T09:52:14.837 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 93 2024-09-11T09:52:14.837 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.837 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:14.837 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.837 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:14.838 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.562067+0000", 2024-09-11T09:52:14.838 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:14.838 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.838 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.838 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:14.838 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.562067+0000", 2024-09-11T09:52:14.838 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:14.838 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.838 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.838 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:14.838 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.562068+0000", 2024-09-11T09:52:14.838 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.0669999999999999e-06 2024-09-11T09:52:14.838 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.838 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.839 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:14.839 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.563170+0000", 2024-09-11T09:52:14.839 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00110256 2024-09-11T09:52:14.839 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.839 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.839 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:14.839 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.563185+0000", 2024-09-11T09:52:14.839 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.4873e-05 2024-09-11T09:52:14.839 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.839 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.839 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:14.839 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.563193+0000", 2024-09-11T09:52:14.839 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.1319999999999994e-06 2024-09-11T09:52:14.840 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.840 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.840 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:14.840 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.563238+0000", 2024-09-11T09:52:14.840 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.4773e-05 2024-09-11T09:52:14.840 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.840 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.840 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:14.840 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.563290+0000", 2024-09-11T09:52:14.840 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.1690000000000001e-05 2024-09-11T09:52:14.840 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.840 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.840 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 2", 2024-09-11T09:52:14.841 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.563365+0000", 2024-09-11T09:52:14.841 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.5023000000000003e-05 2024-09-11T09:52:14.841 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.841 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.841 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:14.841 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.566552+0000", 2024-09-11T09:52:14.841 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0031869960000000001 2024-09-11T09:52:14.841 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.841 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.841 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:52:14.841 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.567048+0000", 2024-09-11T09:52:14.841 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00049605099999999996 2024-09-11T09:52:14.841 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.842 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.842 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:14.842 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.567095+0000", 2024-09-11T09:52:14.842 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.6823999999999998e-05 2024-09-11T09:52:14.842 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:14.842 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:14.842 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:14.842 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.567122+0000", 2024-09-11T09:52:14.842 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.7121e-05 2024-09-11T09:52:14.842 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:14.842 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:14.842 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:14.842 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:14.843 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:14.843 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-11T09:52:14.845 INFO:tasks.workunit.client.0.smithi107.stderr:+ return 0 2024-09-11T09:52:14.845 INFO:tasks.workunit.client.0.smithi107.stderr:+ ps -C rbd-nbd 2024-09-11T09:52:14.845 INFO:tasks.workunit.client.0.smithi107.stderr:+ expect_false grep '^ *29922\b' 2024-09-11T09:52:14.845 INFO:tasks.workunit.client.0.smithi107.stderr:+ grep '^ *29922\b' 2024-09-11T09:52:14.845 INFO:tasks.workunit.client.0.smithi107.stderr:+ return 0 2024-09-11T09:52:14.845 INFO:tasks.workunit.client.0.smithi107.stderr:+ sleep 0.5 2024-09-11T09:52:14.846 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-11T09:52:14.846 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:52:14.845+0000 7f3c96ec9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31909) UID: 0 2024-09-11T09:52:14.875 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-11T09:52:14.918 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T09:52:14.944 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:52:14.945+0000 7f3a7b90a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 42475) UID: 0 2024-09-11T09:52:14.990 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-11T09:52:14.990 INFO:teuthology.orchestra.run.smithi096.stdout: "ops": [], 2024-09-11T09:52:14.990 INFO:teuthology.orchestra.run.smithi096.stdout: "num_ops": 0 2024-09-11T09:52:14.990 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-11T09:52:14.999 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-11T09:52:15.050 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-11T09:52:15.050 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T09:52:15.067 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-11T09:52:15.068 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T09:52:15.083 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-09-11T09:52:15.084+0000 7f7b6c7d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31904) UID: 0 2024-09-11T09:52:15.084 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-11T09:52:15.084 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T09:52:15.102 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-11T09:52:15.103 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T09:52:15.119 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-11T09:52:15.120 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T09:52:15.137 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-11T09:52:15.137 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T09:52:15.145 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:52:15.145 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:52:15.145 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:52:15.146+0000 7f3a7b90a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 42475) UID: 0 2024-09-11T09:52:15.154 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-11T09:52:15.246 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:52:15.247+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:52:15.252 INFO:tasks.workunit.client.0.smithi107.stderr:+ DEV= 2024-09-11T09:52:15.253 INFO:tasks.workunit.client.0.smithi107.stderr:++ rbd snap ls rbd/testrbdnbd27121 2024-09-11T09:52:15.253 INFO:tasks.workunit.client.0.smithi107.stderr:++ awk '$2 == "snap" {print $1}' 2024-09-11T09:52:15.297 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:52:15.297 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:52:15.300 INFO:tasks.workunit.client.0.smithi107.stderr:+ SNAPID=3 2024-09-11T09:52:15.301 INFO:tasks.workunit.client.0.smithi107.stderr:++ _sudo rbd device --device-type nbd --show-cookie map --snap-id 3 rbd/testrbdnbd27121 2024-09-11T09:52:15.301 INFO:tasks.workunit.client.0.smithi107.stderr:++ local cmd 2024-09-11T09:52:15.301 INFO:tasks.workunit.client.0.smithi107.stderr:+++ id -u 2024-09-11T09:52:15.302 INFO:tasks.workunit.client.0.smithi107.stderr:++ '[' 1000 -eq 0 ']' 2024-09-11T09:52:15.302 INFO:tasks.workunit.client.0.smithi107.stderr:+++ which rbd 2024-09-11T09:52:15.304 INFO:tasks.workunit.client.0.smithi107.stderr:++ cmd=/usr/bin/rbd 2024-09-11T09:52:15.304 INFO:tasks.workunit.client.0.smithi107.stderr:++ shift 2024-09-11T09:52:15.304 INFO:tasks.workunit.client.0.smithi107.stderr:++ sudo -nE /usr/bin/rbd device --device-type nbd --show-cookie map --snap-id 3 rbd/testrbdnbd27121 2024-09-11T09:52:15.306 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-11T09:52:15.351 INFO:tasks.ceph.osd.0.smithi017.stderr:2024-09-11T09:52:15.352+0000 7fb73c0f1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31665) UID: 0 2024-09-11T09:52:15.417 INFO:tasks.workunit.client.0.smithi107.stderr:+ OUT='/dev/nbd1 5d713523-0fe6-4bfe-afcd-749f512ef288' 2024-09-11T09:52:15.417 INFO:tasks.workunit.client.0.smithi107.stderr:+ read DEV COOKIE 2024-09-11T09:52:15.417 INFO:tasks.workunit.client.0.smithi107.stderr:+ get_pid rbd 2024-09-11T09:52:15.417 INFO:tasks.workunit.client.0.smithi107.stderr:+ local pool=rbd 2024-09-11T09:52:15.417 INFO:tasks.workunit.client.0.smithi107.stderr:+ local ns= 2024-09-11T09:52:15.417 INFO:tasks.workunit.client.0.smithi107.stderr:++ rbd device --device-type nbd --format xml list 2024-09-11T09:52:15.417 INFO:tasks.workunit.client.0.smithi107.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd27121'\''][device='\''/dev/nbd1'\'']/id' 2024-09-11T09:52:15.418 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:52:15.418 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:52:15.425 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-11T09:52:15.449 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-09-11T09:52:15.450+0000 7f7b6c7d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31904) UID: 0 2024-09-11T09:52:15.506 INFO:tasks.workunit.client.0.smithi107.stderr:+ PID=30010 2024-09-11T09:52:15.506 INFO:tasks.workunit.client.0.smithi107.stderr:+ test -n 30010 2024-09-11T09:52:15.506 INFO:tasks.workunit.client.0.smithi107.stderr:+ ps -p 30010 -C rbd-nbd 2024-09-11T09:52:15.517 INFO:tasks.workunit.client.0.smithi107.stdout: PID TTY TIME CMD 2024-09-11T09:52:15.517 INFO:tasks.workunit.client.0.smithi107.stdout: 30010 ? 00:00:00 rbd-nbd 2024-09-11T09:52:15.518 INFO:tasks.workunit.client.0.smithi107.stderr:+ _sudo rbd device detach rbd/testrbdnbd27121 --snap-id 3 --device-type nbd 2024-09-11T09:52:15.518 INFO:tasks.workunit.client.0.smithi107.stderr:+ local cmd 2024-09-11T09:52:15.518 INFO:tasks.workunit.client.0.smithi107.stderr:++ id -u 2024-09-11T09:52:15.518 INFO:tasks.workunit.client.0.smithi107.stderr:+ '[' 1000 -eq 0 ']' 2024-09-11T09:52:15.519 INFO:tasks.workunit.client.0.smithi107.stderr:++ which rbd 2024-09-11T09:52:15.521 INFO:tasks.workunit.client.0.smithi107.stderr:+ cmd=/usr/bin/rbd 2024-09-11T09:52:15.521 INFO:tasks.workunit.client.0.smithi107.stderr:+ shift 2024-09-11T09:52:15.521 INFO:tasks.workunit.client.0.smithi107.stderr:+ sudo -nE /usr/bin/rbd device detach rbd/testrbdnbd27121 --snap-id 3 --device-type nbd 2024-09-11T09:52:15.550 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:52:15.551+0000 7f3c96ec9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31909) UID: 0 2024-09-11T09:52:15.573 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:52:15.573 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:52:15.582 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-11T09:52:15.631 INFO:tasks.workunit.client.0.smithi107.stderr:+ expect_false get_pid rbd 2024-09-11T09:52:15.632 INFO:tasks.workunit.client.0.smithi107.stderr:+ get_pid rbd 2024-09-11T09:52:15.632 INFO:tasks.workunit.client.0.smithi107.stderr:+ local pool=rbd 2024-09-11T09:52:15.632 INFO:tasks.workunit.client.0.smithi107.stderr:+ local ns= 2024-09-11T09:52:15.632 INFO:tasks.workunit.client.0.smithi107.stderr:++ rbd device --device-type nbd --format xml list 2024-09-11T09:52:15.632 INFO:tasks.workunit.client.0.smithi107.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd27121'\''][device='\''/dev/nbd1'\'']/id' 2024-09-11T09:52:15.651 INFO:tasks.ceph.osd.2.smithi017.stderr:2024-09-11T09:52:15.652+0000 7fd20b170640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31672) UID: 0 2024-09-11T09:52:15.690 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:52:15.690 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:52:15.698 INFO:tasks.workunit.client.0.smithi107.stderr:+ PID= 2024-09-11T09:52:15.698 INFO:tasks.workunit.client.0.smithi107.stderr:+ test -n '' 2024-09-11T09:52:15.698 INFO:tasks.workunit.client.0.smithi107.stderr:+ return 1 2024-09-11T09:52:15.698 INFO:tasks.workunit.client.0.smithi107.stderr:+ return 0 2024-09-11T09:52:15.698 INFO:tasks.workunit.client.0.smithi107.stderr:+ expect_false _sudo rbd device attach --device /dev/nbd1 --snap-id 3 rbd/testrbdnbd27121 --device-type nbd 2024-09-11T09:52:15.698 INFO:tasks.workunit.client.0.smithi107.stderr:+ _sudo rbd device attach --device /dev/nbd1 --snap-id 3 rbd/testrbdnbd27121 --device-type nbd 2024-09-11T09:52:15.698 INFO:tasks.workunit.client.0.smithi107.stderr:+ local cmd 2024-09-11T09:52:15.698 INFO:tasks.workunit.client.0.smithi107.stderr:++ id -u 2024-09-11T09:52:15.699 INFO:tasks.workunit.client.0.smithi107.stderr:+ '[' 1000 -eq 0 ']' 2024-09-11T09:52:15.699 INFO:tasks.workunit.client.0.smithi107.stderr:++ which rbd 2024-09-11T09:52:15.700 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-11T09:52:15.701 INFO:tasks.workunit.client.0.smithi107.stderr:+ cmd=/usr/bin/rbd 2024-09-11T09:52:15.701 INFO:tasks.workunit.client.0.smithi107.stderr:+ shift 2024-09-11T09:52:15.701 INFO:tasks.workunit.client.0.smithi107.stderr:+ sudo -nE /usr/bin/rbd device attach --device /dev/nbd1 --snap-id 3 rbd/testrbdnbd27121 --device-type nbd 2024-09-11T09:52:15.738 INFO:tasks.workunit.client.0.smithi107.stderr:rbd: could not validate attach request 2024-09-11T09:52:15.738 INFO:tasks.workunit.client.0.smithi107.stderr:rbd: mismatching the image and the device may lead to data corruption 2024-09-11T09:52:15.738 INFO:tasks.workunit.client.0.smithi107.stderr:rbd: must specify --cookie or --force to proceed 2024-09-11T09:52:15.741 INFO:tasks.workunit.client.0.smithi107.stderr:+ return 0 2024-09-11T09:52:15.741 INFO:tasks.workunit.client.0.smithi107.stderr:+ '[' -n 5d713523-0fe6-4bfe-afcd-749f512ef288 ']' 2024-09-11T09:52:15.741 INFO:tasks.workunit.client.0.smithi107.stderr:+ _sudo rbd device attach --device /dev/nbd1 --cookie 5d713523-0fe6-4bfe-afcd-749f512ef288 --snap-id 3 rbd/testrbdnbd27121 --device-type nbd 2024-09-11T09:52:15.741 INFO:tasks.workunit.client.0.smithi107.stderr:+ local cmd 2024-09-11T09:52:15.741 INFO:tasks.workunit.client.0.smithi107.stderr:++ id -u 2024-09-11T09:52:15.742 INFO:tasks.workunit.client.0.smithi107.stderr:+ '[' 1000 -eq 0 ']' 2024-09-11T09:52:15.742 INFO:tasks.workunit.client.0.smithi107.stderr:++ which rbd 2024-09-11T09:52:15.744 INFO:tasks.workunit.client.0.smithi107.stderr:+ cmd=/usr/bin/rbd 2024-09-11T09:52:15.744 INFO:tasks.workunit.client.0.smithi107.stderr:+ shift 2024-09-11T09:52:15.744 INFO:tasks.workunit.client.0.smithi107.stderr:+ sudo -nE /usr/bin/rbd device attach --device /dev/nbd1 --cookie 5d713523-0fe6-4bfe-afcd-749f512ef288 --snap-id 3 rbd/testrbdnbd27121 --device-type nbd 2024-09-11T09:52:15.785 INFO:tasks.ceph.osd.2.smithi017.stderr:2024-09-11T09:52:15.786+0000 7fd20b170640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31672) UID: 0 2024-09-11T09:52:15.849 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:52:15.850 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:52:15.854 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-09-11T09:52:15.854+0000 7f7b6c7d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31904) UID: 0 2024-09-11T09:52:15.858 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-11T09:52:15.867 INFO:tasks.workunit.client.0.smithi107.stdout:/dev/nbd1 2024-09-11T09:52:15.872 INFO:teuthology.orchestra.run.smithi017.stderr:noscrub is set 2024-09-11T09:52:15.873 INFO:tasks.workunit.client.0.smithi107.stderr:+ get_pid rbd 2024-09-11T09:52:15.873 INFO:tasks.workunit.client.0.smithi107.stderr:+ local pool=rbd 2024-09-11T09:52:15.873 INFO:tasks.workunit.client.0.smithi107.stderr:+ local ns= 2024-09-11T09:52:15.873 INFO:tasks.workunit.client.0.smithi107.stderr:++ rbd device --device-type nbd --format xml list 2024-09-11T09:52:15.873 INFO:tasks.workunit.client.0.smithi107.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd27121'\''][device='\''/dev/nbd1'\'']/id' 2024-09-11T09:52:15.954 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-09-11T09:52:15.956+0000 7f7b6c7d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31904) UID: 0 2024-09-11T09:52:15.955 INFO:tasks.workunit.client.0.smithi107.stderr:+ PID=30080 2024-09-11T09:52:15.955 INFO:tasks.workunit.client.0.smithi107.stderr:+ test -n 30080 2024-09-11T09:52:15.955 INFO:tasks.workunit.client.0.smithi107.stderr:+ ps -p 30080 -C rbd-nbd 2024-09-11T09:52:15.966 INFO:tasks.workunit.client.0.smithi107.stdout: PID TTY TIME CMD 2024-09-11T09:52:15.966 INFO:tasks.workunit.client.0.smithi107.stdout: 30080 ? 00:00:00 rbd-nbd 2024-09-11T09:52:15.967 INFO:tasks.workunit.client.0.smithi107.stderr:+ _sudo rbd device detach /dev/nbd1 --device-type nbd 2024-09-11T09:52:15.967 INFO:tasks.workunit.client.0.smithi107.stderr:+ local cmd 2024-09-11T09:52:15.967 INFO:tasks.workunit.client.0.smithi107.stderr:++ id -u 2024-09-11T09:52:15.968 INFO:tasks.workunit.client.0.smithi107.stderr:+ '[' 1000 -eq 0 ']' 2024-09-11T09:52:15.968 INFO:tasks.workunit.client.0.smithi107.stderr:++ which rbd 2024-09-11T09:52:15.970 INFO:tasks.workunit.client.0.smithi107.stderr:+ cmd=/usr/bin/rbd 2024-09-11T09:52:15.970 INFO:tasks.workunit.client.0.smithi107.stderr:+ shift 2024-09-11T09:52:15.970 INFO:tasks.workunit.client.0.smithi107.stderr:+ sudo -nE /usr/bin/rbd device detach /dev/nbd1 --device-type nbd 2024-09-11T09:52:15.979 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:52:15.979 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:52:15.989 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-11T09:52:16.055 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:52:16.056+0000 7f3c96ec9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31909) UID: 0 2024-09-11T09:52:16.063 INFO:tasks.workunit.client.0.smithi107.stderr:+ expect_false get_pid rbd 2024-09-11T09:52:16.063 INFO:tasks.workunit.client.0.smithi107.stderr:+ get_pid rbd 2024-09-11T09:52:16.063 INFO:tasks.workunit.client.0.smithi107.stderr:+ local pool=rbd 2024-09-11T09:52:16.063 INFO:tasks.workunit.client.0.smithi107.stderr:+ local ns= 2024-09-11T09:52:16.064 INFO:tasks.workunit.client.0.smithi107.stderr:++ rbd device --device-type nbd --format xml list 2024-09-11T09:52:16.064 INFO:tasks.workunit.client.0.smithi107.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd27121'\''][device='\''/dev/nbd1'\'']/id' 2024-09-11T09:52:16.096 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:52:16.096 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:52:16.105 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-11T09:52:16.131 INFO:tasks.workunit.client.0.smithi107.stderr:+ PID= 2024-09-11T09:52:16.131 INFO:tasks.workunit.client.0.smithi107.stderr:+ test -n '' 2024-09-11T09:52:16.131 INFO:tasks.workunit.client.0.smithi107.stderr:+ return 1 2024-09-11T09:52:16.132 INFO:tasks.workunit.client.0.smithi107.stderr:+ return 0 2024-09-11T09:52:16.132 INFO:tasks.workunit.client.0.smithi107.stderr:+ DEV= 2024-09-11T09:52:16.132 INFO:tasks.workunit.client.0.smithi107.stderr:+ rbd config image set rbd/testrbdnbd27121 rbd_discard_granularity_bytes 4096 2024-09-11T09:52:16.156 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:52:16.157+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:52:16.195 INFO:tasks.workunit.client.0.smithi107.stderr:+ rbd feature enable rbd/testrbdnbd27121 journaling 2024-09-11T09:52:16.214 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:52:16.214 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:52:16.223 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-11T09:52:16.256 INFO:tasks.workunit.client.0.smithi107.stderr:++ _sudo rbd device --device-type nbd map rbd/testrbdnbd27121 2024-09-11T09:52:16.257 INFO:tasks.workunit.client.0.smithi107.stderr:++ local cmd 2024-09-11T09:52:16.257 INFO:tasks.workunit.client.0.smithi107.stderr:+++ id -u 2024-09-11T09:52:16.257 INFO:tasks.workunit.client.0.smithi107.stderr:++ '[' 1000 -eq 0 ']' 2024-09-11T09:52:16.257 INFO:tasks.workunit.client.0.smithi107.stderr:+++ which rbd 2024-09-11T09:52:16.259 INFO:tasks.workunit.client.0.smithi107.stderr:++ cmd=/usr/bin/rbd 2024-09-11T09:52:16.259 INFO:tasks.workunit.client.0.smithi107.stderr:++ shift 2024-09-11T09:52:16.259 INFO:tasks.workunit.client.0.smithi107.stderr:++ sudo -nE /usr/bin/rbd device --device-type nbd map rbd/testrbdnbd27121 2024-09-11T09:52:16.265 INFO:tasks.ceph.osd.2.smithi017.stderr:2024-09-11T09:52:16.266+0000 7fd20b170640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31672) UID: 0 2024-09-11T09:52:16.357 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-09-11T09:52:16.358+0000 7f7b6c7d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31904) UID: 0 2024-09-11T09:52:16.375 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:52:16.375 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:52:16.376 INFO:tasks.workunit.client.0.smithi107.stderr:+ DEV=/dev/nbd2 2024-09-11T09:52:16.376 INFO:tasks.workunit.client.0.smithi107.stderr:+ get_pid rbd 2024-09-11T09:52:16.377 INFO:tasks.workunit.client.0.smithi107.stderr:+ local pool=rbd 2024-09-11T09:52:16.377 INFO:tasks.workunit.client.0.smithi107.stderr:+ local ns= 2024-09-11T09:52:16.377 INFO:tasks.workunit.client.0.smithi107.stderr:++ rbd device --device-type nbd --format xml list 2024-09-11T09:52:16.377 INFO:tasks.workunit.client.0.smithi107.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd27121'\''][device='\''/dev/nbd2'\'']/id' 2024-09-11T09:52:16.384 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-11T09:52:16.458 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:52:16.459+0000 7f3a7b90a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 42475) UID: 0 2024-09-11T09:52:16.468 INFO:tasks.workunit.client.0.smithi107.stderr:+ PID=30183 2024-09-11T09:52:16.468 INFO:tasks.workunit.client.0.smithi107.stderr:+ test -n 30183 2024-09-11T09:52:16.468 INFO:tasks.workunit.client.0.smithi107.stderr:+ ps -p 30183 -C rbd-nbd 2024-09-11T09:52:16.479 INFO:tasks.workunit.client.0.smithi107.stdout: PID TTY TIME CMD 2024-09-11T09:52:16.479 INFO:tasks.workunit.client.0.smithi107.stdout: 30183 ? 00:00:00 rbd-nbd 2024-09-11T09:52:16.479 INFO:tasks.workunit.client.0.smithi107.stderr:+ _sudo blkdiscard --offset 0 --length 3584 /dev/nbd2 2024-09-11T09:52:16.480 INFO:tasks.workunit.client.0.smithi107.stderr:+ local cmd 2024-09-11T09:52:16.480 INFO:tasks.workunit.client.0.smithi107.stderr:++ id -u 2024-09-11T09:52:16.480 INFO:tasks.workunit.client.0.smithi107.stderr:+ '[' 1000 -eq 0 ']' 2024-09-11T09:52:16.480 INFO:tasks.workunit.client.0.smithi107.stderr:++ which blkdiscard 2024-09-11T09:52:16.482 INFO:tasks.workunit.client.0.smithi107.stderr:+ cmd=/usr/sbin/blkdiscard 2024-09-11T09:52:16.482 INFO:tasks.workunit.client.0.smithi107.stderr:+ shift 2024-09-11T09:52:16.482 INFO:tasks.workunit.client.0.smithi107.stderr:+ sudo -nE /usr/sbin/blkdiscard --offset 0 --length 3584 /dev/nbd2 2024-09-11T09:52:16.491 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:52:16.491 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:52:16.501 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-11T09:52:16.505 INFO:tasks.workunit.client.0.smithi107.stderr:blkdiscard: /dev/nbd2 contains existing file system (ext2). 2024-09-11T09:52:16.506 INFO:tasks.workunit.client.0.smithi107.stderr:+ _sudo blkdiscard --offset 0 --length 4096 /dev/nbd2 2024-09-11T09:52:16.507 INFO:tasks.workunit.client.0.smithi107.stderr:+ local cmd 2024-09-11T09:52:16.507 INFO:tasks.workunit.client.0.smithi107.stderr:++ id -u 2024-09-11T09:52:16.507 INFO:tasks.workunit.client.0.smithi107.stderr:+ '[' 1000 -eq 0 ']' 2024-09-11T09:52:16.507 INFO:tasks.workunit.client.0.smithi107.stderr:++ which blkdiscard 2024-09-11T09:52:16.509 INFO:tasks.workunit.client.0.smithi107.stderr:+ cmd=/usr/sbin/blkdiscard 2024-09-11T09:52:16.509 INFO:tasks.workunit.client.0.smithi107.stderr:+ shift 2024-09-11T09:52:16.509 INFO:tasks.workunit.client.0.smithi107.stderr:+ sudo -nE /usr/sbin/blkdiscard --offset 0 --length 4096 /dev/nbd2 2024-09-11T09:52:16.529 INFO:tasks.workunit.client.0.smithi107.stderr:+ _sudo blkdiscard --offset 0 --length 4608 /dev/nbd2 2024-09-11T09:52:16.529 INFO:tasks.workunit.client.0.smithi107.stderr:+ local cmd 2024-09-11T09:52:16.529 INFO:tasks.workunit.client.0.smithi107.stderr:++ id -u 2024-09-11T09:52:16.530 INFO:tasks.workunit.client.0.smithi107.stderr:+ '[' 1000 -eq 0 ']' 2024-09-11T09:52:16.530 INFO:tasks.workunit.client.0.smithi107.stderr:++ which blkdiscard 2024-09-11T09:52:16.532 INFO:tasks.workunit.client.0.smithi107.stderr:+ cmd=/usr/sbin/blkdiscard 2024-09-11T09:52:16.532 INFO:tasks.workunit.client.0.smithi107.stderr:+ shift 2024-09-11T09:52:16.532 INFO:tasks.workunit.client.0.smithi107.stderr:+ sudo -nE /usr/sbin/blkdiscard --offset 0 --length 4608 /dev/nbd2 2024-09-11T09:52:16.547 INFO:tasks.workunit.client.0.smithi107.stderr:+ _sudo blkdiscard --offset 512 --length 7168 /dev/nbd2 2024-09-11T09:52:16.547 INFO:tasks.workunit.client.0.smithi107.stderr:+ local cmd 2024-09-11T09:52:16.547 INFO:tasks.workunit.client.0.smithi107.stderr:++ id -u 2024-09-11T09:52:16.548 INFO:tasks.workunit.client.0.smithi107.stderr:+ '[' 1000 -eq 0 ']' 2024-09-11T09:52:16.548 INFO:tasks.workunit.client.0.smithi107.stderr:++ which blkdiscard 2024-09-11T09:52:16.550 INFO:tasks.workunit.client.0.smithi107.stderr:+ cmd=/usr/sbin/blkdiscard 2024-09-11T09:52:16.550 INFO:tasks.workunit.client.0.smithi107.stderr:+ shift 2024-09-11T09:52:16.550 INFO:tasks.workunit.client.0.smithi107.stderr:+ sudo -nE /usr/sbin/blkdiscard --offset 512 --length 7168 /dev/nbd2 2024-09-11T09:52:16.559 INFO:tasks.ceph.osd.2.smithi017.stderr:2024-09-11T09:52:16.559+0000 7fd20b170640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31672) UID: 0 2024-09-11T09:52:16.562 INFO:tasks.workunit.client.0.smithi107.stderr:+ _sudo blkdiscard --offset 512 --length 7680 /dev/nbd2 2024-09-11T09:52:16.562 INFO:tasks.workunit.client.0.smithi107.stderr:+ local cmd 2024-09-11T09:52:16.562 INFO:tasks.workunit.client.0.smithi107.stderr:++ id -u 2024-09-11T09:52:16.563 INFO:tasks.workunit.client.0.smithi107.stderr:+ '[' 1000 -eq 0 ']' 2024-09-11T09:52:16.563 INFO:tasks.workunit.client.0.smithi107.stderr:++ which blkdiscard 2024-09-11T09:52:16.565 INFO:tasks.workunit.client.0.smithi107.stderr:+ cmd=/usr/sbin/blkdiscard 2024-09-11T09:52:16.565 INFO:tasks.workunit.client.0.smithi107.stderr:+ shift 2024-09-11T09:52:16.565 INFO:tasks.workunit.client.0.smithi107.stderr:+ sudo -nE /usr/sbin/blkdiscard --offset 512 --length 7680 /dev/nbd2 2024-09-11T09:52:16.578 INFO:tasks.workunit.client.0.smithi107.stderr:+ _sudo blkdiscard --offset 512 --length 8192 /dev/nbd2 2024-09-11T09:52:16.579 INFO:tasks.workunit.client.0.smithi107.stderr:+ local cmd 2024-09-11T09:52:16.579 INFO:tasks.workunit.client.0.smithi107.stderr:++ id -u 2024-09-11T09:52:16.579 INFO:tasks.workunit.client.0.smithi107.stderr:+ '[' 1000 -eq 0 ']' 2024-09-11T09:52:16.580 INFO:tasks.workunit.client.0.smithi107.stderr:++ which blkdiscard 2024-09-11T09:52:16.581 INFO:tasks.workunit.client.0.smithi107.stderr:+ cmd=/usr/sbin/blkdiscard 2024-09-11T09:52:16.581 INFO:tasks.workunit.client.0.smithi107.stderr:+ shift 2024-09-11T09:52:16.581 INFO:tasks.workunit.client.0.smithi107.stderr:+ sudo -nE /usr/sbin/blkdiscard --offset 512 --length 8192 /dev/nbd2 2024-09-11T09:52:16.595 INFO:tasks.workunit.client.0.smithi107.stderr:+ tries=0 2024-09-11T09:52:16.595 INFO:tasks.workunit.client.0.smithi107.stderr:++ rbd journal inspect --pool rbd --image testrbdnbd27121 2024-09-11T09:52:16.596 INFO:tasks.workunit.client.0.smithi107.stderr:++ awk '/entries inspected/ {print $1}' 2024-09-11T09:52:16.606 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:52:16.607 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:52:16.616 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-11T09:52:16.652 INFO:tasks.workunit.client.0.smithi107.stderr:2024-09-11T09:52:16.652+0000 7fd2de90e640 -1 JournalMetadata: 0x556958a377f0 failed to locate client: INSPECT 2024-09-11T09:52:16.656 INFO:tasks.workunit.client.0.smithi107.stderr:+ queue_length=4 2024-09-11T09:52:16.656 INFO:tasks.workunit.client.0.smithi107.stderr:+ '[' 0 -lt 10 ']' 2024-09-11T09:52:16.656 INFO:tasks.workunit.client.0.smithi107.stderr:+ '[' 4 -gt 0 ']' 2024-09-11T09:52:16.656 INFO:tasks.workunit.client.0.smithi107.stderr:+ rbd journal inspect --pool rbd --image testrbdnbd27121 --verbose 2024-09-11T09:52:16.662 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:52:16.663+0000 7f3c96ec9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31909) UID: 0 2024-09-11T09:52:16.709 INFO:tasks.workunit.client.0.smithi107.stdout:Entry: tag_id=1, commit_tid=1 2024-09-11T09:52:16.709 INFO:tasks.workunit.client.0.smithi107.stdout:{ 2024-09-11T09:52:16.709 INFO:tasks.workunit.client.0.smithi107.stdout: "event_type": "AioDiscard", 2024-09-11T09:52:16.709 INFO:tasks.workunit.client.0.smithi107.stdout: "offset": 0, 2024-09-11T09:52:16.709 INFO:tasks.workunit.client.0.smithi107.stdout: "length": 4096, 2024-09-11T09:52:16.709 INFO:tasks.workunit.client.0.smithi107.stdout: "discard_granularity_bytes": 4096, 2024-09-11T09:52:16.709 INFO:tasks.workunit.client.0.smithi107.stdout: "timestamp": "2024-09-11T09:52:16.526805+0000" 2024-09-11T09:52:16.709 INFO:tasks.workunit.client.0.smithi107.stdout:} 2024-09-11T09:52:16.709 INFO:tasks.workunit.client.0.smithi107.stdout:Entry: tag_id=1, commit_tid=2 2024-09-11T09:52:16.709 INFO:tasks.workunit.client.0.smithi107.stdout:{ 2024-09-11T09:52:16.709 INFO:tasks.workunit.client.0.smithi107.stdout: "event_type": "AioDiscard", 2024-09-11T09:52:16.709 INFO:tasks.workunit.client.0.smithi107.stdout: "offset": 0, 2024-09-11T09:52:16.710 INFO:tasks.workunit.client.0.smithi107.stdout: "length": 4096, 2024-09-11T09:52:16.710 INFO:tasks.workunit.client.0.smithi107.stdout: "discard_granularity_bytes": 4096, 2024-09-11T09:52:16.710 INFO:tasks.workunit.client.0.smithi107.stdout: "timestamp": "2024-09-11T09:52:16.544819+0000" 2024-09-11T09:52:16.710 INFO:tasks.workunit.client.0.smithi107.stdout:} 2024-09-11T09:52:16.710 INFO:tasks.workunit.client.0.smithi107.stdout:Entry: tag_id=1, commit_tid=3 2024-09-11T09:52:16.710 INFO:tasks.workunit.client.0.smithi107.stdout:{ 2024-09-11T09:52:16.710 INFO:tasks.workunit.client.0.smithi107.stdout: "event_type": "AioDiscard", 2024-09-11T09:52:16.710 INFO:tasks.workunit.client.0.smithi107.stdout: "offset": 4096, 2024-09-11T09:52:16.710 INFO:tasks.workunit.client.0.smithi107.stdout: "length": 4096, 2024-09-11T09:52:16.710 INFO:tasks.workunit.client.0.smithi107.stdout: "discard_granularity_bytes": 4096, 2024-09-11T09:52:16.710 INFO:tasks.workunit.client.0.smithi107.stdout: "timestamp": "2024-09-11T09:52:16.576535+0000" 2024-09-11T09:52:16.710 INFO:tasks.workunit.client.0.smithi107.stdout:} 2024-09-11T09:52:16.710 INFO:tasks.workunit.client.0.smithi107.stdout:Entry: tag_id=1, commit_tid=4 2024-09-11T09:52:16.710 INFO:tasks.workunit.client.0.smithi107.stdout:{ 2024-09-11T09:52:16.710 INFO:tasks.workunit.client.0.smithi107.stdout: "event_type": "AioDiscard", 2024-09-11T09:52:16.711 INFO:tasks.workunit.client.0.smithi107.stdout: "offset": 4096, 2024-09-11T09:52:16.711 INFO:tasks.workunit.client.0.smithi107.stdout: "length": 4096, 2024-09-11T09:52:16.711 INFO:tasks.workunit.client.0.smithi107.stdout: "discard_granularity_bytes": 4096, 2024-09-11T09:52:16.711 INFO:tasks.workunit.client.0.smithi107.stdout: "timestamp": "2024-09-11T09:52:16.593859+0000" 2024-09-11T09:52:16.711 INFO:tasks.workunit.client.0.smithi107.stdout:} 2024-09-11T09:52:16.711 INFO:tasks.workunit.client.0.smithi107.stdout:Summary: 2024-09-11T09:52:16.711 INFO:tasks.workunit.client.0.smithi107.stdout: 4 entries inspected, 0 errors 2024-09-11T09:52:16.711 INFO:tasks.workunit.client.0.smithi107.stderr:2024-09-11T09:52:16.711+0000 7f63627fc640 -1 JournalMetadata: 0x55f23b5d81f0 failed to locate client: INSPECT 2024-09-11T09:52:16.714 INFO:tasks.workunit.client.0.smithi107.stderr:+ sleep 1 2024-09-11T09:52:16.720 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:52:16.720 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:52:16.729 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-11T09:52:16.760 INFO:tasks.ceph.osd.0.smithi017.stderr:2024-09-11T09:52:16.761+0000 7fb73c0f1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31665) UID: 0 2024-09-11T09:52:16.861 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-09-11T09:52:16.862+0000 7f7b6c7d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31904) UID: 0 2024-09-11T09:52:16.872 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:52:16.872 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:52:16.881 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-11T09:52:16.965 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:52:16.966+0000 7f3c96ec9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31909) UID: 0 2024-09-11T09:52:17.024 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:52:17.024 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:52:17.033 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-11T09:52:17.063 INFO:tasks.ceph.osd.2.smithi017.stderr:2024-09-11T09:52:17.063+0000 7fd20b170640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31672) UID: 0 2024-09-11T09:52:17.148 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T09:52:17.176 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:52:17.176 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:52:17.184 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-11T09:52:17.193 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:52:17.194+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:52:17.264 INFO:tasks.ceph.osd.0.smithi017.stderr:2024-09-11T09:52:17.265+0000 7fb73c0f1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31665) UID: 0 2024-09-11T09:52:17.272 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-11T09:52:17.272 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T09:52:17.287 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:52:17.288 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:52:17.289 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-11T09:52:17.289 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T09:52:17.296 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-11T09:52:17.306 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-11T09:52:17.306 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T09:52:17.323 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-11T09:52:17.323 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T09:52:17.340 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-11T09:52:17.340 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T09:52:17.357 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-11T09:52:17.357 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T09:52:17.365 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-09-11T09:52:17.366+0000 7f7b6c7d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31904) UID: 0 2024-09-11T09:52:17.400 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-11T09:52:17.400 INFO:teuthology.orchestra.run.smithi096.stdout: "ops": [], 2024-09-11T09:52:17.400 INFO:teuthology.orchestra.run.smithi096.stdout: "num_ops": 0 2024-09-11T09:52:17.401 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-11T09:52:17.409 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-11T09:52:17.466 INFO:tasks.ceph.osd.2.smithi017.stderr:2024-09-11T09:52:17.467+0000 7fd20b170640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31672) UID: 0 2024-09-11T09:52:17.551 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-11T09:52:17.552 INFO:teuthology.orchestra.run.smithi096.stdout: "ops": [], 2024-09-11T09:52:17.552 INFO:teuthology.orchestra.run.smithi096.stdout: "complaint_time": 30, 2024-09-11T09:52:17.552 INFO:teuthology.orchestra.run.smithi096.stdout: "num_blocked_ops": 0 2024-09-11T09:52:17.552 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-11T09:52:17.560 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-11T09:52:17.604 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:52:17.605+0000 7f3a7b90a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 42475) UID: 0 2024-09-11T09:52:17.664 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-11T09:52:17.664 INFO:teuthology.orchestra.run.smithi096.stdout: "size": 20, 2024-09-11T09:52:17.664 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 600, 2024-09-11T09:52:17.664 INFO:teuthology.orchestra.run.smithi096.stdout: "ops": [ 2024-09-11T09:52:17.664 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:17.664 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "MOSDPGPush(2.4 82/81 [PushOp(2:3bfc818c:ns::rbd_data.1256a0c02879.0000000000000001:head, version: 53'14, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:3bfc818c:ns::rbd_data.1256a0c02879.0000000000000001:head@53'14, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-11T09:52:17.664 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:52:08.947926+0000", 2024-09-11T09:52:17.664 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 8.7157675989999994, 2024-09-11T09:52:17.664 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0020846179999999999, 2024-09-11T09:52:17.664 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:17.665 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:17.665 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "started", 2024-09-11T09:52:17.665 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:17.665 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:17.665 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:17.665 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.947926+0000", 2024-09-11T09:52:17.665 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:17.665 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:17.665 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:17.665 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:17.665 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.947926+0000", 2024-09-11T09:52:17.665 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:17.665 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:17.665 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:17.666 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:17.666 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.947927+0000", 2024-09-11T09:52:17.666 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.9599999999999999e-07 2024-09-11T09:52:17.666 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:17.666 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:17.666 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:17.666 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.949094+0000", 2024-09-11T09:52:17.666 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001167944 2024-09-11T09:52:17.666 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:17.666 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:17.666 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:17.666 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.949110+0000", 2024-09-11T09:52:17.666 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.5650000000000001e-05 2024-09-11T09:52:17.666 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:17.666 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:17.666 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:17.667 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.949138+0000", 2024-09-11T09:52:17.667 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.7515e-05 2024-09-11T09:52:17.667 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:17.667 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:17.667 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:17.667 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.949196+0000", 2024-09-11T09:52:17.667 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.8272e-05 2024-09-11T09:52:17.667 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:17.667 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:17.667 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:17.667 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.949204+0000", 2024-09-11T09:52:17.667 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.2150000000000001e-06 2024-09-11T09:52:17.667 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:17.668 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:17.668 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:17.668 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.950011+0000", 2024-09-11T09:52:17.668 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00080652600000000001 2024-09-11T09:52:17.668 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:17.668 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:17.668 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:17.668 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:17.668 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:17.668 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "MOSDPGPush(2.4 82/81 [PushOp(2:3bfc818c:ns::rbd_data.1256a0c02879.0000000000000001:head, version: 53'14, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:3bfc818c:ns::rbd_data.1256a0c02879.0000000000000001:head@53'14, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-11T09:52:17.668 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:52:08.951738+0000", 2024-09-11T09:52:17.668 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 8.7119560810000003, 2024-09-11T09:52:17.668 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0018266020000000001, 2024-09-11T09:52:17.669 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:17.669 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:17.669 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "started", 2024-09-11T09:52:17.669 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:17.669 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:17.669 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:17.669 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.951738+0000", 2024-09-11T09:52:17.669 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:17.669 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:17.669 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:17.669 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:17.669 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.951738+0000", 2024-09-11T09:52:17.669 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:17.670 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:17.670 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:17.670 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:17.670 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.951739+0000", 2024-09-11T09:52:17.670 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.004e-06 2024-09-11T09:52:17.670 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:17.670 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:17.670 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:17.670 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.952722+0000", 2024-09-11T09:52:17.670 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00098314800000000009 2024-09-11T09:52:17.670 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:17.670 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:17.670 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:17.671 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.952750+0000", 2024-09-11T09:52:17.671 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.8050000000000001e-05 2024-09-11T09:52:17.671 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:17.671 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:17.671 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:17.671 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.952771+0000", 2024-09-11T09:52:17.671 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.0891e-05 2024-09-11T09:52:17.671 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:17.671 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:17.671 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:17.671 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.952844+0000", 2024-09-11T09:52:17.671 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.3064000000000004e-05 2024-09-11T09:52:17.672 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:17.672 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:17.672 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:17.672 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.952853+0000", 2024-09-11T09:52:17.672 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.1160000000000005e-06 2024-09-11T09:52:17.672 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:17.672 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:17.672 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:17.672 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.953564+0000", 2024-09-11T09:52:17.672 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00071132900000000002 2024-09-11T09:52:17.672 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:17.672 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:17.673 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:17.673 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:17.673 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:17.673 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "MOSDPGPush(2.4 82/81 [PushOp(2:3bfc818c:ns::rbd_data.1256a0c02879.0000000000000001:head, version: 53'14, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:3bfc818c:ns::rbd_data.1256a0c02879.0000000000000001:head@53'14, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3145728, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-11T09:52:17.673 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:52:08.955761+0000", 2024-09-11T09:52:17.673 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 8.7079329239999996, 2024-09-11T09:52:17.673 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0019711659999999999, 2024-09-11T09:52:17.673 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:17.673 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:17.673 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "started", 2024-09-11T09:52:17.673 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:17.673 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:17.673 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:17.673 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.955761+0000", 2024-09-11T09:52:17.674 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:17.674 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:17.674 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:17.674 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:17.674 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.955761+0000", 2024-09-11T09:52:17.674 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:17.674 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:17.674 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:17.674 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:17.674 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.955761+0000", 2024-09-11T09:52:17.674 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.2399999999999997e-07 2024-09-11T09:52:17.674 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:17.674 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:17.675 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:17.675 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.956708+0000", 2024-09-11T09:52:17.675 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00094629899999999995 2024-09-11T09:52:17.675 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:17.675 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:17.675 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:17.675 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.956731+0000", 2024-09-11T09:52:17.675 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.3175000000000001e-05 2024-09-11T09:52:17.675 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:17.675 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:17.675 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:17.675 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.956750+0000", 2024-09-11T09:52:17.675 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.8859e-05 2024-09-11T09:52:17.676 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:17.676 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:17.676 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:17.676 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.956828+0000", 2024-09-11T09:52:17.676 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.8567999999999997e-05 2024-09-11T09:52:17.676 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:17.676 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:17.676 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:17.676 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.956841+0000", 2024-09-11T09:52:17.676 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.2489e-05 2024-09-11T09:52:17.676 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:17.676 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:17.676 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:17.677 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.957732+0000", 2024-09-11T09:52:17.677 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00089105200000000003 2024-09-11T09:52:17.677 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:17.677 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:17.677 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:17.677 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:17.677 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:17.677 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "MOSDPGPush(2.4 82/81 [PushOp(2:3bfc818c:ns::rbd_data.1256a0c02879.0000000000000001:head, version: 53'14, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:3bfc818c:ns::rbd_data.1256a0c02879.0000000000000001:head@53'14, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 4194304, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3145728, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-11T09:52:17.677 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:52:08.959483+0000", 2024-09-11T09:52:17.677 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 8.7042103940000004, 2024-09-11T09:52:17.677 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0021404520000000002, 2024-09-11T09:52:17.677 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:17.677 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:17.678 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "started", 2024-09-11T09:52:17.678 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:17.678 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:17.678 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:17.678 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.959483+0000", 2024-09-11T09:52:17.678 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:17.678 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:17.678 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:17.678 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:17.678 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.959483+0000", 2024-09-11T09:52:17.678 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:17.678 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:17.678 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:17.679 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:17.679 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.959484+0000", 2024-09-11T09:52:17.679 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.4900000000000005e-07 2024-09-11T09:52:17.679 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:17.679 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:17.679 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:17.679 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.960450+0000", 2024-09-11T09:52:17.679 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000966141 2024-09-11T09:52:17.679 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:17.679 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:17.679 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:17.679 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.960466+0000", 2024-09-11T09:52:17.679 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.6302000000000001e-05 2024-09-11T09:52:17.680 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:17.680 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:17.680 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:17.680 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.960477+0000", 2024-09-11T09:52:17.680 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.0406e-05 2024-09-11T09:52:17.680 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:17.680 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:17.680 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:17.680 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.960555+0000", 2024-09-11T09:52:17.680 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.7837999999999996e-05 2024-09-11T09:52:17.680 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:17.680 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:17.680 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:17.681 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.960568+0000", 2024-09-11T09:52:17.681 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.2904e-05 2024-09-11T09:52:17.681 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:17.681 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:17.681 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:17.681 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.961624+0000", 2024-09-11T09:52:17.681 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0010561119999999999 2024-09-11T09:52:17.681 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:17.681 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:17.681 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:17.681 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:17.681 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:17.681 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "MOSDPGPush(2.4 82/81 [PushOp(2:2367385a:::rbd_data.1250f5e25b80.0000000000000002:6, version: 79'169, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 1, recovery_info: ObjectRecoveryInfo(2:2367385a:::rbd_data.1250f5e25b80.0000000000000002:6@79'169, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 6=[]:{6: [6, 5, 3]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-11T09:52:17.681 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:52:08.964132+0000", 2024-09-11T09:52:17.682 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 8.6995619130000001, 2024-09-11T09:52:17.682 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0021074240000000001, 2024-09-11T09:52:17.682 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:17.682 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:17.682 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "started", 2024-09-11T09:52:17.682 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:17.682 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:17.682 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:17.682 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.964132+0000", 2024-09-11T09:52:17.682 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:17.682 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:17.682 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:17.682 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:17.683 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.964132+0000", 2024-09-11T09:52:17.683 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:17.683 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:17.683 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:17.683 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:17.683 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.964133+0000", 2024-09-11T09:52:17.683 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.02e-07 2024-09-11T09:52:17.683 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:17.683 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:17.683 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:17.683 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.965104+0000", 2024-09-11T09:52:17.683 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000971849 2024-09-11T09:52:17.683 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:17.683 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:17.684 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:17.684 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.965122+0000", 2024-09-11T09:52:17.684 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.7706999999999998e-05 2024-09-11T09:52:17.684 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:17.684 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:17.684 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:17.684 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.965134+0000", 2024-09-11T09:52:17.684 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1371e-05 2024-09-11T09:52:17.684 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:17.684 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:17.684 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:17.684 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.965221+0000", 2024-09-11T09:52:17.684 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.7037e-05 2024-09-11T09:52:17.685 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:17.685 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:17.685 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:17.685 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.965236+0000", 2024-09-11T09:52:17.685 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.5724999999999999e-05 2024-09-11T09:52:17.685 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:17.685 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:17.685 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:17.685 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.966239+0000", 2024-09-11T09:52:17.685 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0010028330000000001 2024-09-11T09:52:17.686 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:17.686 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:17.686 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:17.686 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:17.686 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:17.686 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "MOSDPGPush(2.4 82/81 [PushOp(2:2367385a:::rbd_data.1250f5e25b80.0000000000000002:6, version: 79'169, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:2367385a:::rbd_data.1250f5e25b80.0000000000000002:6@79'169, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 6=[]:{6: [6, 5, 3]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-11T09:52:17.686 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:52:08.968106+0000", 2024-09-11T09:52:17.686 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 8.6955874459999993, 2024-09-11T09:52:17.686 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.002036936, 2024-09-11T09:52:17.686 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:17.686 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:17.686 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "started", 2024-09-11T09:52:17.686 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:17.687 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:17.687 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:17.687 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.968106+0000", 2024-09-11T09:52:17.687 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:17.687 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:17.687 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:17.687 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:17.687 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.968106+0000", 2024-09-11T09:52:17.687 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:17.687 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:17.687 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:17.687 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:17.687 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.968107+0000", 2024-09-11T09:52:17.688 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.0100000000000001e-06 2024-09-11T09:52:17.688 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:17.688 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:17.688 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:17.688 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.969086+0000", 2024-09-11T09:52:17.688 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00097889099999999996 2024-09-11T09:52:17.688 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:17.688 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:17.688 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:17.688 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.969104+0000", 2024-09-11T09:52:17.688 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.7940000000000001e-05 2024-09-11T09:52:17.688 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:17.688 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:17.689 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:17.689 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.969117+0000", 2024-09-11T09:52:17.689 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.3207000000000001e-05 2024-09-11T09:52:17.689 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:17.689 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:17.689 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:17.689 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.969192+0000", 2024-09-11T09:52:17.689 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.4419999999999996e-05 2024-09-11T09:52:17.689 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:17.689 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:17.689 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:17.689 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.969201+0000", 2024-09-11T09:52:17.690 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.4960000000000002e-06 2024-09-11T09:52:17.690 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:17.690 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:17.690 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:17.690 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.970143+0000", 2024-09-11T09:52:17.690 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00094197200000000001 2024-09-11T09:52:17.690 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:17.690 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:17.690 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:17.690 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:17.690 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:17.690 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "MOSDPGPush(2.4 82/81 [PushOp(2:2367385a:::rbd_data.1250f5e25b80.0000000000000002:6, version: 79'169, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:2367385a:::rbd_data.1250f5e25b80.0000000000000002:6@79'169, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 6=[]:{6: [6, 5, 3]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3145728, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-11T09:52:17.690 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:52:08.971856+0000", 2024-09-11T09:52:17.691 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 8.6918374239999991, 2024-09-11T09:52:17.691 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0018957309999999999, 2024-09-11T09:52:17.691 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:17.691 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:17.691 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "started", 2024-09-11T09:52:17.691 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:17.691 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:17.691 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:17.691 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.971856+0000", 2024-09-11T09:52:17.691 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:17.691 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:17.692 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:17.692 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:17.692 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.971856+0000", 2024-09-11T09:52:17.692 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:17.692 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:17.692 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:17.692 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:17.692 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.971857+0000", 2024-09-11T09:52:17.692 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.4900000000000004e-07 2024-09-11T09:52:17.692 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:17.692 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:17.692 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:17.693 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.972795+0000", 2024-09-11T09:52:17.693 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00093767999999999998 2024-09-11T09:52:17.693 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:17.693 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:17.693 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:17.693 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.972813+0000", 2024-09-11T09:52:17.693 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.8448999999999999e-05 2024-09-11T09:52:17.693 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:17.693 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:17.693 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:17.693 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.972825+0000", 2024-09-11T09:52:17.693 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1837e-05 2024-09-11T09:52:17.694 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:17.694 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:17.694 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:17.694 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.972904+0000", 2024-09-11T09:52:17.694 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.8777999999999997e-05 2024-09-11T09:52:17.694 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:17.694 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:17.694 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:17.694 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.972913+0000", 2024-09-11T09:52:17.694 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.1139999999999995e-06 2024-09-11T09:52:17.694 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:17.694 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:17.695 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:17.695 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.973752+0000", 2024-09-11T09:52:17.695 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00083892400000000001 2024-09-11T09:52:17.695 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:17.695 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:17.695 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:17.695 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:17.695 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:17.695 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "MOSDPGPush(2.4 82/81 [PushOp(2:2367385a:::rbd_data.1250f5e25b80.0000000000000002:6, version: 79'169, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:2367385a:::rbd_data.1250f5e25b80.0000000000000002:6@79'169, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 6=[]:{6: [6, 5, 3]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 4194304, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3145728, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-11T09:52:17.695 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:52:08.975206+0000", 2024-09-11T09:52:17.695 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 8.6884877379999992, 2024-09-11T09:52:17.695 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0020632770000000001, 2024-09-11T09:52:17.695 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:17.696 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:17.696 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "started", 2024-09-11T09:52:17.696 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:17.696 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:17.696 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:17.696 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.975206+0000", 2024-09-11T09:52:17.696 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:17.696 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:17.696 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:17.696 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:17.696 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.975206+0000", 2024-09-11T09:52:17.696 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:17.696 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:17.697 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:17.697 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:17.697 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.975207+0000", 2024-09-11T09:52:17.697 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.2399999999999998e-07 2024-09-11T09:52:17.697 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:17.697 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:17.697 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:17.697 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.976173+0000", 2024-09-11T09:52:17.697 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00096649400000000003 2024-09-11T09:52:17.697 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:17.697 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:17.698 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:17.698 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.976189+0000", 2024-09-11T09:52:17.698 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.6223e-05 2024-09-11T09:52:17.698 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:17.698 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:17.698 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:17.698 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.976200+0000", 2024-09-11T09:52:17.698 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.0729e-05 2024-09-11T09:52:17.698 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:17.698 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:17.698 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:17.698 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.976265+0000", 2024-09-11T09:52:17.698 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.5326000000000004e-05 2024-09-11T09:52:17.698 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:17.699 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:17.699 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:17.699 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.976274+0000", 2024-09-11T09:52:17.699 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.8140000000000007e-06 2024-09-11T09:52:17.699 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:17.699 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:17.699 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:17.699 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.977269+0000", 2024-09-11T09:52:17.699 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00099506700000000009 2024-09-11T09:52:17.699 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:17.699 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:17.699 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:17.699 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:17.699 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:17.700 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "MOSDPGPush(2.4 82/81 [PushOp(2:26d5c2f0:::rbd_data.1250f5e25b80.000000000000000b:6, version: 79'171, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 1, recovery_info: ObjectRecoveryInfo(2:26d5c2f0:::rbd_data.1250f5e25b80.000000000000000b:6@79'171, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 6=[]:{6: [6, 5, 3]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-11T09:52:17.700 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:52:08.979123+0000", 2024-09-11T09:52:17.700 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 8.6845703089999997, 2024-09-11T09:52:17.700 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0019749350000000001, 2024-09-11T09:52:17.700 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:17.700 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:17.700 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "started", 2024-09-11T09:52:17.700 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:17.700 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:17.700 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:17.700 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.979123+0000", 2024-09-11T09:52:17.700 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:17.700 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:17.701 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:17.701 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:17.701 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.979123+0000", 2024-09-11T09:52:17.701 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:17.701 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:17.701 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:17.701 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:17.701 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.979124+0000", 2024-09-11T09:52:17.701 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.5799999999999999e-07 2024-09-11T09:52:17.701 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:17.701 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:17.701 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:17.702 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.980088+0000", 2024-09-11T09:52:17.702 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00096377699999999997 2024-09-11T09:52:17.702 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:17.702 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:17.702 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:17.702 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.980106+0000", 2024-09-11T09:52:17.702 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.7881e-05 2024-09-11T09:52:17.702 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:17.702 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:17.702 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:17.702 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.980117+0000", 2024-09-11T09:52:17.702 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1681e-05 2024-09-11T09:52:17.702 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:17.702 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:17.703 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:17.703 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.980189+0000", 2024-09-11T09:52:17.703 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.1365999999999994e-05 2024-09-11T09:52:17.703 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:17.703 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:17.703 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:17.703 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.980198+0000", 2024-09-11T09:52:17.703 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.3300000000000005e-06 2024-09-11T09:52:17.703 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:17.703 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:17.703 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:17.703 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.981098+0000", 2024-09-11T09:52:17.703 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00090024199999999999 2024-09-11T09:52:17.703 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:17.704 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:17.704 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:17.704 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:17.704 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:17.704 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "MOSDPGPush(2.4 82/81 [PushOp(2:26d5c2f0:::rbd_data.1250f5e25b80.000000000000000b:6, version: 79'171, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:26d5c2f0:::rbd_data.1250f5e25b80.000000000000000b:6@79'171, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 6=[]:{6: [6, 5, 3]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-11T09:52:17.704 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:52:08.982550+0000", 2024-09-11T09:52:17.704 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 8.6811438639999992, 2024-09-11T09:52:17.704 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0020071540000000001, 2024-09-11T09:52:17.704 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:17.704 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:17.704 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "started", 2024-09-11T09:52:17.704 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:17.704 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:17.705 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:17.705 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.982550+0000", 2024-09-11T09:52:17.705 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:17.705 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:17.705 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:17.705 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:17.705 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.982550+0000", 2024-09-11T09:52:17.705 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:17.705 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:17.705 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:17.705 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:17.705 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.982550+0000", 2024-09-11T09:52:17.706 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.89e-07 2024-09-11T09:52:17.706 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:17.706 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:17.706 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:17.706 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.983523+0000", 2024-09-11T09:52:17.706 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00097246400000000003 2024-09-11T09:52:17.706 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:17.706 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:17.706 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:17.706 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.983540+0000", 2024-09-11T09:52:17.706 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.7388999999999999e-05 2024-09-11T09:52:17.706 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:17.706 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:17.706 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:17.707 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.983553+0000", 2024-09-11T09:52:17.707 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.2876e-05 2024-09-11T09:52:17.707 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:17.707 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:17.707 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:17.707 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.983623+0000", 2024-09-11T09:52:17.707 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.0199999999999999e-05 2024-09-11T09:52:17.707 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:17.707 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:17.707 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:17.707 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.983632+0000", 2024-09-11T09:52:17.707 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.8859999999999993e-06 2024-09-11T09:52:17.707 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:17.707 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:17.707 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:17.708 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.984557+0000", 2024-09-11T09:52:17.708 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00092484999999999996 2024-09-11T09:52:17.708 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:17.708 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:17.708 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:17.708 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:17.708 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:17.708 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "MOSDPGPush(2.4 82/81 [PushOp(2:26d5c2f0:::rbd_data.1250f5e25b80.000000000000000b:6, version: 79'171, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:26d5c2f0:::rbd_data.1250f5e25b80.000000000000000b:6@79'171, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 6=[]:{6: [6, 5, 3]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3145728, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-11T09:52:17.708 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:52:08.985896+0000", 2024-09-11T09:52:17.708 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 8.6777972729999995, 2024-09-11T09:52:17.708 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0018739799999999999, 2024-09-11T09:52:17.708 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:17.708 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:17.709 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "started", 2024-09-11T09:52:17.709 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:17.709 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:17.709 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:17.709 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.985896+0000", 2024-09-11T09:52:17.709 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:17.709 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:17.709 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:17.709 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:17.709 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.985896+0000", 2024-09-11T09:52:17.709 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:17.709 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:17.709 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:17.710 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:17.710 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.985897+0000", 2024-09-11T09:52:17.710 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.3099999999999996e-07 2024-09-11T09:52:17.710 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:17.710 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:17.710 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:17.710 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.986883+0000", 2024-09-11T09:52:17.710 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00098564300000000002 2024-09-11T09:52:17.710 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:17.710 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:17.710 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:17.710 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.986900+0000", 2024-09-11T09:52:17.710 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.7475000000000001e-05 2024-09-11T09:52:17.710 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:17.711 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:17.711 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:17.711 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.986912+0000", 2024-09-11T09:52:17.711 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1924e-05 2024-09-11T09:52:17.711 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:17.711 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:17.711 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:17.711 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.986988+0000", 2024-09-11T09:52:17.711 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.6099000000000004e-05 2024-09-11T09:52:17.711 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:17.711 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:17.711 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:17.711 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.986998+0000", 2024-09-11T09:52:17.711 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.5710000000000004e-06 2024-09-11T09:52:17.712 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:17.712 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:17.712 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:17.712 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.987770+0000", 2024-09-11T09:52:17.712 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000772337 2024-09-11T09:52:17.712 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:17.712 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:17.712 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:17.712 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:17.712 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:17.712 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "MOSDPGPush(2.4 82/81 [PushOp(2:26d5c2f0:::rbd_data.1250f5e25b80.000000000000000b:6, version: 79'171, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:26d5c2f0:::rbd_data.1250f5e25b80.000000000000000b:6@79'171, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 6=[]:{6: [6, 5, 3]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 4194304, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3145728, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-11T09:52:17.712 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:52:08.989143+0000", 2024-09-11T09:52:17.712 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 8.6745501730000001, 2024-09-11T09:52:17.713 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0022141589999999998, 2024-09-11T09:52:17.713 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:17.713 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:17.713 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "started", 2024-09-11T09:52:17.713 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:17.713 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:17.713 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:17.713 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.989143+0000", 2024-09-11T09:52:17.713 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:17.713 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:17.713 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:17.713 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:17.713 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.989143+0000", 2024-09-11T09:52:17.714 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:17.714 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:17.714 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:17.714 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:17.714 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.989144+0000", 2024-09-11T09:52:17.714 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.1699999999999997e-07 2024-09-11T09:52:17.714 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:17.714 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:17.714 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:17.714 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.990124+0000", 2024-09-11T09:52:17.714 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00097941400000000007 2024-09-11T09:52:17.714 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:17.714 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:17.715 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:17.715 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.990140+0000", 2024-09-11T09:52:17.715 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.6583e-05 2024-09-11T09:52:17.715 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:17.715 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:17.715 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:17.715 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.990152+0000", 2024-09-11T09:52:17.715 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1331999999999999e-05 2024-09-11T09:52:17.715 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:17.715 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:17.715 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:17.715 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.990220+0000", 2024-09-11T09:52:17.715 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.8777999999999998e-05 2024-09-11T09:52:17.715 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:17.716 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:17.716 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:17.716 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.990229+0000", 2024-09-11T09:52:17.716 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.9590000000000001e-06 2024-09-11T09:52:17.716 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:17.716 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:17.716 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:17.716 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.991358+0000", 2024-09-11T09:52:17.716 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0011283759999999999 2024-09-11T09:52:17.716 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:17.716 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:17.716 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:17.716 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:17.717 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:17.717 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "MOSDPGPush(2.4 82/81 [PushOp(2:239e915b:::rbd_data.1250f5e25b80.000000000000000c:6, version: 79'173, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 1, recovery_info: ObjectRecoveryInfo(2:239e915b:::rbd_data.1250f5e25b80.000000000000000c:6@79'173, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 6=[]:{6: [6, 5, 3]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-11T09:52:17.717 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:52:08.994080+0000", 2024-09-11T09:52:17.717 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 8.6696140360000005, 2024-09-11T09:52:17.717 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0019554189999999999, 2024-09-11T09:52:17.717 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:17.717 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:17.717 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "started", 2024-09-11T09:52:17.717 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:17.717 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:17.717 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:17.717 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.994080+0000", 2024-09-11T09:52:17.717 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:17.718 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:17.718 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:17.718 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:17.718 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.994080+0000", 2024-09-11T09:52:17.718 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:17.718 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:17.718 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:17.718 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:17.718 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.994080+0000", 2024-09-11T09:52:17.718 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.7599999999999997e-07 2024-09-11T09:52:17.718 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:17.718 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:17.718 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:17.718 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.995065+0000", 2024-09-11T09:52:17.719 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00098436199999999991 2024-09-11T09:52:17.719 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:17.719 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:17.719 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:17.719 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.995080+0000", 2024-09-11T09:52:17.719 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.5119e-05 2024-09-11T09:52:17.719 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:17.719 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:17.720 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:17.720 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.995090+0000", 2024-09-11T09:52:17.720 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.0715e-05 2024-09-11T09:52:17.720 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:17.720 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:17.720 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:17.720 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.995152+0000", 2024-09-11T09:52:17.720 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.2089999999999997e-05 2024-09-11T09:52:17.720 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:17.720 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:17.720 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:17.720 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.995159+0000", 2024-09-11T09:52:17.721 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.9210000000000003e-06 2024-09-11T09:52:17.721 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:17.721 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:17.721 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:17.721 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.996035+0000", 2024-09-11T09:52:17.721 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00087563599999999995 2024-09-11T09:52:17.721 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:17.721 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:17.721 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:17.721 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:17.721 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:17.721 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "MOSDPGPush(2.4 82/81 [PushOp(2:239e915b:::rbd_data.1250f5e25b80.000000000000000c:6, version: 79'173, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:239e915b:::rbd_data.1250f5e25b80.000000000000000c:6@79'173, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 6=[]:{6: [6, 5, 3]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-11T09:52:17.722 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:52:08.997853+0000", 2024-09-11T09:52:17.722 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 8.6658410890000006, 2024-09-11T09:52:17.722 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0018792979999999999, 2024-09-11T09:52:17.722 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:17.722 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:17.722 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "started", 2024-09-11T09:52:17.722 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:17.722 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:17.722 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:17.722 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.997853+0000", 2024-09-11T09:52:17.722 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:17.722 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:17.722 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:17.722 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:17.722 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.997853+0000", 2024-09-11T09:52:17.723 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:17.723 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:17.723 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:17.723 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:17.723 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.997853+0000", 2024-09-11T09:52:17.723 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.4000000000000002e-07 2024-09-11T09:52:17.723 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:17.723 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:17.724 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:17.724 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.998831+0000", 2024-09-11T09:52:17.724 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00097792100000000009 2024-09-11T09:52:17.724 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:17.724 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:17.724 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:17.724 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.998846+0000", 2024-09-11T09:52:17.724 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.4987000000000001e-05 2024-09-11T09:52:17.724 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:17.724 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:17.724 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:17.724 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.998856+0000", 2024-09-11T09:52:17.724 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.0448000000000001e-05 2024-09-11T09:52:17.725 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:17.725 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:17.725 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:17.725 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.998942+0000", 2024-09-11T09:52:17.725 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.5563999999999996e-05 2024-09-11T09:52:17.725 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:17.725 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:17.725 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:17.725 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.998952+0000", 2024-09-11T09:52:17.725 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.5170000000000005e-06 2024-09-11T09:52:17.725 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:17.725 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:17.725 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:17.726 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.999732+0000", 2024-09-11T09:52:17.726 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00078032100000000005 2024-09-11T09:52:17.726 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:17.726 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:17.726 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:17.726 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:17.726 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:17.726 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "MOSDPGPush(2.4 82/81 [PushOp(2:239e915b:::rbd_data.1250f5e25b80.000000000000000c:6, version: 79'173, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:239e915b:::rbd_data.1250f5e25b80.000000000000000c:6@79'173, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 6=[]:{6: [6, 5, 3]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3145728, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-11T09:52:17.726 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:52:09.001584+0000", 2024-09-11T09:52:17.726 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 8.6621091690000007, 2024-09-11T09:52:17.726 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0019129119999999999, 2024-09-11T09:52:17.726 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:17.726 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:17.727 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "started", 2024-09-11T09:52:17.727 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:17.727 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:17.727 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:17.727 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.001584+0000", 2024-09-11T09:52:17.727 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:17.727 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:17.727 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:17.727 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:17.727 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.001584+0000", 2024-09-11T09:52:17.727 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:17.727 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:17.727 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:17.727 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:17.728 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.001585+0000", 2024-09-11T09:52:17.728 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.8899999999999999e-07 2024-09-11T09:52:17.728 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:17.728 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:17.728 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:17.728 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.002541+0000", 2024-09-11T09:52:17.728 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00095588800000000001 2024-09-11T09:52:17.728 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:17.728 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:17.728 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:17.728 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.002558+0000", 2024-09-11T09:52:17.728 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.6452000000000001e-05 2024-09-11T09:52:17.728 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:17.729 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:17.729 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:17.729 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.002569+0000", 2024-09-11T09:52:17.729 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.148e-05 2024-09-11T09:52:17.729 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:17.729 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:17.729 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:17.729 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.002647+0000", 2024-09-11T09:52:17.729 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.7570000000000004e-05 2024-09-11T09:52:17.729 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:17.729 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:17.729 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:17.729 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.002655+0000", 2024-09-11T09:52:17.730 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.4610000000000008e-06 2024-09-11T09:52:17.730 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:17.730 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:17.730 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:17.730 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.003497+0000", 2024-09-11T09:52:17.730 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00084237199999999998 2024-09-11T09:52:17.730 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:17.730 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:17.730 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:17.730 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:17.730 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:17.730 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "MOSDPGPush(2.4 82/81 [PushOp(2:239e915b:::rbd_data.1250f5e25b80.000000000000000c:6, version: 79'173, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:239e915b:::rbd_data.1250f5e25b80.000000000000000c:6@79'173, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 6=[]:{6: [6, 5, 3]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 4194304, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3145728, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-11T09:52:17.730 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:52:09.005094+0000", 2024-09-11T09:52:17.731 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 8.6585991749999991, 2024-09-11T09:52:17.731 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0024127419999999998, 2024-09-11T09:52:17.731 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:17.731 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:17.731 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "started", 2024-09-11T09:52:17.731 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:17.731 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:17.731 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:17.731 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.005094+0000", 2024-09-11T09:52:17.731 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:17.732 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:17.732 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:17.732 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:17.732 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.005094+0000", 2024-09-11T09:52:17.732 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:17.732 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:17.732 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:17.732 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:17.732 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.005095+0000", 2024-09-11T09:52:17.732 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.8999999999999997e-07 2024-09-11T09:52:17.732 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:17.732 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:17.732 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:17.733 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.006059+0000", 2024-09-11T09:52:17.733 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00096429100000000004 2024-09-11T09:52:17.733 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:17.733 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:17.733 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:17.733 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.006075+0000", 2024-09-11T09:52:17.733 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.6036e-05 2024-09-11T09:52:17.733 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:17.733 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:17.733 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:17.733 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.006088+0000", 2024-09-11T09:52:17.733 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.2557e-05 2024-09-11T09:52:17.733 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:17.733 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:17.734 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:17.734 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.006167+0000", 2024-09-11T09:52:17.734 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.8843000000000004e-05 2024-09-11T09:52:17.734 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:17.734 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:17.734 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:17.734 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.006182+0000", 2024-09-11T09:52:17.734 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.4968e-05 2024-09-11T09:52:17.734 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:17.734 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:17.734 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:17.734 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.007507+0000", 2024-09-11T09:52:17.734 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001325557 2024-09-11T09:52:17.735 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:17.735 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:17.735 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:17.735 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:17.735 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:17.735 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "MOSDPGPush(2.4 82/81 [PushOp(2:2167012f:::rbd_data.1250f5e25b80.000000000000000d:6, version: 79'175, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 1, recovery_info: ObjectRecoveryInfo(2:2167012f:::rbd_data.1250f5e25b80.000000000000000d:6@79'175, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 6=[]:{6: [6, 5, 3]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-11T09:52:17.735 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:52:09.009862+0000", 2024-09-11T09:52:17.735 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 8.6538317159999991, 2024-09-11T09:52:17.735 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001845522, 2024-09-11T09:52:17.735 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:17.735 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:17.735 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "started", 2024-09-11T09:52:17.735 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:17.735 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:17.736 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:17.736 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.009862+0000", 2024-09-11T09:52:17.736 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:17.736 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:17.736 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:17.736 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:17.736 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.009862+0000", 2024-09-11T09:52:17.736 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:17.736 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:17.736 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:17.736 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:17.736 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.009863+0000", 2024-09-11T09:52:17.736 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.1200000000000002e-07 2024-09-11T09:52:17.737 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:17.737 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:17.737 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:17.737 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.010797+0000", 2024-09-11T09:52:17.737 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00093439599999999999 2024-09-11T09:52:17.737 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:17.737 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:17.737 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:17.737 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.010814+0000", 2024-09-11T09:52:17.737 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.7354e-05 2024-09-11T09:52:17.738 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:17.738 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:17.738 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:17.738 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.010827+0000", 2024-09-11T09:52:17.738 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.3096e-05 2024-09-11T09:52:17.738 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:17.738 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:17.739 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:17.739 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.010894+0000", 2024-09-11T09:52:17.739 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.6501999999999994e-05 2024-09-11T09:52:17.739 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:17.739 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:17.739 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:17.739 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.010903+0000", 2024-09-11T09:52:17.740 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.0899999999999994e-06 2024-09-11T09:52:17.740 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:17.740 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:17.740 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:17.740 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.011707+0000", 2024-09-11T09:52:17.740 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00080437200000000003 2024-09-11T09:52:17.740 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:17.740 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:17.740 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:17.740 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:17.740 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:17.741 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "MOSDPGPush(2.4 82/81 [PushOp(2:2167012f:::rbd_data.1250f5e25b80.000000000000000d:6, version: 79'175, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:2167012f:::rbd_data.1250f5e25b80.000000000000000d:6@79'175, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 6=[]:{6: [6, 5, 3]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-11T09:52:17.741 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:52:09.013431+0000", 2024-09-11T09:52:17.741 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 8.650262841, 2024-09-11T09:52:17.741 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.002012247, 2024-09-11T09:52:17.741 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:17.741 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:17.741 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "started", 2024-09-11T09:52:17.741 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:17.741 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:17.741 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:17.742 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.013431+0000", 2024-09-11T09:52:17.742 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:17.742 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:17.742 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:17.742 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:17.742 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.013431+0000", 2024-09-11T09:52:17.742 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:17.742 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:17.742 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:17.743 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:17.743 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.013432+0000", 2024-09-11T09:52:17.743 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.1900000000000001e-07 2024-09-11T09:52:17.743 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:17.743 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:17.743 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:17.743 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.014359+0000", 2024-09-11T09:52:17.744 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00092687900000000003 2024-09-11T09:52:17.744 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:17.744 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:17.744 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:17.744 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.014377+0000", 2024-09-11T09:52:17.744 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.8085e-05 2024-09-11T09:52:17.744 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:17.744 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:17.745 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:17.745 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.014391+0000", 2024-09-11T09:52:17.745 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.4099e-05 2024-09-11T09:52:17.745 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:17.745 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:17.745 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:17.745 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.014476+0000", 2024-09-11T09:52:17.745 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.4827999999999995e-05 2024-09-11T09:52:17.745 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:17.745 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:17.745 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:17.745 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.014489+0000", 2024-09-11T09:52:17.746 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.3169e-05 2024-09-11T09:52:17.746 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:17.746 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:17.746 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:17.746 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.015443+0000", 2024-09-11T09:52:17.746 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00095436800000000003 2024-09-11T09:52:17.746 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:17.746 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:17.746 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:17.747 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:17.747 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:17.747 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "MOSDPGPush(2.4 82/81 [PushOp(2:2167012f:::rbd_data.1250f5e25b80.000000000000000d:6, version: 79'175, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:2167012f:::rbd_data.1250f5e25b80.000000000000000d:6@79'175, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 6=[]:{6: [6, 5, 3]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3145728, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-11T09:52:17.747 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:52:09.017158+0000", 2024-09-11T09:52:17.747 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 8.6465358529999996, 2024-09-11T09:52:17.747 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0020239889999999999, 2024-09-11T09:52:17.748 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:17.748 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:17.748 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "started", 2024-09-11T09:52:17.748 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:17.748 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:17.748 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:17.748 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.017158+0000", 2024-09-11T09:52:17.748 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:17.748 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:17.749 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:17.749 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:17.749 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.017158+0000", 2024-09-11T09:52:17.749 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:17.749 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:17.749 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:17.749 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:17.749 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.017158+0000", 2024-09-11T09:52:17.749 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.7000000000000004e-07 2024-09-11T09:52:17.749 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:17.750 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:17.750 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:17.750 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.018136+0000", 2024-09-11T09:52:17.750 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00097795 2024-09-11T09:52:17.750 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:17.750 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:17.750 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:17.750 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.018152+0000", 2024-09-11T09:52:17.750 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.5208000000000001e-05 2024-09-11T09:52:17.750 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:17.751 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:17.751 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:17.751 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.018164+0000", 2024-09-11T09:52:17.751 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.2157000000000001e-05 2024-09-11T09:52:17.751 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:17.751 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:17.751 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:17.751 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.018245+0000", 2024-09-11T09:52:17.751 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.1284000000000003e-05 2024-09-11T09:52:17.752 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:17.752 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:17.752 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:17.752 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.018257+0000", 2024-09-11T09:52:17.752 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.2160999999999999e-05 2024-09-11T09:52:17.752 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:17.752 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:17.752 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:17.752 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.019182+0000", 2024-09-11T09:52:17.752 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00092455900000000004 2024-09-11T09:52:17.753 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:17.753 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:17.753 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:17.753 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:17.753 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:17.753 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "MOSDPGPush(2.4 82/81 [PushOp(2:2167012f:::rbd_data.1250f5e25b80.000000000000000d:6, version: 79'175, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:2167012f:::rbd_data.1250f5e25b80.000000000000000d:6@79'175, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 6=[]:{6: [6, 5, 3]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 4194304, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3145728, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-11T09:52:17.753 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:52:09.020895+0000", 2024-09-11T09:52:17.753 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 8.6427987999999996, 2024-09-11T09:52:17.753 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.002075234, 2024-09-11T09:52:17.753 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:17.754 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:17.754 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "started", 2024-09-11T09:52:17.754 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:17.754 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:17.754 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:17.754 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.020895+0000", 2024-09-11T09:52:17.754 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:17.754 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:17.754 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:17.754 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:17.754 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.020895+0000", 2024-09-11T09:52:17.755 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:17.755 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:17.755 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:17.755 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:17.755 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.020895+0000", 2024-09-11T09:52:17.755 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.0999999999999999e-07 2024-09-11T09:52:17.755 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:17.756 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:17.756 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:17.756 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.021855+0000", 2024-09-11T09:52:17.756 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00095982700000000001 2024-09-11T09:52:17.756 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:17.756 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:17.756 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:17.756 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.021870+0000", 2024-09-11T09:52:17.756 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.4609e-05 2024-09-11T09:52:17.757 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:17.757 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:17.757 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:17.757 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.021881+0000", 2024-09-11T09:52:17.757 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.083e-05 2024-09-11T09:52:17.757 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:17.757 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:17.757 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:17.757 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.021943+0000", 2024-09-11T09:52:17.757 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.2780999999999994e-05 2024-09-11T09:52:17.757 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:17.758 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:17.758 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:17.758 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.021952+0000", 2024-09-11T09:52:17.758 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.8629999999999997e-06 2024-09-11T09:52:17.758 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:17.758 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:17.758 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:17.758 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.022970+0000", 2024-09-11T09:52:17.758 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0010178139999999999 2024-09-11T09:52:17.758 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:17.758 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:17.759 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:17.759 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:17.759 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:17.759 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-11T09:52:17.760 INFO:tasks.ceph.osd.2.smithi017.stderr:2024-09-11T09:52:17.761+0000 7fd20b170640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31672) UID: 0 2024-09-11T09:52:17.760 INFO:tasks.workunit.client.0.smithi107.stderr:++ rbd journal inspect --pool rbd --image testrbdnbd27121 2024-09-11T09:52:17.761 INFO:tasks.workunit.client.0.smithi107.stderr:++ awk '/entries inspected/ {print $1}' 2024-09-11T09:52:17.761 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-11T09:52:17.783 INFO:tasks.workunit.client.0.smithi107.stderr:2024-09-11T09:52:17.784+0000 7faeea6a7640 -1 JournalMetadata: 0x557d8c1459d0 failed to locate client: INSPECT 2024-09-11T09:52:17.787 INFO:tasks.workunit.client.0.smithi107.stderr:+ queue_length=4 2024-09-11T09:52:17.787 INFO:tasks.workunit.client.0.smithi107.stderr:+ tries=1 2024-09-11T09:52:17.787 INFO:tasks.workunit.client.0.smithi107.stderr:+ '[' 1 -lt 10 ']' 2024-09-11T09:52:17.788 INFO:tasks.workunit.client.0.smithi107.stderr:+ '[' 4 -gt 0 ']' 2024-09-11T09:52:17.788 INFO:tasks.workunit.client.0.smithi107.stderr:+ rbd journal inspect --pool rbd --image testrbdnbd27121 --verbose 2024-09-11T09:52:17.843 INFO:tasks.workunit.client.0.smithi107.stdout:Entry: tag_id=1, commit_tid=1 2024-09-11T09:52:17.843 INFO:tasks.workunit.client.0.smithi107.stdout:{ 2024-09-11T09:52:17.843 INFO:tasks.workunit.client.0.smithi107.stdout: "event_type": "AioDiscard", 2024-09-11T09:52:17.843 INFO:tasks.workunit.client.0.smithi107.stdout: "offset": 0, 2024-09-11T09:52:17.844 INFO:tasks.workunit.client.0.smithi107.stdout: "length": 4096, 2024-09-11T09:52:17.844 INFO:tasks.workunit.client.0.smithi107.stdout: "discard_granularity_bytes": 4096, 2024-09-11T09:52:17.844 INFO:tasks.workunit.client.0.smithi107.stdout: "timestamp": "2024-09-11T09:52:16.526805+0000" 2024-09-11T09:52:17.844 INFO:tasks.workunit.client.0.smithi107.stdout:} 2024-09-11T09:52:17.844 INFO:tasks.workunit.client.0.smithi107.stdout:Entry: tag_id=1, commit_tid=2 2024-09-11T09:52:17.844 INFO:tasks.workunit.client.0.smithi107.stdout:{ 2024-09-11T09:52:17.844 INFO:tasks.workunit.client.0.smithi107.stdout: "event_type": "AioDiscard", 2024-09-11T09:52:17.844 INFO:tasks.workunit.client.0.smithi107.stdout: "offset": 0, 2024-09-11T09:52:17.844 INFO:tasks.workunit.client.0.smithi107.stdout: "length": 4096, 2024-09-11T09:52:17.844 INFO:tasks.workunit.client.0.smithi107.stdout: "discard_granularity_bytes": 4096, 2024-09-11T09:52:17.844 INFO:tasks.workunit.client.0.smithi107.stdout: "timestamp": "2024-09-11T09:52:16.544819+0000" 2024-09-11T09:52:17.844 INFO:tasks.workunit.client.0.smithi107.stdout:} 2024-09-11T09:52:17.844 INFO:tasks.workunit.client.0.smithi107.stdout:Entry: tag_id=1, commit_tid=3 2024-09-11T09:52:17.845 INFO:tasks.workunit.client.0.smithi107.stdout:{ 2024-09-11T09:52:17.845 INFO:tasks.workunit.client.0.smithi107.stdout: "event_type": "AioDiscard", 2024-09-11T09:52:17.845 INFO:tasks.workunit.client.0.smithi107.stdout: "offset": 4096, 2024-09-11T09:52:17.845 INFO:tasks.workunit.client.0.smithi107.stdout: "length": 4096, 2024-09-11T09:52:17.845 INFO:tasks.workunit.client.0.smithi107.stdout: "discard_granularity_bytes": 4096, 2024-09-11T09:52:17.845 INFO:tasks.workunit.client.0.smithi107.stdout: "timestamp": "2024-09-11T09:52:16.576535+0000" 2024-09-11T09:52:17.845 INFO:tasks.workunit.client.0.smithi107.stdout:} 2024-09-11T09:52:17.845 INFO:tasks.workunit.client.0.smithi107.stdout:Entry: tag_id=1, commit_tid=4 2024-09-11T09:52:17.845 INFO:tasks.workunit.client.0.smithi107.stdout:{ 2024-09-11T09:52:17.845 INFO:tasks.workunit.client.0.smithi107.stdout: "event_type": "AioDiscard", 2024-09-11T09:52:17.845 INFO:tasks.workunit.client.0.smithi107.stdout: "offset": 4096, 2024-09-11T09:52:17.845 INFO:tasks.workunit.client.0.smithi107.stdout: "length": 4096, 2024-09-11T09:52:17.845 INFO:tasks.workunit.client.0.smithi107.stdout: "discard_granularity_bytes": 4096, 2024-09-11T09:52:17.845 INFO:tasks.workunit.client.0.smithi107.stdout: "timestamp": "2024-09-11T09:52:16.593859+0000" 2024-09-11T09:52:17.846 INFO:tasks.workunit.client.0.smithi107.stdout:} 2024-09-11T09:52:17.846 INFO:tasks.workunit.client.0.smithi107.stdout:Summary: 2024-09-11T09:52:17.846 INFO:tasks.workunit.client.0.smithi107.stdout: 4 entries inspected, 0 errors 2024-09-11T09:52:17.846 INFO:tasks.workunit.client.0.smithi107.stderr:2024-09-11T09:52:17.847+0000 7fb3096a7640 -1 JournalMetadata: 0x55db3636b7c0 failed to locate client: INSPECT 2024-09-11T09:52:17.850 INFO:tasks.workunit.client.0.smithi107.stderr:+ sleep 1 2024-09-11T09:52:17.861 INFO:tasks.ceph.osd.0.smithi017.stderr:2024-09-11T09:52:17.862+0000 7fb73c0f1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31665) UID: 0 2024-09-11T09:52:17.878 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:52:17.879 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [], 2024-09-11T09:52:17.879 INFO:teuthology.orchestra.run.smithi017.stdout: "num_ops": 0 2024-09-11T09:52:17.879 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:52:17.887 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-11T09:52:17.888 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-11T09:52:17.962 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:52:17.963+0000 7f3a7b90a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 42475) UID: 0 2024-09-11T09:52:18.037 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:52:18.037 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [], 2024-09-11T09:52:18.037 INFO:teuthology.orchestra.run.smithi017.stdout: "complaint_time": 30, 2024-09-11T09:52:18.037 INFO:teuthology.orchestra.run.smithi017.stdout: "num_blocked_ops": 0 2024-09-11T09:52:18.037 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:52:18.048 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-11T09:52:18.062 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:52:18.063+0000 7f3a7b90a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 42475) UID: 0 2024-09-11T09:52:18.157 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:52:18.157 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 20, 2024-09-11T09:52:18.157 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 600, 2024-09-11T09:52:18.157 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [ 2024-09-11T09:52:18.157 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.157 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4688.0:34 2.4 e53/32 2:2367385a:::rbd_data.1250f5e25b80.0000000000000002:head v 53'6, mlcod=53'4)", 2024-09-11T09:52:18.157 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.907415+0000", 2024-09-11T09:52:18.158 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 55.249082178999998, 2024-09-11T09:52:18.158 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.011285778999999999, 2024-09-11T09:52:18.158 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:18.158 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:18.158 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:18.158 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:18.158 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.158 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:18.158 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.907415+0000", 2024-09-11T09:52:18.158 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:18.158 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.158 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.158 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:18.159 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.907415+0000", 2024-09-11T09:52:18.159 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:18.159 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.159 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.159 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:18.159 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.907417+0000", 2024-09-11T09:52:18.159 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.2910000000000002e-06 2024-09-11T09:52:18.159 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.159 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.159 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:18.159 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.910477+0000", 2024-09-11T09:52:18.159 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0030597829999999999 2024-09-11T09:52:18.159 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.160 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.160 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:18.160 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.910501+0000", 2024-09-11T09:52:18.160 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.3332e-05 2024-09-11T09:52:18.160 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.160 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.160 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:18.160 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.910514+0000", 2024-09-11T09:52:18.160 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3957e-05 2024-09-11T09:52:18.160 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.160 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.160 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:18.160 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.910612+0000", 2024-09-11T09:52:18.161 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.7075000000000001e-05 2024-09-11T09:52:18.161 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.161 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.161 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:18.161 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.910640+0000", 2024-09-11T09:52:18.161 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.8768999999999998e-05 2024-09-11T09:52:18.161 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.161 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.161 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:18.161 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.918640+0000", 2024-09-11T09:52:18.161 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0079991899999999998 2024-09-11T09:52:18.161 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.161 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.162 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:18.162 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.918701+0000", 2024-09-11T09:52:18.162 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.1382000000000003e-05 2024-09-11T09:52:18.162 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:18.162 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:18.162 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:18.162 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.162 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.162 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4688.0:38 2.5 e53/32 2:abde2694:::rbd_data.1250f5e25b80.0000000000000004:head v 53'1, mlcod=0'0)", 2024-09-11T09:52:18.162 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.915432+0000", 2024-09-11T09:52:18.162 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 55.241065798999998, 2024-09-11T09:52:18.162 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.015849652999999998, 2024-09-11T09:52:18.162 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:18.162 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:18.163 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:18.163 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:18.163 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.163 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:18.163 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.915432+0000", 2024-09-11T09:52:18.163 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:18.163 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.163 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.163 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:18.163 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.915432+0000", 2024-09-11T09:52:18.163 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:18.163 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.163 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.164 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:18.164 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.915432+0000", 2024-09-11T09:52:18.164 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.91e-07 2024-09-11T09:52:18.164 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.164 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.164 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:18.164 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.924943+0000", 2024-09-11T09:52:18.164 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0095110630000000002 2024-09-11T09:52:18.164 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.164 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.164 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:18.164 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.924960+0000", 2024-09-11T09:52:18.164 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6915999999999999e-05 2024-09-11T09:52:18.165 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.165 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.165 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:18.165 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.924968+0000", 2024-09-11T09:52:18.165 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.6999999999999997e-06 2024-09-11T09:52:18.165 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.165 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.165 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:18.165 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.925025+0000", 2024-09-11T09:52:18.165 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.6949e-05 2024-09-11T09:52:18.165 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.165 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.165 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:18.165 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.925043+0000", 2024-09-11T09:52:18.166 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7784000000000001e-05 2024-09-11T09:52:18.166 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.166 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.166 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:18.166 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.931237+0000", 2024-09-11T09:52:18.166 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0061935389999999996 2024-09-11T09:52:18.166 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.166 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.166 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:18.166 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.931281+0000", 2024-09-11T09:52:18.166 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.4511000000000001e-05 2024-09-11T09:52:18.166 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:18.166 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:18.167 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:18.167 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.167 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.167 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:40 2.7 2:fe41411c:::rbd_object_map.1250f5e25b80:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:18.167 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.919652+0000", 2024-09-11T09:52:18.167 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 55.236845131999999, 2024-09-11T09:52:18.167 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.012187073, 2024-09-11T09:52:18.167 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:18.167 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:18.167 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:18.167 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:18.167 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:52:18.167 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:52:18.167 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 40 2024-09-11T09:52:18.168 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.168 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:18.168 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.168 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:18.168 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919652+0000", 2024-09-11T09:52:18.168 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:18.168 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.168 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.168 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:18.168 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919652+0000", 2024-09-11T09:52:18.168 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:18.168 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.168 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.168 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:18.169 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919654+0000", 2024-09-11T09:52:18.169 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.88e-06 2024-09-11T09:52:18.169 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.169 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.169 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:18.169 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919669+0000", 2024-09-11T09:52:18.169 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.5271e-05 2024-09-11T09:52:18.169 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.169 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.169 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:18.169 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919687+0000", 2024-09-11T09:52:18.169 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7448999999999998e-05 2024-09-11T09:52:18.169 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.170 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.170 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:18.170 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919695+0000", 2024-09-11T09:52:18.170 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.1149999999999994e-06 2024-09-11T09:52:18.170 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.170 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.170 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:18.170 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919757+0000", 2024-09-11T09:52:18.170 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.2348999999999995e-05 2024-09-11T09:52:18.170 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.170 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.170 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:18.170 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919875+0000", 2024-09-11T09:52:18.171 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00011750499999999999 2024-09-11T09:52:18.171 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.171 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.171 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 5", 2024-09-11T09:52:18.171 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.920201+0000", 2024-09-11T09:52:18.171 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00032628099999999999 2024-09-11T09:52:18.171 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.171 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.171 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:18.171 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.922089+0000", 2024-09-11T09:52:18.171 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.001887978 2024-09-11T09:52:18.171 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.171 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.171 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:18.172 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.931752+0000", 2024-09-11T09:52:18.172 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0096626309999999997 2024-09-11T09:52:18.172 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.172 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.172 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:18.172 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.931794+0000", 2024-09-11T09:52:18.172 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.2849999999999998e-05 2024-09-11T09:52:18.172 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.172 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.172 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:18.172 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.931839+0000", 2024-09-11T09:52:18.172 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.4764e-05 2024-09-11T09:52:18.172 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:18.173 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:18.173 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:18.173 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.173 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.173 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:41 2.7 2:fe41411c:::rbd_object_map.1250f5e25b80:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:18.173 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.919716+0000", 2024-09-11T09:52:18.173 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 55.236781288000003, 2024-09-11T09:52:18.173 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.015641605999999999, 2024-09-11T09:52:18.173 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:18.173 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:18.173 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:18.173 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:18.173 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:52:18.174 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:52:18.174 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 41 2024-09-11T09:52:18.174 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.174 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:18.174 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.174 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:18.174 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919716+0000", 2024-09-11T09:52:18.174 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:18.174 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.174 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.174 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:18.174 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919716+0000", 2024-09-11T09:52:18.174 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:18.174 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.175 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.175 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:18.175 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919716+0000", 2024-09-11T09:52:18.175 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.9999999999999998e-07 2024-09-11T09:52:18.175 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.175 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.175 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:18.175 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919720+0000", 2024-09-11T09:52:18.175 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.6440000000000002e-06 2024-09-11T09:52:18.175 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.175 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.175 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:18.175 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919729+0000", 2024-09-11T09:52:18.175 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.1759999999999999e-06 2024-09-11T09:52:18.176 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.176 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.176 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:18.176 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919736+0000", 2024-09-11T09:52:18.176 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.5390000000000004e-06 2024-09-11T09:52:18.176 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.176 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.176 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:18.176 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.920669+0000", 2024-09-11T09:52:18.176 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00093275599999999997 2024-09-11T09:52:18.176 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.176 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.176 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:18.177 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.920797+0000", 2024-09-11T09:52:18.177 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00012893400000000001 2024-09-11T09:52:18.177 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.177 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.177 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 5", 2024-09-11T09:52:18.177 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.921269+0000", 2024-09-11T09:52:18.177 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00047128600000000001 2024-09-11T09:52:18.177 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.177 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.177 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:18.177 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.923198+0000", 2024-09-11T09:52:18.177 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0019289369999999999 2024-09-11T09:52:18.177 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.178 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.178 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:18.178 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.935300+0000", 2024-09-11T09:52:18.178 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.012102392 2024-09-11T09:52:18.178 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.178 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.178 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:18.178 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.935332+0000", 2024-09-11T09:52:18.178 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.2292000000000002e-05 2024-09-11T09:52:18.178 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.178 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.178 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:18.178 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.935358+0000", 2024-09-11T09:52:18.178 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.525e-05 2024-09-11T09:52:18.179 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:18.179 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:18.179 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:18.179 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.179 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.179 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:48 2.7 2:fe41411c:::rbd_object_map.1250f5e25b80:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:18.179 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.938799+0000", 2024-09-11T09:52:18.179 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 55.217698179000003, 2024-09-11T09:52:18.179 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0041642340000000002, 2024-09-11T09:52:18.179 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:18.179 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:18.179 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:18.179 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:18.180 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:52:18.180 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:52:18.180 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 48 2024-09-11T09:52:18.180 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.180 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:18.180 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.180 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:18.180 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.938799+0000", 2024-09-11T09:52:18.180 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:18.180 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.180 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.180 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:18.180 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.938799+0000", 2024-09-11T09:52:18.180 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:18.181 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.181 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.181 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:18.181 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.938800+0000", 2024-09-11T09:52:18.181 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.71e-07 2024-09-11T09:52:18.181 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.181 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.181 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:18.181 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.938805+0000", 2024-09-11T09:52:18.181 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.3140000000000003e-06 2024-09-11T09:52:18.181 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.181 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.181 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:18.181 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.938816+0000", 2024-09-11T09:52:18.182 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0231000000000001e-05 2024-09-11T09:52:18.182 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.182 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.182 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:18.182 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.938822+0000", 2024-09-11T09:52:18.182 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.9550000000000003e-06 2024-09-11T09:52:18.182 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.182 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.182 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:18.182 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.938868+0000", 2024-09-11T09:52:18.182 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.5562999999999997e-05 2024-09-11T09:52:18.182 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.182 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.183 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:18.183 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.938918+0000", 2024-09-11T09:52:18.183 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.0317e-05 2024-09-11T09:52:18.183 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.183 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.183 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 5", 2024-09-11T09:52:18.183 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.939114+0000", 2024-09-11T09:52:18.183 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00019538300000000001 2024-09-11T09:52:18.183 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.183 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.183 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:18.183 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.939663+0000", 2024-09-11T09:52:18.183 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00054932199999999998 2024-09-11T09:52:18.183 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.184 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.184 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:18.184 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.942910+0000", 2024-09-11T09:52:18.184 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0032473839999999999 2024-09-11T09:52:18.184 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.184 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.184 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:18.184 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.942940+0000", 2024-09-11T09:52:18.184 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.9306999999999999e-05 2024-09-11T09:52:18.184 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.184 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.184 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:18.184 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.942963+0000", 2024-09-11T09:52:18.184 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.3587e-05 2024-09-11T09:52:18.185 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:18.185 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:18.185 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:18.185 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.185 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.185 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4688.0:49 2.5 e53/32 2:b6dd9fc9:::rbd_data.1250f5e25b80.0000000000000009:head v 53'2, mlcod=0'0)", 2024-09-11T09:52:18.185 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.944712+0000", 2024-09-11T09:52:18.185 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 55.211785339999999, 2024-09-11T09:52:18.185 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.016185069999999999, 2024-09-11T09:52:18.185 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:18.185 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:18.185 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:18.185 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:18.186 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.186 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:18.186 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.944712+0000", 2024-09-11T09:52:18.186 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:18.186 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.186 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.186 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:18.186 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.944712+0000", 2024-09-11T09:52:18.186 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:18.186 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.186 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.186 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:18.186 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.944713+0000", 2024-09-11T09:52:18.187 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0139999999999999e-06 2024-09-11T09:52:18.187 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.187 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.187 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:18.187 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.955518+0000", 2024-09-11T09:52:18.187 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.01080542 2024-09-11T09:52:18.187 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.187 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.187 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:18.187 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.955536+0000", 2024-09-11T09:52:18.187 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7818000000000001e-05 2024-09-11T09:52:18.187 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.187 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.188 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:18.188 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.955544+0000", 2024-09-11T09:52:18.188 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.1699999999999997e-06 2024-09-11T09:52:18.188 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.188 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.188 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:18.188 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.955607+0000", 2024-09-11T09:52:18.188 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.2451000000000002e-05 2024-09-11T09:52:18.188 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.188 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.188 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:18.188 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.955632+0000", 2024-09-11T09:52:18.188 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.5335000000000001e-05 2024-09-11T09:52:18.188 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.189 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.189 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:18.189 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.960826+0000", 2024-09-11T09:52:18.189 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0051938560000000002 2024-09-11T09:52:18.189 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.189 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.189 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:18.189 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.960897+0000", 2024-09-11T09:52:18.189 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.1006000000000004e-05 2024-09-11T09:52:18.189 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:18.189 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:18.189 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:18.189 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.190 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.190 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:53 2.7 2:fe41411c:::rbd_object_map.1250f5e25b80:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:18.190 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.951291+0000", 2024-09-11T09:52:18.190 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 55.205206349999997, 2024-09-11T09:52:18.190 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0044964999999999996, 2024-09-11T09:52:18.190 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:18.190 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:18.190 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:18.190 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:18.190 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:52:18.190 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:52:18.190 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 53 2024-09-11T09:52:18.190 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.191 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:18.191 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.191 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:18.191 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951291+0000", 2024-09-11T09:52:18.191 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:18.191 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.191 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.191 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:18.191 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951291+0000", 2024-09-11T09:52:18.191 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:18.191 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.191 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.191 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:18.191 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951293+0000", 2024-09-11T09:52:18.192 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7889999999999999e-06 2024-09-11T09:52:18.192 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.192 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.192 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:18.192 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951302+0000", 2024-09-11T09:52:18.192 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.9819999999999997e-06 2024-09-11T09:52:18.192 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.192 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.192 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:18.192 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951324+0000", 2024-09-11T09:52:18.192 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.2141e-05 2024-09-11T09:52:18.192 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.192 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.193 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:18.193 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951338+0000", 2024-09-11T09:52:18.193 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3750999999999999e-05 2024-09-11T09:52:18.193 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.193 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.193 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:18.193 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951409+0000", 2024-09-11T09:52:18.193 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.1658999999999999e-05 2024-09-11T09:52:18.193 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.193 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.193 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:18.193 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951517+0000", 2024-09-11T09:52:18.193 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00010746799999999999 2024-09-11T09:52:18.193 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.194 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.194 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 5", 2024-09-11T09:52:18.194 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951942+0000", 2024-09-11T09:52:18.194 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00042524300000000001 2024-09-11T09:52:18.194 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.194 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.194 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:18.194 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.952872+0000", 2024-09-11T09:52:18.194 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00092976300000000001 2024-09-11T09:52:18.194 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.194 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.194 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:18.194 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.955682+0000", 2024-09-11T09:52:18.195 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0028099459999999998 2024-09-11T09:52:18.195 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.195 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.195 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:18.195 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.955746+0000", 2024-09-11T09:52:18.195 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.4084999999999994e-05 2024-09-11T09:52:18.195 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.195 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.195 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:18.195 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.955787+0000", 2024-09-11T09:52:18.195 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.1672999999999999e-05 2024-09-11T09:52:18.195 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:18.195 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:18.196 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:18.196 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.196 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.196 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4688.0:52 2.4 e53/32 2:26d5c2f0:::rbd_data.1250f5e25b80.000000000000000b:head v 53'8, mlcod=53'6)", 2024-09-11T09:52:18.196 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.963677+0000", 2024-09-11T09:52:18.196 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 55.192820795000003, 2024-09-11T09:52:18.196 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0089437969999999999, 2024-09-11T09:52:18.196 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:18.196 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:18.196 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:18.196 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:18.196 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.196 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:18.196 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.963677+0000", 2024-09-11T09:52:18.197 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:18.197 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.197 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.197 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:18.197 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.963677+0000", 2024-09-11T09:52:18.197 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:18.197 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.197 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.197 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:18.197 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.963677+0000", 2024-09-11T09:52:18.197 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.7199999999999998e-07 2024-09-11T09:52:18.197 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.197 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.197 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:18.198 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.965506+0000", 2024-09-11T09:52:18.198 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.001829064 2024-09-11T09:52:18.198 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.198 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.198 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:18.198 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.965538+0000", 2024-09-11T09:52:18.198 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.1306999999999997e-05 2024-09-11T09:52:18.198 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.198 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.198 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:18.198 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.965553+0000", 2024-09-11T09:52:18.198 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.4924e-05 2024-09-11T09:52:18.198 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.198 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.199 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:18.199 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.965663+0000", 2024-09-11T09:52:18.199 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000110315 2024-09-11T09:52:18.199 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.199 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.199 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:18.199 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.965692+0000", 2024-09-11T09:52:18.199 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.934e-05 2024-09-11T09:52:18.199 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.199 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.199 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:18.199 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.972537+0000", 2024-09-11T09:52:18.199 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0068444969999999997 2024-09-11T09:52:18.199 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.200 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.200 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:18.200 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.972620+0000", 2024-09-11T09:52:18.200 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.3578000000000005e-05 2024-09-11T09:52:18.200 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:18.200 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:18.200 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:18.200 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.200 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.200 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4688.0:54 2.4 e53/32 2:239e915b:::rbd_data.1250f5e25b80.000000000000000c:head v 53'9, mlcod=53'7)", 2024-09-11T09:52:18.200 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.987028+0000", 2024-09-11T09:52:18.200 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 55.169468878000004, 2024-09-11T09:52:18.200 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0094155650000000007, 2024-09-11T09:52:18.201 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:18.201 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:18.201 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:18.201 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:18.201 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.201 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:18.201 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.987028+0000", 2024-09-11T09:52:18.201 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:18.201 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.201 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.201 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:18.201 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.987028+0000", 2024-09-11T09:52:18.201 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:18.201 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.202 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.202 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:18.202 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.987030+0000", 2024-09-11T09:52:18.202 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6130000000000001e-06 2024-09-11T09:52:18.202 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.202 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.202 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:18.202 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.988575+0000", 2024-09-11T09:52:18.202 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.001545193 2024-09-11T09:52:18.202 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.202 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.202 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:18.202 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.988604+0000", 2024-09-11T09:52:18.202 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.9173e-05 2024-09-11T09:52:18.203 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.203 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.203 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:18.203 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.988620+0000", 2024-09-11T09:52:18.203 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.5517999999999999e-05 2024-09-11T09:52:18.203 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.203 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.203 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:18.203 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.988704+0000", 2024-09-11T09:52:18.203 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.3588e-05 2024-09-11T09:52:18.203 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.203 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.203 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:18.204 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.988735+0000", 2024-09-11T09:52:18.204 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.1643000000000002e-05 2024-09-11T09:52:18.204 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.204 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.204 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:18.204 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.996383+0000", 2024-09-11T09:52:18.204 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0076478099999999997 2024-09-11T09:52:18.204 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.204 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.204 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:18.204 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.996444+0000", 2024-09-11T09:52:18.204 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.1026999999999997e-05 2024-09-11T09:52:18.204 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:18.205 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:18.205 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:18.205 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.205 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.205 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4688.0:56 2.4 e53/32 2:2167012f:::rbd_data.1250f5e25b80.000000000000000d:head v 53'10, mlcod=53'7)", 2024-09-11T09:52:18.205 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.993894+0000", 2024-09-11T09:52:18.205 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 55.162603335999997, 2024-09-11T09:52:18.205 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0099268559999999995, 2024-09-11T09:52:18.205 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:18.205 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:18.205 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:18.205 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:18.205 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.205 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:18.206 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.993894+0000", 2024-09-11T09:52:18.206 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:18.206 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.206 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.206 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:18.206 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.993894+0000", 2024-09-11T09:52:18.206 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:18.206 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.206 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.206 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:18.206 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.993896+0000", 2024-09-11T09:52:18.206 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6249999999999999e-06 2024-09-11T09:52:18.206 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.206 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.207 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:18.207 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.997606+0000", 2024-09-11T09:52:18.207 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.003710737 2024-09-11T09:52:18.207 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.207 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.207 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:18.207 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.997628+0000", 2024-09-11T09:52:18.207 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.1971e-05 2024-09-11T09:52:18.207 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.207 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.207 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:18.207 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.997640+0000", 2024-09-11T09:52:18.207 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1481000000000001e-05 2024-09-11T09:52:18.207 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.207 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.208 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:18.208 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.997712+0000", 2024-09-11T09:52:18.208 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.2255000000000005e-05 2024-09-11T09:52:18.208 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.208 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.208 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:18.208 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.997738+0000", 2024-09-11T09:52:18.208 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.6050999999999998e-05 2024-09-11T09:52:18.208 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.208 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.208 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:18.208 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.003779+0000", 2024-09-11T09:52:18.208 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0060410960000000001 2024-09-11T09:52:18.208 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.209 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.209 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:18.209 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.003821+0000", 2024-09-11T09:52:18.209 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.1640000000000001e-05 2024-09-11T09:52:18.209 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:18.209 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:18.209 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:18.209 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.209 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.209 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "pg_update_log_missing(2.4 epoch 53/32 rep_tid 16 entries 53'11 (0'0) error 2:2906e6a7:::rbd_header.1250f5e25b80:head by client.4688.0:59 0.000000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 53'7)", 2024-09-11T09:52:18.209 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.998956+0000", 2024-09-11T09:52:18.209 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 55.157541166999998, 2024-09-11T09:52:18.209 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.004934127, 2024-09-11T09:52:18.210 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:18.210 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:18.210 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "reached pg", 2024-09-11T09:52:18.210 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:18.210 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.210 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:18.210 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.998956+0000", 2024-09-11T09:52:18.210 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:18.210 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.210 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.210 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:18.210 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.998956+0000", 2024-09-11T09:52:18.210 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:18.210 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.211 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.211 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:18.211 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.998957+0000", 2024-09-11T09:52:18.211 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.139e-06 2024-09-11T09:52:18.211 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.211 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.211 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:18.211 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.998985+0000", 2024-09-11T09:52:18.211 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.7912e-05 2024-09-11T09:52:18.211 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.211 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.211 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:18.211 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.999011+0000", 2024-09-11T09:52:18.211 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.6061e-05 2024-09-11T09:52:18.212 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.212 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.212 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:18.212 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.999028+0000", 2024-09-11T09:52:18.212 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7065999999999999e-05 2024-09-11T09:52:18.212 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.212 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.212 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:18.212 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.001919+0000", 2024-09-11T09:52:18.212 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.002891167 2024-09-11T09:52:18.212 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.212 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.212 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:18.212 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.003890+0000", 2024-09-11T09:52:18.213 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0019707819999999999 2024-09-11T09:52:18.213 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:18.213 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:18.213 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:18.213 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.213 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.213 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4694.0:34 2.7 2:f80bfc0c:ns::rbd_data.1256a0c02879.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:18.213 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.085183+0000", 2024-09-11T09:52:18.213 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 55.071314653999998, 2024-09-11T09:52:18.213 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.020908599, 2024-09-11T09:52:18.213 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:18.213 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:18.213 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:18.214 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:18.214 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4694", 2024-09-11T09:52:18.214 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:52:18.214 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 34 2024-09-11T09:52:18.214 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.214 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:18.214 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.214 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:18.214 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.085183+0000", 2024-09-11T09:52:18.214 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:18.214 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.214 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.214 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:18.215 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.085183+0000", 2024-09-11T09:52:18.215 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:18.215 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.215 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.215 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:18.215 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.085183+0000", 2024-09-11T09:52:18.215 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.4300000000000002e-07 2024-09-11T09:52:18.215 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.215 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.215 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:18.215 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094337+0000", 2024-09-11T09:52:18.215 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0091532769999999996 2024-09-11T09:52:18.215 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.215 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.216 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:18.216 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094362+0000", 2024-09-11T09:52:18.216 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.5287999999999999e-05 2024-09-11T09:52:18.216 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.216 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.216 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:18.216 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094375+0000", 2024-09-11T09:52:18.216 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2446000000000001e-05 2024-09-11T09:52:18.216 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.216 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.216 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:18.216 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094455+0000", 2024-09-11T09:52:18.216 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.0092999999999994e-05 2024-09-11T09:52:18.217 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.217 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.217 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:18.217 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094618+0000", 2024-09-11T09:52:18.217 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000163747 2024-09-11T09:52:18.217 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.217 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.217 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 5", 2024-09-11T09:52:18.217 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094783+0000", 2024-09-11T09:52:18.217 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000165038 2024-09-11T09:52:18.217 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.217 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.217 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:18.218 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.105853+0000", 2024-09-11T09:52:18.218 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.011069294 2024-09-11T09:52:18.218 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.218 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.218 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:18.218 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.106004+0000", 2024-09-11T09:52:18.218 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00015123900000000001 2024-09-11T09:52:18.218 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.218 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.218 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:18.218 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.106062+0000", 2024-09-11T09:52:18.218 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.8205000000000003e-05 2024-09-11T09:52:18.218 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.218 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.219 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:18.219 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.106091+0000", 2024-09-11T09:52:18.219 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.9128999999999998e-05 2024-09-11T09:52:18.219 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:18.219 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:18.219 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:18.219 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.219 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.219 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4694.0:35 2.7 2:e8766325:ns::rbd_object_map.1256a0c02879:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:18.219 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.094405+0000", 2024-09-11T09:52:18.219 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 55.062092018000001, 2024-09-11T09:52:18.219 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.012623564, 2024-09-11T09:52:18.219 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:18.220 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:18.220 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:18.220 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:18.220 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4694", 2024-09-11T09:52:18.220 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:52:18.220 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 35 2024-09-11T09:52:18.220 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.220 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:18.220 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.220 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:18.220 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094405+0000", 2024-09-11T09:52:18.220 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:18.220 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.221 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.221 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:18.221 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094405+0000", 2024-09-11T09:52:18.221 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:18.221 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.221 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.221 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:18.221 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094407+0000", 2024-09-11T09:52:18.221 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.739e-06 2024-09-11T09:52:18.221 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.221 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.221 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:18.221 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094411+0000", 2024-09-11T09:52:18.221 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.3340000000000003e-06 2024-09-11T09:52:18.222 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.222 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.222 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:18.222 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094419+0000", 2024-09-11T09:52:18.222 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.8909999999999995e-06 2024-09-11T09:52:18.222 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.222 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.222 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:18.222 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094424+0000", 2024-09-11T09:52:18.222 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.7639999999999996e-06 2024-09-11T09:52:18.222 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.222 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.222 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:18.222 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.100209+0000", 2024-09-11T09:52:18.223 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0057850489999999996 2024-09-11T09:52:18.223 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.223 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.223 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:18.223 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.100341+0000", 2024-09-11T09:52:18.223 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000131438 2024-09-11T09:52:18.223 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.223 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.223 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 5", 2024-09-11T09:52:18.223 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.100805+0000", 2024-09-11T09:52:18.223 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00046471499999999998 2024-09-11T09:52:18.223 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.223 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.223 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:18.224 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.106148+0000", 2024-09-11T09:52:18.224 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0053426460000000004 2024-09-11T09:52:18.224 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.224 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.224 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:18.224 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.106929+0000", 2024-09-11T09:52:18.224 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00078161300000000001 2024-09-11T09:52:18.224 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.224 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.224 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:18.224 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.106985+0000", 2024-09-11T09:52:18.224 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.5223999999999998e-05 2024-09-11T09:52:18.224 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.225 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.225 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:18.225 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.107029+0000", 2024-09-11T09:52:18.225 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.4150999999999998e-05 2024-09-11T09:52:18.225 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:18.225 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:18.225 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:18.225 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.225 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.225 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4694.0:36 2.7 2:e8766325:ns::rbd_object_map.1256a0c02879:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:18.225 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.094452+0000", 2024-09-11T09:52:18.225 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 55.062045007999998, 2024-09-11T09:52:18.225 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.012680383, 2024-09-11T09:52:18.225 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:18.226 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:18.226 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:18.226 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:18.226 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4694", 2024-09-11T09:52:18.226 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:52:18.226 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 36 2024-09-11T09:52:18.226 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.226 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:18.226 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.226 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:18.226 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094452+0000", 2024-09-11T09:52:18.226 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:18.226 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.227 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.227 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:18.227 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094452+0000", 2024-09-11T09:52:18.227 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:18.227 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.227 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.227 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:18.227 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094453+0000", 2024-09-11T09:52:18.227 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.1800000000000002e-07 2024-09-11T09:52:18.227 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.227 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.227 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:18.227 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094455+0000", 2024-09-11T09:52:18.227 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.7020000000000001e-06 2024-09-11T09:52:18.228 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.228 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.228 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:18.228 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094463+0000", 2024-09-11T09:52:18.228 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.1999999999999997e-06 2024-09-11T09:52:18.228 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.228 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.228 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:18.228 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094467+0000", 2024-09-11T09:52:18.228 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.729e-06 2024-09-11T09:52:18.228 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.228 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.228 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:18.229 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.101269+0000", 2024-09-11T09:52:18.229 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0068018799999999997 2024-09-11T09:52:18.229 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.229 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.229 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:18.229 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.101353+0000", 2024-09-11T09:52:18.229 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.3827999999999998e-05 2024-09-11T09:52:18.229 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.229 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.229 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 5", 2024-09-11T09:52:18.229 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.101588+0000", 2024-09-11T09:52:18.229 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00023467099999999999 2024-09-11T09:52:18.229 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.230 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.230 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:18.230 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.106182+0000", 2024-09-11T09:52:18.230 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0045939489999999999 2024-09-11T09:52:18.230 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.230 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.230 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:18.230 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.107041+0000", 2024-09-11T09:52:18.230 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00085965800000000002 2024-09-11T09:52:18.230 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.230 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.230 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:18.230 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.107084+0000", 2024-09-11T09:52:18.230 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.261e-05 2024-09-11T09:52:18.231 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.231 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.231 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:18.231 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.107133+0000", 2024-09-11T09:52:18.231 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.8838000000000002e-05 2024-09-11T09:52:18.231 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:18.231 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:18.231 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:18.231 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.231 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.231 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4694.0:37 2.7 2:e8766325:ns::rbd_object_map.1256a0c02879:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:18.232 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.094473+0000", 2024-09-11T09:52:18.232 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 55.062023857, 2024-09-11T09:52:18.232 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.012778484999999999, 2024-09-11T09:52:18.232 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:18.232 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:18.232 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:18.232 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:18.232 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4694", 2024-09-11T09:52:18.232 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:52:18.232 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 37 2024-09-11T09:52:18.232 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.232 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:18.233 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.233 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:18.233 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094473+0000", 2024-09-11T09:52:18.233 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:18.233 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.233 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.233 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:18.233 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094473+0000", 2024-09-11T09:52:18.233 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:18.233 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.233 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.233 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:18.233 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094474+0000", 2024-09-11T09:52:18.233 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.9699999999999996e-07 2024-09-11T09:52:18.234 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.234 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.234 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:18.234 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094477+0000", 2024-09-11T09:52:18.234 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.151e-06 2024-09-11T09:52:18.234 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.234 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.234 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:18.234 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094483+0000", 2024-09-11T09:52:18.234 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.6330000000000001e-06 2024-09-11T09:52:18.234 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.235 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.235 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:18.235 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094487+0000", 2024-09-11T09:52:18.235 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.5399999999999997e-06 2024-09-11T09:52:18.235 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.235 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.235 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:18.235 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.101866+0000", 2024-09-11T09:52:18.235 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.007378768 2024-09-11T09:52:18.235 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.235 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.235 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:18.235 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.101960+0000", 2024-09-11T09:52:18.235 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.3861000000000001e-05 2024-09-11T09:52:18.236 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.236 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.236 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 5", 2024-09-11T09:52:18.236 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.102197+0000", 2024-09-11T09:52:18.236 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00023723200000000001 2024-09-11T09:52:18.236 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.236 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.236 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:18.236 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.106222+0000", 2024-09-11T09:52:18.236 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0040249689999999998 2024-09-11T09:52:18.236 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.236 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.236 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:18.237 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.107141+0000", 2024-09-11T09:52:18.237 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00091913900000000002 2024-09-11T09:52:18.237 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.237 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.237 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:18.237 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.107198+0000", 2024-09-11T09:52:18.237 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.6564999999999998e-05 2024-09-11T09:52:18.237 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.237 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.237 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:18.237 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.107252+0000", 2024-09-11T09:52:18.237 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.4030000000000003e-05 2024-09-11T09:52:18.237 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:18.237 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:18.238 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:18.238 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.238 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.238 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4694.0:33 2.4 e53/32 2:3bfc818c:ns::rbd_data.1256a0c02879.0000000000000001:head v 53'14, mlcod=53'12)", 2024-09-11T09:52:18.238 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.096496+0000", 2024-09-11T09:52:18.238 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 55.060001284000002, 2024-09-11T09:52:18.238 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0099139310000000008, 2024-09-11T09:52:18.238 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:18.238 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:18.238 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:18.238 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:18.238 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.238 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:18.238 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.096496+0000", 2024-09-11T09:52:18.239 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:18.239 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.239 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.239 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:18.239 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.096496+0000", 2024-09-11T09:52:18.239 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:18.239 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.239 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.239 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:18.239 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.096498+0000", 2024-09-11T09:52:18.239 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6309999999999999e-06 2024-09-11T09:52:18.239 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.239 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.240 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:18.240 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.098087+0000", 2024-09-11T09:52:18.240 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0015889019999999999 2024-09-11T09:52:18.240 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.240 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.240 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:18.240 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.098112+0000", 2024-09-11T09:52:18.240 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.5369000000000001e-05 2024-09-11T09:52:18.240 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.240 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.240 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:18.240 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.098127+0000", 2024-09-11T09:52:18.240 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.5218999999999999e-05 2024-09-11T09:52:18.241 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.241 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.241 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:18.241 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.098209+0000", 2024-09-11T09:52:18.241 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.2120999999999995e-05 2024-09-11T09:52:18.241 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.241 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.241 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:18.241 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.098237+0000", 2024-09-11T09:52:18.241 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.7657999999999998e-05 2024-09-11T09:52:18.241 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.241 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.241 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:18.241 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.106376+0000", 2024-09-11T09:52:18.242 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0081386749999999997 2024-09-11T09:52:18.242 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.242 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.242 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:18.242 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.106410+0000", 2024-09-11T09:52:18.242 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.4356000000000001e-05 2024-09-11T09:52:18.242 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:18.242 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:18.242 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:18.242 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.242 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.242 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4694.0:49 2.7 2:e8766325:ns::rbd_object_map.1256a0c02879:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:18.242 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.134806+0000", 2024-09-11T09:52:18.243 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 55.021691214999997, 2024-09-11T09:52:18.243 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0061629049999999998, 2024-09-11T09:52:18.243 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:18.243 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:18.243 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:18.243 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:18.243 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4694", 2024-09-11T09:52:18.243 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:52:18.243 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 49 2024-09-11T09:52:18.243 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.243 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:18.243 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.243 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:18.244 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.134806+0000", 2024-09-11T09:52:18.244 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:18.244 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.244 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.244 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:18.244 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.134806+0000", 2024-09-11T09:52:18.244 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:18.244 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.244 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.244 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:18.244 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.134808+0000", 2024-09-11T09:52:18.244 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.9870000000000002e-06 2024-09-11T09:52:18.244 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.244 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.245 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:18.245 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.134818+0000", 2024-09-11T09:52:18.245 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.5599999999999999e-06 2024-09-11T09:52:18.245 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.245 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.245 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:18.245 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.134835+0000", 2024-09-11T09:52:18.245 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7399000000000001e-05 2024-09-11T09:52:18.245 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.245 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.245 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:18.245 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.134845+0000", 2024-09-11T09:52:18.245 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0285000000000001e-05 2024-09-11T09:52:18.246 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.246 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.246 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:18.246 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.134911+0000", 2024-09-11T09:52:18.246 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.5654999999999994e-05 2024-09-11T09:52:18.246 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.246 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.246 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:18.246 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.135023+0000", 2024-09-11T09:52:18.246 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00011186799999999999 2024-09-11T09:52:18.246 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.246 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.246 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 5", 2024-09-11T09:52:18.247 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.135290+0000", 2024-09-11T09:52:18.247 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00026725300000000001 2024-09-11T09:52:18.247 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.247 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.247 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:18.247 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.136506+0000", 2024-09-11T09:52:18.247 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0012161139999999999 2024-09-11T09:52:18.247 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.247 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.247 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:18.247 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.140884+0000", 2024-09-11T09:52:18.247 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0043781990000000002 2024-09-11T09:52:18.247 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.247 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.248 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:18.248 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.140923+0000", 2024-09-11T09:52:18.248 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.8813999999999998e-05 2024-09-11T09:52:18.248 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.248 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.248 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:18.248 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.140969+0000", 2024-09-11T09:52:18.248 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.5770999999999999e-05 2024-09-11T09:52:18.248 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:18.248 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:18.248 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:18.248 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.248 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.249 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4694.0:42 2.5 e53/32 2:a4fc00ec:ns::rbd_data.1256a0c02879.0000000000000004:head v 53'3, mlcod=53'1)", 2024-09-11T09:52:18.249 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.136646+0000", 2024-09-11T09:52:18.249 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 55.019850908999999, 2024-09-11T09:52:18.249 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.012545501000000001, 2024-09-11T09:52:18.249 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:18.249 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:18.249 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:18.249 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:18.249 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.249 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:18.249 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.136646+0000", 2024-09-11T09:52:18.249 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:18.249 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.250 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.250 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:18.250 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.136646+0000", 2024-09-11T09:52:18.250 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:18.250 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.250 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.250 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:18.250 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.136647+0000", 2024-09-11T09:52:18.250 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.2900000000000002e-07 2024-09-11T09:52:18.250 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.250 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.250 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:18.250 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.143620+0000", 2024-09-11T09:52:18.251 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0069727440000000003 2024-09-11T09:52:18.251 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.251 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.251 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:18.251 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.143642+0000", 2024-09-11T09:52:18.251 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.1846000000000001e-05 2024-09-11T09:52:18.251 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.251 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.251 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:18.251 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.143654+0000", 2024-09-11T09:52:18.251 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2081e-05 2024-09-11T09:52:18.251 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.251 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.252 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:18.252 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.143717+0000", 2024-09-11T09:52:18.252 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.3443999999999998e-05 2024-09-11T09:52:18.252 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.252 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.252 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:18.252 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.143740+0000", 2024-09-11T09:52:18.252 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.2626999999999999e-05 2024-09-11T09:52:18.252 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.252 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.252 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:18.252 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.149142+0000", 2024-09-11T09:52:18.252 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0054017630000000004 2024-09-11T09:52:18.253 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.253 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.253 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:18.253 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.149192+0000", 2024-09-11T09:52:18.253 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.0067e-05 2024-09-11T09:52:18.253 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:18.253 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:18.253 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:18.253 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.253 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.253 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4694.0:53 2.7 2:e8766325:ns::rbd_object_map.1256a0c02879:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:18.253 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.147978+0000", 2024-09-11T09:52:18.253 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 55.008519425000003, 2024-09-11T09:52:18.254 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0044549769999999997, 2024-09-11T09:52:18.254 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:18.254 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:18.254 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:18.254 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:18.254 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4694", 2024-09-11T09:52:18.254 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:52:18.254 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 53 2024-09-11T09:52:18.254 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.254 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:18.254 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.254 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:18.254 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.147978+0000", 2024-09-11T09:52:18.255 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:18.255 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.255 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.255 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:18.255 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.147978+0000", 2024-09-11T09:52:18.255 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:18.255 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.255 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.255 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:18.255 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.147980+0000", 2024-09-11T09:52:18.255 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.8309999999999999e-06 2024-09-11T09:52:18.255 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.255 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.256 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:18.256 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.147989+0000", 2024-09-11T09:52:18.256 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.9639999999999992e-06 2024-09-11T09:52:18.256 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.256 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.256 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:18.256 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.148007+0000", 2024-09-11T09:52:18.256 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7884999999999999e-05 2024-09-11T09:52:18.256 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.256 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.256 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:18.256 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.148017+0000", 2024-09-11T09:52:18.256 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0895e-05 2024-09-11T09:52:18.256 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.257 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.257 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:18.257 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.148091+0000", 2024-09-11T09:52:18.257 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.3709999999999997e-05 2024-09-11T09:52:18.257 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.257 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.257 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:18.257 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.148189+0000", 2024-09-11T09:52:18.257 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.7403000000000002e-05 2024-09-11T09:52:18.257 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.257 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.257 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 5", 2024-09-11T09:52:18.258 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.148504+0000", 2024-09-11T09:52:18.258 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00031589900000000002 2024-09-11T09:52:18.258 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.258 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.258 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:18.258 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.149514+0000", 2024-09-11T09:52:18.258 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0010096 2024-09-11T09:52:18.258 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.258 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.258 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:18.258 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.152367+0000", 2024-09-11T09:52:18.258 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0028532309999999999 2024-09-11T09:52:18.258 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.259 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.259 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:18.259 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.152402+0000", 2024-09-11T09:52:18.259 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.4232e-05 2024-09-11T09:52:18.259 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.259 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.259 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:18.259 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.152433+0000", 2024-09-11T09:52:18.259 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.1327e-05 2024-09-11T09:52:18.259 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:18.259 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:18.259 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:18.259 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.259 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.259 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4694.0:55 2.7 2:e8766325:ns::rbd_object_map.1256a0c02879:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:18.259 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.149838+0000", 2024-09-11T09:52:18.260 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 55.006659691999999, 2024-09-11T09:52:18.260 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0060695159999999996, 2024-09-11T09:52:18.260 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:18.260 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:18.260 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:18.260 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:18.260 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4694", 2024-09-11T09:52:18.260 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:52:18.260 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 55 2024-09-11T09:52:18.260 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.260 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:18.260 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.260 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:18.260 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.149838+0000", 2024-09-11T09:52:18.261 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:18.261 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.261 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.261 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:18.261 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.149838+0000", 2024-09-11T09:52:18.261 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:18.261 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.261 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.261 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:18.261 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.149839+0000", 2024-09-11T09:52:18.261 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.119e-06 2024-09-11T09:52:18.261 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.261 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.261 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:18.262 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.149845+0000", 2024-09-11T09:52:18.262 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.7810000000000002e-06 2024-09-11T09:52:18.262 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.262 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.262 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:18.262 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.149856+0000", 2024-09-11T09:52:18.262 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1199999999999999e-05 2024-09-11T09:52:18.262 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.262 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.262 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:18.262 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.149863+0000", 2024-09-11T09:52:18.264 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.4109999999999999e-06 2024-09-11T09:52:18.264 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.264 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.264 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:18.264 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.149908+0000", 2024-09-11T09:52:18.264 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.5184e-05 2024-09-11T09:52:18.264 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.264 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.264 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:18.265 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.149964+0000", 2024-09-11T09:52:18.265 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.5395000000000001e-05 2024-09-11T09:52:18.265 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.265 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.265 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 5", 2024-09-11T09:52:18.265 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.150157+0000", 2024-09-11T09:52:18.265 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000193489 2024-09-11T09:52:18.265 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.265 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.265 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:18.265 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.152238+0000", 2024-09-11T09:52:18.265 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0020803480000000001 2024-09-11T09:52:18.265 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.265 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.266 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:18.266 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.155828+0000", 2024-09-11T09:52:18.266 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0035902210000000002 2024-09-11T09:52:18.266 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.266 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.266 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:18.266 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.155873+0000", 2024-09-11T09:52:18.266 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.5145999999999998e-05 2024-09-11T09:52:18.266 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.266 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.266 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:18.266 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.155907+0000", 2024-09-11T09:52:18.266 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.4221999999999999e-05 2024-09-11T09:52:18.266 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:18.267 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:18.267 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:18.267 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:18.267 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:18.267 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:52:18.268 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-11T09:52:18.269 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:52:18.269+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:52:18.369 INFO:tasks.ceph.osd.2.smithi017.stderr:2024-09-11T09:52:18.370+0000 7fd20b170640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31672) UID: 0 2024-09-11T09:52:18.416 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:52:18.416 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [], 2024-09-11T09:52:18.416 INFO:teuthology.orchestra.run.smithi017.stdout: "num_ops": 0 2024-09-11T09:52:18.416 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:52:18.425 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-11T09:52:18.469 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:52:18.470+0000 7f3a7b90a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 42475) UID: 0 2024-09-11T09:52:18.570 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-09-11T09:52:18.571+0000 7f7b6c7d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31904) UID: 0 2024-09-11T09:52:18.574 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:52:18.574 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [], 2024-09-11T09:52:18.574 INFO:teuthology.orchestra.run.smithi017.stdout: "complaint_time": 30, 2024-09-11T09:52:18.574 INFO:teuthology.orchestra.run.smithi017.stdout: "num_blocked_ops": 0 2024-09-11T09:52:18.575 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:52:18.583 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-11T09:52:18.670 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:52:18.672+0000 7f3c96ec9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31909) UID: 0 2024-09-11T09:52:18.692 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:52:18.692 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 20, 2024-09-11T09:52:18.692 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 600, 2024-09-11T09:52:18.692 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [ 2024-09-11T09:52:18.692 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.692 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:34 2.4 2:2367385a:::rbd_data.1250f5e25b80.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:18.692 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.897717+0000", 2024-09-11T09:52:18.692 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 55.794024227000001, 2024-09-11T09:52:18.692 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.021323661000000001, 2024-09-11T09:52:18.692 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:18.692 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:18.693 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:18.693 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:18.693 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:52:18.693 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:52:18.693 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 34 2024-09-11T09:52:18.693 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.693 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:18.693 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.693 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:18.693 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.897717+0000", 2024-09-11T09:52:18.693 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:18.693 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.693 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.694 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:18.694 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.897717+0000", 2024-09-11T09:52:18.694 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:18.694 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.694 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.694 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:18.694 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.897718+0000", 2024-09-11T09:52:18.694 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.1800000000000004e-07 2024-09-11T09:52:18.694 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.694 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.694 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:18.694 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.905339+0000", 2024-09-11T09:52:18.694 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0076210239999999997 2024-09-11T09:52:18.695 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.695 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.695 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:18.695 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.905366+0000", 2024-09-11T09:52:18.695 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.6959000000000001e-05 2024-09-11T09:52:18.695 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.695 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.695 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:18.695 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.905375+0000", 2024-09-11T09:52:18.695 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.516e-06 2024-09-11T09:52:18.695 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.695 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.695 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:18.696 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.905475+0000", 2024-09-11T09:52:18.696 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.9738e-05 2024-09-11T09:52:18.696 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.696 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.696 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:18.696 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.905660+0000", 2024-09-11T09:52:18.696 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00018554199999999999 2024-09-11T09:52:18.696 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.696 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.696 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 0", 2024-09-11T09:52:18.696 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.905808+0000", 2024-09-11T09:52:18.696 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000147386 2024-09-11T09:52:18.696 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.697 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.697 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:18.697 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.918230+0000", 2024-09-11T09:52:18.697 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.012421690000000001 2024-09-11T09:52:18.697 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.697 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.697 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:18.697 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.918932+0000", 2024-09-11T09:52:18.697 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00070267300000000001 2024-09-11T09:52:18.697 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.697 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.697 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:18.697 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.918989+0000", 2024-09-11T09:52:18.697 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.7268999999999997e-05 2024-09-11T09:52:18.698 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.698 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.698 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:18.698 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919040+0000", 2024-09-11T09:52:18.698 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.0946000000000003e-05 2024-09-11T09:52:18.698 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:18.698 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:18.698 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:18.698 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.698 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.698 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4688.0:31 2.1 e53/15 2:8654721b:::rbd_data.1250f5e25b80.0000000000000000:head v 53'2, mlcod=0'0)", 2024-09-11T09:52:18.698 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.904661+0000", 2024-09-11T09:52:18.698 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 55.787079738000003, 2024-09-11T09:52:18.699 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.014152804999999999, 2024-09-11T09:52:18.699 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:18.699 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:18.699 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:18.699 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:18.699 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.699 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:18.699 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.904661+0000", 2024-09-11T09:52:18.699 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:18.699 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.699 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.700 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:18.700 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.904661+0000", 2024-09-11T09:52:18.700 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:18.700 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.700 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.700 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:18.700 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.904662+0000", 2024-09-11T09:52:18.700 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0759999999999999e-06 2024-09-11T09:52:18.700 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.700 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.700 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:18.700 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.907298+0000", 2024-09-11T09:52:18.700 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0026361240000000001 2024-09-11T09:52:18.701 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.701 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.701 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:18.701 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.907320+0000", 2024-09-11T09:52:18.701 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.1268999999999999e-05 2024-09-11T09:52:18.701 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.701 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.701 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:18.701 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.907331+0000", 2024-09-11T09:52:18.701 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1031e-05 2024-09-11T09:52:18.701 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.701 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.701 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:18.702 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.907403+0000", 2024-09-11T09:52:18.702 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.2571999999999996e-05 2024-09-11T09:52:18.702 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.702 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.702 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:18.702 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.907428+0000", 2024-09-11T09:52:18.702 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.4380999999999999e-05 2024-09-11T09:52:18.702 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.702 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.702 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:18.702 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.918750+0000", 2024-09-11T09:52:18.702 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.011322211 2024-09-11T09:52:18.702 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.703 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.703 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:18.703 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.918814+0000", 2024-09-11T09:52:18.703 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.4140999999999995e-05 2024-09-11T09:52:18.703 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:18.703 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:18.703 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:18.703 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.703 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.703 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:36 2.6 2:61e1b442:::rbd_data.1250f5e25b80.0000000000000003:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:18.703 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.905404+0000", 2024-09-11T09:52:18.703 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 55.786336867000003, 2024-09-11T09:52:18.703 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.030912479999999999, 2024-09-11T09:52:18.703 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:18.704 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:18.704 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:18.704 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:18.704 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:52:18.704 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:52:18.704 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 36 2024-09-11T09:52:18.704 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.704 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:18.704 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.704 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:18.704 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.905404+0000", 2024-09-11T09:52:18.704 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:18.704 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.705 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.705 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:18.705 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.905404+0000", 2024-09-11T09:52:18.705 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:18.705 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.705 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.705 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:18.705 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.905405+0000", 2024-09-11T09:52:18.705 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2270000000000001e-06 2024-09-11T09:52:18.705 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.705 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.705 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:18.705 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.914859+0000", 2024-09-11T09:52:18.705 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0094541190000000004 2024-09-11T09:52:18.706 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.706 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.706 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:18.706 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.914879+0000", 2024-09-11T09:52:18.706 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.0052999999999999e-05 2024-09-11T09:52:18.706 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.706 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.706 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:18.706 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.914897+0000", 2024-09-11T09:52:18.706 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7331000000000001e-05 2024-09-11T09:52:18.706 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.706 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.706 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:18.707 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.915019+0000", 2024-09-11T09:52:18.707 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00012168 2024-09-11T09:52:18.707 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.707 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.707 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:18.707 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.915269+0000", 2024-09-11T09:52:18.707 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00025054899999999998 2024-09-11T09:52:18.707 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.707 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.707 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 3", 2024-09-11T09:52:18.707 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.915479+0000", 2024-09-11T09:52:18.707 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00021017799999999999 2024-09-11T09:52:18.707 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.707 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.708 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:18.708 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.926070+0000", 2024-09-11T09:52:18.708 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.010590624999999999 2024-09-11T09:52:18.708 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.708 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.708 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:18.708 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.936198+0000", 2024-09-11T09:52:18.708 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.010128194 2024-09-11T09:52:18.708 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.708 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.708 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:18.708 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.936263+0000", 2024-09-11T09:52:18.709 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.5366999999999999e-05 2024-09-11T09:52:18.709 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.709 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.709 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:18.709 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.936317+0000", 2024-09-11T09:52:18.709 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.3156999999999999e-05 2024-09-11T09:52:18.709 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:18.709 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:18.709 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:18.709 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.709 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.709 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:39 2.6 2:743e7616:::rbd_data.1250f5e25b80.0000000000000005:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:18.709 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.914960+0000", 2024-09-11T09:52:18.710 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 55.776780565000003, 2024-09-11T09:52:18.710 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.022371518, 2024-09-11T09:52:18.710 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:18.710 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:18.710 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:18.710 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:18.710 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:52:18.710 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:52:18.710 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 39 2024-09-11T09:52:18.710 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.710 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:18.710 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.710 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:18.710 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.914960+0000", 2024-09-11T09:52:18.711 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:18.711 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.711 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.711 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:18.711 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.914960+0000", 2024-09-11T09:52:18.711 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:18.711 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.711 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.711 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:18.711 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.914963+0000", 2024-09-11T09:52:18.711 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.1119999999999999e-06 2024-09-11T09:52:18.711 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.711 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.712 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:18.712 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.920417+0000", 2024-09-11T09:52:18.712 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0054543969999999997 2024-09-11T09:52:18.712 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.712 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.712 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:18.712 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.920430+0000", 2024-09-11T09:52:18.712 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3414000000000001e-05 2024-09-11T09:52:18.712 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.712 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.712 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:18.712 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.920437+0000", 2024-09-11T09:52:18.712 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.6159999999999998e-06 2024-09-11T09:52:18.712 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.713 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.713 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:18.713 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.921325+0000", 2024-09-11T09:52:18.713 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000887731 2024-09-11T09:52:18.713 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.713 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.713 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:18.713 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.921422+0000", 2024-09-11T09:52:18.713 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.7201999999999995e-05 2024-09-11T09:52:18.713 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.713 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.713 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 3", 2024-09-11T09:52:18.713 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.921511+0000", 2024-09-11T09:52:18.714 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.8880000000000003e-05 2024-09-11T09:52:18.714 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.714 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.714 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:18.714 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.928092+0000", 2024-09-11T09:52:18.714 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0065816290000000003 2024-09-11T09:52:18.714 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.714 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.714 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:18.714 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.937240+0000", 2024-09-11T09:52:18.714 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0091481070000000008 2024-09-11T09:52:18.714 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.714 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.714 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:18.715 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.937294+0000", 2024-09-11T09:52:18.715 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.3100000000000003e-05 2024-09-11T09:52:18.715 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.715 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.715 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:18.715 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.937332+0000", 2024-09-11T09:52:18.715 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.8330000000000001e-05 2024-09-11T09:52:18.715 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:18.715 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:18.715 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:18.715 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.715 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.715 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4688.0:32 2.2 e53/15 2:45941fd2:::rbd_data.1250f5e25b80.0000000000000001:head v 53'3, mlcod=17'1)", 2024-09-11T09:52:18.715 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.916509+0000", 2024-09-11T09:52:18.716 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 55.775231482999999, 2024-09-11T09:52:18.716 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.015885318999999998, 2024-09-11T09:52:18.716 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:18.716 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:18.716 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:18.716 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:18.716 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.716 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:18.716 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.916509+0000", 2024-09-11T09:52:18.716 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:18.716 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.716 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.716 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:18.717 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.916509+0000", 2024-09-11T09:52:18.717 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:18.717 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.717 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.717 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:18.717 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.916511+0000", 2024-09-11T09:52:18.717 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.052e-06 2024-09-11T09:52:18.717 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.717 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.717 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:18.717 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.927019+0000", 2024-09-11T09:52:18.717 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.010508224 2024-09-11T09:52:18.717 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.718 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.718 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:18.718 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.927038+0000", 2024-09-11T09:52:18.718 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.9590999999999999e-05 2024-09-11T09:52:18.718 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.718 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.718 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:18.718 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.927050+0000", 2024-09-11T09:52:18.718 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1439e-05 2024-09-11T09:52:18.718 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.718 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.718 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:18.718 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.927115+0000", 2024-09-11T09:52:18.718 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.5301000000000004e-05 2024-09-11T09:52:18.719 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.719 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.719 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:18.719 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.927132+0000", 2024-09-11T09:52:18.719 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7297000000000001e-05 2024-09-11T09:52:18.719 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.719 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.719 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:18.719 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.932354+0000", 2024-09-11T09:52:18.719 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0052211870000000004 2024-09-11T09:52:18.719 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.719 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.719 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:18.720 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.932395+0000", 2024-09-11T09:52:18.720 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.1227999999999999e-05 2024-09-11T09:52:18.720 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:18.720 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:18.720 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:18.720 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.720 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.720 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:44 2.6 2:67246a22:::rbd_data.1250f5e25b80.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:18.720 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.943181+0000", 2024-09-11T09:52:18.720 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 55.748560271999999, 2024-09-11T09:52:18.720 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.022629371999999998, 2024-09-11T09:52:18.720 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:18.720 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:18.720 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:18.721 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:18.721 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:52:18.721 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:52:18.721 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 44 2024-09-11T09:52:18.721 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.721 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:18.721 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.721 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:18.721 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.943181+0000", 2024-09-11T09:52:18.721 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:18.721 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.721 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.721 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:18.722 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.943181+0000", 2024-09-11T09:52:18.722 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:18.722 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.722 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.722 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:18.722 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.943182+0000", 2024-09-11T09:52:18.722 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.0500000000000002e-07 2024-09-11T09:52:18.722 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.722 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.722 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:18.722 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951240+0000", 2024-09-11T09:52:18.722 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.008058529 2024-09-11T09:52:18.722 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.722 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.723 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:18.723 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951260+0000", 2024-09-11T09:52:18.723 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.9570000000000001e-05 2024-09-11T09:52:18.723 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.723 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.723 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:18.723 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951270+0000", 2024-09-11T09:52:18.723 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0018e-05 2024-09-11T09:52:18.723 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.723 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.723 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:18.723 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951349+0000", 2024-09-11T09:52:18.723 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.8801000000000007e-05 2024-09-11T09:52:18.724 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.724 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.724 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:18.724 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951483+0000", 2024-09-11T09:52:18.724 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00013427 2024-09-11T09:52:18.724 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.724 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.724 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 3", 2024-09-11T09:52:18.724 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951593+0000", 2024-09-11T09:52:18.724 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00011031000000000001 2024-09-11T09:52:18.724 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.724 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.724 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:18.725 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.957303+0000", 2024-09-11T09:52:18.725 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0057097959999999996 2024-09-11T09:52:18.725 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.725 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.725 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:18.725 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.965690+0000", 2024-09-11T09:52:18.725 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0083867420000000008 2024-09-11T09:52:18.725 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.725 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.725 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:18.725 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.965749+0000", 2024-09-11T09:52:18.725 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.8981999999999999e-05 2024-09-11T09:52:18.725 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.726 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.726 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:18.726 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.965810+0000", 2024-09-11T09:52:18.726 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.1449000000000001e-05 2024-09-11T09:52:18.726 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:18.726 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:18.726 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:18.726 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.726 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.726 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:51 2.6 2:6b5894fc:::rbd_data.1250f5e25b80.000000000000000a:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:18.726 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.951324+0000", 2024-09-11T09:52:18.726 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 55.740417198999999, 2024-09-11T09:52:18.726 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.021375273, 2024-09-11T09:52:18.726 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:18.727 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:18.727 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:18.727 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:18.727 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:52:18.727 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:52:18.727 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 51 2024-09-11T09:52:18.727 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.727 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:18.727 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.727 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:18.727 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951324+0000", 2024-09-11T09:52:18.727 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:18.728 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.728 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.728 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:18.728 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951324+0000", 2024-09-11T09:52:18.728 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:18.728 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.728 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.728 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:18.728 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951324+0000", 2024-09-11T09:52:18.728 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.39e-07 2024-09-11T09:52:18.729 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.729 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.729 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:18.729 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.958537+0000", 2024-09-11T09:52:18.729 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0072126769999999998 2024-09-11T09:52:18.729 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.729 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.729 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:18.730 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.958554+0000", 2024-09-11T09:52:18.730 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7232e-05 2024-09-11T09:52:18.730 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.730 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.730 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:18.730 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.958570+0000", 2024-09-11T09:52:18.730 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.5747999999999999e-05 2024-09-11T09:52:18.730 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.730 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.730 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:18.730 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.958638+0000", 2024-09-11T09:52:18.730 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.7810000000000003e-05 2024-09-11T09:52:18.731 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.731 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.731 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:18.731 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.958758+0000", 2024-09-11T09:52:18.731 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000120689 2024-09-11T09:52:18.731 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.731 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.731 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 3", 2024-09-11T09:52:18.731 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.958865+0000", 2024-09-11T09:52:18.731 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00010692 2024-09-11T09:52:18.731 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.731 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.731 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:18.732 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.970712+0000", 2024-09-11T09:52:18.732 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.011846327 2024-09-11T09:52:18.732 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.732 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.732 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:18.732 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.972603+0000", 2024-09-11T09:52:18.732 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0018918610000000001 2024-09-11T09:52:18.732 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.732 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.732 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:18.732 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.972653+0000", 2024-09-11T09:52:18.732 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.9762999999999998e-05 2024-09-11T09:52:18.732 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.732 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.733 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:18.733 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.972699+0000", 2024-09-11T09:52:18.733 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.5806999999999997e-05 2024-09-11T09:52:18.733 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:18.733 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:18.733 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:18.733 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.733 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.733 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:52 2.4 2:26d5c2f0:::rbd_data.1250f5e25b80.000000000000000b:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:18.733 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.958596+0000", 2024-09-11T09:52:18.733 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 55.733145004000001, 2024-09-11T09:52:18.733 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.016477849999999999, 2024-09-11T09:52:18.733 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:18.733 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:18.734 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:18.734 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:18.734 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:52:18.734 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:52:18.734 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 52 2024-09-11T09:52:18.734 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.734 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:18.734 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.734 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:18.734 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.958596+0000", 2024-09-11T09:52:18.734 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:18.734 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.734 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.734 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:18.735 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.958596+0000", 2024-09-11T09:52:18.735 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:18.735 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.735 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.735 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:18.735 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.958597+0000", 2024-09-11T09:52:18.735 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1960000000000001e-06 2024-09-11T09:52:18.735 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.735 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.735 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:18.735 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.963157+0000", 2024-09-11T09:52:18.735 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0045603029999999999 2024-09-11T09:52:18.735 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.736 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.736 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:18.736 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.963171+0000", 2024-09-11T09:52:18.736 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3777999999999999e-05 2024-09-11T09:52:18.736 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.736 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.736 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:18.736 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.963181+0000", 2024-09-11T09:52:18.736 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.9010000000000006e-06 2024-09-11T09:52:18.736 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.736 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.736 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:18.736 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.963246+0000", 2024-09-11T09:52:18.737 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.5351000000000005e-05 2024-09-11T09:52:18.737 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.737 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.737 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:18.737 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.963419+0000", 2024-09-11T09:52:18.737 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00017299700000000001 2024-09-11T09:52:18.737 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.737 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.737 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 0", 2024-09-11T09:52:18.737 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.963570+0000", 2024-09-11T09:52:18.737 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00015066999999999999 2024-09-11T09:52:18.737 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.737 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.738 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:18.738 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.973881+0000", 2024-09-11T09:52:18.738 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.010311268 2024-09-11T09:52:18.738 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.738 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.738 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:18.738 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.974963+0000", 2024-09-11T09:52:18.738 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.001081493 2024-09-11T09:52:18.738 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.738 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.738 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:18.738 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.975012+0000", 2024-09-11T09:52:18.739 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.9471000000000001e-05 2024-09-11T09:52:18.739 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.739 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.739 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:18.739 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.975074+0000", 2024-09-11T09:52:18.739 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.1421999999999996e-05 2024-09-11T09:52:18.739 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:18.739 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:18.739 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:18.739 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.739 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.739 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4688.0:47 2.1 e53/15 2:95f53b3e:::rbd_data.1250f5e25b80.0000000000000008:head v 53'3, mlcod=53'1)", 2024-09-11T09:52:18.739 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.960861+0000", 2024-09-11T09:52:18.739 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 55.730879586, 2024-09-11T09:52:18.740 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.012346107, 2024-09-11T09:52:18.740 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:18.740 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:18.740 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:18.740 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:18.740 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.740 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:18.740 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.960861+0000", 2024-09-11T09:52:18.740 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:18.740 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.740 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.740 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:18.740 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.960861+0000", 2024-09-11T09:52:18.740 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:18.741 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.741 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.741 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:18.741 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.960863+0000", 2024-09-11T09:52:18.741 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7150000000000001e-06 2024-09-11T09:52:18.741 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.741 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.741 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:18.741 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.962428+0000", 2024-09-11T09:52:18.741 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.001564668 2024-09-11T09:52:18.741 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.741 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.741 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:18.742 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.962446+0000", 2024-09-11T09:52:18.742 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.8476999999999999e-05 2024-09-11T09:52:18.742 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.742 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.742 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:18.742 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.962458+0000", 2024-09-11T09:52:18.742 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1263999999999999e-05 2024-09-11T09:52:18.742 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.742 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.742 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:18.742 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.962575+0000", 2024-09-11T09:52:18.742 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000117025 2024-09-11T09:52:18.742 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.742 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.743 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:18.743 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.962601+0000", 2024-09-11T09:52:18.743 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.5984999999999999e-05 2024-09-11T09:52:18.743 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.743 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.743 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:18.743 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.973065+0000", 2024-09-11T09:52:18.743 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.010464753 2024-09-11T09:52:18.743 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.743 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.743 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:18.743 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.973207+0000", 2024-09-11T09:52:18.743 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00014222 2024-09-11T09:52:18.744 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:18.744 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:18.744 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:18.744 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.744 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.744 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:54 2.4 2.da8979c4 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:18.744 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.963207+0000", 2024-09-11T09:52:18.744 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 55.728534148000001, 2024-09-11T09:52:18.744 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.010641575, 2024-09-11T09:52:18.744 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:18.744 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:18.744 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "reached pg", 2024-09-11T09:52:18.744 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:18.745 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:52:18.745 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:52:18.745 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 54 2024-09-11T09:52:18.745 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.745 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:18.745 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.745 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:18.745 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.963207+0000", 2024-09-11T09:52:18.745 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:18.745 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.745 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.745 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:18.745 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.963207+0000", 2024-09-11T09:52:18.746 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:18.746 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.746 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.746 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:18.746 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.963208+0000", 2024-09-11T09:52:18.746 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2979999999999999e-06 2024-09-11T09:52:18.746 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.746 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.746 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:18.746 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.967419+0000", 2024-09-11T09:52:18.746 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0042111750000000002 2024-09-11T09:52:18.746 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.746 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.747 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:18.747 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.967439+0000", 2024-09-11T09:52:18.747 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.0129999999999999e-05 2024-09-11T09:52:18.747 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.747 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.747 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:18.747 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.967453+0000", 2024-09-11T09:52:18.747 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3781000000000001e-05 2024-09-11T09:52:18.747 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.747 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.747 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:18.747 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.973840+0000", 2024-09-11T09:52:18.747 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0063872269999999997 2024-09-11T09:52:18.747 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.748 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.748 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:18.748 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.973848+0000", 2024-09-11T09:52:18.748 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.9640000000000003e-06 2024-09-11T09:52:18.748 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:18.748 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:18.748 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:18.748 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.748 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.748 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:54 2.4 2:239e915b:::rbd_data.1250f5e25b80.000000000000000c:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:18.748 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.982401+0000", 2024-09-11T09:52:18.748 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 55.709340220999998, 2024-09-11T09:52:18.748 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.016817347, 2024-09-11T09:52:18.748 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:18.749 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:18.749 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:18.749 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:18.749 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:52:18.749 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:52:18.749 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 54 2024-09-11T09:52:18.749 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.749 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:18.749 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.749 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:18.749 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.982401+0000", 2024-09-11T09:52:18.749 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:18.749 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.749 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.750 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:18.750 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.982401+0000", 2024-09-11T09:52:18.750 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:18.750 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.750 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.750 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:18.750 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.982402+0000", 2024-09-11T09:52:18.750 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.8999999999999995e-07 2024-09-11T09:52:18.750 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.750 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.750 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:18.750 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.986535+0000", 2024-09-11T09:52:18.751 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0041333689999999996 2024-09-11T09:52:18.751 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.751 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.751 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:18.751 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.986555+0000", 2024-09-11T09:52:18.751 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.9851e-05 2024-09-11T09:52:18.751 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.751 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.751 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:18.751 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.986569+0000", 2024-09-11T09:52:18.751 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3784000000000001e-05 2024-09-11T09:52:18.751 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.751 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.751 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:18.751 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.986683+0000", 2024-09-11T09:52:18.752 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000114474 2024-09-11T09:52:18.752 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.752 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.752 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:18.752 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.986815+0000", 2024-09-11T09:52:18.752 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00013217799999999999 2024-09-11T09:52:18.752 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.752 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.752 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 0", 2024-09-11T09:52:18.752 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.986918+0000", 2024-09-11T09:52:18.752 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00010235599999999999 2024-09-11T09:52:18.753 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.753 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.753 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:18.753 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.998983+0000", 2024-09-11T09:52:18.753 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.012065352 2024-09-11T09:52:18.753 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.753 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.753 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:18.753 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.999081+0000", 2024-09-11T09:52:18.753 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.8091000000000007e-05 2024-09-11T09:52:18.753 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.753 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.753 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:18.753 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.999159+0000", 2024-09-11T09:52:18.753 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.7823000000000004e-05 2024-09-11T09:52:18.754 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.754 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.754 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:18.754 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.999218+0000", 2024-09-11T09:52:18.754 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.9178999999999997e-05 2024-09-11T09:52:18.754 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:18.754 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:18.754 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:18.754 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.754 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.754 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:56 2.4 2:2167012f:::rbd_data.1250f5e25b80.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:18.754 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.986661+0000", 2024-09-11T09:52:18.754 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 55.705079767999997, 2024-09-11T09:52:18.754 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.017353539000000001, 2024-09-11T09:52:18.754 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:18.755 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:18.755 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:18.755 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:18.755 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:52:18.755 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:52:18.755 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 56 2024-09-11T09:52:18.755 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.755 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:18.755 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.755 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:18.755 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.986661+0000", 2024-09-11T09:52:18.755 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:18.755 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.755 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.755 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:18.756 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.986661+0000", 2024-09-11T09:52:18.756 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:18.756 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.756 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.756 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:18.756 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.986663+0000", 2024-09-11T09:52:18.756 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.8989999999999999e-06 2024-09-11T09:52:18.756 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.756 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.757 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:18.757 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.990444+0000", 2024-09-11T09:52:18.757 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0037808569999999999 2024-09-11T09:52:18.757 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.757 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.757 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:18.757 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.990462+0000", 2024-09-11T09:52:18.757 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.8450000000000001e-05 2024-09-11T09:52:18.757 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.757 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.757 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:18.757 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.990476+0000", 2024-09-11T09:52:18.757 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3565e-05 2024-09-11T09:52:18.757 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.757 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.758 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:18.758 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.993582+0000", 2024-09-11T09:52:18.758 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0031059109999999998 2024-09-11T09:52:18.758 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.758 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.758 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:18.758 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.993686+0000", 2024-09-11T09:52:18.758 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000104424 2024-09-11T09:52:18.758 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.758 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.758 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 0", 2024-09-11T09:52:18.758 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.993799+0000", 2024-09-11T09:52:18.758 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000112985 2024-09-11T09:52:18.758 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.758 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.759 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:18.759 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.002449+0000", 2024-09-11T09:52:18.759 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.008650151 2024-09-11T09:52:18.759 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.759 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.759 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:18.759 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.003948+0000", 2024-09-11T09:52:18.759 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.001498636 2024-09-11T09:52:18.759 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.759 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.759 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:18.759 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.003985+0000", 2024-09-11T09:52:18.760 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.7054999999999997e-05 2024-09-11T09:52:18.760 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.760 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.760 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:18.760 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.004015+0000", 2024-09-11T09:52:18.760 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.9606000000000001e-05 2024-09-11T09:52:18.760 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:18.760 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:18.760 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:18.760 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.760 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.760 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:58 2.6 2:6277c644:::rbd_data.1250f5e25b80.000000000000000e:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:18.761 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.990540+0000", 2024-09-11T09:52:18.761 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 55.701200972999999, 2024-09-11T09:52:18.761 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.016232068999999998, 2024-09-11T09:52:18.761 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:18.761 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:18.761 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:18.761 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:18.761 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:52:18.761 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:52:18.761 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 58 2024-09-11T09:52:18.761 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.761 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:18.761 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.761 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:18.761 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.990540+0000", 2024-09-11T09:52:18.762 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:18.762 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.762 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.762 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:18.762 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.990540+0000", 2024-09-11T09:52:18.762 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:18.762 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.762 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.762 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:18.762 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.990541+0000", 2024-09-11T09:52:18.762 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2359999999999999e-06 2024-09-11T09:52:18.762 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.762 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.763 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:18.763 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.994626+0000", 2024-09-11T09:52:18.763 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.004084865 2024-09-11T09:52:18.763 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.763 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.763 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:18.763 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.994644+0000", 2024-09-11T09:52:18.763 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7866000000000001e-05 2024-09-11T09:52:18.763 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.763 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.763 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:18.763 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.994653+0000", 2024-09-11T09:52:18.763 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.2159999999999995e-06 2024-09-11T09:52:18.763 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.763 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.764 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:18.764 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.994732+0000", 2024-09-11T09:52:18.764 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.8954000000000003e-05 2024-09-11T09:52:18.764 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.764 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.764 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:18.764 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.994906+0000", 2024-09-11T09:52:18.764 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0001741 2024-09-11T09:52:18.764 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.764 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.764 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 3", 2024-09-11T09:52:18.764 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.995083+0000", 2024-09-11T09:52:18.764 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00017637100000000001 2024-09-11T09:52:18.764 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.765 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.765 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:18.765 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.003066+0000", 2024-09-11T09:52:18.765 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0079833360000000006 2024-09-11T09:52:18.765 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.765 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.765 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:18.765 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.006634+0000", 2024-09-11T09:52:18.765 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.003568309 2024-09-11T09:52:18.765 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.765 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.765 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:18.765 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.006716+0000", 2024-09-11T09:52:18.766 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.1377000000000004e-05 2024-09-11T09:52:18.766 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.766 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.766 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:18.766 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.006772+0000", 2024-09-11T09:52:18.766 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.6439e-05 2024-09-11T09:52:18.766 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:18.766 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:18.766 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:18.766 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.766 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.766 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:59 2.4 2:2906e6a7:::rbd_header.1250f5e25b80:head [watch reconnect cookie 140424358411408 gen 2] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:18.766 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.994684+0000", 2024-09-11T09:52:18.767 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 55.697057342000001, 2024-09-11T09:52:18.767 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0094134349999999995, 2024-09-11T09:52:18.767 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:18.767 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:18.767 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "started", 2024-09-11T09:52:18.767 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:18.767 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:52:18.767 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:52:18.767 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 59 2024-09-11T09:52:18.767 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.767 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:18.767 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.767 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:18.767 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.994684+0000", 2024-09-11T09:52:18.767 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:18.768 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.768 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.768 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:18.768 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.994684+0000", 2024-09-11T09:52:18.768 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:18.768 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.768 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.768 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:18.768 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.994686+0000", 2024-09-11T09:52:18.768 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.8159999999999998e-06 2024-09-11T09:52:18.768 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.768 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.768 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:18.768 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.994691+0000", 2024-09-11T09:52:18.769 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.5009999999999998e-06 2024-09-11T09:52:18.769 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.769 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.769 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:18.769 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.994698+0000", 2024-09-11T09:52:18.769 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.3039999999999999e-06 2024-09-11T09:52:18.769 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.769 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.769 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:18.769 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.994703+0000", 2024-09-11T09:52:18.769 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.5190000000000003e-06 2024-09-11T09:52:18.769 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.769 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.769 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:18.769 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.998678+0000", 2024-09-11T09:52:18.770 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0039752340000000002 2024-09-11T09:52:18.770 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.770 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.770 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:18.770 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.998748+0000", 2024-09-11T09:52:18.770 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.0114e-05 2024-09-11T09:52:18.770 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.770 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.770 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:18.770 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.004097+0000", 2024-09-11T09:52:18.770 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0053489469999999997 2024-09-11T09:52:18.770 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:18.771 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:18.771 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:18.771 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.771 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.771 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4694.0:33 2.4 2:3bfc818c:ns::rbd_data.1256a0c02879.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:18.771 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.084537+0000", 2024-09-11T09:52:18.771 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 55.607203914000003, 2024-09-11T09:52:18.771 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.022075773999999999, 2024-09-11T09:52:18.771 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:18.771 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:18.771 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:18.771 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:18.772 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4694", 2024-09-11T09:52:18.772 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:52:18.772 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 33 2024-09-11T09:52:18.772 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.772 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:18.772 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.772 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:18.772 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.084537+0000", 2024-09-11T09:52:18.772 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:18.772 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.772 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.773 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:18.773 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.084537+0000", 2024-09-11T09:52:18.773 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:18.773 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.773 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.773 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:18.773 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.084538+0000", 2024-09-11T09:52:18.773 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.5000000000000001e-07 2024-09-11T09:52:18.773 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.773 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.773 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:18.773 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.095984+0000", 2024-09-11T09:52:18.773 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.011446543 2024-09-11T09:52:18.773 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.773 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.774 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:18.774 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.096014+0000", 2024-09-11T09:52:18.774 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.9745000000000001e-05 2024-09-11T09:52:18.774 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.774 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.774 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:18.774 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.096030+0000", 2024-09-11T09:52:18.774 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6152999999999999e-05 2024-09-11T09:52:18.774 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.774 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.774 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:18.774 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.096113+0000", 2024-09-11T09:52:18.774 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.2731999999999993e-05 2024-09-11T09:52:18.774 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.775 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.775 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:18.775 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.096273+0000", 2024-09-11T09:52:18.775 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00015973699999999999 2024-09-11T09:52:18.775 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.775 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.775 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 0", 2024-09-11T09:52:18.775 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.096394+0000", 2024-09-11T09:52:18.775 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000120883 2024-09-11T09:52:18.775 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.775 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.775 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:18.775 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.106048+0000", 2024-09-11T09:52:18.775 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0096548020000000005 2024-09-11T09:52:18.775 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.776 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.776 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:18.776 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.106531+0000", 2024-09-11T09:52:18.776 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00048264299999999999 2024-09-11T09:52:18.776 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.776 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.776 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:18.776 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.106565+0000", 2024-09-11T09:52:18.776 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.4343e-05 2024-09-11T09:52:18.776 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.776 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.776 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:18.776 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.106613+0000", 2024-09-11T09:52:18.777 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.7342999999999997e-05 2024-09-11T09:52:18.777 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:18.777 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:18.777 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:18.777 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.777 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.777 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4694.0:32 2.1 e53/15 2:8a862b48:ns::rbd_data.1256a0c02879.0000000000000000:head v 53'8, mlcod=53'6)", 2024-09-11T09:52:18.777 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.091418+0000", 2024-09-11T09:52:18.777 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 55.600322745, 2024-09-11T09:52:18.777 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.013027372000000001, 2024-09-11T09:52:18.777 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:18.777 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:18.777 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:18.777 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:18.777 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.778 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:18.778 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.091418+0000", 2024-09-11T09:52:18.778 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:18.778 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.778 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.778 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:18.778 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.091418+0000", 2024-09-11T09:52:18.778 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:18.778 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.778 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.778 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:18.778 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.091420+0000", 2024-09-11T09:52:18.778 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6899999999999999e-06 2024-09-11T09:52:18.778 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.778 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.779 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:18.779 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.093086+0000", 2024-09-11T09:52:18.779 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0016662510000000001 2024-09-11T09:52:18.779 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.779 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.779 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:18.779 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.093107+0000", 2024-09-11T09:52:18.779 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.0999e-05 2024-09-11T09:52:18.779 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.779 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.779 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:18.779 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.093120+0000", 2024-09-11T09:52:18.779 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2544999999999999e-05 2024-09-11T09:52:18.779 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.779 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.780 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:18.780 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.093186+0000", 2024-09-11T09:52:18.780 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.6595999999999997e-05 2024-09-11T09:52:18.780 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.780 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.780 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:18.780 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.093208+0000", 2024-09-11T09:52:18.780 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.1624000000000002e-05 2024-09-11T09:52:18.780 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.780 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.780 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:18.780 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.104392+0000", 2024-09-11T09:52:18.780 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.011184265000000001 2024-09-11T09:52:18.780 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.781 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.781 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:18.781 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.104446+0000", 2024-09-11T09:52:18.781 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.3402000000000001e-05 2024-09-11T09:52:18.781 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:18.781 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:18.781 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:18.781 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.781 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.781 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4694.0:45 2.6 2:671fb830:ns::rbd_data.1256a0c02879.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:18.781 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.109629+0000", 2024-09-11T09:52:18.781 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 55.582111904000001, 2024-09-11T09:52:18.781 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.036403143999999998, 2024-09-11T09:52:18.781 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:18.781 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:18.782 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:18.782 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:18.782 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4694", 2024-09-11T09:52:18.782 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:52:18.782 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 45 2024-09-11T09:52:18.782 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.782 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:18.782 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.782 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:18.782 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.109629+0000", 2024-09-11T09:52:18.782 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:18.782 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.782 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.782 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:18.782 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.109629+0000", 2024-09-11T09:52:18.783 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:18.783 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.783 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.783 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:18.783 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.109630+0000", 2024-09-11T09:52:18.783 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1200000000000001e-06 2024-09-11T09:52:18.783 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.783 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.783 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:18.783 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.121370+0000", 2024-09-11T09:52:18.783 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.011739571000000001 2024-09-11T09:52:18.783 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.783 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.783 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:18.783 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.121391+0000", 2024-09-11T09:52:18.783 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.1175e-05 2024-09-11T09:52:18.784 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.784 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.784 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:18.784 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.121404+0000", 2024-09-11T09:52:18.784 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2807e-05 2024-09-11T09:52:18.784 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.784 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.784 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:18.784 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.121498+0000", 2024-09-11T09:52:18.784 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.3770999999999994e-05 2024-09-11T09:52:18.784 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.784 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.784 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:18.785 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.121612+0000", 2024-09-11T09:52:18.785 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000114666 2024-09-11T09:52:18.785 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.785 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.785 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 3", 2024-09-11T09:52:18.785 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.121711+0000", 2024-09-11T09:52:18.785 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.8647999999999995e-05 2024-09-11T09:52:18.785 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.785 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.785 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:18.785 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.126558+0000", 2024-09-11T09:52:18.785 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0048473769999999999 2024-09-11T09:52:18.785 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.785 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.785 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:18.786 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.145912+0000", 2024-09-11T09:52:18.786 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.019354281000000001 2024-09-11T09:52:18.786 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.786 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.786 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:18.786 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.145977+0000", 2024-09-11T09:52:18.786 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.4135999999999997e-05 2024-09-11T09:52:18.786 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.786 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.786 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:18.786 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.146032+0000", 2024-09-11T09:52:18.786 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.5591999999999999e-05 2024-09-11T09:52:18.786 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:18.786 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:18.787 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:18.787 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.787 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.787 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4694.0:41 2.2 e53/15 2:5e2505c4:ns::rbd_data.1256a0c02879.0000000000000003:head v 53'4, mlcod=18'2)", 2024-09-11T09:52:18.787 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.113199+0000", 2024-09-11T09:52:18.787 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 55.578541494, 2024-09-11T09:52:18.787 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.011404512, 2024-09-11T09:52:18.787 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:18.787 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:18.787 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:18.787 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:18.787 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.787 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:18.787 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.113199+0000", 2024-09-11T09:52:18.788 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:18.788 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.788 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.788 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:18.788 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.113199+0000", 2024-09-11T09:52:18.788 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:18.788 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.788 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.788 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:18.788 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.113200+0000", 2024-09-11T09:52:18.788 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.2799999999999995e-07 2024-09-11T09:52:18.788 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.789 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.789 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:18.789 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.118872+0000", 2024-09-11T09:52:18.789 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0056718089999999999 2024-09-11T09:52:18.789 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.789 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.789 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:18.789 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.118889+0000", 2024-09-11T09:52:18.789 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7428999999999999e-05 2024-09-11T09:52:18.789 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.789 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.789 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:18.789 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.118900+0000", 2024-09-11T09:52:18.790 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0243e-05 2024-09-11T09:52:18.790 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.790 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.790 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:18.790 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.118959+0000", 2024-09-11T09:52:18.790 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.9364000000000003e-05 2024-09-11T09:52:18.790 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.790 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.790 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:18.790 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.118976+0000", 2024-09-11T09:52:18.790 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7124000000000001e-05 2024-09-11T09:52:18.790 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.790 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.791 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:18.791 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.124557+0000", 2024-09-11T09:52:18.791 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0055812659999999997 2024-09-11T09:52:18.791 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.791 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.791 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:18.791 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.124604+0000", 2024-09-11T09:52:18.791 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.6548999999999998e-05 2024-09-11T09:52:18.791 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:18.791 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:18.791 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:18.791 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.791 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.792 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4694.0:43 2.0 e53/15 2:16c8ae84:ns::rbd_data.1256a0c02879.0000000000000005:head v 53'2, mlcod=0'0)", 2024-09-11T09:52:18.792 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.158559+0000", 2024-09-11T09:52:18.792 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 55.533182168000003, 2024-09-11T09:52:18.792 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.016713677999999999, 2024-09-11T09:52:18.792 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:18.792 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:18.792 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:18.792 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:18.792 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.792 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:18.792 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.158559+0000", 2024-09-11T09:52:18.792 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:18.792 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.792 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.793 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:18.793 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.158559+0000", 2024-09-11T09:52:18.793 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:18.793 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.793 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.793 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:18.793 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.158560+0000", 2024-09-11T09:52:18.793 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1459999999999999e-06 2024-09-11T09:52:18.793 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.793 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.793 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:18.793 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.166465+0000", 2024-09-11T09:52:18.793 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.007904767 2024-09-11T09:52:18.793 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.793 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.794 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:18.794 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.166491+0000", 2024-09-11T09:52:18.794 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.6308e-05 2024-09-11T09:52:18.794 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.794 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.794 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:18.794 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.166502+0000", 2024-09-11T09:52:18.794 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0736e-05 2024-09-11T09:52:18.794 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.794 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.794 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:18.794 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.166564+0000", 2024-09-11T09:52:18.794 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.2578999999999998e-05 2024-09-11T09:52:18.794 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.794 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.795 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:18.795 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.166585+0000", 2024-09-11T09:52:18.795 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.0395999999999999e-05 2024-09-11T09:52:18.795 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.795 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.795 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:18.795 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.175227+0000", 2024-09-11T09:52:18.795 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0086426420000000007 2024-09-11T09:52:18.795 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.795 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.795 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:18.795 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.175272+0000", 2024-09-11T09:52:18.795 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.5104e-05 2024-09-11T09:52:18.795 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:18.795 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:18.796 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:18.796 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.796 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.796 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4694.0:59 2.2 e53/15 2:41adefc9:ns::rbd_data.1256a0c02879.000000000000000e:head v 53'5, mlcod=53'3)", 2024-09-11T09:52:18.796 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.163389+0000", 2024-09-11T09:52:18.796 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 55.528352192, 2024-09-11T09:52:18.796 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.013095719, 2024-09-11T09:52:18.796 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:18.796 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:18.796 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:18.796 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:18.796 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.796 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:18.796 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.163389+0000", 2024-09-11T09:52:18.796 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:18.797 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.797 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.797 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:18.797 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.163389+0000", 2024-09-11T09:52:18.797 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:18.797 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.797 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.797 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:18.797 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.163390+0000", 2024-09-11T09:52:18.797 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.336e-06 2024-09-11T09:52:18.797 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.798 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.798 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:18.798 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.170335+0000", 2024-09-11T09:52:18.798 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0069451080000000002 2024-09-11T09:52:18.798 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.798 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.798 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:18.798 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.170354+0000", 2024-09-11T09:52:18.798 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.8423e-05 2024-09-11T09:52:18.798 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.798 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.798 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:18.798 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.170366+0000", 2024-09-11T09:52:18.798 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2235999999999999e-05 2024-09-11T09:52:18.798 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.799 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.799 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:18.799 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.170439+0000", 2024-09-11T09:52:18.799 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.2989000000000002e-05 2024-09-11T09:52:18.799 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.799 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.799 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:18.799 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.170460+0000", 2024-09-11T09:52:18.799 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.1211000000000001e-05 2024-09-11T09:52:18.799 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.799 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.799 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:18.799 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.176442+0000", 2024-09-11T09:52:18.800 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0059821969999999999 2024-09-11T09:52:18.800 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:18.800 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:18.800 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:18.800 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.176484+0000", 2024-09-11T09:52:18.800 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.2218999999999997e-05 2024-09-11T09:52:18.800 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:18.800 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:18.800 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:18.800 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:18.800 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:18.800 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:52:18.802 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-11T09:52:18.802 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:52:18.803+0000 7f3a7b90a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 42475) UID: 0 2024-09-11T09:52:18.852 INFO:tasks.workunit.client.0.smithi107.stderr:++ rbd journal inspect --pool rbd --image testrbdnbd27121 2024-09-11T09:52:18.852 INFO:tasks.workunit.client.0.smithi107.stderr:++ awk '/entries inspected/ {print $1}' 2024-09-11T09:52:18.880 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 5] out_osds: [0] dead_osds: [] live_osds: [0, 1, 2, 3, 5, 4] 2024-09-11T09:52:18.880 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-09-11T09:52:18.881 INFO:teuthology.orchestra.run.smithi017.stderr:nodeep-scrub is set 2024-09-11T09:52:18.917 INFO:tasks.workunit.client.0.smithi107.stderr:2024-09-11T09:52:18.918+0000 7f5974d9c640 -1 JournalMetadata: 0x555601a7aa00 failed to locate client: INSPECT 2024-09-11T09:52:18.918 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:52:18.918 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [], 2024-09-11T09:52:18.918 INFO:teuthology.orchestra.run.smithi017.stdout: "num_ops": 0 2024-09-11T09:52:18.919 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:52:18.919 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:52:18.920+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:52:18.921 INFO:tasks.workunit.client.0.smithi107.stderr:+ queue_length=4 2024-09-11T09:52:18.921 INFO:tasks.workunit.client.0.smithi107.stderr:+ tries=2 2024-09-11T09:52:18.921 INFO:tasks.workunit.client.0.smithi107.stderr:+ '[' 2 -lt 10 ']' 2024-09-11T09:52:18.921 INFO:tasks.workunit.client.0.smithi107.stderr:+ '[' 4 -gt 0 ']' 2024-09-11T09:52:18.921 INFO:tasks.workunit.client.0.smithi107.stderr:+ rbd journal inspect --pool rbd --image testrbdnbd27121 --verbose 2024-09-11T09:52:18.928 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-11T09:52:18.978 INFO:tasks.workunit.client.0.smithi107.stdout:Entry: tag_id=1, commit_tid=1 2024-09-11T09:52:18.978 INFO:tasks.workunit.client.0.smithi107.stdout:{ 2024-09-11T09:52:18.978 INFO:tasks.workunit.client.0.smithi107.stdout: "event_type": "AioDiscard", 2024-09-11T09:52:18.978 INFO:tasks.workunit.client.0.smithi107.stdout: "offset": 0, 2024-09-11T09:52:18.978 INFO:tasks.workunit.client.0.smithi107.stdout: "length": 4096, 2024-09-11T09:52:18.978 INFO:tasks.workunit.client.0.smithi107.stdout: "discard_granularity_bytes": 4096, 2024-09-11T09:52:18.978 INFO:tasks.workunit.client.0.smithi107.stdout: "timestamp": "2024-09-11T09:52:16.526805+0000" 2024-09-11T09:52:18.978 INFO:tasks.workunit.client.0.smithi107.stdout:} 2024-09-11T09:52:18.979 INFO:tasks.workunit.client.0.smithi107.stdout:Entry: tag_id=1, commit_tid=2 2024-09-11T09:52:18.979 INFO:tasks.workunit.client.0.smithi107.stdout:{ 2024-09-11T09:52:18.979 INFO:tasks.workunit.client.0.smithi107.stdout: "event_type": "AioDiscard", 2024-09-11T09:52:18.979 INFO:tasks.workunit.client.0.smithi107.stdout: "offset": 0, 2024-09-11T09:52:18.979 INFO:tasks.workunit.client.0.smithi107.stdout: "length": 4096, 2024-09-11T09:52:18.979 INFO:tasks.workunit.client.0.smithi107.stdout: "discard_granularity_bytes": 4096, 2024-09-11T09:52:18.979 INFO:tasks.workunit.client.0.smithi107.stdout: "timestamp": "2024-09-11T09:52:16.544819+0000" 2024-09-11T09:52:18.979 INFO:tasks.workunit.client.0.smithi107.stdout:} 2024-09-11T09:52:18.979 INFO:tasks.workunit.client.0.smithi107.stdout:Entry: tag_id=1, commit_tid=3 2024-09-11T09:52:18.979 INFO:tasks.workunit.client.0.smithi107.stdout:{ 2024-09-11T09:52:18.979 INFO:tasks.workunit.client.0.smithi107.stdout: "event_type": "AioDiscard", 2024-09-11T09:52:18.979 INFO:tasks.workunit.client.0.smithi107.stdout: "offset": 4096, 2024-09-11T09:52:18.979 INFO:tasks.workunit.client.0.smithi107.stdout: "length": 4096, 2024-09-11T09:52:18.979 INFO:tasks.workunit.client.0.smithi107.stdout: "discard_granularity_bytes": 4096, 2024-09-11T09:52:18.979 INFO:tasks.workunit.client.0.smithi107.stdout: "timestamp": "2024-09-11T09:52:16.576535+0000" 2024-09-11T09:52:18.980 INFO:tasks.workunit.client.0.smithi107.stdout:} 2024-09-11T09:52:18.980 INFO:tasks.workunit.client.0.smithi107.stdout:Entry: tag_id=1, commit_tid=4 2024-09-11T09:52:18.980 INFO:tasks.workunit.client.0.smithi107.stdout:{ 2024-09-11T09:52:18.980 INFO:tasks.workunit.client.0.smithi107.stdout: "event_type": "AioDiscard", 2024-09-11T09:52:18.980 INFO:tasks.workunit.client.0.smithi107.stdout: "offset": 4096, 2024-09-11T09:52:18.980 INFO:tasks.workunit.client.0.smithi107.stdout: "length": 4096, 2024-09-11T09:52:18.980 INFO:tasks.workunit.client.0.smithi107.stdout: "discard_granularity_bytes": 4096, 2024-09-11T09:52:18.980 INFO:tasks.workunit.client.0.smithi107.stdout: "timestamp": "2024-09-11T09:52:16.593859+0000" 2024-09-11T09:52:18.980 INFO:tasks.workunit.client.0.smithi107.stdout:} 2024-09-11T09:52:18.980 INFO:tasks.workunit.client.0.smithi107.stdout:Summary: 2024-09-11T09:52:18.980 INFO:tasks.workunit.client.0.smithi107.stdout: 4 entries inspected, 0 errors 2024-09-11T09:52:18.982 INFO:tasks.workunit.client.0.smithi107.stderr:2024-09-11T09:52:18.983+0000 7fdf2cf0e640 -1 JournalMetadata: 0x55b661074a50 failed to locate client: INSPECT 2024-09-11T09:52:18.986 INFO:tasks.workunit.client.0.smithi107.stderr:+ sleep 1 2024-09-11T09:52:19.003 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:52:19.004+0000 7f3c96ec9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31909) UID: 0 2024-09-11T09:52:19.079 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:52:19.079 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [], 2024-09-11T09:52:19.079 INFO:teuthology.orchestra.run.smithi017.stdout: "complaint_time": 30, 2024-09-11T09:52:19.079 INFO:teuthology.orchestra.run.smithi017.stdout: "num_blocked_ops": 0 2024-09-11T09:52:19.079 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:52:19.088 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-11T09:52:19.132 INFO:tasks.ceph.osd.0.smithi017.stderr:2024-09-11T09:52:19.133+0000 7fb73c0f1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31665) UID: 0 2024-09-11T09:52:19.197 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:52:19.197 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 20, 2024-09-11T09:52:19.198 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 600, 2024-09-11T09:52:19.198 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [ 2024-09-11T09:52:19.198 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.198 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:31 2.1 2:8654721b:::rbd_data.1250f5e25b80.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:19.198 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.896024+0000", 2024-09-11T09:52:19.198 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 56.301312021000001, 2024-09-11T09:52:19.198 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.023148401999999998, 2024-09-11T09:52:19.198 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:19.198 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:19.198 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:19.198 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:19.198 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:52:19.198 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:52:19.198 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 31 2024-09-11T09:52:19.199 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.199 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:19.199 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.199 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:19.199 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.896024+0000", 2024-09-11T09:52:19.199 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:19.199 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.199 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.199 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:19.199 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.896024+0000", 2024-09-11T09:52:19.199 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:19.199 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.199 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.200 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:19.200 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.896025+0000", 2024-09-11T09:52:19.200 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.54e-07 2024-09-11T09:52:19.200 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.200 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.200 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:19.200 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.904287+0000", 2024-09-11T09:52:19.200 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0082624729999999993 2024-09-11T09:52:19.200 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.200 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.200 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:19.200 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.904305+0000", 2024-09-11T09:52:19.200 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7912999999999999e-05 2024-09-11T09:52:19.200 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.201 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.201 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:19.201 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.904312+0000", 2024-09-11T09:52:19.201 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.0230000000000004e-06 2024-09-11T09:52:19.201 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.201 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.201 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:19.201 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.904395+0000", 2024-09-11T09:52:19.201 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.2355999999999996e-05 2024-09-11T09:52:19.201 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.201 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.201 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:19.201 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.904499+0000", 2024-09-11T09:52:19.201 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000104672 2024-09-11T09:52:19.202 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.202 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.202 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1", 2024-09-11T09:52:19.202 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.904594+0000", 2024-09-11T09:52:19.202 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.5129000000000003e-05 2024-09-11T09:52:19.202 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.202 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.202 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:19.202 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.918183+0000", 2024-09-11T09:52:19.202 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.013588355999999999 2024-09-11T09:52:19.202 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.202 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.202 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:19.202 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919053+0000", 2024-09-11T09:52:19.203 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00087066800000000001 2024-09-11T09:52:19.203 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.203 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.203 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:19.203 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919125+0000", 2024-09-11T09:52:19.203 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.1333000000000003e-05 2024-09-11T09:52:19.203 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.203 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.203 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:19.203 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919172+0000", 2024-09-11T09:52:19.203 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.7725000000000001e-05 2024-09-11T09:52:19.203 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:19.203 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:19.203 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:19.204 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.204 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.204 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4688.0:42 2.3 e53/15 2:c54742d8:::rbd_data.1250f5e25b80.0000000000000006:head v 53'2, mlcod=0'0)", 2024-09-11T09:52:19.204 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.939272+0000", 2024-09-11T09:52:19.204 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 56.258064036999997, 2024-09-11T09:52:19.204 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.010821958, 2024-09-11T09:52:19.204 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:19.204 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:19.204 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:19.204 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:19.204 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.204 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:19.204 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.939272+0000", 2024-09-11T09:52:19.204 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:19.205 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.205 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.205 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:19.205 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.939272+0000", 2024-09-11T09:52:19.205 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:19.205 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.205 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.205 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:19.205 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.939273+0000", 2024-09-11T09:52:19.205 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.7899999999999997e-07 2024-09-11T09:52:19.205 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.205 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.205 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:19.205 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.944403+0000", 2024-09-11T09:52:19.206 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0051302129999999998 2024-09-11T09:52:19.206 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.206 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.206 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:19.206 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.944413+0000", 2024-09-11T09:52:19.206 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0182e-05 2024-09-11T09:52:19.206 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.206 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.206 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:19.206 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.944420+0000", 2024-09-11T09:52:19.206 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.7000000000000002e-06 2024-09-11T09:52:19.206 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.206 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.207 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:19.207 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.944482+0000", 2024-09-11T09:52:19.207 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.1911999999999999e-05 2024-09-11T09:52:19.207 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.207 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.207 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:19.207 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.944508+0000", 2024-09-11T09:52:19.207 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.6582000000000001e-05 2024-09-11T09:52:19.207 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.207 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.207 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:19.207 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.950050+0000", 2024-09-11T09:52:19.207 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0055414080000000003 2024-09-11T09:52:19.207 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.208 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.208 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:19.208 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.950094+0000", 2024-09-11T09:52:19.208 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.4082000000000002e-05 2024-09-11T09:52:19.208 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:19.208 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:19.208 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:19.208 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.208 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.208 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:47 2.1 2:95f53b3e:::rbd_data.1250f5e25b80.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:19.208 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.948330+0000", 2024-09-11T09:52:19.208 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 56.249005693000001, 2024-09-11T09:52:19.208 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.025156603999999999, 2024-09-11T09:52:19.208 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:19.209 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:19.209 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:19.209 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:19.209 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:52:19.209 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:52:19.209 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 47 2024-09-11T09:52:19.209 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.209 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:19.209 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.209 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:19.209 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.948330+0000", 2024-09-11T09:52:19.209 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:19.209 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.209 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.210 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:19.210 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.948330+0000", 2024-09-11T09:52:19.210 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:19.210 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.210 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.210 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:19.210 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.948333+0000", 2024-09-11T09:52:19.210 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.481e-06 2024-09-11T09:52:19.210 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.210 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.210 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:19.210 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.960263+0000", 2024-09-11T09:52:19.210 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.011930671 2024-09-11T09:52:19.210 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.211 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.211 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:19.211 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.960289+0000", 2024-09-11T09:52:19.211 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.5573000000000001e-05 2024-09-11T09:52:19.211 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.211 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.211 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:19.211 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.960304+0000", 2024-09-11T09:52:19.211 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.4647000000000001e-05 2024-09-11T09:52:19.211 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.211 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.211 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:19.211 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.960390+0000", 2024-09-11T09:52:19.212 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.6088999999999995e-05 2024-09-11T09:52:19.212 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.212 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.212 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:19.212 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.960562+0000", 2024-09-11T09:52:19.212 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00017237400000000001 2024-09-11T09:52:19.212 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.212 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.212 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1", 2024-09-11T09:52:19.212 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.960725+0000", 2024-09-11T09:52:19.212 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00016321599999999999 2024-09-11T09:52:19.212 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.212 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.212 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:19.213 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.969465+0000", 2024-09-11T09:52:19.213 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0087396899999999996 2024-09-11T09:52:19.213 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.213 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.213 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:19.213 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.973393+0000", 2024-09-11T09:52:19.213 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0039274669999999996 2024-09-11T09:52:19.213 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.213 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.213 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:19.213 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.973443+0000", 2024-09-11T09:52:19.213 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.0340000000000003e-05 2024-09-11T09:52:19.213 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.213 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.214 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:19.214 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.973487+0000", 2024-09-11T09:52:19.214 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.4056e-05 2024-09-11T09:52:19.214 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:19.214 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:19.214 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:19.214 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.214 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.214 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:61 2.1 2:9af038b1:::rbd_data.1250f5e25b80.000000000000000f:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:19.214 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.975764+0000", 2024-09-11T09:52:19.214 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 56.221572168999998, 2024-09-11T09:52:19.214 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.016158891000000002, 2024-09-11T09:52:19.214 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:19.214 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:19.215 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:19.215 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:19.215 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:52:19.215 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:52:19.215 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 61 2024-09-11T09:52:19.215 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.215 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:19.215 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.215 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:19.215 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.975764+0000", 2024-09-11T09:52:19.215 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:19.215 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.215 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.216 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:19.216 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.975764+0000", 2024-09-11T09:52:19.216 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:19.216 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.216 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.216 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:19.216 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.975766+0000", 2024-09-11T09:52:19.216 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.767e-06 2024-09-11T09:52:19.216 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.216 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.216 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:19.216 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.983204+0000", 2024-09-11T09:52:19.216 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0074388400000000004 2024-09-11T09:52:19.216 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.217 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.217 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:19.217 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.983226+0000", 2024-09-11T09:52:19.217 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.1279000000000001e-05 2024-09-11T09:52:19.217 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.217 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.217 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:19.217 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.983238+0000", 2024-09-11T09:52:19.217 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2563e-05 2024-09-11T09:52:19.217 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.217 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.217 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:19.217 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.983318+0000", 2024-09-11T09:52:19.218 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.0075999999999997e-05 2024-09-11T09:52:19.218 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.218 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.218 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:19.218 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.983429+0000", 2024-09-11T09:52:19.218 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000110618 2024-09-11T09:52:19.218 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.218 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.218 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1", 2024-09-11T09:52:19.218 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.983525+0000", 2024-09-11T09:52:19.218 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.603e-05 2024-09-11T09:52:19.218 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.218 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.218 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:19.219 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.989004+0000", 2024-09-11T09:52:19.219 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0054785429999999998 2024-09-11T09:52:19.219 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.219 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.219 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:19.219 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.991788+0000", 2024-09-11T09:52:19.219 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0027845729999999998 2024-09-11T09:52:19.219 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.219 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.219 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:19.219 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.991870+0000", 2024-09-11T09:52:19.219 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.1567e-05 2024-09-11T09:52:19.219 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.220 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.220 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:19.220 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.991923+0000", 2024-09-11T09:52:19.220 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.3035000000000003e-05 2024-09-11T09:52:19.220 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:19.220 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:19.220 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:19.220 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.220 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.220 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4694.0:32 2.1 2:8a862b48:ns::rbd_data.1256a0c02879.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:19.220 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.084089+0000", 2024-09-11T09:52:19.220 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 56.113246695000001, 2024-09-11T09:52:19.220 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.020630553999999999, 2024-09-11T09:52:19.221 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:19.221 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:19.221 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:19.221 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:19.221 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4694", 2024-09-11T09:52:19.221 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:52:19.221 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 32 2024-09-11T09:52:19.221 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.221 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:19.221 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.221 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:19.221 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.084089+0000", 2024-09-11T09:52:19.221 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:19.221 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.222 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.222 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:19.222 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.084089+0000", 2024-09-11T09:52:19.222 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:19.222 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.222 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.222 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:19.222 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.084090+0000", 2024-09-11T09:52:19.222 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.2900000000000003e-07 2024-09-11T09:52:19.222 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.222 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.222 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:19.222 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.090866+0000", 2024-09-11T09:52:19.223 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0067756609999999997 2024-09-11T09:52:19.223 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.223 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.223 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:19.223 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.090891+0000", 2024-09-11T09:52:19.223 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.5119000000000001e-05 2024-09-11T09:52:19.223 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.223 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.223 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:19.223 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.090900+0000", 2024-09-11T09:52:19.223 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.4110000000000002e-06 2024-09-11T09:52:19.223 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.223 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.223 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:19.224 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.090968+0000", 2024-09-11T09:52:19.224 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.8041999999999997e-05 2024-09-11T09:52:19.224 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.224 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.224 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:19.224 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.091115+0000", 2024-09-11T09:52:19.224 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00014633499999999999 2024-09-11T09:52:19.224 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.224 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.224 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1", 2024-09-11T09:52:19.224 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.091284+0000", 2024-09-11T09:52:19.224 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000169836 2024-09-11T09:52:19.224 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.224 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.225 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:19.225 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.100896+0000", 2024-09-11T09:52:19.225 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0096118639999999995 2024-09-11T09:52:19.225 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.225 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.225 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:19.225 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.104604+0000", 2024-09-11T09:52:19.225 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0037074619999999999 2024-09-11T09:52:19.225 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.225 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.225 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:19.225 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.104669+0000", 2024-09-11T09:52:19.225 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.4836000000000001e-05 2024-09-11T09:52:19.225 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.226 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.226 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:19.226 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.104720+0000", 2024-09-11T09:52:19.226 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.1258999999999998e-05 2024-09-11T09:52:19.226 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:19.226 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:19.226 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:19.226 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.226 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.226 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4694.0:44 2.3 e53/15 2:cd1a7d1c:ns::rbd_data.1256a0c02879.0000000000000006:head v 53'3, mlcod=16'1)", 2024-09-11T09:52:19.226 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.120387+0000", 2024-09-11T09:52:19.226 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 56.076949042000003, 2024-09-11T09:52:19.226 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.011988468, 2024-09-11T09:52:19.226 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:19.227 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:19.227 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:19.227 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:19.227 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.227 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:19.227 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.120387+0000", 2024-09-11T09:52:19.227 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:19.227 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.227 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.227 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:19.227 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.120387+0000", 2024-09-11T09:52:19.227 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:19.227 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.227 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.228 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:19.228 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.120389+0000", 2024-09-11T09:52:19.228 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.018e-06 2024-09-11T09:52:19.228 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.228 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.228 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:19.228 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.127403+0000", 2024-09-11T09:52:19.228 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0070144769999999999 2024-09-11T09:52:19.228 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.228 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.228 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:19.228 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.127414+0000", 2024-09-11T09:52:19.229 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0991e-05 2024-09-11T09:52:19.229 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.229 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.229 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:19.229 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.127423+0000", 2024-09-11T09:52:19.229 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.8580000000000006e-06 2024-09-11T09:52:19.229 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.229 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.229 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:19.229 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.127468+0000", 2024-09-11T09:52:19.229 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.4830999999999998e-05 2024-09-11T09:52:19.229 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.229 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.229 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:19.230 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.127485+0000", 2024-09-11T09:52:19.230 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6838999999999999e-05 2024-09-11T09:52:19.230 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.230 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.230 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:19.230 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.132341+0000", 2024-09-11T09:52:19.230 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0048557959999999999 2024-09-11T09:52:19.230 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.230 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.230 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:19.230 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.132375+0000", 2024-09-11T09:52:19.230 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.4657999999999999e-05 2024-09-11T09:52:19.230 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:19.230 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:19.231 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:19.231 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.231 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.231 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4694.0:46 2.3 e53/15 2:ce0e1469:ns::rbd_data.1256a0c02879.0000000000000008:head v 53'4, mlcod=16'1)", 2024-09-11T09:52:19.231 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.129008+0000", 2024-09-11T09:52:19.231 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 56.068328456000003, 2024-09-11T09:52:19.231 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0087151220000000005, 2024-09-11T09:52:19.231 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:19.231 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:19.231 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:19.231 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:19.231 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.231 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:19.231 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.129008+0000", 2024-09-11T09:52:19.232 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:19.232 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.232 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.232 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:19.232 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.129008+0000", 2024-09-11T09:52:19.232 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:19.232 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.232 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.232 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:19.232 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.129009+0000", 2024-09-11T09:52:19.232 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.367e-06 2024-09-11T09:52:19.232 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.232 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.232 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:19.233 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.132839+0000", 2024-09-11T09:52:19.233 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0038297610000000001 2024-09-11T09:52:19.233 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.233 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.233 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:19.233 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.132849+0000", 2024-09-11T09:52:19.233 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0672e-05 2024-09-11T09:52:19.233 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.233 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.233 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:19.233 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.132856+0000", 2024-09-11T09:52:19.233 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.1380000000000004e-06 2024-09-11T09:52:19.233 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.234 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.234 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:19.234 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.132891+0000", 2024-09-11T09:52:19.234 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.5712e-05 2024-09-11T09:52:19.234 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.234 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.234 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:19.234 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.132907+0000", 2024-09-11T09:52:19.234 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6166000000000001e-05 2024-09-11T09:52:19.234 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.234 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.234 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:19.234 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.137681+0000", 2024-09-11T09:52:19.234 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0047735299999999998 2024-09-11T09:52:19.235 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.235 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.235 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:19.235 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.137723+0000", 2024-09-11T09:52:19.235 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.1776000000000001e-05 2024-09-11T09:52:19.235 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:19.235 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:19.235 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:19.235 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.235 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.235 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4694.0:48 2.3 e53/15 2:d2090b70:ns::rbd_data.1256a0c02879.0000000000000009:head v 53'5, mlcod=53'2)", 2024-09-11T09:52:19.235 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.135108+0000", 2024-09-11T09:52:19.235 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 56.062227559, 2024-09-11T09:52:19.236 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.010710679000000001, 2024-09-11T09:52:19.236 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:19.236 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:19.236 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:19.236 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:19.236 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.236 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:19.236 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.135108+0000", 2024-09-11T09:52:19.236 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:19.236 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.236 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.236 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:19.236 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.135108+0000", 2024-09-11T09:52:19.237 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:19.237 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.237 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.237 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:19.237 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.135110+0000", 2024-09-11T09:52:19.237 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.122e-06 2024-09-11T09:52:19.237 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.237 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.237 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:19.237 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.140876+0000", 2024-09-11T09:52:19.237 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.005766162 2024-09-11T09:52:19.237 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.237 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.237 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:19.238 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.140888+0000", 2024-09-11T09:52:19.238 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.201e-05 2024-09-11T09:52:19.238 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.238 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.238 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:19.238 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.140894+0000", 2024-09-11T09:52:19.238 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.1739999999999997e-06 2024-09-11T09:52:19.238 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.238 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.238 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:19.238 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.140971+0000", 2024-09-11T09:52:19.238 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.7495000000000003e-05 2024-09-11T09:52:19.238 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.238 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.239 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:19.239 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.140995+0000", 2024-09-11T09:52:19.239 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.3204999999999999e-05 2024-09-11T09:52:19.239 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.239 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.239 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:19.239 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.145757+0000", 2024-09-11T09:52:19.239 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0047627190000000003 2024-09-11T09:52:19.239 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.239 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.239 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:19.239 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.145819+0000", 2024-09-11T09:52:19.239 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.1791999999999994e-05 2024-09-11T09:52:19.240 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:19.240 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:19.240 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:19.240 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.240 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.240 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4694.0:50 2.1 2:83078786:ns::rbd_data.1256a0c02879.000000000000000a:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:19.240 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.141467+0000", 2024-09-11T09:52:19.240 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 56.055868834999998, 2024-09-11T09:52:19.240 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.012785889, 2024-09-11T09:52:19.240 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:19.240 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:19.240 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:19.240 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:19.241 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4694", 2024-09-11T09:52:19.241 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:52:19.241 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 50 2024-09-11T09:52:19.241 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.241 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:19.241 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.241 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:19.241 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.141467+0000", 2024-09-11T09:52:19.241 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:19.241 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.241 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.241 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:19.241 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.141467+0000", 2024-09-11T09:52:19.241 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:19.242 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.242 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.242 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:19.242 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.141468+0000", 2024-09-11T09:52:19.242 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1009999999999999e-06 2024-09-11T09:52:19.242 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.242 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.242 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:19.242 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.146432+0000", 2024-09-11T09:52:19.242 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0049640539999999999 2024-09-11T09:52:19.242 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.242 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.242 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:19.242 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.146446+0000", 2024-09-11T09:52:19.243 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3254e-05 2024-09-11T09:52:19.243 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.243 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.243 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:19.243 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.146452+0000", 2024-09-11T09:52:19.243 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.207e-06 2024-09-11T09:52:19.243 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.243 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.243 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:19.243 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.146510+0000", 2024-09-11T09:52:19.243 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.8322999999999997e-05 2024-09-11T09:52:19.243 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.243 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.243 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:19.244 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.146650+0000", 2024-09-11T09:52:19.244 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00013947100000000001 2024-09-11T09:52:19.244 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.244 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.244 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1", 2024-09-11T09:52:19.244 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.146779+0000", 2024-09-11T09:52:19.244 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000129059 2024-09-11T09:52:19.244 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.244 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.244 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:19.244 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.153778+0000", 2024-09-11T09:52:19.244 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0069997130000000003 2024-09-11T09:52:19.244 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.245 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.245 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:19.245 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.154180+0000", 2024-09-11T09:52:19.245 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00040159800000000003 2024-09-11T09:52:19.245 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.245 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.245 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:19.245 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.154218+0000", 2024-09-11T09:52:19.245 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.7795e-05 2024-09-11T09:52:19.245 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.245 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.245 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:19.245 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.154253+0000", 2024-09-11T09:52:19.246 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.5314000000000001e-05 2024-09-11T09:52:19.246 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:19.246 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:19.246 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:19.246 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.246 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.246 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4694.0:54 2.3 e53/15 2:c44f0253:ns::rbd_data.1256a0c02879.000000000000000b:head v 53'6, mlcod=53'4)", 2024-09-11T09:52:19.246 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.154926+0000", 2024-09-11T09:52:19.246 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 56.042410459999999, 2024-09-11T09:52:19.246 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0089497099999999996, 2024-09-11T09:52:19.246 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:19.246 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:19.246 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:19.246 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:19.247 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.247 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:19.247 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.154926+0000", 2024-09-11T09:52:19.247 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:19.247 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.247 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.247 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:19.247 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.154926+0000", 2024-09-11T09:52:19.247 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:19.247 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.247 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.247 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:19.247 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.154927+0000", 2024-09-11T09:52:19.248 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.7000000000000003e-07 2024-09-11T09:52:19.248 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.248 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.248 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:19.248 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.158712+0000", 2024-09-11T09:52:19.248 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0037855469999999998 2024-09-11T09:52:19.248 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.248 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.248 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:19.248 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.158726+0000", 2024-09-11T09:52:19.248 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3621000000000001e-05 2024-09-11T09:52:19.248 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.248 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.249 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:19.249 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.158732+0000", 2024-09-11T09:52:19.249 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.9089999999999997e-06 2024-09-11T09:52:19.249 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.249 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.249 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:19.249 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.158781+0000", 2024-09-11T09:52:19.249 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.8900000000000003e-05 2024-09-11T09:52:19.249 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.249 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.249 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:19.249 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.158805+0000", 2024-09-11T09:52:19.249 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.44e-05 2024-09-11T09:52:19.250 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.250 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.250 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:19.250 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.163826+0000", 2024-09-11T09:52:19.250 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0050206399999999998 2024-09-11T09:52:19.250 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.250 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.250 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:19.250 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.163875+0000", 2024-09-11T09:52:19.250 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.9722999999999998e-05 2024-09-11T09:52:19.250 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:19.250 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:19.250 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:19.250 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.251 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.251 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4694.0:56 2.3 e53/15 2:c462f029:ns::rbd_data.1256a0c02879.000000000000000c:head v 53'7, mlcod=53'4)", 2024-09-11T09:52:19.251 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.159549+0000", 2024-09-11T09:52:19.251 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 56.037786754000003, 2024-09-11T09:52:19.251 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.015591666000000001, 2024-09-11T09:52:19.251 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:19.251 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:19.251 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:19.251 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:19.251 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.251 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:19.251 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.159549+0000", 2024-09-11T09:52:19.251 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:19.251 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.252 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.252 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:19.252 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.159549+0000", 2024-09-11T09:52:19.252 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:19.252 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.252 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.252 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:19.252 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.159550+0000", 2024-09-11T09:52:19.252 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.1500000000000004e-07 2024-09-11T09:52:19.252 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.252 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.252 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:19.252 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.167834+0000", 2024-09-11T09:52:19.252 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0082840220000000003 2024-09-11T09:52:19.253 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.253 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.253 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:19.253 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.167857+0000", 2024-09-11T09:52:19.253 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.3337e-05 2024-09-11T09:52:19.253 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.253 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.253 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:19.253 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.167869+0000", 2024-09-11T09:52:19.253 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1556000000000001e-05 2024-09-11T09:52:19.253 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.253 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.253 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:19.253 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.167939+0000", 2024-09-11T09:52:19.254 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.0674999999999997e-05 2024-09-11T09:52:19.254 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.254 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.254 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:19.254 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.167967+0000", 2024-09-11T09:52:19.254 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.7322999999999999e-05 2024-09-11T09:52:19.254 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.254 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.254 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:19.254 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.175079+0000", 2024-09-11T09:52:19.254 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0071126150000000001 2024-09-11T09:52:19.254 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.254 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.255 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:19.255 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.175141+0000", 2024-09-11T09:52:19.255 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.1523e-05 2024-09-11T09:52:19.255 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:19.255 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:19.255 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:19.255 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.255 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.255 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.5314.0:31 2.1 2:8654721b:::rbd_data.1250f5e25b80.0000000000000000:head [write 393216~131072] snapc 3=[3] ondisk+write+known_if_redirected+supports_pool_eio e68)", 2024-09-11T09:52:19.255 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:50.230850+0000", 2024-09-11T09:52:19.255 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 28.966485992999999, 2024-09-11T09:52:19.255 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0073772539999999998, 2024-09-11T09:52:19.255 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:19.255 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:19.256 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:19.256 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:19.256 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.5314", 2024-09-11T09:52:19.256 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3853995732", 2024-09-11T09:52:19.256 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 31 2024-09-11T09:52:19.256 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.256 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:19.256 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.256 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:19.256 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.230850+0000", 2024-09-11T09:52:19.256 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:19.256 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.256 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.257 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:19.257 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.230850+0000", 2024-09-11T09:52:19.257 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:19.257 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.257 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.257 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:19.257 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.230850+0000", 2024-09-11T09:52:19.257 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.32e-07 2024-09-11T09:52:19.257 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.257 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.257 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:19.257 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.230908+0000", 2024-09-11T09:52:19.257 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.7698000000000002e-05 2024-09-11T09:52:19.257 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.258 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.258 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:19.258 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.230915+0000", 2024-09-11T09:52:19.258 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.81e-06 2024-09-11T09:52:19.258 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.258 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.258 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:19.258 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.230920+0000", 2024-09-11T09:52:19.258 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.9660000000000004e-06 2024-09-11T09:52:19.258 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.258 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.258 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:19.258 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.235055+0000", 2024-09-11T09:52:19.258 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0041354629999999998 2024-09-11T09:52:19.259 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.259 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.259 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:19.259 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.235145+0000", 2024-09-11T09:52:19.259 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.9362000000000003e-05 2024-09-11T09:52:19.259 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.259 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.259 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1", 2024-09-11T09:52:19.259 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.235282+0000", 2024-09-11T09:52:19.259 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00013710999999999999 2024-09-11T09:52:19.259 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.259 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.259 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:19.259 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.237335+0000", 2024-09-11T09:52:19.260 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.002053165 2024-09-11T09:52:19.260 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.260 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.260 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:19.260 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.238106+0000", 2024-09-11T09:52:19.260 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000771015 2024-09-11T09:52:19.260 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.260 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.260 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:19.260 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.238165+0000", 2024-09-11T09:52:19.260 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.8588000000000002e-05 2024-09-11T09:52:19.260 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.260 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.261 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:19.261 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.238227+0000", 2024-09-11T09:52:19.261 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.2644999999999994e-05 2024-09-11T09:52:19.261 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:19.261 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:19.261 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:19.261 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.261 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.261 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.5314.0:32 2.1 2:8654721b:::rbd_data.1250f5e25b80.0000000000000000:head [write 524288~131072] snapc 3=[3] ondisk+write+known_if_redirected+supports_pool_eio e68)", 2024-09-11T09:52:19.262 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:50.230932+0000", 2024-09-11T09:52:19.262 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 28.966404367999999, 2024-09-11T09:52:19.262 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0074882129999999996, 2024-09-11T09:52:19.262 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:19.262 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:19.262 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:19.262 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:19.262 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.5314", 2024-09-11T09:52:19.262 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3853995732", 2024-09-11T09:52:19.262 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 32 2024-09-11T09:52:19.262 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.262 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:19.262 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.262 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:19.263 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.230932+0000", 2024-09-11T09:52:19.263 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:19.263 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.263 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.263 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:19.263 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.230932+0000", 2024-09-11T09:52:19.263 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:19.263 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.263 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.263 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:19.263 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.230932+0000", 2024-09-11T09:52:19.263 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.9999999999999999e-07 2024-09-11T09:52:19.263 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.263 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.264 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:19.264 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.231022+0000", 2024-09-11T09:52:19.264 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.0439999999999995e-05 2024-09-11T09:52:19.264 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.264 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.264 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:19.264 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.231030+0000", 2024-09-11T09:52:19.264 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.3350000000000001e-06 2024-09-11T09:52:19.264 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.264 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.264 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:19.264 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.231034+0000", 2024-09-11T09:52:19.264 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.9759999999999997e-06 2024-09-11T09:52:19.265 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.265 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.265 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:19.265 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.235837+0000", 2024-09-11T09:52:19.265 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0048031910000000001 2024-09-11T09:52:19.265 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.265 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.265 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:19.265 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.235904+0000", 2024-09-11T09:52:19.265 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.7181999999999995e-05 2024-09-11T09:52:19.265 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.265 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.265 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1", 2024-09-11T09:52:19.265 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.236040+0000", 2024-09-11T09:52:19.266 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00013545199999999999 2024-09-11T09:52:19.266 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.266 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.266 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:19.266 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.238242+0000", 2024-09-11T09:52:19.266 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.002202809 2024-09-11T09:52:19.266 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.266 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.266 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:19.266 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.238301+0000", 2024-09-11T09:52:19.266 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.8573999999999999e-05 2024-09-11T09:52:19.266 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.266 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.266 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:19.267 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.238378+0000", 2024-09-11T09:52:19.267 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.7323999999999994e-05 2024-09-11T09:52:19.267 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.267 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.267 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:19.267 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.238420+0000", 2024-09-11T09:52:19.267 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.163e-05 2024-09-11T09:52:19.267 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:19.267 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:19.267 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:19.267 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.267 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.267 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.5314.0:33 2.1 2:8654721b:::rbd_data.1250f5e25b80.0000000000000000:head [write 655360~131072] snapc 3=[3] ondisk+write+known_if_redirected+supports_pool_eio e68)", 2024-09-11T09:52:19.267 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:50.231041+0000", 2024-09-11T09:52:19.268 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 28.966295208999998, 2024-09-11T09:52:19.268 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0075627139999999999, 2024-09-11T09:52:19.268 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:19.268 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:19.268 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:19.268 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:19.268 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.5314", 2024-09-11T09:52:19.268 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3853995732", 2024-09-11T09:52:19.268 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 33 2024-09-11T09:52:19.268 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.268 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:19.268 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.268 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:19.269 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.231041+0000", 2024-09-11T09:52:19.269 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:19.269 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.269 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.269 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:19.269 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.231041+0000", 2024-09-11T09:52:19.269 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:19.269 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.269 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.269 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:19.269 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.231041+0000", 2024-09-11T09:52:19.269 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.9499999999999998e-07 2024-09-11T09:52:19.269 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.269 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.270 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:19.270 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.231111+0000", 2024-09-11T09:52:19.270 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.0166999999999994e-05 2024-09-11T09:52:19.270 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.270 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.270 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:19.270 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.231122+0000", 2024-09-11T09:52:19.270 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0518e-05 2024-09-11T09:52:19.270 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.270 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.270 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:19.270 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.231128+0000", 2024-09-11T09:52:19.270 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.3509999999999998e-06 2024-09-11T09:52:19.270 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.271 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.271 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:19.271 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.236622+0000", 2024-09-11T09:52:19.271 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0054934989999999998 2024-09-11T09:52:19.271 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.271 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.271 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:19.271 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.236719+0000", 2024-09-11T09:52:19.271 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.7497999999999994e-05 2024-09-11T09:52:19.271 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.271 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.271 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1", 2024-09-11T09:52:19.271 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.236854+0000", 2024-09-11T09:52:19.271 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000134535 2024-09-11T09:52:19.272 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.272 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.272 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:19.272 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.238491+0000", 2024-09-11T09:52:19.272 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0016368610000000001 2024-09-11T09:52:19.272 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.272 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.272 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:19.272 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.238504+0000", 2024-09-11T09:52:19.272 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3927999999999999e-05 2024-09-11T09:52:19.272 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.272 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.272 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:19.272 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.238561+0000", 2024-09-11T09:52:19.273 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.6029999999999997e-05 2024-09-11T09:52:19.273 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.273 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.273 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:19.273 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.238604+0000", 2024-09-11T09:52:19.273 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.3031999999999997e-05 2024-09-11T09:52:19.273 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:19.273 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:19.273 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:19.273 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.273 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.273 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.5314.0:34 2.1 2:8654721b:::rbd_data.1250f5e25b80.0000000000000000:head [write 786432~131072] snapc 3=[3] ondisk+write+known_if_redirected+supports_pool_eio e68)", 2024-09-11T09:52:19.273 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:50.231139+0000", 2024-09-11T09:52:19.274 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 28.966196606, 2024-09-11T09:52:19.274 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0083689690000000004, 2024-09-11T09:52:19.274 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:19.274 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:19.274 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:19.274 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:19.274 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.5314", 2024-09-11T09:52:19.274 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3853995732", 2024-09-11T09:52:19.274 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 34 2024-09-11T09:52:19.274 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.274 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:19.274 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.274 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:19.274 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.231139+0000", 2024-09-11T09:52:19.275 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:19.275 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.275 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.275 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:19.275 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.231139+0000", 2024-09-11T09:52:19.275 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:19.275 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.275 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.275 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:19.275 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.231140+0000", 2024-09-11T09:52:19.275 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.1699999999999998e-07 2024-09-11T09:52:19.275 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.275 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.276 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:19.276 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.231247+0000", 2024-09-11T09:52:19.276 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000107088 2024-09-11T09:52:19.276 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.276 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.276 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:19.276 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.231260+0000", 2024-09-11T09:52:19.276 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.289e-05 2024-09-11T09:52:19.276 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.276 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.276 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:19.276 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.231272+0000", 2024-09-11T09:52:19.276 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2371999999999999e-05 2024-09-11T09:52:19.276 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.277 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.277 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:19.277 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.237380+0000", 2024-09-11T09:52:19.277 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.006107683 2024-09-11T09:52:19.277 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.277 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.277 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:19.277 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.237434+0000", 2024-09-11T09:52:19.277 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.4143e-05 2024-09-11T09:52:19.277 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.277 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.277 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1", 2024-09-11T09:52:19.277 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.237541+0000", 2024-09-11T09:52:19.278 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00010676800000000001 2024-09-11T09:52:19.278 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.278 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.278 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:19.278 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.239390+0000", 2024-09-11T09:52:19.278 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.001848861 2024-09-11T09:52:19.278 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.278 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.278 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:19.278 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.239409+0000", 2024-09-11T09:52:19.278 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.9505e-05 2024-09-11T09:52:19.278 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.278 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.278 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:19.279 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.239463+0000", 2024-09-11T09:52:19.279 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.3838000000000002e-05 2024-09-11T09:52:19.279 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.279 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.279 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:19.279 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.239508+0000", 2024-09-11T09:52:19.279 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.5303999999999999e-05 2024-09-11T09:52:19.279 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:19.279 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:19.279 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:19.279 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.279 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.279 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.5314.0:35 2.1 2:8654721b:::rbd_data.1250f5e25b80.0000000000000000:head [write 917504~131072] snapc 3=[3] ondisk+write+known_if_redirected+supports_pool_eio e68)", 2024-09-11T09:52:19.279 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:50.231287+0000", 2024-09-11T09:52:19.280 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 28.966049271999999, 2024-09-11T09:52:19.280 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0086930730000000008, 2024-09-11T09:52:19.280 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:19.280 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:19.280 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:19.280 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:19.280 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.5314", 2024-09-11T09:52:19.280 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3853995732", 2024-09-11T09:52:19.280 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 35 2024-09-11T09:52:19.280 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.280 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:19.280 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.280 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:19.280 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.231287+0000", 2024-09-11T09:52:19.281 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:19.281 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.281 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.281 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:19.281 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.231287+0000", 2024-09-11T09:52:19.281 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:19.281 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.281 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.281 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:19.281 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.231287+0000", 2024-09-11T09:52:19.281 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.7700000000000004e-07 2024-09-11T09:52:19.281 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.281 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.281 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:19.282 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.231354+0000", 2024-09-11T09:52:19.282 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.6304999999999996e-05 2024-09-11T09:52:19.282 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.282 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.282 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:19.282 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.231367+0000", 2024-09-11T09:52:19.282 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2952999999999999e-05 2024-09-11T09:52:19.282 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.282 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.282 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:19.282 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.231374+0000", 2024-09-11T09:52:19.282 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.0469999999999996e-06 2024-09-11T09:52:19.282 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.282 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.283 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:19.283 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.238650+0000", 2024-09-11T09:52:19.283 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0072760280000000004 2024-09-11T09:52:19.283 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.283 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.283 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:19.283 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.238732+0000", 2024-09-11T09:52:19.283 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.2194000000000006e-05 2024-09-11T09:52:19.283 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.283 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.283 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1", 2024-09-11T09:52:19.283 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.238876+0000", 2024-09-11T09:52:19.283 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00014458 2024-09-11T09:52:19.283 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.284 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.284 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:19.284 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.239739+0000", 2024-09-11T09:52:19.284 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00086244499999999999 2024-09-11T09:52:19.284 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.284 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.284 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:19.284 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.239917+0000", 2024-09-11T09:52:19.284 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00017777299999999999 2024-09-11T09:52:19.284 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.284 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.284 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:19.284 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.239954+0000", 2024-09-11T09:52:19.285 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.7379999999999998e-05 2024-09-11T09:52:19.285 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.285 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.285 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:19.285 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.239980+0000", 2024-09-11T09:52:19.285 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.5791000000000001e-05 2024-09-11T09:52:19.285 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:19.285 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:19.285 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:19.285 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.285 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.285 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.5314.0:70 2.1 2:8654721b:::rbd_data.1250f5e25b80.0000000000000000:head [write 524288~131072] snapc 5=[5,3] ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-09-11T09:52:19.285 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:50.824556+0000", 2024-09-11T09:52:19.285 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 28.372780470999999, 2024-09-11T09:52:19.286 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0072794469999999997, 2024-09-11T09:52:19.286 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:19.286 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:19.286 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:19.286 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:19.286 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.5314", 2024-09-11T09:52:19.286 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3853995732", 2024-09-11T09:52:19.286 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 70 2024-09-11T09:52:19.286 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.286 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:19.286 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.286 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:19.286 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.824556+0000", 2024-09-11T09:52:19.286 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:19.287 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.287 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.287 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:19.287 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.824556+0000", 2024-09-11T09:52:19.287 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:19.287 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.287 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.287 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:19.287 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.824564+0000", 2024-09-11T09:52:19.287 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.2409999999999995e-06 2024-09-11T09:52:19.287 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.287 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.287 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:19.287 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.824613+0000", 2024-09-11T09:52:19.288 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.9579000000000001e-05 2024-09-11T09:52:19.288 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.288 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.288 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:19.288 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.824638+0000", 2024-09-11T09:52:19.288 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.4510000000000001e-05 2024-09-11T09:52:19.288 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.288 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.288 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:19.288 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.824653+0000", 2024-09-11T09:52:19.288 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.4831000000000001e-05 2024-09-11T09:52:19.288 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.288 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.288 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:19.289 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.829072+0000", 2024-09-11T09:52:19.289 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0044195700000000003 2024-09-11T09:52:19.289 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.289 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.289 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:19.289 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.829166+0000", 2024-09-11T09:52:19.289 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.3212000000000001e-05 2024-09-11T09:52:19.289 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.289 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.289 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1", 2024-09-11T09:52:19.289 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.829309+0000", 2024-09-11T09:52:19.289 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00014301900000000001 2024-09-11T09:52:19.289 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.290 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.290 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:19.290 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.831074+0000", 2024-09-11T09:52:19.290 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.001765305 2024-09-11T09:52:19.290 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.290 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.290 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:19.290 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.831751+0000", 2024-09-11T09:52:19.290 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00067694000000000001 2024-09-11T09:52:19.290 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.290 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.290 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:19.290 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.831803+0000", 2024-09-11T09:52:19.291 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.2441000000000002e-05 2024-09-11T09:52:19.291 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.291 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.291 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:19.291 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.831835+0000", 2024-09-11T09:52:19.291 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.1798999999999998e-05 2024-09-11T09:52:19.291 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:19.291 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:19.291 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:19.291 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.291 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.291 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.5314.0:71 2.1 2:8654721b:::rbd_data.1250f5e25b80.0000000000000000:head [write 655360~131072] snapc 5=[5,3] ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-09-11T09:52:19.291 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:50.824662+0000", 2024-09-11T09:52:19.291 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 28.372674355000001, 2024-09-11T09:52:19.292 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0082172219999999997, 2024-09-11T09:52:19.292 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:19.292 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:19.292 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:19.292 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:19.292 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.5314", 2024-09-11T09:52:19.292 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3853995732", 2024-09-11T09:52:19.292 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 71 2024-09-11T09:52:19.292 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.292 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:19.292 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.292 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:19.292 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.824662+0000", 2024-09-11T09:52:19.293 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:19.293 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.293 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.293 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:19.293 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.824662+0000", 2024-09-11T09:52:19.293 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:19.293 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.293 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.293 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:19.293 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.824662+0000", 2024-09-11T09:52:19.293 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.0699999999999998e-07 2024-09-11T09:52:19.293 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.293 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.294 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:19.294 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.824859+0000", 2024-09-11T09:52:19.294 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00019725799999999999 2024-09-11T09:52:19.294 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.294 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.294 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:19.294 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.824906+0000", 2024-09-11T09:52:19.294 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.7162e-05 2024-09-11T09:52:19.294 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.294 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.294 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:19.294 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.824926+0000", 2024-09-11T09:52:19.294 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.0038999999999999e-05 2024-09-11T09:52:19.294 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.295 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.295 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:19.295 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.830313+0000", 2024-09-11T09:52:19.295 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0053868409999999999 2024-09-11T09:52:19.295 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.295 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.295 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:19.295 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.830359+0000", 2024-09-11T09:52:19.295 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.5370000000000001e-05 2024-09-11T09:52:19.295 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.295 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.295 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1", 2024-09-11T09:52:19.295 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.830460+0000", 2024-09-11T09:52:19.295 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00010171299999999999 2024-09-11T09:52:19.296 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.296 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.296 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:19.296 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.831843+0000", 2024-09-11T09:52:19.296 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.001382501 2024-09-11T09:52:19.296 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.296 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.296 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:19.296 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.832769+0000", 2024-09-11T09:52:19.296 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00092638099999999999 2024-09-11T09:52:19.296 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.296 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.296 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:19.297 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.832835+0000", 2024-09-11T09:52:19.297 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.6106999999999996e-05 2024-09-11T09:52:19.297 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.297 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.297 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:19.297 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.832879+0000", 2024-09-11T09:52:19.297 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.3543e-05 2024-09-11T09:52:19.297 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:19.297 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:19.297 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:19.297 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.297 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.297 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.5314.0:72 2.1 2:8654721b:::rbd_data.1250f5e25b80.0000000000000000:head [write 786432~131072] snapc 5=[5,3] ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-09-11T09:52:19.298 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:50.824938+0000", 2024-09-11T09:52:19.298 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 28.372398053000001, 2024-09-11T09:52:19.298 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0082389799999999999, 2024-09-11T09:52:19.298 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:19.298 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:19.298 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:19.298 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:19.298 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.5314", 2024-09-11T09:52:19.298 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3853995732", 2024-09-11T09:52:19.298 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 72 2024-09-11T09:52:19.298 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.298 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:19.298 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.298 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:19.299 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.824938+0000", 2024-09-11T09:52:19.299 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:19.299 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.299 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.299 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:19.299 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.824938+0000", 2024-09-11T09:52:19.299 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:19.299 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.299 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.299 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:19.299 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.824939+0000", 2024-09-11T09:52:19.299 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.6199999999999997e-07 2024-09-11T09:52:19.299 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.300 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.300 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:19.300 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.825028+0000", 2024-09-11T09:52:19.300 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.9105999999999997e-05 2024-09-11T09:52:19.300 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.300 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.300 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:19.300 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.825042+0000", 2024-09-11T09:52:19.300 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3760999999999999e-05 2024-09-11T09:52:19.300 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.300 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.300 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:19.300 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.825061+0000", 2024-09-11T09:52:19.300 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.9088000000000001e-05 2024-09-11T09:52:19.301 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.301 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.301 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:19.301 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.831122+0000", 2024-09-11T09:52:19.301 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.006061299 2024-09-11T09:52:19.301 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.301 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.301 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:19.301 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.831195+0000", 2024-09-11T09:52:19.301 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.2957e-05 2024-09-11T09:52:19.301 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.301 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.301 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1", 2024-09-11T09:52:19.301 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.831301+0000", 2024-09-11T09:52:19.302 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000106049 2024-09-11T09:52:19.302 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.302 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.302 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:19.302 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.832893+0000", 2024-09-11T09:52:19.302 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0015923770000000001 2024-09-11T09:52:19.302 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.302 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.302 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:19.302 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.833076+0000", 2024-09-11T09:52:19.302 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000182855 2024-09-11T09:52:19.302 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.302 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.302 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:19.303 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.833137+0000", 2024-09-11T09:52:19.303 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.1193999999999997e-05 2024-09-11T09:52:19.303 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.303 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.303 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:19.303 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.833177+0000", 2024-09-11T09:52:19.303 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.9632000000000003e-05 2024-09-11T09:52:19.303 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:19.303 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:19.303 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:19.303 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.303 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.303 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.5314.0:73 2.1 2:8654721b:::rbd_data.1250f5e25b80.0000000000000000:head [write 917504~131072] snapc 5=[5,3] ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-09-11T09:52:19.303 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:50.825071+0000", 2024-09-11T09:52:19.304 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 28.372264546, 2024-09-11T09:52:19.304 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0084039160000000009, 2024-09-11T09:52:19.304 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:19.304 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:19.304 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:19.304 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:19.304 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.5314", 2024-09-11T09:52:19.304 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3853995732", 2024-09-11T09:52:19.304 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 73 2024-09-11T09:52:19.304 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.304 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:19.304 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.304 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:19.304 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.825071+0000", 2024-09-11T09:52:19.305 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:19.305 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.305 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.305 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:19.305 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.825071+0000", 2024-09-11T09:52:19.305 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:19.305 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.305 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.305 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:19.305 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.825072+0000", 2024-09-11T09:52:19.305 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.5499999999999998e-07 2024-09-11T09:52:19.305 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.305 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.306 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:19.306 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.825124+0000", 2024-09-11T09:52:19.306 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.2197000000000002e-05 2024-09-11T09:52:19.306 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.306 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.306 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:19.306 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.825132+0000", 2024-09-11T09:52:19.306 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.0290000000000005e-06 2024-09-11T09:52:19.306 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.306 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.306 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:19.306 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.825136+0000", 2024-09-11T09:52:19.306 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.9380000000000002e-06 2024-09-11T09:52:19.306 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.306 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.307 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:19.307 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.832001+0000", 2024-09-11T09:52:19.307 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0068650949999999999 2024-09-11T09:52:19.307 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.307 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.307 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:19.307 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.832091+0000", 2024-09-11T09:52:19.307 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.9524999999999994e-05 2024-09-11T09:52:19.307 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.307 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.307 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1", 2024-09-11T09:52:19.307 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.832234+0000", 2024-09-11T09:52:19.307 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000142783 2024-09-11T09:52:19.307 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.308 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.308 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:19.308 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.833198+0000", 2024-09-11T09:52:19.308 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00096476899999999998 2024-09-11T09:52:19.308 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.308 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.308 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:19.308 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.833386+0000", 2024-09-11T09:52:19.308 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00018772 2024-09-11T09:52:19.308 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.308 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.308 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:19.308 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.833440+0000", 2024-09-11T09:52:19.308 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.4228999999999999e-05 2024-09-11T09:52:19.309 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:19.309 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:19.309 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:19.309 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.833475+0000", 2024-09-11T09:52:19.309 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.5076000000000001e-05 2024-09-11T09:52:19.309 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:19.309 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:19.309 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:19.309 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:19.309 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:19.309 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:52:19.311 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-11T09:52:19.311 INFO:tasks.ceph.osd.0.smithi017.stderr:2024-09-11T09:52:19.311+0000 7fb73c0f1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31665) UID: 0 2024-09-11T09:52:19.368 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T09:52:19.412 INFO:tasks.ceph.osd.2.smithi017.stderr:2024-09-11T09:52:19.413+0000 7fd20b170640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31672) UID: 0 2024-09-11T09:52:19.422 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-11T09:52:19.422 INFO:teuthology.orchestra.run.smithi096.stdout: "ops": [], 2024-09-11T09:52:19.422 INFO:teuthology.orchestra.run.smithi096.stdout: "num_ops": 0 2024-09-11T09:52:19.422 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-11T09:52:19.431 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-11T09:52:19.492 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-11T09:52:19.492 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T09:52:19.508 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-11T09:52:19.509 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T09:52:19.513 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-09-11T09:52:19.514+0000 7f7b6c7d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31904) UID: 0 2024-09-11T09:52:19.525 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-11T09:52:19.525 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T09:52:19.533 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-11T09:52:19.533 INFO:teuthology.orchestra.run.smithi096.stdout: "ops": [], 2024-09-11T09:52:19.533 INFO:teuthology.orchestra.run.smithi096.stdout: "complaint_time": 30, 2024-09-11T09:52:19.534 INFO:teuthology.orchestra.run.smithi096.stdout: "num_blocked_ops": 0 2024-09-11T09:52:19.534 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-11T09:52:19.542 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-11T09:52:19.543 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-11T09:52:19.543 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T09:52:19.560 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-11T09:52:19.560 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T09:52:19.577 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-11T09:52:19.577 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T09:52:19.630 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:52:19.631+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:52:19.685 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:52:19.685 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:52:19.694 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-11T09:52:19.737 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:52:19.738+0000 7f3a7b90a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 42475) UID: 0 2024-09-11T09:52:19.797 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:52:19.797 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:52:19.806 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-11T09:52:19.815 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:52:19.816+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:52:19.881 INFO:teuthology.orchestra.run.smithi017.stderr:reweighted osd.1 to 1 (10000) 2024-09-11T09:52:19.895 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-09-11T09:52:19.937 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:52:19.938+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:52:19.949 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:52:19.950 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:52:19.958 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-11T09:52:19.987 INFO:tasks.workunit.client.0.smithi107.stderr:++ rbd journal inspect --pool rbd --image testrbdnbd27121 2024-09-11T09:52:19.987 INFO:tasks.workunit.client.0.smithi107.stderr:++ awk '/entries inspected/ {print $1}' 2024-09-11T09:52:20.017 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:52:20.017+0000 7f3c96ec9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31909) UID: 0 2024-09-11T09:52:20.043 INFO:tasks.workunit.client.0.smithi107.stderr:2024-09-11T09:52:20.044+0000 7f63467fc640 -1 JournalMetadata: 0x55e9572f9230 failed to locate client: INSPECT 2024-09-11T09:52:20.047 INFO:tasks.workunit.client.0.smithi107.stderr:+ queue_length=4 2024-09-11T09:52:20.047 INFO:tasks.workunit.client.0.smithi107.stderr:+ tries=3 2024-09-11T09:52:20.047 INFO:tasks.workunit.client.0.smithi107.stderr:+ '[' 3 -lt 10 ']' 2024-09-11T09:52:20.047 INFO:tasks.workunit.client.0.smithi107.stderr:+ '[' 4 -gt 0 ']' 2024-09-11T09:52:20.047 INFO:tasks.workunit.client.0.smithi107.stderr:+ rbd journal inspect --pool rbd --image testrbdnbd27121 --verbose 2024-09-11T09:52:20.063 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:52:20.063 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:52:20.072 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-11T09:52:20.100 INFO:tasks.workunit.client.0.smithi107.stdout:Entry: tag_id=1, commit_tid=1 2024-09-11T09:52:20.100 INFO:tasks.workunit.client.0.smithi107.stdout:{ 2024-09-11T09:52:20.100 INFO:tasks.workunit.client.0.smithi107.stdout: "event_type": "AioDiscard", 2024-09-11T09:52:20.100 INFO:tasks.workunit.client.0.smithi107.stdout: "offset": 0, 2024-09-11T09:52:20.101 INFO:tasks.workunit.client.0.smithi107.stdout: "length": 4096, 2024-09-11T09:52:20.101 INFO:tasks.workunit.client.0.smithi107.stdout: "discard_granularity_bytes": 4096, 2024-09-11T09:52:20.101 INFO:tasks.workunit.client.0.smithi107.stdout: "timestamp": "2024-09-11T09:52:16.526805+0000" 2024-09-11T09:52:20.101 INFO:tasks.workunit.client.0.smithi107.stdout:} 2024-09-11T09:52:20.101 INFO:tasks.workunit.client.0.smithi107.stdout:Entry: tag_id=1, commit_tid=2 2024-09-11T09:52:20.101 INFO:tasks.workunit.client.0.smithi107.stdout:{ 2024-09-11T09:52:20.101 INFO:tasks.workunit.client.0.smithi107.stdout: "event_type": "AioDiscard", 2024-09-11T09:52:20.101 INFO:tasks.workunit.client.0.smithi107.stdout: "offset": 0, 2024-09-11T09:52:20.101 INFO:tasks.workunit.client.0.smithi107.stdout: "length": 4096, 2024-09-11T09:52:20.101 INFO:tasks.workunit.client.0.smithi107.stdout: "discard_granularity_bytes": 4096, 2024-09-11T09:52:20.101 INFO:tasks.workunit.client.0.smithi107.stdout: "timestamp": "2024-09-11T09:52:16.544819+0000" 2024-09-11T09:52:20.101 INFO:tasks.workunit.client.0.smithi107.stdout:} 2024-09-11T09:52:20.102 INFO:tasks.workunit.client.0.smithi107.stdout:Entry: tag_id=1, commit_tid=3 2024-09-11T09:52:20.102 INFO:tasks.workunit.client.0.smithi107.stdout:{ 2024-09-11T09:52:20.102 INFO:tasks.workunit.client.0.smithi107.stdout: "event_type": "AioDiscard", 2024-09-11T09:52:20.102 INFO:tasks.workunit.client.0.smithi107.stdout: "offset": 4096, 2024-09-11T09:52:20.102 INFO:tasks.workunit.client.0.smithi107.stdout: "length": 4096, 2024-09-11T09:52:20.102 INFO:tasks.workunit.client.0.smithi107.stdout: "discard_granularity_bytes": 4096, 2024-09-11T09:52:20.102 INFO:tasks.workunit.client.0.smithi107.stdout: "timestamp": "2024-09-11T09:52:16.576535+0000" 2024-09-11T09:52:20.102 INFO:tasks.workunit.client.0.smithi107.stdout:} 2024-09-11T09:52:20.102 INFO:tasks.workunit.client.0.smithi107.stdout:Entry: tag_id=1, commit_tid=4 2024-09-11T09:52:20.102 INFO:tasks.workunit.client.0.smithi107.stdout:{ 2024-09-11T09:52:20.102 INFO:tasks.workunit.client.0.smithi107.stdout: "event_type": "AioDiscard", 2024-09-11T09:52:20.102 INFO:tasks.workunit.client.0.smithi107.stdout: "offset": 4096, 2024-09-11T09:52:20.103 INFO:tasks.workunit.client.0.smithi107.stdout: "length": 4096, 2024-09-11T09:52:20.103 INFO:tasks.workunit.client.0.smithi107.stdout: "discard_granularity_bytes": 4096, 2024-09-11T09:52:20.103 INFO:tasks.workunit.client.0.smithi107.stdout: "timestamp": "2024-09-11T09:52:16.593859+0000" 2024-09-11T09:52:20.103 INFO:tasks.workunit.client.0.smithi107.stdout:} 2024-09-11T09:52:20.103 INFO:tasks.workunit.client.0.smithi107.stdout:Summary: 2024-09-11T09:52:20.103 INFO:tasks.workunit.client.0.smithi107.stdout: 4 entries inspected, 0 errors 2024-09-11T09:52:20.103 INFO:tasks.workunit.client.0.smithi107.stderr:2024-09-11T09:52:20.103+0000 7f7ff679b640 -1 JournalMetadata: 0x55772bcd5a40 failed to locate client: INSPECT 2024-09-11T09:52:20.106 INFO:tasks.workunit.client.0.smithi107.stderr:+ sleep 1 2024-09-11T09:52:20.155 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:52:20.156+0000 7f3a7b90a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 42475) UID: 0 2024-09-11T09:52:20.215 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:52:20.215 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:52:20.219 INFO:tasks.ceph.osd.2.smithi017.stderr:2024-09-11T09:52:20.220+0000 7fd20b170640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31672) UID: 0 2024-09-11T09:52:20.224 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-11T09:52:20.319 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:52:20.320+0000 7f3a7b90a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 42475) UID: 0 2024-09-11T09:52:20.367 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:52:20.367 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:52:20.376 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-11T09:52:20.460 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-09-11T09:52:20.461+0000 7f7b6c7d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31904) UID: 0 2024-09-11T09:52:20.519 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:52:20.520 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:52:20.521 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:52:20.521+0000 7f3c96ec9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31909) UID: 0 2024-09-11T09:52:20.528 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-11T09:52:20.622 INFO:tasks.ceph.osd.0.smithi017.stderr:2024-09-11T09:52:20.622+0000 7fb73c0f1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31665) UID: 0 2024-09-11T09:52:20.639 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:52:20.639 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:52:20.647 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-11T09:52:20.732 INFO:tasks.ceph.osd.0.smithi017.stderr:2024-09-11T09:52:20.733+0000 7fb73c0f1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31665) UID: 0 2024-09-11T09:52:20.796 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:52:20.796 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:52:20.805 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-11T09:52:20.823 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:52:20.824+0000 7f3a7b90a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 42475) UID: 0 2024-09-11T09:52:20.886 INFO:teuthology.orchestra.run.smithi017.stderr:reweighted osd.2 to 1 (10000) 2024-09-11T09:52:20.894 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-11T09:52:20.901 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-09-11T09:52:20.925 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-09-11T09:52:20.926+0000 7f7b6c7d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31904) UID: 0 2024-09-11T09:52:20.956 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:52:20.956 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:52:20.966 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-11T09:52:21.025 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:52:21.026+0000 7f3a7b90a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 42475) UID: 0 2024-09-11T09:52:21.108 INFO:tasks.workunit.client.0.smithi107.stderr:++ rbd journal inspect --pool rbd --image testrbdnbd27121 2024-09-11T09:52:21.108 INFO:tasks.workunit.client.0.smithi107.stderr:++ awk '/entries inspected/ {print $1}' 2024-09-11T09:52:21.118 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:52:21.118 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:52:21.126 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-09-11T09:52:21.127+0000 7f7b6c7d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31904) UID: 0 2024-09-11T09:52:21.128 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-11T09:52:21.170 INFO:tasks.workunit.client.0.smithi107.stderr:2024-09-11T09:52:21.170+0000 7f5d2e0c4640 -1 JournalMetadata: 0x5625f15b9810 failed to locate client: INSPECT 2024-09-11T09:52:21.174 INFO:tasks.workunit.client.0.smithi107.stderr:+ queue_length=4 2024-09-11T09:52:21.174 INFO:tasks.workunit.client.0.smithi107.stderr:+ tries=4 2024-09-11T09:52:21.175 INFO:tasks.workunit.client.0.smithi107.stderr:+ '[' 4 -lt 10 ']' 2024-09-11T09:52:21.175 INFO:tasks.workunit.client.0.smithi107.stderr:+ '[' 4 -gt 0 ']' 2024-09-11T09:52:21.175 INFO:tasks.workunit.client.0.smithi107.stderr:+ rbd journal inspect --pool rbd --image testrbdnbd27121 --verbose 2024-09-11T09:52:21.227 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:52:21.228+0000 7f3c96ec9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31909) UID: 0 2024-09-11T09:52:21.232 INFO:tasks.workunit.client.0.smithi107.stdout:Entry: tag_id=1, commit_tid=1 2024-09-11T09:52:21.232 INFO:tasks.workunit.client.0.smithi107.stdout:{ 2024-09-11T09:52:21.232 INFO:tasks.workunit.client.0.smithi107.stdout: "event_type": "AioDiscard", 2024-09-11T09:52:21.232 INFO:tasks.workunit.client.0.smithi107.stdout: "offset": 0, 2024-09-11T09:52:21.232 INFO:tasks.workunit.client.0.smithi107.stdout: "length": 4096, 2024-09-11T09:52:21.232 INFO:tasks.workunit.client.0.smithi107.stdout: "discard_granularity_bytes": 4096, 2024-09-11T09:52:21.232 INFO:tasks.workunit.client.0.smithi107.stdout: "timestamp": "2024-09-11T09:52:16.526805+0000" 2024-09-11T09:52:21.232 INFO:tasks.workunit.client.0.smithi107.stdout:} 2024-09-11T09:52:21.232 INFO:tasks.workunit.client.0.smithi107.stdout:Entry: tag_id=1, commit_tid=2 2024-09-11T09:52:21.233 INFO:tasks.workunit.client.0.smithi107.stdout:{ 2024-09-11T09:52:21.233 INFO:tasks.workunit.client.0.smithi107.stdout: "event_type": "AioDiscard", 2024-09-11T09:52:21.233 INFO:tasks.workunit.client.0.smithi107.stdout: "offset": 0, 2024-09-11T09:52:21.233 INFO:tasks.workunit.client.0.smithi107.stdout: "length": 4096, 2024-09-11T09:52:21.233 INFO:tasks.workunit.client.0.smithi107.stdout: "discard_granularity_bytes": 4096, 2024-09-11T09:52:21.233 INFO:tasks.workunit.client.0.smithi107.stdout: "timestamp": "2024-09-11T09:52:16.544819+0000" 2024-09-11T09:52:21.233 INFO:tasks.workunit.client.0.smithi107.stdout:} 2024-09-11T09:52:21.233 INFO:tasks.workunit.client.0.smithi107.stdout:Entry: tag_id=1, commit_tid=3 2024-09-11T09:52:21.233 INFO:tasks.workunit.client.0.smithi107.stdout:{ 2024-09-11T09:52:21.233 INFO:tasks.workunit.client.0.smithi107.stdout: "event_type": "AioDiscard", 2024-09-11T09:52:21.233 INFO:tasks.workunit.client.0.smithi107.stdout: "offset": 4096, 2024-09-11T09:52:21.233 INFO:tasks.workunit.client.0.smithi107.stdout: "length": 4096, 2024-09-11T09:52:21.233 INFO:tasks.workunit.client.0.smithi107.stdout: "discard_granularity_bytes": 4096, 2024-09-11T09:52:21.233 INFO:tasks.workunit.client.0.smithi107.stdout: "timestamp": "2024-09-11T09:52:16.576535+0000" 2024-09-11T09:52:21.234 INFO:tasks.workunit.client.0.smithi107.stdout:} 2024-09-11T09:52:21.234 INFO:tasks.workunit.client.0.smithi107.stdout:Entry: tag_id=1, commit_tid=4 2024-09-11T09:52:21.234 INFO:tasks.workunit.client.0.smithi107.stdout:{ 2024-09-11T09:52:21.234 INFO:tasks.workunit.client.0.smithi107.stdout: "event_type": "AioDiscard", 2024-09-11T09:52:21.234 INFO:tasks.workunit.client.0.smithi107.stdout: "offset": 4096, 2024-09-11T09:52:21.234 INFO:tasks.workunit.client.0.smithi107.stdout: "length": 4096, 2024-09-11T09:52:21.234 INFO:tasks.workunit.client.0.smithi107.stdout: "discard_granularity_bytes": 4096, 2024-09-11T09:52:21.234 INFO:tasks.workunit.client.0.smithi107.stdout: "timestamp": "2024-09-11T09:52:16.593859+0000" 2024-09-11T09:52:21.234 INFO:tasks.workunit.client.0.smithi107.stdout:} 2024-09-11T09:52:21.234 INFO:tasks.workunit.client.0.smithi107.stdout:Summary: 2024-09-11T09:52:21.234 INFO:tasks.workunit.client.0.smithi107.stdout: 4 entries inspected, 0 errors 2024-09-11T09:52:21.236 INFO:tasks.workunit.client.0.smithi107.stderr:2024-09-11T09:52:21.237+0000 7f89ff7fe640 -1 JournalMetadata: 0x5580ca4368b0 failed to locate client: INSPECT 2024-09-11T09:52:21.239 INFO:tasks.workunit.client.0.smithi107.stderr:+ sleep 1 2024-09-11T09:52:21.243 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:52:21.243 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:52:21.252 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-11T09:52:21.328 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-09-11T09:52:21.329+0000 7f7b6c7d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31904) UID: 0 2024-09-11T09:52:21.359 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:52:21.360 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:52:21.368 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-11T09:52:21.429 INFO:tasks.ceph.osd.2.smithi017.stderr:2024-09-11T09:52:21.430+0000 7fd20b170640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31672) UID: 0 2024-09-11T09:52:21.476 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:52:21.477 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:52:21.485 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-11T09:52:21.569 INFO:tasks.ceph.osd.2.smithi017.stderr:2024-09-11T09:52:21.570+0000 7fd20b170640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31672) UID: 0 2024-09-11T09:52:21.589 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T09:52:21.631 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-09-11T09:52:21.632+0000 7f7b6c7d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31904) UID: 0 2024-09-11T09:52:21.633 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:52:21.634 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:52:21.643 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-11T09:52:21.715 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-11T09:52:21.715 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T09:52:21.732 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-09-11T09:52:21.733+0000 7f7b6c7d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31904) UID: 0 2024-09-11T09:52:21.732 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-11T09:52:21.733 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T09:52:21.749 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-11T09:52:21.749 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T09:52:21.755 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:52:21.755 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 20, 2024-09-11T09:52:21.755 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 600, 2024-09-11T09:52:21.755 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [ 2024-09-11T09:52:21.755 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.755 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:31 2.1 2:8654721b:::rbd_data.1250f5e25b80.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:21.755 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.896024+0000", 2024-09-11T09:52:21.755 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 58.858821081000002, 2024-09-11T09:52:21.755 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.023148401999999998, 2024-09-11T09:52:21.755 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:21.755 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:21.756 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:21.756 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:21.756 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:52:21.756 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:52:21.756 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 31 2024-09-11T09:52:21.756 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.756 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:21.756 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.756 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:21.756 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.896024+0000", 2024-09-11T09:52:21.756 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:21.756 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.756 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.756 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:21.757 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.896024+0000", 2024-09-11T09:52:21.757 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:21.757 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.757 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.757 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:21.757 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.896025+0000", 2024-09-11T09:52:21.757 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.54e-07 2024-09-11T09:52:21.757 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.757 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.757 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:21.757 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.904287+0000", 2024-09-11T09:52:21.757 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0082624729999999993 2024-09-11T09:52:21.757 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.757 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.758 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:21.758 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.904305+0000", 2024-09-11T09:52:21.758 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7912999999999999e-05 2024-09-11T09:52:21.758 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.758 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.758 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:21.758 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.904312+0000", 2024-09-11T09:52:21.758 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.0230000000000004e-06 2024-09-11T09:52:21.758 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.758 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.758 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:21.758 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.904395+0000", 2024-09-11T09:52:21.758 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.2355999999999996e-05 2024-09-11T09:52:21.758 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.758 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.759 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:21.759 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.904499+0000", 2024-09-11T09:52:21.759 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000104672 2024-09-11T09:52:21.759 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.759 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.759 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1", 2024-09-11T09:52:21.759 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.904594+0000", 2024-09-11T09:52:21.759 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.5129000000000003e-05 2024-09-11T09:52:21.759 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.759 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.759 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:21.759 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.918183+0000", 2024-09-11T09:52:21.759 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.013588355999999999 2024-09-11T09:52:21.759 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.760 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.760 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:21.760 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919053+0000", 2024-09-11T09:52:21.760 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00087066800000000001 2024-09-11T09:52:21.760 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.760 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.760 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:21.760 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919125+0000", 2024-09-11T09:52:21.760 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.1333000000000003e-05 2024-09-11T09:52:21.760 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.760 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.760 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:21.760 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919172+0000", 2024-09-11T09:52:21.760 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.7725000000000001e-05 2024-09-11T09:52:21.760 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:21.761 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:21.761 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:21.761 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.761 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.761 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4688.0:42 2.3 e53/15 2:c54742d8:::rbd_data.1250f5e25b80.0000000000000006:head v 53'2, mlcod=0'0)", 2024-09-11T09:52:21.761 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.939272+0000", 2024-09-11T09:52:21.761 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 58.815573096999998, 2024-09-11T09:52:21.761 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.010821958, 2024-09-11T09:52:21.761 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:21.761 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:21.761 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:21.761 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:21.761 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.762 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:21.762 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.939272+0000", 2024-09-11T09:52:21.762 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:21.762 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.762 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.762 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:21.762 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.939272+0000", 2024-09-11T09:52:21.762 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:21.762 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.762 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.762 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:21.762 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.939273+0000", 2024-09-11T09:52:21.762 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.7899999999999997e-07 2024-09-11T09:52:21.762 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.763 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.763 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:21.763 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.944403+0000", 2024-09-11T09:52:21.763 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0051302129999999998 2024-09-11T09:52:21.763 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.763 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.763 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:21.763 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.944413+0000", 2024-09-11T09:52:21.763 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0182e-05 2024-09-11T09:52:21.763 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.763 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.763 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:21.763 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.944420+0000", 2024-09-11T09:52:21.763 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.7000000000000002e-06 2024-09-11T09:52:21.763 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.764 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.764 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:21.764 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.944482+0000", 2024-09-11T09:52:21.764 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.1911999999999999e-05 2024-09-11T09:52:21.764 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.764 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.764 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:21.764 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.944508+0000", 2024-09-11T09:52:21.764 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.6582000000000001e-05 2024-09-11T09:52:21.764 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.764 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.764 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:21.764 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.950050+0000", 2024-09-11T09:52:21.764 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0055414080000000003 2024-09-11T09:52:21.764 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.765 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.765 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:21.765 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.950094+0000", 2024-09-11T09:52:21.765 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.4082000000000002e-05 2024-09-11T09:52:21.765 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:21.765 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:21.765 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:21.765 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.765 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.765 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:47 2.1 2:95f53b3e:::rbd_data.1250f5e25b80.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:21.765 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.948330+0000", 2024-09-11T09:52:21.765 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 58.806514753000002, 2024-09-11T09:52:21.765 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.025156603999999999, 2024-09-11T09:52:21.765 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:21.765 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:21.766 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:21.766 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:21.766 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:52:21.766 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:52:21.766 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 47 2024-09-11T09:52:21.766 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.766 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:21.766 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.766 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:21.766 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.948330+0000", 2024-09-11T09:52:21.766 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:21.766 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.766 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.766 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:21.766 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.948330+0000", 2024-09-11T09:52:21.767 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:21.767 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.767 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.767 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:21.767 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.948333+0000", 2024-09-11T09:52:21.767 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.481e-06 2024-09-11T09:52:21.767 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.767 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.767 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:21.767 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.960263+0000", 2024-09-11T09:52:21.767 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.011930671 2024-09-11T09:52:21.767 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.767 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.768 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:21.768 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.960289+0000", 2024-09-11T09:52:21.768 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.5573000000000001e-05 2024-09-11T09:52:21.768 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.768 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.768 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:21.768 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.960304+0000", 2024-09-11T09:52:21.768 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.4647000000000001e-05 2024-09-11T09:52:21.768 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.768 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.768 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:21.768 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.960390+0000", 2024-09-11T09:52:21.768 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.6088999999999995e-05 2024-09-11T09:52:21.769 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.769 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.769 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:21.769 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.960562+0000", 2024-09-11T09:52:21.769 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00017237400000000001 2024-09-11T09:52:21.769 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.769 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.769 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1", 2024-09-11T09:52:21.769 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.960725+0000", 2024-09-11T09:52:21.769 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00016321599999999999 2024-09-11T09:52:21.769 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.769 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.769 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:21.770 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.969465+0000", 2024-09-11T09:52:21.770 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0087396899999999996 2024-09-11T09:52:21.770 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.770 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.770 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:21.770 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.973393+0000", 2024-09-11T09:52:21.770 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0039274669999999996 2024-09-11T09:52:21.770 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.770 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.770 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:21.770 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.973443+0000", 2024-09-11T09:52:21.770 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.0340000000000003e-05 2024-09-11T09:52:21.770 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.771 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.771 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:21.771 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.973487+0000", 2024-09-11T09:52:21.771 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.4056e-05 2024-09-11T09:52:21.771 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:21.771 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:21.771 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:21.771 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.771 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.771 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:61 2.1 2:9af038b1:::rbd_data.1250f5e25b80.000000000000000f:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:21.771 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.975764+0000", 2024-09-11T09:52:21.771 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 58.779081228999999, 2024-09-11T09:52:21.771 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.016158891000000002, 2024-09-11T09:52:21.771 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:21.772 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:21.772 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:21.772 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:21.772 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:52:21.772 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:52:21.772 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 61 2024-09-11T09:52:21.772 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.772 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:21.772 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.772 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:21.772 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.975764+0000", 2024-09-11T09:52:21.773 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:21.773 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.773 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.773 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:21.773 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.975764+0000", 2024-09-11T09:52:21.773 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:21.773 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.773 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.773 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:21.773 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.975766+0000", 2024-09-11T09:52:21.773 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.767e-06 2024-09-11T09:52:21.773 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.774 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.774 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:21.774 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.983204+0000", 2024-09-11T09:52:21.774 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0074388400000000004 2024-09-11T09:52:21.774 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.774 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.774 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:21.774 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.983226+0000", 2024-09-11T09:52:21.774 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.1279000000000001e-05 2024-09-11T09:52:21.774 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.774 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.774 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:21.774 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.983238+0000", 2024-09-11T09:52:21.774 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2563e-05 2024-09-11T09:52:21.774 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.775 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.775 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:21.775 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.983318+0000", 2024-09-11T09:52:21.775 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.0075999999999997e-05 2024-09-11T09:52:21.775 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.775 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.775 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:21.775 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.983429+0000", 2024-09-11T09:52:21.775 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000110618 2024-09-11T09:52:21.775 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.775 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.775 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1", 2024-09-11T09:52:21.775 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.983525+0000", 2024-09-11T09:52:21.775 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.603e-05 2024-09-11T09:52:21.775 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.776 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.776 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:21.776 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.989004+0000", 2024-09-11T09:52:21.776 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0054785429999999998 2024-09-11T09:52:21.776 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.776 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.776 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:21.776 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.991788+0000", 2024-09-11T09:52:21.776 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0027845729999999998 2024-09-11T09:52:21.776 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.776 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.776 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:21.776 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.991870+0000", 2024-09-11T09:52:21.777 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.1567e-05 2024-09-11T09:52:21.777 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.777 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.777 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:21.777 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.991923+0000", 2024-09-11T09:52:21.777 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.3035000000000003e-05 2024-09-11T09:52:21.777 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:21.777 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:21.777 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:21.777 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.777 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.777 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4694.0:32 2.1 2:8a862b48:ns::rbd_data.1256a0c02879.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:21.777 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.084089+0000", 2024-09-11T09:52:21.777 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 58.670755755000002, 2024-09-11T09:52:21.778 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.020630553999999999, 2024-09-11T09:52:21.778 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:21.778 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:21.778 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:21.778 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:21.778 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4694", 2024-09-11T09:52:21.778 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:52:21.778 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 32 2024-09-11T09:52:21.778 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.778 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:21.778 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.778 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:21.778 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.084089+0000", 2024-09-11T09:52:21.778 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:21.779 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.779 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.779 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:21.779 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.084089+0000", 2024-09-11T09:52:21.779 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:21.779 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.779 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.779 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:21.779 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.084090+0000", 2024-09-11T09:52:21.779 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.2900000000000003e-07 2024-09-11T09:52:21.779 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.779 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.779 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:21.780 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.090866+0000", 2024-09-11T09:52:21.780 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0067756609999999997 2024-09-11T09:52:21.780 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.780 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.780 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:21.780 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.090891+0000", 2024-09-11T09:52:21.780 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.5119000000000001e-05 2024-09-11T09:52:21.780 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.780 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.780 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:21.780 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.090900+0000", 2024-09-11T09:52:21.780 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.4110000000000002e-06 2024-09-11T09:52:21.781 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.781 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.781 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:21.781 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.090968+0000", 2024-09-11T09:52:21.781 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.8041999999999997e-05 2024-09-11T09:52:21.781 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.781 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.781 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:21.781 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.091115+0000", 2024-09-11T09:52:21.781 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00014633499999999999 2024-09-11T09:52:21.781 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.781 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.781 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1", 2024-09-11T09:52:21.781 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.091284+0000", 2024-09-11T09:52:21.782 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000169836 2024-09-11T09:52:21.782 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.782 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.782 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:21.782 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.100896+0000", 2024-09-11T09:52:21.782 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0096118639999999995 2024-09-11T09:52:21.782 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.782 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.782 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:21.782 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.104604+0000", 2024-09-11T09:52:21.782 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0037074619999999999 2024-09-11T09:52:21.782 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.782 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.782 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:21.782 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.104669+0000", 2024-09-11T09:52:21.783 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.4836000000000001e-05 2024-09-11T09:52:21.783 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.783 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.783 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:21.783 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.104720+0000", 2024-09-11T09:52:21.783 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.1258999999999998e-05 2024-09-11T09:52:21.783 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:21.783 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:21.783 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:21.783 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.783 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.783 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4694.0:44 2.3 e53/15 2:cd1a7d1c:ns::rbd_data.1256a0c02879.0000000000000006:head v 53'3, mlcod=16'1)", 2024-09-11T09:52:21.783 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.120387+0000", 2024-09-11T09:52:21.783 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 58.634458101999996, 2024-09-11T09:52:21.784 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.011988468, 2024-09-11T09:52:21.784 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:21.784 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:21.784 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:21.784 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:21.784 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.784 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:21.784 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.120387+0000", 2024-09-11T09:52:21.784 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:21.784 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.784 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.784 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:21.784 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.120387+0000", 2024-09-11T09:52:21.784 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:21.785 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.785 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.785 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:21.785 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.120389+0000", 2024-09-11T09:52:21.785 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.018e-06 2024-09-11T09:52:21.785 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.785 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.785 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:21.785 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.127403+0000", 2024-09-11T09:52:21.785 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0070144769999999999 2024-09-11T09:52:21.785 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.785 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.785 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:21.785 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.127414+0000", 2024-09-11T09:52:21.786 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0991e-05 2024-09-11T09:52:21.786 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.786 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.786 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:21.786 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.127423+0000", 2024-09-11T09:52:21.786 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.8580000000000006e-06 2024-09-11T09:52:21.786 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.786 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.786 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:21.786 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.127468+0000", 2024-09-11T09:52:21.786 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.4830999999999998e-05 2024-09-11T09:52:21.786 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.786 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.787 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:21.787 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.127485+0000", 2024-09-11T09:52:21.787 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6838999999999999e-05 2024-09-11T09:52:21.787 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.787 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.787 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:21.787 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.132341+0000", 2024-09-11T09:52:21.787 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0048557959999999999 2024-09-11T09:52:21.787 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.787 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.787 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:21.787 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.132375+0000", 2024-09-11T09:52:21.787 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.4657999999999999e-05 2024-09-11T09:52:21.787 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:21.787 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:21.788 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:21.788 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.788 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.788 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4694.0:46 2.3 e53/15 2:ce0e1469:ns::rbd_data.1256a0c02879.0000000000000008:head v 53'4, mlcod=16'1)", 2024-09-11T09:52:21.788 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.129008+0000", 2024-09-11T09:52:21.788 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 58.625837515999997, 2024-09-11T09:52:21.788 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0087151220000000005, 2024-09-11T09:52:21.788 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:21.788 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:21.788 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:21.788 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:21.788 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.788 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:21.789 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.129008+0000", 2024-09-11T09:52:21.789 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:21.789 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.789 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.789 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:21.789 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.129008+0000", 2024-09-11T09:52:21.789 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:21.789 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.789 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.789 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:21.789 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.129009+0000", 2024-09-11T09:52:21.789 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.367e-06 2024-09-11T09:52:21.789 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.789 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.790 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:21.790 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.132839+0000", 2024-09-11T09:52:21.790 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0038297610000000001 2024-09-11T09:52:21.790 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.790 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.790 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:21.790 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.132849+0000", 2024-09-11T09:52:21.790 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0672e-05 2024-09-11T09:52:21.790 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.790 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.790 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:21.790 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.132856+0000", 2024-09-11T09:52:21.790 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.1380000000000004e-06 2024-09-11T09:52:21.790 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.790 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.791 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:21.791 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.132891+0000", 2024-09-11T09:52:21.791 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.5712e-05 2024-09-11T09:52:21.791 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.791 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.791 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:21.791 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.132907+0000", 2024-09-11T09:52:21.791 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6166000000000001e-05 2024-09-11T09:52:21.791 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.791 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.791 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:21.791 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.137681+0000", 2024-09-11T09:52:21.791 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0047735299999999998 2024-09-11T09:52:21.791 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.792 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.792 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:21.792 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.137723+0000", 2024-09-11T09:52:21.792 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.1776000000000001e-05 2024-09-11T09:52:21.792 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:21.792 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:21.792 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:21.792 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.792 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.792 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4694.0:48 2.3 e53/15 2:d2090b70:ns::rbd_data.1256a0c02879.0000000000000009:head v 53'5, mlcod=53'2)", 2024-09-11T09:52:21.792 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.135108+0000", 2024-09-11T09:52:21.792 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 58.619736619000001, 2024-09-11T09:52:21.792 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.010710679000000001, 2024-09-11T09:52:21.792 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:21.793 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:21.793 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:21.793 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:21.793 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.793 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:21.793 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.135108+0000", 2024-09-11T09:52:21.793 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:21.793 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.793 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.793 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:21.793 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.135108+0000", 2024-09-11T09:52:21.793 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:21.793 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.794 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.794 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:21.794 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.135110+0000", 2024-09-11T09:52:21.794 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.122e-06 2024-09-11T09:52:21.794 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.794 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.794 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:21.794 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.140876+0000", 2024-09-11T09:52:21.794 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.005766162 2024-09-11T09:52:21.794 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.794 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.794 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:21.794 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.140888+0000", 2024-09-11T09:52:21.795 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.201e-05 2024-09-11T09:52:21.795 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.795 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.795 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:21.795 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.140894+0000", 2024-09-11T09:52:21.795 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.1739999999999997e-06 2024-09-11T09:52:21.795 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.795 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.795 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:21.795 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.140971+0000", 2024-09-11T09:52:21.795 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.7495000000000003e-05 2024-09-11T09:52:21.795 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.795 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.795 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:21.796 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.140995+0000", 2024-09-11T09:52:21.796 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.3204999999999999e-05 2024-09-11T09:52:21.796 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.796 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.796 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:21.796 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.145757+0000", 2024-09-11T09:52:21.796 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0047627190000000003 2024-09-11T09:52:21.796 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.796 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.796 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:21.796 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.145819+0000", 2024-09-11T09:52:21.796 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.1791999999999994e-05 2024-09-11T09:52:21.796 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:21.797 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:21.797 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:21.797 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.797 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.797 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4694.0:50 2.1 2:83078786:ns::rbd_data.1256a0c02879.000000000000000a:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:21.797 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.141467+0000", 2024-09-11T09:52:21.797 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 58.613377894999999, 2024-09-11T09:52:21.797 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.012785889, 2024-09-11T09:52:21.797 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:21.797 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:21.797 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:21.797 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:21.797 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4694", 2024-09-11T09:52:21.798 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:52:21.798 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 50 2024-09-11T09:52:21.798 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.798 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:21.798 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.798 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:21.798 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.141467+0000", 2024-09-11T09:52:21.798 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:21.798 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.798 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.798 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:21.798 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.141467+0000", 2024-09-11T09:52:21.798 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:21.798 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.799 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.799 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:21.799 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.141468+0000", 2024-09-11T09:52:21.799 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1009999999999999e-06 2024-09-11T09:52:21.799 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.799 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.799 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:21.799 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.146432+0000", 2024-09-11T09:52:21.799 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0049640539999999999 2024-09-11T09:52:21.799 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.799 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.799 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:21.799 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.146446+0000", 2024-09-11T09:52:21.799 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3254e-05 2024-09-11T09:52:21.799 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.800 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.800 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:21.800 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.146452+0000", 2024-09-11T09:52:21.800 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.207e-06 2024-09-11T09:52:21.800 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.800 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.800 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:21.800 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.146510+0000", 2024-09-11T09:52:21.800 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.8322999999999997e-05 2024-09-11T09:52:21.800 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.800 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.800 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:21.800 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.146650+0000", 2024-09-11T09:52:21.800 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00013947100000000001 2024-09-11T09:52:21.800 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.801 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.801 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1", 2024-09-11T09:52:21.801 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.146779+0000", 2024-09-11T09:52:21.801 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000129059 2024-09-11T09:52:21.801 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.801 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.801 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:21.801 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.153778+0000", 2024-09-11T09:52:21.801 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0069997130000000003 2024-09-11T09:52:21.801 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.801 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.801 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:21.801 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.154180+0000", 2024-09-11T09:52:21.801 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00040159800000000003 2024-09-11T09:52:21.802 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.802 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.802 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:21.802 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.154218+0000", 2024-09-11T09:52:21.802 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.7795e-05 2024-09-11T09:52:21.802 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.802 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.802 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:21.802 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.154253+0000", 2024-09-11T09:52:21.802 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.5314000000000001e-05 2024-09-11T09:52:21.802 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:21.802 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:21.802 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:21.802 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.802 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.803 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4694.0:54 2.3 e53/15 2:c44f0253:ns::rbd_data.1256a0c02879.000000000000000b:head v 53'6, mlcod=53'4)", 2024-09-11T09:52:21.803 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.154926+0000", 2024-09-11T09:52:21.803 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 58.59991952, 2024-09-11T09:52:21.803 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0089497099999999996, 2024-09-11T09:52:21.803 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:21.803 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:21.803 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:21.803 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:21.803 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.803 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:21.803 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.154926+0000", 2024-09-11T09:52:21.803 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:21.803 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.803 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.803 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:21.804 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.154926+0000", 2024-09-11T09:52:21.804 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:21.804 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.804 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.804 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:21.804 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.154927+0000", 2024-09-11T09:52:21.804 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.7000000000000003e-07 2024-09-11T09:52:21.804 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.804 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.804 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:21.804 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.158712+0000", 2024-09-11T09:52:21.804 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0037855469999999998 2024-09-11T09:52:21.804 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.804 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.805 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:21.805 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.158726+0000", 2024-09-11T09:52:21.805 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3621000000000001e-05 2024-09-11T09:52:21.805 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.805 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.805 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:21.805 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.158732+0000", 2024-09-11T09:52:21.805 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.9089999999999997e-06 2024-09-11T09:52:21.805 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.805 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.805 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:21.805 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.158781+0000", 2024-09-11T09:52:21.805 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.8900000000000003e-05 2024-09-11T09:52:21.805 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.806 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.806 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:21.806 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.158805+0000", 2024-09-11T09:52:21.806 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.44e-05 2024-09-11T09:52:21.806 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.806 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.806 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:21.806 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.163826+0000", 2024-09-11T09:52:21.806 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0050206399999999998 2024-09-11T09:52:21.806 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.806 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.806 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:21.806 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.163875+0000", 2024-09-11T09:52:21.806 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.9722999999999998e-05 2024-09-11T09:52:21.807 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:21.807 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:21.807 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:21.807 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.807 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.807 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4694.0:56 2.3 e53/15 2:c462f029:ns::rbd_data.1256a0c02879.000000000000000c:head v 53'7, mlcod=53'4)", 2024-09-11T09:52:21.807 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.159549+0000", 2024-09-11T09:52:21.807 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 58.595295813999996, 2024-09-11T09:52:21.807 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.015591666000000001, 2024-09-11T09:52:21.807 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:21.807 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:21.807 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:21.807 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:21.807 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.808 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:21.808 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.159549+0000", 2024-09-11T09:52:21.808 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:21.808 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.808 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.808 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:21.808 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.159549+0000", 2024-09-11T09:52:21.808 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:21.808 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.808 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.808 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:21.808 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.159550+0000", 2024-09-11T09:52:21.808 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.1500000000000004e-07 2024-09-11T09:52:21.808 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.809 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.809 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:21.809 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.167834+0000", 2024-09-11T09:52:21.809 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0082840220000000003 2024-09-11T09:52:21.809 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.809 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.809 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:21.809 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.167857+0000", 2024-09-11T09:52:21.809 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.3337e-05 2024-09-11T09:52:21.809 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.809 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.809 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:21.809 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.167869+0000", 2024-09-11T09:52:21.810 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1556000000000001e-05 2024-09-11T09:52:21.810 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.810 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.810 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:21.810 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.167939+0000", 2024-09-11T09:52:21.810 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.0674999999999997e-05 2024-09-11T09:52:21.810 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.810 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.810 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:21.810 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.167967+0000", 2024-09-11T09:52:21.810 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.7322999999999999e-05 2024-09-11T09:52:21.810 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.810 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.811 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:21.811 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.175079+0000", 2024-09-11T09:52:21.811 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0071126150000000001 2024-09-11T09:52:21.811 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.811 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.811 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:21.811 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.175141+0000", 2024-09-11T09:52:21.811 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.1523e-05 2024-09-11T09:52:21.811 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:21.811 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:21.811 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:21.811 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.811 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.811 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.5314.0:31 2.1 2:8654721b:::rbd_data.1250f5e25b80.0000000000000000:head [write 393216~131072] snapc 3=[3] ondisk+write+known_if_redirected+supports_pool_eio e68)", 2024-09-11T09:52:21.812 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:50.230850+0000", 2024-09-11T09:52:21.812 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 31.523995053, 2024-09-11T09:52:21.812 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0073772539999999998, 2024-09-11T09:52:21.812 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:21.812 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:21.812 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:21.812 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:21.812 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.5314", 2024-09-11T09:52:21.812 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3853995732", 2024-09-11T09:52:21.812 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 31 2024-09-11T09:52:21.812 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.812 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:21.813 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.813 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:21.813 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.230850+0000", 2024-09-11T09:52:21.813 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:21.813 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.813 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.813 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:21.813 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.230850+0000", 2024-09-11T09:52:21.813 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:21.813 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.813 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.813 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:21.813 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.230850+0000", 2024-09-11T09:52:21.813 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.32e-07 2024-09-11T09:52:21.814 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.814 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.814 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:21.814 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.230908+0000", 2024-09-11T09:52:21.814 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.7698000000000002e-05 2024-09-11T09:52:21.814 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.814 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.814 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:21.814 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.230915+0000", 2024-09-11T09:52:21.814 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.81e-06 2024-09-11T09:52:21.814 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.814 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.814 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:21.814 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.230920+0000", 2024-09-11T09:52:21.815 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.9660000000000004e-06 2024-09-11T09:52:21.815 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.815 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.815 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:21.815 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.235055+0000", 2024-09-11T09:52:21.815 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0041354629999999998 2024-09-11T09:52:21.815 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.815 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.815 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:21.815 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.235145+0000", 2024-09-11T09:52:21.815 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.9362000000000003e-05 2024-09-11T09:52:21.815 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.815 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.816 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1", 2024-09-11T09:52:21.816 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.235282+0000", 2024-09-11T09:52:21.816 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00013710999999999999 2024-09-11T09:52:21.816 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.816 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.816 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:21.816 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.237335+0000", 2024-09-11T09:52:21.816 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.002053165 2024-09-11T09:52:21.816 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.816 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.816 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:21.816 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.238106+0000", 2024-09-11T09:52:21.816 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000771015 2024-09-11T09:52:21.817 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.817 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.817 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:21.817 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.238165+0000", 2024-09-11T09:52:21.817 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.8588000000000002e-05 2024-09-11T09:52:21.817 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.817 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.817 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:21.817 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.238227+0000", 2024-09-11T09:52:21.817 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.2644999999999994e-05 2024-09-11T09:52:21.817 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:21.817 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:21.817 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:21.817 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.818 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.818 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.5314.0:32 2.1 2:8654721b:::rbd_data.1250f5e25b80.0000000000000000:head [write 524288~131072] snapc 3=[3] ondisk+write+known_if_redirected+supports_pool_eio e68)", 2024-09-11T09:52:21.818 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:50.230932+0000", 2024-09-11T09:52:21.818 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 31.523913428, 2024-09-11T09:52:21.818 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0074882129999999996, 2024-09-11T09:52:21.818 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:21.818 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:21.818 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:21.818 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:21.818 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.5314", 2024-09-11T09:52:21.818 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3853995732", 2024-09-11T09:52:21.818 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 32 2024-09-11T09:52:21.818 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.818 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:21.819 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.819 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:21.819 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.230932+0000", 2024-09-11T09:52:21.819 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:21.819 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.819 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.819 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:21.819 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.230932+0000", 2024-09-11T09:52:21.819 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:21.819 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.819 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.819 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:21.819 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.230932+0000", 2024-09-11T09:52:21.819 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.9999999999999999e-07 2024-09-11T09:52:21.820 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.820 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.820 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:21.820 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.231022+0000", 2024-09-11T09:52:21.820 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.0439999999999995e-05 2024-09-11T09:52:21.820 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.820 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.820 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:21.820 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.231030+0000", 2024-09-11T09:52:21.820 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.3350000000000001e-06 2024-09-11T09:52:21.820 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.820 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.820 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:21.820 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.231034+0000", 2024-09-11T09:52:21.820 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.9759999999999997e-06 2024-09-11T09:52:21.821 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.821 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.821 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:21.821 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.235837+0000", 2024-09-11T09:52:21.821 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0048031910000000001 2024-09-11T09:52:21.821 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.821 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.821 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:21.821 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.235904+0000", 2024-09-11T09:52:21.821 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.7181999999999995e-05 2024-09-11T09:52:21.821 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.821 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.822 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1", 2024-09-11T09:52:21.822 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.236040+0000", 2024-09-11T09:52:21.822 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00013545199999999999 2024-09-11T09:52:21.822 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.822 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.822 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:21.822 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.238242+0000", 2024-09-11T09:52:21.822 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.002202809 2024-09-11T09:52:21.822 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.822 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.822 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:21.822 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.238301+0000", 2024-09-11T09:52:21.822 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.8573999999999999e-05 2024-09-11T09:52:21.822 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.823 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.823 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:21.823 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.238378+0000", 2024-09-11T09:52:21.823 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.7323999999999994e-05 2024-09-11T09:52:21.823 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.823 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.823 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:21.823 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.238420+0000", 2024-09-11T09:52:21.823 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.163e-05 2024-09-11T09:52:21.823 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:21.823 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:21.823 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:21.823 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.823 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.824 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.5314.0:33 2.1 2:8654721b:::rbd_data.1250f5e25b80.0000000000000000:head [write 655360~131072] snapc 3=[3] ondisk+write+known_if_redirected+supports_pool_eio e68)", 2024-09-11T09:52:21.824 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:50.231041+0000", 2024-09-11T09:52:21.824 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 31.523804268999999, 2024-09-11T09:52:21.824 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0075627139999999999, 2024-09-11T09:52:21.824 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:21.824 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:21.824 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:21.824 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:21.824 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.5314", 2024-09-11T09:52:21.824 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3853995732", 2024-09-11T09:52:21.824 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 33 2024-09-11T09:52:21.824 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.825 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:21.825 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.825 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:21.825 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.231041+0000", 2024-09-11T09:52:21.825 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:21.825 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.825 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.825 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:21.825 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.231041+0000", 2024-09-11T09:52:21.825 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:21.825 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.825 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.825 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:21.826 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.231041+0000", 2024-09-11T09:52:21.826 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.9499999999999998e-07 2024-09-11T09:52:21.826 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.826 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.826 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:21.826 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.231111+0000", 2024-09-11T09:52:21.826 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.0166999999999994e-05 2024-09-11T09:52:21.826 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.826 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.826 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:21.826 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.231122+0000", 2024-09-11T09:52:21.826 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0518e-05 2024-09-11T09:52:21.827 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.827 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.827 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:21.827 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.231128+0000", 2024-09-11T09:52:21.827 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.3509999999999998e-06 2024-09-11T09:52:21.827 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.827 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.827 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:21.827 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.236622+0000", 2024-09-11T09:52:21.827 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0054934989999999998 2024-09-11T09:52:21.827 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.827 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.827 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:21.828 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.236719+0000", 2024-09-11T09:52:21.828 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.7497999999999994e-05 2024-09-11T09:52:21.828 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.828 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.828 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1", 2024-09-11T09:52:21.828 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.236854+0000", 2024-09-11T09:52:21.828 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000134535 2024-09-11T09:52:21.828 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.828 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.828 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:21.828 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.238491+0000", 2024-09-11T09:52:21.828 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0016368610000000001 2024-09-11T09:52:21.829 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.829 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.829 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:21.829 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.238504+0000", 2024-09-11T09:52:21.829 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3927999999999999e-05 2024-09-11T09:52:21.829 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.829 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.829 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:21.829 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.238561+0000", 2024-09-11T09:52:21.829 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.6029999999999997e-05 2024-09-11T09:52:21.829 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.829 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.829 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:21.830 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.238604+0000", 2024-09-11T09:52:21.830 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.3031999999999997e-05 2024-09-11T09:52:21.830 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:21.830 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:21.830 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:21.830 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.830 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.830 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.5314.0:34 2.1 2:8654721b:::rbd_data.1250f5e25b80.0000000000000000:head [write 786432~131072] snapc 3=[3] ondisk+write+known_if_redirected+supports_pool_eio e68)", 2024-09-11T09:52:21.830 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:50.231139+0000", 2024-09-11T09:52:21.830 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 31.523705666000001, 2024-09-11T09:52:21.830 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0083689690000000004, 2024-09-11T09:52:21.830 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:21.830 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:21.830 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:21.831 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:21.831 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.5314", 2024-09-11T09:52:21.831 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3853995732", 2024-09-11T09:52:21.831 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 34 2024-09-11T09:52:21.831 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.831 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:21.831 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.831 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:21.831 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.231139+0000", 2024-09-11T09:52:21.831 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:21.831 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.831 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.831 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:21.831 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.231139+0000", 2024-09-11T09:52:21.831 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:21.831 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.832 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.832 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:21.832 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.231140+0000", 2024-09-11T09:52:21.832 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.1699999999999998e-07 2024-09-11T09:52:21.832 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.832 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.832 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:21.832 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.231247+0000", 2024-09-11T09:52:21.832 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000107088 2024-09-11T09:52:21.832 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.832 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.832 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:21.832 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.231260+0000", 2024-09-11T09:52:21.833 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.289e-05 2024-09-11T09:52:21.833 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.833 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.833 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:21.833 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.231272+0000", 2024-09-11T09:52:21.833 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2371999999999999e-05 2024-09-11T09:52:21.833 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.833 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.833 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:21.833 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.237380+0000", 2024-09-11T09:52:21.833 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.006107683 2024-09-11T09:52:21.833 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.833 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.834 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:21.834 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.237434+0000", 2024-09-11T09:52:21.834 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.4143e-05 2024-09-11T09:52:21.834 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.834 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.834 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1", 2024-09-11T09:52:21.834 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.237541+0000", 2024-09-11T09:52:21.834 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00010676800000000001 2024-09-11T09:52:21.834 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.834 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.834 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:21.834 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.239390+0000", 2024-09-11T09:52:21.834 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.001848861 2024-09-11T09:52:21.834 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.834 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.835 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:21.835 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.239409+0000", 2024-09-11T09:52:21.835 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.9505e-05 2024-09-11T09:52:21.835 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.835 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.835 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:21.835 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.239463+0000", 2024-09-11T09:52:21.835 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.3838000000000002e-05 2024-09-11T09:52:21.835 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.835 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.835 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:21.835 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.239508+0000", 2024-09-11T09:52:21.835 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.5303999999999999e-05 2024-09-11T09:52:21.835 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:21.835 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:21.836 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:21.836 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.836 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.836 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.5314.0:35 2.1 2:8654721b:::rbd_data.1250f5e25b80.0000000000000000:head [write 917504~131072] snapc 3=[3] ondisk+write+known_if_redirected+supports_pool_eio e68)", 2024-09-11T09:52:21.836 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:50.231287+0000", 2024-09-11T09:52:21.836 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 31.523558332, 2024-09-11T09:52:21.836 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0086930730000000008, 2024-09-11T09:52:21.836 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:21.836 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:21.836 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:21.836 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:21.836 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.5314", 2024-09-11T09:52:21.836 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3853995732", 2024-09-11T09:52:21.836 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 35 2024-09-11T09:52:21.836 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.837 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:21.837 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.837 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:21.837 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.231287+0000", 2024-09-11T09:52:21.837 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:21.837 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.837 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.837 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:21.837 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.231287+0000", 2024-09-11T09:52:21.837 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:21.837 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.837 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.837 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:21.837 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.231287+0000", 2024-09-11T09:52:21.838 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.7700000000000004e-07 2024-09-11T09:52:21.838 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.838 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.838 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:21.838 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.231354+0000", 2024-09-11T09:52:21.838 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.6304999999999996e-05 2024-09-11T09:52:21.838 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.838 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.838 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:21.838 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.231367+0000", 2024-09-11T09:52:21.838 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2952999999999999e-05 2024-09-11T09:52:21.838 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.838 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.839 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:21.839 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.231374+0000", 2024-09-11T09:52:21.839 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.0469999999999996e-06 2024-09-11T09:52:21.839 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.839 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.839 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:21.839 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.238650+0000", 2024-09-11T09:52:21.839 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0072760280000000004 2024-09-11T09:52:21.839 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.839 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.839 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:21.839 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.238732+0000", 2024-09-11T09:52:21.839 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.2194000000000006e-05 2024-09-11T09:52:21.839 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.840 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.840 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1", 2024-09-11T09:52:21.840 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.238876+0000", 2024-09-11T09:52:21.840 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00014458 2024-09-11T09:52:21.840 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.840 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.840 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:21.840 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.239739+0000", 2024-09-11T09:52:21.840 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00086244499999999999 2024-09-11T09:52:21.840 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.840 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.840 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:21.840 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.239917+0000", 2024-09-11T09:52:21.840 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00017777299999999999 2024-09-11T09:52:21.841 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.841 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.841 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:21.841 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.239954+0000", 2024-09-11T09:52:21.841 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.7379999999999998e-05 2024-09-11T09:52:21.841 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.841 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.841 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:21.841 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.239980+0000", 2024-09-11T09:52:21.841 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.5791000000000001e-05 2024-09-11T09:52:21.841 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:21.841 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:21.841 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:21.842 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.842 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.842 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.5314.0:70 2.1 2:8654721b:::rbd_data.1250f5e25b80.0000000000000000:head [write 524288~131072] snapc 5=[5,3] ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-09-11T09:52:21.842 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:50.824556+0000", 2024-09-11T09:52:21.842 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 30.930289531, 2024-09-11T09:52:21.842 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0072794469999999997, 2024-09-11T09:52:21.842 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:21.842 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:21.842 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:21.842 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:21.842 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.5314", 2024-09-11T09:52:21.842 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3853995732", 2024-09-11T09:52:21.842 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 70 2024-09-11T09:52:21.842 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.843 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:21.843 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.843 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:21.843 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.824556+0000", 2024-09-11T09:52:21.843 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:21.843 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.843 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.843 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:21.843 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.824556+0000", 2024-09-11T09:52:21.843 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:21.843 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.843 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.843 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:21.844 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.824564+0000", 2024-09-11T09:52:21.844 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.2409999999999995e-06 2024-09-11T09:52:21.844 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.844 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.844 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:21.844 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.824613+0000", 2024-09-11T09:52:21.844 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.9579000000000001e-05 2024-09-11T09:52:21.844 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.844 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.844 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:21.844 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.824638+0000", 2024-09-11T09:52:21.844 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.4510000000000001e-05 2024-09-11T09:52:21.844 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.844 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.845 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:21.845 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.824653+0000", 2024-09-11T09:52:21.845 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.4831000000000001e-05 2024-09-11T09:52:21.845 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.845 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.845 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:21.845 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.829072+0000", 2024-09-11T09:52:21.845 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0044195700000000003 2024-09-11T09:52:21.845 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.845 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.845 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:21.845 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.829166+0000", 2024-09-11T09:52:21.845 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.3212000000000001e-05 2024-09-11T09:52:21.845 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.846 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.846 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1", 2024-09-11T09:52:21.846 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.829309+0000", 2024-09-11T09:52:21.846 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00014301900000000001 2024-09-11T09:52:21.846 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.846 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.846 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:21.846 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.831074+0000", 2024-09-11T09:52:21.846 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.001765305 2024-09-11T09:52:21.846 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.846 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.846 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:21.846 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.831751+0000", 2024-09-11T09:52:21.846 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00067694000000000001 2024-09-11T09:52:21.847 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.847 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.847 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:21.847 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.831803+0000", 2024-09-11T09:52:21.847 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.2441000000000002e-05 2024-09-11T09:52:21.847 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.847 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.847 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:21.847 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.831835+0000", 2024-09-11T09:52:21.847 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.1798999999999998e-05 2024-09-11T09:52:21.847 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:21.847 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:21.847 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:21.847 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.848 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.848 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.5314.0:71 2.1 2:8654721b:::rbd_data.1250f5e25b80.0000000000000000:head [write 655360~131072] snapc 5=[5,3] ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-09-11T09:52:21.848 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:50.824662+0000", 2024-09-11T09:52:21.848 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 30.930183414999998, 2024-09-11T09:52:21.848 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0082172219999999997, 2024-09-11T09:52:21.848 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:21.848 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:21.848 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:21.848 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:21.848 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.5314", 2024-09-11T09:52:21.848 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3853995732", 2024-09-11T09:52:21.848 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 71 2024-09-11T09:52:21.848 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.848 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:21.848 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.849 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:21.849 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.824662+0000", 2024-09-11T09:52:21.849 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:21.849 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.849 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.849 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:21.849 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.824662+0000", 2024-09-11T09:52:21.849 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:21.849 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.849 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.849 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:21.849 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.824662+0000", 2024-09-11T09:52:21.849 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.0699999999999998e-07 2024-09-11T09:52:21.850 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.850 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.850 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:21.850 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.824859+0000", 2024-09-11T09:52:21.850 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00019725799999999999 2024-09-11T09:52:21.850 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.850 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.850 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:21.850 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.824906+0000", 2024-09-11T09:52:21.850 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.7162e-05 2024-09-11T09:52:21.850 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.850 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.850 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:21.850 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.824926+0000", 2024-09-11T09:52:21.850 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.0038999999999999e-05 2024-09-11T09:52:21.850 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.851 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.851 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:21.851 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.830313+0000", 2024-09-11T09:52:21.851 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0053868409999999999 2024-09-11T09:52:21.851 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.851 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.851 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:21.851 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.830359+0000", 2024-09-11T09:52:21.851 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.5370000000000001e-05 2024-09-11T09:52:21.851 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.851 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.851 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1", 2024-09-11T09:52:21.851 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.830460+0000", 2024-09-11T09:52:21.851 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00010171299999999999 2024-09-11T09:52:21.851 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.852 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.852 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:21.852 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.831843+0000", 2024-09-11T09:52:21.852 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.001382501 2024-09-11T09:52:21.852 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.852 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.852 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:21.852 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.832769+0000", 2024-09-11T09:52:21.852 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00092638099999999999 2024-09-11T09:52:21.852 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.852 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.852 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:21.852 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.832835+0000", 2024-09-11T09:52:21.852 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.6106999999999996e-05 2024-09-11T09:52:21.853 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.853 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.853 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:21.853 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.832879+0000", 2024-09-11T09:52:21.853 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.3543e-05 2024-09-11T09:52:21.853 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:21.853 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:21.853 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:21.853 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.853 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.853 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.5314.0:72 2.1 2:8654721b:::rbd_data.1250f5e25b80.0000000000000000:head [write 786432~131072] snapc 5=[5,3] ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-09-11T09:52:21.853 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:50.824938+0000", 2024-09-11T09:52:21.853 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 30.929907112999999, 2024-09-11T09:52:21.853 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0082389799999999999, 2024-09-11T09:52:21.854 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:21.854 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:21.854 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:21.854 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:21.854 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.5314", 2024-09-11T09:52:21.854 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3853995732", 2024-09-11T09:52:21.854 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 72 2024-09-11T09:52:21.854 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.854 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:21.854 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.854 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:21.854 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.824938+0000", 2024-09-11T09:52:21.854 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:21.855 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.855 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.855 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:21.855 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.824938+0000", 2024-09-11T09:52:21.855 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:21.855 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.855 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.855 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:21.855 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.824939+0000", 2024-09-11T09:52:21.855 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.6199999999999997e-07 2024-09-11T09:52:21.855 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.855 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.855 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:21.856 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.825028+0000", 2024-09-11T09:52:21.856 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.9105999999999997e-05 2024-09-11T09:52:21.856 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.856 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.856 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:21.856 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.825042+0000", 2024-09-11T09:52:21.856 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3760999999999999e-05 2024-09-11T09:52:21.856 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.856 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.856 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:21.856 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.825061+0000", 2024-09-11T09:52:21.856 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.9088000000000001e-05 2024-09-11T09:52:21.856 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.856 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.856 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:21.857 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.831122+0000", 2024-09-11T09:52:21.857 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.006061299 2024-09-11T09:52:21.857 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.857 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.857 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:21.857 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.831195+0000", 2024-09-11T09:52:21.857 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.2957e-05 2024-09-11T09:52:21.857 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.857 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.857 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1", 2024-09-11T09:52:21.857 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.831301+0000", 2024-09-11T09:52:21.857 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000106049 2024-09-11T09:52:21.857 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.857 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.858 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:21.858 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.832893+0000", 2024-09-11T09:52:21.858 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0015923770000000001 2024-09-11T09:52:21.858 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.858 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.858 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:21.858 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.833076+0000", 2024-09-11T09:52:21.858 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000182855 2024-09-11T09:52:21.858 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.858 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.858 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:21.858 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.833137+0000", 2024-09-11T09:52:21.858 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.1193999999999997e-05 2024-09-11T09:52:21.858 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.858 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.859 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:21.859 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.833177+0000", 2024-09-11T09:52:21.859 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.9632000000000003e-05 2024-09-11T09:52:21.859 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:21.859 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:21.859 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:21.859 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.859 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.859 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.5314.0:73 2.1 2:8654721b:::rbd_data.1250f5e25b80.0000000000000000:head [write 917504~131072] snapc 5=[5,3] ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-09-11T09:52:21.859 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:50.825071+0000", 2024-09-11T09:52:21.859 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 30.929773606000001, 2024-09-11T09:52:21.859 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0084039160000000009, 2024-09-11T09:52:21.859 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:21.859 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:21.859 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:21.860 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:21.860 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.5314", 2024-09-11T09:52:21.860 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3853995732", 2024-09-11T09:52:21.860 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 73 2024-09-11T09:52:21.860 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.860 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:21.860 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.860 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:21.860 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.825071+0000", 2024-09-11T09:52:21.860 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:21.860 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.860 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.860 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:21.860 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.825071+0000", 2024-09-11T09:52:21.861 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:21.861 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.861 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.861 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:21.861 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.825072+0000", 2024-09-11T09:52:21.861 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.5499999999999998e-07 2024-09-11T09:52:21.861 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.861 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.861 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:21.861 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.825124+0000", 2024-09-11T09:52:21.861 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.2197000000000002e-05 2024-09-11T09:52:21.861 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.861 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.861 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:21.861 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.825132+0000", 2024-09-11T09:52:21.862 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.0290000000000005e-06 2024-09-11T09:52:21.862 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.862 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.862 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:21.862 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.825136+0000", 2024-09-11T09:52:21.862 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.9380000000000002e-06 2024-09-11T09:52:21.862 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.862 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.862 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:21.862 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.832001+0000", 2024-09-11T09:52:21.862 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0068650949999999999 2024-09-11T09:52:21.862 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.862 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.862 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:21.862 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.832091+0000", 2024-09-11T09:52:21.862 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.9524999999999994e-05 2024-09-11T09:52:21.863 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.863 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.863 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1", 2024-09-11T09:52:21.863 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.832234+0000", 2024-09-11T09:52:21.863 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000142783 2024-09-11T09:52:21.863 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.863 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.863 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:21.863 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.833198+0000", 2024-09-11T09:52:21.863 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00096476899999999998 2024-09-11T09:52:21.863 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.863 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.863 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:21.863 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.833386+0000", 2024-09-11T09:52:21.863 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00018772 2024-09-11T09:52:21.864 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.864 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.864 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:21.864 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.833440+0000", 2024-09-11T09:52:21.864 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.4228999999999999e-05 2024-09-11T09:52:21.864 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:21.864 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:21.864 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:21.864 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.833475+0000", 2024-09-11T09:52:21.864 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.5076000000000001e-05 2024-09-11T09:52:21.864 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:21.864 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:21.864 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:21.864 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:21.864 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:21.865 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:52:21.866 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-11T09:52:21.866 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-11T09:52:21.867 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T09:52:21.867 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-11T09:52:21.867 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T09:52:21.868 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-11T09:52:21.868 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T09:52:21.868 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:52:21.867+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:52:21.893 INFO:teuthology.orchestra.run.smithi017.stderr:reweighted osd.3 to 1 (10000) 2024-09-11T09:52:21.893 INFO:teuthology.orchestra.run.smithi017.stderr:noscrub is unset 2024-09-11T09:52:21.911 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 4 1 2024-09-11T09:52:21.983 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-11T09:52:21.983 INFO:teuthology.orchestra.run.smithi096.stdout: "ops": [], 2024-09-11T09:52:21.983 INFO:teuthology.orchestra.run.smithi096.stdout: "num_ops": 0 2024-09-11T09:52:21.983 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-11T09:52:21.992 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-11T09:52:21.995 INFO:tasks.ceph.osd.0.smithi017.stderr:2024-09-11T09:52:21.996+0000 7fb73c0f1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31665) UID: 0 2024-09-11T09:52:22.067 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-09-11T09:52:22.068+0000 7f7b6c7d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31904) UID: 0 2024-09-11T09:52:22.097 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-11T09:52:22.097 INFO:teuthology.orchestra.run.smithi096.stdout: "ops": [], 2024-09-11T09:52:22.097 INFO:teuthology.orchestra.run.smithi096.stdout: "complaint_time": 30, 2024-09-11T09:52:22.097 INFO:teuthology.orchestra.run.smithi096.stdout: "num_blocked_ops": 0 2024-09-11T09:52:22.097 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-11T09:52:22.105 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-11T09:52:22.189 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:52:22.190+0000 7f3c96ec9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31909) UID: 0 2024-09-11T09:52:22.241 INFO:tasks.workunit.client.0.smithi107.stderr:++ rbd journal inspect --pool rbd --image testrbdnbd27121 2024-09-11T09:52:22.241 INFO:tasks.workunit.client.0.smithi107.stderr:++ awk '/entries inspected/ {print $1}' 2024-09-11T09:52:22.249 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-11T09:52:22.249 INFO:teuthology.orchestra.run.smithi096.stdout: "size": 20, 2024-09-11T09:52:22.249 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 600, 2024-09-11T09:52:22.249 INFO:teuthology.orchestra.run.smithi096.stdout: "ops": [ 2024-09-11T09:52:22.249 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.250 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:111 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:52:22.250 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.711668+0000", 2024-09-11T09:52:22.250 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 130.53767526199999, 2024-09-11T09:52:22.250 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0077145570000000004, 2024-09-11T09:52:22.250 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:22.250 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:22.250 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:22.250 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:52:22.250 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:52:22.250 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:52:22.250 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 111 2024-09-11T09:52:22.250 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.250 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:22.251 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.251 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:22.251 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711668+0000", 2024-09-11T09:52:22.251 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:22.251 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.251 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.251 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:22.251 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711668+0000", 2024-09-11T09:52:22.251 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:22.251 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.251 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.251 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:22.251 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711669+0000", 2024-09-11T09:52:22.251 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.99e-07 2024-09-11T09:52:22.252 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.252 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.252 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:22.252 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711673+0000", 2024-09-11T09:52:22.252 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.7440000000000001e-06 2024-09-11T09:52:22.252 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.252 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.252 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:22.252 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711686+0000", 2024-09-11T09:52:22.252 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.3317e-05 2024-09-11T09:52:22.252 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.252 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.252 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:22.252 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711697+0000", 2024-09-11T09:52:22.253 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1022000000000001e-05 2024-09-11T09:52:22.253 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.253 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.253 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:22.253 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.717065+0000", 2024-09-11T09:52:22.253 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.005367531 2024-09-11T09:52:22.253 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.253 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.253 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:22.253 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.717114+0000", 2024-09-11T09:52:22.253 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.9707999999999999e-05 2024-09-11T09:52:22.253 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.253 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.254 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:52:22.254 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.717190+0000", 2024-09-11T09:52:22.254 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.5394000000000004e-05 2024-09-11T09:52:22.254 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.254 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.254 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:22.254 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718793+0000", 2024-09-11T09:52:22.254 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001603653 2024-09-11T09:52:22.254 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.254 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.254 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:52:22.254 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719329+0000", 2024-09-11T09:52:22.254 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00053543799999999995 2024-09-11T09:52:22.254 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.254 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.255 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:22.255 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719359+0000", 2024-09-11T09:52:22.255 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.0420999999999999e-05 2024-09-11T09:52:22.255 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.255 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.255 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:22.255 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719383+0000", 2024-09-11T09:52:22.255 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.3730000000000001e-05 2024-09-11T09:52:22.255 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:22.255 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:22.255 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:22.255 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.255 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.256 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:112 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:52:22.256 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.711712+0000", 2024-09-11T09:52:22.256 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 130.537631702, 2024-09-11T09:52:22.256 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.008162084, 2024-09-11T09:52:22.256 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:22.256 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:22.256 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:22.256 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:52:22.256 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:52:22.256 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:52:22.256 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 112 2024-09-11T09:52:22.256 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.256 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:22.256 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.257 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:22.257 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711712+0000", 2024-09-11T09:52:22.257 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:22.257 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.257 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.257 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:22.257 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711712+0000", 2024-09-11T09:52:22.257 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:22.257 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.257 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.257 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:22.257 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711712+0000", 2024-09-11T09:52:22.257 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.6000000000000004e-07 2024-09-11T09:52:22.258 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.258 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.258 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:22.258 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711808+0000", 2024-09-11T09:52:22.258 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.5977999999999995e-05 2024-09-11T09:52:22.258 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.258 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.258 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:22.258 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711819+0000", 2024-09-11T09:52:22.258 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.0755e-05 2024-09-11T09:52:22.258 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.258 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.258 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:22.259 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711826+0000", 2024-09-11T09:52:22.259 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.7970000000000003e-06 2024-09-11T09:52:22.259 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.259 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.259 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:22.259 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.717435+0000", 2024-09-11T09:52:22.259 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0056090239999999998 2024-09-11T09:52:22.259 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.259 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.259 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:22.259 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.717496+0000", 2024-09-11T09:52:22.259 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.0543e-05 2024-09-11T09:52:22.259 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.259 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.260 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:52:22.260 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.717591+0000", 2024-09-11T09:52:22.260 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.5667000000000004e-05 2024-09-11T09:52:22.260 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.260 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.260 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:22.260 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718831+0000", 2024-09-11T09:52:22.260 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001240204 2024-09-11T09:52:22.260 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.260 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.260 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:52:22.260 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719819+0000", 2024-09-11T09:52:22.260 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00098751500000000001 2024-09-11T09:52:22.260 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.261 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.261 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:22.261 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719850+0000", 2024-09-11T09:52:22.261 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.1535999999999998e-05 2024-09-11T09:52:22.261 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.261 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.261 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:22.261 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719874+0000", 2024-09-11T09:52:22.261 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.3504999999999999e-05 2024-09-11T09:52:22.261 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:22.261 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:22.261 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:22.261 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.261 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.262 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:113 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:52:22.262 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.711842+0000", 2024-09-11T09:52:22.262 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 130.53750120300001, 2024-09-11T09:52:22.262 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0081160099999999999, 2024-09-11T09:52:22.262 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:22.262 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:22.262 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:22.262 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:52:22.262 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:52:22.262 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:52:22.262 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 113 2024-09-11T09:52:22.262 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.262 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:22.263 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.263 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:22.263 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711842+0000", 2024-09-11T09:52:22.263 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:22.263 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.263 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.263 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:22.263 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711842+0000", 2024-09-11T09:52:22.263 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:22.263 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.263 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.263 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:22.263 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711843+0000", 2024-09-11T09:52:22.263 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.4399999999999999e-07 2024-09-11T09:52:22.264 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.264 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.264 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:22.264 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711848+0000", 2024-09-11T09:52:22.264 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.4159999999999997e-06 2024-09-11T09:52:22.264 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.264 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.264 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:22.264 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711856+0000", 2024-09-11T09:52:22.264 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.7870000000000007e-06 2024-09-11T09:52:22.264 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.264 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.264 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:22.265 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711862+0000", 2024-09-11T09:52:22.265 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.3689999999999998e-06 2024-09-11T09:52:22.265 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.265 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.265 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:22.265 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.717965+0000", 2024-09-11T09:52:22.265 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0061028899999999997 2024-09-11T09:52:22.265 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.265 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.265 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:22.265 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718023+0000", 2024-09-11T09:52:22.265 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.8313000000000002e-05 2024-09-11T09:52:22.265 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.265 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.266 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:52:22.266 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718109+0000", 2024-09-11T09:52:22.266 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.6205000000000006e-05 2024-09-11T09:52:22.266 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.266 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.266 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:22.266 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719808+0000", 2024-09-11T09:52:22.266 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0016990060000000001 2024-09-11T09:52:22.266 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.266 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.266 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:52:22.266 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719883+0000", 2024-09-11T09:52:22.266 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.4624999999999998e-05 2024-09-11T09:52:22.266 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.266 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.267 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:22.267 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719910+0000", 2024-09-11T09:52:22.267 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.7512e-05 2024-09-11T09:52:22.267 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.267 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.267 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:22.267 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719958+0000", 2024-09-11T09:52:22.267 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.8143000000000003e-05 2024-09-11T09:52:22.267 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:22.267 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:22.267 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:22.267 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.267 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.268 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:52:22.268 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.711887+0000", 2024-09-11T09:52:22.268 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 130.53745658899999, 2024-09-11T09:52:22.268 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0086063589999999992, 2024-09-11T09:52:22.268 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:22.268 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:22.268 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:22.268 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:52:22.268 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:52:22.268 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:52:22.268 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 114 2024-09-11T09:52:22.268 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.268 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:22.269 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.269 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:22.269 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711887+0000", 2024-09-11T09:52:22.269 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:22.269 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.269 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.269 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:22.269 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711887+0000", 2024-09-11T09:52:22.269 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:22.269 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.269 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.269 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:22.269 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711887+0000", 2024-09-11T09:52:22.270 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.1500000000000005e-07 2024-09-11T09:52:22.270 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.270 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.270 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:22.270 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711897+0000", 2024-09-11T09:52:22.270 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.1900000000000001e-06 2024-09-11T09:52:22.270 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.270 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.270 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:22.270 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711902+0000", 2024-09-11T09:52:22.270 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.2349999999999999e-06 2024-09-11T09:52:22.270 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.270 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.270 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:22.271 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711906+0000", 2024-09-11T09:52:22.271 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.7129999999999999e-06 2024-09-11T09:52:22.271 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.271 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.271 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:22.271 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718429+0000", 2024-09-11T09:52:22.271 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0065231960000000002 2024-09-11T09:52:22.271 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.271 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.271 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:22.271 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718480+0000", 2024-09-11T09:52:22.271 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.1161e-05 2024-09-11T09:52:22.271 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.272 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.272 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:52:22.272 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718558+0000", 2024-09-11T09:52:22.272 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.7519000000000001e-05 2024-09-11T09:52:22.272 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.272 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.272 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:22.272 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720038+0000", 2024-09-11T09:52:22.272 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001480016 2024-09-11T09:52:22.272 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.272 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.272 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:52:22.272 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720417+0000", 2024-09-11T09:52:22.273 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00037943699999999999 2024-09-11T09:52:22.273 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.273 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.273 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:22.273 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720452+0000", 2024-09-11T09:52:22.273 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.4595000000000003e-05 2024-09-11T09:52:22.273 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.273 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.273 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:22.273 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720493+0000", 2024-09-11T09:52:22.274 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.1782000000000001e-05 2024-09-11T09:52:22.274 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:22.274 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:22.274 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:22.274 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.274 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.274 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:52:22.274 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.711912+0000", 2024-09-11T09:52:22.274 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 130.53743191999999, 2024-09-11T09:52:22.274 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0086864750000000008, 2024-09-11T09:52:22.274 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:22.274 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:22.275 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:22.275 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:52:22.275 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:52:22.275 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:52:22.275 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 115 2024-09-11T09:52:22.275 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.275 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:22.275 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.275 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:22.275 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711912+0000", 2024-09-11T09:52:22.275 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:22.275 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.275 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.275 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:22.275 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711912+0000", 2024-09-11T09:52:22.276 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:22.276 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.276 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.276 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:22.276 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711912+0000", 2024-09-11T09:52:22.276 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.3900000000000001e-07 2024-09-11T09:52:22.276 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.276 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.276 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:22.276 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711969+0000", 2024-09-11T09:52:22.276 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.7501999999999999e-05 2024-09-11T09:52:22.276 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.276 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.276 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:22.277 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711986+0000", 2024-09-11T09:52:22.277 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.6402999999999999e-05 2024-09-11T09:52:22.277 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.277 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.277 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:22.277 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711990+0000", 2024-09-11T09:52:22.277 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.929e-06 2024-09-11T09:52:22.277 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.277 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.277 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:22.277 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718861+0000", 2024-09-11T09:52:22.277 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0068710530000000002 2024-09-11T09:52:22.277 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.277 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.278 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:22.278 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718908+0000", 2024-09-11T09:52:22.278 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.7431e-05 2024-09-11T09:52:22.278 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.278 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.278 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:52:22.278 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718984+0000", 2024-09-11T09:52:22.278 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.5500999999999994e-05 2024-09-11T09:52:22.278 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.278 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.278 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:52:22.278 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720500+0000", 2024-09-11T09:52:22.278 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0015158820000000001 2024-09-11T09:52:22.278 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.279 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.279 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:22.279 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720536+0000", 2024-09-11T09:52:22.279 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.6118000000000002e-05 2024-09-11T09:52:22.279 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.279 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.279 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:22.279 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720570+0000", 2024-09-11T09:52:22.279 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.4387999999999997e-05 2024-09-11T09:52:22.279 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.279 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.279 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:22.279 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720598+0000", 2024-09-11T09:52:22.280 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.8028999999999999e-05 2024-09-11T09:52:22.280 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:22.280 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:22.280 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:22.280 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.280 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.280 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:52:22.280 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.711998+0000", 2024-09-11T09:52:22.280 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 130.537345741, 2024-09-11T09:52:22.280 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.010785813999999999, 2024-09-11T09:52:22.280 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:22.280 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:22.280 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:22.281 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:52:22.281 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:52:22.281 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:52:22.281 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 116 2024-09-11T09:52:22.281 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.281 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:22.281 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.281 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:22.281 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711998+0000", 2024-09-11T09:52:22.281 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:22.281 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.281 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.281 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:22.281 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711998+0000", 2024-09-11T09:52:22.282 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:22.282 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.282 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.282 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:22.282 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711998+0000", 2024-09-11T09:52:22.282 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.3300000000000003e-07 2024-09-11T09:52:22.282 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.282 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.282 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:22.282 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712001+0000", 2024-09-11T09:52:22.282 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.9349999999999999e-06 2024-09-11T09:52:22.282 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.282 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.282 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:22.283 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712006+0000", 2024-09-11T09:52:22.283 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.1630000000000004e-06 2024-09-11T09:52:22.283 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.283 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.283 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:22.283 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712010+0000", 2024-09-11T09:52:22.283 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.574e-06 2024-09-11T09:52:22.283 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.283 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.283 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:22.283 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719415+0000", 2024-09-11T09:52:22.283 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0074049099999999998 2024-09-11T09:52:22.283 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.283 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.284 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:22.284 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719465+0000", 2024-09-11T09:52:22.284 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.0129000000000001e-05 2024-09-11T09:52:22.284 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.284 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.284 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:52:22.284 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719545+0000", 2024-09-11T09:52:22.284 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.0253999999999994e-05 2024-09-11T09:52:22.284 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.284 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.284 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:22.284 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720636+0000", 2024-09-11T09:52:22.284 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0010909330000000001 2024-09-11T09:52:22.285 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.285 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.285 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:52:22.285 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.722707+0000", 2024-09-11T09:52:22.285 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0020713300000000001 2024-09-11T09:52:22.285 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.285 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.285 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:22.285 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.722743+0000", 2024-09-11T09:52:22.285 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.5351000000000001e-05 2024-09-11T09:52:22.285 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.285 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.285 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:22.286 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.722784+0000", 2024-09-11T09:52:22.286 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.0802e-05 2024-09-11T09:52:22.286 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:22.286 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:22.286 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:22.286 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.286 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.286 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:52:22.286 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.712014+0000", 2024-09-11T09:52:22.286 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 130.537329701, 2024-09-11T09:52:22.286 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.013420938, 2024-09-11T09:52:22.286 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:22.286 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:22.287 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:22.287 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:52:22.287 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:52:22.287 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:52:22.287 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 117 2024-09-11T09:52:22.287 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.287 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:22.287 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.287 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:22.287 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712014+0000", 2024-09-11T09:52:22.287 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:22.287 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.287 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.288 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:22.288 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712014+0000", 2024-09-11T09:52:22.288 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:22.288 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.288 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.288 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:22.288 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712014+0000", 2024-09-11T09:52:22.288 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.2000000000000001e-07 2024-09-11T09:52:22.288 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.288 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.288 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:22.288 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712016+0000", 2024-09-11T09:52:22.288 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.2309999999999999e-06 2024-09-11T09:52:22.288 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.289 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.289 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:22.289 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712020+0000", 2024-09-11T09:52:22.289 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.3830000000000001e-06 2024-09-11T09:52:22.289 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.289 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.289 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:22.289 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712023+0000", 2024-09-11T09:52:22.289 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.3340000000000002e-06 2024-09-11T09:52:22.289 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.289 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.289 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:22.289 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720083+0000", 2024-09-11T09:52:22.290 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0080601189999999993 2024-09-11T09:52:22.290 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.290 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.290 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:22.290 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720125+0000", 2024-09-11T09:52:22.290 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.2029e-05 2024-09-11T09:52:22.290 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.290 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.290 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:52:22.290 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720205+0000", 2024-09-11T09:52:22.290 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.0010000000000001e-05 2024-09-11T09:52:22.290 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.290 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.291 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:22.291 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720849+0000", 2024-09-11T09:52:22.291 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00064369899999999996 2024-09-11T09:52:22.291 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.291 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.291 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:52:22.291 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.725357+0000", 2024-09-11T09:52:22.291 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0045077379999999998 2024-09-11T09:52:22.291 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.291 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.291 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:22.291 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.725392+0000", 2024-09-11T09:52:22.291 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.5862999999999999e-05 2024-09-11T09:52:22.291 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.291 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.292 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:22.292 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.725435+0000", 2024-09-11T09:52:22.292 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.2311999999999997e-05 2024-09-11T09:52:22.292 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:22.292 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:22.292 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:22.292 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.292 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.292 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:159 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:52:22.292 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:12.452099+0000", 2024-09-11T09:52:22.292 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 129.797244113, 2024-09-11T09:52:22.292 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0069885950000000002, 2024-09-11T09:52:22.292 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:22.293 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:22.293 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:22.293 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:52:22.293 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:52:22.293 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:52:22.293 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 159 2024-09-11T09:52:22.293 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.293 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:22.293 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.293 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:22.293 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452099+0000", 2024-09-11T09:52:22.293 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:22.293 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.294 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.294 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:22.294 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452099+0000", 2024-09-11T09:52:22.294 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:22.294 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.294 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.294 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:22.294 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452100+0000", 2024-09-11T09:52:22.294 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.2800000000000003e-07 2024-09-11T09:52:22.294 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.294 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.294 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:22.294 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452102+0000", 2024-09-11T09:52:22.294 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.824e-06 2024-09-11T09:52:22.295 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.295 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.295 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:22.295 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452105+0000", 2024-09-11T09:52:22.295 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.0460000000000001e-06 2024-09-11T09:52:22.295 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.295 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.295 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:22.295 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452107+0000", 2024-09-11T09:52:22.295 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.543e-06 2024-09-11T09:52:22.295 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.295 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.295 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:22.295 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.455484+0000", 2024-09-11T09:52:22.296 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0033769759999999998 2024-09-11T09:52:22.296 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.296 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.296 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:22.296 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.455527+0000", 2024-09-11T09:52:22.296 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.2589999999999997e-05 2024-09-11T09:52:22.296 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.296 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.296 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:52:22.296 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.455594+0000", 2024-09-11T09:52:22.296 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.6775999999999999e-05 2024-09-11T09:52:22.296 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.296 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.297 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:22.297 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.456312+0000", 2024-09-11T09:52:22.297 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00071874700000000005 2024-09-11T09:52:22.297 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.297 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.297 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:52:22.297 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.459015+0000", 2024-09-11T09:52:22.297 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0027028540000000002 2024-09-11T09:52:22.297 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.297 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.297 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:22.297 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.459054+0000", 2024-09-11T09:52:22.297 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.8840000000000001e-05 2024-09-11T09:52:22.298 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.298 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.298 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:22.298 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.459088+0000", 2024-09-11T09:52:22.298 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.4071e-05 2024-09-11T09:52:22.298 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:22.298 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:22.298 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:22.298 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.298 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.298 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4688.0:38 2.5 2:abde2694:::rbd_data.1250f5e25b80.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:22.298 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:22.903926+0000", 2024-09-11T09:52:22.298 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 59.345417437000002, 2024-09-11T09:52:22.298 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.028303676, 2024-09-11T09:52:22.299 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:22.299 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:22.299 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:22.299 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:52:22.299 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4688", 2024-09-11T09:52:22.299 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:52:22.299 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 38 2024-09-11T09:52:22.299 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.299 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:22.299 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.299 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:22.299 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.903926+0000", 2024-09-11T09:52:22.299 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:22.300 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.300 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.300 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:22.300 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.903926+0000", 2024-09-11T09:52:22.300 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:22.300 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.300 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.300 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:22.300 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.903928+0000", 2024-09-11T09:52:22.300 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.4249999999999999e-06 2024-09-11T09:52:22.300 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.300 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.300 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:22.300 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.914774+0000", 2024-09-11T09:52:22.301 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.010846613 2024-09-11T09:52:22.301 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.301 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.301 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:22.301 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.914793+0000", 2024-09-11T09:52:22.301 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.8879999999999999e-05 2024-09-11T09:52:22.301 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.301 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.301 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:22.301 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.914801+0000", 2024-09-11T09:52:22.301 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.2230000000000007e-06 2024-09-11T09:52:22.301 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.301 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.302 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:22.302 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.914854+0000", 2024-09-11T09:52:22.302 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.2883999999999997e-05 2024-09-11T09:52:22.302 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.302 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.302 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:22.302 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.914951+0000", 2024-09-11T09:52:22.302 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.6731000000000006e-05 2024-09-11T09:52:22.302 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.302 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.302 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:52:22.302 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.915080+0000", 2024-09-11T09:52:22.302 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00012926999999999999 2024-09-11T09:52:22.303 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.303 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.303 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:22.303 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.931469+0000", 2024-09-11T09:52:22.303 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.016389107 2024-09-11T09:52:22.303 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.303 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.303 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:52:22.303 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.932118+0000", 2024-09-11T09:52:22.303 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00064915600000000002 2024-09-11T09:52:22.303 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.303 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.303 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:22.303 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.932177+0000", 2024-09-11T09:52:22.304 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.9032e-05 2024-09-11T09:52:22.304 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.304 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.304 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:22.304 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.932230+0000", 2024-09-11T09:52:22.304 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.2355000000000003e-05 2024-09-11T09:52:22.304 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:22.304 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:22.304 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:22.304 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.304 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.304 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_repop(client.4688.0:36 2.6 e53/15 2:61e1b442:::rbd_data.1250f5e25b80.0000000000000003:head v 53'2, mlcod=0'0)", 2024-09-11T09:52:22.304 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:22.915597+0000", 2024-09-11T09:52:22.305 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 59.333746886999997, 2024-09-11T09:52:22.305 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.020430798, 2024-09-11T09:52:22.305 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:22.305 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:22.305 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:22.305 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:22.305 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.305 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:22.305 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.915597+0000", 2024-09-11T09:52:22.305 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:22.305 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.305 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.305 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:22.306 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.915597+0000", 2024-09-11T09:52:22.306 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:22.306 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.306 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.306 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:22.306 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.915598+0000", 2024-09-11T09:52:22.306 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.7150000000000001e-06 2024-09-11T09:52:22.306 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.306 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.306 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:22.306 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921741+0000", 2024-09-11T09:52:22.306 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0061424339999999996 2024-09-11T09:52:22.306 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.307 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.307 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:22.307 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921757+0000", 2024-09-11T09:52:22.307 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.6144e-05 2024-09-11T09:52:22.307 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.307 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.307 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:22.307 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921765+0000", 2024-09-11T09:52:22.307 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.0809999999999994e-06 2024-09-11T09:52:22.307 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.307 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.307 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:22.307 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921826+0000", 2024-09-11T09:52:22.308 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.0878999999999998e-05 2024-09-11T09:52:22.308 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.308 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.308 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:22.308 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921849+0000", 2024-09-11T09:52:22.308 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.2966999999999999e-05 2024-09-11T09:52:22.308 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.308 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.308 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:22.308 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.935968+0000", 2024-09-11T09:52:22.308 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.014119191 2024-09-11T09:52:22.308 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.308 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.309 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:22.309 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.936027+0000", 2024-09-11T09:52:22.309 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.9386999999999999e-05 2024-09-11T09:52:22.309 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:22.309 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:22.309 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:22.309 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.309 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.309 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_repop(client.4688.0:39 2.6 e53/15 2:743e7616:::rbd_data.1250f5e25b80.0000000000000005:head v 53'3, mlcod=0'0)", 2024-09-11T09:52:22.309 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:22.921806+0000", 2024-09-11T09:52:22.309 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 59.327537837000001, 2024-09-11T09:52:22.309 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.015275018, 2024-09-11T09:52:22.309 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:22.310 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:22.310 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:22.310 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:22.310 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.310 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:22.310 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921806+0000", 2024-09-11T09:52:22.310 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:22.310 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.310 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.310 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:22.310 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921806+0000", 2024-09-11T09:52:22.310 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:22.310 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.310 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.311 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:22.311 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921806+0000", 2024-09-11T09:52:22.311 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.2200000000000003e-07 2024-09-11T09:52:22.311 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.311 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.311 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:22.311 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.925590+0000", 2024-09-11T09:52:22.311 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0037838949999999998 2024-09-11T09:52:22.311 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.311 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.311 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:22.311 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.925605+0000", 2024-09-11T09:52:22.311 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.4321999999999999e-05 2024-09-11T09:52:22.312 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.312 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.312 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:22.312 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.925613+0000", 2024-09-11T09:52:22.312 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.1980000000000001e-06 2024-09-11T09:52:22.312 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.312 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.312 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:22.312 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.925857+0000", 2024-09-11T09:52:22.312 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000244064 2024-09-11T09:52:22.312 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.312 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.312 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:22.313 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.925875+0000", 2024-09-11T09:52:22.313 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.838e-05 2024-09-11T09:52:22.313 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.313 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.313 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:22.313 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.937042+0000", 2024-09-11T09:52:22.313 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.011166416 2024-09-11T09:52:22.313 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.313 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.313 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:22.313 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.937081+0000", 2024-09-11T09:52:22.313 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.9020999999999998e-05 2024-09-11T09:52:22.313 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:22.314 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:22.314 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:22.314 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.314 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.314 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4688.0:49 2.5 2:b6dd9fc9:::rbd_data.1250f5e25b80.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:22.314 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:22.938548+0000", 2024-09-11T09:52:22.314 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 59.310795833999997, 2024-09-11T09:52:22.314 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.022766950000000001, 2024-09-11T09:52:22.314 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:22.314 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:22.314 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:22.314 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:52:22.314 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4688", 2024-09-11T09:52:22.315 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:52:22.315 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 49 2024-09-11T09:52:22.315 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.315 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:22.315 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.315 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:22.315 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.938548+0000", 2024-09-11T09:52:22.315 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:22.315 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.315 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.315 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:22.315 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.938548+0000", 2024-09-11T09:52:22.315 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:22.315 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.316 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.316 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:22.316 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.938550+0000", 2024-09-11T09:52:22.316 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.7710000000000001e-06 2024-09-11T09:52:22.316 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.316 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.316 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:22.316 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.944237+0000", 2024-09-11T09:52:22.316 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0056879790000000001 2024-09-11T09:52:22.316 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.316 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.316 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:22.316 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.944262+0000", 2024-09-11T09:52:22.317 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.4247e-05 2024-09-11T09:52:22.317 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.317 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.317 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:22.317 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.944274+0000", 2024-09-11T09:52:22.317 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.2639999999999999e-05 2024-09-11T09:52:22.317 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.317 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.317 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:22.317 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.944344+0000", 2024-09-11T09:52:22.317 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.9574999999999997e-05 2024-09-11T09:52:22.317 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.317 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.318 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:22.318 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.944474+0000", 2024-09-11T09:52:22.318 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000130464 2024-09-11T09:52:22.318 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.318 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.318 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:52:22.318 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.944585+0000", 2024-09-11T09:52:22.318 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000110096 2024-09-11T09:52:22.318 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.318 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.318 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:52:22.318 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.952724+0000", 2024-09-11T09:52:22.318 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0081398900000000003 2024-09-11T09:52:22.318 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.318 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.319 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:22.319 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.961219+0000", 2024-09-11T09:52:22.319 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0084949789999999997 2024-09-11T09:52:22.319 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.319 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.319 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:22.319 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.961280+0000", 2024-09-11T09:52:22.319 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.0404e-05 2024-09-11T09:52:22.319 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.319 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.319 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:22.319 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.961315+0000", 2024-09-11T09:52:22.319 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.4904999999999999e-05 2024-09-11T09:52:22.320 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:22.320 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:22.320 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:22.320 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.320 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.320 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_repop(client.4688.0:44 2.6 e53/15 2:67246a22:::rbd_data.1250f5e25b80.0000000000000007:head v 53'4, mlcod=53'2)", 2024-09-11T09:52:22.320 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:22.951721+0000", 2024-09-11T09:52:22.320 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 59.297622562000001, 2024-09-11T09:52:22.320 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.01080981, 2024-09-11T09:52:22.320 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:22.320 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:22.320 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:22.320 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:22.321 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.321 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:22.321 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.951721+0000", 2024-09-11T09:52:22.321 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:22.321 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.321 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.321 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:22.321 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.951721+0000", 2024-09-11T09:52:22.321 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:22.321 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.321 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.321 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:22.321 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.951722+0000", 2024-09-11T09:52:22.322 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.0279999999999999e-06 2024-09-11T09:52:22.322 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.322 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.322 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:22.322 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.955612+0000", 2024-09-11T09:52:22.322 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0038896920000000001 2024-09-11T09:52:22.322 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.322 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.322 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:22.322 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.955628+0000", 2024-09-11T09:52:22.322 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.5945000000000001e-05 2024-09-11T09:52:22.322 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.322 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.323 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:22.323 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.955635+0000", 2024-09-11T09:52:22.323 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.8080000000000005e-06 2024-09-11T09:52:22.323 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.323 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.323 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:22.323 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.955700+0000", 2024-09-11T09:52:22.323 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.4114000000000003e-05 2024-09-11T09:52:22.323 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.323 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.323 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:22.323 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.955724+0000", 2024-09-11T09:52:22.323 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.3989999999999999e-05 2024-09-11T09:52:22.323 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.324 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.324 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:22.324 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.962490+0000", 2024-09-11T09:52:22.324 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0067662599999999996 2024-09-11T09:52:22.324 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.324 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.324 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:22.324 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.962531+0000", 2024-09-11T09:52:22.324 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.0973000000000002e-05 2024-09-11T09:52:22.324 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:22.324 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:22.324 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:22.324 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.325 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.325 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_repop(client.4688.0:51 2.6 e53/15 2:6b5894fc:::rbd_data.1250f5e25b80.000000000000000a:head v 53'5, mlcod=53'2)", 2024-09-11T09:52:22.325 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:22.959030+0000", 2024-09-11T09:52:22.325 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 59.290313388000001, 2024-09-11T09:52:22.325 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.011503638, 2024-09-11T09:52:22.325 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:22.325 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:22.325 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:22.325 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:22.325 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.325 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:22.325 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.959030+0000", 2024-09-11T09:52:22.325 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:22.326 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.326 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.326 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:22.326 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.959030+0000", 2024-09-11T09:52:22.326 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:22.326 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.326 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.326 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:22.326 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.959032+0000", 2024-09-11T09:52:22.326 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.356e-06 2024-09-11T09:52:22.326 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.326 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.326 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:22.326 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.962899+0000", 2024-09-11T09:52:22.327 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.003867639 2024-09-11T09:52:22.327 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.327 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.327 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:22.327 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.962909+0000", 2024-09-11T09:52:22.327 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.0105999999999999e-05 2024-09-11T09:52:22.327 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.327 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.327 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:22.327 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.962915+0000", 2024-09-11T09:52:22.327 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.8869999999999997e-06 2024-09-11T09:52:22.327 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.327 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.327 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:22.328 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.962962+0000", 2024-09-11T09:52:22.328 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.6511999999999998e-05 2024-09-11T09:52:22.328 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.328 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.328 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:22.328 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.962989+0000", 2024-09-11T09:52:22.328 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.7702999999999999e-05 2024-09-11T09:52:22.328 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.328 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.328 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:22.328 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.970495+0000", 2024-09-11T09:52:22.328 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0075052449999999998 2024-09-11T09:52:22.328 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.329 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.329 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:22.329 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.970534+0000", 2024-09-11T09:52:22.329 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.9190000000000003e-05 2024-09-11T09:52:22.329 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:22.329 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:22.329 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:22.329 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.329 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.329 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_repop(client.4688.0:58 2.6 e53/15 2:6277c644:::rbd_data.1250f5e25b80.000000000000000e:head v 53'6, mlcod=53'4)", 2024-09-11T09:52:22.329 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:22.996199+0000", 2024-09-11T09:52:22.329 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 59.253144468000002, 2024-09-11T09:52:22.329 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.010145573999999999, 2024-09-11T09:52:22.329 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:22.330 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:22.330 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:22.330 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:22.330 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.330 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:22.330 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.996199+0000", 2024-09-11T09:52:22.330 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:22.330 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.330 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.330 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:22.330 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.996199+0000", 2024-09-11T09:52:22.330 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:22.330 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.331 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.331 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:22.331 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.996200+0000", 2024-09-11T09:52:22.331 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.85e-07 2024-09-11T09:52:22.331 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.331 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.331 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:22.331 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.999914+0000", 2024-09-11T09:52:22.331 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.003714406 2024-09-11T09:52:22.331 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.331 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.331 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:22.331 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.999928+0000", 2024-09-11T09:52:22.332 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.3149000000000001e-05 2024-09-11T09:52:22.332 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.332 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.332 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:22.332 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.999951+0000", 2024-09-11T09:52:22.332 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.3116e-05 2024-09-11T09:52:22.332 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.332 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.332 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:22.332 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.000003+0000", 2024-09-11T09:52:22.332 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.2169000000000001e-05 2024-09-11T09:52:22.332 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.332 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.333 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:22.333 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.000018+0000", 2024-09-11T09:52:22.333 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.5500000000000001e-05 2024-09-11T09:52:22.333 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.333 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.333 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:22.333 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.006303+0000", 2024-09-11T09:52:22.333 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0062846710000000004 2024-09-11T09:52:22.333 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.333 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.333 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:22.333 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.006345+0000", 2024-09-11T09:52:22.333 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.1678000000000003e-05 2024-09-11T09:52:22.334 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:22.334 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:22.334 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:22.334 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.334 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.334 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4694.0:42 2.5 2:a4fc00ec:ns::rbd_data.1256a0c02879.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:22.334 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.111570+0000", 2024-09-11T09:52:22.334 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 59.137773361999997, 2024-09-11T09:52:22.334 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.038002197000000001, 2024-09-11T09:52:22.334 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:22.334 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:22.334 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:22.334 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:52:22.334 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4694", 2024-09-11T09:52:22.335 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:52:22.335 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 42 2024-09-11T09:52:22.335 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.335 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:22.335 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.335 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:22.335 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.111570+0000", 2024-09-11T09:52:22.335 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:22.335 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.335 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.335 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:22.335 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.111570+0000", 2024-09-11T09:52:22.335 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:22.335 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.336 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.336 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:22.336 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.111572+0000", 2024-09-11T09:52:22.336 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.3909999999999999e-06 2024-09-11T09:52:22.336 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.336 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.336 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:22.336 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.135929+0000", 2024-09-11T09:52:22.336 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.024357410999999999 2024-09-11T09:52:22.336 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.336 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.336 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:22.336 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.135951+0000", 2024-09-11T09:52:22.337 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.1775000000000001e-05 2024-09-11T09:52:22.337 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.337 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.337 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:22.337 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.135962+0000", 2024-09-11T09:52:22.337 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.0903e-05 2024-09-11T09:52:22.337 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.337 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.337 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:22.337 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.136045+0000", 2024-09-11T09:52:22.337 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.3128999999999996e-05 2024-09-11T09:52:22.337 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.337 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.337 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:22.338 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.136229+0000", 2024-09-11T09:52:22.338 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00018393700000000001 2024-09-11T09:52:22.338 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.338 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.338 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:52:22.338 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.136394+0000", 2024-09-11T09:52:22.338 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000165527 2024-09-11T09:52:22.338 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.338 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.338 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:22.338 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.149320+0000", 2024-09-11T09:52:22.338 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.01292568 2024-09-11T09:52:22.338 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.339 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.339 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:52:22.339 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.149493+0000", 2024-09-11T09:52:22.339 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00017285999999999999 2024-09-11T09:52:22.339 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.339 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.339 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:22.339 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.149539+0000", 2024-09-11T09:52:22.339 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.6628000000000001e-05 2024-09-11T09:52:22.339 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.339 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.339 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:22.339 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.149572+0000", 2024-09-11T09:52:22.340 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.2956000000000001e-05 2024-09-11T09:52:22.340 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:22.340 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:22.340 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:22.340 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.340 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.340 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_repop(client.4694.0:45 2.6 e53/15 2:671fb830:ns::rbd_data.1256a0c02879.0000000000000007:head v 53'9, mlcod=53'7)", 2024-09-11T09:52:22.340 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.121833+0000", 2024-09-11T09:52:22.340 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 59.127511003999999, 2024-09-11T09:52:22.340 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.023876192000000001, 2024-09-11T09:52:22.340 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:22.340 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:22.341 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:22.341 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:22.341 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.341 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:22.341 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.121833+0000", 2024-09-11T09:52:22.341 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:22.341 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.341 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.341 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:22.341 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.121833+0000", 2024-09-11T09:52:22.341 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:22.341 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.341 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.342 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:22.342 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.121834+0000", 2024-09-11T09:52:22.342 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.471e-06 2024-09-11T09:52:22.342 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.342 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.342 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:22.342 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.128990+0000", 2024-09-11T09:52:22.342 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0071557879999999997 2024-09-11T09:52:22.342 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.342 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.342 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:22.342 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.129011+0000", 2024-09-11T09:52:22.342 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.0760000000000001e-05 2024-09-11T09:52:22.343 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.343 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.343 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:22.343 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.129022+0000", 2024-09-11T09:52:22.343 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1056000000000001e-05 2024-09-11T09:52:22.343 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.343 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.343 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:22.343 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.129097+0000", 2024-09-11T09:52:22.343 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.5082999999999999e-05 2024-09-11T09:52:22.343 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.343 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.343 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:22.344 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.129130+0000", 2024-09-11T09:52:22.344 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.3345e-05 2024-09-11T09:52:22.344 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.344 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.344 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:22.344 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.145657+0000", 2024-09-11T09:52:22.344 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.016527224 2024-09-11T09:52:22.344 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.344 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.344 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:22.344 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.145709+0000", 2024-09-11T09:52:22.344 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.1465000000000003e-05 2024-09-11T09:52:22.344 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:22.345 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:22.345 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:22.345 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.345 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.345 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4694.0:43 2.0 2:16c8ae84:ns::rbd_data.1256a0c02879.0000000000000005:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:22.345 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.135999+0000", 2024-09-11T09:52:22.345 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 59.113345054, 2024-09-11T09:52:22.345 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.039468309, 2024-09-11T09:52:22.345 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:22.345 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:22.345 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:22.345 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:52:22.345 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4694", 2024-09-11T09:52:22.346 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:52:22.346 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 43 2024-09-11T09:52:22.346 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.346 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:22.346 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.346 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:22.346 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.135999+0000", 2024-09-11T09:52:22.346 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:22.346 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.346 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.346 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:22.346 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.135999+0000", 2024-09-11T09:52:22.346 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:22.347 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.347 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.347 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:22.347 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.136000+0000", 2024-09-11T09:52:22.347 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.6330000000000001e-06 2024-09-11T09:52:22.347 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.347 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.347 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:22.347 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.157961+0000", 2024-09-11T09:52:22.347 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0219606 2024-09-11T09:52:22.347 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.347 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.347 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:22.348 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.157995+0000", 2024-09-11T09:52:22.348 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.4391999999999999e-05 2024-09-11T09:52:22.348 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.348 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.348 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:22.348 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.158011+0000", 2024-09-11T09:52:22.348 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.5794000000000002e-05 2024-09-11T09:52:22.348 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.348 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.348 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:22.348 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.158083+0000", 2024-09-11T09:52:22.348 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.2273000000000004e-05 2024-09-11T09:52:22.348 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.348 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.349 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:22.349 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.158239+0000", 2024-09-11T09:52:22.349 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000155372 2024-09-11T09:52:22.349 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.349 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.349 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 1", 2024-09-11T09:52:22.349 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.158403+0000", 2024-09-11T09:52:22.349 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00016408199999999999 2024-09-11T09:52:22.349 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.349 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.349 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:52:22.349 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.173860+0000", 2024-09-11T09:52:22.349 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.015457347 2024-09-11T09:52:22.350 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.350 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.350 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:22.350 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.175400+0000", 2024-09-11T09:52:22.350 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001540006 2024-09-11T09:52:22.350 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.350 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.350 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:22.350 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.175438+0000", 2024-09-11T09:52:22.350 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.8135e-05 2024-09-11T09:52:22.350 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.350 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.350 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:22.350 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.175467+0000", 2024-09-11T09:52:22.351 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.8674999999999999e-05 2024-09-11T09:52:22.351 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:22.351 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:22.351 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:22.351 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.351 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.351 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4694.0:57 2.0 2:04b5549c:ns::rbd_data.1256a0c02879.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:22.351 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.158037+0000", 2024-09-11T09:52:22.351 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 59.091306316000001, 2024-09-11T09:52:22.351 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.027748790999999998, 2024-09-11T09:52:22.351 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:22.351 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:22.351 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:22.352 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:52:22.352 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4694", 2024-09-11T09:52:22.352 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:52:22.352 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 57 2024-09-11T09:52:22.352 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.352 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:22.352 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.352 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:22.352 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.158037+0000", 2024-09-11T09:52:22.352 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:22.352 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.352 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.352 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:22.352 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.158037+0000", 2024-09-11T09:52:22.353 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:22.353 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.353 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.353 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:22.353 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.158039+0000", 2024-09-11T09:52:22.353 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.356e-06 2024-09-11T09:52:22.353 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.353 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.353 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:22.353 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.176367+0000", 2024-09-11T09:52:22.353 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.018328739 2024-09-11T09:52:22.353 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.353 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.353 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:22.354 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.176378+0000", 2024-09-11T09:52:22.354 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.0713999999999999e-05 2024-09-11T09:52:22.354 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.354 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.354 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:22.354 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.176383+0000", 2024-09-11T09:52:22.354 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.0209999999999999e-06 2024-09-11T09:52:22.354 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.354 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.354 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:22.354 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.176426+0000", 2024-09-11T09:52:22.354 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.2623999999999997e-05 2024-09-11T09:52:22.354 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.355 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.355 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:22.355 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.176552+0000", 2024-09-11T09:52:22.355 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000126076 2024-09-11T09:52:22.355 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.355 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.355 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 1", 2024-09-11T09:52:22.355 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.176665+0000", 2024-09-11T09:52:22.355 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000112995 2024-09-11T09:52:22.355 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.355 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.355 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:52:22.355 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.183639+0000", 2024-09-11T09:52:22.356 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0069740039999999998 2024-09-11T09:52:22.356 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.356 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.356 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:22.356 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.185689+0000", 2024-09-11T09:52:22.356 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.002049852 2024-09-11T09:52:22.356 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.356 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.356 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:22.356 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.185725+0000", 2024-09-11T09:52:22.356 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.6804999999999997e-05 2024-09-11T09:52:22.356 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.356 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.356 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:22.357 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.185786+0000", 2024-09-11T09:52:22.357 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.0605e-05 2024-09-11T09:52:22.357 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:22.357 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:22.357 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:22.357 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.357 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.357 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4694.0:60 2.0 2:054f2af4:ns::rbd_data.1256a0c02879.000000000000000f:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:22.357 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.186855+0000", 2024-09-11T09:52:22.357 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 59.062488299999998, 2024-09-11T09:52:22.357 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.013682913, 2024-09-11T09:52:22.357 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:22.357 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:22.358 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:22.358 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:52:22.358 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4694", 2024-09-11T09:52:22.358 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:52:22.358 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 60 2024-09-11T09:52:22.358 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.358 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:22.358 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.358 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:22.358 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.186855+0000", 2024-09-11T09:52:22.358 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:22.358 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.358 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.359 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:22.359 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.186855+0000", 2024-09-11T09:52:22.359 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:22.359 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.359 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.359 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:22.359 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.186856+0000", 2024-09-11T09:52:22.359 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.1200000000000001e-07 2024-09-11T09:52:22.359 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.359 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.359 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:22.359 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.191484+0000", 2024-09-11T09:52:22.359 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0046274159999999996 2024-09-11T09:52:22.359 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.360 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.360 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:22.360 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.191496+0000", 2024-09-11T09:52:22.360 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.2149e-05 2024-09-11T09:52:22.360 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.360 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.360 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:22.360 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.191502+0000", 2024-09-11T09:52:22.360 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.1890000000000004e-06 2024-09-11T09:52:22.360 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.360 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.360 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:22.360 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.191544+0000", 2024-09-11T09:52:22.361 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.1724999999999998e-05 2024-09-11T09:52:22.361 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.361 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.361 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:22.361 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.191641+0000", 2024-09-11T09:52:22.361 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.6939000000000001e-05 2024-09-11T09:52:22.361 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.361 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.364 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 1", 2024-09-11T09:52:22.364 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.191731+0000", 2024-09-11T09:52:22.364 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.0651999999999999e-05 2024-09-11T09:52:22.364 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.364 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.364 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:52:22.364 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.199852+0000", 2024-09-11T09:52:22.364 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0081204529999999997 2024-09-11T09:52:22.364 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.365 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.365 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:22.365 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.200461+0000", 2024-09-11T09:52:22.365 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00060900899999999998 2024-09-11T09:52:22.365 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.365 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.365 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:22.365 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.200497+0000", 2024-09-11T09:52:22.365 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.5800000000000003e-05 2024-09-11T09:52:22.365 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.365 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.365 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:22.365 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.200538+0000", 2024-09-11T09:52:22.366 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.1668999999999997e-05 2024-09-11T09:52:22.366 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:22.366 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:22.366 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:22.366 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:22.366 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:22.366 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-11T09:52:22.367 INFO:tasks.ceph.osd.0.smithi017.stderr:2024-09-11T09:52:22.368+0000 7fb73c0f1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31665) UID: 0 2024-09-11T09:52:22.369 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-11T09:52:22.370 INFO:tasks.workunit.client.0.smithi107.stderr:2024-09-11T09:52:22.304+0000 7f68b37fe640 -1 JournalMetadata: 0x55bf04444320 failed to locate client: INSPECT 2024-09-11T09:52:22.370 INFO:tasks.workunit.client.0.smithi107.stderr:+ queue_length=0 2024-09-11T09:52:22.370 INFO:tasks.workunit.client.0.smithi107.stderr:+ tries=5 2024-09-11T09:52:22.370 INFO:tasks.workunit.client.0.smithi107.stderr:+ '[' 5 -lt 10 ']' 2024-09-11T09:52:22.370 INFO:tasks.workunit.client.0.smithi107.stderr:+ '[' 0 -gt 0 ']' 2024-09-11T09:52:22.370 INFO:tasks.workunit.client.0.smithi107.stderr:+ '[' 0 -eq 0 ']' 2024-09-11T09:52:22.370 INFO:tasks.workunit.client.0.smithi107.stderr:+ unmap_device /dev/nbd2 30183 2024-09-11T09:52:22.370 INFO:tasks.workunit.client.0.smithi107.stderr:+ local args=/dev/nbd2 2024-09-11T09:52:22.370 INFO:tasks.workunit.client.0.smithi107.stderr:+ local pid=30183 2024-09-11T09:52:22.370 INFO:tasks.workunit.client.0.smithi107.stderr:+ _sudo rbd device --device-type nbd unmap /dev/nbd2 2024-09-11T09:52:22.371 INFO:tasks.workunit.client.0.smithi107.stderr:+ local cmd 2024-09-11T09:52:22.371 INFO:tasks.workunit.client.0.smithi107.stderr:++ id -u 2024-09-11T09:52:22.371 INFO:tasks.workunit.client.0.smithi107.stderr:+ '[' 1000 -eq 0 ']' 2024-09-11T09:52:22.371 INFO:tasks.workunit.client.0.smithi107.stderr:++ which rbd 2024-09-11T09:52:22.371 INFO:tasks.workunit.client.0.smithi107.stderr:+ cmd=/usr/bin/rbd 2024-09-11T09:52:22.371 INFO:tasks.workunit.client.0.smithi107.stderr:+ shift 2024-09-11T09:52:22.371 INFO:tasks.workunit.client.0.smithi107.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd unmap /dev/nbd2 2024-09-11T09:52:22.420 INFO:tasks.workunit.client.0.smithi107.stderr:+ rbd device --device-type nbd list 2024-09-11T09:52:22.420 INFO:tasks.workunit.client.0.smithi107.stderr:+ expect_false grep '^30183\b' 2024-09-11T09:52:22.421 INFO:tasks.workunit.client.0.smithi107.stderr:+ grep '^30183\b' 2024-09-11T09:52:22.468 INFO:tasks.ceph.osd.2.smithi017.stderr:2024-09-11T09:52:22.469+0000 7fd20b170640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31672) UID: 0 2024-09-11T09:52:22.480 INFO:tasks.workunit.client.0.smithi107.stderr:+ return 0 2024-09-11T09:52:22.480 INFO:tasks.workunit.client.0.smithi107.stderr:+ ps -C rbd-nbd 2024-09-11T09:52:22.481 INFO:tasks.workunit.client.0.smithi107.stderr:+ expect_false grep '^ *30183\b' 2024-09-11T09:52:22.481 INFO:tasks.workunit.client.0.smithi107.stderr:+ grep '^ *30183\b' 2024-09-11T09:52:22.491 INFO:tasks.workunit.client.0.smithi107.stderr:+ return 0 2024-09-11T09:52:22.492 INFO:tasks.workunit.client.0.smithi107.stderr:+ sleep 0.5 2024-09-11T09:52:22.511 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-11T09:52:22.511 INFO:teuthology.orchestra.run.smithi096.stdout: "ops": [], 2024-09-11T09:52:22.511 INFO:teuthology.orchestra.run.smithi096.stdout: "num_ops": 0 2024-09-11T09:52:22.511 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-11T09:52:22.520 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-11T09:52:22.569 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:52:22.569+0000 7f3c96ec9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31909) UID: 0 2024-09-11T09:52:22.623 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-11T09:52:22.623 INFO:teuthology.orchestra.run.smithi096.stdout: "ops": [], 2024-09-11T09:52:22.623 INFO:teuthology.orchestra.run.smithi096.stdout: "complaint_time": 30, 2024-09-11T09:52:22.623 INFO:teuthology.orchestra.run.smithi096.stdout: "num_blocked_ops": 0 2024-09-11T09:52:22.623 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-11T09:52:22.631 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-11T09:52:22.672 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:52:22.673+0000 7f3a7b90a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 42475) UID: 0 2024-09-11T09:52:22.771 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-09-11T09:52:22.771+0000 7f7b6c7d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31904) UID: 0 2024-09-11T09:52:22.775 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-11T09:52:22.776 INFO:teuthology.orchestra.run.smithi096.stdout: "size": 20, 2024-09-11T09:52:22.776 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 600, 2024-09-11T09:52:22.776 INFO:teuthology.orchestra.run.smithi096.stdout: "ops": [ 2024-09-11T09:52:22.776 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.776 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4230.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-11T09:52:22.776 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:18.490736+0000", 2024-09-11T09:52:22.776 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 124.284627753, 2024-09-11T09:52:22.776 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0059097569999999999, 2024-09-11T09:52:22.776 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:22.776 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:22.776 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:22.776 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:52:22.777 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4230", 2024-09-11T09:52:22.777 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.17:0/4101109576", 2024-09-11T09:52:22.777 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 1 2024-09-11T09:52:22.777 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.777 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:22.777 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.777 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:22.777 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.490736+0000", 2024-09-11T09:52:22.777 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:22.777 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.777 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.777 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:22.777 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.490736+0000", 2024-09-11T09:52:22.778 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:22.778 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.778 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.778 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:22.778 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.490742+0000", 2024-09-11T09:52:22.778 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.48e-06 2024-09-11T09:52:22.778 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.778 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.778 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:22.778 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.490749+0000", 2024-09-11T09:52:22.778 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.4569999999999999e-06 2024-09-11T09:52:22.778 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.778 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.779 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:22.779 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.490762+0000", 2024-09-11T09:52:22.779 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.2721000000000001e-05 2024-09-11T09:52:22.779 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.779 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.779 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:22.779 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.490774+0000", 2024-09-11T09:52:22.779 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1823e-05 2024-09-11T09:52:22.779 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.779 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.779 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:22.779 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.490846+0000", 2024-09-11T09:52:22.779 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.1878000000000005e-05 2024-09-11T09:52:22.779 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.780 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.780 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for peered", 2024-09-11T09:52:22.780 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.490857+0000", 2024-09-11T09:52:22.780 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1017e-05 2024-09-11T09:52:22.780 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.780 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.780 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:22.780 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.493856+0000", 2024-09-11T09:52:22.780 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00299885 2024-09-11T09:52:22.780 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.780 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.780 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:22.780 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.494020+0000", 2024-09-11T09:52:22.780 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00016436599999999999 2024-09-11T09:52:22.781 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.781 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.781 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 2", 2024-09-11T09:52:22.781 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.494186+0000", 2024-09-11T09:52:22.781 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00016562799999999999 2024-09-11T09:52:22.781 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.781 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.781 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:22.781 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.495407+0000", 2024-09-11T09:52:22.781 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0012217420000000001 2024-09-11T09:52:22.781 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.781 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.781 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:52:22.781 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.496579+0000", 2024-09-11T09:52:22.782 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001171844 2024-09-11T09:52:22.782 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.782 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.782 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:22.782 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.496620+0000", 2024-09-11T09:52:22.782 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.0479000000000003e-05 2024-09-11T09:52:22.782 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.782 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.782 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:22.782 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.496646+0000", 2024-09-11T09:52:22.782 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.6472e-05 2024-09-11T09:52:22.782 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:22.782 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:22.782 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:22.782 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.783 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.783 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4230.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-09-11T09:52:22.783 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:20.498634+0000", 2024-09-11T09:52:22.783 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 122.27672973200001, 2024-09-11T09:52:22.783 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0050648539999999997, 2024-09-11T09:52:22.783 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:22.783 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:22.783 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:22.783 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:52:22.783 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4230", 2024-09-11T09:52:22.783 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.17:0/4101109576", 2024-09-11T09:52:22.783 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 7 2024-09-11T09:52:22.783 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.783 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:22.783 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.784 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:22.784 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.498634+0000", 2024-09-11T09:52:22.784 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:22.784 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.784 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.784 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:22.784 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.498634+0000", 2024-09-11T09:52:22.784 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:22.784 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.784 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.784 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:22.784 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.498637+0000", 2024-09-11T09:52:22.784 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.1919999999999999e-06 2024-09-11T09:52:22.785 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.785 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.785 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:22.785 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.498648+0000", 2024-09-11T09:52:22.785 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1116e-05 2024-09-11T09:52:22.785 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.785 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.785 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:22.785 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.498666+0000", 2024-09-11T09:52:22.785 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.8411999999999999e-05 2024-09-11T09:52:22.785 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.785 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.785 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:22.785 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.498675+0000", 2024-09-11T09:52:22.786 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.4059999999999994e-06 2024-09-11T09:52:22.786 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.786 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.786 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:22.786 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.498745+0000", 2024-09-11T09:52:22.786 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.9764000000000005e-05 2024-09-11T09:52:22.786 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.786 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.786 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op must wait for map", 2024-09-11T09:52:22.786 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.498754+0000", 2024-09-11T09:52:22.786 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.5969999999999992e-06 2024-09-11T09:52:22.786 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.786 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.787 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:22.787 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.500385+0000", 2024-09-11T09:52:22.787 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0016308360000000001 2024-09-11T09:52:22.787 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.787 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.787 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:22.787 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.500473+0000", 2024-09-11T09:52:22.787 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.7966000000000004e-05 2024-09-11T09:52:22.787 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.787 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.787 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 1", 2024-09-11T09:52:22.787 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.500591+0000", 2024-09-11T09:52:22.787 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000117895 2024-09-11T09:52:22.788 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.788 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.788 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:22.788 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.501685+0000", 2024-09-11T09:52:22.788 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001094886 2024-09-11T09:52:22.788 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.788 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.788 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:52:22.788 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.503627+0000", 2024-09-11T09:52:22.788 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001941199 2024-09-11T09:52:22.788 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.789 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.789 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:22.789 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.503667+0000", 2024-09-11T09:52:22.789 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.0645000000000001e-05 2024-09-11T09:52:22.789 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.789 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.789 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:22.789 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.503699+0000", 2024-09-11T09:52:22.789 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.1940000000000003e-05 2024-09-11T09:52:22.789 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:22.789 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:22.789 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:22.789 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.790 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.790 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4688.0:32 2.2 2:45941fd2:::rbd_data.1250f5e25b80.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:22.790 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:22.896607+0000", 2024-09-11T09:52:22.790 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 59.878757129, 2024-09-11T09:52:22.790 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.038655179999999997, 2024-09-11T09:52:22.790 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:22.790 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:22.790 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:22.790 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:52:22.790 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4688", 2024-09-11T09:52:22.790 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:52:22.790 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 32 2024-09-11T09:52:22.790 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.791 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:22.791 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.791 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:22.791 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.896607+0000", 2024-09-11T09:52:22.791 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:22.791 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.791 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.791 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:22.791 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.896607+0000", 2024-09-11T09:52:22.791 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:22.791 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.791 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.791 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:22.792 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.896608+0000", 2024-09-11T09:52:22.792 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.0690000000000001e-06 2024-09-11T09:52:22.792 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.792 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.792 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:22.792 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.914461+0000", 2024-09-11T09:52:22.792 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.017852903 2024-09-11T09:52:22.792 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.792 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.792 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:22.792 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.914495+0000", 2024-09-11T09:52:22.792 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.4301000000000002e-05 2024-09-11T09:52:22.792 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.793 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.793 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:22.793 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.914522+0000", 2024-09-11T09:52:22.793 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.7127e-05 2024-09-11T09:52:22.793 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.793 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.793 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:22.793 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.914583+0000", 2024-09-11T09:52:22.793 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.0730999999999998e-05 2024-09-11T09:52:22.793 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.793 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.793 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:22.793 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.914709+0000", 2024-09-11T09:52:22.793 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00012590400000000001 2024-09-11T09:52:22.793 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.794 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.794 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 1", 2024-09-11T09:52:22.794 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.914809+0000", 2024-09-11T09:52:22.794 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.9882000000000004e-05 2024-09-11T09:52:22.794 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.794 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.794 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:22.794 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.932551+0000", 2024-09-11T09:52:22.794 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.017742048999999999 2024-09-11T09:52:22.794 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.794 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.794 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:52:22.794 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.935144+0000", 2024-09-11T09:52:22.794 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0025927630000000001 2024-09-11T09:52:22.795 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.795 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.795 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:22.795 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.935205+0000", 2024-09-11T09:52:22.795 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.1634e-05 2024-09-11T09:52:22.795 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.795 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.795 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:22.795 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.935262+0000", 2024-09-11T09:52:22.795 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.6817000000000002e-05 2024-09-11T09:52:22.795 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:22.795 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:22.795 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:22.795 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.795 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.795 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_repop(client.4688.0:40 2.7 e53/38 2:fe41411c:::rbd_object_map.1250f5e25b80:head v 53'9, mlcod=53'7)", 2024-09-11T09:52:22.796 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:22.920457+0000", 2024-09-11T09:52:22.796 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 59.854907038, 2024-09-11T09:52:22.796 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.011132817, 2024-09-11T09:52:22.796 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:22.796 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:22.796 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:22.796 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:22.796 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.796 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:22.796 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.920457+0000", 2024-09-11T09:52:22.796 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:22.796 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.796 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.796 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:22.796 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.920457+0000", 2024-09-11T09:52:22.797 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:22.797 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.797 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.797 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:22.797 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.920459+0000", 2024-09-11T09:52:22.797 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.6729999999999999e-06 2024-09-11T09:52:22.797 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.797 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.797 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:22.797 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.920469+0000", 2024-09-11T09:52:22.797 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.944e-06 2024-09-11T09:52:22.797 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.797 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.797 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:22.797 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.920495+0000", 2024-09-11T09:52:22.798 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.6401999999999999e-05 2024-09-11T09:52:22.798 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.798 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.798 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:22.798 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.920506+0000", 2024-09-11T09:52:22.798 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.133e-05 2024-09-11T09:52:22.798 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.798 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.798 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:22.798 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.920581+0000", 2024-09-11T09:52:22.798 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.4932999999999996e-05 2024-09-11T09:52:22.798 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.798 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.798 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:22.798 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.920612+0000", 2024-09-11T09:52:22.799 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.0537999999999998e-05 2024-09-11T09:52:22.799 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.799 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.799 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:22.799 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.931537+0000", 2024-09-11T09:52:22.799 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.010925239 2024-09-11T09:52:22.799 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.799 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.799 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:22.799 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.931590+0000", 2024-09-11T09:52:22.799 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.2757999999999999e-05 2024-09-11T09:52:22.799 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:22.799 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:22.799 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:22.799 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.799 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.800 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_repop(client.4688.0:41 2.7 e53/38 2:fe41411c:::rbd_object_map.1250f5e25b80:head v 53'10, mlcod=53'7)", 2024-09-11T09:52:22.800 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:22.921422+0000", 2024-09-11T09:52:22.800 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 59.853941622000001, 2024-09-11T09:52:22.800 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.013734942, 2024-09-11T09:52:22.800 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:22.800 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:22.800 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:22.800 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:22.800 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.800 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:22.800 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921422+0000", 2024-09-11T09:52:22.800 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:22.800 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.800 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.800 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:22.801 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921422+0000", 2024-09-11T09:52:22.801 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:22.801 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.801 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.801 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:22.801 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921426+0000", 2024-09-11T09:52:22.801 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.6150000000000001e-06 2024-09-11T09:52:22.801 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.801 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.801 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:22.801 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921435+0000", 2024-09-11T09:52:22.801 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.9840000000000007e-06 2024-09-11T09:52:22.801 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.801 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.801 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:22.802 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921444+0000", 2024-09-11T09:52:22.802 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.6000000000000007e-06 2024-09-11T09:52:22.802 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.802 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.802 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:22.802 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921450+0000", 2024-09-11T09:52:22.802 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.4690000000000002e-06 2024-09-11T09:52:22.802 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.802 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.802 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:22.802 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921511+0000", 2024-09-11T09:52:22.802 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.0779999999999997e-05 2024-09-11T09:52:22.802 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.802 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.802 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:22.803 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921531+0000", 2024-09-11T09:52:22.803 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.0302999999999999e-05 2024-09-11T09:52:22.803 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.803 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.803 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:22.803 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.935115+0000", 2024-09-11T09:52:22.803 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.013583471 2024-09-11T09:52:22.803 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.803 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.803 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:22.803 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.935157+0000", 2024-09-11T09:52:22.803 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.2719999999999998e-05 2024-09-11T09:52:22.803 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:22.803 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:22.803 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:22.803 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.804 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.804 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4688.0:42 2.3 2:c54742d8:::rbd_data.1250f5e25b80.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:22.804 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:22.932560+0000", 2024-09-11T09:52:22.804 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 59.842803879000002, 2024-09-11T09:52:22.804 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.017798198000000001, 2024-09-11T09:52:22.804 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:22.804 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:22.804 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:22.804 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:52:22.804 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4688", 2024-09-11T09:52:22.804 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:52:22.804 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 42 2024-09-11T09:52:22.804 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.804 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:22.804 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.805 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:22.805 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.932560+0000", 2024-09-11T09:52:22.805 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:22.805 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.805 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.805 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:22.805 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.932560+0000", 2024-09-11T09:52:22.805 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:22.805 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.805 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.805 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:22.805 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.932562+0000", 2024-09-11T09:52:22.805 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.17e-06 2024-09-11T09:52:22.805 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.806 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.806 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:22.806 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.938851+0000", 2024-09-11T09:52:22.806 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.006288971 2024-09-11T09:52:22.806 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.806 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.806 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:22.806 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.938865+0000", 2024-09-11T09:52:22.806 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.365e-05 2024-09-11T09:52:22.806 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.806 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.806 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:22.806 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.938871+0000", 2024-09-11T09:52:22.806 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.8850000000000004e-06 2024-09-11T09:52:22.806 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.807 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.807 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:22.807 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.938926+0000", 2024-09-11T09:52:22.807 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.4691999999999997e-05 2024-09-11T09:52:22.807 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.807 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.807 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:22.807 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.939059+0000", 2024-09-11T09:52:22.807 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00013310899999999999 2024-09-11T09:52:22.807 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.807 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.807 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 2", 2024-09-11T09:52:22.807 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.939171+0000", 2024-09-11T09:52:22.807 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000112718 2024-09-11T09:52:22.807 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.808 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.808 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:22.808 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.950262+0000", 2024-09-11T09:52:22.808 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.011091063 2024-09-11T09:52:22.808 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.808 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.808 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:52:22.808 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.950274+0000", 2024-09-11T09:52:22.808 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1963e-05 2024-09-11T09:52:22.808 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.808 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.808 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:22.809 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.950318+0000", 2024-09-11T09:52:22.809 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.3519999999999997e-05 2024-09-11T09:52:22.809 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.809 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.809 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:22.809 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.950358+0000", 2024-09-11T09:52:22.809 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.0457000000000002e-05 2024-09-11T09:52:22.809 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:22.809 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:22.809 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:22.809 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.809 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.810 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_repop(client.4694.0:34 2.7 e53/38 2:f80bfc0c:ns::rbd_data.1256a0c02879.0000000000000002:head v 53'25, mlcod=53'23)", 2024-09-11T09:52:22.810 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.094944+0000", 2024-09-11T09:52:22.810 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 59.680420486999999, 2024-09-11T09:52:22.810 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.010896785000000001, 2024-09-11T09:52:22.810 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:22.810 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:22.810 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:22.810 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:22.810 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.810 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:22.810 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.094944+0000", 2024-09-11T09:52:22.810 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:22.810 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.810 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.811 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:22.811 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.094944+0000", 2024-09-11T09:52:22.811 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:22.811 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.811 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.811 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:22.811 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.094944+0000", 2024-09-11T09:52:22.811 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.0699999999999996e-07 2024-09-11T09:52:22.811 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.811 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.811 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:22.811 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.099060+0000", 2024-09-11T09:52:22.811 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0041155580000000001 2024-09-11T09:52:22.811 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.812 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.812 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:22.812 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.099082+0000", 2024-09-11T09:52:22.812 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.2323999999999998e-05 2024-09-11T09:52:22.812 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.812 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.812 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:22.812 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.099094+0000", 2024-09-11T09:52:22.812 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1535000000000001e-05 2024-09-11T09:52:22.812 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.812 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.812 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:22.812 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.099160+0000", 2024-09-11T09:52:22.812 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.6449e-05 2024-09-11T09:52:22.813 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.813 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.813 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:22.813 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.099183+0000", 2024-09-11T09:52:22.813 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.3108e-05 2024-09-11T09:52:22.813 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.813 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.813 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:22.813 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.105776+0000", 2024-09-11T09:52:22.813 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0065922760000000002 2024-09-11T09:52:22.813 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.813 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.813 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:22.813 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.105840+0000", 2024-09-11T09:52:22.813 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.4727999999999994e-05 2024-09-11T09:52:22.814 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:22.814 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:22.814 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:22.814 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.814 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.814 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_repop(client.4694.0:35 2.7 e53/38 2:e8766325:ns::rbd_object_map.1256a0c02879:head v 53'26, mlcod=53'23)", 2024-09-11T09:52:22.814 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.100964+0000", 2024-09-11T09:52:22.814 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 59.674399958999999, 2024-09-11T09:52:22.814 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0049649359999999997, 2024-09-11T09:52:22.814 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:22.814 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:22.814 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:22.814 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:22.814 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.814 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:22.815 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.100964+0000", 2024-09-11T09:52:22.815 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:22.815 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.815 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.815 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:22.815 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.100964+0000", 2024-09-11T09:52:22.815 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:22.815 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.815 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.815 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:22.815 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.100966+0000", 2024-09-11T09:52:22.815 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.6330000000000001e-06 2024-09-11T09:52:22.815 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.815 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.815 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:22.816 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.100979+0000", 2024-09-11T09:52:22.816 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.2813000000000001e-05 2024-09-11T09:52:22.816 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.816 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.816 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:22.816 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.100992+0000", 2024-09-11T09:52:22.816 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.3414999999999999e-05 2024-09-11T09:52:22.816 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.816 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.816 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:22.816 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.101002+0000", 2024-09-11T09:52:22.816 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.0159e-05 2024-09-11T09:52:22.816 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.816 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.816 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:22.817 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.102365+0000", 2024-09-11T09:52:22.817 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001362912 2024-09-11T09:52:22.817 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.817 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.817 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:22.817 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.102382+0000", 2024-09-11T09:52:22.817 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.7433000000000001e-05 2024-09-11T09:52:22.817 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.817 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.817 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:22.817 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.105901+0000", 2024-09-11T09:52:22.817 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.003518476 2024-09-11T09:52:22.817 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.817 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.818 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:22.818 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.105929+0000", 2024-09-11T09:52:22.818 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.8095000000000001e-05 2024-09-11T09:52:22.818 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:22.818 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:22.818 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:22.818 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.818 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.818 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4694.0:41 2.2 2:5e2505c4:ns::rbd_data.1256a0c02879.0000000000000003:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:22.818 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.107511+0000", 2024-09-11T09:52:22.818 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 59.667852805999999, 2024-09-11T09:52:22.818 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.018571066000000001, 2024-09-11T09:52:22.818 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:22.818 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:22.818 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:22.818 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:52:22.819 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4694", 2024-09-11T09:52:22.819 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:52:22.819 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 41 2024-09-11T09:52:22.819 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.819 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:22.819 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.819 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:22.819 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.107511+0000", 2024-09-11T09:52:22.819 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:22.819 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.819 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.819 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:22.819 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.107511+0000", 2024-09-11T09:52:22.819 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:22.819 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.820 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.820 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:22.820 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.107513+0000", 2024-09-11T09:52:22.820 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.3489999999999999e-06 2024-09-11T09:52:22.820 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.820 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.820 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:22.820 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.112704+0000", 2024-09-11T09:52:22.820 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0051918509999999999 2024-09-11T09:52:22.820 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.820 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.820 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:22.820 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.112718+0000", 2024-09-11T09:52:22.820 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.359e-05 2024-09-11T09:52:22.820 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.821 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.821 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:22.821 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.112725+0000", 2024-09-11T09:52:22.821 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.9940000000000003e-06 2024-09-11T09:52:22.821 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.821 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.821 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:22.821 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.112788+0000", 2024-09-11T09:52:22.821 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.2447999999999996e-05 2024-09-11T09:52:22.821 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.821 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.821 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:22.821 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.112937+0000", 2024-09-11T09:52:22.821 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00014921499999999999 2024-09-11T09:52:22.822 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.822 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.822 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 1", 2024-09-11T09:52:22.822 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.113085+0000", 2024-09-11T09:52:22.822 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000148085 2024-09-11T09:52:22.822 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.822 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.822 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:52:22.822 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.124318+0000", 2024-09-11T09:52:22.822 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.011233158 2024-09-11T09:52:22.822 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.822 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.822 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:22.822 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.125957+0000", 2024-09-11T09:52:22.822 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001638701 2024-09-11T09:52:22.822 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.823 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.823 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:22.823 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.126029+0000", 2024-09-11T09:52:22.823 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.2125000000000005e-05 2024-09-11T09:52:22.823 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.823 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.823 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:22.823 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.126082+0000", 2024-09-11T09:52:22.823 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.3550000000000001e-05 2024-09-11T09:52:22.823 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:22.823 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:22.823 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:22.823 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.823 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.823 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4694.0:44 2.3 2:cd1a7d1c:ns::rbd_data.1256a0c02879.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:22.824 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.112744+0000", 2024-09-11T09:52:22.824 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 59.662620238999999, 2024-09-11T09:52:22.824 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.021672212999999999, 2024-09-11T09:52:22.824 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:22.824 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:22.824 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:22.824 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:52:22.824 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4694", 2024-09-11T09:52:22.824 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:52:22.824 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 44 2024-09-11T09:52:22.824 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.824 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:22.824 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.824 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:22.824 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.112744+0000", 2024-09-11T09:52:22.825 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:22.825 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.825 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.825 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:22.825 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.112744+0000", 2024-09-11T09:52:22.825 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:22.825 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.825 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.825 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:22.825 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.112745+0000", 2024-09-11T09:52:22.825 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.2900000000000002e-07 2024-09-11T09:52:22.825 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.825 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.825 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:22.825 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.119929+0000", 2024-09-11T09:52:22.826 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0071846230000000002 2024-09-11T09:52:22.826 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.826 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.826 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:22.826 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.119949+0000", 2024-09-11T09:52:22.826 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.9264000000000001e-05 2024-09-11T09:52:22.826 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.826 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.826 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:22.826 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.119959+0000", 2024-09-11T09:52:22.826 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.0465999999999999e-05 2024-09-11T09:52:22.826 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.826 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.826 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:22.826 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.120028+0000", 2024-09-11T09:52:22.827 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.9318000000000003e-05 2024-09-11T09:52:22.827 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.827 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.827 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:22.827 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.120160+0000", 2024-09-11T09:52:22.827 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000132015 2024-09-11T09:52:22.827 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.827 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.827 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 2", 2024-09-11T09:52:22.827 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.120260+0000", 2024-09-11T09:52:22.827 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.9397999999999999e-05 2024-09-11T09:52:22.827 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.827 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.827 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:52:22.827 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.134246+0000", 2024-09-11T09:52:22.827 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.013986721000000001 2024-09-11T09:52:22.828 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.828 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.828 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:22.828 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.134295+0000", 2024-09-11T09:52:22.828 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.8649000000000001e-05 2024-09-11T09:52:22.828 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.828 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.828 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:22.828 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.134361+0000", 2024-09-11T09:52:22.828 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.6081999999999995e-05 2024-09-11T09:52:22.828 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.828 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.828 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:22.828 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.134416+0000", 2024-09-11T09:52:22.829 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.4848000000000001e-05 2024-09-11T09:52:22.829 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:22.829 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:22.829 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:22.829 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.829 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.829 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4694.0:46 2.3 2:ce0e1469:ns::rbd_data.1256a0c02879.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:22.829 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.119993+0000", 2024-09-11T09:52:22.829 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 59.655370763000001, 2024-09-11T09:52:22.829 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.025840610999999999, 2024-09-11T09:52:22.829 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:22.829 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:22.829 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:22.829 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:52:22.830 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4694", 2024-09-11T09:52:22.830 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:52:22.830 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 46 2024-09-11T09:52:22.830 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.830 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:22.830 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.830 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:22.830 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.119993+0000", 2024-09-11T09:52:22.830 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:22.830 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.830 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.830 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:22.830 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.119993+0000", 2024-09-11T09:52:22.830 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:22.831 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.831 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.831 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:22.831 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.119995+0000", 2024-09-11T09:52:22.831 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1930000000000001e-06 2024-09-11T09:52:22.831 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.831 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.831 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:22.831 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.128458+0000", 2024-09-11T09:52:22.831 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0084632630000000004 2024-09-11T09:52:22.831 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.831 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.831 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:22.831 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.128478+0000", 2024-09-11T09:52:22.832 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.0353e-05 2024-09-11T09:52:22.832 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.832 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.832 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:22.832 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.128491+0000", 2024-09-11T09:52:22.832 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.2954e-05 2024-09-11T09:52:22.832 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.832 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.832 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:22.832 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.128558+0000", 2024-09-11T09:52:22.832 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.7076999999999995e-05 2024-09-11T09:52:22.832 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.832 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.832 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:22.832 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.128708+0000", 2024-09-11T09:52:22.833 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00014939599999999999 2024-09-11T09:52:22.833 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.833 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.833 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 2", 2024-09-11T09:52:22.833 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.128866+0000", 2024-09-11T09:52:22.833 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000158145 2024-09-11T09:52:22.833 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.833 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.833 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:22.833 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.141045+0000", 2024-09-11T09:52:22.833 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.012179331999999999 2024-09-11T09:52:22.833 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.833 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.834 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:52:22.834 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.145742+0000", 2024-09-11T09:52:22.834 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0046972050000000003 2024-09-11T09:52:22.834 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.834 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.834 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:22.834 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.145779+0000", 2024-09-11T09:52:22.834 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.6544000000000001e-05 2024-09-11T09:52:22.834 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.834 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.834 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:22.834 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.145834+0000", 2024-09-11T09:52:22.834 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.5148999999999997e-05 2024-09-11T09:52:22.834 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:22.835 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:22.835 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:22.835 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.835 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.835 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4694.0:48 2.3 2:d2090b70:ns::rbd_data.1256a0c02879.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:22.835 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.130540+0000", 2024-09-11T09:52:22.835 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 59.644824497000002, 2024-09-11T09:52:22.835 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.017167789999999999, 2024-09-11T09:52:22.835 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:22.835 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:22.835 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:22.835 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:52:22.835 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4694", 2024-09-11T09:52:22.835 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:52:22.836 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 48 2024-09-11T09:52:22.836 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.836 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:22.836 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.836 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:22.836 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.130540+0000", 2024-09-11T09:52:22.836 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:22.836 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.836 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.836 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:22.836 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.130540+0000", 2024-09-11T09:52:22.836 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:22.836 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.836 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.836 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:22.837 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.130542+0000", 2024-09-11T09:52:22.837 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.9209999999999999e-06 2024-09-11T09:52:22.837 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.837 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.837 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:22.837 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.134691+0000", 2024-09-11T09:52:22.837 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0041498739999999996 2024-09-11T09:52:22.837 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.837 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.837 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:22.837 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.134705+0000", 2024-09-11T09:52:22.837 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.3669000000000001e-05 2024-09-11T09:52:22.837 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.838 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.838 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:22.838 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.134712+0000", 2024-09-11T09:52:22.838 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.8410000000000002e-06 2024-09-11T09:52:22.838 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.838 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.838 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:22.838 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.134754+0000", 2024-09-11T09:52:22.838 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.2524999999999997e-05 2024-09-11T09:52:22.838 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.838 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.838 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:22.838 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.134875+0000", 2024-09-11T09:52:22.838 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000120332 2024-09-11T09:52:22.839 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.839 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.839 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 2", 2024-09-11T09:52:22.839 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.135005+0000", 2024-09-11T09:52:22.839 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00013024900000000001 2024-09-11T09:52:22.839 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.839 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.839 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:22.839 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.145971+0000", 2024-09-11T09:52:22.839 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.010966203000000001 2024-09-11T09:52:22.839 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.839 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.839 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:52:22.839 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.147600+0000", 2024-09-11T09:52:22.839 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001628382 2024-09-11T09:52:22.840 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.840 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.840 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:22.840 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.147635+0000", 2024-09-11T09:52:22.840 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.5420000000000003e-05 2024-09-11T09:52:22.840 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.840 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.840 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:22.840 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.147707+0000", 2024-09-11T09:52:22.840 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.2373999999999995e-05 2024-09-11T09:52:22.840 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:22.840 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:22.840 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:22.840 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.841 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.841 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_repop(client.4694.0:49 2.7 e53/38 2:e8766325:ns::rbd_object_map.1256a0c02879:head v 53'33, mlcod=53'31)", 2024-09-11T09:52:22.841 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.135391+0000", 2024-09-11T09:52:22.841 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 59.639972931999999, 2024-09-11T09:52:22.841 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0051189859999999998, 2024-09-11T09:52:22.841 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:22.841 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:22.841 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:22.841 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:22.841 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.841 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:22.841 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.135391+0000", 2024-09-11T09:52:22.841 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:22.841 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.842 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.842 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:22.842 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.135391+0000", 2024-09-11T09:52:22.842 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:22.842 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.842 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.842 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:22.842 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.135393+0000", 2024-09-11T09:52:22.842 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.4020000000000001e-06 2024-09-11T09:52:22.842 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.842 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.842 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:22.842 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.135400+0000", 2024-09-11T09:52:22.843 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.5129999999999999e-06 2024-09-11T09:52:22.843 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.843 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.843 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:22.843 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.135411+0000", 2024-09-11T09:52:22.843 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1038e-05 2024-09-11T09:52:22.843 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.843 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.843 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:22.843 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.135418+0000", 2024-09-11T09:52:22.843 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.1489999999999997e-06 2024-09-11T09:52:22.843 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.843 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.843 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:22.843 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.135470+0000", 2024-09-11T09:52:22.844 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.2021999999999998e-05 2024-09-11T09:52:22.844 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.844 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.844 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:22.844 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.135501+0000", 2024-09-11T09:52:22.844 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.0349999999999999e-05 2024-09-11T09:52:22.844 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.844 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.844 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:22.844 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.140453+0000", 2024-09-11T09:52:22.844 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0049522409999999996 2024-09-11T09:52:22.844 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.844 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.844 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:22.845 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.140510+0000", 2024-09-11T09:52:22.845 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.7271000000000001e-05 2024-09-11T09:52:22.845 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:22.845 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:22.845 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:22.845 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.845 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.845 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4694.0:54 2.3 2:c44f0253:ns::rbd_data.1256a0c02879.000000000000000b:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:22.845 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.150151+0000", 2024-09-11T09:52:22.845 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 59.625213080999998, 2024-09-11T09:52:22.845 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.015103484, 2024-09-11T09:52:22.845 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:22.845 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:22.845 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:22.846 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:52:22.846 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4694", 2024-09-11T09:52:22.846 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:52:22.846 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 54 2024-09-11T09:52:22.846 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.846 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:22.846 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.846 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:22.846 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.150151+0000", 2024-09-11T09:52:22.846 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:22.846 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.846 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.846 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:22.847 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.150151+0000", 2024-09-11T09:52:22.847 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:22.847 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.847 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.847 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:22.847 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.150152+0000", 2024-09-11T09:52:22.847 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.2550000000000001e-06 2024-09-11T09:52:22.847 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.847 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.847 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:22.847 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.154528+0000", 2024-09-11T09:52:22.847 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0043758219999999997 2024-09-11T09:52:22.847 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.848 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.848 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:22.848 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.154541+0000", 2024-09-11T09:52:22.848 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.3414999999999999e-05 2024-09-11T09:52:22.848 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.848 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.848 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:22.848 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.154551+0000", 2024-09-11T09:52:22.848 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.4350000000000003e-06 2024-09-11T09:52:22.848 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.848 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.848 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:22.848 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.154615+0000", 2024-09-11T09:52:22.849 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.3627000000000006e-05 2024-09-11T09:52:22.849 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.849 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.849 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:22.849 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.154717+0000", 2024-09-11T09:52:22.849 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000102729 2024-09-11T09:52:22.849 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.849 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.849 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 2", 2024-09-11T09:52:22.849 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.154826+0000", 2024-09-11T09:52:22.849 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000108409 2024-09-11T09:52:22.849 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.849 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.849 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:52:22.850 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.165094+0000", 2024-09-11T09:52:22.850 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.010268210999999999 2024-09-11T09:52:22.850 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.850 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.850 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:22.850 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.165138+0000", 2024-09-11T09:52:22.850 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.4078e-05 2024-09-11T09:52:22.850 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.850 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.850 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:22.850 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.165204+0000", 2024-09-11T09:52:22.850 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.5716000000000006e-05 2024-09-11T09:52:22.850 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.850 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.851 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:22.851 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.165254+0000", 2024-09-11T09:52:22.851 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.0787e-05 2024-09-11T09:52:22.851 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:22.851 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:22.851 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:22.851 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.851 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.851 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_repop(client.4694.0:55 2.7 e53/38 2:e8766325:ns::rbd_object_map.1256a0c02879:head v 53'37, mlcod=53'34)", 2024-09-11T09:52:22.851 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.150221+0000", 2024-09-11T09:52:22.851 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 59.625143215999998, 2024-09-11T09:52:22.851 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00530015, 2024-09-11T09:52:22.851 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:22.851 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:22.852 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:22.852 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:22.852 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.852 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:22.852 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.150221+0000", 2024-09-11T09:52:22.852 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:22.852 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.852 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.852 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:22.852 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.150221+0000", 2024-09-11T09:52:22.852 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:22.852 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.852 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.852 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:22.852 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.150221+0000", 2024-09-11T09:52:22.853 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.9100000000000004e-07 2024-09-11T09:52:22.853 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.853 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.853 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:22.853 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.150226+0000", 2024-09-11T09:52:22.853 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.2250000000000002e-06 2024-09-11T09:52:22.853 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.853 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.853 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:22.853 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.150251+0000", 2024-09-11T09:52:22.853 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.4987e-05 2024-09-11T09:52:22.853 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.853 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.853 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:22.854 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.150258+0000", 2024-09-11T09:52:22.854 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.0790000000000003e-06 2024-09-11T09:52:22.854 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.854 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.854 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:22.854 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.150305+0000", 2024-09-11T09:52:22.854 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.6879000000000003e-05 2024-09-11T09:52:22.854 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.854 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.854 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:22.854 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.150320+0000", 2024-09-11T09:52:22.854 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.5268999999999999e-05 2024-09-11T09:52:22.855 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.855 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.855 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:22.855 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.155493+0000", 2024-09-11T09:52:22.855 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0051731390000000002 2024-09-11T09:52:22.855 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.855 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.855 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:22.855 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.155521+0000", 2024-09-11T09:52:22.855 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.8081000000000001e-05 2024-09-11T09:52:22.855 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:22.855 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:22.855 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:22.855 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.856 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.856 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4694.0:56 2.3 2:c462f029:ns::rbd_data.1256a0c02879.000000000000000c:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:22.856 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.154576+0000", 2024-09-11T09:52:22.856 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 59.620788240000003, 2024-09-11T09:52:22.856 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.020781621, 2024-09-11T09:52:22.856 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:22.856 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:22.856 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:22.856 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:52:22.856 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4694", 2024-09-11T09:52:22.856 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:52:22.856 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 56 2024-09-11T09:52:22.856 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.857 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:22.857 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.857 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:22.857 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.154576+0000", 2024-09-11T09:52:22.857 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:22.857 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.857 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.857 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:22.857 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.154576+0000", 2024-09-11T09:52:22.857 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:22.857 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.857 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.857 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:22.857 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.154577+0000", 2024-09-11T09:52:22.858 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.2249999999999999e-06 2024-09-11T09:52:22.858 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.858 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.858 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:22.858 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.158783+0000", 2024-09-11T09:52:22.858 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0042063889999999996 2024-09-11T09:52:22.858 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.858 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.858 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:22.858 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.158803+0000", 2024-09-11T09:52:22.858 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.9576999999999999e-05 2024-09-11T09:52:22.858 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.858 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.859 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:22.859 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.158815+0000", 2024-09-11T09:52:22.859 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.2221999999999999e-05 2024-09-11T09:52:22.859 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.859 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.859 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:22.859 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.158902+0000", 2024-09-11T09:52:22.859 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.6390000000000005e-05 2024-09-11T09:52:22.859 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.859 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.859 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:22.859 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.159141+0000", 2024-09-11T09:52:22.859 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000238886 2024-09-11T09:52:22.860 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.860 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.860 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 2", 2024-09-11T09:52:22.860 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.159279+0000", 2024-09-11T09:52:22.860 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000138073 2024-09-11T09:52:22.860 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.860 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.860 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:52:22.860 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.173297+0000", 2024-09-11T09:52:22.860 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.014018113 2024-09-11T09:52:22.860 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.860 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.861 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:22.861 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.175278+0000", 2024-09-11T09:52:22.861 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0019817390000000002 2024-09-11T09:52:22.861 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.861 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.861 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:22.861 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.175316+0000", 2024-09-11T09:52:22.861 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.7829e-05 2024-09-11T09:52:22.861 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.861 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.861 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:22.861 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.175357+0000", 2024-09-11T09:52:22.861 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.1177999999999998e-05 2024-09-11T09:52:22.862 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:22.862 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:22.862 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:22.862 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.862 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.862 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4694.0:59 2.2 2:41adefc9:ns::rbd_data.1256a0c02879.000000000000000e:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:22.862 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.158843+0000", 2024-09-11T09:52:22.862 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 59.616520674, 2024-09-11T09:52:22.862 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.017874207, 2024-09-11T09:52:22.862 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:22.862 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:22.862 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:22.862 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:52:22.863 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4694", 2024-09-11T09:52:22.863 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:52:22.863 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 59 2024-09-11T09:52:22.863 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.863 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:22.863 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.863 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:22.863 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.158843+0000", 2024-09-11T09:52:22.863 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:22.863 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.863 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.863 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:22.863 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.158843+0000", 2024-09-11T09:52:22.863 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:22.863 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.864 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.864 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:22.864 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.158845+0000", 2024-09-11T09:52:22.864 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.7439999999999999e-06 2024-09-11T09:52:22.864 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.864 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.864 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:22.864 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.162713+0000", 2024-09-11T09:52:22.864 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0038678829999999999 2024-09-11T09:52:22.864 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.864 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.864 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:22.864 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.162728+0000", 2024-09-11T09:52:22.865 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.5114e-05 2024-09-11T09:52:22.865 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.865 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.865 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:22.865 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.162737+0000", 2024-09-11T09:52:22.865 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.1099999999999992e-06 2024-09-11T09:52:22.865 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.865 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.865 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:22.865 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.162799+0000", 2024-09-11T09:52:22.865 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.1934999999999996e-05 2024-09-11T09:52:22.865 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.865 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.866 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:22.866 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.162911+0000", 2024-09-11T09:52:22.866 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000111867 2024-09-11T09:52:22.866 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.866 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.866 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 1", 2024-09-11T09:52:22.866 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.163030+0000", 2024-09-11T09:52:22.866 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000119108 2024-09-11T09:52:22.866 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.866 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.866 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:52:22.866 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.174086+0000", 2024-09-11T09:52:22.866 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.01105543 2024-09-11T09:52:22.867 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.867 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.867 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:22.867 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.176614+0000", 2024-09-11T09:52:22.867 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0025284940000000001 2024-09-11T09:52:22.867 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.867 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.867 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:22.867 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.176672+0000", 2024-09-11T09:52:22.867 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.7892000000000001e-05 2024-09-11T09:52:22.867 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.867 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.867 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:22.868 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.176718+0000", 2024-09-11T09:52:22.868 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.5630000000000002e-05 2024-09-11T09:52:22.868 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:22.868 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:22.868 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:22.868 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.868 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.868 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.5002.0:89 2.3 2:c54742d8:::rbd_data.1250f5e25b80.0000000000000006:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-09-11T09:52:22.868 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:38.546016+0000", 2024-09-11T09:52:22.868 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 44.229348127999998, 2024-09-11T09:52:22.868 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0049098550000000003, 2024-09-11T09:52:22.868 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:22.868 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:22.868 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:22.869 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:52:22.869 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.5002", 2024-09-11T09:52:22.869 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/2204440692", 2024-09-11T09:52:22.869 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 89 2024-09-11T09:52:22.869 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.869 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:22.869 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.869 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:22.869 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.546016+0000", 2024-09-11T09:52:22.869 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:22.869 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.869 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.869 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:22.869 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.546016+0000", 2024-09-11T09:52:22.870 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:22.870 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.870 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.870 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:22.870 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.546017+0000", 2024-09-11T09:52:22.870 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.3399999999999998e-07 2024-09-11T09:52:22.870 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.870 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.870 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:22.870 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.546839+0000", 2024-09-11T09:52:22.870 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00082222800000000004 2024-09-11T09:52:22.870 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.870 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.870 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:22.871 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.546851+0000", 2024-09-11T09:52:22.871 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.2075999999999999e-05 2024-09-11T09:52:22.871 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.871 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.871 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:22.871 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.546856+0000", 2024-09-11T09:52:22.871 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.4009999999999996e-06 2024-09-11T09:52:22.871 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.871 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.871 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:22.871 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.546897+0000", 2024-09-11T09:52:22.871 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.0113e-05 2024-09-11T09:52:22.871 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.871 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.872 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:22.872 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.546954+0000", 2024-09-11T09:52:22.872 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.7825000000000003e-05 2024-09-11T09:52:22.872 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.872 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.872 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 2", 2024-09-11T09:52:22.872 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.547055+0000", 2024-09-11T09:52:22.872 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000100269 2024-09-11T09:52:22.872 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.872 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.872 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:22.872 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.549675+0000", 2024-09-11T09:52:22.872 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0026206609999999998 2024-09-11T09:52:22.873 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.873 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.873 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:52:22.873 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.550858+0000", 2024-09-11T09:52:22.873 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0011825270000000001 2024-09-11T09:52:22.873 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.873 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.873 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:22.873 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.550894+0000", 2024-09-11T09:52:22.873 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.5896999999999999e-05 2024-09-11T09:52:22.873 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.873 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.873 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:22.874 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.550926+0000", 2024-09-11T09:52:22.874 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.2023999999999997e-05 2024-09-11T09:52:22.874 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:22.874 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:22.874 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:22.874 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.874 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.874 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.5002.0:91 2.3 2:c54742d8:::rbd_data.1250f5e25b80.0000000000000006:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-09-11T09:52:22.874 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:38.553743+0000", 2024-09-11T09:52:22.874 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 44.221621085000002, 2024-09-11T09:52:22.874 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0051633629999999998, 2024-09-11T09:52:22.874 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:22.874 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:22.874 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:22.875 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:52:22.875 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.5002", 2024-09-11T09:52:22.875 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/2204440692", 2024-09-11T09:52:22.875 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 91 2024-09-11T09:52:22.875 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.875 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:22.875 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.875 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:22.875 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.553743+0000", 2024-09-11T09:52:22.875 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:22.875 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.875 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.875 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:22.875 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.553743+0000", 2024-09-11T09:52:22.876 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:22.876 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.876 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.876 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:22.876 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.553745+0000", 2024-09-11T09:52:22.876 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.5120000000000001e-06 2024-09-11T09:52:22.876 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.876 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.876 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:22.876 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.554588+0000", 2024-09-11T09:52:22.876 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00084352299999999999 2024-09-11T09:52:22.876 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.876 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.877 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:22.877 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.554604+0000", 2024-09-11T09:52:22.877 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.643e-05 2024-09-11T09:52:22.877 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.877 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.877 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:22.877 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.554613+0000", 2024-09-11T09:52:22.877 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.5669999999999995e-06 2024-09-11T09:52:22.877 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.877 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.877 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:22.877 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.554662+0000", 2024-09-11T09:52:22.877 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.8510000000000001e-05 2024-09-11T09:52:22.877 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.878 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.878 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:22.878 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.554732+0000", 2024-09-11T09:52:22.878 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.0297999999999997e-05 2024-09-11T09:52:22.878 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.878 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.878 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 2", 2024-09-11T09:52:22.878 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.554870+0000", 2024-09-11T09:52:22.878 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00013856599999999999 2024-09-11T09:52:22.878 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.878 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.878 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:22.878 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.557875+0000", 2024-09-11T09:52:22.879 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0030042760000000002 2024-09-11T09:52:22.879 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.879 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.879 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:52:22.879 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.558835+0000", 2024-09-11T09:52:22.879 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00096057900000000003 2024-09-11T09:52:22.879 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.879 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.879 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:22.879 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.558874+0000", 2024-09-11T09:52:22.879 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.8992999999999997e-05 2024-09-11T09:52:22.879 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.879 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.879 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:22.879 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.558906+0000", 2024-09-11T09:52:22.880 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.2109e-05 2024-09-11T09:52:22.880 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:22.880 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:22.880 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:22.880 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.880 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.880 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.5002.0:93 2.3 2:c54742d8:::rbd_data.1250f5e25b80.0000000000000006:head [write 2228224~917504] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e61)", 2024-09-11T09:52:22.880 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:38.562067+0000", 2024-09-11T09:52:22.880 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 44.213297488999999, 2024-09-11T09:52:22.880 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0050551099999999998, 2024-09-11T09:52:22.880 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:22.880 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:22.880 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:22.881 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:52:22.881 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.5002", 2024-09-11T09:52:22.881 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/2204440692", 2024-09-11T09:52:22.881 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 93 2024-09-11T09:52:22.881 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.881 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:22.881 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.881 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:22.881 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.562067+0000", 2024-09-11T09:52:22.881 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:22.881 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.881 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.881 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:22.881 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.562067+0000", 2024-09-11T09:52:22.882 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:22.882 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.882 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.882 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:22.882 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.562068+0000", 2024-09-11T09:52:22.882 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.0669999999999999e-06 2024-09-11T09:52:22.882 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.882 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.882 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:22.882 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.563170+0000", 2024-09-11T09:52:22.882 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00110256 2024-09-11T09:52:22.882 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.882 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.882 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:22.883 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.563185+0000", 2024-09-11T09:52:22.883 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.4873e-05 2024-09-11T09:52:22.883 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.883 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.883 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:22.883 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.563193+0000", 2024-09-11T09:52:22.883 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.1319999999999994e-06 2024-09-11T09:52:22.883 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.883 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.883 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:22.883 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.563238+0000", 2024-09-11T09:52:22.883 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.4773e-05 2024-09-11T09:52:22.883 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.883 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.883 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:22.884 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.563290+0000", 2024-09-11T09:52:22.884 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.1690000000000001e-05 2024-09-11T09:52:22.884 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.884 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.884 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 2", 2024-09-11T09:52:22.884 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.563365+0000", 2024-09-11T09:52:22.884 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.5023000000000003e-05 2024-09-11T09:52:22.884 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.884 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.884 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:22.884 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.566552+0000", 2024-09-11T09:52:22.884 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0031869960000000001 2024-09-11T09:52:22.884 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.884 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.884 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:52:22.885 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.567048+0000", 2024-09-11T09:52:22.885 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00049605099999999996 2024-09-11T09:52:22.885 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.885 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.885 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:22.885 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.567095+0000", 2024-09-11T09:52:22.885 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.6823999999999998e-05 2024-09-11T09:52:22.885 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:22.885 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:22.885 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:22.885 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.567122+0000", 2024-09-11T09:52:22.885 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.7121e-05 2024-09-11T09:52:22.885 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:22.885 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:22.886 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:22.886 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:22.886 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:22.886 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-11T09:52:22.887 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-11T09:52:22.888 INFO:tasks.ceph.osd.0.smithi017.stderr:2024-09-11T09:52:22.888+0000 7fb73c0f1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31665) UID: 0 2024-09-11T09:52:22.892 INFO:teuthology.orchestra.run.smithi017.stderr:reweighted osd.4 to 1 (10000) 2024-09-11T09:52:22.907 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 5 1 2024-09-11T09:52:22.988 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:52:22.989+0000 7f3a7b90a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 42475) UID: 0 2024-09-11T09:52:22.993 INFO:tasks.workunit.client.0.smithi107.stderr:+ DEV= 2024-09-11T09:52:22.993 INFO:tasks.workunit.client.0.smithi107.stderr:+ rbd feature disable rbd/testrbdnbd27121 journaling 2024-09-11T09:52:23.032 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-11T09:52:23.032 INFO:teuthology.orchestra.run.smithi096.stdout: "ops": [], 2024-09-11T09:52:23.032 INFO:teuthology.orchestra.run.smithi096.stdout: "num_ops": 0 2024-09-11T09:52:23.032 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-11T09:52:23.041 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-11T09:52:23.058 INFO:tasks.workunit.client.0.smithi107.stderr:+ rbd config image rm rbd/testrbdnbd27121 rbd_discard_granularity_bytes 2024-09-11T09:52:23.123 INFO:tasks.workunit.client.0.smithi107.stderr:++ _sudo rbd device --device-type nbd map rbd/testrbdnbd27121 2024-09-11T09:52:23.123 INFO:tasks.workunit.client.0.smithi107.stderr:++ local cmd 2024-09-11T09:52:23.123 INFO:tasks.workunit.client.0.smithi107.stderr:+++ id -u 2024-09-11T09:52:23.123 INFO:tasks.workunit.client.0.smithi107.stderr:++ '[' 1000 -eq 0 ']' 2024-09-11T09:52:23.124 INFO:tasks.workunit.client.0.smithi107.stderr:+++ which rbd 2024-09-11T09:52:23.125 INFO:tasks.workunit.client.0.smithi107.stderr:++ cmd=/usr/bin/rbd 2024-09-11T09:52:23.126 INFO:tasks.workunit.client.0.smithi107.stderr:++ shift 2024-09-11T09:52:23.126 INFO:tasks.workunit.client.0.smithi107.stderr:++ sudo -nE /usr/bin/rbd device --device-type nbd map rbd/testrbdnbd27121 2024-09-11T09:52:23.126 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:52:23.127+0000 7f3c96ec9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31909) UID: 0 2024-09-11T09:52:23.184 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-11T09:52:23.185 INFO:teuthology.orchestra.run.smithi096.stdout: "ops": [], 2024-09-11T09:52:23.185 INFO:teuthology.orchestra.run.smithi096.stdout: "complaint_time": 30, 2024-09-11T09:52:23.185 INFO:teuthology.orchestra.run.smithi096.stdout: "num_blocked_ops": 0 2024-09-11T09:52:23.185 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-11T09:52:23.190 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:52:23.191+0000 7f3a7b90a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 42475) UID: 0 2024-09-11T09:52:23.193 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-11T09:52:23.232 INFO:tasks.workunit.client.0.smithi107.stderr:+ DEV=/dev/nbd2 2024-09-11T09:52:23.232 INFO:tasks.workunit.client.0.smithi107.stderr:+ get_pid rbd 2024-09-11T09:52:23.232 INFO:tasks.workunit.client.0.smithi107.stderr:+ local pool=rbd 2024-09-11T09:52:23.232 INFO:tasks.workunit.client.0.smithi107.stderr:+ local ns= 2024-09-11T09:52:23.232 INFO:tasks.workunit.client.0.smithi107.stderr:++ rbd device --device-type nbd --format xml list 2024-09-11T09:52:23.232 INFO:tasks.workunit.client.0.smithi107.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd27121'\''][device='\''/dev/nbd2'\'']/id' 2024-09-11T09:52:23.290 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-09-11T09:52:23.291+0000 7f7b6c7d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31904) UID: 0 2024-09-11T09:52:23.322 INFO:tasks.workunit.client.0.smithi107.stderr:+ PID=30702 2024-09-11T09:52:23.322 INFO:tasks.workunit.client.0.smithi107.stderr:+ test -n 30702 2024-09-11T09:52:23.322 INFO:tasks.workunit.client.0.smithi107.stderr:+ ps -p 30702 -C rbd-nbd 2024-09-11T09:52:23.333 INFO:tasks.workunit.client.0.smithi107.stdout: PID TTY TIME CMD 2024-09-11T09:52:23.333 INFO:tasks.workunit.client.0.smithi107.stdout: 30702 ? 00:00:00 rbd-nbd 2024-09-11T09:52:23.333 INFO:tasks.workunit.client.0.smithi107.stderr:+ rbd feature enable rbd/testrbdnbd27121 journaling 2024-09-11T09:52:23.339 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-11T09:52:23.339 INFO:teuthology.orchestra.run.smithi096.stdout: "size": 20, 2024-09-11T09:52:23.339 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 600, 2024-09-11T09:52:23.339 INFO:teuthology.orchestra.run.smithi096.stdout: "ops": [ 2024-09-11T09:52:23.339 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:23.339 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "MOSDPGPush(2.4 82/81 [PushOp(2:3bfc818c:ns::rbd_data.1256a0c02879.0000000000000001:head, version: 53'14, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:3bfc818c:ns::rbd_data.1256a0c02879.0000000000000001:head@53'14, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-11T09:52:23.339 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:52:08.947926+0000", 2024-09-11T09:52:23.339 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 14.39114854, 2024-09-11T09:52:23.339 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0020846179999999999, 2024-09-11T09:52:23.340 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:23.340 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:23.340 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "started", 2024-09-11T09:52:23.340 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:23.340 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:23.340 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:23.340 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.947926+0000", 2024-09-11T09:52:23.340 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:23.340 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:23.340 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:23.340 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:23.340 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.947926+0000", 2024-09-11T09:52:23.340 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:23.341 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:23.341 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:23.341 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:23.341 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.947927+0000", 2024-09-11T09:52:23.341 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.9599999999999999e-07 2024-09-11T09:52:23.341 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:23.341 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:23.341 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:23.341 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.949094+0000", 2024-09-11T09:52:23.341 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001167944 2024-09-11T09:52:23.341 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:23.341 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:23.342 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:23.342 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.949110+0000", 2024-09-11T09:52:23.342 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.5650000000000001e-05 2024-09-11T09:52:23.342 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:23.342 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:23.342 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:23.342 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.949138+0000", 2024-09-11T09:52:23.342 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.7515e-05 2024-09-11T09:52:23.342 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:23.342 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:23.342 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:23.342 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.949196+0000", 2024-09-11T09:52:23.342 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.8272e-05 2024-09-11T09:52:23.342 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:23.343 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:23.343 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:23.343 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.949204+0000", 2024-09-11T09:52:23.343 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.2150000000000001e-06 2024-09-11T09:52:23.343 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:23.343 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:23.343 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:23.343 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.950011+0000", 2024-09-11T09:52:23.343 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00080652600000000001 2024-09-11T09:52:23.343 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:23.343 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:23.343 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:23.344 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:23.344 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:23.344 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "MOSDPGPush(2.4 82/81 [PushOp(2:3bfc818c:ns::rbd_data.1256a0c02879.0000000000000001:head, version: 53'14, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:3bfc818c:ns::rbd_data.1256a0c02879.0000000000000001:head@53'14, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-11T09:52:23.344 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:52:08.951738+0000", 2024-09-11T09:52:23.344 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 14.387337022000001, 2024-09-11T09:52:23.344 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0018266020000000001, 2024-09-11T09:52:23.344 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:23.344 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:23.344 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "started", 2024-09-11T09:52:23.345 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:23.345 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:23.345 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:23.345 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.951738+0000", 2024-09-11T09:52:23.345 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:23.345 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:23.345 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:23.345 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:23.345 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.951738+0000", 2024-09-11T09:52:23.345 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:23.345 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:23.345 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:23.345 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:23.345 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.951739+0000", 2024-09-11T09:52:23.346 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.004e-06 2024-09-11T09:52:23.346 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:23.346 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:23.346 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:23.346 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.952722+0000", 2024-09-11T09:52:23.346 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00098314800000000009 2024-09-11T09:52:23.346 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:23.346 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:23.346 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:23.346 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.952750+0000", 2024-09-11T09:52:23.346 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.8050000000000001e-05 2024-09-11T09:52:23.346 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:23.346 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:23.347 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:23.347 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.952771+0000", 2024-09-11T09:52:23.347 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.0891e-05 2024-09-11T09:52:23.347 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:23.347 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:23.347 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:23.347 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.952844+0000", 2024-09-11T09:52:23.347 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.3064000000000004e-05 2024-09-11T09:52:23.347 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:23.347 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:23.348 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:23.348 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.952853+0000", 2024-09-11T09:52:23.348 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.1160000000000005e-06 2024-09-11T09:52:23.348 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:23.348 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:23.348 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:23.348 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.953564+0000", 2024-09-11T09:52:23.348 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00071132900000000002 2024-09-11T09:52:23.348 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:23.348 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:23.348 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:23.348 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:23.349 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:23.349 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "MOSDPGPush(2.4 82/81 [PushOp(2:3bfc818c:ns::rbd_data.1256a0c02879.0000000000000001:head, version: 53'14, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:3bfc818c:ns::rbd_data.1256a0c02879.0000000000000001:head@53'14, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3145728, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-11T09:52:23.349 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:52:08.955761+0000", 2024-09-11T09:52:23.349 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 14.383313865, 2024-09-11T09:52:23.349 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0019711659999999999, 2024-09-11T09:52:23.349 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:23.349 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:23.349 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "started", 2024-09-11T09:52:23.349 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:23.349 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:23.349 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:23.349 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.955761+0000", 2024-09-11T09:52:23.350 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:23.350 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:23.350 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:23.350 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:23.350 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.955761+0000", 2024-09-11T09:52:23.350 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:23.350 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:23.350 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:23.350 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:23.350 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.955761+0000", 2024-09-11T09:52:23.350 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.2399999999999997e-07 2024-09-11T09:52:23.350 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:23.350 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:23.350 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:23.351 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.956708+0000", 2024-09-11T09:52:23.351 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00094629899999999995 2024-09-11T09:52:23.351 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:23.351 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:23.351 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:23.351 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.956731+0000", 2024-09-11T09:52:23.351 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.3175000000000001e-05 2024-09-11T09:52:23.351 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:23.351 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:23.351 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:23.351 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.956750+0000", 2024-09-11T09:52:23.351 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.8859e-05 2024-09-11T09:52:23.351 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:23.352 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:23.352 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:23.352 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.956828+0000", 2024-09-11T09:52:23.352 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.8567999999999997e-05 2024-09-11T09:52:23.352 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:23.352 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:23.352 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:23.352 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.956841+0000", 2024-09-11T09:52:23.352 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.2489e-05 2024-09-11T09:52:23.352 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:23.352 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:23.352 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:23.352 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.957732+0000", 2024-09-11T09:52:23.352 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00089105200000000003 2024-09-11T09:52:23.353 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:23.353 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:23.353 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:23.353 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:23.353 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:23.353 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "MOSDPGPush(2.4 82/81 [PushOp(2:3bfc818c:ns::rbd_data.1256a0c02879.0000000000000001:head, version: 53'14, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:3bfc818c:ns::rbd_data.1256a0c02879.0000000000000001:head@53'14, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 4194304, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3145728, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-11T09:52:23.353 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:52:08.959483+0000", 2024-09-11T09:52:23.353 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 14.379591335000001, 2024-09-11T09:52:23.353 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0021404520000000002, 2024-09-11T09:52:23.353 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:23.353 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:23.353 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "started", 2024-09-11T09:52:23.353 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:23.354 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:23.354 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:23.354 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.959483+0000", 2024-09-11T09:52:23.354 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:23.354 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:23.354 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:23.354 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:23.354 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.959483+0000", 2024-09-11T09:52:23.354 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:23.354 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:23.354 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:23.354 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:23.354 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.959484+0000", 2024-09-11T09:52:23.355 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.4900000000000005e-07 2024-09-11T09:52:23.355 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:23.355 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:23.355 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:23.355 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.960450+0000", 2024-09-11T09:52:23.355 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000966141 2024-09-11T09:52:23.355 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:23.355 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:23.355 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:23.355 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.960466+0000", 2024-09-11T09:52:23.355 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.6302000000000001e-05 2024-09-11T09:52:23.355 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:23.355 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:23.356 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:23.356 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.960477+0000", 2024-09-11T09:52:23.356 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.0406e-05 2024-09-11T09:52:23.356 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:23.356 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:23.356 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:23.356 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.960555+0000", 2024-09-11T09:52:23.356 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.7837999999999996e-05 2024-09-11T09:52:23.356 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:23.356 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:23.356 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:23.356 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.960568+0000", 2024-09-11T09:52:23.356 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.2904e-05 2024-09-11T09:52:23.357 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:23.357 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:23.357 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:23.357 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.961624+0000", 2024-09-11T09:52:23.357 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0010561119999999999 2024-09-11T09:52:23.357 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:23.357 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:23.357 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:23.357 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:23.357 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:23.357 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "MOSDPGPush(2.4 82/81 [PushOp(2:2367385a:::rbd_data.1250f5e25b80.0000000000000002:6, version: 79'169, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 1, recovery_info: ObjectRecoveryInfo(2:2367385a:::rbd_data.1250f5e25b80.0000000000000002:6@79'169, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 6=[]:{6: [6, 5, 3]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-11T09:52:23.357 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:52:08.964132+0000", 2024-09-11T09:52:23.357 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 14.374942854, 2024-09-11T09:52:23.357 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0021074240000000001, 2024-09-11T09:52:23.358 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:23.358 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:23.358 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "started", 2024-09-11T09:52:23.358 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:23.358 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:23.358 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:23.358 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.964132+0000", 2024-09-11T09:52:23.358 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:23.358 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:23.358 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:23.358 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:23.358 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.964132+0000", 2024-09-11T09:52:23.358 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:23.358 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:23.359 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:23.359 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:23.359 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.964133+0000", 2024-09-11T09:52:23.359 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.02e-07 2024-09-11T09:52:23.359 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:23.359 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:23.359 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:23.359 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.965104+0000", 2024-09-11T09:52:23.359 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000971849 2024-09-11T09:52:23.359 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:23.360 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:23.360 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:23.360 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.965122+0000", 2024-09-11T09:52:23.360 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.7706999999999998e-05 2024-09-11T09:52:23.360 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:23.360 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:23.360 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:23.360 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.965134+0000", 2024-09-11T09:52:23.360 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1371e-05 2024-09-11T09:52:23.360 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:23.360 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:23.360 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:23.360 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.965221+0000", 2024-09-11T09:52:23.361 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.7037e-05 2024-09-11T09:52:23.361 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:23.361 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:23.361 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:23.361 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.965236+0000", 2024-09-11T09:52:23.361 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.5724999999999999e-05 2024-09-11T09:52:23.361 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:23.361 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:23.361 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:23.361 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.966239+0000", 2024-09-11T09:52:23.361 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0010028330000000001 2024-09-11T09:52:23.361 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:23.361 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:23.362 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:23.362 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:23.362 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:23.362 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "MOSDPGPush(2.4 82/81 [PushOp(2:2367385a:::rbd_data.1250f5e25b80.0000000000000002:6, version: 79'169, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:2367385a:::rbd_data.1250f5e25b80.0000000000000002:6@79'169, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 6=[]:{6: [6, 5, 3]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-11T09:52:23.362 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:52:08.968106+0000", 2024-09-11T09:52:23.362 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 14.370968387, 2024-09-11T09:52:23.362 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.002036936, 2024-09-11T09:52:23.362 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:23.362 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:23.362 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "started", 2024-09-11T09:52:23.362 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:23.362 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:23.362 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:23.362 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.968106+0000", 2024-09-11T09:52:23.363 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:23.363 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:23.363 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:23.363 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:23.363 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.968106+0000", 2024-09-11T09:52:23.363 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:23.363 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:23.363 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:23.363 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:23.363 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.968107+0000", 2024-09-11T09:52:23.363 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.0100000000000001e-06 2024-09-11T09:52:23.363 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:23.364 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:23.364 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:23.364 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.969086+0000", 2024-09-11T09:52:23.364 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00097889099999999996 2024-09-11T09:52:23.364 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:23.364 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:23.364 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:23.364 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.969104+0000", 2024-09-11T09:52:23.364 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.7940000000000001e-05 2024-09-11T09:52:23.364 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:23.364 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:23.364 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:23.364 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.969117+0000", 2024-09-11T09:52:23.365 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.3207000000000001e-05 2024-09-11T09:52:23.365 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:23.365 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:23.365 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:23.365 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.969192+0000", 2024-09-11T09:52:23.365 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.4419999999999996e-05 2024-09-11T09:52:23.365 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:23.365 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:23.365 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:23.365 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.969201+0000", 2024-09-11T09:52:23.365 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.4960000000000002e-06 2024-09-11T09:52:23.365 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:23.366 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:23.366 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:23.366 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.970143+0000", 2024-09-11T09:52:23.366 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00094197200000000001 2024-09-11T09:52:23.366 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:23.366 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:23.366 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:23.366 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:23.366 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:23.366 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "MOSDPGPush(2.4 82/81 [PushOp(2:2367385a:::rbd_data.1250f5e25b80.0000000000000002:6, version: 79'169, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:2367385a:::rbd_data.1250f5e25b80.0000000000000002:6@79'169, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 6=[]:{6: [6, 5, 3]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3145728, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-11T09:52:23.366 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:52:08.971856+0000", 2024-09-11T09:52:23.366 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 14.367218364999999, 2024-09-11T09:52:23.367 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0018957309999999999, 2024-09-11T09:52:23.367 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:23.367 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:23.367 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "started", 2024-09-11T09:52:23.367 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:23.367 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:23.367 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:23.367 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.971856+0000", 2024-09-11T09:52:23.367 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:23.367 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:23.367 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:23.367 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:23.367 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.971856+0000", 2024-09-11T09:52:23.368 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:23.368 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:23.368 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:23.368 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:23.368 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.971857+0000", 2024-09-11T09:52:23.368 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.4900000000000004e-07 2024-09-11T09:52:23.368 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:23.368 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:23.368 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:23.368 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.972795+0000", 2024-09-11T09:52:23.368 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00093767999999999998 2024-09-11T09:52:23.368 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:23.368 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:23.369 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:23.369 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.972813+0000", 2024-09-11T09:52:23.369 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.8448999999999999e-05 2024-09-11T09:52:23.369 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:23.369 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:23.369 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:23.369 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.972825+0000", 2024-09-11T09:52:23.369 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1837e-05 2024-09-11T09:52:23.369 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:23.369 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:23.369 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:23.369 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.972904+0000", 2024-09-11T09:52:23.369 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.8777999999999997e-05 2024-09-11T09:52:23.369 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:23.370 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:23.370 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:23.370 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.972913+0000", 2024-09-11T09:52:23.370 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.1139999999999995e-06 2024-09-11T09:52:23.370 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:23.370 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:23.370 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:23.370 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.973752+0000", 2024-09-11T09:52:23.370 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00083892400000000001 2024-09-11T09:52:23.370 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:23.370 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:23.370 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:23.370 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:23.370 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:23.371 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "MOSDPGPush(2.4 82/81 [PushOp(2:2367385a:::rbd_data.1250f5e25b80.0000000000000002:6, version: 79'169, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:2367385a:::rbd_data.1250f5e25b80.0000000000000002:6@79'169, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 6=[]:{6: [6, 5, 3]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 4194304, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3145728, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-11T09:52:23.371 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:52:08.975206+0000", 2024-09-11T09:52:23.371 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 14.363868678999999, 2024-09-11T09:52:23.371 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0020632770000000001, 2024-09-11T09:52:23.371 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:23.371 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:23.371 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "started", 2024-09-11T09:52:23.371 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:23.371 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:23.371 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:23.371 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.975206+0000", 2024-09-11T09:52:23.371 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:23.371 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:23.372 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:23.372 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:23.372 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.975206+0000", 2024-09-11T09:52:23.372 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:23.372 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:23.372 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:23.372 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:23.372 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.975207+0000", 2024-09-11T09:52:23.372 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.2399999999999998e-07 2024-09-11T09:52:23.372 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:23.372 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:23.372 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:23.372 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.976173+0000", 2024-09-11T09:52:23.373 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00096649400000000003 2024-09-11T09:52:23.373 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:23.373 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:23.373 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:23.373 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.976189+0000", 2024-09-11T09:52:23.373 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.6223e-05 2024-09-11T09:52:23.373 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:23.373 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:23.373 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:23.373 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.976200+0000", 2024-09-11T09:52:23.373 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.0729e-05 2024-09-11T09:52:23.373 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:23.373 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:23.373 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:23.374 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.976265+0000", 2024-09-11T09:52:23.374 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.5326000000000004e-05 2024-09-11T09:52:23.374 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:23.374 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:23.374 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:23.374 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.976274+0000", 2024-09-11T09:52:23.374 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.8140000000000007e-06 2024-09-11T09:52:23.374 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:23.374 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:23.374 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:23.374 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.977269+0000", 2024-09-11T09:52:23.374 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00099506700000000009 2024-09-11T09:52:23.375 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:23.375 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:23.375 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:23.375 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:23.375 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:23.375 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "MOSDPGPush(2.4 82/81 [PushOp(2:26d5c2f0:::rbd_data.1250f5e25b80.000000000000000b:6, version: 79'171, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 1, recovery_info: ObjectRecoveryInfo(2:26d5c2f0:::rbd_data.1250f5e25b80.000000000000000b:6@79'171, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 6=[]:{6: [6, 5, 3]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-11T09:52:23.375 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:52:08.979123+0000", 2024-09-11T09:52:23.375 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 14.35995125, 2024-09-11T09:52:23.375 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0019749350000000001, 2024-09-11T09:52:23.375 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:23.375 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:23.375 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "started", 2024-09-11T09:52:23.375 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:23.375 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:23.376 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:23.376 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.979123+0000", 2024-09-11T09:52:23.376 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:23.376 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:23.376 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:23.376 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:23.376 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.979123+0000", 2024-09-11T09:52:23.376 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:23.376 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:23.376 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:23.376 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:23.376 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.979124+0000", 2024-09-11T09:52:23.376 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.5799999999999999e-07 2024-09-11T09:52:23.377 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:23.377 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:23.377 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:23.377 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.980088+0000", 2024-09-11T09:52:23.377 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00096377699999999997 2024-09-11T09:52:23.377 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:23.377 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:23.377 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:23.377 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.980106+0000", 2024-09-11T09:52:23.377 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.7881e-05 2024-09-11T09:52:23.378 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:23.378 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:23.378 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:23.378 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.980117+0000", 2024-09-11T09:52:23.378 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1681e-05 2024-09-11T09:52:23.378 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:23.378 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:23.378 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:23.378 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.980189+0000", 2024-09-11T09:52:23.378 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.1365999999999994e-05 2024-09-11T09:52:23.378 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:23.378 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:23.378 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:23.379 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.980198+0000", 2024-09-11T09:52:23.379 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.3300000000000005e-06 2024-09-11T09:52:23.379 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:23.379 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:23.379 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:23.379 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.981098+0000", 2024-09-11T09:52:23.379 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00090024199999999999 2024-09-11T09:52:23.379 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:23.379 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:23.379 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:23.379 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:23.379 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:23.379 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "MOSDPGPush(2.4 82/81 [PushOp(2:26d5c2f0:::rbd_data.1250f5e25b80.000000000000000b:6, version: 79'171, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:26d5c2f0:::rbd_data.1250f5e25b80.000000000000000b:6@79'171, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 6=[]:{6: [6, 5, 3]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-11T09:52:23.379 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:52:08.982550+0000", 2024-09-11T09:52:23.380 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 14.356524804999999, 2024-09-11T09:52:23.380 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0020071540000000001, 2024-09-11T09:52:23.380 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:23.380 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:23.380 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "started", 2024-09-11T09:52:23.380 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:23.380 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:23.380 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:23.380 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.982550+0000", 2024-09-11T09:52:23.380 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:23.380 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:23.380 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:23.380 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:23.381 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.982550+0000", 2024-09-11T09:52:23.381 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:23.381 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:23.381 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:23.381 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:23.381 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.982550+0000", 2024-09-11T09:52:23.381 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.89e-07 2024-09-11T09:52:23.381 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:23.381 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:23.381 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:23.381 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.983523+0000", 2024-09-11T09:52:23.381 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00097246400000000003 2024-09-11T09:52:23.381 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:23.381 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:23.382 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:23.382 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.983540+0000", 2024-09-11T09:52:23.382 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.7388999999999999e-05 2024-09-11T09:52:23.382 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:23.382 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:23.382 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:23.382 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.983553+0000", 2024-09-11T09:52:23.382 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.2876e-05 2024-09-11T09:52:23.382 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:23.382 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:23.382 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:23.382 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.983623+0000", 2024-09-11T09:52:23.382 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.0199999999999999e-05 2024-09-11T09:52:23.383 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:23.383 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:23.383 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:23.383 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.983632+0000", 2024-09-11T09:52:23.383 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.8859999999999993e-06 2024-09-11T09:52:23.383 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:23.383 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:23.383 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:23.383 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.984557+0000", 2024-09-11T09:52:23.383 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00092484999999999996 2024-09-11T09:52:23.383 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:23.383 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:23.383 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:23.384 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:23.384 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:23.384 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "MOSDPGPush(2.4 82/81 [PushOp(2:26d5c2f0:::rbd_data.1250f5e25b80.000000000000000b:6, version: 79'171, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:26d5c2f0:::rbd_data.1250f5e25b80.000000000000000b:6@79'171, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 6=[]:{6: [6, 5, 3]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3145728, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-11T09:52:23.384 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:52:08.985896+0000", 2024-09-11T09:52:23.384 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 14.353178214, 2024-09-11T09:52:23.384 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0018739799999999999, 2024-09-11T09:52:23.384 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:23.384 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:23.384 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "started", 2024-09-11T09:52:23.384 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:23.384 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:23.384 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:23.384 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.985896+0000", 2024-09-11T09:52:23.384 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:23.385 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:23.385 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:23.385 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:23.385 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.985896+0000", 2024-09-11T09:52:23.385 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:23.388 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:23.388 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:23.388 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:23.388 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.985897+0000", 2024-09-11T09:52:23.388 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.3099999999999996e-07 2024-09-11T09:52:23.388 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:23.388 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:23.388 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:23.388 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.986883+0000", 2024-09-11T09:52:23.388 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00098564300000000002 2024-09-11T09:52:23.388 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:23.388 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:23.389 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:23.389 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.986900+0000", 2024-09-11T09:52:23.389 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.7475000000000001e-05 2024-09-11T09:52:23.389 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:23.389 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:23.389 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:23.389 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.986912+0000", 2024-09-11T09:52:23.389 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1924e-05 2024-09-11T09:52:23.389 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:23.389 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:23.389 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:23.389 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.986988+0000", 2024-09-11T09:52:23.389 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.6099000000000004e-05 2024-09-11T09:52:23.390 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:23.390 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:23.390 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:23.390 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.986998+0000", 2024-09-11T09:52:23.390 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.5710000000000004e-06 2024-09-11T09:52:23.390 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:23.390 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:23.390 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:23.390 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.987770+0000", 2024-09-11T09:52:23.390 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000772337 2024-09-11T09:52:23.390 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:23.390 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:23.390 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:23.390 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:23.390 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:23.391 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "MOSDPGPush(2.4 82/81 [PushOp(2:26d5c2f0:::rbd_data.1250f5e25b80.000000000000000b:6, version: 79'171, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:26d5c2f0:::rbd_data.1250f5e25b80.000000000000000b:6@79'171, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 6=[]:{6: [6, 5, 3]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 4194304, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3145728, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-11T09:52:23.391 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:52:08.989143+0000", 2024-09-11T09:52:23.391 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 14.349931114, 2024-09-11T09:52:23.391 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0022141589999999998, 2024-09-11T09:52:23.391 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:23.391 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:23.391 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "started", 2024-09-11T09:52:23.391 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:23.391 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:23.391 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:23.391 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.989143+0000", 2024-09-11T09:52:23.391 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:23.391 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:23.392 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:23.392 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:23.392 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.989143+0000", 2024-09-11T09:52:23.392 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:23.392 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:23.392 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:23.392 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:23.392 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.989144+0000", 2024-09-11T09:52:23.392 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.1699999999999997e-07 2024-09-11T09:52:23.392 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:23.392 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:23.392 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:23.392 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.990124+0000", 2024-09-11T09:52:23.392 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00097941400000000007 2024-09-11T09:52:23.393 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:23.393 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:23.393 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:23.393 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.990140+0000", 2024-09-11T09:52:23.393 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.6583e-05 2024-09-11T09:52:23.393 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:23.393 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:23.393 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:23.393 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.990152+0000", 2024-09-11T09:52:23.393 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1331999999999999e-05 2024-09-11T09:52:23.393 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:23.393 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:23.393 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:23.394 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.990220+0000", 2024-09-11T09:52:23.394 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.8777999999999998e-05 2024-09-11T09:52:23.394 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:23.394 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:23.394 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:23.394 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.990229+0000", 2024-09-11T09:52:23.394 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.9590000000000001e-06 2024-09-11T09:52:23.394 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:23.394 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:23.394 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:23.394 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.991358+0000", 2024-09-11T09:52:23.394 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0011283759999999999 2024-09-11T09:52:23.394 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:23.394 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:23.395 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:23.395 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:23.395 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:23.395 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "MOSDPGPush(2.4 82/81 [PushOp(2:239e915b:::rbd_data.1250f5e25b80.000000000000000c:6, version: 79'173, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 1, recovery_info: ObjectRecoveryInfo(2:239e915b:::rbd_data.1250f5e25b80.000000000000000c:6@79'173, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 6=[]:{6: [6, 5, 3]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-11T09:52:23.395 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:52:08.994080+0000", 2024-09-11T09:52:23.395 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 14.344994977000001, 2024-09-11T09:52:23.395 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0019554189999999999, 2024-09-11T09:52:23.395 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:23.395 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:23.395 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "started", 2024-09-11T09:52:23.395 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:23.395 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:23.395 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:23.395 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.994080+0000", 2024-09-11T09:52:23.396 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:23.396 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:23.396 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:23.396 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:23.396 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.994080+0000", 2024-09-11T09:52:23.396 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:23.396 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:23.396 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:23.396 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:23.396 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.994080+0000", 2024-09-11T09:52:23.396 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.7599999999999997e-07 2024-09-11T09:52:23.396 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:23.396 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:23.397 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:23.397 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.995065+0000", 2024-09-11T09:52:23.397 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00098436199999999991 2024-09-11T09:52:23.397 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:23.397 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:23.397 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:23.397 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.995080+0000", 2024-09-11T09:52:23.397 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.5119e-05 2024-09-11T09:52:23.397 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:23.397 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:23.397 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:23.397 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.995090+0000", 2024-09-11T09:52:23.397 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.0715e-05 2024-09-11T09:52:23.398 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:23.398 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:23.398 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:23.398 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.995152+0000", 2024-09-11T09:52:23.398 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.2089999999999997e-05 2024-09-11T09:52:23.398 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:23.398 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:23.398 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:23.398 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.995159+0000", 2024-09-11T09:52:23.398 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.9210000000000003e-06 2024-09-11T09:52:23.398 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:23.398 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:23.398 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:23.398 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.996035+0000", 2024-09-11T09:52:23.399 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00087563599999999995 2024-09-11T09:52:23.399 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:23.399 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:23.399 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:23.399 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:23.399 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:23.399 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "MOSDPGPush(2.4 82/81 [PushOp(2:239e915b:::rbd_data.1250f5e25b80.000000000000000c:6, version: 79'173, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:239e915b:::rbd_data.1250f5e25b80.000000000000000c:6@79'173, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 6=[]:{6: [6, 5, 3]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-11T09:52:23.399 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:52:08.997853+0000", 2024-09-11T09:52:23.399 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 14.341222030000001, 2024-09-11T09:52:23.399 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0018792979999999999, 2024-09-11T09:52:23.399 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:23.399 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:23.399 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "started", 2024-09-11T09:52:23.399 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:23.400 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:23.400 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:23.400 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.997853+0000", 2024-09-11T09:52:23.400 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:23.400 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:23.400 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:23.400 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:23.400 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.997853+0000", 2024-09-11T09:52:23.400 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:23.400 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:23.400 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:23.400 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:23.400 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.997853+0000", 2024-09-11T09:52:23.400 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.4000000000000002e-07 2024-09-11T09:52:23.400 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:23.401 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:23.401 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:23.401 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.998831+0000", 2024-09-11T09:52:23.401 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00097792100000000009 2024-09-11T09:52:23.401 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:23.401 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:23.401 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:23.401 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.998846+0000", 2024-09-11T09:52:23.401 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.4987000000000001e-05 2024-09-11T09:52:23.401 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:23.401 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:23.401 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:23.401 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.998856+0000", 2024-09-11T09:52:23.402 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.0448000000000001e-05 2024-09-11T09:52:23.402 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:23.402 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:23.402 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:23.402 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.998942+0000", 2024-09-11T09:52:23.402 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.5563999999999996e-05 2024-09-11T09:52:23.402 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:23.402 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:23.402 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:23.402 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.998952+0000", 2024-09-11T09:52:23.402 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.5170000000000005e-06 2024-09-11T09:52:23.402 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:23.402 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:23.402 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:23.403 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.999732+0000", 2024-09-11T09:52:23.403 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00078032100000000005 2024-09-11T09:52:23.403 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:23.403 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:23.403 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:23.403 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:23.403 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:23.403 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "MOSDPGPush(2.4 82/81 [PushOp(2:239e915b:::rbd_data.1250f5e25b80.000000000000000c:6, version: 79'173, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:239e915b:::rbd_data.1250f5e25b80.000000000000000c:6@79'173, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 6=[]:{6: [6, 5, 3]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3145728, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-11T09:52:23.403 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:52:09.001584+0000", 2024-09-11T09:52:23.403 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 14.337490109999999, 2024-09-11T09:52:23.403 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0019129119999999999, 2024-09-11T09:52:23.403 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:23.403 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:23.403 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "started", 2024-09-11T09:52:23.404 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:23.404 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:23.404 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:23.404 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.001584+0000", 2024-09-11T09:52:23.404 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:23.404 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:23.404 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:23.404 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:23.404 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.001584+0000", 2024-09-11T09:52:23.404 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:23.404 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:23.404 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:23.404 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:23.404 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.001585+0000", 2024-09-11T09:52:23.405 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.8899999999999999e-07 2024-09-11T09:52:23.405 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:23.405 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:23.405 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:23.405 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.002541+0000", 2024-09-11T09:52:23.405 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00095588800000000001 2024-09-11T09:52:23.405 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:23.405 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:23.405 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:23.405 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.002558+0000", 2024-09-11T09:52:23.405 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.6452000000000001e-05 2024-09-11T09:52:23.405 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:23.405 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:23.406 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:23.406 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.002569+0000", 2024-09-11T09:52:23.406 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.148e-05 2024-09-11T09:52:23.406 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:23.406 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:23.406 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:23.406 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.002647+0000", 2024-09-11T09:52:23.406 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.7570000000000004e-05 2024-09-11T09:52:23.406 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:23.406 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:23.406 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:23.406 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.002655+0000", 2024-09-11T09:52:23.406 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.4610000000000008e-06 2024-09-11T09:52:23.407 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:23.407 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:23.407 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:23.407 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.003497+0000", 2024-09-11T09:52:23.407 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00084237199999999998 2024-09-11T09:52:23.407 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:23.407 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:23.407 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:23.407 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:23.407 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:23.407 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "MOSDPGPush(2.4 82/81 [PushOp(2:239e915b:::rbd_data.1250f5e25b80.000000000000000c:6, version: 79'173, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:239e915b:::rbd_data.1250f5e25b80.000000000000000c:6@79'173, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 6=[]:{6: [6, 5, 3]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 4194304, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3145728, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-11T09:52:23.407 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:52:09.005094+0000", 2024-09-11T09:52:23.407 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 14.333980115999999, 2024-09-11T09:52:23.407 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0024127419999999998, 2024-09-11T09:52:23.408 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:23.408 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:23.408 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "started", 2024-09-11T09:52:23.408 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:23.408 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:23.408 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:23.408 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.005094+0000", 2024-09-11T09:52:23.408 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:23.408 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:23.408 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:23.408 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:23.408 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.005094+0000", 2024-09-11T09:52:23.408 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:23.408 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:23.409 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:23.409 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:23.409 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.005095+0000", 2024-09-11T09:52:23.409 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.8999999999999997e-07 2024-09-11T09:52:23.409 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:23.409 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:23.409 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:23.409 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.006059+0000", 2024-09-11T09:52:23.409 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00096429100000000004 2024-09-11T09:52:23.409 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:23.409 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:23.409 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:23.410 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.006075+0000", 2024-09-11T09:52:23.410 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.6036e-05 2024-09-11T09:52:23.410 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:23.410 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:23.410 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:23.410 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.006088+0000", 2024-09-11T09:52:23.410 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.2557e-05 2024-09-11T09:52:23.410 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:23.410 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:23.410 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:23.410 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.006167+0000", 2024-09-11T09:52:23.410 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.8843000000000004e-05 2024-09-11T09:52:23.410 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:23.410 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:23.411 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:23.411 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.006182+0000", 2024-09-11T09:52:23.411 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.4968e-05 2024-09-11T09:52:23.411 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:23.411 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:23.411 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:23.411 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.007507+0000", 2024-09-11T09:52:23.411 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001325557 2024-09-11T09:52:23.411 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:23.411 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:23.411 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:23.411 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:23.411 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:23.412 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "MOSDPGPush(2.4 82/81 [PushOp(2:2167012f:::rbd_data.1250f5e25b80.000000000000000d:6, version: 79'175, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 1, recovery_info: ObjectRecoveryInfo(2:2167012f:::rbd_data.1250f5e25b80.000000000000000d:6@79'175, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 6=[]:{6: [6, 5, 3]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-11T09:52:23.412 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:52:09.009862+0000", 2024-09-11T09:52:23.412 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 14.329212656999999, 2024-09-11T09:52:23.412 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001845522, 2024-09-11T09:52:23.412 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:23.412 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:23.412 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "started", 2024-09-11T09:52:23.412 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:23.412 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:23.412 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:23.412 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.009862+0000", 2024-09-11T09:52:23.412 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:23.412 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:23.413 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:23.413 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:23.413 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.009862+0000", 2024-09-11T09:52:23.413 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:23.413 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:23.413 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:23.413 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:23.413 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.009863+0000", 2024-09-11T09:52:23.413 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.1200000000000002e-07 2024-09-11T09:52:23.413 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:23.413 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:23.413 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:23.413 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.010797+0000", 2024-09-11T09:52:23.414 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00093439599999999999 2024-09-11T09:52:23.414 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:23.414 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:23.414 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:23.414 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.010814+0000", 2024-09-11T09:52:23.414 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.7354e-05 2024-09-11T09:52:23.414 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:23.414 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:23.414 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:23.414 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.010827+0000", 2024-09-11T09:52:23.414 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.3096e-05 2024-09-11T09:52:23.414 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:23.414 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:23.415 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:23.415 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.010894+0000", 2024-09-11T09:52:23.415 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.6501999999999994e-05 2024-09-11T09:52:23.415 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:23.415 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:23.415 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:23.415 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.010903+0000", 2024-09-11T09:52:23.415 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.0899999999999994e-06 2024-09-11T09:52:23.415 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:23.415 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:23.415 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:23.415 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.011707+0000", 2024-09-11T09:52:23.415 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00080437200000000003 2024-09-11T09:52:23.415 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:23.416 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:23.416 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:23.416 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:23.416 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:23.416 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "MOSDPGPush(2.4 82/81 [PushOp(2:2167012f:::rbd_data.1250f5e25b80.000000000000000d:6, version: 79'175, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:2167012f:::rbd_data.1250f5e25b80.000000000000000d:6@79'175, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 6=[]:{6: [6, 5, 3]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-11T09:52:23.416 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:52:09.013431+0000", 2024-09-11T09:52:23.416 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 14.325643782, 2024-09-11T09:52:23.416 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.002012247, 2024-09-11T09:52:23.416 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:23.416 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:23.416 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "started", 2024-09-11T09:52:23.416 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:23.416 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:23.417 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:23.417 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.013431+0000", 2024-09-11T09:52:23.417 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:23.417 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:23.417 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:23.417 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:23.417 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.013431+0000", 2024-09-11T09:52:23.417 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:23.417 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:23.417 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:23.417 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:23.417 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.013432+0000", 2024-09-11T09:52:23.417 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.1900000000000001e-07 2024-09-11T09:52:23.417 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:23.418 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:23.418 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:23.418 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.014359+0000", 2024-09-11T09:52:23.418 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00092687900000000003 2024-09-11T09:52:23.418 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:23.418 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:23.418 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:23.418 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.014377+0000", 2024-09-11T09:52:23.418 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.8085e-05 2024-09-11T09:52:23.418 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:23.418 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:23.418 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:23.418 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.014391+0000", 2024-09-11T09:52:23.418 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.4099e-05 2024-09-11T09:52:23.419 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:23.419 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:23.419 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:23.419 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.014476+0000", 2024-09-11T09:52:23.419 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.4827999999999995e-05 2024-09-11T09:52:23.419 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:23.419 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:23.419 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:23.419 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.014489+0000", 2024-09-11T09:52:23.419 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.3169e-05 2024-09-11T09:52:23.419 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:23.419 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:23.419 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:23.419 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.015443+0000", 2024-09-11T09:52:23.420 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00095436800000000003 2024-09-11T09:52:23.420 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:23.420 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:23.420 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:23.420 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:23.420 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:23.420 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "MOSDPGPush(2.4 82/81 [PushOp(2:2167012f:::rbd_data.1250f5e25b80.000000000000000d:6, version: 79'175, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:2167012f:::rbd_data.1250f5e25b80.000000000000000d:6@79'175, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 6=[]:{6: [6, 5, 3]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3145728, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-11T09:52:23.420 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:52:09.017158+0000", 2024-09-11T09:52:23.420 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 14.321916794, 2024-09-11T09:52:23.420 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0020239889999999999, 2024-09-11T09:52:23.420 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:23.420 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:23.420 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "started", 2024-09-11T09:52:23.420 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:23.421 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:23.421 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:23.421 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.017158+0000", 2024-09-11T09:52:23.421 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:23.421 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:23.421 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:23.421 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:23.421 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.017158+0000", 2024-09-11T09:52:23.421 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:23.421 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:23.421 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:23.421 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:23.421 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.017158+0000", 2024-09-11T09:52:23.421 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.7000000000000004e-07 2024-09-11T09:52:23.422 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:23.422 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:23.422 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:23.422 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.018136+0000", 2024-09-11T09:52:23.422 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00097795 2024-09-11T09:52:23.422 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:23.422 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:23.422 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:23.422 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.018152+0000", 2024-09-11T09:52:23.422 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.5208000000000001e-05 2024-09-11T09:52:23.422 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:23.422 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:23.422 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:23.422 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.018164+0000", 2024-09-11T09:52:23.422 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.2157000000000001e-05 2024-09-11T09:52:23.423 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:23.423 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:23.423 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:23.423 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.018245+0000", 2024-09-11T09:52:23.423 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.1284000000000003e-05 2024-09-11T09:52:23.423 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:23.423 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:23.423 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:23.423 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.018257+0000", 2024-09-11T09:52:23.423 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.2160999999999999e-05 2024-09-11T09:52:23.423 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:23.423 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:23.423 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:23.423 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.019182+0000", 2024-09-11T09:52:23.424 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00092455900000000004 2024-09-11T09:52:23.424 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:23.424 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:23.424 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:23.424 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:23.424 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:23.424 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "MOSDPGPush(2.4 82/81 [PushOp(2:2167012f:::rbd_data.1250f5e25b80.000000000000000d:6, version: 79'175, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:2167012f:::rbd_data.1250f5e25b80.000000000000000d:6@79'175, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 6=[]:{6: [6, 5, 3]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 4194304, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3145728, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-11T09:52:23.424 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:52:09.020895+0000", 2024-09-11T09:52:23.424 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 14.318179741, 2024-09-11T09:52:23.424 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.002075234, 2024-09-11T09:52:23.424 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:23.424 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:23.424 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "started", 2024-09-11T09:52:23.425 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:23.425 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:23.425 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:23.425 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.020895+0000", 2024-09-11T09:52:23.425 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:23.425 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:23.425 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:23.425 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:23.425 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.020895+0000", 2024-09-11T09:52:23.425 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:23.425 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:23.425 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:23.425 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:23.425 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.020895+0000", 2024-09-11T09:52:23.426 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.0999999999999999e-07 2024-09-11T09:52:23.426 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:23.426 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:23.426 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:23.426 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.021855+0000", 2024-09-11T09:52:23.426 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00095982700000000001 2024-09-11T09:52:23.426 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:23.426 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:23.426 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:23.426 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.021870+0000", 2024-09-11T09:52:23.426 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.4609e-05 2024-09-11T09:52:23.426 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:23.426 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:23.426 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:23.427 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.021881+0000", 2024-09-11T09:52:23.427 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.083e-05 2024-09-11T09:52:23.427 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:23.427 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:23.427 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:23.427 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.021943+0000", 2024-09-11T09:52:23.427 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.2780999999999994e-05 2024-09-11T09:52:23.427 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:23.427 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:23.427 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:23.427 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.021952+0000", 2024-09-11T09:52:23.427 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.8629999999999997e-06 2024-09-11T09:52:23.427 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:23.427 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:23.428 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:23.428 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.022970+0000", 2024-09-11T09:52:23.428 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0010178139999999999 2024-09-11T09:52:23.428 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:23.428 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:23.428 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:23.428 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:23.428 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:23.428 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-11T09:52:23.430 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:52:23.430+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:52:23.431 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-11T09:52:23.431 INFO:tasks.workunit.client.0.smithi107.stderr:+ _sudo blkdiscard --offset 0 --length 4096 /dev/nbd2 2024-09-11T09:52:23.431 INFO:tasks.workunit.client.0.smithi107.stderr:+ local cmd 2024-09-11T09:52:23.431 INFO:tasks.workunit.client.0.smithi107.stderr:++ id -u 2024-09-11T09:52:23.431 INFO:tasks.workunit.client.0.smithi107.stderr:+ '[' 1000 -eq 0 ']' 2024-09-11T09:52:23.432 INFO:tasks.workunit.client.0.smithi107.stderr:++ which blkdiscard 2024-09-11T09:52:23.432 INFO:tasks.workunit.client.0.smithi107.stderr:+ cmd=/usr/sbin/blkdiscard 2024-09-11T09:52:23.432 INFO:tasks.workunit.client.0.smithi107.stderr:+ shift 2024-09-11T09:52:23.432 INFO:tasks.workunit.client.0.smithi107.stderr:+ sudo -nE /usr/sbin/blkdiscard --offset 0 --length 4096 /dev/nbd2 2024-09-11T09:52:23.530 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-09-11T09:52:23.531+0000 7f7b6c7d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31904) UID: 0 2024-09-11T09:52:23.539 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:52:23.539 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [], 2024-09-11T09:52:23.539 INFO:teuthology.orchestra.run.smithi017.stdout: "num_ops": 0 2024-09-11T09:52:23.539 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:52:23.548 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-11T09:52:23.631 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:52:23.632+0000 7f3a7b90a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 42475) UID: 0 2024-09-11T09:52:23.639 INFO:tasks.workunit.client.0.smithi107.stderr:+ rbd feature disable rbd/testrbdnbd27121 journaling 2024-09-11T09:52:23.657 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:52:23.657 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [], 2024-09-11T09:52:23.657 INFO:teuthology.orchestra.run.smithi017.stdout: "complaint_time": 30, 2024-09-11T09:52:23.657 INFO:teuthology.orchestra.run.smithi017.stdout: "num_blocked_ops": 0 2024-09-11T09:52:23.657 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:52:23.666 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-11T09:52:23.699 INFO:tasks.workunit.client.0.smithi107.stderr:+ unmap_device /dev/nbd2 30702 2024-09-11T09:52:23.699 INFO:tasks.workunit.client.0.smithi107.stderr:+ local args=/dev/nbd2 2024-09-11T09:52:23.699 INFO:tasks.workunit.client.0.smithi107.stderr:+ local pid=30702 2024-09-11T09:52:23.699 INFO:tasks.workunit.client.0.smithi107.stderr:+ _sudo rbd device --device-type nbd unmap /dev/nbd2 2024-09-11T09:52:23.699 INFO:tasks.workunit.client.0.smithi107.stderr:+ local cmd 2024-09-11T09:52:23.699 INFO:tasks.workunit.client.0.smithi107.stderr:++ id -u 2024-09-11T09:52:23.700 INFO:tasks.workunit.client.0.smithi107.stderr:+ '[' 1000 -eq 0 ']' 2024-09-11T09:52:23.700 INFO:tasks.workunit.client.0.smithi107.stderr:++ which rbd 2024-09-11T09:52:23.702 INFO:tasks.workunit.client.0.smithi107.stderr:+ cmd=/usr/bin/rbd 2024-09-11T09:52:23.702 INFO:tasks.workunit.client.0.smithi107.stderr:+ shift 2024-09-11T09:52:23.702 INFO:tasks.workunit.client.0.smithi107.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd unmap /dev/nbd2 2024-09-11T09:52:23.732 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-09-11T09:52:23.734+0000 7f7b6c7d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31904) UID: 0 2024-09-11T09:52:23.777 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:52:23.777 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 20, 2024-09-11T09:52:23.777 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 600, 2024-09-11T09:52:23.777 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [ 2024-09-11T09:52:23.777 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.777 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4688.0:34 2.4 e53/32 2:2367385a:::rbd_data.1250f5e25b80.0000000000000002:head v 53'6, mlcod=53'4)", 2024-09-11T09:52:23.777 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.907415+0000", 2024-09-11T09:52:23.777 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 60.869350461000003, 2024-09-11T09:52:23.777 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.011285778999999999, 2024-09-11T09:52:23.777 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:23.777 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:23.778 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:23.778 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:23.778 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.778 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:23.778 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.907415+0000", 2024-09-11T09:52:23.778 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:23.778 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.778 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.778 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:23.778 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.907415+0000", 2024-09-11T09:52:23.778 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:23.778 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.779 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.779 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:23.779 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.907417+0000", 2024-09-11T09:52:23.779 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.2910000000000002e-06 2024-09-11T09:52:23.779 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.779 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.779 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:23.779 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.910477+0000", 2024-09-11T09:52:23.779 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0030597829999999999 2024-09-11T09:52:23.779 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.779 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.780 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:23.780 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.910501+0000", 2024-09-11T09:52:23.780 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.3332e-05 2024-09-11T09:52:23.780 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.780 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.780 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:23.780 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.910514+0000", 2024-09-11T09:52:23.780 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3957e-05 2024-09-11T09:52:23.780 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.780 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.780 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:23.780 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.910612+0000", 2024-09-11T09:52:23.780 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.7075000000000001e-05 2024-09-11T09:52:23.781 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.781 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.781 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:23.781 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.910640+0000", 2024-09-11T09:52:23.781 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.8768999999999998e-05 2024-09-11T09:52:23.781 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.781 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.781 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:23.781 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.918640+0000", 2024-09-11T09:52:23.781 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0079991899999999998 2024-09-11T09:52:23.781 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.781 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.782 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:23.782 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.918701+0000", 2024-09-11T09:52:23.782 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.1382000000000003e-05 2024-09-11T09:52:23.782 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:23.782 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:23.782 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:23.782 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.782 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.782 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4688.0:38 2.5 e53/32 2:abde2694:::rbd_data.1250f5e25b80.0000000000000004:head v 53'1, mlcod=0'0)", 2024-09-11T09:52:23.782 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.915432+0000", 2024-09-11T09:52:23.782 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 60.861334081000003, 2024-09-11T09:52:23.782 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.015849652999999998, 2024-09-11T09:52:23.783 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:23.783 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:23.783 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:23.783 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:23.783 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.783 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:23.783 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.915432+0000", 2024-09-11T09:52:23.783 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:23.783 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.783 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.783 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:23.783 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.915432+0000", 2024-09-11T09:52:23.784 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:23.784 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.784 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.784 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:23.784 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.915432+0000", 2024-09-11T09:52:23.784 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.91e-07 2024-09-11T09:52:23.784 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.784 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.784 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:23.784 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.924943+0000", 2024-09-11T09:52:23.784 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0095110630000000002 2024-09-11T09:52:23.784 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.785 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.785 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:23.785 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.924960+0000", 2024-09-11T09:52:23.785 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6915999999999999e-05 2024-09-11T09:52:23.785 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.785 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.785 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:23.785 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.924968+0000", 2024-09-11T09:52:23.785 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.6999999999999997e-06 2024-09-11T09:52:23.785 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.785 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.785 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:23.785 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.925025+0000", 2024-09-11T09:52:23.785 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.6949e-05 2024-09-11T09:52:23.786 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.786 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.786 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:23.786 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.925043+0000", 2024-09-11T09:52:23.786 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7784000000000001e-05 2024-09-11T09:52:23.786 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.786 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.786 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:23.786 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.931237+0000", 2024-09-11T09:52:23.786 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0061935389999999996 2024-09-11T09:52:23.786 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.786 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.786 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:23.786 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.931281+0000", 2024-09-11T09:52:23.787 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.4511000000000001e-05 2024-09-11T09:52:23.787 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:23.787 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:23.787 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:23.787 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.787 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.787 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:40 2.7 2:fe41411c:::rbd_object_map.1250f5e25b80:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:23.787 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.919652+0000", 2024-09-11T09:52:23.787 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 60.857113413999997, 2024-09-11T09:52:23.787 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.012187073, 2024-09-11T09:52:23.787 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:23.787 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:23.787 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:23.788 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:23.788 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:52:23.788 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:52:23.788 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 40 2024-09-11T09:52:23.788 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.788 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:23.788 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.788 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:23.788 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919652+0000", 2024-09-11T09:52:23.788 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:23.788 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.788 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.789 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:23.789 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919652+0000", 2024-09-11T09:52:23.789 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:23.789 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.789 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.789 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:23.789 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919654+0000", 2024-09-11T09:52:23.789 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.88e-06 2024-09-11T09:52:23.789 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.789 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.789 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:23.789 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919669+0000", 2024-09-11T09:52:23.789 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.5271e-05 2024-09-11T09:52:23.790 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.790 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.790 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:23.790 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919687+0000", 2024-09-11T09:52:23.790 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7448999999999998e-05 2024-09-11T09:52:23.790 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.790 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.790 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:23.790 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919695+0000", 2024-09-11T09:52:23.790 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.1149999999999994e-06 2024-09-11T09:52:23.790 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.790 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.791 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:23.791 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919757+0000", 2024-09-11T09:52:23.791 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.2348999999999995e-05 2024-09-11T09:52:23.791 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.791 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.791 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:23.791 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919875+0000", 2024-09-11T09:52:23.791 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00011750499999999999 2024-09-11T09:52:23.791 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.791 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.791 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 5", 2024-09-11T09:52:23.792 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.920201+0000", 2024-09-11T09:52:23.792 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00032628099999999999 2024-09-11T09:52:23.792 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.792 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.792 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:23.792 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.922089+0000", 2024-09-11T09:52:23.792 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.001887978 2024-09-11T09:52:23.792 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.792 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.792 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:23.792 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.931752+0000", 2024-09-11T09:52:23.792 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0096626309999999997 2024-09-11T09:52:23.792 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.793 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.793 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:23.793 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.931794+0000", 2024-09-11T09:52:23.793 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.2849999999999998e-05 2024-09-11T09:52:23.793 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.793 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.793 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:23.793 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.931839+0000", 2024-09-11T09:52:23.793 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.4764e-05 2024-09-11T09:52:23.793 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:23.793 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:23.793 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:23.793 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.793 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.794 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:41 2.7 2:fe41411c:::rbd_object_map.1250f5e25b80:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:23.794 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.919716+0000", 2024-09-11T09:52:23.794 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 60.857049570000001, 2024-09-11T09:52:23.794 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.015641605999999999, 2024-09-11T09:52:23.794 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:23.794 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:23.794 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:23.794 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:23.794 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:52:23.794 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:52:23.794 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 41 2024-09-11T09:52:23.794 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.794 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:23.794 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.794 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:23.795 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919716+0000", 2024-09-11T09:52:23.795 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:23.795 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.795 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.795 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:23.795 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919716+0000", 2024-09-11T09:52:23.795 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:23.795 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.795 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.795 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:23.795 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919716+0000", 2024-09-11T09:52:23.795 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.9999999999999998e-07 2024-09-11T09:52:23.795 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.795 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.796 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:23.796 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919720+0000", 2024-09-11T09:52:23.796 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.6440000000000002e-06 2024-09-11T09:52:23.796 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.796 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.796 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:23.796 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919729+0000", 2024-09-11T09:52:23.796 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.1759999999999999e-06 2024-09-11T09:52:23.796 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.796 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.796 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:23.796 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919736+0000", 2024-09-11T09:52:23.796 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.5390000000000004e-06 2024-09-11T09:52:23.796 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.797 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.797 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:23.797 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.920669+0000", 2024-09-11T09:52:23.797 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00093275599999999997 2024-09-11T09:52:23.797 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.797 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.797 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:23.797 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.920797+0000", 2024-09-11T09:52:23.797 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00012893400000000001 2024-09-11T09:52:23.797 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.797 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.797 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 5", 2024-09-11T09:52:23.797 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.921269+0000", 2024-09-11T09:52:23.798 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00047128600000000001 2024-09-11T09:52:23.798 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.798 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.798 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:23.798 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.923198+0000", 2024-09-11T09:52:23.798 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0019289369999999999 2024-09-11T09:52:23.798 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.798 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.798 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:23.798 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.935300+0000", 2024-09-11T09:52:23.798 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.012102392 2024-09-11T09:52:23.798 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.798 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.798 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:23.799 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.935332+0000", 2024-09-11T09:52:23.799 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.2292000000000002e-05 2024-09-11T09:52:23.799 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.799 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.799 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:23.799 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.935358+0000", 2024-09-11T09:52:23.799 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.525e-05 2024-09-11T09:52:23.799 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:23.799 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:23.799 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:23.799 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.799 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.799 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:48 2.7 2:fe41411c:::rbd_object_map.1250f5e25b80:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:23.800 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.938799+0000", 2024-09-11T09:52:23.800 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 60.837966461000001, 2024-09-11T09:52:23.800 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0041642340000000002, 2024-09-11T09:52:23.800 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:23.800 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:23.800 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:23.800 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:23.800 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:52:23.800 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:52:23.800 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 48 2024-09-11T09:52:23.800 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.800 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:23.800 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.801 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:23.801 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.938799+0000", 2024-09-11T09:52:23.801 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:23.801 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.801 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.801 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:23.801 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.938799+0000", 2024-09-11T09:52:23.801 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:23.801 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.801 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.801 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:23.801 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.938800+0000", 2024-09-11T09:52:23.801 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.71e-07 2024-09-11T09:52:23.801 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.802 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.802 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:23.802 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.938805+0000", 2024-09-11T09:52:23.802 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.3140000000000003e-06 2024-09-11T09:52:23.802 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.802 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.802 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:23.802 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.938816+0000", 2024-09-11T09:52:23.802 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0231000000000001e-05 2024-09-11T09:52:23.802 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.802 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.802 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:23.802 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.938822+0000", 2024-09-11T09:52:23.802 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.9550000000000003e-06 2024-09-11T09:52:23.803 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.803 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.803 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:23.803 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.938868+0000", 2024-09-11T09:52:23.803 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.5562999999999997e-05 2024-09-11T09:52:23.803 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.803 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.803 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:23.803 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.938918+0000", 2024-09-11T09:52:23.803 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.0317e-05 2024-09-11T09:52:23.803 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.803 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.803 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 5", 2024-09-11T09:52:23.803 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.939114+0000", 2024-09-11T09:52:23.804 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00019538300000000001 2024-09-11T09:52:23.804 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.804 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.804 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:23.804 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.939663+0000", 2024-09-11T09:52:23.804 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00054932199999999998 2024-09-11T09:52:23.804 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.804 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.804 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:23.804 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.942910+0000", 2024-09-11T09:52:23.804 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0032473839999999999 2024-09-11T09:52:23.804 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.804 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.804 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:23.805 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.942940+0000", 2024-09-11T09:52:23.805 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.9306999999999999e-05 2024-09-11T09:52:23.805 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.805 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.805 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:23.805 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.942963+0000", 2024-09-11T09:52:23.805 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.3587e-05 2024-09-11T09:52:23.805 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:23.805 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:23.805 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:23.805 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.805 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.805 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4688.0:49 2.5 e53/32 2:b6dd9fc9:::rbd_data.1250f5e25b80.0000000000000009:head v 53'2, mlcod=0'0)", 2024-09-11T09:52:23.806 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.944712+0000", 2024-09-11T09:52:23.806 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 60.832053621999997, 2024-09-11T09:52:23.806 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.016185069999999999, 2024-09-11T09:52:23.806 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:23.806 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:23.806 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:23.806 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:23.806 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.806 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:23.806 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.944712+0000", 2024-09-11T09:52:23.806 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:23.806 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.806 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.807 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:23.807 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.944712+0000", 2024-09-11T09:52:23.807 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:23.807 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.807 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.807 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:23.807 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.944713+0000", 2024-09-11T09:52:23.807 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0139999999999999e-06 2024-09-11T09:52:23.807 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.807 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.807 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:23.807 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.955518+0000", 2024-09-11T09:52:23.807 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.01080542 2024-09-11T09:52:23.808 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.808 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.808 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:23.808 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.955536+0000", 2024-09-11T09:52:23.808 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7818000000000001e-05 2024-09-11T09:52:23.808 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.808 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.808 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:23.808 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.955544+0000", 2024-09-11T09:52:23.808 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.1699999999999997e-06 2024-09-11T09:52:23.808 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.808 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.808 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:23.809 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.955607+0000", 2024-09-11T09:52:23.809 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.2451000000000002e-05 2024-09-11T09:52:23.809 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.809 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.809 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:23.809 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.955632+0000", 2024-09-11T09:52:23.809 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.5335000000000001e-05 2024-09-11T09:52:23.809 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.809 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.809 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:23.809 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.960826+0000", 2024-09-11T09:52:23.809 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0051938560000000002 2024-09-11T09:52:23.809 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.810 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.810 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:23.810 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.960897+0000", 2024-09-11T09:52:23.810 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.1006000000000004e-05 2024-09-11T09:52:23.810 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:23.810 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:23.810 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:23.810 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.810 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.810 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:53 2.7 2:fe41411c:::rbd_object_map.1250f5e25b80:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:23.810 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.951291+0000", 2024-09-11T09:52:23.810 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 60.825474632000002, 2024-09-11T09:52:23.810 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0044964999999999996, 2024-09-11T09:52:23.811 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:23.811 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:23.811 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:23.811 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:23.811 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:52:23.811 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:52:23.811 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 53 2024-09-11T09:52:23.811 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.811 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:23.811 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.811 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:23.811 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951291+0000", 2024-09-11T09:52:23.811 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:23.812 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.812 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.812 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:23.812 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951291+0000", 2024-09-11T09:52:23.812 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:23.812 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.812 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.812 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:23.812 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951293+0000", 2024-09-11T09:52:23.812 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7889999999999999e-06 2024-09-11T09:52:23.812 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.812 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.812 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:23.813 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951302+0000", 2024-09-11T09:52:23.813 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.9819999999999997e-06 2024-09-11T09:52:23.813 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.813 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.813 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:23.813 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951324+0000", 2024-09-11T09:52:23.813 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.2141e-05 2024-09-11T09:52:23.813 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.813 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.813 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:23.813 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951338+0000", 2024-09-11T09:52:23.813 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3750999999999999e-05 2024-09-11T09:52:23.813 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.814 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.814 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:23.814 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951409+0000", 2024-09-11T09:52:23.814 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.1658999999999999e-05 2024-09-11T09:52:23.814 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.814 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.814 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:23.814 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951517+0000", 2024-09-11T09:52:23.814 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00010746799999999999 2024-09-11T09:52:23.814 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.814 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.814 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 5", 2024-09-11T09:52:23.814 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951942+0000", 2024-09-11T09:52:23.814 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00042524300000000001 2024-09-11T09:52:23.815 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.815 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.815 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:23.815 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.952872+0000", 2024-09-11T09:52:23.815 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00092976300000000001 2024-09-11T09:52:23.815 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.815 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.815 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:23.815 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.955682+0000", 2024-09-11T09:52:23.815 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0028099459999999998 2024-09-11T09:52:23.815 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.815 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.815 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:23.815 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.955746+0000", 2024-09-11T09:52:23.816 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.4084999999999994e-05 2024-09-11T09:52:23.816 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.816 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.816 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:23.816 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.955787+0000", 2024-09-11T09:52:23.816 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.1672999999999999e-05 2024-09-11T09:52:23.816 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:23.816 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:23.816 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:23.816 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.816 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.816 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4688.0:52 2.4 e53/32 2:26d5c2f0:::rbd_data.1250f5e25b80.000000000000000b:head v 53'8, mlcod=53'6)", 2024-09-11T09:52:23.816 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.963677+0000", 2024-09-11T09:52:23.816 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 60.813089077000001, 2024-09-11T09:52:23.817 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0089437969999999999, 2024-09-11T09:52:23.817 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:23.817 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:23.817 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:23.817 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:23.817 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.817 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:23.817 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.963677+0000", 2024-09-11T09:52:23.817 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:23.817 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.817 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.817 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:23.817 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.963677+0000", 2024-09-11T09:52:23.818 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:23.818 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.818 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.818 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:23.818 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.963677+0000", 2024-09-11T09:52:23.818 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.7199999999999998e-07 2024-09-11T09:52:23.818 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.818 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.818 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:23.818 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.965506+0000", 2024-09-11T09:52:23.818 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.001829064 2024-09-11T09:52:23.818 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.818 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.819 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:23.819 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.965538+0000", 2024-09-11T09:52:23.819 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.1306999999999997e-05 2024-09-11T09:52:23.819 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.819 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.819 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:23.819 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.965553+0000", 2024-09-11T09:52:23.819 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.4924e-05 2024-09-11T09:52:23.819 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.819 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.819 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:23.819 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.965663+0000", 2024-09-11T09:52:23.819 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000110315 2024-09-11T09:52:23.819 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.820 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.820 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:23.820 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.965692+0000", 2024-09-11T09:52:23.820 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.934e-05 2024-09-11T09:52:23.820 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.820 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.820 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:23.820 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.972537+0000", 2024-09-11T09:52:23.820 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0068444969999999997 2024-09-11T09:52:23.820 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.820 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.820 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:23.820 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.972620+0000", 2024-09-11T09:52:23.821 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.3578000000000005e-05 2024-09-11T09:52:23.821 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:23.821 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:23.821 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:23.821 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.821 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.821 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4688.0:54 2.4 e53/32 2:239e915b:::rbd_data.1250f5e25b80.000000000000000c:head v 53'9, mlcod=53'7)", 2024-09-11T09:52:23.821 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.987028+0000", 2024-09-11T09:52:23.821 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 60.789737160000001, 2024-09-11T09:52:23.821 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0094155650000000007, 2024-09-11T09:52:23.821 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:23.821 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:23.821 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:23.822 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:23.822 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.822 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:23.822 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.987028+0000", 2024-09-11T09:52:23.822 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:23.822 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.822 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.822 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:23.822 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.987028+0000", 2024-09-11T09:52:23.822 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:23.822 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.822 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.822 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:23.822 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.987030+0000", 2024-09-11T09:52:23.823 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6130000000000001e-06 2024-09-11T09:52:23.823 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.823 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.823 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:23.823 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.988575+0000", 2024-09-11T09:52:23.823 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.001545193 2024-09-11T09:52:23.823 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.823 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.823 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:23.823 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.988604+0000", 2024-09-11T09:52:23.823 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.9173e-05 2024-09-11T09:52:23.823 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.823 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.824 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:23.824 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.988620+0000", 2024-09-11T09:52:23.824 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.5517999999999999e-05 2024-09-11T09:52:23.824 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.824 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.824 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:23.824 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.988704+0000", 2024-09-11T09:52:23.824 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.3588e-05 2024-09-11T09:52:23.824 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.824 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.824 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:23.824 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.988735+0000", 2024-09-11T09:52:23.824 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.1643000000000002e-05 2024-09-11T09:52:23.824 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.825 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.825 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:23.825 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.996383+0000", 2024-09-11T09:52:23.825 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0076478099999999997 2024-09-11T09:52:23.825 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.825 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.825 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:23.825 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.996444+0000", 2024-09-11T09:52:23.825 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.1026999999999997e-05 2024-09-11T09:52:23.825 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:23.825 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:23.825 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:23.825 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.826 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.826 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4688.0:56 2.4 e53/32 2:2167012f:::rbd_data.1250f5e25b80.000000000000000d:head v 53'10, mlcod=53'7)", 2024-09-11T09:52:23.826 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.993894+0000", 2024-09-11T09:52:23.826 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 60.782871618000001, 2024-09-11T09:52:23.826 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0099268559999999995, 2024-09-11T09:52:23.826 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:23.826 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:23.826 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:23.826 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:23.826 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.826 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:23.826 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.993894+0000", 2024-09-11T09:52:23.826 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:23.826 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.826 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.827 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:23.827 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.993894+0000", 2024-09-11T09:52:23.827 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:23.827 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.827 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.827 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:23.827 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.993896+0000", 2024-09-11T09:52:23.827 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6249999999999999e-06 2024-09-11T09:52:23.827 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.827 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.827 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:23.827 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.997606+0000", 2024-09-11T09:52:23.827 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.003710737 2024-09-11T09:52:23.827 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.828 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.828 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:23.828 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.997628+0000", 2024-09-11T09:52:23.828 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.1971e-05 2024-09-11T09:52:23.828 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.828 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.828 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:23.828 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.997640+0000", 2024-09-11T09:52:23.828 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1481000000000001e-05 2024-09-11T09:52:23.828 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.828 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.828 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:23.828 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.997712+0000", 2024-09-11T09:52:23.828 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.2255000000000005e-05 2024-09-11T09:52:23.828 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.829 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.829 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:23.829 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.997738+0000", 2024-09-11T09:52:23.829 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.6050999999999998e-05 2024-09-11T09:52:23.829 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.829 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.829 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:23.829 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.003779+0000", 2024-09-11T09:52:23.829 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0060410960000000001 2024-09-11T09:52:23.829 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.829 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.829 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:23.829 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.003821+0000", 2024-09-11T09:52:23.830 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.1640000000000001e-05 2024-09-11T09:52:23.830 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:23.830 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:23.830 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:23.830 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.830 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.830 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "pg_update_log_missing(2.4 epoch 53/32 rep_tid 16 entries 53'11 (0'0) error 2:2906e6a7:::rbd_header.1250f5e25b80:head by client.4688.0:59 0.000000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 53'7)", 2024-09-11T09:52:23.830 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.998956+0000", 2024-09-11T09:52:23.830 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 60.777809449000003, 2024-09-11T09:52:23.830 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.004934127, 2024-09-11T09:52:23.830 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:23.830 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:23.830 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "reached pg", 2024-09-11T09:52:23.830 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:23.831 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.831 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:23.831 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.998956+0000", 2024-09-11T09:52:23.831 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:23.831 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.831 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.831 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:23.831 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.998956+0000", 2024-09-11T09:52:23.831 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:23.831 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.831 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.831 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:23.831 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.998957+0000", 2024-09-11T09:52:23.832 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.139e-06 2024-09-11T09:52:23.832 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.832 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.832 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:23.832 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.998985+0000", 2024-09-11T09:52:23.832 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.7912e-05 2024-09-11T09:52:23.832 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.832 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.832 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:23.832 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.999011+0000", 2024-09-11T09:52:23.832 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.6061e-05 2024-09-11T09:52:23.832 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.832 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.832 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:23.833 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.999028+0000", 2024-09-11T09:52:23.833 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7065999999999999e-05 2024-09-11T09:52:23.833 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.833 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.833 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:23.833 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.001919+0000", 2024-09-11T09:52:23.833 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.002891167 2024-09-11T09:52:23.833 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.833 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.833 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:23.833 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.003890+0000", 2024-09-11T09:52:23.833 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0019707819999999999 2024-09-11T09:52:23.833 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:23.834 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:23.834 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:23.834 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.834 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.834 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4694.0:34 2.7 2:f80bfc0c:ns::rbd_data.1256a0c02879.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:23.834 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.085183+0000", 2024-09-11T09:52:23.834 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 60.691582936000003, 2024-09-11T09:52:23.834 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.020908599, 2024-09-11T09:52:23.834 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:23.834 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:23.834 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:23.834 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:23.834 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4694", 2024-09-11T09:52:23.834 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:52:23.835 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 34 2024-09-11T09:52:23.835 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.835 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:23.835 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.835 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:23.835 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.085183+0000", 2024-09-11T09:52:23.835 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:23.835 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.835 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.835 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:23.835 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.085183+0000", 2024-09-11T09:52:23.835 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:23.835 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.836 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.836 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:23.836 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.085183+0000", 2024-09-11T09:52:23.836 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.4300000000000002e-07 2024-09-11T09:52:23.836 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.836 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.836 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:23.836 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094337+0000", 2024-09-11T09:52:23.836 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0091532769999999996 2024-09-11T09:52:23.836 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.836 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.836 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:23.836 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094362+0000", 2024-09-11T09:52:23.836 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.5287999999999999e-05 2024-09-11T09:52:23.837 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.837 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.837 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:23.837 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094375+0000", 2024-09-11T09:52:23.837 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2446000000000001e-05 2024-09-11T09:52:23.837 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.837 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.837 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:23.837 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094455+0000", 2024-09-11T09:52:23.837 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.0092999999999994e-05 2024-09-11T09:52:23.837 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.837 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.837 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:23.838 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094618+0000", 2024-09-11T09:52:23.838 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000163747 2024-09-11T09:52:23.838 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.838 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.838 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 5", 2024-09-11T09:52:23.838 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094783+0000", 2024-09-11T09:52:23.838 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000165038 2024-09-11T09:52:23.838 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.838 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.838 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:23.838 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.105853+0000", 2024-09-11T09:52:23.838 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.011069294 2024-09-11T09:52:23.838 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.838 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.839 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:23.839 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.106004+0000", 2024-09-11T09:52:23.839 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00015123900000000001 2024-09-11T09:52:23.839 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.839 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.839 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:23.839 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.106062+0000", 2024-09-11T09:52:23.839 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.8205000000000003e-05 2024-09-11T09:52:23.839 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.839 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.839 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:23.839 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.106091+0000", 2024-09-11T09:52:23.839 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.9128999999999998e-05 2024-09-11T09:52:23.839 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:23.839 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:23.840 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:23.840 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.840 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.840 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4694.0:35 2.7 2:e8766325:ns::rbd_object_map.1256a0c02879:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:23.840 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.094405+0000", 2024-09-11T09:52:23.840 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 60.682360299999999, 2024-09-11T09:52:23.840 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.012623564, 2024-09-11T09:52:23.840 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:23.840 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:23.840 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:23.840 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:23.840 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4694", 2024-09-11T09:52:23.840 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:52:23.840 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 35 2024-09-11T09:52:23.841 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.841 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:23.841 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.841 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:23.841 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094405+0000", 2024-09-11T09:52:23.841 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:23.841 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.841 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.841 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:23.841 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094405+0000", 2024-09-11T09:52:23.841 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:23.841 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.841 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.841 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:23.841 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094407+0000", 2024-09-11T09:52:23.842 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.739e-06 2024-09-11T09:52:23.842 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.842 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.842 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:23.842 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094411+0000", 2024-09-11T09:52:23.842 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.3340000000000003e-06 2024-09-11T09:52:23.842 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.842 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.842 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:23.842 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094419+0000", 2024-09-11T09:52:23.842 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.8909999999999995e-06 2024-09-11T09:52:23.842 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.842 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.843 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:23.843 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094424+0000", 2024-09-11T09:52:23.843 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.7639999999999996e-06 2024-09-11T09:52:23.843 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.843 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.843 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:23.843 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.100209+0000", 2024-09-11T09:52:23.843 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0057850489999999996 2024-09-11T09:52:23.843 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.843 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.843 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:23.843 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.100341+0000", 2024-09-11T09:52:23.844 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000131438 2024-09-11T09:52:23.844 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.844 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.844 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 5", 2024-09-11T09:52:23.844 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.100805+0000", 2024-09-11T09:52:23.844 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00046471499999999998 2024-09-11T09:52:23.844 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.844 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.844 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:23.844 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.106148+0000", 2024-09-11T09:52:23.844 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0053426460000000004 2024-09-11T09:52:23.844 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.844 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.844 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:23.845 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.106929+0000", 2024-09-11T09:52:23.845 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00078161300000000001 2024-09-11T09:52:23.845 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.845 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.845 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:23.845 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.106985+0000", 2024-09-11T09:52:23.845 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.5223999999999998e-05 2024-09-11T09:52:23.845 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.845 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.845 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:23.845 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.107029+0000", 2024-09-11T09:52:23.845 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.4150999999999998e-05 2024-09-11T09:52:23.845 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:23.845 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:23.846 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:23.846 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.846 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.846 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4694.0:36 2.7 2:e8766325:ns::rbd_object_map.1256a0c02879:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:23.846 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.094452+0000", 2024-09-11T09:52:23.846 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 60.682313290000003, 2024-09-11T09:52:23.846 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.012680383, 2024-09-11T09:52:23.846 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:23.846 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:23.846 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:23.846 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:23.846 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4694", 2024-09-11T09:52:23.847 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:52:23.847 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 36 2024-09-11T09:52:23.847 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.847 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:23.847 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.847 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:23.847 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094452+0000", 2024-09-11T09:52:23.847 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:23.847 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.847 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.847 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:23.847 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094452+0000", 2024-09-11T09:52:23.847 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:23.847 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.848 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.848 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:23.848 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094453+0000", 2024-09-11T09:52:23.848 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.1800000000000002e-07 2024-09-11T09:52:23.848 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.848 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.848 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:23.848 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094455+0000", 2024-09-11T09:52:23.848 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.7020000000000001e-06 2024-09-11T09:52:23.848 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.848 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.848 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:23.848 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094463+0000", 2024-09-11T09:52:23.848 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.1999999999999997e-06 2024-09-11T09:52:23.849 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.849 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.849 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:23.849 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094467+0000", 2024-09-11T09:52:23.849 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.729e-06 2024-09-11T09:52:23.849 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.849 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.849 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:23.849 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.101269+0000", 2024-09-11T09:52:23.849 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0068018799999999997 2024-09-11T09:52:23.849 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.849 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.849 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:23.849 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.101353+0000", 2024-09-11T09:52:23.850 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.3827999999999998e-05 2024-09-11T09:52:23.850 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.850 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.850 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 5", 2024-09-11T09:52:23.850 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.101588+0000", 2024-09-11T09:52:23.850 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00023467099999999999 2024-09-11T09:52:23.850 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.850 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.850 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:23.850 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.106182+0000", 2024-09-11T09:52:23.850 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0045939489999999999 2024-09-11T09:52:23.850 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.850 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.850 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:23.850 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.107041+0000", 2024-09-11T09:52:23.851 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00085965800000000002 2024-09-11T09:52:23.851 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.851 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.851 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:23.851 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.107084+0000", 2024-09-11T09:52:23.851 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.261e-05 2024-09-11T09:52:23.851 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.851 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.851 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:23.851 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.107133+0000", 2024-09-11T09:52:23.851 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.8838000000000002e-05 2024-09-11T09:52:23.851 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:23.851 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:23.852 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:23.852 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.852 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.852 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4694.0:37 2.7 2:e8766325:ns::rbd_object_map.1256a0c02879:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:23.852 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.094473+0000", 2024-09-11T09:52:23.852 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 60.682292138999998, 2024-09-11T09:52:23.852 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.012778484999999999, 2024-09-11T09:52:23.852 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:23.852 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:23.852 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:23.852 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:23.852 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4694", 2024-09-11T09:52:23.852 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:52:23.852 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 37 2024-09-11T09:52:23.853 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.853 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:23.853 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.853 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:23.853 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094473+0000", 2024-09-11T09:52:23.853 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:23.853 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.853 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.853 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:23.853 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094473+0000", 2024-09-11T09:52:23.853 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:23.853 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.853 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.854 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:23.854 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094474+0000", 2024-09-11T09:52:23.854 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.9699999999999996e-07 2024-09-11T09:52:23.854 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.854 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.854 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:23.854 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094477+0000", 2024-09-11T09:52:23.854 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.151e-06 2024-09-11T09:52:23.854 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.854 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.854 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:23.854 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094483+0000", 2024-09-11T09:52:23.854 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.6330000000000001e-06 2024-09-11T09:52:23.854 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.855 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.855 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:23.855 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094487+0000", 2024-09-11T09:52:23.855 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.5399999999999997e-06 2024-09-11T09:52:23.855 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.855 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.855 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:23.855 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.101866+0000", 2024-09-11T09:52:23.855 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.007378768 2024-09-11T09:52:23.855 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.855 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.855 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:23.855 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.101960+0000", 2024-09-11T09:52:23.855 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.3861000000000001e-05 2024-09-11T09:52:23.856 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.856 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.856 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 5", 2024-09-11T09:52:23.856 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.102197+0000", 2024-09-11T09:52:23.856 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00023723200000000001 2024-09-11T09:52:23.856 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.856 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.856 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:23.856 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.106222+0000", 2024-09-11T09:52:23.856 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0040249689999999998 2024-09-11T09:52:23.856 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.856 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.856 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:23.856 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.107141+0000", 2024-09-11T09:52:23.857 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00091913900000000002 2024-09-11T09:52:23.857 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.857 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.857 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:23.857 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.107198+0000", 2024-09-11T09:52:23.857 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.6564999999999998e-05 2024-09-11T09:52:23.857 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.857 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.857 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:23.857 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.107252+0000", 2024-09-11T09:52:23.857 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.4030000000000003e-05 2024-09-11T09:52:23.857 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:23.857 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:23.857 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:23.858 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.858 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.858 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4694.0:33 2.4 e53/32 2:3bfc818c:ns::rbd_data.1256a0c02879.0000000000000001:head v 53'14, mlcod=53'12)", 2024-09-11T09:52:23.858 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.096496+0000", 2024-09-11T09:52:23.858 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 60.680269566, 2024-09-11T09:52:23.858 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0099139310000000008, 2024-09-11T09:52:23.858 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:23.858 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:23.858 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:23.858 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:23.858 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.858 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:23.858 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.096496+0000", 2024-09-11T09:52:23.858 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:23.859 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.859 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.859 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:23.859 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.096496+0000", 2024-09-11T09:52:23.859 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:23.859 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.859 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.859 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:23.859 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.096498+0000", 2024-09-11T09:52:23.859 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6309999999999999e-06 2024-09-11T09:52:23.859 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.859 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.859 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:23.859 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.098087+0000", 2024-09-11T09:52:23.859 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0015889019999999999 2024-09-11T09:52:23.860 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.860 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.860 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:23.860 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.098112+0000", 2024-09-11T09:52:23.860 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.5369000000000001e-05 2024-09-11T09:52:23.860 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.860 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.860 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:23.860 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.098127+0000", 2024-09-11T09:52:23.860 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.5218999999999999e-05 2024-09-11T09:52:23.860 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.860 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.860 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:23.860 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.098209+0000", 2024-09-11T09:52:23.860 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.2120999999999995e-05 2024-09-11T09:52:23.861 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.861 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.861 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:23.861 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.098237+0000", 2024-09-11T09:52:23.861 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.7657999999999998e-05 2024-09-11T09:52:23.861 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.861 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.861 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:23.861 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.106376+0000", 2024-09-11T09:52:23.861 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0081386749999999997 2024-09-11T09:52:23.861 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.861 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.861 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:23.861 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.106410+0000", 2024-09-11T09:52:23.861 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.4356000000000001e-05 2024-09-11T09:52:23.862 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:23.862 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:23.862 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:23.862 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.862 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.862 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4694.0:49 2.7 2:e8766325:ns::rbd_object_map.1256a0c02879:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:23.862 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.134806+0000", 2024-09-11T09:52:23.862 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 60.641959497000002, 2024-09-11T09:52:23.862 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0061629049999999998, 2024-09-11T09:52:23.862 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:23.862 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:23.862 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:23.862 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:23.862 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4694", 2024-09-11T09:52:23.863 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:52:23.863 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 49 2024-09-11T09:52:23.863 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.863 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:23.863 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.863 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:23.863 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.134806+0000", 2024-09-11T09:52:23.863 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:23.863 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.863 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.863 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:23.863 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.134806+0000", 2024-09-11T09:52:23.863 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:23.863 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.864 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.864 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:23.864 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.134808+0000", 2024-09-11T09:52:23.864 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.9870000000000002e-06 2024-09-11T09:52:23.864 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.864 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.864 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:23.864 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.134818+0000", 2024-09-11T09:52:23.864 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.5599999999999999e-06 2024-09-11T09:52:23.864 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.864 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.864 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:23.864 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.134835+0000", 2024-09-11T09:52:23.864 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7399000000000001e-05 2024-09-11T09:52:23.864 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.865 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.865 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:23.865 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.134845+0000", 2024-09-11T09:52:23.865 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0285000000000001e-05 2024-09-11T09:52:23.865 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.865 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.865 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:23.865 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.134911+0000", 2024-09-11T09:52:23.865 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.5654999999999994e-05 2024-09-11T09:52:23.865 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.865 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.865 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:23.865 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.135023+0000", 2024-09-11T09:52:23.865 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00011186799999999999 2024-09-11T09:52:23.865 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.866 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.866 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 5", 2024-09-11T09:52:23.866 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.135290+0000", 2024-09-11T09:52:23.866 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00026725300000000001 2024-09-11T09:52:23.866 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.866 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.866 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:23.866 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.136506+0000", 2024-09-11T09:52:23.866 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0012161139999999999 2024-09-11T09:52:23.866 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.866 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.866 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:23.866 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.140884+0000", 2024-09-11T09:52:23.866 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0043781990000000002 2024-09-11T09:52:23.866 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.867 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.867 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:23.867 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.140923+0000", 2024-09-11T09:52:23.867 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.8813999999999998e-05 2024-09-11T09:52:23.867 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.867 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.867 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:23.867 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.140969+0000", 2024-09-11T09:52:23.867 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.5770999999999999e-05 2024-09-11T09:52:23.867 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:23.867 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:23.867 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:23.867 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.867 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.867 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4694.0:42 2.5 e53/32 2:a4fc00ec:ns::rbd_data.1256a0c02879.0000000000000004:head v 53'3, mlcod=53'1)", 2024-09-11T09:52:23.867 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.136646+0000", 2024-09-11T09:52:23.868 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 60.640119190999997, 2024-09-11T09:52:23.868 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.012545501000000001, 2024-09-11T09:52:23.868 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:23.868 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:23.868 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:23.868 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:23.868 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.868 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:23.868 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.136646+0000", 2024-09-11T09:52:23.868 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:23.868 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.868 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.868 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:23.868 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.136646+0000", 2024-09-11T09:52:23.869 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:23.869 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.869 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.869 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:23.869 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.136647+0000", 2024-09-11T09:52:23.869 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.2900000000000002e-07 2024-09-11T09:52:23.869 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.869 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.869 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:23.869 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.143620+0000", 2024-09-11T09:52:23.869 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0069727440000000003 2024-09-11T09:52:23.869 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.869 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.869 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:23.870 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.143642+0000", 2024-09-11T09:52:23.870 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.1846000000000001e-05 2024-09-11T09:52:23.870 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.870 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.870 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:23.870 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.143654+0000", 2024-09-11T09:52:23.870 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2081e-05 2024-09-11T09:52:23.870 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.870 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.870 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:23.870 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.143717+0000", 2024-09-11T09:52:23.870 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.3443999999999998e-05 2024-09-11T09:52:23.871 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.871 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.871 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:23.871 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.143740+0000", 2024-09-11T09:52:23.871 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.2626999999999999e-05 2024-09-11T09:52:23.871 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.871 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.871 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:23.871 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.149142+0000", 2024-09-11T09:52:23.871 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0054017630000000004 2024-09-11T09:52:23.871 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.871 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.871 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:23.871 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.149192+0000", 2024-09-11T09:52:23.871 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.0067e-05 2024-09-11T09:52:23.872 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:23.872 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:23.872 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:23.872 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.872 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.872 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4694.0:53 2.7 2:e8766325:ns::rbd_object_map.1256a0c02879:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:23.872 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.147978+0000", 2024-09-11T09:52:23.872 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 60.628787707000001, 2024-09-11T09:52:23.872 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0044549769999999997, 2024-09-11T09:52:23.872 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:23.872 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:23.872 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:23.873 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:23.873 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4694", 2024-09-11T09:52:23.873 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:52:23.873 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 53 2024-09-11T09:52:23.873 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.873 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:23.873 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.873 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:23.873 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.147978+0000", 2024-09-11T09:52:23.873 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:23.873 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.873 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.873 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:23.873 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.147978+0000", 2024-09-11T09:52:23.873 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:23.874 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.874 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.874 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:23.874 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.147980+0000", 2024-09-11T09:52:23.874 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.8309999999999999e-06 2024-09-11T09:52:23.874 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.874 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.874 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:23.874 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.147989+0000", 2024-09-11T09:52:23.874 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.9639999999999992e-06 2024-09-11T09:52:23.874 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.874 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.874 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:23.875 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.148007+0000", 2024-09-11T09:52:23.875 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7884999999999999e-05 2024-09-11T09:52:23.875 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.875 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.875 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:23.875 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.148017+0000", 2024-09-11T09:52:23.875 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0895e-05 2024-09-11T09:52:23.875 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.875 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.875 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:23.875 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.148091+0000", 2024-09-11T09:52:23.875 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.3709999999999997e-05 2024-09-11T09:52:23.875 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.875 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.876 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:23.876 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.148189+0000", 2024-09-11T09:52:23.876 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.7403000000000002e-05 2024-09-11T09:52:23.876 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.876 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.876 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 5", 2024-09-11T09:52:23.876 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.148504+0000", 2024-09-11T09:52:23.876 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00031589900000000002 2024-09-11T09:52:23.876 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.876 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.876 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:23.876 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.149514+0000", 2024-09-11T09:52:23.876 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0010096 2024-09-11T09:52:23.876 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.877 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.877 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:23.877 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.152367+0000", 2024-09-11T09:52:23.877 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0028532309999999999 2024-09-11T09:52:23.877 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.877 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.877 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:23.877 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.152402+0000", 2024-09-11T09:52:23.877 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.4232e-05 2024-09-11T09:52:23.877 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.877 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.877 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:23.877 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.152433+0000", 2024-09-11T09:52:23.877 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.1327e-05 2024-09-11T09:52:23.877 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:23.878 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:23.878 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:23.878 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.878 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.878 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4694.0:55 2.7 2:e8766325:ns::rbd_object_map.1256a0c02879:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:23.878 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.149838+0000", 2024-09-11T09:52:23.878 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 60.626927973999997, 2024-09-11T09:52:23.878 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0060695159999999996, 2024-09-11T09:52:23.878 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:23.878 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:23.878 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:23.878 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:23.878 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4694", 2024-09-11T09:52:23.878 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:52:23.879 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 55 2024-09-11T09:52:23.879 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.879 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:23.879 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.879 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:23.879 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.149838+0000", 2024-09-11T09:52:23.879 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:23.879 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.879 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.879 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:23.879 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.149838+0000", 2024-09-11T09:52:23.879 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:23.879 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.880 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.880 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:23.880 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.149839+0000", 2024-09-11T09:52:23.880 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.119e-06 2024-09-11T09:52:23.880 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.880 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.880 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:23.880 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.149845+0000", 2024-09-11T09:52:23.880 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.7810000000000002e-06 2024-09-11T09:52:23.880 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.880 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.880 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:23.880 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.149856+0000", 2024-09-11T09:52:23.880 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1199999999999999e-05 2024-09-11T09:52:23.881 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.881 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.881 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:23.881 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.149863+0000", 2024-09-11T09:52:23.881 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.4109999999999999e-06 2024-09-11T09:52:23.881 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.881 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.881 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:23.881 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.149908+0000", 2024-09-11T09:52:23.881 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.5184e-05 2024-09-11T09:52:23.881 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.881 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.881 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:23.882 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.149964+0000", 2024-09-11T09:52:23.882 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.5395000000000001e-05 2024-09-11T09:52:23.882 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.882 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.882 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 5", 2024-09-11T09:52:23.882 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.150157+0000", 2024-09-11T09:52:23.882 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000193489 2024-09-11T09:52:23.882 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.882 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.882 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:23.882 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.152238+0000", 2024-09-11T09:52:23.882 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0020803480000000001 2024-09-11T09:52:23.883 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.883 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.883 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:23.883 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.155828+0000", 2024-09-11T09:52:23.883 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0035902210000000002 2024-09-11T09:52:23.883 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.883 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.883 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:23.883 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.155873+0000", 2024-09-11T09:52:23.883 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.5145999999999998e-05 2024-09-11T09:52:23.883 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:23.883 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:23.883 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:23.883 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.155907+0000", 2024-09-11T09:52:23.884 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.4221999999999999e-05 2024-09-11T09:52:23.884 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:23.884 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:23.884 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:23.884 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:23.884 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:23.884 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:52:23.885 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T09:52:23.887 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-11T09:52:23.887 INFO:tasks.ceph.osd.2.smithi017.stderr:2024-09-11T09:52:23.886+0000 7fd20b170640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31672) UID: 0 2024-09-11T09:52:23.887 INFO:tasks.workunit.client.0.smithi107.stderr:+ rbd device --device-type nbd list 2024-09-11T09:52:23.887 INFO:tasks.workunit.client.0.smithi107.stderr:+ expect_false grep '^30702\b' 2024-09-11T09:52:23.887 INFO:tasks.workunit.client.0.smithi107.stderr:+ grep '^30702\b' 2024-09-11T09:52:23.888 INFO:tasks.workunit.client.0.smithi107.stderr:+ return 0 2024-09-11T09:52:23.888 INFO:tasks.workunit.client.0.smithi107.stderr:+ ps -C rbd-nbd 2024-09-11T09:52:23.888 INFO:tasks.workunit.client.0.smithi107.stderr:+ expect_false grep '^ *30702\b' 2024-09-11T09:52:23.888 INFO:tasks.workunit.client.0.smithi107.stderr:+ grep '^ *30702\b' 2024-09-11T09:52:23.888 INFO:tasks.workunit.client.0.smithi107.stderr:+ return 0 2024-09-11T09:52:23.888 INFO:tasks.workunit.client.0.smithi107.stderr:+ sleep 0.5 2024-09-11T09:52:23.900 INFO:teuthology.orchestra.run.smithi017.stderr:reweighted osd.5 to 1 (10000) 2024-09-11T09:52:23.911 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-11T09:52:23.915 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-09-11T09:52:23.915 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T09:52:23.999 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:52:24.000+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:52:24.013 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:52:24.013 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [], 2024-09-11T09:52:24.013 INFO:teuthology.orchestra.run.smithi017.stdout: "num_ops": 0 2024-09-11T09:52:24.013 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:52:24.022 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-11T09:52:24.027 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-11T09:52:24.027 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T09:52:24.045 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-11T09:52:24.045 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T09:52:24.066 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-11T09:52:24.066 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T09:52:24.086 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:52:24.088+0000 7f3a7b90a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 42475) UID: 0 2024-09-11T09:52:24.099 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-11T09:52:24.099 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T09:52:24.131 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-11T09:52:24.131 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T09:52:24.164 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-11T09:52:24.164 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T09:52:24.174 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:52:24.175 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:52:24.185 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-11T09:52:24.187 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:52:24.189+0000 7f3c96ec9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31909) UID: 0 2024-09-11T09:52:24.254 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-11T09:52:24.254 INFO:teuthology.orchestra.run.smithi017.stderr:dumped all 2024-09-11T09:52:24.267 INFO:teuthology.orchestra.run.smithi017.stdout:{"pg_ready":true,"pg_map":{"version":154,"stamp":"2024-09-11T09:52:23.900393+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":149531867,"num_objects":81,"num_object_clones":19,"num_object_copies":162,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":81,"num_whiteouts":7,"num_read":6042,"num_read_kb":169249,"num_write":1001,"num_write_kb":275544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":41107170,"num_keys_recovered":14,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":971,"ondisk_log_size":971,"up":34,"acting":34,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":34,"num_osds":5,"num_per_pool_osds":5,"num_per_pool_omap_osds":5,"kb":471859200,"kb_used":437732,"kb_used_data":276528,"kb_used_omap":10,"kb_used_meta":161077,"kb_avail":471421468,"statfs":{"total":483183820800,"available":482735583232,"internally_reserved":0,"allocated":283164672,"data_stored":303404609,"data_compressed":587091,"data_compressed_allocated":671744,"data_compressed_original":11749376,"omap_allocated":10808,"internal_metadata":164943304},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":6456,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":1019,"num_read_kb":1733,"num_write":44,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.019312"},"pg_stats":[{"pgid":"1.0","version":"13'117","reported_seq":352,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-11T09:52:20.887613+0000","last_change":"2024-09-11T09:52:10.859352+0000","last_active":"2024-09-11T09:52:20.887613+0000","last_peered":"2024-09-11T09:52:20.887613+0000","last_clean":"2024-09-11T09:52:20.887613+0000","last_became_active":"2024-09-11T09:52:10.858918+0000","last_became_peered":"2024-09-11T09:52:10.858918+0000","last_unstale":"2024-09-11T09:52:20.887613+0000","last_undegraded":"2024-09-11T09:52:20.887613+0000","last_fullsized":"2024-09-11T09:52:20.887613+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:50:10.457262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:50:10.457262+0000","last_clean_scrub_stamp":"2024-09-11T09:50:10.457262+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:32:35.326531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"79'32","reported_seq":242,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-11T09:52:19.935661+0000","last_change":"2024-09-11T09:50:20.500687+0000","last_active":"2024-09-11T09:52:19.935661+0000","last_peered":"2024-09-11T09:52:19.935661+0000","last_clean":"2024-09-11T09:52:19.935661+0000","last_became_active":"2024-09-11T09:50:18.493650+0000","last_became_peered":"2024-09-11T09:50:18.493650+0000","last_unstale":"2024-09-11T09:52:19.935661+0000","last_undegraded":"2024-09-11T09:52:19.935661+0000","last_fullsized":"2024-09-11T09:52:19.935661+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_clean_scrub_stamp":"2024-09-11T09:50:17.477890+0000","objects_scrubbed":0,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:44:39.593950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041261100000000001,"stat_sum":{"num_bytes":25956352,"num_objects":8,"num_object_clones":1,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":47,"num_read_kb":12306,"num_write":30,"num_write_kb":33288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"67'3","reported_seq":89,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-11T09:52:19.882335+0000","last_change":"2024-09-11T09:51:27.719936+0000","last_active":"2024-09-11T09:52:19.882335+0000","last_peered":"2024-09-11T09:52:19.882335+0000","last_clean":"2024-09-11T09:52:19.882335+0000","last_became_active":"2024-09-11T09:51:24.714496+0000","last_became_peered":"2024-09-11T09:51:24.714496+0000","last_unstale":"2024-09-11T09:52:19.882335+0000","last_undegraded":"2024-09-11T09:52:19.882335+0000","last_fullsized":"2024-09-11T09:52:19.882335+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:51:23.699749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:51:23.699749+0000","last_clean_scrub_stamp":"2024-09-11T09:51:23.699749+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T20:58:07.190652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00068421499999999995,"stat_sum":{"num_bytes":54,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":5,"num_read_kb":4,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"74'6","reported_seq":330,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-11T09:52:20.887611+0000","last_change":"2024-09-11T09:50:20.498284+0000","last_active":"2024-09-11T09:52:20.887611+0000","last_peered":"2024-09-11T09:52:20.887611+0000","last_clean":"2024-09-11T09:52:20.887611+0000","last_became_active":"2024-09-11T09:50:18.493210+0000","last_became_peered":"2024-09-11T09:50:18.493210+0000","last_unstale":"2024-09-11T09:52:20.887611+0000","last_undegraded":"2024-09-11T09:52:20.887611+0000","last_fullsized":"2024-09-11T09:52:20.887611+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_clean_scrub_stamp":"2024-09-11T09:50:17.477890+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:06:18.020045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079749499999999995,"stat_sum":{"num_bytes":12582988,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":161,"num_read_kb":2852,"num_write":5,"num_write_kb":12290,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"67'1","reported_seq":76,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-11T09:52:22.896112+0000","last_change":"2024-09-11T09:52:07.849281+0000","last_active":"2024-09-11T09:52:22.896112+0000","last_peered":"2024-09-11T09:52:22.896112+0000","last_clean":"2024-09-11T09:52:22.896112+0000","last_became_active":"2024-09-11T09:52:07.845118+0000","last_became_peered":"2024-09-11T09:52:07.845118+0000","last_unstale":"2024-09-11T09:52:22.896112+0000","last_undegraded":"2024-09-11T09:52:22.896112+0000","last_fullsized":"2024-09-11T09:52:22.896112+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:51:23.699749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:51:23.699749+0000","last_clean_scrub_stamp":"2024-09-11T09:51:23.699749+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T21:07:40.901843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041844700000000001,"stat_sum":{"num_bytes":16,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":16,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1","version":"89'221","reported_seq":1818,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-11T09:52:20.105388+0000","last_change":"2024-09-11T09:50:21.030566+0000","last_active":"2024-09-11T09:52:20.105388+0000","last_peered":"2024-09-11T09:52:20.105388+0000","last_clean":"2024-09-11T09:52:20.105388+0000","last_became_active":"2024-09-11T09:50:18.493109+0000","last_became_peered":"2024-09-11T09:50:18.493109+0000","last_unstale":"2024-09-11T09:52:20.105388+0000","last_undegraded":"2024-09-11T09:52:20.105388+0000","last_fullsized":"2024-09-11T09:52:20.105388+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_clean_scrub_stamp":"2024-09-11T09:50:17.477890+0000","objects_scrubbed":0,"log_size":221,"log_dups_size":0,"ondisk_log_size":221,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T20:07:51.753387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041008499999999997,"stat_sum":{"num_bytes":31477930,"num_objects":17,"num_object_clones":6,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":1726,"num_read_kb":44414,"num_write":190,"num_write_kb":52637,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":82,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-11T09:52:19.882197+0000","last_change":"2024-09-11T09:51:27.719437+0000","last_active":"2024-09-11T09:52:19.882197+0000","last_peered":"2024-09-11T09:52:19.882197+0000","last_clean":"2024-09-11T09:52:19.882197+0000","last_became_active":"2024-09-11T09:51:24.714263+0000","last_became_peered":"2024-09-11T09:51:24.714263+0000","last_unstale":"2024-09-11T09:52:19.882197+0000","last_undegraded":"2024-09-11T09:52:19.882197+0000","last_fullsized":"2024-09-11T09:52:19.882197+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:51:23.699749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:51:23.699749+0000","last_clean_scrub_stamp":"2024-09-11T09:51:23.699749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:06:36.882446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045627500000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"87'25","reported_seq":351,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-11T09:52:20.089820+0000","last_change":"2024-09-11T09:50:20.501218+0000","last_active":"2024-09-11T09:52:20.089820+0000","last_peered":"2024-09-11T09:52:20.089820+0000","last_clean":"2024-09-11T09:52:20.089820+0000","last_became_active":"2024-09-11T09:50:18.493165+0000","last_became_peered":"2024-09-11T09:50:18.493165+0000","last_unstale":"2024-09-11T09:52:20.089820+0000","last_undegraded":"2024-09-11T09:52:20.089820+0000","last_fullsized":"2024-09-11T09:52:20.089820+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_clean_scrub_stamp":"2024-09-11T09:50:17.477890+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T19:11:24.668941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00094463900000000005,"stat_sum":{"num_bytes":12583009,"num_objects":6,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":164,"num_read_kb":8357,"num_write":24,"num_write_kb":20483,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"56'1","reported_seq":78,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-11T09:52:22.896112+0000","last_change":"2024-09-11T09:52:07.851088+0000","last_active":"2024-09-11T09:52:22.896112+0000","last_peered":"2024-09-11T09:52:22.896112+0000","last_clean":"2024-09-11T09:52:22.896112+0000","last_became_active":"2024-09-11T09:52:07.848151+0000","last_became_peered":"2024-09-11T09:52:07.848151+0000","last_unstale":"2024-09-11T09:52:22.896112+0000","last_undegraded":"2024-09-11T09:52:22.896112+0000","last_fullsized":"2024-09-11T09:52:22.896112+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:51:23.699749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:51:23.699749+0000","last_clean_scrub_stamp":"2024-09-11T09:51:23.699749+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T20:26:55.098291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040505599999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":82,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-11T09:52:19.935657+0000","last_change":"2024-09-11T09:51:27.729396+0000","last_active":"2024-09-11T09:52:19.935657+0000","last_peered":"2024-09-11T09:52:19.935657+0000","last_clean":"2024-09-11T09:52:19.935657+0000","last_became_active":"2024-09-11T09:51:24.715777+0000","last_became_peered":"2024-09-11T09:51:24.715777+0000","last_unstale":"2024-09-11T09:52:19.935657+0000","last_undegraded":"2024-09-11T09:52:19.935657+0000","last_fullsized":"2024-09-11T09:52:19.935657+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:51:23.699749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:51:23.699749+0000","last_clean_scrub_stamp":"2024-09-11T09:51:23.699749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:56:41.450651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026452099999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"87'226","reported_seq":1365,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-11T09:52:20.090158+0000","last_change":"2024-09-11T09:52:10.859987+0000","last_active":"2024-09-11T09:52:20.090158+0000","last_peered":"2024-09-11T09:52:20.090158+0000","last_clean":"2024-09-11T09:52:20.090158+0000","last_became_active":"2024-09-11T09:52:10.859011+0000","last_became_peered":"2024-09-11T09:52:10.859011+0000","last_unstale":"2024-09-11T09:52:20.090158+0000","last_undegraded":"2024-09-11T09:52:20.090158+0000","last_fullsized":"2024-09-11T09:52:20.090158+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_clean_scrub_stamp":"2024-09-11T09:50:17.477890+0000","objects_scrubbed":0,"log_size":226,"log_dups_size":0,"ondisk_log_size":226,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:33:06.639833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00050337500000000002,"stat_sum":{"num_bytes":22289408,"num_objects":12,"num_object_clones":4,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":2,"num_read":2614,"num_read_kb":34894,"num_write":134,"num_write_kb":54299,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":22031360,"num_keys_recovered":14,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"68'4","reported_seq":96,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-11T09:52:19.882556+0000","last_change":"2024-09-11T09:51:27.720403+0000","last_active":"2024-09-11T09:52:19.882556+0000","last_peered":"2024-09-11T09:52:19.882556+0000","last_clean":"2024-09-11T09:52:19.882556+0000","last_became_active":"2024-09-11T09:51:24.714345+0000","last_became_peered":"2024-09-11T09:51:24.714345+0000","last_unstale":"2024-09-11T09:52:19.882556+0000","last_undegraded":"2024-09-11T09:52:19.882556+0000","last_fullsized":"2024-09-11T09:52:19.882556+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:51:23.699749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:51:23.699749+0000","last_clean_scrub_stamp":"2024-09-11T09:51:23.699749+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T20:24:54.033307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040133200000000001,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":28,"num_read_kb":25,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"79'46","reported_seq":310,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-11T09:52:20.887559+0000","last_change":"2024-09-11T09:52:07.917418+0000","last_active":"2024-09-11T09:52:20.887559+0000","last_peered":"2024-09-11T09:52:20.887559+0000","last_clean":"2024-09-11T09:52:20.887559+0000","last_became_active":"2024-09-11T09:52:07.853491+0000","last_became_peered":"2024-09-11T09:52:07.853491+0000","last_unstale":"2024-09-11T09:52:20.887559+0000","last_undegraded":"2024-09-11T09:52:20.887559+0000","last_fullsized":"2024-09-11T09:52:20.887559+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_clean_scrub_stamp":"2024-09-11T09:50:17.477890+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T19:04:30.211623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045546699999999998,"stat_sum":{"num_bytes":13110272,"num_objects":5,"num_object_clones":2,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":73,"num_read_kb":24582,"num_write":44,"num_write_kb":29188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":13110272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":83,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-11T09:52:19.882434+0000","last_change":"2024-09-11T09:51:27.870068+0000","last_active":"2024-09-11T09:52:19.882434+0000","last_peered":"2024-09-11T09:52:19.882434+0000","last_clean":"2024-09-11T09:52:19.882434+0000","last_became_active":"2024-09-11T09:51:24.716325+0000","last_became_peered":"2024-09-11T09:51:24.716325+0000","last_unstale":"2024-09-11T09:52:19.882434+0000","last_undegraded":"2024-09-11T09:52:19.882434+0000","last_fullsized":"2024-09-11T09:52:19.882434+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:51:23.699749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:51:23.699749+0000","last_clean_scrub_stamp":"2024-09-11T09:51:23.699749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T19:44:13.062941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035292500000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.6","version":"87'117","reported_seq":483,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-11T09:52:20.041535+0000","last_change":"2024-09-11T09:50:20.497446+0000","last_active":"2024-09-11T09:52:20.041535+0000","last_peered":"2024-09-11T09:52:20.041535+0000","last_clean":"2024-09-11T09:52:20.041535+0000","last_became_active":"2024-09-11T09:50:18.497481+0000","last_became_peered":"2024-09-11T09:50:18.497481+0000","last_unstale":"2024-09-11T09:52:20.041535+0000","last_undegraded":"2024-09-11T09:52:20.041535+0000","last_fullsized":"2024-09-11T09:52:20.041535+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_clean_scrub_stamp":"2024-09-11T09:50:17.477890+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T18:30:25.298071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00050697600000000002,"stat_sum":{"num_bytes":26746974,"num_objects":14,"num_object_clones":5,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":3,"num_read":220,"num_read_kb":41002,"num_write":112,"num_write_kb":66580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-11T09:52:19.882286+0000","last_change":"2024-09-11T09:52:07.845599+0000","last_active":"2024-09-11T09:52:19.882286+0000","last_peered":"2024-09-11T09:52:19.882286+0000","last_clean":"2024-09-11T09:52:19.882286+0000","last_became_active":"2024-09-11T09:52:07.845316+0000","last_became_peered":"2024-09-11T09:52:07.845316+0000","last_unstale":"2024-09-11T09:52:19.882286+0000","last_undegraded":"2024-09-11T09:52:19.882286+0000","last_fullsized":"2024-09-11T09:52:19.882286+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:51:23.699749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:51:23.699749+0000","last_clean_scrub_stamp":"2024-09-11T09:51:23.699749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:25:27.357405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"87'172","reported_seq":341,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-11T09:52:21.233260+0000","last_change":"2024-09-11T09:52:04.840883+0000","last_active":"2024-09-11T09:52:21.233260+0000","last_peered":"2024-09-11T09:52:21.233260+0000","last_clean":"2024-09-11T09:52:21.233260+0000","last_became_active":"2024-09-11T09:52:04.840466+0000","last_became_peered":"2024-09-11T09:52:04.840466+0000","last_unstale":"2024-09-11T09:52:21.233260+0000","last_undegraded":"2024-09-11T09:52:21.233260+0000","last_fullsized":"2024-09-11T09:52:21.233260+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_clean_scrub_stamp":"2024-09-11T09:50:17.477890+0000","objects_scrubbed":0,"log_size":172,"log_dups_size":0,"ondisk_log_size":172,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:09:32.466344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00046094899999999999,"stat_sum":{"num_bytes":4194496,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":886,"num_read_kb":713,"num_write":297,"num_write_kb":4357,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":4194418,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":8,"stat_sum":{"num_bytes":70,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":6,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":16,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":156,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":16,"acting":16,"num_store_stats":6},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":148941429,"num_objects":73,"num_object_clones":19,"num_object_copies":146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73,"num_whiteouts":7,"num_read":5891,"num_read_kb":169120,"num_write":836,"num_write_kb":273122,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":39336050,"num_keys_recovered":14,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":311308288,"data_stored":333745308,"data_compressed":520463,"data_compressed_allocated":581632,"data_compressed_original":10586112,"omap_allocated":0,"internal_metadata":0},"log_size":845,"ondisk_log_size":845,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1771104,"data_compressed":100760,"data_compressed_allocated":135168,"data_compressed_original":1753088,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":12,"seq":51539607582,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":85704,"kb_used_data":54508,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94286136,"statfs":{"total":96636764160,"available":96549003264,"internally_reserved":0,"allocated":55816192,"data_stored":57116569,"data_compressed":105864,"data_compressed_allocated":143360,"data_compressed_original":2129920,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":51539607581,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":11,"seq":47244640286,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69132,"kb_used_data":37940,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94302708,"statfs":{"total":96636764160,"available":96565972992,"internally_reserved":0,"allocated":38850560,"data_stored":38683956,"data_compressed":24712,"data_compressed_allocated":32768,"data_compressed_original":524288,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":66,"seq":283467841545,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66164,"kb_used_data":38048,"kb_used_omap":4,"kb_used_meta":28091,"kb_avail":94305676,"statfs":{"total":96636764160,"available":96569012224,"internally_reserved":0,"allocated":38961152,"data_stored":39753589,"data_compressed":74161,"data_compressed_allocated":98304,"data_compressed_original":1572864,"omap_allocated":4432,"internal_metadata":28765872},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640285,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":85736,"kb_used_data":50384,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94286104,"statfs":{"total":96636764160,"available":96548970496,"internally_reserved":0,"allocated":51593216,"data_stored":59765079,"data_compressed":137427,"data_compressed_allocated":126976,"data_compressed_original":2688000,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640285,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":130996,"kb_used_data":95648,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94240844,"statfs":{"total":96636764160,"available":96502624256,"internally_reserved":0,"allocated":97943552,"data_stored":108085416,"data_compressed":244927,"data_compressed_allocated":270336,"data_compressed_original":4834304,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":34132,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":34132,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32496,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":33554432,"data_stored":33554432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":96841728,"data_stored":107092405,"data_compressed":210795,"data_compressed_allocated":225280,"data_compressed_original":4244480,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":50544640,"data_stored":59362474,"data_compressed":137427,"data_compressed_allocated":126976,"data_compressed_original":2688000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54693888,"data_stored":56118634,"data_compressed":73368,"data_compressed_allocated":98304,"data_compressed_original":1556480,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":37875712,"data_stored":39336050,"data_compressed":74161,"data_compressed_allocated":98304,"data_compressed_original":1572864,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":37797888,"data_stored":38281313,"data_compressed":24712,"data_compressed_allocated":32768,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":16,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":54,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":16,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":16,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":54,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T09:52:24.268 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T09:52:24.308 INFO:tasks.ceph.osd.2.smithi017.stderr:2024-09-11T09:52:24.309+0000 7fd20b170640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31672) UID: 0 2024-09-11T09:52:24.333 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:52:24.333 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:52:24.342 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-11T09:52:24.383 INFO:tasks.workunit.client.0.smithi107.stderr:+ DEV= 2024-09-11T09:52:24.383 INFO:tasks.workunit.client.0.smithi107.stderr:++ ceph-conf --show-config-value librados_thread_count 2024-09-11T09:52:24.402 INFO:tasks.workunit.client.0.smithi107.stderr:+ EXPECTED=2 2024-09-11T09:52:24.403 INFO:tasks.workunit.client.0.smithi107.stderr:++ _sudo rbd device --device-type nbd map rbd/testrbdnbd27121 2024-09-11T09:52:24.403 INFO:tasks.workunit.client.0.smithi107.stderr:++ local cmd 2024-09-11T09:52:24.403 INFO:tasks.workunit.client.0.smithi107.stderr:+++ id -u 2024-09-11T09:52:24.404 INFO:tasks.workunit.client.0.smithi107.stderr:++ '[' 1000 -eq 0 ']' 2024-09-11T09:52:24.404 INFO:tasks.workunit.client.0.smithi107.stderr:+++ which rbd 2024-09-11T09:52:24.406 INFO:tasks.workunit.client.0.smithi107.stderr:++ cmd=/usr/bin/rbd 2024-09-11T09:52:24.406 INFO:tasks.workunit.client.0.smithi107.stderr:++ shift 2024-09-11T09:52:24.406 INFO:tasks.workunit.client.0.smithi107.stderr:++ sudo -nE /usr/bin/rbd device --device-type nbd map rbd/testrbdnbd27121 2024-09-11T09:52:24.426 INFO:tasks.ceph.osd.0.smithi017.stderr:2024-09-11T09:52:24.427+0000 7fb73c0f1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31665) UID: 0 2024-09-11T09:52:24.489 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:52:24.490+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:52:24.512 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:52:24.513 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:52:24.522 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-11T09:52:24.606 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:52:24.607+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:52:24.625 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-11T09:52:24.626 INFO:teuthology.orchestra.run.smithi017.stderr:dumped all 2024-09-11T09:52:24.638 INFO:teuthology.orchestra.run.smithi017.stdout:{"pg_ready":true,"pg_map":{"version":155,"stamp":"2024-09-11T09:52:24.437629+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":149531867,"num_objects":81,"num_object_clones":19,"num_object_copies":162,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":81,"num_whiteouts":7,"num_read":6042,"num_read_kb":169249,"num_write":1001,"num_write_kb":275544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":41107170,"num_keys_recovered":14,"num_objects_omap":8,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":971,"ondisk_log_size":971,"up":34,"acting":34,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":34,"num_osds":5,"num_per_pool_osds":5,"num_per_pool_omap_osds":5,"kb":471859200,"kb_used":437732,"kb_used_data":276528,"kb_used_omap":10,"kb_used_meta":161077,"kb_avail":471421468,"statfs":{"total":483183820800,"available":482735583232,"internally_reserved":0,"allocated":283164672,"data_stored":303404609,"data_compressed":587091,"data_compressed_allocated":671744,"data_compressed_original":11749376,"omap_allocated":10808,"internal_metadata":164943304},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":6456,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":1019,"num_read_kb":1733,"num_write":44,"num_write_kb":27,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000528"},"pg_stats":[{"pgid":"1.0","version":"13'117","reported_seq":352,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-11T09:52:20.887613+0000","last_change":"2024-09-11T09:52:10.859352+0000","last_active":"2024-09-11T09:52:20.887613+0000","last_peered":"2024-09-11T09:52:20.887613+0000","last_clean":"2024-09-11T09:52:20.887613+0000","last_became_active":"2024-09-11T09:52:10.858918+0000","last_became_peered":"2024-09-11T09:52:10.858918+0000","last_unstale":"2024-09-11T09:52:20.887613+0000","last_undegraded":"2024-09-11T09:52:20.887613+0000","last_fullsized":"2024-09-11T09:52:20.887613+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:50:10.457262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:50:10.457262+0000","last_clean_scrub_stamp":"2024-09-11T09:50:10.457262+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:32:35.326531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"79'32","reported_seq":242,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-11T09:52:19.935661+0000","last_change":"2024-09-11T09:50:20.500687+0000","last_active":"2024-09-11T09:52:19.935661+0000","last_peered":"2024-09-11T09:52:19.935661+0000","last_clean":"2024-09-11T09:52:19.935661+0000","last_became_active":"2024-09-11T09:50:18.493650+0000","last_became_peered":"2024-09-11T09:50:18.493650+0000","last_unstale":"2024-09-11T09:52:19.935661+0000","last_undegraded":"2024-09-11T09:52:19.935661+0000","last_fullsized":"2024-09-11T09:52:19.935661+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_clean_scrub_stamp":"2024-09-11T09:50:17.477890+0000","objects_scrubbed":0,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:44:39.593950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041261100000000001,"stat_sum":{"num_bytes":25956352,"num_objects":8,"num_object_clones":1,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":47,"num_read_kb":12306,"num_write":30,"num_write_kb":33288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"67'3","reported_seq":89,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-11T09:52:19.882335+0000","last_change":"2024-09-11T09:51:27.719936+0000","last_active":"2024-09-11T09:52:19.882335+0000","last_peered":"2024-09-11T09:52:19.882335+0000","last_clean":"2024-09-11T09:52:19.882335+0000","last_became_active":"2024-09-11T09:51:24.714496+0000","last_became_peered":"2024-09-11T09:51:24.714496+0000","last_unstale":"2024-09-11T09:52:19.882335+0000","last_undegraded":"2024-09-11T09:52:19.882335+0000","last_fullsized":"2024-09-11T09:52:19.882335+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:51:23.699749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:51:23.699749+0000","last_clean_scrub_stamp":"2024-09-11T09:51:23.699749+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T20:58:07.190652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00068421499999999995,"stat_sum":{"num_bytes":54,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":5,"num_read_kb":4,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"74'6","reported_seq":330,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-11T09:52:20.887611+0000","last_change":"2024-09-11T09:50:20.498284+0000","last_active":"2024-09-11T09:52:20.887611+0000","last_peered":"2024-09-11T09:52:20.887611+0000","last_clean":"2024-09-11T09:52:20.887611+0000","last_became_active":"2024-09-11T09:50:18.493210+0000","last_became_peered":"2024-09-11T09:50:18.493210+0000","last_unstale":"2024-09-11T09:52:20.887611+0000","last_undegraded":"2024-09-11T09:52:20.887611+0000","last_fullsized":"2024-09-11T09:52:20.887611+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_clean_scrub_stamp":"2024-09-11T09:50:17.477890+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:06:18.020045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079749499999999995,"stat_sum":{"num_bytes":12582988,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":161,"num_read_kb":2852,"num_write":5,"num_write_kb":12290,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1","version":"67'1","reported_seq":76,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-11T09:52:22.896112+0000","last_change":"2024-09-11T09:52:07.849281+0000","last_active":"2024-09-11T09:52:22.896112+0000","last_peered":"2024-09-11T09:52:22.896112+0000","last_clean":"2024-09-11T09:52:22.896112+0000","last_became_active":"2024-09-11T09:52:07.845118+0000","last_became_peered":"2024-09-11T09:52:07.845118+0000","last_unstale":"2024-09-11T09:52:22.896112+0000","last_undegraded":"2024-09-11T09:52:22.896112+0000","last_fullsized":"2024-09-11T09:52:22.896112+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:51:23.699749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:51:23.699749+0000","last_clean_scrub_stamp":"2024-09-11T09:51:23.699749+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T21:07:40.901843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041844700000000001,"stat_sum":{"num_bytes":16,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":16,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1","version":"89'221","reported_seq":1818,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-11T09:52:20.105388+0000","last_change":"2024-09-11T09:50:21.030566+0000","last_active":"2024-09-11T09:52:20.105388+0000","last_peered":"2024-09-11T09:52:20.105388+0000","last_clean":"2024-09-11T09:52:20.105388+0000","last_became_active":"2024-09-11T09:50:18.493109+0000","last_became_peered":"2024-09-11T09:50:18.493109+0000","last_unstale":"2024-09-11T09:52:20.105388+0000","last_undegraded":"2024-09-11T09:52:20.105388+0000","last_fullsized":"2024-09-11T09:52:20.105388+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_clean_scrub_stamp":"2024-09-11T09:50:17.477890+0000","objects_scrubbed":0,"log_size":221,"log_dups_size":0,"ondisk_log_size":221,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T20:07:51.753387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041008499999999997,"stat_sum":{"num_bytes":31477930,"num_objects":17,"num_object_clones":6,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":1726,"num_read_kb":44414,"num_write":190,"num_write_kb":52637,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":82,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-11T09:52:19.882197+0000","last_change":"2024-09-11T09:51:27.719437+0000","last_active":"2024-09-11T09:52:19.882197+0000","last_peered":"2024-09-11T09:52:19.882197+0000","last_clean":"2024-09-11T09:52:19.882197+0000","last_became_active":"2024-09-11T09:51:24.714263+0000","last_became_peered":"2024-09-11T09:51:24.714263+0000","last_unstale":"2024-09-11T09:52:19.882197+0000","last_undegraded":"2024-09-11T09:52:19.882197+0000","last_fullsized":"2024-09-11T09:52:19.882197+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:51:23.699749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:51:23.699749+0000","last_clean_scrub_stamp":"2024-09-11T09:51:23.699749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:06:36.882446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045627500000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"87'25","reported_seq":351,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-11T09:52:20.089820+0000","last_change":"2024-09-11T09:50:20.501218+0000","last_active":"2024-09-11T09:52:20.089820+0000","last_peered":"2024-09-11T09:52:20.089820+0000","last_clean":"2024-09-11T09:52:20.089820+0000","last_became_active":"2024-09-11T09:50:18.493165+0000","last_became_peered":"2024-09-11T09:50:18.493165+0000","last_unstale":"2024-09-11T09:52:20.089820+0000","last_undegraded":"2024-09-11T09:52:20.089820+0000","last_fullsized":"2024-09-11T09:52:20.089820+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_clean_scrub_stamp":"2024-09-11T09:50:17.477890+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T19:11:24.668941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00094463900000000005,"stat_sum":{"num_bytes":12583009,"num_objects":6,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":164,"num_read_kb":8357,"num_write":24,"num_write_kb":20483,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"56'1","reported_seq":78,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-09-11T09:52:22.896112+0000","last_change":"2024-09-11T09:52:07.851088+0000","last_active":"2024-09-11T09:52:22.896112+0000","last_peered":"2024-09-11T09:52:22.896112+0000","last_clean":"2024-09-11T09:52:22.896112+0000","last_became_active":"2024-09-11T09:52:07.848151+0000","last_became_peered":"2024-09-11T09:52:07.848151+0000","last_unstale":"2024-09-11T09:52:22.896112+0000","last_undegraded":"2024-09-11T09:52:22.896112+0000","last_fullsized":"2024-09-11T09:52:22.896112+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:51:23.699749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:51:23.699749+0000","last_clean_scrub_stamp":"2024-09-11T09:51:23.699749+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T20:26:55.098291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040505599999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":82,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-11T09:52:19.935657+0000","last_change":"2024-09-11T09:51:27.729396+0000","last_active":"2024-09-11T09:52:19.935657+0000","last_peered":"2024-09-11T09:52:19.935657+0000","last_clean":"2024-09-11T09:52:19.935657+0000","last_became_active":"2024-09-11T09:51:24.715777+0000","last_became_peered":"2024-09-11T09:51:24.715777+0000","last_unstale":"2024-09-11T09:52:19.935657+0000","last_undegraded":"2024-09-11T09:52:19.935657+0000","last_fullsized":"2024-09-11T09:52:19.935657+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:51:23.699749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:51:23.699749+0000","last_clean_scrub_stamp":"2024-09-11T09:51:23.699749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:56:41.450651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026452099999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"87'226","reported_seq":1365,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-11T09:52:20.090158+0000","last_change":"2024-09-11T09:52:10.859987+0000","last_active":"2024-09-11T09:52:20.090158+0000","last_peered":"2024-09-11T09:52:20.090158+0000","last_clean":"2024-09-11T09:52:20.090158+0000","last_became_active":"2024-09-11T09:52:10.859011+0000","last_became_peered":"2024-09-11T09:52:10.859011+0000","last_unstale":"2024-09-11T09:52:20.090158+0000","last_undegraded":"2024-09-11T09:52:20.090158+0000","last_fullsized":"2024-09-11T09:52:20.090158+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_clean_scrub_stamp":"2024-09-11T09:50:17.477890+0000","objects_scrubbed":0,"log_size":226,"log_dups_size":0,"ondisk_log_size":226,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:33:06.639833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00050337500000000002,"stat_sum":{"num_bytes":22289408,"num_objects":12,"num_object_clones":4,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":2,"num_read":2614,"num_read_kb":34894,"num_write":134,"num_write_kb":54299,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":22031360,"num_keys_recovered":14,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"68'4","reported_seq":96,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-11T09:52:19.882556+0000","last_change":"2024-09-11T09:51:27.720403+0000","last_active":"2024-09-11T09:52:19.882556+0000","last_peered":"2024-09-11T09:52:19.882556+0000","last_clean":"2024-09-11T09:52:19.882556+0000","last_became_active":"2024-09-11T09:51:24.714345+0000","last_became_peered":"2024-09-11T09:51:24.714345+0000","last_unstale":"2024-09-11T09:52:19.882556+0000","last_undegraded":"2024-09-11T09:52:19.882556+0000","last_fullsized":"2024-09-11T09:52:19.882556+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:51:23.699749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:51:23.699749+0000","last_clean_scrub_stamp":"2024-09-11T09:51:23.699749+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T20:24:54.033307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040133200000000001,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":28,"num_read_kb":25,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"79'46","reported_seq":310,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-11T09:52:20.887559+0000","last_change":"2024-09-11T09:52:07.917418+0000","last_active":"2024-09-11T09:52:20.887559+0000","last_peered":"2024-09-11T09:52:20.887559+0000","last_clean":"2024-09-11T09:52:20.887559+0000","last_became_active":"2024-09-11T09:52:07.853491+0000","last_became_peered":"2024-09-11T09:52:07.853491+0000","last_unstale":"2024-09-11T09:52:20.887559+0000","last_undegraded":"2024-09-11T09:52:20.887559+0000","last_fullsized":"2024-09-11T09:52:20.887559+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_clean_scrub_stamp":"2024-09-11T09:50:17.477890+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T19:04:30.211623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045546699999999998,"stat_sum":{"num_bytes":13110272,"num_objects":5,"num_object_clones":2,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":73,"num_read_kb":24582,"num_write":44,"num_write_kb":29188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":13110272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":83,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-11T09:52:19.882434+0000","last_change":"2024-09-11T09:51:27.870068+0000","last_active":"2024-09-11T09:52:19.882434+0000","last_peered":"2024-09-11T09:52:19.882434+0000","last_clean":"2024-09-11T09:52:19.882434+0000","last_became_active":"2024-09-11T09:51:24.716325+0000","last_became_peered":"2024-09-11T09:51:24.716325+0000","last_unstale":"2024-09-11T09:52:19.882434+0000","last_undegraded":"2024-09-11T09:52:19.882434+0000","last_fullsized":"2024-09-11T09:52:19.882434+0000","mapping_epoch":54,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":55,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:51:23.699749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:51:23.699749+0000","last_clean_scrub_stamp":"2024-09-11T09:51:23.699749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T19:44:13.062941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035292500000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.6","version":"87'117","reported_seq":483,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-11T09:52:20.041535+0000","last_change":"2024-09-11T09:50:20.497446+0000","last_active":"2024-09-11T09:52:20.041535+0000","last_peered":"2024-09-11T09:52:20.041535+0000","last_clean":"2024-09-11T09:52:20.041535+0000","last_became_active":"2024-09-11T09:50:18.497481+0000","last_became_peered":"2024-09-11T09:50:18.497481+0000","last_unstale":"2024-09-11T09:52:20.041535+0000","last_undegraded":"2024-09-11T09:52:20.041535+0000","last_fullsized":"2024-09-11T09:52:20.041535+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_clean_scrub_stamp":"2024-09-11T09:50:17.477890+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T18:30:25.298071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00050697600000000002,"stat_sum":{"num_bytes":26746974,"num_objects":14,"num_object_clones":5,"num_object_copies":28,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":3,"num_read":220,"num_read_kb":41002,"num_write":112,"num_write_kb":66580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-09-11T09:52:19.882286+0000","last_change":"2024-09-11T09:52:07.845599+0000","last_active":"2024-09-11T09:52:19.882286+0000","last_peered":"2024-09-11T09:52:19.882286+0000","last_clean":"2024-09-11T09:52:19.882286+0000","last_became_active":"2024-09-11T09:52:07.845316+0000","last_became_peered":"2024-09-11T09:52:07.845316+0000","last_unstale":"2024-09-11T09:52:19.882286+0000","last_undegraded":"2024-09-11T09:52:19.882286+0000","last_fullsized":"2024-09-11T09:52:19.882286+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":54,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:51:23.699749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:51:23.699749+0000","last_clean_scrub_stamp":"2024-09-11T09:51:23.699749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:25:27.357405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"87'172","reported_seq":341,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-11T09:52:21.233260+0000","last_change":"2024-09-11T09:52:04.840883+0000","last_active":"2024-09-11T09:52:21.233260+0000","last_peered":"2024-09-11T09:52:21.233260+0000","last_clean":"2024-09-11T09:52:21.233260+0000","last_became_active":"2024-09-11T09:52:04.840466+0000","last_became_peered":"2024-09-11T09:52:04.840466+0000","last_unstale":"2024-09-11T09:52:21.233260+0000","last_undegraded":"2024-09-11T09:52:21.233260+0000","last_fullsized":"2024-09-11T09:52:21.233260+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_clean_scrub_stamp":"2024-09-11T09:50:17.477890+0000","objects_scrubbed":0,"log_size":172,"log_dups_size":0,"ondisk_log_size":172,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:09:32.466344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00046094899999999999,"stat_sum":{"num_bytes":4194496,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":886,"num_read_kb":713,"num_write":297,"num_write_kb":4357,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":4194418,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":8,"stat_sum":{"num_bytes":70,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":6,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":16,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":156,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":16,"acting":16,"num_store_stats":6},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":148941429,"num_objects":73,"num_object_clones":19,"num_object_copies":146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73,"num_whiteouts":7,"num_read":5891,"num_read_kb":169120,"num_write":836,"num_write_kb":273122,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":39336050,"num_keys_recovered":14,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":311308288,"data_stored":333745308,"data_compressed":520463,"data_compressed_allocated":581632,"data_compressed_original":10586112,"omap_allocated":0,"internal_metadata":0},"log_size":845,"ondisk_log_size":845,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1771104,"data_compressed":100760,"data_compressed_allocated":135168,"data_compressed_original":1753088,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":12,"seq":51539607582,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":85704,"kb_used_data":54508,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94286136,"statfs":{"total":96636764160,"available":96549003264,"internally_reserved":0,"allocated":55816192,"data_stored":57116569,"data_compressed":105864,"data_compressed_allocated":143360,"data_compressed_original":2129920,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":51539607581,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":11,"seq":47244640286,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69132,"kb_used_data":37940,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94302708,"statfs":{"total":96636764160,"available":96565972992,"internally_reserved":0,"allocated":38850560,"data_stored":38683956,"data_compressed":24712,"data_compressed_allocated":32768,"data_compressed_original":524288,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":66,"seq":283467841545,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66164,"kb_used_data":38048,"kb_used_omap":4,"kb_used_meta":28091,"kb_avail":94305676,"statfs":{"total":96636764160,"available":96569012224,"internally_reserved":0,"allocated":38961152,"data_stored":39753589,"data_compressed":74161,"data_compressed_allocated":98304,"data_compressed_original":1572864,"omap_allocated":4432,"internal_metadata":28765872},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640285,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":85736,"kb_used_data":50384,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94286104,"statfs":{"total":96636764160,"available":96548970496,"internally_reserved":0,"allocated":51593216,"data_stored":59765079,"data_compressed":137427,"data_compressed_allocated":126976,"data_compressed_original":2688000,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640285,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":130996,"kb_used_data":95648,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94240844,"statfs":{"total":96636764160,"available":96502624256,"internally_reserved":0,"allocated":97943552,"data_stored":108085416,"data_compressed":244927,"data_compressed_allocated":270336,"data_compressed_original":4834304,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":34132,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":34132,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32496,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":33554432,"data_stored":33554432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":96841728,"data_stored":107092405,"data_compressed":210795,"data_compressed_allocated":225280,"data_compressed_original":4244480,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":50544640,"data_stored":59362474,"data_compressed":137427,"data_compressed_allocated":126976,"data_compressed_original":2688000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":54693888,"data_stored":56118634,"data_compressed":73368,"data_compressed_allocated":98304,"data_compressed_original":1556480,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":37875712,"data_stored":39336050,"data_compressed":74161,"data_compressed_allocated":98304,"data_compressed_original":1572864,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":37797888,"data_stored":38281313,"data_compressed":24712,"data_compressed_allocated":32768,"data_compressed_original":524288,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":16,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":54,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":16,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":16,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":54,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T09:52:24.640 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-09-11T09:52:24.640 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-11T09:52:24.640 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-09-11T09:52:24.640 INFO:tasks.thrashosds.thrasher:Setting osd 1 primary_affinity to 0.000000 2024-09-11T09:52:24.640 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 1 0 2024-09-11T09:52:24.670 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:52:24.670 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:52:24.679 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-11T09:52:24.705 INFO:tasks.workunit.client.0.smithi107.stderr:+ DEV=/dev/nbd2 2024-09-11T09:52:24.705 INFO:tasks.workunit.client.0.smithi107.stderr:+ get_pid rbd 2024-09-11T09:52:24.705 INFO:tasks.workunit.client.0.smithi107.stderr:+ local pool=rbd 2024-09-11T09:52:24.706 INFO:tasks.workunit.client.0.smithi107.stderr:+ local ns= 2024-09-11T09:52:24.706 INFO:tasks.workunit.client.0.smithi107.stderr:++ rbd device --device-type nbd --format xml list 2024-09-11T09:52:24.706 INFO:tasks.workunit.client.0.smithi107.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd27121'\''][device='\''/dev/nbd2'\'']/id' 2024-09-11T09:52:24.720 INFO:tasks.ceph.osd.2.smithi017.stderr:2024-09-11T09:52:24.721+0000 7fd20b170640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31672) UID: 0 2024-09-11T09:52:24.792 INFO:tasks.ceph.osd.0.smithi017.stderr:2024-09-11T09:52:24.793+0000 7fb73c0f1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31665) UID: 0 2024-09-11T09:52:24.798 INFO:tasks.workunit.client.0.smithi107.stderr:+ PID=30819 2024-09-11T09:52:24.798 INFO:tasks.workunit.client.0.smithi107.stderr:+ test -n 30819 2024-09-11T09:52:24.798 INFO:tasks.workunit.client.0.smithi107.stderr:+ ps -p 30819 -C rbd-nbd 2024-09-11T09:52:24.809 INFO:tasks.workunit.client.0.smithi107.stdout: PID TTY TIME CMD 2024-09-11T09:52:24.809 INFO:tasks.workunit.client.0.smithi107.stdout: 30819 ? 00:00:00 rbd-nbd 2024-09-11T09:52:24.810 INFO:tasks.workunit.client.0.smithi107.stderr:++ ps -p 30819 -T 2024-09-11T09:52:24.810 INFO:tasks.workunit.client.0.smithi107.stderr:++ grep -c io_context_pool 2024-09-11T09:52:24.821 INFO:tasks.workunit.client.0.smithi107.stderr:+ ACTUAL=2 2024-09-11T09:52:24.821 INFO:tasks.workunit.client.0.smithi107.stderr:+ '[' 2 -eq 2 ']' 2024-09-11T09:52:24.821 INFO:tasks.workunit.client.0.smithi107.stderr:+ unmap_device /dev/nbd2 30819 2024-09-11T09:52:24.822 INFO:tasks.workunit.client.0.smithi107.stderr:+ local args=/dev/nbd2 2024-09-11T09:52:24.822 INFO:tasks.workunit.client.0.smithi107.stderr:+ local pid=30819 2024-09-11T09:52:24.822 INFO:tasks.workunit.client.0.smithi107.stderr:+ _sudo rbd device --device-type nbd unmap /dev/nbd2 2024-09-11T09:52:24.822 INFO:tasks.workunit.client.0.smithi107.stderr:+ local cmd 2024-09-11T09:52:24.822 INFO:tasks.workunit.client.0.smithi107.stderr:++ id -u 2024-09-11T09:52:24.822 INFO:tasks.workunit.client.0.smithi107.stderr:+ '[' 1000 -eq 0 ']' 2024-09-11T09:52:24.822 INFO:tasks.workunit.client.0.smithi107.stderr:++ which rbd 2024-09-11T09:52:24.824 INFO:tasks.workunit.client.0.smithi107.stderr:+ cmd=/usr/bin/rbd 2024-09-11T09:52:24.824 INFO:tasks.workunit.client.0.smithi107.stderr:+ shift 2024-09-11T09:52:24.824 INFO:tasks.workunit.client.0.smithi107.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd unmap /dev/nbd2 2024-09-11T09:52:24.845 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:52:24.846 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:52:24.854 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-11T09:52:24.899 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:52:24.899+0000 7f3a7b90a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 42475) UID: 0 2024-09-11T09:52:24.902 INFO:teuthology.orchestra.run.smithi017.stderr:nodeep-scrub is unset 2024-09-11T09:52:24.932 INFO:tasks.workunit.client.0.smithi107.stderr:+ rbd device --device-type nbd list 2024-09-11T09:52:24.932 INFO:tasks.workunit.client.0.smithi107.stderr:+ expect_false grep '^30819\b' 2024-09-11T09:52:24.932 INFO:tasks.workunit.client.0.smithi107.stderr:+ grep '^30819\b' 2024-09-11T09:52:24.970 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:52:24.970 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:52:24.979 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-11T09:52:24.993 INFO:tasks.workunit.client.0.smithi107.stderr:+ return 0 2024-09-11T09:52:24.994 INFO:tasks.workunit.client.0.smithi107.stderr:+ ps -C rbd-nbd 2024-09-11T09:52:24.994 INFO:tasks.workunit.client.0.smithi107.stderr:+ expect_false grep '^ *30819\b' 2024-09-11T09:52:24.994 INFO:tasks.workunit.client.0.smithi107.stderr:+ grep '^ *30819\b' 2024-09-11T09:52:25.005 INFO:tasks.workunit.client.0.smithi107.stderr:+ return 0 2024-09-11T09:52:25.005 INFO:tasks.workunit.client.0.smithi107.stderr:+ sleep 0.5 2024-09-11T09:52:25.020 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:52:25.021+0000 7f3a7b90a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 42475) UID: 0 2024-09-11T09:52:25.095 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:52:25.096+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:52:25.123 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:52:25.123 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:52:25.131 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-11T09:52:25.196 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:52:25.196+0000 7f3a7b90a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 42475) UID: 0 2024-09-11T09:52:25.235 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:52:25.236 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:52:25.245 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-11T09:52:25.297 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:52:25.298+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:52:25.388 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:52:25.388 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:52:25.397 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-11T09:52:25.398 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:52:25.398+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:52:25.498 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-09-11T09:52:25.498+0000 7f7b6c7d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31904) UID: 0 2024-09-11T09:52:25.500 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:52:25.500 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:52:25.506 INFO:tasks.workunit.client.0.smithi107.stderr:+ EXPECTED=7 2024-09-11T09:52:25.506 INFO:tasks.workunit.client.0.smithi107.stderr:++ _sudo rbd device --device-type nbd --rbd-op-threads 7 map rbd/testrbdnbd27121 2024-09-11T09:52:25.506 INFO:tasks.workunit.client.0.smithi107.stderr:++ local cmd 2024-09-11T09:52:25.507 INFO:tasks.workunit.client.0.smithi107.stderr:+++ id -u 2024-09-11T09:52:25.507 INFO:tasks.workunit.client.0.smithi107.stderr:++ '[' 1000 -eq 0 ']' 2024-09-11T09:52:25.508 INFO:tasks.workunit.client.0.smithi107.stderr:+++ which rbd 2024-09-11T09:52:25.509 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-11T09:52:25.510 INFO:tasks.workunit.client.0.smithi107.stderr:++ cmd=/usr/bin/rbd 2024-09-11T09:52:25.510 INFO:tasks.workunit.client.0.smithi107.stderr:++ shift 2024-09-11T09:52:25.510 INFO:tasks.workunit.client.0.smithi107.stderr:++ sudo -nE /usr/bin/rbd device --device-type nbd --rbd-op-threads 7 map rbd/testrbdnbd27121 2024-09-11T09:52:25.599 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-09-11T09:52:25.599+0000 7f7b6c7d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31904) UID: 0 2024-09-11T09:52:25.624 INFO:tasks.workunit.client.0.smithi107.stderr:+ DEV=/dev/nbd2 2024-09-11T09:52:25.624 INFO:tasks.workunit.client.0.smithi107.stderr:+ get_pid rbd 2024-09-11T09:52:25.624 INFO:tasks.workunit.client.0.smithi107.stderr:+ local pool=rbd 2024-09-11T09:52:25.625 INFO:tasks.workunit.client.0.smithi107.stderr:+ local ns= 2024-09-11T09:52:25.625 INFO:tasks.workunit.client.0.smithi107.stderr:++ rbd device --device-type nbd --format xml list 2024-09-11T09:52:25.625 INFO:tasks.workunit.client.0.smithi107.stderr:++ xmlstarlet sel -t -v '//devices/device[pool='\''rbd'\''][namespace='\'''\''][image='\''testrbdnbd27121'\''][device='\''/dev/nbd2'\'']/id' 2024-09-11T09:52:25.653 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:52:25.653 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:52:25.662 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-11T09:52:25.703 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:52:25.704+0000 7f3a7b90a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 42475) UID: 0 2024-09-11T09:52:25.719 INFO:tasks.workunit.client.0.smithi107.stderr:+ PID=30888 2024-09-11T09:52:25.719 INFO:tasks.workunit.client.0.smithi107.stderr:+ test -n 30888 2024-09-11T09:52:25.719 INFO:tasks.workunit.client.0.smithi107.stderr:+ ps -p 30888 -C rbd-nbd 2024-09-11T09:52:25.730 INFO:tasks.workunit.client.0.smithi107.stdout: PID TTY TIME CMD 2024-09-11T09:52:25.730 INFO:tasks.workunit.client.0.smithi107.stdout: 30888 ? 00:00:00 rbd-nbd 2024-09-11T09:52:25.730 INFO:tasks.workunit.client.0.smithi107.stderr:++ ps -p 30888 -T 2024-09-11T09:52:25.730 INFO:tasks.workunit.client.0.smithi107.stderr:++ grep -c io_context_pool 2024-09-11T09:52:25.742 INFO:tasks.workunit.client.0.smithi107.stderr:+ ACTUAL=7 2024-09-11T09:52:25.742 INFO:tasks.workunit.client.0.smithi107.stderr:+ '[' 7 -eq 7 ']' 2024-09-11T09:52:25.742 INFO:tasks.workunit.client.0.smithi107.stderr:+ unmap_device /dev/nbd2 30888 2024-09-11T09:52:25.742 INFO:tasks.workunit.client.0.smithi107.stderr:+ local args=/dev/nbd2 2024-09-11T09:52:25.742 INFO:tasks.workunit.client.0.smithi107.stderr:+ local pid=30888 2024-09-11T09:52:25.742 INFO:tasks.workunit.client.0.smithi107.stderr:+ _sudo rbd device --device-type nbd unmap /dev/nbd2 2024-09-11T09:52:25.743 INFO:tasks.workunit.client.0.smithi107.stderr:+ local cmd 2024-09-11T09:52:25.743 INFO:tasks.workunit.client.0.smithi107.stderr:++ id -u 2024-09-11T09:52:25.743 INFO:tasks.workunit.client.0.smithi107.stderr:+ '[' 1000 -eq 0 ']' 2024-09-11T09:52:25.743 INFO:tasks.workunit.client.0.smithi107.stderr:++ which rbd 2024-09-11T09:52:25.745 INFO:tasks.workunit.client.0.smithi107.stderr:+ cmd=/usr/bin/rbd 2024-09-11T09:52:25.745 INFO:tasks.workunit.client.0.smithi107.stderr:+ shift 2024-09-11T09:52:25.745 INFO:tasks.workunit.client.0.smithi107.stderr:+ sudo -nE /usr/bin/rbd device --device-type nbd unmap /dev/nbd2 2024-09-11T09:52:25.801 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:52:25.802+0000 7f3c96ec9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31909) UID: 0 2024-09-11T09:52:25.805 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:52:25.805 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:52:25.814 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-11T09:52:25.855 INFO:tasks.workunit.client.0.smithi107.stderr:+ rbd device --device-type nbd list 2024-09-11T09:52:25.855 INFO:tasks.workunit.client.0.smithi107.stderr:+ expect_false grep '^30888\b' 2024-09-11T09:52:25.855 INFO:tasks.workunit.client.0.smithi107.stderr:+ grep '^30888\b' 2024-09-11T09:52:25.902 INFO:tasks.ceph.osd.0.smithi017.stderr:2024-09-11T09:52:25.903+0000 7fb73c0f1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31665) UID: 0 2024-09-11T09:52:25.909 INFO:teuthology.orchestra.run.smithi017.stderr:set osd.1 primary-affinity to 0 (802) 2024-09-11T09:52:25.916 INFO:tasks.workunit.client.0.smithi107.stderr:+ return 0 2024-09-11T09:52:25.916 INFO:tasks.workunit.client.0.smithi107.stderr:+ ps -C rbd-nbd 2024-09-11T09:52:25.917 INFO:tasks.workunit.client.0.smithi107.stderr:+ expect_false grep '^ *30888\b' 2024-09-11T09:52:25.917 INFO:tasks.workunit.client.0.smithi107.stderr:+ grep '^ *30888\b' 2024-09-11T09:52:25.927 INFO:tasks.workunit.client.0.smithi107.stderr:+ return 0 2024-09-11T09:52:25.927 INFO:tasks.workunit.client.0.smithi107.stderr:+ sleep 0.5 2024-09-11T09:52:25.960 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:52:25.960 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:52:25.969 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-11T09:52:26.012 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:52:26.013+0000 7f3c96ec9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31909) UID: 0 2024-09-11T09:52:26.070 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:52:26.070 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:52:26.079 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-11T09:52:26.123 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:52:26.124+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:52:26.177 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T09:52:26.200 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:52:26.200 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:52:26.205 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-09-11T09:52:26.206+0000 7f7b6c7d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31904) UID: 0 2024-09-11T09:52:26.211 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-11T09:52:26.306 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-11T09:52:26.306 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T09:52:26.306 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-09-11T09:52:26.307+0000 7f7b6c7d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31904) UID: 0 2024-09-11T09:52:26.324 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-11T09:52:26.324 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T09:52:26.342 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-11T09:52:26.342 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T09:52:26.359 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-11T09:52:26.359 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T09:52:26.364 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:52:26.364 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [], 2024-09-11T09:52:26.364 INFO:teuthology.orchestra.run.smithi017.stdout: "complaint_time": 30, 2024-09-11T09:52:26.364 INFO:teuthology.orchestra.run.smithi017.stdout: "num_blocked_ops": 0 2024-09-11T09:52:26.364 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:52:26.373 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-11T09:52:26.376 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-11T09:52:26.376 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T09:52:26.393 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-11T09:52:26.394 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T09:52:26.429 INFO:tasks.workunit.client.0.smithi107.stderr:+ DEV= 2024-09-11T09:52:26.429 INFO:tasks.workunit.client.0.smithi107.stdout:OK 2024-09-11T09:52:26.430 INFO:tasks.workunit.client.0.smithi107.stderr:+ echo OK 2024-09-11T09:52:26.430 INFO:tasks.workunit.client.0.smithi107.stderr:+ cleanup 2024-09-11T09:52:26.430 INFO:tasks.workunit.client.0.smithi107.stderr:+ local ns s 2024-09-11T09:52:26.430 INFO:tasks.workunit.client.0.smithi107.stderr:+ set +e 2024-09-11T09:52:26.430 INFO:tasks.workunit.client.0.smithi107.stderr:+ mount 2024-09-11T09:52:26.430 INFO:tasks.workunit.client.0.smithi107.stderr:+ fgrep /tmp/tmp.rvrcDirNue/mnt 2024-09-11T09:52:26.446 INFO:tasks.ceph.osd.0.smithi017.stderr:2024-09-11T09:52:26.447+0000 7fb73c0f1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31665) UID: 0 2024-09-11T09:52:26.448 INFO:tasks.workunit.client.0.smithi107.stderr:+ rm -Rf /tmp/tmp.rvrcDirNue 2024-09-11T09:52:26.457 INFO:tasks.workunit.client.0.smithi107.stderr:+ '[' -n '' ']' 2024-09-11T09:52:26.457 INFO:tasks.workunit.client.0.smithi107.stderr:+ for ns in '' ${NS} 2024-09-11T09:52:26.457 INFO:tasks.workunit.client.0.smithi107.stderr:+ rbd -p rbd --namespace '' status testrbdnbd27121 2024-09-11T09:52:26.483 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:52:26.483 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 20, 2024-09-11T09:52:26.483 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 600, 2024-09-11T09:52:26.483 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [ 2024-09-11T09:52:26.483 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.483 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4688.0:34 2.4 e53/32 2:2367385a:::rbd_data.1250f5e25b80.0000000000000002:head v 53'6, mlcod=53'4)", 2024-09-11T09:52:26.483 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.907415+0000", 2024-09-11T09:52:26.484 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 63.575711358, 2024-09-11T09:52:26.484 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.011285778999999999, 2024-09-11T09:52:26.484 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:26.484 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:26.484 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:26.484 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:26.484 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.484 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:26.484 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.907415+0000", 2024-09-11T09:52:26.484 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:26.484 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.484 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.485 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:26.485 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.907415+0000", 2024-09-11T09:52:26.485 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:26.485 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.485 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.485 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:26.485 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.907417+0000", 2024-09-11T09:52:26.485 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.2910000000000002e-06 2024-09-11T09:52:26.485 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.485 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.485 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:26.485 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.910477+0000", 2024-09-11T09:52:26.485 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0030597829999999999 2024-09-11T09:52:26.485 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.486 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.486 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:26.486 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.910501+0000", 2024-09-11T09:52:26.486 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.3332e-05 2024-09-11T09:52:26.486 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.486 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.486 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:26.486 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.910514+0000", 2024-09-11T09:52:26.486 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3957e-05 2024-09-11T09:52:26.486 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.486 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.486 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:26.486 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.910612+0000", 2024-09-11T09:52:26.486 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.7075000000000001e-05 2024-09-11T09:52:26.487 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.487 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.487 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:26.487 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.910640+0000", 2024-09-11T09:52:26.487 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.8768999999999998e-05 2024-09-11T09:52:26.487 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.487 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.487 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:26.487 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.918640+0000", 2024-09-11T09:52:26.487 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0079991899999999998 2024-09-11T09:52:26.487 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.487 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.487 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:26.488 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.918701+0000", 2024-09-11T09:52:26.488 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.1382000000000003e-05 2024-09-11T09:52:26.488 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:26.488 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:26.488 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:26.488 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.488 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.488 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4688.0:38 2.5 e53/32 2:abde2694:::rbd_data.1250f5e25b80.0000000000000004:head v 53'1, mlcod=0'0)", 2024-09-11T09:52:26.488 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.915432+0000", 2024-09-11T09:52:26.489 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 63.567694977999999, 2024-09-11T09:52:26.489 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.015849652999999998, 2024-09-11T09:52:26.489 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:26.489 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:26.489 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:26.489 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:26.489 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.489 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:26.489 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.915432+0000", 2024-09-11T09:52:26.489 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:26.489 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.489 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.489 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:26.490 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.915432+0000", 2024-09-11T09:52:26.490 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:26.490 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.490 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.490 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:26.490 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.915432+0000", 2024-09-11T09:52:26.490 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.91e-07 2024-09-11T09:52:26.490 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.490 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.490 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:26.490 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.924943+0000", 2024-09-11T09:52:26.490 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0095110630000000002 2024-09-11T09:52:26.490 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.491 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.491 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:26.491 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.924960+0000", 2024-09-11T09:52:26.491 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6915999999999999e-05 2024-09-11T09:52:26.491 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.491 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.491 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:26.491 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.924968+0000", 2024-09-11T09:52:26.491 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.6999999999999997e-06 2024-09-11T09:52:26.491 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.491 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.491 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:26.491 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.925025+0000", 2024-09-11T09:52:26.491 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.6949e-05 2024-09-11T09:52:26.492 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.492 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.492 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:26.492 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.925043+0000", 2024-09-11T09:52:26.492 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7784000000000001e-05 2024-09-11T09:52:26.492 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.492 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.492 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:26.492 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.931237+0000", 2024-09-11T09:52:26.492 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0061935389999999996 2024-09-11T09:52:26.492 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.492 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.492 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:26.493 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.931281+0000", 2024-09-11T09:52:26.493 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.4511000000000001e-05 2024-09-11T09:52:26.493 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:26.493 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:26.493 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:26.493 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.493 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.493 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:40 2.7 2:fe41411c:::rbd_object_map.1250f5e25b80:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:26.493 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.919652+0000", 2024-09-11T09:52:26.493 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 63.563474311, 2024-09-11T09:52:26.493 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.012187073, 2024-09-11T09:52:26.493 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:26.493 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:26.494 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:26.494 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:26.494 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:52:26.494 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:52:26.494 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 40 2024-09-11T09:52:26.494 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.494 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:26.494 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.494 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:26.494 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919652+0000", 2024-09-11T09:52:26.494 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:26.494 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.494 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.494 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:26.495 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919652+0000", 2024-09-11T09:52:26.495 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:26.495 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.495 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.495 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:26.495 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919654+0000", 2024-09-11T09:52:26.495 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.88e-06 2024-09-11T09:52:26.495 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.495 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.495 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:26.495 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919669+0000", 2024-09-11T09:52:26.495 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.5271e-05 2024-09-11T09:52:26.495 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.496 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.496 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:26.496 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919687+0000", 2024-09-11T09:52:26.496 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7448999999999998e-05 2024-09-11T09:52:26.496 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.496 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.496 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:26.496 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919695+0000", 2024-09-11T09:52:26.496 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.1149999999999994e-06 2024-09-11T09:52:26.496 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.496 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.496 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:26.496 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919757+0000", 2024-09-11T09:52:26.496 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.2348999999999995e-05 2024-09-11T09:52:26.497 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.497 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.497 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:26.497 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919875+0000", 2024-09-11T09:52:26.497 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00011750499999999999 2024-09-11T09:52:26.497 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.497 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.497 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 5", 2024-09-11T09:52:26.497 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.920201+0000", 2024-09-11T09:52:26.497 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00032628099999999999 2024-09-11T09:52:26.497 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.497 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.497 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:26.497 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.922089+0000", 2024-09-11T09:52:26.498 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.001887978 2024-09-11T09:52:26.498 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.498 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.498 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:26.498 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.931752+0000", 2024-09-11T09:52:26.498 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0096626309999999997 2024-09-11T09:52:26.498 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.498 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.498 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:26.498 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.931794+0000", 2024-09-11T09:52:26.498 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.2849999999999998e-05 2024-09-11T09:52:26.498 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.498 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.499 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:26.499 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.931839+0000", 2024-09-11T09:52:26.499 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.4764e-05 2024-09-11T09:52:26.499 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:26.499 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:26.499 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:26.499 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.499 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.499 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:41 2.7 2:fe41411c:::rbd_object_map.1250f5e25b80:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:26.499 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.919716+0000", 2024-09-11T09:52:26.499 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 63.563410466999997, 2024-09-11T09:52:26.499 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.015641605999999999, 2024-09-11T09:52:26.499 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:26.499 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:26.500 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:26.500 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:26.500 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:52:26.500 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:52:26.500 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 41 2024-09-11T09:52:26.500 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.500 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:26.500 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.500 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:26.500 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919716+0000", 2024-09-11T09:52:26.500 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:26.500 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.501 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.501 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:26.501 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919716+0000", 2024-09-11T09:52:26.501 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:26.501 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.501 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.501 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:26.501 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919716+0000", 2024-09-11T09:52:26.501 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.9999999999999998e-07 2024-09-11T09:52:26.501 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.501 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.501 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:26.501 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919720+0000", 2024-09-11T09:52:26.501 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.6440000000000002e-06 2024-09-11T09:52:26.502 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.502 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.502 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:26.502 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919729+0000", 2024-09-11T09:52:26.502 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.1759999999999999e-06 2024-09-11T09:52:26.502 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.502 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.502 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:26.502 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919736+0000", 2024-09-11T09:52:26.502 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.5390000000000004e-06 2024-09-11T09:52:26.502 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.502 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.502 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:26.502 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.920669+0000", 2024-09-11T09:52:26.503 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00093275599999999997 2024-09-11T09:52:26.503 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.503 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.503 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:26.503 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.920797+0000", 2024-09-11T09:52:26.503 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00012893400000000001 2024-09-11T09:52:26.503 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.503 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.503 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 5", 2024-09-11T09:52:26.503 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.921269+0000", 2024-09-11T09:52:26.503 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00047128600000000001 2024-09-11T09:52:26.503 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.503 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.503 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:26.504 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.923198+0000", 2024-09-11T09:52:26.504 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0019289369999999999 2024-09-11T09:52:26.504 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.504 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.504 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:26.504 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.935300+0000", 2024-09-11T09:52:26.504 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.012102392 2024-09-11T09:52:26.504 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.504 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.504 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:26.504 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.935332+0000", 2024-09-11T09:52:26.504 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.2292000000000002e-05 2024-09-11T09:52:26.504 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.505 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.505 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:26.505 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.935358+0000", 2024-09-11T09:52:26.505 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.525e-05 2024-09-11T09:52:26.505 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:26.505 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:26.505 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:26.505 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.505 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.505 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:48 2.7 2:fe41411c:::rbd_object_map.1250f5e25b80:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:26.505 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.938799+0000", 2024-09-11T09:52:26.505 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 63.544327357999997, 2024-09-11T09:52:26.505 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0041642340000000002, 2024-09-11T09:52:26.506 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:26.506 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:26.506 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:26.506 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:26.506 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:52:26.506 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:52:26.506 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 48 2024-09-11T09:52:26.506 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.506 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:26.506 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.506 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:26.506 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.938799+0000", 2024-09-11T09:52:26.506 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:26.506 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.507 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.507 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:26.507 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.938799+0000", 2024-09-11T09:52:26.507 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:26.507 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.507 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.507 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:26.507 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.938800+0000", 2024-09-11T09:52:26.507 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.71e-07 2024-09-11T09:52:26.507 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.507 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.507 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:26.507 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.938805+0000", 2024-09-11T09:52:26.507 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.3140000000000003e-06 2024-09-11T09:52:26.507 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.508 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.508 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:26.508 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.938816+0000", 2024-09-11T09:52:26.508 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0231000000000001e-05 2024-09-11T09:52:26.508 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.508 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.508 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:26.508 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.938822+0000", 2024-09-11T09:52:26.508 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.9550000000000003e-06 2024-09-11T09:52:26.508 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.508 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.508 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:26.508 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.938868+0000", 2024-09-11T09:52:26.508 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.5562999999999997e-05 2024-09-11T09:52:26.509 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.509 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.509 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:26.509 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.938918+0000", 2024-09-11T09:52:26.509 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.0317e-05 2024-09-11T09:52:26.509 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.509 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.509 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 5", 2024-09-11T09:52:26.509 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.939114+0000", 2024-09-11T09:52:26.509 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00019538300000000001 2024-09-11T09:52:26.509 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.509 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.509 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:26.509 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.939663+0000", 2024-09-11T09:52:26.510 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00054932199999999998 2024-09-11T09:52:26.510 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.510 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.510 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:26.510 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.942910+0000", 2024-09-11T09:52:26.510 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0032473839999999999 2024-09-11T09:52:26.510 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.510 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.510 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:26.510 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.942940+0000", 2024-09-11T09:52:26.510 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.9306999999999999e-05 2024-09-11T09:52:26.510 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.510 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.510 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:26.511 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.942963+0000", 2024-09-11T09:52:26.511 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.3587e-05 2024-09-11T09:52:26.511 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:26.511 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:26.511 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:26.511 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.511 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.511 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4688.0:49 2.5 e53/32 2:b6dd9fc9:::rbd_data.1250f5e25b80.0000000000000009:head v 53'2, mlcod=0'0)", 2024-09-11T09:52:26.511 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.944712+0000", 2024-09-11T09:52:26.511 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 63.538414519, 2024-09-11T09:52:26.511 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.016185069999999999, 2024-09-11T09:52:26.511 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:26.511 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:26.512 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:26.512 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:26.512 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.512 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:26.512 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.944712+0000", 2024-09-11T09:52:26.512 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:26.512 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.512 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.512 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:26.512 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.944712+0000", 2024-09-11T09:52:26.512 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:26.512 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.512 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.513 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:26.513 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.944713+0000", 2024-09-11T09:52:26.513 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0139999999999999e-06 2024-09-11T09:52:26.513 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.513 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.513 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:26.513 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.955518+0000", 2024-09-11T09:52:26.513 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.01080542 2024-09-11T09:52:26.513 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.513 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.513 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:26.513 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.955536+0000", 2024-09-11T09:52:26.513 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7818000000000001e-05 2024-09-11T09:52:26.514 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.514 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.514 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:26.514 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.955544+0000", 2024-09-11T09:52:26.514 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.1699999999999997e-06 2024-09-11T09:52:26.514 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.514 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.514 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:26.514 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.955607+0000", 2024-09-11T09:52:26.514 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.2451000000000002e-05 2024-09-11T09:52:26.514 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.514 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.514 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:26.514 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.955632+0000", 2024-09-11T09:52:26.515 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.5335000000000001e-05 2024-09-11T09:52:26.515 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.515 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.515 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:26.515 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.960826+0000", 2024-09-11T09:52:26.515 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0051938560000000002 2024-09-11T09:52:26.515 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.515 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.515 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:26.515 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.960897+0000", 2024-09-11T09:52:26.515 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.1006000000000004e-05 2024-09-11T09:52:26.515 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:26.515 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:26.515 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:26.516 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.516 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.516 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:53 2.7 2:fe41411c:::rbd_object_map.1250f5e25b80:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:26.516 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.951291+0000", 2024-09-11T09:52:26.516 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 63.531835528999999, 2024-09-11T09:52:26.516 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0044964999999999996, 2024-09-11T09:52:26.516 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:26.516 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:26.516 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:26.516 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:26.516 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:52:26.516 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:52:26.516 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 53 2024-09-11T09:52:26.517 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.517 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:26.517 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.517 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:26.517 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951291+0000", 2024-09-11T09:52:26.517 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:26.517 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.517 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.517 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:26.517 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951291+0000", 2024-09-11T09:52:26.517 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:26.517 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.517 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.518 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:26.518 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951293+0000", 2024-09-11T09:52:26.518 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7889999999999999e-06 2024-09-11T09:52:26.518 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.518 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.518 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:26.518 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951302+0000", 2024-09-11T09:52:26.518 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.9819999999999997e-06 2024-09-11T09:52:26.518 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.518 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.518 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:26.518 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951324+0000", 2024-09-11T09:52:26.518 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.2141e-05 2024-09-11T09:52:26.518 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.518 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.519 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:26.519 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951338+0000", 2024-09-11T09:52:26.519 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3750999999999999e-05 2024-09-11T09:52:26.519 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.519 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.519 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:26.519 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951409+0000", 2024-09-11T09:52:26.519 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.1658999999999999e-05 2024-09-11T09:52:26.519 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.519 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.519 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:26.519 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951517+0000", 2024-09-11T09:52:26.519 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00010746799999999999 2024-09-11T09:52:26.519 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.519 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.520 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 5", 2024-09-11T09:52:26.520 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951942+0000", 2024-09-11T09:52:26.520 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00042524300000000001 2024-09-11T09:52:26.520 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.520 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.520 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:26.520 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.952872+0000", 2024-09-11T09:52:26.520 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00092976300000000001 2024-09-11T09:52:26.520 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.520 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.520 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:26.520 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.955682+0000", 2024-09-11T09:52:26.520 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0028099459999999998 2024-09-11T09:52:26.521 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.521 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.521 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:26.521 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.955746+0000", 2024-09-11T09:52:26.521 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.4084999999999994e-05 2024-09-11T09:52:26.521 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.521 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.521 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:26.521 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.955787+0000", 2024-09-11T09:52:26.521 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.1672999999999999e-05 2024-09-11T09:52:26.521 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:26.521 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:26.521 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:26.522 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.522 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.522 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4688.0:52 2.4 e53/32 2:26d5c2f0:::rbd_data.1250f5e25b80.000000000000000b:head v 53'8, mlcod=53'6)", 2024-09-11T09:52:26.522 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.963677+0000", 2024-09-11T09:52:26.522 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 63.519449973999997, 2024-09-11T09:52:26.522 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0089437969999999999, 2024-09-11T09:52:26.522 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:26.522 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:26.522 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:26.522 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:26.522 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.522 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:26.522 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.963677+0000", 2024-09-11T09:52:26.522 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:26.522 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.523 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.523 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:26.523 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.963677+0000", 2024-09-11T09:52:26.523 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:26.523 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.523 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.523 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:26.523 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.963677+0000", 2024-09-11T09:52:26.523 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.7199999999999998e-07 2024-09-11T09:52:26.523 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.523 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.523 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:26.523 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.965506+0000", 2024-09-11T09:52:26.523 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.001829064 2024-09-11T09:52:26.524 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.524 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.524 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:26.524 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.965538+0000", 2024-09-11T09:52:26.524 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.1306999999999997e-05 2024-09-11T09:52:26.524 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.524 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.524 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:26.524 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.965553+0000", 2024-09-11T09:52:26.524 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.4924e-05 2024-09-11T09:52:26.524 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.524 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.524 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:26.524 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.965663+0000", 2024-09-11T09:52:26.525 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000110315 2024-09-11T09:52:26.525 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.525 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.525 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:26.525 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.965692+0000", 2024-09-11T09:52:26.525 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.934e-05 2024-09-11T09:52:26.525 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.525 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.525 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:26.525 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.972537+0000", 2024-09-11T09:52:26.525 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0068444969999999997 2024-09-11T09:52:26.525 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.525 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.526 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:26.526 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.972620+0000", 2024-09-11T09:52:26.526 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.3578000000000005e-05 2024-09-11T09:52:26.526 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:26.526 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:26.526 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:26.526 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.526 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.526 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4688.0:54 2.4 e53/32 2:239e915b:::rbd_data.1250f5e25b80.000000000000000c:head v 53'9, mlcod=53'7)", 2024-09-11T09:52:26.526 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.987028+0000", 2024-09-11T09:52:26.526 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 63.496098056999998, 2024-09-11T09:52:26.526 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0094155650000000007, 2024-09-11T09:52:26.526 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:26.526 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:26.527 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:26.527 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:26.527 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.527 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:26.527 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.987028+0000", 2024-09-11T09:52:26.527 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:26.527 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.527 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.527 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:26.527 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.987028+0000", 2024-09-11T09:52:26.527 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:26.527 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.527 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.528 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:26.528 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.987030+0000", 2024-09-11T09:52:26.528 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6130000000000001e-06 2024-09-11T09:52:26.528 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.528 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.528 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:26.528 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.988575+0000", 2024-09-11T09:52:26.528 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.001545193 2024-09-11T09:52:26.528 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.528 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.528 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:26.528 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.988604+0000", 2024-09-11T09:52:26.528 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.9173e-05 2024-09-11T09:52:26.529 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.529 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.529 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:26.529 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.988620+0000", 2024-09-11T09:52:26.529 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.5517999999999999e-05 2024-09-11T09:52:26.529 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.529 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.529 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:26.529 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.988704+0000", 2024-09-11T09:52:26.529 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.3588e-05 2024-09-11T09:52:26.529 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.529 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.529 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:26.529 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.988735+0000", 2024-09-11T09:52:26.530 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.1643000000000002e-05 2024-09-11T09:52:26.530 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.530 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.530 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:26.530 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.996383+0000", 2024-09-11T09:52:26.530 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0076478099999999997 2024-09-11T09:52:26.530 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.530 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.530 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:26.530 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.996444+0000", 2024-09-11T09:52:26.530 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.1026999999999997e-05 2024-09-11T09:52:26.530 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:26.530 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:26.530 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:26.530 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.531 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.531 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4688.0:56 2.4 e53/32 2:2167012f:::rbd_data.1250f5e25b80.000000000000000d:head v 53'10, mlcod=53'7)", 2024-09-11T09:52:26.531 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.993894+0000", 2024-09-11T09:52:26.531 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 63.489232514999998, 2024-09-11T09:52:26.531 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0099268559999999995, 2024-09-11T09:52:26.531 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:26.531 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:26.531 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:26.531 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:26.531 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.531 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:26.531 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.993894+0000", 2024-09-11T09:52:26.531 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:26.532 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.532 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.532 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:26.532 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.993894+0000", 2024-09-11T09:52:26.532 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:26.532 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.532 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.532 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:26.532 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.993896+0000", 2024-09-11T09:52:26.532 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6249999999999999e-06 2024-09-11T09:52:26.532 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.532 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.532 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:26.532 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.997606+0000", 2024-09-11T09:52:26.533 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.003710737 2024-09-11T09:52:26.533 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.533 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.533 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:26.533 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.997628+0000", 2024-09-11T09:52:26.533 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.1971e-05 2024-09-11T09:52:26.533 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.533 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.533 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:26.533 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.997640+0000", 2024-09-11T09:52:26.533 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1481000000000001e-05 2024-09-11T09:52:26.533 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.533 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.534 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:26.534 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.997712+0000", 2024-09-11T09:52:26.534 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.2255000000000005e-05 2024-09-11T09:52:26.534 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.534 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.534 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:26.534 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.997738+0000", 2024-09-11T09:52:26.534 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.6050999999999998e-05 2024-09-11T09:52:26.534 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.534 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.534 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:26.534 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.003779+0000", 2024-09-11T09:52:26.534 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0060410960000000001 2024-09-11T09:52:26.534 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.535 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.535 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:26.535 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.003821+0000", 2024-09-11T09:52:26.535 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.1640000000000001e-05 2024-09-11T09:52:26.535 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:26.535 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:26.535 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:26.535 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.535 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.535 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "pg_update_log_missing(2.4 epoch 53/32 rep_tid 16 entries 53'11 (0'0) error 2:2906e6a7:::rbd_header.1250f5e25b80:head by client.4688.0:59 0.000000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 53'7)", 2024-09-11T09:52:26.535 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.998956+0000", 2024-09-11T09:52:26.535 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 63.484170345999999, 2024-09-11T09:52:26.535 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.004934127, 2024-09-11T09:52:26.535 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:26.535 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:26.536 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "reached pg", 2024-09-11T09:52:26.536 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:26.536 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.536 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:26.536 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.998956+0000", 2024-09-11T09:52:26.536 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:26.536 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.536 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.536 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:26.536 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.998956+0000", 2024-09-11T09:52:26.536 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:26.536 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.536 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.536 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:26.536 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.998957+0000", 2024-09-11T09:52:26.537 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.139e-06 2024-09-11T09:52:26.537 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.537 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.537 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:26.537 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.998985+0000", 2024-09-11T09:52:26.537 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.7912e-05 2024-09-11T09:52:26.537 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.537 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.537 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:26.537 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.999011+0000", 2024-09-11T09:52:26.537 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.6061e-05 2024-09-11T09:52:26.537 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.537 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.537 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:26.537 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.999028+0000", 2024-09-11T09:52:26.538 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7065999999999999e-05 2024-09-11T09:52:26.538 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.538 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.538 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:26.538 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.001919+0000", 2024-09-11T09:52:26.538 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.002891167 2024-09-11T09:52:26.538 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.538 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.538 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:26.538 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.003890+0000", 2024-09-11T09:52:26.538 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0019707819999999999 2024-09-11T09:52:26.538 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:26.538 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:26.538 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:26.538 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.538 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.539 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4694.0:34 2.7 2:f80bfc0c:ns::rbd_data.1256a0c02879.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:26.539 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.085183+0000", 2024-09-11T09:52:26.539 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 63.397943832999999, 2024-09-11T09:52:26.539 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.020908599, 2024-09-11T09:52:26.539 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:26.539 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:26.539 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:26.539 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:26.539 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4694", 2024-09-11T09:52:26.539 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:52:26.539 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 34 2024-09-11T09:52:26.539 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.539 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:26.540 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.540 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:26.540 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.085183+0000", 2024-09-11T09:52:26.540 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:26.540 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.540 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.540 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:26.540 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.085183+0000", 2024-09-11T09:52:26.540 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:26.540 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.540 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.540 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:26.540 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.085183+0000", 2024-09-11T09:52:26.541 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.4300000000000002e-07 2024-09-11T09:52:26.541 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.541 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.541 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:26.541 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094337+0000", 2024-09-11T09:52:26.541 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0091532769999999996 2024-09-11T09:52:26.541 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.541 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.541 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:26.541 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094362+0000", 2024-09-11T09:52:26.541 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.5287999999999999e-05 2024-09-11T09:52:26.541 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.541 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.541 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:26.542 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094375+0000", 2024-09-11T09:52:26.542 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2446000000000001e-05 2024-09-11T09:52:26.542 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.542 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.542 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:26.542 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094455+0000", 2024-09-11T09:52:26.542 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.0092999999999994e-05 2024-09-11T09:52:26.542 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.542 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.542 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:26.542 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094618+0000", 2024-09-11T09:52:26.542 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000163747 2024-09-11T09:52:26.542 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.543 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.543 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 5", 2024-09-11T09:52:26.543 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094783+0000", 2024-09-11T09:52:26.543 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000165038 2024-09-11T09:52:26.543 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.543 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.543 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:26.543 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.105853+0000", 2024-09-11T09:52:26.543 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.011069294 2024-09-11T09:52:26.543 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.543 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.543 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:26.543 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.106004+0000", 2024-09-11T09:52:26.543 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00015123900000000001 2024-09-11T09:52:26.544 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.544 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.544 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:26.544 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.106062+0000", 2024-09-11T09:52:26.544 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.8205000000000003e-05 2024-09-11T09:52:26.544 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.544 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.544 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:26.544 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.106091+0000", 2024-09-11T09:52:26.544 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.9128999999999998e-05 2024-09-11T09:52:26.544 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:26.544 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:26.545 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:26.545 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.545 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.545 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4694.0:35 2.7 2:e8766325:ns::rbd_object_map.1256a0c02879:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:26.545 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.094405+0000", 2024-09-11T09:52:26.545 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 63.388721197000002, 2024-09-11T09:52:26.545 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.012623564, 2024-09-11T09:52:26.545 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:26.545 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:26.545 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:26.545 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:26.545 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4694", 2024-09-11T09:52:26.545 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:52:26.546 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 35 2024-09-11T09:52:26.546 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.546 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:26.546 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.546 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:26.546 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094405+0000", 2024-09-11T09:52:26.546 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:26.546 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.546 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.546 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:26.546 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094405+0000", 2024-09-11T09:52:26.546 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:26.546 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.547 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.547 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:26.547 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094407+0000", 2024-09-11T09:52:26.547 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.739e-06 2024-09-11T09:52:26.547 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.547 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.547 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:26.547 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094411+0000", 2024-09-11T09:52:26.547 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.3340000000000003e-06 2024-09-11T09:52:26.547 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.547 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.547 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:26.547 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094419+0000", 2024-09-11T09:52:26.547 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.8909999999999995e-06 2024-09-11T09:52:26.548 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.548 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.548 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:26.548 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094424+0000", 2024-09-11T09:52:26.548 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.7639999999999996e-06 2024-09-11T09:52:26.548 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.548 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.548 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:26.548 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.100209+0000", 2024-09-11T09:52:26.548 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0057850489999999996 2024-09-11T09:52:26.548 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.548 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.548 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:26.548 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.100341+0000", 2024-09-11T09:52:26.549 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000131438 2024-09-11T09:52:26.549 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.549 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.549 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 5", 2024-09-11T09:52:26.549 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.100805+0000", 2024-09-11T09:52:26.549 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00046471499999999998 2024-09-11T09:52:26.549 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.549 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.549 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:26.549 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.106148+0000", 2024-09-11T09:52:26.549 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0053426460000000004 2024-09-11T09:52:26.549 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.550 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.550 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:26.550 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.106929+0000", 2024-09-11T09:52:26.550 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00078161300000000001 2024-09-11T09:52:26.550 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.550 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.550 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:26.550 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.106985+0000", 2024-09-11T09:52:26.550 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.5223999999999998e-05 2024-09-11T09:52:26.550 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.550 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.550 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:26.550 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.107029+0000", 2024-09-11T09:52:26.550 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.4150999999999998e-05 2024-09-11T09:52:26.551 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:26.551 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:26.551 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:26.551 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.551 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.551 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4694.0:36 2.7 2:e8766325:ns::rbd_object_map.1256a0c02879:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:26.551 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.094452+0000", 2024-09-11T09:52:26.551 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 63.388674186999999, 2024-09-11T09:52:26.551 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.012680383, 2024-09-11T09:52:26.551 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:26.551 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:26.551 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:26.551 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:26.551 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4694", 2024-09-11T09:52:26.551 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:52:26.552 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 36 2024-09-11T09:52:26.552 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.552 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:26.552 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.552 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:26.552 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094452+0000", 2024-09-11T09:52:26.552 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:26.552 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.552 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.552 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:26.552 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094452+0000", 2024-09-11T09:52:26.552 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:26.552 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.552 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.553 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:26.553 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094453+0000", 2024-09-11T09:52:26.553 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.1800000000000002e-07 2024-09-11T09:52:26.553 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.553 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.553 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:26.553 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094455+0000", 2024-09-11T09:52:26.553 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.7020000000000001e-06 2024-09-11T09:52:26.553 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.553 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.553 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:26.553 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094463+0000", 2024-09-11T09:52:26.553 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.1999999999999997e-06 2024-09-11T09:52:26.554 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.554 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.554 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:26.554 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094467+0000", 2024-09-11T09:52:26.554 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.729e-06 2024-09-11T09:52:26.554 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.554 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.554 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:26.554 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.101269+0000", 2024-09-11T09:52:26.554 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0068018799999999997 2024-09-11T09:52:26.554 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.554 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.554 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:26.555 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.101353+0000", 2024-09-11T09:52:26.555 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.3827999999999998e-05 2024-09-11T09:52:26.555 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.555 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.555 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 5", 2024-09-11T09:52:26.555 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.101588+0000", 2024-09-11T09:52:26.555 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00023467099999999999 2024-09-11T09:52:26.555 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.555 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.556 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:26.556 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.106182+0000", 2024-09-11T09:52:26.556 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0045939489999999999 2024-09-11T09:52:26.556 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.556 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.556 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:26.556 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.107041+0000", 2024-09-11T09:52:26.556 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00085965800000000002 2024-09-11T09:52:26.556 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.556 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.556 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:26.556 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.107084+0000", 2024-09-11T09:52:26.556 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.261e-05 2024-09-11T09:52:26.556 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.556 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.557 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:26.557 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.107133+0000", 2024-09-11T09:52:26.557 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.8838000000000002e-05 2024-09-11T09:52:26.557 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:26.557 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:26.557 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:26.557 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.557 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.557 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4694.0:37 2.7 2:e8766325:ns::rbd_object_map.1256a0c02879:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:26.557 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.094473+0000", 2024-09-11T09:52:26.557 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 63.388653036000001, 2024-09-11T09:52:26.557 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.012778484999999999, 2024-09-11T09:52:26.557 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:26.557 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:26.558 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:26.558 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:26.558 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4694", 2024-09-11T09:52:26.558 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:52:26.558 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 37 2024-09-11T09:52:26.558 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.558 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:26.558 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.558 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:26.558 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094473+0000", 2024-09-11T09:52:26.558 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:26.558 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.558 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.559 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:26.559 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094473+0000", 2024-09-11T09:52:26.559 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:26.559 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.559 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.559 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:26.559 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094474+0000", 2024-09-11T09:52:26.559 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.9699999999999996e-07 2024-09-11T09:52:26.559 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.559 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.559 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:26.559 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094477+0000", 2024-09-11T09:52:26.559 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.151e-06 2024-09-11T09:52:26.560 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.560 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.560 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:26.560 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094483+0000", 2024-09-11T09:52:26.560 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.6330000000000001e-06 2024-09-11T09:52:26.560 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.560 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.560 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:26.560 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094487+0000", 2024-09-11T09:52:26.560 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.5399999999999997e-06 2024-09-11T09:52:26.560 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.560 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.560 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:26.560 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.101866+0000", 2024-09-11T09:52:26.561 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.007378768 2024-09-11T09:52:26.561 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.561 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.561 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:26.561 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.101960+0000", 2024-09-11T09:52:26.561 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.3861000000000001e-05 2024-09-11T09:52:26.561 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.561 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.561 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 5", 2024-09-11T09:52:26.561 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.102197+0000", 2024-09-11T09:52:26.561 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00023723200000000001 2024-09-11T09:52:26.561 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.561 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.561 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:26.562 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.106222+0000", 2024-09-11T09:52:26.562 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0040249689999999998 2024-09-11T09:52:26.562 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.562 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.562 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:26.562 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.107141+0000", 2024-09-11T09:52:26.562 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00091913900000000002 2024-09-11T09:52:26.562 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.562 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.562 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:26.562 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.107198+0000", 2024-09-11T09:52:26.562 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.6564999999999998e-05 2024-09-11T09:52:26.563 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.563 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.563 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:26.563 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.107252+0000", 2024-09-11T09:52:26.563 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.4030000000000003e-05 2024-09-11T09:52:26.563 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:26.563 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:26.563 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:26.563 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.563 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.563 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4694.0:33 2.4 e53/32 2:3bfc818c:ns::rbd_data.1256a0c02879.0000000000000001:head v 53'14, mlcod=53'12)", 2024-09-11T09:52:26.563 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.096496+0000", 2024-09-11T09:52:26.563 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 63.386630463000003, 2024-09-11T09:52:26.563 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0099139310000000008, 2024-09-11T09:52:26.564 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:26.564 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:26.564 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:26.564 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:26.564 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.564 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:26.564 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.096496+0000", 2024-09-11T09:52:26.564 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:26.564 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.564 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.564 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:26.564 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.096496+0000", 2024-09-11T09:52:26.564 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:26.564 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.565 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.565 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:26.565 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.096498+0000", 2024-09-11T09:52:26.565 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6309999999999999e-06 2024-09-11T09:52:26.565 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.565 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.565 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:26.565 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.098087+0000", 2024-09-11T09:52:26.565 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0015889019999999999 2024-09-11T09:52:26.565 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.565 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.565 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:26.565 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.098112+0000", 2024-09-11T09:52:26.565 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.5369000000000001e-05 2024-09-11T09:52:26.566 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.566 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.566 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:26.566 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.098127+0000", 2024-09-11T09:52:26.566 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.5218999999999999e-05 2024-09-11T09:52:26.566 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.566 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.566 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:26.566 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.098209+0000", 2024-09-11T09:52:26.566 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.2120999999999995e-05 2024-09-11T09:52:26.566 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.566 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.566 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:26.566 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.098237+0000", 2024-09-11T09:52:26.566 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.7657999999999998e-05 2024-09-11T09:52:26.567 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.567 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.567 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:26.567 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.106376+0000", 2024-09-11T09:52:26.567 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0081386749999999997 2024-09-11T09:52:26.567 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.567 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.567 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:26.567 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.106410+0000", 2024-09-11T09:52:26.567 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.4356000000000001e-05 2024-09-11T09:52:26.567 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:26.567 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:26.567 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:26.567 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.567 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.568 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4694.0:49 2.7 2:e8766325:ns::rbd_object_map.1256a0c02879:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:26.568 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.134806+0000", 2024-09-11T09:52:26.568 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 63.348320393999998, 2024-09-11T09:52:26.568 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0061629049999999998, 2024-09-11T09:52:26.568 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:26.568 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:26.568 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:26.568 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:26.568 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4694", 2024-09-11T09:52:26.568 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:52:26.568 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 49 2024-09-11T09:52:26.568 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.568 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:26.568 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.569 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:26.569 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.134806+0000", 2024-09-11T09:52:26.569 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:26.569 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.569 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.569 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:26.569 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.134806+0000", 2024-09-11T09:52:26.569 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:26.569 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.569 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.569 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:26.569 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.134808+0000", 2024-09-11T09:52:26.569 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.9870000000000002e-06 2024-09-11T09:52:26.569 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.569 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.570 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:26.570 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.134818+0000", 2024-09-11T09:52:26.570 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.5599999999999999e-06 2024-09-11T09:52:26.570 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.570 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.570 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:26.570 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.134835+0000", 2024-09-11T09:52:26.570 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7399000000000001e-05 2024-09-11T09:52:26.570 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.570 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.570 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:26.570 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.134845+0000", 2024-09-11T09:52:26.570 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0285000000000001e-05 2024-09-11T09:52:26.571 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.571 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.571 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:26.571 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.134911+0000", 2024-09-11T09:52:26.571 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.5654999999999994e-05 2024-09-11T09:52:26.571 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.571 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.571 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:26.571 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.135023+0000", 2024-09-11T09:52:26.571 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00011186799999999999 2024-09-11T09:52:26.571 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.571 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.571 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 5", 2024-09-11T09:52:26.572 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.135290+0000", 2024-09-11T09:52:26.572 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00026725300000000001 2024-09-11T09:52:26.572 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.572 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.572 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:26.572 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.136506+0000", 2024-09-11T09:52:26.572 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0012161139999999999 2024-09-11T09:52:26.572 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.572 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.572 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:26.572 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.140884+0000", 2024-09-11T09:52:26.572 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0043781990000000002 2024-09-11T09:52:26.572 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.572 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.573 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:26.573 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.140923+0000", 2024-09-11T09:52:26.573 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.8813999999999998e-05 2024-09-11T09:52:26.573 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.573 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.573 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:26.573 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.140969+0000", 2024-09-11T09:52:26.573 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.5770999999999999e-05 2024-09-11T09:52:26.573 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:26.573 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:26.573 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:26.573 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.574 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.574 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4694.0:42 2.5 e53/32 2:a4fc00ec:ns::rbd_data.1256a0c02879.0000000000000004:head v 53'3, mlcod=53'1)", 2024-09-11T09:52:26.574 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.136646+0000", 2024-09-11T09:52:26.574 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 63.346480088, 2024-09-11T09:52:26.574 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.012545501000000001, 2024-09-11T09:52:26.574 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:26.574 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:26.574 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:26.574 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:26.574 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.574 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:26.574 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.136646+0000", 2024-09-11T09:52:26.574 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:26.574 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.575 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.575 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:26.575 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.136646+0000", 2024-09-11T09:52:26.575 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:26.575 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.575 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.575 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:26.575 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.136647+0000", 2024-09-11T09:52:26.575 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.2900000000000002e-07 2024-09-11T09:52:26.575 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.575 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.575 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:26.575 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.143620+0000", 2024-09-11T09:52:26.575 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0069727440000000003 2024-09-11T09:52:26.576 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.576 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.576 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:26.576 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.143642+0000", 2024-09-11T09:52:26.576 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.1846000000000001e-05 2024-09-11T09:52:26.576 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.576 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.576 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:26.576 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.143654+0000", 2024-09-11T09:52:26.576 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2081e-05 2024-09-11T09:52:26.576 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.576 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.576 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:26.577 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.143717+0000", 2024-09-11T09:52:26.577 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.3443999999999998e-05 2024-09-11T09:52:26.577 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.577 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.577 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:26.577 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.143740+0000", 2024-09-11T09:52:26.577 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.2626999999999999e-05 2024-09-11T09:52:26.577 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.577 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.577 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:26.577 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.149142+0000", 2024-09-11T09:52:26.577 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0054017630000000004 2024-09-11T09:52:26.577 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.577 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.578 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:26.578 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.149192+0000", 2024-09-11T09:52:26.578 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.0067e-05 2024-09-11T09:52:26.578 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:26.578 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:26.578 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:26.578 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.578 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.578 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4694.0:53 2.7 2:e8766325:ns::rbd_object_map.1256a0c02879:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:26.578 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.147978+0000", 2024-09-11T09:52:26.578 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 63.335148603999997, 2024-09-11T09:52:26.578 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0044549769999999997, 2024-09-11T09:52:26.578 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:26.578 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:26.578 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:26.579 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:26.579 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4694", 2024-09-11T09:52:26.579 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:52:26.579 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 53 2024-09-11T09:52:26.579 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.579 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:26.579 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.579 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:26.579 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.147978+0000", 2024-09-11T09:52:26.579 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:26.579 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.579 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.579 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:26.579 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.147978+0000", 2024-09-11T09:52:26.580 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:26.580 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.580 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.580 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:26.580 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.147980+0000", 2024-09-11T09:52:26.580 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.8309999999999999e-06 2024-09-11T09:52:26.580 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.580 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.580 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:26.580 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.147989+0000", 2024-09-11T09:52:26.580 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.9639999999999992e-06 2024-09-11T09:52:26.580 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.580 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.580 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:26.580 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.148007+0000", 2024-09-11T09:52:26.581 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7884999999999999e-05 2024-09-11T09:52:26.581 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.581 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.581 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:26.581 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.148017+0000", 2024-09-11T09:52:26.581 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0895e-05 2024-09-11T09:52:26.581 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.581 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.581 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:26.581 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.148091+0000", 2024-09-11T09:52:26.581 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.3709999999999997e-05 2024-09-11T09:52:26.581 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.581 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.581 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:26.582 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.148189+0000", 2024-09-11T09:52:26.582 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.7403000000000002e-05 2024-09-11T09:52:26.582 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.582 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.582 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 5", 2024-09-11T09:52:26.582 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.148504+0000", 2024-09-11T09:52:26.582 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00031589900000000002 2024-09-11T09:52:26.582 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.582 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.582 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:26.582 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.149514+0000", 2024-09-11T09:52:26.582 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0010096 2024-09-11T09:52:26.582 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.582 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.583 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:26.583 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.152367+0000", 2024-09-11T09:52:26.583 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0028532309999999999 2024-09-11T09:52:26.583 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.583 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.583 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:26.583 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.152402+0000", 2024-09-11T09:52:26.583 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.4232e-05 2024-09-11T09:52:26.583 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.583 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.583 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:26.583 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.152433+0000", 2024-09-11T09:52:26.583 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.1327e-05 2024-09-11T09:52:26.584 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:26.584 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:26.584 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:26.584 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.584 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.584 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4694.0:55 2.7 2:e8766325:ns::rbd_object_map.1256a0c02879:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:26.584 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.149838+0000", 2024-09-11T09:52:26.584 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 63.333288871000001, 2024-09-11T09:52:26.584 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0060695159999999996, 2024-09-11T09:52:26.584 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:26.584 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:26.584 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:26.584 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:26.584 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4694", 2024-09-11T09:52:26.585 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:52:26.585 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 55 2024-09-11T09:52:26.585 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.585 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:26.585 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.585 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:26.585 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.149838+0000", 2024-09-11T09:52:26.585 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:26.585 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.585 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.585 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:26.585 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.149838+0000", 2024-09-11T09:52:26.585 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:26.585 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.585 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.586 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:26.586 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.149839+0000", 2024-09-11T09:52:26.586 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.119e-06 2024-09-11T09:52:26.586 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.586 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.586 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:26.586 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.149845+0000", 2024-09-11T09:52:26.586 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.7810000000000002e-06 2024-09-11T09:52:26.586 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.586 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.586 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:26.586 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.149856+0000", 2024-09-11T09:52:26.586 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1199999999999999e-05 2024-09-11T09:52:26.586 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.587 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.587 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:26.587 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.149863+0000", 2024-09-11T09:52:26.587 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.4109999999999999e-06 2024-09-11T09:52:26.587 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.587 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.587 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:26.587 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.149908+0000", 2024-09-11T09:52:26.587 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.5184e-05 2024-09-11T09:52:26.587 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.587 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.587 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:26.587 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.149964+0000", 2024-09-11T09:52:26.587 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.5395000000000001e-05 2024-09-11T09:52:26.587 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.587 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.588 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 5", 2024-09-11T09:52:26.588 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.150157+0000", 2024-09-11T09:52:26.588 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000193489 2024-09-11T09:52:26.588 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.588 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.588 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:26.588 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.152238+0000", 2024-09-11T09:52:26.588 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0020803480000000001 2024-09-11T09:52:26.588 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.589 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.589 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:26.589 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.155828+0000", 2024-09-11T09:52:26.589 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0035902210000000002 2024-09-11T09:52:26.589 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.589 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.589 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:26.589 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.155873+0000", 2024-09-11T09:52:26.589 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.5145999999999998e-05 2024-09-11T09:52:26.589 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.589 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.589 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:26.589 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.155907+0000", 2024-09-11T09:52:26.590 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.4221999999999999e-05 2024-09-11T09:52:26.590 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:26.590 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:26.590 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:26.590 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:26.590 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:26.590 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:52:26.591 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-11T09:52:26.592 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:52:26.592+0000 7f3c96ec9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31909) UID: 0 2024-09-11T09:52:26.692 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:52:26.692+0000 7f3c96ec9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31909) UID: 0 2024-09-11T09:52:26.740 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:52:26.740 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [], 2024-09-11T09:52:26.740 INFO:teuthology.orchestra.run.smithi017.stdout: "num_ops": 0 2024-09-11T09:52:26.740 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:52:26.748 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-11T09:52:26.793 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-09-11T09:52:26.794+0000 7f7b6c7d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31904) UID: 0 2024-09-11T09:52:26.855 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:52:26.855 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [], 2024-09-11T09:52:26.855 INFO:teuthology.orchestra.run.smithi017.stdout: "complaint_time": 30, 2024-09-11T09:52:26.855 INFO:teuthology.orchestra.run.smithi017.stdout: "num_blocked_ops": 0 2024-09-11T09:52:26.855 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:52:26.864 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-11T09:52:26.909 INFO:tasks.ceph.osd.0.smithi017.stderr:2024-09-11T09:52:26.910+0000 7fb73c0f1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31665) UID: 0 2024-09-11T09:52:26.919 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-11T09:52:26.923 INFO:tasks.workunit.client.0.smithi107.stdout:Watchers: none 2024-09-11T09:52:26.927 INFO:tasks.workunit.client.0.smithi107.stderr:+ for s in 0.5 1 2 4 8 16 32 2024-09-11T09:52:26.927 INFO:tasks.workunit.client.0.smithi107.stderr:+ sleep 0.5 2024-09-11T09:52:26.979 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:52:26.979 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 20, 2024-09-11T09:52:26.979 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 600, 2024-09-11T09:52:26.979 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [ 2024-09-11T09:52:26.979 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.979 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:34 2.4 2:2367385a:::rbd_data.1250f5e25b80.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:26.979 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.897717+0000", 2024-09-11T09:52:26.979 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 64.080918080999993, 2024-09-11T09:52:26.979 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.021323661000000001, 2024-09-11T09:52:26.979 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:26.980 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:26.980 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:26.980 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:26.980 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:52:26.980 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:52:26.980 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 34 2024-09-11T09:52:26.980 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.980 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:26.980 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.980 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:26.980 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.897717+0000", 2024-09-11T09:52:26.980 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:26.980 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.980 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.981 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:26.981 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.897717+0000", 2024-09-11T09:52:26.981 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:26.981 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.981 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.981 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:26.981 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.897718+0000", 2024-09-11T09:52:26.981 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.1800000000000004e-07 2024-09-11T09:52:26.981 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.981 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.981 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:26.981 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.905339+0000", 2024-09-11T09:52:26.982 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0076210239999999997 2024-09-11T09:52:26.982 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.982 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.982 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:26.982 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.905366+0000", 2024-09-11T09:52:26.982 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.6959000000000001e-05 2024-09-11T09:52:26.982 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.982 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.982 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:26.982 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.905375+0000", 2024-09-11T09:52:26.982 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.516e-06 2024-09-11T09:52:26.982 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.982 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.983 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:26.983 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.905475+0000", 2024-09-11T09:52:26.983 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.9738e-05 2024-09-11T09:52:26.983 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.983 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.983 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:26.983 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.905660+0000", 2024-09-11T09:52:26.983 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00018554199999999999 2024-09-11T09:52:26.983 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.983 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.983 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 0", 2024-09-11T09:52:26.983 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.905808+0000", 2024-09-11T09:52:26.983 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000147386 2024-09-11T09:52:26.984 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.984 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.984 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:26.984 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.918230+0000", 2024-09-11T09:52:26.984 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.012421690000000001 2024-09-11T09:52:26.984 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.984 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.984 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:26.984 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.918932+0000", 2024-09-11T09:52:26.984 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00070267300000000001 2024-09-11T09:52:26.984 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.984 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.984 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:26.985 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.918989+0000", 2024-09-11T09:52:26.985 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.7268999999999997e-05 2024-09-11T09:52:26.985 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.985 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.985 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:26.985 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919040+0000", 2024-09-11T09:52:26.985 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.0946000000000003e-05 2024-09-11T09:52:26.985 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:26.985 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:26.985 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:26.985 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.985 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.985 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4688.0:31 2.1 e53/15 2:8654721b:::rbd_data.1250f5e25b80.0000000000000000:head v 53'2, mlcod=0'0)", 2024-09-11T09:52:26.986 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.904661+0000", 2024-09-11T09:52:26.986 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 64.073973592000002, 2024-09-11T09:52:26.986 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.014152804999999999, 2024-09-11T09:52:26.986 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:26.986 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:26.986 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:26.986 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:26.986 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.986 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:26.986 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.904661+0000", 2024-09-11T09:52:26.986 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:26.986 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.986 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.986 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:26.986 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.904661+0000", 2024-09-11T09:52:26.987 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:26.987 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.987 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.987 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:26.987 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.904662+0000", 2024-09-11T09:52:26.987 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0759999999999999e-06 2024-09-11T09:52:26.987 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.987 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.987 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:26.987 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.907298+0000", 2024-09-11T09:52:26.987 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0026361240000000001 2024-09-11T09:52:26.987 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.987 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.987 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:26.988 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.907320+0000", 2024-09-11T09:52:26.988 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.1268999999999999e-05 2024-09-11T09:52:26.988 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.988 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.988 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:26.988 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.907331+0000", 2024-09-11T09:52:26.988 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1031e-05 2024-09-11T09:52:26.988 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.988 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.988 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:26.988 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.907403+0000", 2024-09-11T09:52:26.988 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.2571999999999996e-05 2024-09-11T09:52:26.988 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.988 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.989 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:26.989 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.907428+0000", 2024-09-11T09:52:26.989 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.4380999999999999e-05 2024-09-11T09:52:26.989 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.989 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.989 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:26.989 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.918750+0000", 2024-09-11T09:52:26.989 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.011322211 2024-09-11T09:52:26.989 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.989 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.989 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:26.989 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.918814+0000", 2024-09-11T09:52:26.990 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.4140999999999995e-05 2024-09-11T09:52:26.990 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:26.990 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:26.990 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:26.990 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.990 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.990 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:36 2.6 2:61e1b442:::rbd_data.1250f5e25b80.0000000000000003:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:26.991 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.905404+0000", 2024-09-11T09:52:26.991 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 64.073230721000002, 2024-09-11T09:52:26.991 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.030912479999999999, 2024-09-11T09:52:26.991 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:26.991 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:26.991 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:26.991 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:26.991 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:52:26.991 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:52:26.991 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 36 2024-09-11T09:52:26.991 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.991 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:26.992 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.992 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:26.992 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.905404+0000", 2024-09-11T09:52:26.992 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:26.992 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.992 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.992 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:26.992 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.905404+0000", 2024-09-11T09:52:26.992 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:26.992 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.992 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.992 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:26.992 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.905405+0000", 2024-09-11T09:52:26.993 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2270000000000001e-06 2024-09-11T09:52:26.993 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.993 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.993 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:26.993 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.914859+0000", 2024-09-11T09:52:26.993 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0094541190000000004 2024-09-11T09:52:26.993 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.993 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.993 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:26.993 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.914879+0000", 2024-09-11T09:52:26.993 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.0052999999999999e-05 2024-09-11T09:52:26.993 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.993 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.993 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:26.994 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.914897+0000", 2024-09-11T09:52:26.994 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7331000000000001e-05 2024-09-11T09:52:26.994 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.994 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.994 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:26.994 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.915019+0000", 2024-09-11T09:52:26.994 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00012168 2024-09-11T09:52:26.994 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.994 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.994 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:26.994 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.915269+0000", 2024-09-11T09:52:26.994 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00025054899999999998 2024-09-11T09:52:26.994 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.995 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.995 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 3", 2024-09-11T09:52:26.995 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.915479+0000", 2024-09-11T09:52:26.995 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00021017799999999999 2024-09-11T09:52:26.995 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.995 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.995 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:26.995 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.926070+0000", 2024-09-11T09:52:26.995 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.010590624999999999 2024-09-11T09:52:26.995 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.995 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.995 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:26.995 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.936198+0000", 2024-09-11T09:52:26.996 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.010128194 2024-09-11T09:52:26.996 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.996 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.996 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:26.996 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.936263+0000", 2024-09-11T09:52:26.996 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.5366999999999999e-05 2024-09-11T09:52:26.996 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.996 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.996 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:26.996 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.936317+0000", 2024-09-11T09:52:26.996 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.3156999999999999e-05 2024-09-11T09:52:26.996 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:26.996 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:26.996 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:26.997 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.997 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.997 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:39 2.6 2:743e7616:::rbd_data.1250f5e25b80.0000000000000005:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:26.997 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.914960+0000", 2024-09-11T09:52:26.997 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 64.063674418999994, 2024-09-11T09:52:26.997 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.022371518, 2024-09-11T09:52:26.997 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:26.997 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:26.997 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:26.997 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:26.997 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:52:26.997 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:52:26.997 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 39 2024-09-11T09:52:26.997 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.998 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:26.998 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.998 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:26.998 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.914960+0000", 2024-09-11T09:52:26.998 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:26.998 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.998 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.998 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:26.998 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.914960+0000", 2024-09-11T09:52:26.998 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:26.998 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.998 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.998 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:26.998 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.914963+0000", 2024-09-11T09:52:26.999 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.1119999999999999e-06 2024-09-11T09:52:26.999 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.999 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.999 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:26.999 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.920417+0000", 2024-09-11T09:52:26.999 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0054543969999999997 2024-09-11T09:52:26.999 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.999 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.999 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:26.999 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.920430+0000", 2024-09-11T09:52:26.999 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3414000000000001e-05 2024-09-11T09:52:26.999 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:26.999 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:26.999 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:26.999 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.920437+0000", 2024-09-11T09:52:27.000 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.6159999999999998e-06 2024-09-11T09:52:27.000 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.000 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.000 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:27.000 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.921325+0000", 2024-09-11T09:52:27.000 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000887731 2024-09-11T09:52:27.000 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.000 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.000 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:27.000 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.921422+0000", 2024-09-11T09:52:27.000 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.7201999999999995e-05 2024-09-11T09:52:27.000 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.000 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.000 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 3", 2024-09-11T09:52:27.001 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.921511+0000", 2024-09-11T09:52:27.001 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.8880000000000003e-05 2024-09-11T09:52:27.001 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.001 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.001 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:27.001 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.928092+0000", 2024-09-11T09:52:27.001 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0065816290000000003 2024-09-11T09:52:27.001 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.001 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.001 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:27.001 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.937240+0000", 2024-09-11T09:52:27.001 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0091481070000000008 2024-09-11T09:52:27.001 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.001 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.002 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:27.002 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.937294+0000", 2024-09-11T09:52:27.002 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.3100000000000003e-05 2024-09-11T09:52:27.002 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.002 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.002 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:27.002 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.937332+0000", 2024-09-11T09:52:27.002 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.8330000000000001e-05 2024-09-11T09:52:27.002 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:27.002 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:27.002 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:27.002 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.003 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.003 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4688.0:32 2.2 e53/15 2:45941fd2:::rbd_data.1250f5e25b80.0000000000000001:head v 53'3, mlcod=17'1)", 2024-09-11T09:52:27.003 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.916509+0000", 2024-09-11T09:52:27.003 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 64.062125336999998, 2024-09-11T09:52:27.003 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.015885318999999998, 2024-09-11T09:52:27.003 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:27.003 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:27.003 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:27.003 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:27.003 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.003 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:27.003 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.916509+0000", 2024-09-11T09:52:27.003 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:27.003 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.003 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.004 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:27.004 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.916509+0000", 2024-09-11T09:52:27.004 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:27.004 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.004 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.004 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:27.004 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.916511+0000", 2024-09-11T09:52:27.004 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.052e-06 2024-09-11T09:52:27.004 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.004 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.004 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:27.004 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.927019+0000", 2024-09-11T09:52:27.004 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.010508224 2024-09-11T09:52:27.005 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.005 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.005 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:27.005 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.927038+0000", 2024-09-11T09:52:27.005 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.9590999999999999e-05 2024-09-11T09:52:27.005 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.005 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.005 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:27.005 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.927050+0000", 2024-09-11T09:52:27.005 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1439e-05 2024-09-11T09:52:27.005 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.005 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.005 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:27.006 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.927115+0000", 2024-09-11T09:52:27.006 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.5301000000000004e-05 2024-09-11T09:52:27.006 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.006 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.006 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:27.006 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.927132+0000", 2024-09-11T09:52:27.006 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7297000000000001e-05 2024-09-11T09:52:27.006 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.006 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.006 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:27.006 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.932354+0000", 2024-09-11T09:52:27.006 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0052211870000000004 2024-09-11T09:52:27.006 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.007 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.007 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:27.007 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.932395+0000", 2024-09-11T09:52:27.007 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.1227999999999999e-05 2024-09-11T09:52:27.007 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:27.007 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:27.007 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:27.007 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.007 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.007 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:44 2.6 2:67246a22:::rbd_data.1250f5e25b80.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:27.007 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.943181+0000", 2024-09-11T09:52:27.008 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 64.035454126000005, 2024-09-11T09:52:27.008 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.022629371999999998, 2024-09-11T09:52:27.008 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:27.008 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:27.008 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:27.008 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:27.008 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:52:27.008 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:52:27.008 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 44 2024-09-11T09:52:27.008 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.008 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:27.008 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.009 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:27.009 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.943181+0000", 2024-09-11T09:52:27.009 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:27.009 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.009 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.009 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:27.009 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.943181+0000", 2024-09-11T09:52:27.009 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:27.009 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.009 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.009 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:27.009 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.943182+0000", 2024-09-11T09:52:27.010 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.0500000000000002e-07 2024-09-11T09:52:27.010 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.010 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.010 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:27.010 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951240+0000", 2024-09-11T09:52:27.010 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.008058529 2024-09-11T09:52:27.010 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.010 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.010 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:27.010 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951260+0000", 2024-09-11T09:52:27.010 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.9570000000000001e-05 2024-09-11T09:52:27.010 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.011 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.011 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:27.011 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951270+0000", 2024-09-11T09:52:27.011 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0018e-05 2024-09-11T09:52:27.011 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.011 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.011 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:27.011 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951349+0000", 2024-09-11T09:52:27.011 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.8801000000000007e-05 2024-09-11T09:52:27.011 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.011 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.011 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:27.012 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951483+0000", 2024-09-11T09:52:27.012 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00013427 2024-09-11T09:52:27.012 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.012 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.012 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 3", 2024-09-11T09:52:27.012 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951593+0000", 2024-09-11T09:52:27.012 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00011031000000000001 2024-09-11T09:52:27.012 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.012 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.012 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:27.012 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.957303+0000", 2024-09-11T09:52:27.013 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0057097959999999996 2024-09-11T09:52:27.013 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.013 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.013 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:27.013 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.965690+0000", 2024-09-11T09:52:27.013 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0083867420000000008 2024-09-11T09:52:27.013 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.013 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.013 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:27.013 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.965749+0000", 2024-09-11T09:52:27.013 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.8981999999999999e-05 2024-09-11T09:52:27.013 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.014 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.014 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:27.014 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.965810+0000", 2024-09-11T09:52:27.014 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.1449000000000001e-05 2024-09-11T09:52:27.014 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:27.014 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:27.014 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:27.014 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.014 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.014 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:51 2.6 2:6b5894fc:::rbd_data.1250f5e25b80.000000000000000a:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:27.014 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.951324+0000", 2024-09-11T09:52:27.014 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 64.027311053000005, 2024-09-11T09:52:27.015 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.021375273, 2024-09-11T09:52:27.015 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:27.015 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:27.015 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:27.015 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:27.015 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:52:27.015 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:52:27.015 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 51 2024-09-11T09:52:27.015 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.015 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:27.015 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.016 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:27.016 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951324+0000", 2024-09-11T09:52:27.016 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:27.016 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.016 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.016 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:27.016 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951324+0000", 2024-09-11T09:52:27.016 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:27.016 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.016 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.016 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:27.016 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951324+0000", 2024-09-11T09:52:27.016 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.39e-07 2024-09-11T09:52:27.017 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.017 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.017 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:27.017 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.958537+0000", 2024-09-11T09:52:27.017 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0072126769999999998 2024-09-11T09:52:27.017 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.017 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.017 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:27.017 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.958554+0000", 2024-09-11T09:52:27.017 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7232e-05 2024-09-11T09:52:27.017 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.017 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.017 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:27.018 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.958570+0000", 2024-09-11T09:52:27.018 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.5747999999999999e-05 2024-09-11T09:52:27.018 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.018 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.018 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:27.018 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.958638+0000", 2024-09-11T09:52:27.018 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.7810000000000003e-05 2024-09-11T09:52:27.018 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.018 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.018 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:27.018 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.958758+0000", 2024-09-11T09:52:27.018 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000120689 2024-09-11T09:52:27.018 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.018 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.019 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 3", 2024-09-11T09:52:27.019 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.958865+0000", 2024-09-11T09:52:27.019 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00010692 2024-09-11T09:52:27.019 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.019 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.019 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:27.019 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.970712+0000", 2024-09-11T09:52:27.019 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.011846327 2024-09-11T09:52:27.019 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.019 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.019 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:27.019 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.972603+0000", 2024-09-11T09:52:27.020 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0018918610000000001 2024-09-11T09:52:27.020 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.020 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.020 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:27.020 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.972653+0000", 2024-09-11T09:52:27.020 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.9762999999999998e-05 2024-09-11T09:52:27.020 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.020 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.020 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:27.020 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.972699+0000", 2024-09-11T09:52:27.020 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.5806999999999997e-05 2024-09-11T09:52:27.020 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:27.020 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:27.021 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:27.021 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.021 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.021 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:52 2.4 2:26d5c2f0:::rbd_data.1250f5e25b80.000000000000000b:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:27.021 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.958596+0000", 2024-09-11T09:52:27.021 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 64.020038858000007, 2024-09-11T09:52:27.021 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.016477849999999999, 2024-09-11T09:52:27.021 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:27.021 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:27.021 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:27.021 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:27.021 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:52:27.021 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:52:27.022 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 52 2024-09-11T09:52:27.022 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.022 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:27.022 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.022 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:27.022 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.958596+0000", 2024-09-11T09:52:27.022 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:27.022 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.022 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.022 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:27.022 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.958596+0000", 2024-09-11T09:52:27.022 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:27.022 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.023 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.023 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:27.023 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.958597+0000", 2024-09-11T09:52:27.023 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1960000000000001e-06 2024-09-11T09:52:27.023 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.023 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.023 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:27.023 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.963157+0000", 2024-09-11T09:52:27.023 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0045603029999999999 2024-09-11T09:52:27.023 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.023 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.023 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:27.023 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.963171+0000", 2024-09-11T09:52:27.024 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3777999999999999e-05 2024-09-11T09:52:27.024 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.024 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.024 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:27.024 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.963181+0000", 2024-09-11T09:52:27.024 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.9010000000000006e-06 2024-09-11T09:52:27.024 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.024 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.024 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:27.024 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.963246+0000", 2024-09-11T09:52:27.024 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.5351000000000005e-05 2024-09-11T09:52:27.024 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.024 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.025 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:27.025 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.963419+0000", 2024-09-11T09:52:27.025 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00017299700000000001 2024-09-11T09:52:27.025 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.025 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.025 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 0", 2024-09-11T09:52:27.025 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.963570+0000", 2024-09-11T09:52:27.025 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00015066999999999999 2024-09-11T09:52:27.025 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.025 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.025 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:27.025 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.973881+0000", 2024-09-11T09:52:27.025 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.010311268 2024-09-11T09:52:27.026 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.026 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.026 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:27.026 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.974963+0000", 2024-09-11T09:52:27.026 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.001081493 2024-09-11T09:52:27.026 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.026 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.026 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:27.026 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.975012+0000", 2024-09-11T09:52:27.026 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.9471000000000001e-05 2024-09-11T09:52:27.026 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.026 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.026 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:27.026 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.975074+0000", 2024-09-11T09:52:27.027 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.1421999999999996e-05 2024-09-11T09:52:27.027 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:27.027 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:27.027 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:27.027 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.027 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.027 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4688.0:47 2.1 e53/15 2:95f53b3e:::rbd_data.1250f5e25b80.0000000000000008:head v 53'3, mlcod=53'1)", 2024-09-11T09:52:27.027 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.960861+0000", 2024-09-11T09:52:27.027 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 64.017773439999999, 2024-09-11T09:52:27.027 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.012346107, 2024-09-11T09:52:27.027 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:27.027 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:27.027 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:27.027 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:27.028 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.028 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:27.028 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.960861+0000", 2024-09-11T09:52:27.028 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:27.028 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.028 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.028 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:27.028 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.960861+0000", 2024-09-11T09:52:27.028 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:27.028 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.028 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.028 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:27.028 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.960863+0000", 2024-09-11T09:52:27.029 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7150000000000001e-06 2024-09-11T09:52:27.029 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.029 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.029 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:27.029 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.962428+0000", 2024-09-11T09:52:27.029 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.001564668 2024-09-11T09:52:27.029 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.029 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.029 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:27.029 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.962446+0000", 2024-09-11T09:52:27.029 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.8476999999999999e-05 2024-09-11T09:52:27.029 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.029 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.030 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:27.030 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.962458+0000", 2024-09-11T09:52:27.030 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1263999999999999e-05 2024-09-11T09:52:27.030 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.030 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.030 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:27.030 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.962575+0000", 2024-09-11T09:52:27.030 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000117025 2024-09-11T09:52:27.030 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.030 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.030 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:27.030 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.962601+0000", 2024-09-11T09:52:27.030 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.5984999999999999e-05 2024-09-11T09:52:27.030 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.030 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.030 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:27.031 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.973065+0000", 2024-09-11T09:52:27.031 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.010464753 2024-09-11T09:52:27.031 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.031 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.031 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:27.031 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.973207+0000", 2024-09-11T09:52:27.031 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00014222 2024-09-11T09:52:27.031 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:27.031 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:27.031 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:27.031 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.031 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.031 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:54 2.4 2.da8979c4 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:27.031 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.963207+0000", 2024-09-11T09:52:27.032 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 64.015428001999993, 2024-09-11T09:52:27.032 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.010641575, 2024-09-11T09:52:27.032 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:27.032 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:27.032 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "reached pg", 2024-09-11T09:52:27.032 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:27.032 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:52:27.032 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:52:27.032 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 54 2024-09-11T09:52:27.032 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.032 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:27.032 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.033 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:27.033 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.963207+0000", 2024-09-11T09:52:27.033 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:27.033 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.033 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.033 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:27.033 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.963207+0000", 2024-09-11T09:52:27.033 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:27.033 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.033 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.033 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:27.033 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.963208+0000", 2024-09-11T09:52:27.033 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2979999999999999e-06 2024-09-11T09:52:27.033 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.033 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.034 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:27.034 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.967419+0000", 2024-09-11T09:52:27.034 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0042111750000000002 2024-09-11T09:52:27.034 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.034 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.034 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:27.034 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.967439+0000", 2024-09-11T09:52:27.034 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.0129999999999999e-05 2024-09-11T09:52:27.034 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.034 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.034 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:27.034 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.967453+0000", 2024-09-11T09:52:27.034 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3781000000000001e-05 2024-09-11T09:52:27.034 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.035 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.035 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:27.035 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.973840+0000", 2024-09-11T09:52:27.035 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0063872269999999997 2024-09-11T09:52:27.035 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.035 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.035 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:27.035 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.973848+0000", 2024-09-11T09:52:27.035 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.9640000000000003e-06 2024-09-11T09:52:27.035 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:27.035 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:27.035 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:27.035 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.035 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.035 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:54 2.4 2:239e915b:::rbd_data.1250f5e25b80.000000000000000c:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:27.035 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.982401+0000", 2024-09-11T09:52:27.036 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 63.996234074999997, 2024-09-11T09:52:27.036 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.016817347, 2024-09-11T09:52:27.036 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:27.036 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:27.036 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:27.036 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:27.036 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:52:27.036 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:52:27.036 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 54 2024-09-11T09:52:27.036 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.036 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:27.036 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.036 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:27.036 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.982401+0000", 2024-09-11T09:52:27.037 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:27.037 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.037 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.037 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:27.037 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.982401+0000", 2024-09-11T09:52:27.037 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:27.037 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.037 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.037 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:27.037 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.982402+0000", 2024-09-11T09:52:27.037 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.8999999999999995e-07 2024-09-11T09:52:27.037 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.038 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.038 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:27.038 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.986535+0000", 2024-09-11T09:52:27.038 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0041333689999999996 2024-09-11T09:52:27.038 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.038 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.038 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:27.038 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.986555+0000", 2024-09-11T09:52:27.038 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.9851e-05 2024-09-11T09:52:27.038 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.038 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.038 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:27.038 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.986569+0000", 2024-09-11T09:52:27.038 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3784000000000001e-05 2024-09-11T09:52:27.038 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.039 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.039 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:27.039 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.986683+0000", 2024-09-11T09:52:27.039 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000114474 2024-09-11T09:52:27.039 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.039 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.039 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:27.039 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.986815+0000", 2024-09-11T09:52:27.039 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00013217799999999999 2024-09-11T09:52:27.039 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.039 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.039 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 0", 2024-09-11T09:52:27.039 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.986918+0000", 2024-09-11T09:52:27.039 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00010235599999999999 2024-09-11T09:52:27.039 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.040 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.040 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:27.040 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.998983+0000", 2024-09-11T09:52:27.040 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.012065352 2024-09-11T09:52:27.040 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.040 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.040 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:27.040 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.999081+0000", 2024-09-11T09:52:27.040 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.8091000000000007e-05 2024-09-11T09:52:27.040 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.040 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.040 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:27.040 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.999159+0000", 2024-09-11T09:52:27.040 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.7823000000000004e-05 2024-09-11T09:52:27.041 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.041 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.041 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:27.041 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.999218+0000", 2024-09-11T09:52:27.041 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.9178999999999997e-05 2024-09-11T09:52:27.041 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:27.041 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:27.041 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:27.041 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.041 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.041 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:56 2.4 2:2167012f:::rbd_data.1250f5e25b80.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:27.041 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.986661+0000", 2024-09-11T09:52:27.041 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 63.991973622000003, 2024-09-11T09:52:27.041 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.017353539000000001, 2024-09-11T09:52:27.041 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:27.042 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:27.042 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:27.042 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:27.042 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:52:27.042 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:52:27.042 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 56 2024-09-11T09:52:27.042 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.042 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:27.042 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.042 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:27.042 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.986661+0000", 2024-09-11T09:52:27.042 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:27.042 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.042 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.042 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:27.043 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.986661+0000", 2024-09-11T09:52:27.043 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:27.043 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.043 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.043 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:27.043 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.986663+0000", 2024-09-11T09:52:27.043 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.8989999999999999e-06 2024-09-11T09:52:27.043 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.043 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.043 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:27.043 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.990444+0000", 2024-09-11T09:52:27.043 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0037808569999999999 2024-09-11T09:52:27.043 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.043 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.043 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:27.044 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.990462+0000", 2024-09-11T09:52:27.044 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.8450000000000001e-05 2024-09-11T09:52:27.044 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.044 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.044 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:27.044 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.990476+0000", 2024-09-11T09:52:27.044 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3565e-05 2024-09-11T09:52:27.044 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.044 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.044 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:27.044 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.993582+0000", 2024-09-11T09:52:27.044 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0031059109999999998 2024-09-11T09:52:27.044 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.044 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.045 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:27.045 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.993686+0000", 2024-09-11T09:52:27.045 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000104424 2024-09-11T09:52:27.045 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.045 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.045 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 0", 2024-09-11T09:52:27.045 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.993799+0000", 2024-09-11T09:52:27.045 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000112985 2024-09-11T09:52:27.045 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.045 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.045 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:27.045 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.002449+0000", 2024-09-11T09:52:27.045 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.008650151 2024-09-11T09:52:27.045 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.046 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.046 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:27.046 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.003948+0000", 2024-09-11T09:52:27.046 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.001498636 2024-09-11T09:52:27.046 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.046 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.046 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:27.046 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.003985+0000", 2024-09-11T09:52:27.046 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.7054999999999997e-05 2024-09-11T09:52:27.046 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.046 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.046 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:27.046 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.004015+0000", 2024-09-11T09:52:27.046 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.9606000000000001e-05 2024-09-11T09:52:27.046 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:27.047 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:27.047 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:27.047 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.047 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.047 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:58 2.6 2:6277c644:::rbd_data.1250f5e25b80.000000000000000e:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:27.047 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.990540+0000", 2024-09-11T09:52:27.047 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 63.988094826999998, 2024-09-11T09:52:27.047 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.016232068999999998, 2024-09-11T09:52:27.047 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:27.047 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:27.047 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:27.047 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:27.047 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:52:27.047 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:52:27.047 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 58 2024-09-11T09:52:27.047 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.048 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:27.048 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.048 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:27.048 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.990540+0000", 2024-09-11T09:52:27.048 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:27.048 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.048 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.048 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:27.048 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.990540+0000", 2024-09-11T09:52:27.048 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:27.048 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.048 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.048 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:27.048 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.990541+0000", 2024-09-11T09:52:27.049 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2359999999999999e-06 2024-09-11T09:52:27.049 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.049 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.049 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:27.049 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.994626+0000", 2024-09-11T09:52:27.049 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.004084865 2024-09-11T09:52:27.049 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.049 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.049 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:27.049 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.994644+0000", 2024-09-11T09:52:27.049 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7866000000000001e-05 2024-09-11T09:52:27.049 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.049 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.049 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:27.050 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.994653+0000", 2024-09-11T09:52:27.050 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.2159999999999995e-06 2024-09-11T09:52:27.050 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.050 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.050 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:27.050 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.994732+0000", 2024-09-11T09:52:27.050 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.8954000000000003e-05 2024-09-11T09:52:27.050 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.050 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.050 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:27.050 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.994906+0000", 2024-09-11T09:52:27.050 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0001741 2024-09-11T09:52:27.050 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.051 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.051 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 3", 2024-09-11T09:52:27.051 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.995083+0000", 2024-09-11T09:52:27.051 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00017637100000000001 2024-09-11T09:52:27.051 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.051 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.051 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:27.051 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.003066+0000", 2024-09-11T09:52:27.051 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0079833360000000006 2024-09-11T09:52:27.051 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.051 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.051 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:27.051 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.006634+0000", 2024-09-11T09:52:27.051 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.003568309 2024-09-11T09:52:27.052 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.052 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.052 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:27.052 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.006716+0000", 2024-09-11T09:52:27.052 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.1377000000000004e-05 2024-09-11T09:52:27.052 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.052 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.052 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:27.052 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.006772+0000", 2024-09-11T09:52:27.052 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.6439e-05 2024-09-11T09:52:27.052 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:27.052 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:27.053 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:27.053 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.053 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.053 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:59 2.4 2:2906e6a7:::rbd_header.1250f5e25b80:head [watch reconnect cookie 140424358411408 gen 2] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:27.053 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.994684+0000", 2024-09-11T09:52:27.053 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 63.983951196, 2024-09-11T09:52:27.053 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0094134349999999995, 2024-09-11T09:52:27.053 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:27.053 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:27.053 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "started", 2024-09-11T09:52:27.053 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:27.053 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:52:27.053 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:52:27.054 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 59 2024-09-11T09:52:27.054 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.054 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:27.054 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.054 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:27.054 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.994684+0000", 2024-09-11T09:52:27.054 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:27.054 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.054 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.054 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:27.054 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.994684+0000", 2024-09-11T09:52:27.054 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:27.054 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.055 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.055 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:27.055 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.994686+0000", 2024-09-11T09:52:27.055 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.8159999999999998e-06 2024-09-11T09:52:27.055 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.055 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.055 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:27.055 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.994691+0000", 2024-09-11T09:52:27.055 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.5009999999999998e-06 2024-09-11T09:52:27.055 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.055 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.055 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:27.055 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.994698+0000", 2024-09-11T09:52:27.056 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.3039999999999999e-06 2024-09-11T09:52:27.056 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.056 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.056 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:27.056 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.994703+0000", 2024-09-11T09:52:27.056 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.5190000000000003e-06 2024-09-11T09:52:27.056 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.056 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.056 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:27.056 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.998678+0000", 2024-09-11T09:52:27.056 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0039752340000000002 2024-09-11T09:52:27.056 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.056 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.057 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:27.057 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.998748+0000", 2024-09-11T09:52:27.057 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.0114e-05 2024-09-11T09:52:27.057 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.057 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.057 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:27.057 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.004097+0000", 2024-09-11T09:52:27.057 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0053489469999999997 2024-09-11T09:52:27.057 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:27.057 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:27.057 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:27.057 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.058 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.058 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4694.0:33 2.4 2:3bfc818c:ns::rbd_data.1256a0c02879.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:27.058 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.084537+0000", 2024-09-11T09:52:27.058 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 63.894097768000002, 2024-09-11T09:52:27.058 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.022075773999999999, 2024-09-11T09:52:27.058 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:27.058 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:27.058 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:27.058 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:27.058 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4694", 2024-09-11T09:52:27.058 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:52:27.058 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 33 2024-09-11T09:52:27.059 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.059 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:27.059 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.059 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:27.059 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.084537+0000", 2024-09-11T09:52:27.059 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:27.059 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.059 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.059 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:27.059 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.084537+0000", 2024-09-11T09:52:27.059 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:27.059 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.059 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.059 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:27.060 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.084538+0000", 2024-09-11T09:52:27.060 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.5000000000000001e-07 2024-09-11T09:52:27.060 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.060 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.060 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:27.060 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.095984+0000", 2024-09-11T09:52:27.060 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.011446543 2024-09-11T09:52:27.060 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.060 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.060 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:27.060 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.096014+0000", 2024-09-11T09:52:27.060 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.9745000000000001e-05 2024-09-11T09:52:27.060 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.061 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.061 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:27.061 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.096030+0000", 2024-09-11T09:52:27.061 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6152999999999999e-05 2024-09-11T09:52:27.061 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.061 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.061 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:27.061 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.096113+0000", 2024-09-11T09:52:27.061 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.2731999999999993e-05 2024-09-11T09:52:27.061 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.061 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.061 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:27.061 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.096273+0000", 2024-09-11T09:52:27.061 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00015973699999999999 2024-09-11T09:52:27.062 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.062 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.062 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 0", 2024-09-11T09:52:27.062 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.096394+0000", 2024-09-11T09:52:27.062 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000120883 2024-09-11T09:52:27.062 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.062 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.062 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:27.062 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.106048+0000", 2024-09-11T09:52:27.062 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0096548020000000005 2024-09-11T09:52:27.062 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.062 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.062 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:27.062 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.106531+0000", 2024-09-11T09:52:27.063 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00048264299999999999 2024-09-11T09:52:27.063 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.063 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.063 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:27.063 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.106565+0000", 2024-09-11T09:52:27.063 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.4343e-05 2024-09-11T09:52:27.063 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.063 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.063 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:27.063 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.106613+0000", 2024-09-11T09:52:27.063 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.7342999999999997e-05 2024-09-11T09:52:27.063 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:27.063 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:27.063 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:27.064 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.064 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.064 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4694.0:32 2.1 e53/15 2:8a862b48:ns::rbd_data.1256a0c02879.0000000000000000:head v 53'8, mlcod=53'6)", 2024-09-11T09:52:27.064 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.091418+0000", 2024-09-11T09:52:27.064 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 63.887216598999998, 2024-09-11T09:52:27.064 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.013027372000000001, 2024-09-11T09:52:27.064 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:27.064 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:27.064 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:27.064 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:27.064 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.064 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:27.064 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.091418+0000", 2024-09-11T09:52:27.064 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:27.065 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.065 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.065 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:27.065 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.091418+0000", 2024-09-11T09:52:27.065 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:27.065 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.065 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.065 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:27.065 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.091420+0000", 2024-09-11T09:52:27.065 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6899999999999999e-06 2024-09-11T09:52:27.065 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.065 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.065 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:27.065 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.093086+0000", 2024-09-11T09:52:27.065 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0016662510000000001 2024-09-11T09:52:27.066 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.066 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.066 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:27.066 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.093107+0000", 2024-09-11T09:52:27.066 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.0999e-05 2024-09-11T09:52:27.066 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.066 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.066 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:27.066 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.093120+0000", 2024-09-11T09:52:27.066 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2544999999999999e-05 2024-09-11T09:52:27.066 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.066 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.066 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:27.066 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.093186+0000", 2024-09-11T09:52:27.066 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.6595999999999997e-05 2024-09-11T09:52:27.067 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.067 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.067 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:27.067 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.093208+0000", 2024-09-11T09:52:27.067 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.1624000000000002e-05 2024-09-11T09:52:27.067 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.067 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.067 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:27.067 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.104392+0000", 2024-09-11T09:52:27.067 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.011184265000000001 2024-09-11T09:52:27.067 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.067 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.067 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:27.067 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.104446+0000", 2024-09-11T09:52:27.067 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.3402000000000001e-05 2024-09-11T09:52:27.068 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:27.068 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:27.068 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:27.068 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.068 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.068 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4694.0:45 2.6 2:671fb830:ns::rbd_data.1256a0c02879.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:27.068 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.109629+0000", 2024-09-11T09:52:27.068 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 63.869005758, 2024-09-11T09:52:27.068 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.036403143999999998, 2024-09-11T09:52:27.068 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:27.068 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:27.068 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:27.068 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:27.068 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4694", 2024-09-11T09:52:27.069 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:52:27.069 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 45 2024-09-11T09:52:27.069 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.069 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:27.069 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.069 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:27.069 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.109629+0000", 2024-09-11T09:52:27.069 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:27.069 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.069 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.069 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:27.069 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.109629+0000", 2024-09-11T09:52:27.069 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:27.069 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.070 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.070 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:27.070 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.109630+0000", 2024-09-11T09:52:27.070 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1200000000000001e-06 2024-09-11T09:52:27.070 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.070 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.070 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:27.070 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.121370+0000", 2024-09-11T09:52:27.070 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.011739571000000001 2024-09-11T09:52:27.070 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.070 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.070 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:27.070 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.121391+0000", 2024-09-11T09:52:27.070 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.1175e-05 2024-09-11T09:52:27.070 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.071 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.071 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:27.071 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.121404+0000", 2024-09-11T09:52:27.071 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2807e-05 2024-09-11T09:52:27.071 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.071 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.071 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:27.071 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.121498+0000", 2024-09-11T09:52:27.071 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.3770999999999994e-05 2024-09-11T09:52:27.071 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.071 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.071 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:27.071 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.121612+0000", 2024-09-11T09:52:27.072 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000114666 2024-09-11T09:52:27.072 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.072 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.072 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 3", 2024-09-11T09:52:27.072 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.121711+0000", 2024-09-11T09:52:27.072 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.8647999999999995e-05 2024-09-11T09:52:27.072 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.072 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.072 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:27.072 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.126558+0000", 2024-09-11T09:52:27.072 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0048473769999999999 2024-09-11T09:52:27.072 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.072 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.072 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:27.073 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.145912+0000", 2024-09-11T09:52:27.073 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.019354281000000001 2024-09-11T09:52:27.073 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.073 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.073 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:27.073 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.145977+0000", 2024-09-11T09:52:27.073 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.4135999999999997e-05 2024-09-11T09:52:27.073 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.073 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.073 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:27.073 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.146032+0000", 2024-09-11T09:52:27.073 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.5591999999999999e-05 2024-09-11T09:52:27.073 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:27.073 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:27.074 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:27.074 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.074 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.074 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4694.0:41 2.2 e53/15 2:5e2505c4:ns::rbd_data.1256a0c02879.0000000000000003:head v 53'4, mlcod=18'2)", 2024-09-11T09:52:27.074 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.113199+0000", 2024-09-11T09:52:27.074 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 63.865435347999998, 2024-09-11T09:52:27.074 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.011404512, 2024-09-11T09:52:27.074 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:27.074 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:27.074 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:27.074 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:27.074 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.074 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:27.074 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.113199+0000", 2024-09-11T09:52:27.074 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:27.075 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.075 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.075 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:27.075 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.113199+0000", 2024-09-11T09:52:27.075 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:27.075 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.075 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.075 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:27.075 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.113200+0000", 2024-09-11T09:52:27.075 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.2799999999999995e-07 2024-09-11T09:52:27.075 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.075 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.075 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:27.075 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.118872+0000", 2024-09-11T09:52:27.075 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0056718089999999999 2024-09-11T09:52:27.076 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.076 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.076 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:27.076 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.118889+0000", 2024-09-11T09:52:27.076 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7428999999999999e-05 2024-09-11T09:52:27.076 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.076 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.076 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:27.076 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.118900+0000", 2024-09-11T09:52:27.076 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0243e-05 2024-09-11T09:52:27.076 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.076 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.076 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:27.076 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.118959+0000", 2024-09-11T09:52:27.077 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.9364000000000003e-05 2024-09-11T09:52:27.077 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.077 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.077 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:27.077 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.118976+0000", 2024-09-11T09:52:27.077 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7124000000000001e-05 2024-09-11T09:52:27.077 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.077 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.077 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:27.077 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.124557+0000", 2024-09-11T09:52:27.077 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0055812659999999997 2024-09-11T09:52:27.077 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.077 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.077 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:27.078 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.124604+0000", 2024-09-11T09:52:27.078 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.6548999999999998e-05 2024-09-11T09:52:27.078 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:27.078 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:27.078 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:27.078 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.078 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.078 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4694.0:43 2.0 e53/15 2:16c8ae84:ns::rbd_data.1256a0c02879.0000000000000005:head v 53'2, mlcod=0'0)", 2024-09-11T09:52:27.078 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.158559+0000", 2024-09-11T09:52:27.078 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 63.820076022000002, 2024-09-11T09:52:27.078 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.016713677999999999, 2024-09-11T09:52:27.078 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:27.078 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:27.079 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:27.079 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:27.079 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.079 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:27.079 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.158559+0000", 2024-09-11T09:52:27.079 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:27.079 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.079 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.079 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:27.079 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.158559+0000", 2024-09-11T09:52:27.079 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:27.079 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.079 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.080 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:27.080 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.158560+0000", 2024-09-11T09:52:27.080 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1459999999999999e-06 2024-09-11T09:52:27.080 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.080 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.080 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:27.080 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.166465+0000", 2024-09-11T09:52:27.080 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.007904767 2024-09-11T09:52:27.080 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.080 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.080 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:27.080 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.166491+0000", 2024-09-11T09:52:27.080 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.6308e-05 2024-09-11T09:52:27.080 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.080 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.081 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:27.081 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.166502+0000", 2024-09-11T09:52:27.081 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0736e-05 2024-09-11T09:52:27.081 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.081 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.081 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:27.081 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.166564+0000", 2024-09-11T09:52:27.081 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.2578999999999998e-05 2024-09-11T09:52:27.081 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.081 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.081 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:27.081 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.166585+0000", 2024-09-11T09:52:27.081 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.0395999999999999e-05 2024-09-11T09:52:27.081 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.081 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.082 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:27.082 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.175227+0000", 2024-09-11T09:52:27.082 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0086426420000000007 2024-09-11T09:52:27.082 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.082 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.082 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:27.082 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.175272+0000", 2024-09-11T09:52:27.082 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.5104e-05 2024-09-11T09:52:27.082 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:27.082 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:27.082 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:27.082 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.082 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.083 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4694.0:59 2.2 e53/15 2:41adefc9:ns::rbd_data.1256a0c02879.000000000000000e:head v 53'5, mlcod=53'3)", 2024-09-11T09:52:27.083 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.163389+0000", 2024-09-11T09:52:27.083 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 63.815246045999999, 2024-09-11T09:52:27.083 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.013095719, 2024-09-11T09:52:27.083 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:27.083 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:27.083 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:27.083 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:27.083 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.083 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:27.083 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.163389+0000", 2024-09-11T09:52:27.083 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:27.083 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.083 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.083 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:27.084 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.163389+0000", 2024-09-11T09:52:27.084 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:27.084 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.084 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.084 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:27.084 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.163390+0000", 2024-09-11T09:52:27.084 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.336e-06 2024-09-11T09:52:27.084 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.084 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.084 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:27.084 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.170335+0000", 2024-09-11T09:52:27.084 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0069451080000000002 2024-09-11T09:52:27.084 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.084 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.085 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:27.085 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.170354+0000", 2024-09-11T09:52:27.085 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.8423e-05 2024-09-11T09:52:27.085 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.085 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.085 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:27.085 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.170366+0000", 2024-09-11T09:52:27.085 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2235999999999999e-05 2024-09-11T09:52:27.085 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.085 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.085 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:27.085 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.170439+0000", 2024-09-11T09:52:27.085 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.2989000000000002e-05 2024-09-11T09:52:27.085 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.085 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.086 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:27.086 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.170460+0000", 2024-09-11T09:52:27.086 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.1211000000000001e-05 2024-09-11T09:52:27.086 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.086 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.086 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:27.086 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.176442+0000", 2024-09-11T09:52:27.086 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0059821969999999999 2024-09-11T09:52:27.086 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.086 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.086 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:27.086 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.176484+0000", 2024-09-11T09:52:27.086 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.2218999999999997e-05 2024-09-11T09:52:27.086 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:27.086 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:27.087 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:27.087 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:27.087 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:27.087 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:52:27.088 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-11T09:52:27.089 INFO:tasks.ceph.osd.2.smithi017.stderr:2024-09-11T09:52:27.089+0000 7fd20b170640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31672) UID: 0 2024-09-11T09:52:27.189 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:52:27.190+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:52:27.236 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:52:27.236 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [], 2024-09-11T09:52:27.237 INFO:teuthology.orchestra.run.smithi017.stdout: "num_ops": 0 2024-09-11T09:52:27.237 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:52:27.245 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-11T09:52:27.290 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-09-11T09:52:27.291+0000 7f7b6c7d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31904) UID: 0 2024-09-11T09:52:27.391 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:52:27.392+0000 7f3a7b90a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 42475) UID: 0 2024-09-11T09:52:27.392 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:52:27.392 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [], 2024-09-11T09:52:27.392 INFO:teuthology.orchestra.run.smithi017.stdout: "complaint_time": 30, 2024-09-11T09:52:27.392 INFO:teuthology.orchestra.run.smithi017.stdout: "num_blocked_ops": 0 2024-09-11T09:52:27.392 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:52:27.401 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-11T09:52:27.428 INFO:tasks.workunit.client.0.smithi107.stderr:+ rbd -p rbd --namespace '' status testrbdnbd27121 2024-09-11T09:52:27.428 INFO:tasks.workunit.client.0.smithi107.stderr:+ grep 'Watchers: none' 2024-09-11T09:52:27.482 INFO:tasks.workunit.client.0.smithi107.stdout:Watchers: none 2024-09-11T09:52:27.482 INFO:tasks.workunit.client.0.smithi107.stderr:+ break 2024-09-11T09:52:27.482 INFO:tasks.workunit.client.0.smithi107.stderr:+ rbd -p rbd --namespace '' snap purge testrbdnbd27121 2024-09-11T09:52:27.492 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:52:27.492+0000 7f3a7b90a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 42475) UID: 0 2024-09-11T09:52:27.510 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:52:27.510 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 20, 2024-09-11T09:52:27.510 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 600, 2024-09-11T09:52:27.510 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [ 2024-09-11T09:52:27.510 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.510 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:31 2.1 2:8654721b:::rbd_data.1250f5e25b80.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:27.510 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.896024+0000", 2024-09-11T09:52:27.511 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 64.614063212000005, 2024-09-11T09:52:27.511 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.023148401999999998, 2024-09-11T09:52:27.511 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:27.511 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:27.511 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:27.511 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:27.511 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:52:27.511 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:52:27.511 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 31 2024-09-11T09:52:27.511 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.511 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:27.511 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.511 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:27.512 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.896024+0000", 2024-09-11T09:52:27.512 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:27.512 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.512 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.512 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:27.512 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.896024+0000", 2024-09-11T09:52:27.512 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:27.512 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.512 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.512 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:27.512 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.896025+0000", 2024-09-11T09:52:27.512 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.54e-07 2024-09-11T09:52:27.512 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.512 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.513 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:27.513 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.904287+0000", 2024-09-11T09:52:27.513 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0082624729999999993 2024-09-11T09:52:27.513 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.513 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.513 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:27.513 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.904305+0000", 2024-09-11T09:52:27.513 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7912999999999999e-05 2024-09-11T09:52:27.513 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.513 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.513 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:27.513 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.904312+0000", 2024-09-11T09:52:27.513 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.0230000000000004e-06 2024-09-11T09:52:27.514 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.514 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.514 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:27.514 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.904395+0000", 2024-09-11T09:52:27.514 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.2355999999999996e-05 2024-09-11T09:52:27.514 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.514 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.514 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:27.514 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.904499+0000", 2024-09-11T09:52:27.514 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000104672 2024-09-11T09:52:27.514 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.514 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.514 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1", 2024-09-11T09:52:27.514 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.904594+0000", 2024-09-11T09:52:27.515 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.5129000000000003e-05 2024-09-11T09:52:27.515 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.515 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.515 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:27.515 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.918183+0000", 2024-09-11T09:52:27.515 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.013588355999999999 2024-09-11T09:52:27.515 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.515 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.515 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:27.515 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919053+0000", 2024-09-11T09:52:27.515 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00087066800000000001 2024-09-11T09:52:27.515 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.515 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.515 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:27.515 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919125+0000", 2024-09-11T09:52:27.516 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.1333000000000003e-05 2024-09-11T09:52:27.516 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.516 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.516 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:27.516 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919172+0000", 2024-09-11T09:52:27.516 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.7725000000000001e-05 2024-09-11T09:52:27.516 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:27.516 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:27.516 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:27.516 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.516 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.516 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4688.0:42 2.3 e53/15 2:c54742d8:::rbd_data.1250f5e25b80.0000000000000006:head v 53'2, mlcod=0'0)", 2024-09-11T09:52:27.516 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.939272+0000", 2024-09-11T09:52:27.516 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 64.570815228000001, 2024-09-11T09:52:27.517 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.010821958, 2024-09-11T09:52:27.517 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:27.517 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:27.517 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:27.517 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:27.517 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.517 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:27.517 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.939272+0000", 2024-09-11T09:52:27.517 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:27.517 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.517 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.517 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:27.517 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.939272+0000", 2024-09-11T09:52:27.517 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:27.518 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.518 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.518 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:27.518 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.939273+0000", 2024-09-11T09:52:27.518 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.7899999999999997e-07 2024-09-11T09:52:27.518 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.518 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.518 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:27.518 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.944403+0000", 2024-09-11T09:52:27.518 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0051302129999999998 2024-09-11T09:52:27.518 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.518 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.518 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:27.518 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.944413+0000", 2024-09-11T09:52:27.519 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0182e-05 2024-09-11T09:52:27.519 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.519 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.519 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:27.519 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.944420+0000", 2024-09-11T09:52:27.519 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.7000000000000002e-06 2024-09-11T09:52:27.519 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.519 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.519 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:27.519 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.944482+0000", 2024-09-11T09:52:27.519 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.1911999999999999e-05 2024-09-11T09:52:27.519 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.519 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.519 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:27.520 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.944508+0000", 2024-09-11T09:52:27.520 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.6582000000000001e-05 2024-09-11T09:52:27.520 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.520 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.520 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:27.520 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.950050+0000", 2024-09-11T09:52:27.520 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0055414080000000003 2024-09-11T09:52:27.520 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.520 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.520 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:27.520 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.950094+0000", 2024-09-11T09:52:27.520 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.4082000000000002e-05 2024-09-11T09:52:27.520 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:27.520 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:27.521 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:27.521 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.521 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.521 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:47 2.1 2:95f53b3e:::rbd_data.1250f5e25b80.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:27.521 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.948330+0000", 2024-09-11T09:52:27.521 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 64.561756884000005, 2024-09-11T09:52:27.521 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.025156603999999999, 2024-09-11T09:52:27.521 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:27.521 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:27.521 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:27.521 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:27.521 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:52:27.521 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:52:27.521 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 47 2024-09-11T09:52:27.522 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.522 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:27.522 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.522 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:27.522 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.948330+0000", 2024-09-11T09:52:27.522 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:27.522 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.522 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.522 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:27.522 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.948330+0000", 2024-09-11T09:52:27.522 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:27.522 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.522 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.522 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:27.523 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.948333+0000", 2024-09-11T09:52:27.523 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.481e-06 2024-09-11T09:52:27.523 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.523 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.523 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:27.523 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.960263+0000", 2024-09-11T09:52:27.523 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.011930671 2024-09-11T09:52:27.523 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.523 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.523 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:27.523 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.960289+0000", 2024-09-11T09:52:27.523 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.5573000000000001e-05 2024-09-11T09:52:27.523 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.523 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.524 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:27.524 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.960304+0000", 2024-09-11T09:52:27.524 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.4647000000000001e-05 2024-09-11T09:52:27.524 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.524 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.524 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:27.524 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.960390+0000", 2024-09-11T09:52:27.524 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.6088999999999995e-05 2024-09-11T09:52:27.524 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.524 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.524 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:27.524 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.960562+0000", 2024-09-11T09:52:27.524 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00017237400000000001 2024-09-11T09:52:27.524 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.525 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.525 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1", 2024-09-11T09:52:27.525 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.960725+0000", 2024-09-11T09:52:27.525 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00016321599999999999 2024-09-11T09:52:27.525 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.525 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.525 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:27.525 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.969465+0000", 2024-09-11T09:52:27.525 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0087396899999999996 2024-09-11T09:52:27.525 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.525 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.525 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:27.526 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.973393+0000", 2024-09-11T09:52:27.526 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0039274669999999996 2024-09-11T09:52:27.526 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.526 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.526 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:27.526 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.973443+0000", 2024-09-11T09:52:27.526 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.0340000000000003e-05 2024-09-11T09:52:27.526 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.526 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.526 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:27.526 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.973487+0000", 2024-09-11T09:52:27.526 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.4056e-05 2024-09-11T09:52:27.526 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:27.526 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:27.527 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:27.527 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.527 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.527 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:61 2.1 2:9af038b1:::rbd_data.1250f5e25b80.000000000000000f:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:27.527 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.975764+0000", 2024-09-11T09:52:27.527 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 64.534323360000002, 2024-09-11T09:52:27.527 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.016158891000000002, 2024-09-11T09:52:27.527 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:27.527 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:27.527 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:27.527 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:27.527 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:52:27.527 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:52:27.527 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 61 2024-09-11T09:52:27.528 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.528 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:27.528 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.528 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:27.528 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.975764+0000", 2024-09-11T09:52:27.528 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:27.528 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.528 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.528 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:27.528 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.975764+0000", 2024-09-11T09:52:27.528 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:27.528 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.528 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.529 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:27.529 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.975766+0000", 2024-09-11T09:52:27.529 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.767e-06 2024-09-11T09:52:27.529 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.529 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.529 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:27.529 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.983204+0000", 2024-09-11T09:52:27.529 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0074388400000000004 2024-09-11T09:52:27.529 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.529 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.529 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:27.529 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.983226+0000", 2024-09-11T09:52:27.529 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.1279000000000001e-05 2024-09-11T09:52:27.530 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.530 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.530 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:27.530 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.983238+0000", 2024-09-11T09:52:27.530 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2563e-05 2024-09-11T09:52:27.530 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.530 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.530 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:27.530 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.983318+0000", 2024-09-11T09:52:27.530 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.0075999999999997e-05 2024-09-11T09:52:27.530 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.530 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.530 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:27.531 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.983429+0000", 2024-09-11T09:52:27.531 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000110618 2024-09-11T09:52:27.531 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.531 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.531 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1", 2024-09-11T09:52:27.531 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.983525+0000", 2024-09-11T09:52:27.531 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.603e-05 2024-09-11T09:52:27.531 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.531 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.531 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:27.531 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.989004+0000", 2024-09-11T09:52:27.531 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0054785429999999998 2024-09-11T09:52:27.531 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.531 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.531 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:27.532 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.991788+0000", 2024-09-11T09:52:27.532 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0027845729999999998 2024-09-11T09:52:27.532 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.532 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.532 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:27.532 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.991870+0000", 2024-09-11T09:52:27.532 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.1567e-05 2024-09-11T09:52:27.532 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.532 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.532 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:27.532 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.991923+0000", 2024-09-11T09:52:27.532 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.3035000000000003e-05 2024-09-11T09:52:27.532 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:27.533 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:27.533 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:27.533 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.533 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.533 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4694.0:32 2.1 2:8a862b48:ns::rbd_data.1256a0c02879.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:27.533 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.084089+0000", 2024-09-11T09:52:27.533 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 64.425997886000005, 2024-09-11T09:52:27.533 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.020630553999999999, 2024-09-11T09:52:27.533 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:27.533 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:27.533 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:27.533 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:27.533 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4694", 2024-09-11T09:52:27.534 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:52:27.534 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 32 2024-09-11T09:52:27.534 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.534 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:27.534 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.534 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:27.534 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.084089+0000", 2024-09-11T09:52:27.534 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:27.534 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.534 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.534 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:27.534 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.084089+0000", 2024-09-11T09:52:27.534 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:27.534 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.534 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.535 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:27.535 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.084090+0000", 2024-09-11T09:52:27.535 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.2900000000000003e-07 2024-09-11T09:52:27.535 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.535 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.535 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:27.535 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.090866+0000", 2024-09-11T09:52:27.535 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0067756609999999997 2024-09-11T09:52:27.535 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.535 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.535 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:27.535 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.090891+0000", 2024-09-11T09:52:27.535 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.5119000000000001e-05 2024-09-11T09:52:27.535 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.535 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.536 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:27.536 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.090900+0000", 2024-09-11T09:52:27.536 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.4110000000000002e-06 2024-09-11T09:52:27.536 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.536 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.536 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:27.536 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.090968+0000", 2024-09-11T09:52:27.536 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.8041999999999997e-05 2024-09-11T09:52:27.536 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.536 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.536 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:27.536 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.091115+0000", 2024-09-11T09:52:27.536 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00014633499999999999 2024-09-11T09:52:27.536 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.537 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.537 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1", 2024-09-11T09:52:27.537 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.091284+0000", 2024-09-11T09:52:27.537 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000169836 2024-09-11T09:52:27.537 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.537 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.537 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:27.537 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.100896+0000", 2024-09-11T09:52:27.537 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0096118639999999995 2024-09-11T09:52:27.537 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.537 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.537 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:27.537 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.104604+0000", 2024-09-11T09:52:27.537 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0037074619999999999 2024-09-11T09:52:27.538 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.538 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.538 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:27.538 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.104669+0000", 2024-09-11T09:52:27.538 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.4836000000000001e-05 2024-09-11T09:52:27.538 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.538 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.538 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:27.538 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.104720+0000", 2024-09-11T09:52:27.538 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.1258999999999998e-05 2024-09-11T09:52:27.538 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:27.538 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:27.538 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:27.538 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.539 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.539 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4694.0:44 2.3 e53/15 2:cd1a7d1c:ns::rbd_data.1256a0c02879.0000000000000006:head v 53'3, mlcod=16'1)", 2024-09-11T09:52:27.539 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.120387+0000", 2024-09-11T09:52:27.539 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 64.389700232999999, 2024-09-11T09:52:27.539 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.011988468, 2024-09-11T09:52:27.539 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:27.539 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:27.539 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:27.539 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:27.539 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.539 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:27.539 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.120387+0000", 2024-09-11T09:52:27.540 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:27.540 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.540 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.540 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:27.540 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.120387+0000", 2024-09-11T09:52:27.540 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:27.540 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.540 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.540 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:27.540 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.120389+0000", 2024-09-11T09:52:27.540 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.018e-06 2024-09-11T09:52:27.540 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.541 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.541 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:27.541 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.127403+0000", 2024-09-11T09:52:27.541 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0070144769999999999 2024-09-11T09:52:27.541 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.541 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.541 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:27.541 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.127414+0000", 2024-09-11T09:52:27.541 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0991e-05 2024-09-11T09:52:27.541 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.541 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.541 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:27.541 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.127423+0000", 2024-09-11T09:52:27.542 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.8580000000000006e-06 2024-09-11T09:52:27.542 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.542 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.542 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:27.542 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.127468+0000", 2024-09-11T09:52:27.542 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.4830999999999998e-05 2024-09-11T09:52:27.542 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.542 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.542 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:27.542 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.127485+0000", 2024-09-11T09:52:27.542 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6838999999999999e-05 2024-09-11T09:52:27.542 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.542 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.542 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:27.543 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.132341+0000", 2024-09-11T09:52:27.543 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0048557959999999999 2024-09-11T09:52:27.543 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.543 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.543 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:27.543 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.132375+0000", 2024-09-11T09:52:27.543 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.4657999999999999e-05 2024-09-11T09:52:27.543 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:27.543 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:27.543 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:27.543 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.543 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.543 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4694.0:46 2.3 e53/15 2:ce0e1469:ns::rbd_data.1256a0c02879.0000000000000008:head v 53'4, mlcod=16'1)", 2024-09-11T09:52:27.543 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.129008+0000", 2024-09-11T09:52:27.544 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 64.381079647000007, 2024-09-11T09:52:27.544 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0087151220000000005, 2024-09-11T09:52:27.544 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:27.544 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:27.544 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:27.544 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:27.544 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.544 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:27.544 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.129008+0000", 2024-09-11T09:52:27.544 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:27.544 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.544 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.544 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:27.544 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.129008+0000", 2024-09-11T09:52:27.544 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:27.545 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.545 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.545 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:27.545 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.129009+0000", 2024-09-11T09:52:27.545 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.367e-06 2024-09-11T09:52:27.545 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.545 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.545 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:27.545 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.132839+0000", 2024-09-11T09:52:27.545 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0038297610000000001 2024-09-11T09:52:27.545 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.545 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.545 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:27.545 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.132849+0000", 2024-09-11T09:52:27.546 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0672e-05 2024-09-11T09:52:27.546 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.546 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.546 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:27.546 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.132856+0000", 2024-09-11T09:52:27.546 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.1380000000000004e-06 2024-09-11T09:52:27.546 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.546 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.546 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:27.546 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.132891+0000", 2024-09-11T09:52:27.546 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.5712e-05 2024-09-11T09:52:27.546 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.546 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.546 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:27.546 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.132907+0000", 2024-09-11T09:52:27.546 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6166000000000001e-05 2024-09-11T09:52:27.547 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.547 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.547 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:27.547 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.137681+0000", 2024-09-11T09:52:27.547 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0047735299999999998 2024-09-11T09:52:27.547 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.547 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.547 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:27.547 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.137723+0000", 2024-09-11T09:52:27.547 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.1776000000000001e-05 2024-09-11T09:52:27.547 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:27.547 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:27.547 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:27.547 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.547 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.548 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4694.0:48 2.3 e53/15 2:d2090b70:ns::rbd_data.1256a0c02879.0000000000000009:head v 53'5, mlcod=53'2)", 2024-09-11T09:52:27.548 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.135108+0000", 2024-09-11T09:52:27.548 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 64.374978749999997, 2024-09-11T09:52:27.548 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.010710679000000001, 2024-09-11T09:52:27.548 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:27.548 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:27.548 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:27.548 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:27.548 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.548 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:27.548 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.135108+0000", 2024-09-11T09:52:27.548 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:27.548 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.549 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.549 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:27.549 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.135108+0000", 2024-09-11T09:52:27.549 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:27.549 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.549 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.549 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:27.549 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.135110+0000", 2024-09-11T09:52:27.549 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.122e-06 2024-09-11T09:52:27.549 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.549 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.549 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:27.550 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.140876+0000", 2024-09-11T09:52:27.550 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.005766162 2024-09-11T09:52:27.550 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.550 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.550 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:27.550 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.140888+0000", 2024-09-11T09:52:27.550 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.201e-05 2024-09-11T09:52:27.550 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.550 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.550 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:27.550 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.140894+0000", 2024-09-11T09:52:27.550 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.1739999999999997e-06 2024-09-11T09:52:27.550 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.550 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.551 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:27.551 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.140971+0000", 2024-09-11T09:52:27.551 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.7495000000000003e-05 2024-09-11T09:52:27.551 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.551 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.551 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:27.551 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.140995+0000", 2024-09-11T09:52:27.551 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.3204999999999999e-05 2024-09-11T09:52:27.551 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.551 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.551 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:27.551 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.145757+0000", 2024-09-11T09:52:27.551 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0047627190000000003 2024-09-11T09:52:27.552 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.552 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.552 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:27.552 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.145819+0000", 2024-09-11T09:52:27.552 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.1791999999999994e-05 2024-09-11T09:52:27.552 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:27.552 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:27.552 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:27.552 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.552 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.552 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4694.0:50 2.1 2:83078786:ns::rbd_data.1256a0c02879.000000000000000a:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:27.552 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.141467+0000", 2024-09-11T09:52:27.552 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 64.368620026000002, 2024-09-11T09:52:27.552 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.012785889, 2024-09-11T09:52:27.553 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:27.553 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:27.553 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:27.553 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:27.553 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4694", 2024-09-11T09:52:27.553 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:52:27.553 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 50 2024-09-11T09:52:27.553 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.553 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:27.553 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.553 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:27.553 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.141467+0000", 2024-09-11T09:52:27.553 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:27.554 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.554 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.554 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:27.554 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.141467+0000", 2024-09-11T09:52:27.554 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:27.554 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.554 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.554 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:27.554 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.141468+0000", 2024-09-11T09:52:27.554 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1009999999999999e-06 2024-09-11T09:52:27.554 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.554 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.554 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:27.554 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.146432+0000", 2024-09-11T09:52:27.554 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0049640539999999999 2024-09-11T09:52:27.555 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.555 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.555 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:27.555 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.146446+0000", 2024-09-11T09:52:27.555 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3254e-05 2024-09-11T09:52:27.555 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.555 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.555 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:27.555 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.146452+0000", 2024-09-11T09:52:27.555 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.207e-06 2024-09-11T09:52:27.555 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.555 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.555 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:27.555 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.146510+0000", 2024-09-11T09:52:27.556 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.8322999999999997e-05 2024-09-11T09:52:27.556 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.556 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.556 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:27.556 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.146650+0000", 2024-09-11T09:52:27.556 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00013947100000000001 2024-09-11T09:52:27.556 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.556 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.556 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1", 2024-09-11T09:52:27.556 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.146779+0000", 2024-09-11T09:52:27.556 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000129059 2024-09-11T09:52:27.556 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.556 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.556 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:27.557 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.153778+0000", 2024-09-11T09:52:27.557 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0069997130000000003 2024-09-11T09:52:27.557 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.557 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.557 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:27.557 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.154180+0000", 2024-09-11T09:52:27.557 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00040159800000000003 2024-09-11T09:52:27.557 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.557 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.557 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:27.557 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.154218+0000", 2024-09-11T09:52:27.557 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.7795e-05 2024-09-11T09:52:27.557 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.557 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.557 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:27.558 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.154253+0000", 2024-09-11T09:52:27.558 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.5314000000000001e-05 2024-09-11T09:52:27.558 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:27.558 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:27.558 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:27.558 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.558 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.558 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4694.0:54 2.3 e53/15 2:c44f0253:ns::rbd_data.1256a0c02879.000000000000000b:head v 53'6, mlcod=53'4)", 2024-09-11T09:52:27.558 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.154926+0000", 2024-09-11T09:52:27.558 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 64.355161651000003, 2024-09-11T09:52:27.558 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0089497099999999996, 2024-09-11T09:52:27.558 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:27.558 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:27.558 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:27.558 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:27.559 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.559 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:27.559 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.154926+0000", 2024-09-11T09:52:27.559 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:27.559 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.559 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.559 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:27.559 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.154926+0000", 2024-09-11T09:52:27.559 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:27.559 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.559 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.559 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:27.559 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.154927+0000", 2024-09-11T09:52:27.559 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.7000000000000003e-07 2024-09-11T09:52:27.560 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.560 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.560 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:27.560 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.158712+0000", 2024-09-11T09:52:27.560 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0037855469999999998 2024-09-11T09:52:27.560 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.560 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.560 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:27.560 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.158726+0000", 2024-09-11T09:52:27.560 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3621000000000001e-05 2024-09-11T09:52:27.560 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.560 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.560 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:27.560 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.158732+0000", 2024-09-11T09:52:27.561 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.9089999999999997e-06 2024-09-11T09:52:27.561 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.561 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.561 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:27.561 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.158781+0000", 2024-09-11T09:52:27.561 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.8900000000000003e-05 2024-09-11T09:52:27.561 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.561 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.561 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:27.561 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.158805+0000", 2024-09-11T09:52:27.561 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.44e-05 2024-09-11T09:52:27.561 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.561 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.561 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:27.562 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.163826+0000", 2024-09-11T09:52:27.562 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0050206399999999998 2024-09-11T09:52:27.562 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.562 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.562 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:27.562 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.163875+0000", 2024-09-11T09:52:27.562 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.9722999999999998e-05 2024-09-11T09:52:27.562 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:27.562 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:27.562 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:27.562 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.562 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.562 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4694.0:56 2.3 e53/15 2:c462f029:ns::rbd_data.1256a0c02879.000000000000000c:head v 53'7, mlcod=53'4)", 2024-09-11T09:52:27.562 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.159549+0000", 2024-09-11T09:52:27.563 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 64.350537944999999, 2024-09-11T09:52:27.563 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.015591666000000001, 2024-09-11T09:52:27.563 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:27.563 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:27.563 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:27.563 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:27.563 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.563 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:27.563 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.159549+0000", 2024-09-11T09:52:27.563 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:27.563 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.563 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.564 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:27.564 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.159549+0000", 2024-09-11T09:52:27.564 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:27.564 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.564 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.564 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:27.564 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.159550+0000", 2024-09-11T09:52:27.564 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.1500000000000004e-07 2024-09-11T09:52:27.564 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.564 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.564 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:27.564 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.167834+0000", 2024-09-11T09:52:27.564 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0082840220000000003 2024-09-11T09:52:27.565 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.565 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.565 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:27.565 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.167857+0000", 2024-09-11T09:52:27.565 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.3337e-05 2024-09-11T09:52:27.565 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.565 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.565 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:27.565 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.167869+0000", 2024-09-11T09:52:27.565 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1556000000000001e-05 2024-09-11T09:52:27.565 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.565 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.565 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:27.565 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.167939+0000", 2024-09-11T09:52:27.566 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.0674999999999997e-05 2024-09-11T09:52:27.566 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.566 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.566 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:27.566 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.167967+0000", 2024-09-11T09:52:27.566 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.7322999999999999e-05 2024-09-11T09:52:27.566 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.566 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.566 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:27.566 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.175079+0000", 2024-09-11T09:52:27.566 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0071126150000000001 2024-09-11T09:52:27.566 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.566 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.566 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:27.566 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.175141+0000", 2024-09-11T09:52:27.567 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.1523e-05 2024-09-11T09:52:27.567 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:27.567 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:27.567 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:27.567 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.567 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.567 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.5314.0:31 2.1 2:8654721b:::rbd_data.1250f5e25b80.0000000000000000:head [write 393216~131072] snapc 3=[3] ondisk+write+known_if_redirected+supports_pool_eio e68)", 2024-09-11T09:52:27.567 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:50.230850+0000", 2024-09-11T09:52:27.567 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 37.279237184000003, 2024-09-11T09:52:27.567 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0073772539999999998, 2024-09-11T09:52:27.567 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:27.567 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:27.567 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:27.567 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:27.567 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.5314", 2024-09-11T09:52:27.567 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3853995732", 2024-09-11T09:52:27.568 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 31 2024-09-11T09:52:27.568 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.568 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:27.568 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.568 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:27.568 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.230850+0000", 2024-09-11T09:52:27.568 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:27.568 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.568 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.568 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:27.568 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.230850+0000", 2024-09-11T09:52:27.568 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:27.568 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.568 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.568 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:27.569 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.230850+0000", 2024-09-11T09:52:27.569 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.32e-07 2024-09-11T09:52:27.569 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.569 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.569 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:27.569 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.230908+0000", 2024-09-11T09:52:27.569 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.7698000000000002e-05 2024-09-11T09:52:27.569 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.569 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.569 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:27.569 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.230915+0000", 2024-09-11T09:52:27.569 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.81e-06 2024-09-11T09:52:27.569 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.569 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.569 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:27.570 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.230920+0000", 2024-09-11T09:52:27.570 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.9660000000000004e-06 2024-09-11T09:52:27.570 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.570 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.570 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:27.570 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.235055+0000", 2024-09-11T09:52:27.570 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0041354629999999998 2024-09-11T09:52:27.570 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.570 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.570 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:27.570 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.235145+0000", 2024-09-11T09:52:27.570 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.9362000000000003e-05 2024-09-11T09:52:27.570 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.570 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.570 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1", 2024-09-11T09:52:27.571 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.235282+0000", 2024-09-11T09:52:27.571 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00013710999999999999 2024-09-11T09:52:27.571 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.571 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.571 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:27.571 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.237335+0000", 2024-09-11T09:52:27.571 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.002053165 2024-09-11T09:52:27.571 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.571 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.571 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:27.571 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.238106+0000", 2024-09-11T09:52:27.571 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000771015 2024-09-11T09:52:27.571 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.571 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.571 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:27.572 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.238165+0000", 2024-09-11T09:52:27.572 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.8588000000000002e-05 2024-09-11T09:52:27.572 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.572 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.572 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:27.572 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.238227+0000", 2024-09-11T09:52:27.572 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.2644999999999994e-05 2024-09-11T09:52:27.572 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:27.572 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:27.572 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:27.572 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.572 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.572 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.5314.0:32 2.1 2:8654721b:::rbd_data.1250f5e25b80.0000000000000000:head [write 524288~131072] snapc 3=[3] ondisk+write+known_if_redirected+supports_pool_eio e68)", 2024-09-11T09:52:27.573 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:50.230932+0000", 2024-09-11T09:52:27.573 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 37.279155559000003, 2024-09-11T09:52:27.573 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0074882129999999996, 2024-09-11T09:52:27.573 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:27.573 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:27.573 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:27.573 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:27.573 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.5314", 2024-09-11T09:52:27.573 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3853995732", 2024-09-11T09:52:27.573 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 32 2024-09-11T09:52:27.573 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.573 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:27.573 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.573 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:27.574 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.230932+0000", 2024-09-11T09:52:27.574 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:27.574 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.574 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.574 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:27.574 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.230932+0000", 2024-09-11T09:52:27.574 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:27.574 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.574 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.574 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:27.574 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.230932+0000", 2024-09-11T09:52:27.574 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.9999999999999999e-07 2024-09-11T09:52:27.574 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.575 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.575 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:27.575 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.231022+0000", 2024-09-11T09:52:27.575 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.0439999999999995e-05 2024-09-11T09:52:27.575 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.575 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.575 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:27.575 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.231030+0000", 2024-09-11T09:52:27.575 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.3350000000000001e-06 2024-09-11T09:52:27.575 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.575 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.575 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:27.575 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.231034+0000", 2024-09-11T09:52:27.575 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.9759999999999997e-06 2024-09-11T09:52:27.575 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.576 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.576 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:27.576 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.235837+0000", 2024-09-11T09:52:27.576 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0048031910000000001 2024-09-11T09:52:27.576 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.576 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.576 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:27.576 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.235904+0000", 2024-09-11T09:52:27.576 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.7181999999999995e-05 2024-09-11T09:52:27.576 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.576 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.576 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1", 2024-09-11T09:52:27.576 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.236040+0000", 2024-09-11T09:52:27.576 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00013545199999999999 2024-09-11T09:52:27.576 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.577 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.577 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:27.577 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.238242+0000", 2024-09-11T09:52:27.577 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.002202809 2024-09-11T09:52:27.577 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.577 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.577 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:27.577 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.238301+0000", 2024-09-11T09:52:27.577 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.8573999999999999e-05 2024-09-11T09:52:27.577 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.577 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.577 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:27.577 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.238378+0000", 2024-09-11T09:52:27.577 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.7323999999999994e-05 2024-09-11T09:52:27.577 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.578 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.578 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:27.578 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.238420+0000", 2024-09-11T09:52:27.578 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.163e-05 2024-09-11T09:52:27.578 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:27.578 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:27.578 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:27.578 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.578 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.578 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.5314.0:33 2.1 2:8654721b:::rbd_data.1250f5e25b80.0000000000000000:head [write 655360~131072] snapc 3=[3] ondisk+write+known_if_redirected+supports_pool_eio e68)", 2024-09-11T09:52:27.578 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:50.231041+0000", 2024-09-11T09:52:27.578 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 37.279046399999999, 2024-09-11T09:52:27.578 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0075627139999999999, 2024-09-11T09:52:27.578 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:27.579 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:27.579 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:27.579 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:27.579 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.5314", 2024-09-11T09:52:27.579 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3853995732", 2024-09-11T09:52:27.579 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 33 2024-09-11T09:52:27.579 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.579 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:27.579 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.579 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:27.579 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.231041+0000", 2024-09-11T09:52:27.579 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:27.579 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.579 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.579 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:27.580 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.231041+0000", 2024-09-11T09:52:27.580 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:27.580 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.580 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.580 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:27.580 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.231041+0000", 2024-09-11T09:52:27.580 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.9499999999999998e-07 2024-09-11T09:52:27.580 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.580 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.580 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:27.580 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.231111+0000", 2024-09-11T09:52:27.580 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.0166999999999994e-05 2024-09-11T09:52:27.580 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.580 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.580 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:27.581 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.231122+0000", 2024-09-11T09:52:27.581 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0518e-05 2024-09-11T09:52:27.581 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.581 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.581 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:27.581 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.231128+0000", 2024-09-11T09:52:27.581 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.3509999999999998e-06 2024-09-11T09:52:27.581 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.581 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.581 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:27.581 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.236622+0000", 2024-09-11T09:52:27.581 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0054934989999999998 2024-09-11T09:52:27.581 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.581 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.581 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:27.582 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.236719+0000", 2024-09-11T09:52:27.582 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.7497999999999994e-05 2024-09-11T09:52:27.582 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.582 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.582 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1", 2024-09-11T09:52:27.582 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.236854+0000", 2024-09-11T09:52:27.582 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000134535 2024-09-11T09:52:27.582 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.582 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.582 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:27.582 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.238491+0000", 2024-09-11T09:52:27.582 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0016368610000000001 2024-09-11T09:52:27.582 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.582 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.583 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:27.583 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.238504+0000", 2024-09-11T09:52:27.583 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3927999999999999e-05 2024-09-11T09:52:27.583 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.583 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.583 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:27.583 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.238561+0000", 2024-09-11T09:52:27.583 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.6029999999999997e-05 2024-09-11T09:52:27.583 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.583 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.583 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:27.583 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.238604+0000", 2024-09-11T09:52:27.583 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.3031999999999997e-05 2024-09-11T09:52:27.584 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:27.584 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:27.584 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:27.584 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.584 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.584 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.5314.0:34 2.1 2:8654721b:::rbd_data.1250f5e25b80.0000000000000000:head [write 786432~131072] snapc 3=[3] ondisk+write+known_if_redirected+supports_pool_eio e68)", 2024-09-11T09:52:27.584 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:50.231139+0000", 2024-09-11T09:52:27.584 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 37.278947797000001, 2024-09-11T09:52:27.584 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0083689690000000004, 2024-09-11T09:52:27.584 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:27.584 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:27.584 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:27.584 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:27.584 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.5314", 2024-09-11T09:52:27.585 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3853995732", 2024-09-11T09:52:27.585 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 34 2024-09-11T09:52:27.585 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.585 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:27.585 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.586 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:27.586 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.231139+0000", 2024-09-11T09:52:27.586 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:27.586 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.586 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.586 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:27.586 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.231139+0000", 2024-09-11T09:52:27.587 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:27.587 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.587 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.587 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:27.587 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.231140+0000", 2024-09-11T09:52:27.587 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.1699999999999998e-07 2024-09-11T09:52:27.587 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.587 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.587 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:27.587 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.231247+0000", 2024-09-11T09:52:27.587 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000107088 2024-09-11T09:52:27.587 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.587 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.587 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:27.588 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.231260+0000", 2024-09-11T09:52:27.588 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.289e-05 2024-09-11T09:52:27.588 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.588 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.588 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:27.588 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.231272+0000", 2024-09-11T09:52:27.588 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2371999999999999e-05 2024-09-11T09:52:27.588 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.588 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.588 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:27.588 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.237380+0000", 2024-09-11T09:52:27.588 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.006107683 2024-09-11T09:52:27.588 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.588 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.589 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:27.589 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.237434+0000", 2024-09-11T09:52:27.589 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.4143e-05 2024-09-11T09:52:27.589 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.589 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.589 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1", 2024-09-11T09:52:27.589 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.237541+0000", 2024-09-11T09:52:27.589 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00010676800000000001 2024-09-11T09:52:27.589 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.589 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.589 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:27.589 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.239390+0000", 2024-09-11T09:52:27.590 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.001848861 2024-09-11T09:52:27.590 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.590 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.590 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:27.590 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.239409+0000", 2024-09-11T09:52:27.590 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.9505e-05 2024-09-11T09:52:27.590 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.590 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.590 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:27.590 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.239463+0000", 2024-09-11T09:52:27.590 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.3838000000000002e-05 2024-09-11T09:52:27.590 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.591 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.591 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:27.591 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.239508+0000", 2024-09-11T09:52:27.591 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.5303999999999999e-05 2024-09-11T09:52:27.591 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:27.591 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:27.591 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:27.591 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.591 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.591 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.5314.0:35 2.1 2:8654721b:::rbd_data.1250f5e25b80.0000000000000000:head [write 917504~131072] snapc 3=[3] ondisk+write+known_if_redirected+supports_pool_eio e68)", 2024-09-11T09:52:27.591 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:50.231287+0000", 2024-09-11T09:52:27.591 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 37.278800463000003, 2024-09-11T09:52:27.591 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0086930730000000008, 2024-09-11T09:52:27.591 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:27.592 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:27.592 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:27.592 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:27.592 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.5314", 2024-09-11T09:52:27.592 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3853995732", 2024-09-11T09:52:27.592 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 35 2024-09-11T09:52:27.592 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.592 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:27.592 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.592 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:27.592 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.231287+0000", 2024-09-11T09:52:27.592 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:27.592 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.592 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.593 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:27.593 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.231287+0000", 2024-09-11T09:52:27.593 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:27.593 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.593 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.593 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:27.593 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.231287+0000", 2024-09-11T09:52:27.593 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.7700000000000004e-07 2024-09-11T09:52:27.593 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.593 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.593 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:27.593 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.231354+0000", 2024-09-11T09:52:27.593 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.6304999999999996e-05 2024-09-11T09:52:27.594 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.594 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.594 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:27.594 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.231367+0000", 2024-09-11T09:52:27.594 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2952999999999999e-05 2024-09-11T09:52:27.594 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.594 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.594 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:27.594 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.231374+0000", 2024-09-11T09:52:27.594 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.0469999999999996e-06 2024-09-11T09:52:27.594 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.594 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.594 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:27.594 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.238650+0000", 2024-09-11T09:52:27.595 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0072760280000000004 2024-09-11T09:52:27.595 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.595 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.595 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:27.595 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.238732+0000", 2024-09-11T09:52:27.595 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.2194000000000006e-05 2024-09-11T09:52:27.595 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.595 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.595 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1", 2024-09-11T09:52:27.595 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.238876+0000", 2024-09-11T09:52:27.595 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00014458 2024-09-11T09:52:27.595 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.595 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.595 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:27.595 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.239739+0000", 2024-09-11T09:52:27.596 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00086244499999999999 2024-09-11T09:52:27.596 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.596 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.596 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:27.596 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.239917+0000", 2024-09-11T09:52:27.596 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00017777299999999999 2024-09-11T09:52:27.596 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.596 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.596 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:27.596 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.239954+0000", 2024-09-11T09:52:27.596 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.7379999999999998e-05 2024-09-11T09:52:27.596 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.596 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.596 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:27.596 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.239980+0000", 2024-09-11T09:52:27.597 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.5791000000000001e-05 2024-09-11T09:52:27.597 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:27.597 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:27.597 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:27.597 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.597 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.597 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.5314.0:70 2.1 2:8654721b:::rbd_data.1250f5e25b80.0000000000000000:head [write 524288~131072] snapc 5=[5,3] ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-09-11T09:52:27.597 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:50.824556+0000", 2024-09-11T09:52:27.597 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 36.685531662000002, 2024-09-11T09:52:27.597 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0072794469999999997, 2024-09-11T09:52:27.597 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:27.597 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:27.597 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:27.597 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:27.598 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.5314", 2024-09-11T09:52:27.598 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3853995732", 2024-09-11T09:52:27.598 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 70 2024-09-11T09:52:27.598 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.598 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:27.598 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.598 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:27.598 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.824556+0000", 2024-09-11T09:52:27.598 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:27.598 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.598 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.598 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:27.598 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.824556+0000", 2024-09-11T09:52:27.598 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:27.598 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.599 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.599 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:27.599 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.824564+0000", 2024-09-11T09:52:27.599 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.2409999999999995e-06 2024-09-11T09:52:27.599 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.599 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.599 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:27.599 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.824613+0000", 2024-09-11T09:52:27.599 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.9579000000000001e-05 2024-09-11T09:52:27.599 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.599 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.599 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:27.599 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.824638+0000", 2024-09-11T09:52:27.599 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.4510000000000001e-05 2024-09-11T09:52:27.599 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.599 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.600 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:27.600 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.824653+0000", 2024-09-11T09:52:27.600 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.4831000000000001e-05 2024-09-11T09:52:27.600 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.600 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.600 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:27.600 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.829072+0000", 2024-09-11T09:52:27.600 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0044195700000000003 2024-09-11T09:52:27.600 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.600 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.600 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:27.600 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.829166+0000", 2024-09-11T09:52:27.600 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.3212000000000001e-05 2024-09-11T09:52:27.600 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.601 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.601 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1", 2024-09-11T09:52:27.601 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.829309+0000", 2024-09-11T09:52:27.601 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00014301900000000001 2024-09-11T09:52:27.601 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.601 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.601 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:27.601 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.831074+0000", 2024-09-11T09:52:27.601 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.001765305 2024-09-11T09:52:27.601 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.601 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.601 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:27.601 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.831751+0000", 2024-09-11T09:52:27.601 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00067694000000000001 2024-09-11T09:52:27.601 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.602 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.602 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:27.602 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.831803+0000", 2024-09-11T09:52:27.602 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.2441000000000002e-05 2024-09-11T09:52:27.602 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.602 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.602 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:27.602 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.831835+0000", 2024-09-11T09:52:27.602 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.1798999999999998e-05 2024-09-11T09:52:27.602 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:27.602 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:27.602 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:27.602 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.603 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.603 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.5314.0:71 2.1 2:8654721b:::rbd_data.1250f5e25b80.0000000000000000:head [write 655360~131072] snapc 5=[5,3] ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-09-11T09:52:27.603 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:50.824662+0000", 2024-09-11T09:52:27.603 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 36.685425545999998, 2024-09-11T09:52:27.603 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0082172219999999997, 2024-09-11T09:52:27.603 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:27.603 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:27.603 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:27.603 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:27.603 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.5314", 2024-09-11T09:52:27.603 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3853995732", 2024-09-11T09:52:27.603 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 71 2024-09-11T09:52:27.603 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.603 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:27.604 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.604 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:27.604 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.824662+0000", 2024-09-11T09:52:27.604 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:27.604 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.604 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.604 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:27.604 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.824662+0000", 2024-09-11T09:52:27.604 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:27.604 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.604 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.604 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:27.604 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.824662+0000", 2024-09-11T09:52:27.604 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.0699999999999998e-07 2024-09-11T09:52:27.604 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.605 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.605 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:27.605 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.824859+0000", 2024-09-11T09:52:27.605 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00019725799999999999 2024-09-11T09:52:27.605 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.605 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.605 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:27.605 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.824906+0000", 2024-09-11T09:52:27.605 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.7162e-05 2024-09-11T09:52:27.605 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.605 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.605 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:27.605 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.824926+0000", 2024-09-11T09:52:27.605 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.0038999999999999e-05 2024-09-11T09:52:27.605 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.606 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.606 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:27.606 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.830313+0000", 2024-09-11T09:52:27.606 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0053868409999999999 2024-09-11T09:52:27.606 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.606 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.606 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:27.606 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.830359+0000", 2024-09-11T09:52:27.606 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.5370000000000001e-05 2024-09-11T09:52:27.606 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.606 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.606 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1", 2024-09-11T09:52:27.606 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.830460+0000", 2024-09-11T09:52:27.606 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00010171299999999999 2024-09-11T09:52:27.606 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.607 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.607 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:27.607 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.831843+0000", 2024-09-11T09:52:27.607 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.001382501 2024-09-11T09:52:27.607 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.607 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.607 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:27.607 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.832769+0000", 2024-09-11T09:52:27.607 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00092638099999999999 2024-09-11T09:52:27.607 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.607 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.607 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:27.607 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.832835+0000", 2024-09-11T09:52:27.607 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.6106999999999996e-05 2024-09-11T09:52:27.607 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.608 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.608 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:27.608 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.832879+0000", 2024-09-11T09:52:27.608 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.3543e-05 2024-09-11T09:52:27.608 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:27.608 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:27.608 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:27.608 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.608 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.608 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.5314.0:72 2.1 2:8654721b:::rbd_data.1250f5e25b80.0000000000000000:head [write 786432~131072] snapc 5=[5,3] ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-09-11T09:52:27.608 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:50.824938+0000", 2024-09-11T09:52:27.608 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 36.685149244000002, 2024-09-11T09:52:27.608 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0082389799999999999, 2024-09-11T09:52:27.608 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:27.609 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:27.609 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:27.609 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:27.609 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.5314", 2024-09-11T09:52:27.609 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3853995732", 2024-09-11T09:52:27.609 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 72 2024-09-11T09:52:27.609 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.609 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:27.609 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.609 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:27.609 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.824938+0000", 2024-09-11T09:52:27.609 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:27.609 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.609 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.609 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:27.610 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.824938+0000", 2024-09-11T09:52:27.610 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:27.610 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.610 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.610 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:27.610 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.824939+0000", 2024-09-11T09:52:27.610 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.6199999999999997e-07 2024-09-11T09:52:27.610 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.610 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.610 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:27.610 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.825028+0000", 2024-09-11T09:52:27.610 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.9105999999999997e-05 2024-09-11T09:52:27.610 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.610 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.610 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:27.611 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.825042+0000", 2024-09-11T09:52:27.611 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3760999999999999e-05 2024-09-11T09:52:27.611 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.611 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.611 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:27.611 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.825061+0000", 2024-09-11T09:52:27.611 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.9088000000000001e-05 2024-09-11T09:52:27.611 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.611 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.611 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:27.611 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.831122+0000", 2024-09-11T09:52:27.611 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.006061299 2024-09-11T09:52:27.611 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.611 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.611 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:27.612 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.831195+0000", 2024-09-11T09:52:27.612 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.2957e-05 2024-09-11T09:52:27.612 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.612 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.612 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1", 2024-09-11T09:52:27.612 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.831301+0000", 2024-09-11T09:52:27.612 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000106049 2024-09-11T09:52:27.612 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.612 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.612 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:27.612 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.832893+0000", 2024-09-11T09:52:27.612 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0015923770000000001 2024-09-11T09:52:27.612 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.612 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.613 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:27.613 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.833076+0000", 2024-09-11T09:52:27.613 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000182855 2024-09-11T09:52:27.613 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.613 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.613 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:27.613 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.833137+0000", 2024-09-11T09:52:27.613 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.1193999999999997e-05 2024-09-11T09:52:27.613 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.613 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.613 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:27.613 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.833177+0000", 2024-09-11T09:52:27.613 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.9632000000000003e-05 2024-09-11T09:52:27.613 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:27.613 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:27.614 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:27.614 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.614 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.614 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.5314.0:73 2.1 2:8654721b:::rbd_data.1250f5e25b80.0000000000000000:head [write 917504~131072] snapc 5=[5,3] ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-09-11T09:52:27.614 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:50.825071+0000", 2024-09-11T09:52:27.614 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 36.685015737000001, 2024-09-11T09:52:27.614 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0084039160000000009, 2024-09-11T09:52:27.614 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:27.614 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:27.614 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:27.614 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:27.614 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.5314", 2024-09-11T09:52:27.614 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3853995732", 2024-09-11T09:52:27.614 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 73 2024-09-11T09:52:27.615 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.615 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:27.615 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.615 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:27.615 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.825071+0000", 2024-09-11T09:52:27.615 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:27.615 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.615 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.615 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:27.615 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.825071+0000", 2024-09-11T09:52:27.615 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:27.615 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.615 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.615 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:27.615 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.825072+0000", 2024-09-11T09:52:27.616 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.5499999999999998e-07 2024-09-11T09:52:27.616 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.616 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.616 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:27.616 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.825124+0000", 2024-09-11T09:52:27.616 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.2197000000000002e-05 2024-09-11T09:52:27.616 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.616 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.616 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:27.616 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.825132+0000", 2024-09-11T09:52:27.616 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.0290000000000005e-06 2024-09-11T09:52:27.616 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.616 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.616 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:27.616 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.825136+0000", 2024-09-11T09:52:27.617 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.9380000000000002e-06 2024-09-11T09:52:27.617 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.617 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.617 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:27.617 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.832001+0000", 2024-09-11T09:52:27.617 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0068650949999999999 2024-09-11T09:52:27.617 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.617 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.617 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:27.617 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.832091+0000", 2024-09-11T09:52:27.617 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.9524999999999994e-05 2024-09-11T09:52:27.617 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.617 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.617 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1", 2024-09-11T09:52:27.618 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.832234+0000", 2024-09-11T09:52:27.618 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000142783 2024-09-11T09:52:27.618 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.618 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.618 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:27.618 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.833198+0000", 2024-09-11T09:52:27.618 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00096476899999999998 2024-09-11T09:52:27.618 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.618 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.618 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:27.618 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.833386+0000", 2024-09-11T09:52:27.618 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00018772 2024-09-11T09:52:27.618 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.618 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.618 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:27.618 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.833440+0000", 2024-09-11T09:52:27.619 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.4228999999999999e-05 2024-09-11T09:52:27.619 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:27.619 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:27.619 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:27.619 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.833475+0000", 2024-09-11T09:52:27.619 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.5076000000000001e-05 2024-09-11T09:52:27.619 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:27.619 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:27.619 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:27.619 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:27.619 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:27.619 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:52:27.621 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-11T09:52:27.621 INFO:tasks.ceph.osd.2.smithi017.stderr:2024-09-11T09:52:27.621+0000 7fd20b170640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31672) UID: 0 2024-09-11T09:52:27.721 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:52:27.722+0000 7f3c96ec9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31909) UID: 0 2024-09-11T09:52:27.734 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-11T09:52:27.734 INFO:teuthology.orchestra.run.smithi096.stdout: "ops": [], 2024-09-11T09:52:27.734 INFO:teuthology.orchestra.run.smithi096.stdout: "num_ops": 0 2024-09-11T09:52:27.734 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-11T09:52:27.743 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-11T09:52:27.828 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:52:27.829+0000 7f3a7b90a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 42475) UID: 0 2024-09-11T09:52:27.888 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-11T09:52:27.888 INFO:teuthology.orchestra.run.smithi096.stdout: "ops": [], 2024-09-11T09:52:27.888 INFO:teuthology.orchestra.run.smithi096.stdout: "complaint_time": 30, 2024-09-11T09:52:27.888 INFO:teuthology.orchestra.run.smithi096.stdout: "num_blocked_ops": 0 2024-09-11T09:52:27.888 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-11T09:52:27.897 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-11T09:52:27.914 INFO:teuthology.orchestra.run.smithi017.stderr:noscrub is set 2024-09-11T09:52:27.937 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:52:27.938+0000 7f3a7b90a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 42475) UID: 0 2024-09-11T09:52:28.024 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:52:28.025+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:52:28.041 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-11T09:52:28.041 INFO:teuthology.orchestra.run.smithi096.stdout: "size": 20, 2024-09-11T09:52:28.041 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 600, 2024-09-11T09:52:28.041 INFO:teuthology.orchestra.run.smithi096.stdout: "ops": [ 2024-09-11T09:52:28.041 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.041 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:111 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:52:28.041 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.711668+0000", 2024-09-11T09:52:28.041 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 136.328793851, 2024-09-11T09:52:28.041 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0077145570000000004, 2024-09-11T09:52:28.041 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:28.041 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:28.041 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:28.042 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:52:28.042 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:52:28.042 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:52:28.042 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 111 2024-09-11T09:52:28.042 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.042 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:28.042 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.042 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:28.042 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711668+0000", 2024-09-11T09:52:28.042 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:28.042 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.043 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.043 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:28.043 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711668+0000", 2024-09-11T09:52:28.043 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:28.043 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.043 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.043 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:28.043 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711669+0000", 2024-09-11T09:52:28.043 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.99e-07 2024-09-11T09:52:28.043 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.043 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.043 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:28.043 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711673+0000", 2024-09-11T09:52:28.044 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.7440000000000001e-06 2024-09-11T09:52:28.044 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.044 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.044 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:28.044 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711686+0000", 2024-09-11T09:52:28.044 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.3317e-05 2024-09-11T09:52:28.044 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.044 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.044 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:28.044 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711697+0000", 2024-09-11T09:52:28.044 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1022000000000001e-05 2024-09-11T09:52:28.044 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.044 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.045 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:28.045 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.717065+0000", 2024-09-11T09:52:28.045 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.005367531 2024-09-11T09:52:28.045 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.045 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.045 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:28.045 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.717114+0000", 2024-09-11T09:52:28.045 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.9707999999999999e-05 2024-09-11T09:52:28.045 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.045 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.045 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:52:28.045 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.717190+0000", 2024-09-11T09:52:28.046 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.5394000000000004e-05 2024-09-11T09:52:28.046 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.046 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.046 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:28.046 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718793+0000", 2024-09-11T09:52:28.046 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001603653 2024-09-11T09:52:28.046 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.046 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.046 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:52:28.046 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719329+0000", 2024-09-11T09:52:28.046 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00053543799999999995 2024-09-11T09:52:28.046 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.046 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.047 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:28.047 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719359+0000", 2024-09-11T09:52:28.047 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.0420999999999999e-05 2024-09-11T09:52:28.047 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.047 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.047 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:28.047 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719383+0000", 2024-09-11T09:52:28.047 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.3730000000000001e-05 2024-09-11T09:52:28.047 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:28.047 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:28.047 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:28.047 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.047 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.047 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:112 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:52:28.048 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.711712+0000", 2024-09-11T09:52:28.048 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 136.32875029100001, 2024-09-11T09:52:28.048 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.008162084, 2024-09-11T09:52:28.048 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:28.048 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:28.048 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:28.048 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:52:28.048 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:52:28.048 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:52:28.048 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 112 2024-09-11T09:52:28.048 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.048 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:28.048 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.048 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:28.049 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711712+0000", 2024-09-11T09:52:28.049 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:28.049 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.049 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.049 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:28.049 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711712+0000", 2024-09-11T09:52:28.049 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:28.049 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.049 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.049 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:28.049 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711712+0000", 2024-09-11T09:52:28.050 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.6000000000000004e-07 2024-09-11T09:52:28.050 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.050 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.050 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:28.050 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711808+0000", 2024-09-11T09:52:28.050 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.5977999999999995e-05 2024-09-11T09:52:28.050 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.050 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.050 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:28.050 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711819+0000", 2024-09-11T09:52:28.050 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.0755e-05 2024-09-11T09:52:28.050 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.050 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.050 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:28.050 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711826+0000", 2024-09-11T09:52:28.051 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.7970000000000003e-06 2024-09-11T09:52:28.051 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.051 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.051 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:28.051 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.717435+0000", 2024-09-11T09:52:28.051 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0056090239999999998 2024-09-11T09:52:28.051 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.051 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.051 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:28.051 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.717496+0000", 2024-09-11T09:52:28.051 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.0543e-05 2024-09-11T09:52:28.051 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.051 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.051 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:52:28.052 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.717591+0000", 2024-09-11T09:52:28.052 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.5667000000000004e-05 2024-09-11T09:52:28.052 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.052 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.052 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:28.052 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718831+0000", 2024-09-11T09:52:28.052 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001240204 2024-09-11T09:52:28.052 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.052 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.052 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:52:28.052 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719819+0000", 2024-09-11T09:52:28.052 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00098751500000000001 2024-09-11T09:52:28.053 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.053 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.053 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:28.053 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719850+0000", 2024-09-11T09:52:28.053 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.1535999999999998e-05 2024-09-11T09:52:28.053 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.053 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.053 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:28.053 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719874+0000", 2024-09-11T09:52:28.053 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.3504999999999999e-05 2024-09-11T09:52:28.053 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:28.053 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:28.054 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:28.054 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.054 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.054 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:113 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:52:28.054 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.711842+0000", 2024-09-11T09:52:28.054 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 136.32861979200001, 2024-09-11T09:52:28.054 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0081160099999999999, 2024-09-11T09:52:28.054 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:28.054 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:28.054 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:28.054 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:52:28.054 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:52:28.055 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:52:28.055 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 113 2024-09-11T09:52:28.055 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.055 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:28.055 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.055 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:28.055 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711842+0000", 2024-09-11T09:52:28.055 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:28.055 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.055 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.055 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:28.055 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711842+0000", 2024-09-11T09:52:28.056 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:28.056 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.056 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.056 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:28.056 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711843+0000", 2024-09-11T09:52:28.056 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.4399999999999999e-07 2024-09-11T09:52:28.056 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.056 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.056 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:28.056 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711848+0000", 2024-09-11T09:52:28.056 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.4159999999999997e-06 2024-09-11T09:52:28.056 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.056 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.057 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:28.057 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711856+0000", 2024-09-11T09:52:28.057 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.7870000000000007e-06 2024-09-11T09:52:28.057 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.057 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.057 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:28.057 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711862+0000", 2024-09-11T09:52:28.057 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.3689999999999998e-06 2024-09-11T09:52:28.057 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.057 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.057 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:28.057 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.717965+0000", 2024-09-11T09:52:28.058 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0061028899999999997 2024-09-11T09:52:28.058 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.058 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.058 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:28.058 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718023+0000", 2024-09-11T09:52:28.058 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.8313000000000002e-05 2024-09-11T09:52:28.058 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.058 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.058 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:52:28.058 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718109+0000", 2024-09-11T09:52:28.058 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.6205000000000006e-05 2024-09-11T09:52:28.058 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.058 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.059 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:28.059 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719808+0000", 2024-09-11T09:52:28.059 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0016990060000000001 2024-09-11T09:52:28.059 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.059 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.059 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:52:28.059 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719883+0000", 2024-09-11T09:52:28.059 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.4624999999999998e-05 2024-09-11T09:52:28.059 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.059 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.059 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:28.059 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719910+0000", 2024-09-11T09:52:28.060 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.7512e-05 2024-09-11T09:52:28.060 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.060 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.060 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:28.060 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719958+0000", 2024-09-11T09:52:28.060 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.8143000000000003e-05 2024-09-11T09:52:28.060 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:28.060 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:28.060 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:28.060 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.060 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.060 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:52:28.060 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.711887+0000", 2024-09-11T09:52:28.061 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 136.32857517799999, 2024-09-11T09:52:28.061 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0086063589999999992, 2024-09-11T09:52:28.061 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:28.061 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:28.061 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:28.061 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:52:28.061 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:52:28.061 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:52:28.061 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 114 2024-09-11T09:52:28.061 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.061 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:28.061 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.062 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:28.062 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711887+0000", 2024-09-11T09:52:28.062 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:28.062 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.062 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.062 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:28.062 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711887+0000", 2024-09-11T09:52:28.062 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:28.062 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.063 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.063 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:28.063 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711887+0000", 2024-09-11T09:52:28.063 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.1500000000000005e-07 2024-09-11T09:52:28.063 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.063 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.063 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:28.063 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711897+0000", 2024-09-11T09:52:28.063 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.1900000000000001e-06 2024-09-11T09:52:28.063 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.063 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.063 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:28.064 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711902+0000", 2024-09-11T09:52:28.064 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.2349999999999999e-06 2024-09-11T09:52:28.064 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.064 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.064 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:28.064 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711906+0000", 2024-09-11T09:52:28.064 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.7129999999999999e-06 2024-09-11T09:52:28.064 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.064 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.064 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:28.064 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718429+0000", 2024-09-11T09:52:28.064 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0065231960000000002 2024-09-11T09:52:28.065 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.065 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.065 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:28.065 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718480+0000", 2024-09-11T09:52:28.065 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.1161e-05 2024-09-11T09:52:28.065 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.065 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.065 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:52:28.065 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718558+0000", 2024-09-11T09:52:28.065 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.7519000000000001e-05 2024-09-11T09:52:28.065 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.065 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.066 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:28.066 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720038+0000", 2024-09-11T09:52:28.066 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001480016 2024-09-11T09:52:28.066 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.066 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.066 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:52:28.066 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720417+0000", 2024-09-11T09:52:28.066 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00037943699999999999 2024-09-11T09:52:28.066 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.066 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.066 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:28.066 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720452+0000", 2024-09-11T09:52:28.067 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.4595000000000003e-05 2024-09-11T09:52:28.067 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.067 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.067 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:28.067 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720493+0000", 2024-09-11T09:52:28.067 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.1782000000000001e-05 2024-09-11T09:52:28.067 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:28.067 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:28.067 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:28.067 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.067 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.067 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:52:28.067 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.711912+0000", 2024-09-11T09:52:28.068 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 136.328550509, 2024-09-11T09:52:28.068 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0086864750000000008, 2024-09-11T09:52:28.068 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:28.068 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:28.068 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:28.068 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:52:28.068 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:52:28.068 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:52:28.068 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 115 2024-09-11T09:52:28.068 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.068 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:28.068 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.069 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:28.069 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711912+0000", 2024-09-11T09:52:28.069 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:28.069 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.069 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.069 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:28.069 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711912+0000", 2024-09-11T09:52:28.069 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:28.069 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.069 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.069 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:28.069 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711912+0000", 2024-09-11T09:52:28.069 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.3900000000000001e-07 2024-09-11T09:52:28.069 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.070 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.070 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:28.070 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711969+0000", 2024-09-11T09:52:28.070 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.7501999999999999e-05 2024-09-11T09:52:28.070 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.070 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.070 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:28.070 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711986+0000", 2024-09-11T09:52:28.070 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.6402999999999999e-05 2024-09-11T09:52:28.070 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.070 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.070 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:28.070 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711990+0000", 2024-09-11T09:52:28.070 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.929e-06 2024-09-11T09:52:28.071 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.071 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.071 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:28.071 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718861+0000", 2024-09-11T09:52:28.071 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0068710530000000002 2024-09-11T09:52:28.071 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.071 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.071 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:28.071 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718908+0000", 2024-09-11T09:52:28.071 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.7431e-05 2024-09-11T09:52:28.071 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.071 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.071 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:52:28.072 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718984+0000", 2024-09-11T09:52:28.072 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.5500999999999994e-05 2024-09-11T09:52:28.072 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.072 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.072 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:52:28.072 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720500+0000", 2024-09-11T09:52:28.072 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0015158820000000001 2024-09-11T09:52:28.072 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.072 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.072 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:28.072 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720536+0000", 2024-09-11T09:52:28.072 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.6118000000000002e-05 2024-09-11T09:52:28.072 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.073 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.073 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:28.073 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720570+0000", 2024-09-11T09:52:28.073 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.4387999999999997e-05 2024-09-11T09:52:28.073 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.073 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.073 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:28.073 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720598+0000", 2024-09-11T09:52:28.073 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.8028999999999999e-05 2024-09-11T09:52:28.073 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:28.073 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:28.073 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:28.073 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.073 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.074 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:52:28.074 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.711998+0000", 2024-09-11T09:52:28.074 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 136.32846433, 2024-09-11T09:52:28.074 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.010785813999999999, 2024-09-11T09:52:28.074 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:28.074 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:28.074 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:28.074 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:52:28.074 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:52:28.074 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:52:28.074 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 116 2024-09-11T09:52:28.074 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.074 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:28.075 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.075 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:28.075 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711998+0000", 2024-09-11T09:52:28.075 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:28.075 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.075 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.075 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:28.075 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711998+0000", 2024-09-11T09:52:28.075 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:28.075 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.075 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.075 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:28.075 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711998+0000", 2024-09-11T09:52:28.075 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.3300000000000003e-07 2024-09-11T09:52:28.076 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.076 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.076 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:28.076 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712001+0000", 2024-09-11T09:52:28.076 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.9349999999999999e-06 2024-09-11T09:52:28.076 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.076 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.076 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:28.076 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712006+0000", 2024-09-11T09:52:28.076 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.1630000000000004e-06 2024-09-11T09:52:28.076 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.076 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.076 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:28.077 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712010+0000", 2024-09-11T09:52:28.077 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.574e-06 2024-09-11T09:52:28.077 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.077 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.077 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:28.077 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719415+0000", 2024-09-11T09:52:28.077 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0074049099999999998 2024-09-11T09:52:28.077 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.077 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.077 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:28.077 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719465+0000", 2024-09-11T09:52:28.077 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.0129000000000001e-05 2024-09-11T09:52:28.077 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.077 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.078 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:52:28.078 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719545+0000", 2024-09-11T09:52:28.078 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.0253999999999994e-05 2024-09-11T09:52:28.078 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.078 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.078 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:28.078 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720636+0000", 2024-09-11T09:52:28.078 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0010909330000000001 2024-09-11T09:52:28.078 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.078 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.078 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:52:28.078 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.722707+0000", 2024-09-11T09:52:28.078 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0020713300000000001 2024-09-11T09:52:28.079 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.079 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.079 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:28.079 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.722743+0000", 2024-09-11T09:52:28.079 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.5351000000000001e-05 2024-09-11T09:52:28.079 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.079 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.079 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:28.079 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.722784+0000", 2024-09-11T09:52:28.079 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.0802e-05 2024-09-11T09:52:28.079 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:28.079 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:28.079 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:28.079 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.080 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.080 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:52:28.080 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.712014+0000", 2024-09-11T09:52:28.080 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 136.32844829000001, 2024-09-11T09:52:28.080 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.013420938, 2024-09-11T09:52:28.080 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:28.080 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:28.080 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:28.080 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:52:28.080 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:52:28.080 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:52:28.080 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 117 2024-09-11T09:52:28.080 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.080 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:28.081 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.081 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:28.081 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712014+0000", 2024-09-11T09:52:28.081 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:28.081 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.081 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.081 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:28.081 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712014+0000", 2024-09-11T09:52:28.081 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:28.081 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.081 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.081 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:28.081 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712014+0000", 2024-09-11T09:52:28.081 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.2000000000000001e-07 2024-09-11T09:52:28.082 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.082 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.082 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:28.082 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712016+0000", 2024-09-11T09:52:28.082 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.2309999999999999e-06 2024-09-11T09:52:28.082 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.082 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.082 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:28.082 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712020+0000", 2024-09-11T09:52:28.082 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.3830000000000001e-06 2024-09-11T09:52:28.082 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.082 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.082 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:28.083 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712023+0000", 2024-09-11T09:52:28.083 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.3340000000000002e-06 2024-09-11T09:52:28.083 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.083 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.083 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:28.083 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720083+0000", 2024-09-11T09:52:28.083 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0080601189999999993 2024-09-11T09:52:28.083 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.083 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.083 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:28.083 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720125+0000", 2024-09-11T09:52:28.083 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.2029e-05 2024-09-11T09:52:28.083 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.084 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.084 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:52:28.084 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720205+0000", 2024-09-11T09:52:28.084 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.0010000000000001e-05 2024-09-11T09:52:28.084 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.084 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.084 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:28.084 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720849+0000", 2024-09-11T09:52:28.084 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00064369899999999996 2024-09-11T09:52:28.084 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.084 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.084 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:52:28.084 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.725357+0000", 2024-09-11T09:52:28.084 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0045077379999999998 2024-09-11T09:52:28.085 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.085 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.085 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:28.085 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.725392+0000", 2024-09-11T09:52:28.085 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.5862999999999999e-05 2024-09-11T09:52:28.085 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.085 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.085 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:28.085 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.725435+0000", 2024-09-11T09:52:28.085 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.2311999999999997e-05 2024-09-11T09:52:28.085 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:28.085 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:28.086 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:28.086 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.086 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.086 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:159 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:52:28.086 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:12.452099+0000", 2024-09-11T09:52:28.086 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 135.58836270200001, 2024-09-11T09:52:28.086 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0069885950000000002, 2024-09-11T09:52:28.086 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:28.086 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:28.086 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:28.086 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:52:28.086 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:52:28.087 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:52:28.087 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 159 2024-09-11T09:52:28.087 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.087 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:28.087 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.087 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:28.087 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452099+0000", 2024-09-11T09:52:28.087 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:28.087 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.087 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.087 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:28.087 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452099+0000", 2024-09-11T09:52:28.087 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:28.087 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.088 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.088 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:28.088 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452100+0000", 2024-09-11T09:52:28.088 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.2800000000000003e-07 2024-09-11T09:52:28.088 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.088 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.088 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:28.088 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452102+0000", 2024-09-11T09:52:28.088 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.824e-06 2024-09-11T09:52:28.088 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.088 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.088 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:28.088 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452105+0000", 2024-09-11T09:52:28.088 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.0460000000000001e-06 2024-09-11T09:52:28.089 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.089 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.089 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:28.089 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452107+0000", 2024-09-11T09:52:28.089 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.543e-06 2024-09-11T09:52:28.089 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.089 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.089 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:28.089 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.455484+0000", 2024-09-11T09:52:28.089 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0033769759999999998 2024-09-11T09:52:28.089 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.089 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.089 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:28.090 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.455527+0000", 2024-09-11T09:52:28.090 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.2589999999999997e-05 2024-09-11T09:52:28.090 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.090 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.090 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:52:28.090 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.455594+0000", 2024-09-11T09:52:28.090 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.6775999999999999e-05 2024-09-11T09:52:28.090 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.090 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.090 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:28.090 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.456312+0000", 2024-09-11T09:52:28.090 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00071874700000000005 2024-09-11T09:52:28.090 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.090 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.090 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:52:28.091 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.459015+0000", 2024-09-11T09:52:28.091 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0027028540000000002 2024-09-11T09:52:28.091 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.091 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.091 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:28.091 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.459054+0000", 2024-09-11T09:52:28.091 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.8840000000000001e-05 2024-09-11T09:52:28.091 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.091 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.091 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:28.091 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.459088+0000", 2024-09-11T09:52:28.091 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.4071e-05 2024-09-11T09:52:28.091 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:28.091 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:28.092 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:28.092 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.092 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.092 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4688.0:38 2.5 2:abde2694:::rbd_data.1250f5e25b80.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:28.092 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:22.903926+0000", 2024-09-11T09:52:28.092 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 65.136536026000002, 2024-09-11T09:52:28.092 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.028303676, 2024-09-11T09:52:28.092 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:28.092 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:28.092 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:28.092 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:52:28.092 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4688", 2024-09-11T09:52:28.092 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:52:28.093 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 38 2024-09-11T09:52:28.093 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.093 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:28.093 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.093 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:28.093 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.903926+0000", 2024-09-11T09:52:28.093 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:28.093 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.093 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.094 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:28.094 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.903926+0000", 2024-09-11T09:52:28.094 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:28.094 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.094 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.094 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:28.094 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.903928+0000", 2024-09-11T09:52:28.094 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.4249999999999999e-06 2024-09-11T09:52:28.094 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.094 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.094 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:28.094 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.914774+0000", 2024-09-11T09:52:28.094 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.010846613 2024-09-11T09:52:28.095 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.095 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.095 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:28.095 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.914793+0000", 2024-09-11T09:52:28.095 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.8879999999999999e-05 2024-09-11T09:52:28.095 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.095 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.095 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:28.095 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.914801+0000", 2024-09-11T09:52:28.095 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.2230000000000007e-06 2024-09-11T09:52:28.095 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.095 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.096 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:28.096 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.914854+0000", 2024-09-11T09:52:28.096 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.2883999999999997e-05 2024-09-11T09:52:28.096 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.096 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.096 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:28.096 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.914951+0000", 2024-09-11T09:52:28.096 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.6731000000000006e-05 2024-09-11T09:52:28.096 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.096 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.096 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:52:28.096 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.915080+0000", 2024-09-11T09:52:28.096 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00012926999999999999 2024-09-11T09:52:28.097 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.097 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.097 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:28.097 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.931469+0000", 2024-09-11T09:52:28.097 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.016389107 2024-09-11T09:52:28.097 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.097 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.097 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:52:28.097 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.932118+0000", 2024-09-11T09:52:28.097 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00064915600000000002 2024-09-11T09:52:28.097 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.097 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.097 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:28.097 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.932177+0000", 2024-09-11T09:52:28.098 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.9032e-05 2024-09-11T09:52:28.098 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.098 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.098 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:28.098 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.932230+0000", 2024-09-11T09:52:28.098 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.2355000000000003e-05 2024-09-11T09:52:28.098 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:28.098 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:28.098 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:28.098 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.098 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.098 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_repop(client.4688.0:36 2.6 e53/15 2:61e1b442:::rbd_data.1250f5e25b80.0000000000000003:head v 53'2, mlcod=0'0)", 2024-09-11T09:52:28.098 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:22.915597+0000", 2024-09-11T09:52:28.098 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 65.124865475999997, 2024-09-11T09:52:28.099 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.020430798, 2024-09-11T09:52:28.099 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:28.099 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:28.099 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:28.099 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:28.099 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.099 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:28.099 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.915597+0000", 2024-09-11T09:52:28.099 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:28.099 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.099 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.099 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:28.099 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.915597+0000", 2024-09-11T09:52:28.100 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:28.100 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.100 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.100 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:28.100 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.915598+0000", 2024-09-11T09:52:28.100 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.7150000000000001e-06 2024-09-11T09:52:28.100 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.100 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.100 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:28.100 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921741+0000", 2024-09-11T09:52:28.100 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0061424339999999996 2024-09-11T09:52:28.100 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.100 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.100 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:28.101 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921757+0000", 2024-09-11T09:52:28.101 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.6144e-05 2024-09-11T09:52:28.101 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.101 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.101 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:28.101 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921765+0000", 2024-09-11T09:52:28.101 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.0809999999999994e-06 2024-09-11T09:52:28.101 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.101 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.101 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:28.101 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921826+0000", 2024-09-11T09:52:28.101 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.0878999999999998e-05 2024-09-11T09:52:28.101 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.102 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.102 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:28.102 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921849+0000", 2024-09-11T09:52:28.102 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.2966999999999999e-05 2024-09-11T09:52:28.102 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.102 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.102 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:28.102 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.935968+0000", 2024-09-11T09:52:28.102 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.014119191 2024-09-11T09:52:28.102 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.102 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.102 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:28.102 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.936027+0000", 2024-09-11T09:52:28.103 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.9386999999999999e-05 2024-09-11T09:52:28.103 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:28.103 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:28.103 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:28.103 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.103 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.103 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_repop(client.4688.0:39 2.6 e53/15 2:743e7616:::rbd_data.1250f5e25b80.0000000000000005:head v 53'3, mlcod=0'0)", 2024-09-11T09:52:28.103 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:22.921806+0000", 2024-09-11T09:52:28.103 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 65.118656426000001, 2024-09-11T09:52:28.103 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.015275018, 2024-09-11T09:52:28.103 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:28.103 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:28.104 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:28.104 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:28.104 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.104 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:28.104 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921806+0000", 2024-09-11T09:52:28.104 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:28.104 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.104 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.104 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:28.104 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921806+0000", 2024-09-11T09:52:28.104 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:28.104 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.104 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.105 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:28.105 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921806+0000", 2024-09-11T09:52:28.105 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.2200000000000003e-07 2024-09-11T09:52:28.105 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.105 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.105 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:28.105 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.925590+0000", 2024-09-11T09:52:28.105 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0037838949999999998 2024-09-11T09:52:28.105 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.105 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.105 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:28.105 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.925605+0000", 2024-09-11T09:52:28.105 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.4321999999999999e-05 2024-09-11T09:52:28.106 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.106 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.106 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:28.106 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.925613+0000", 2024-09-11T09:52:28.106 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.1980000000000001e-06 2024-09-11T09:52:28.106 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.106 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.106 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:28.106 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.925857+0000", 2024-09-11T09:52:28.106 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000244064 2024-09-11T09:52:28.106 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.106 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.106 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:28.107 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.925875+0000", 2024-09-11T09:52:28.107 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.838e-05 2024-09-11T09:52:28.107 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.107 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.107 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:28.107 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.937042+0000", 2024-09-11T09:52:28.107 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.011166416 2024-09-11T09:52:28.107 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.107 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.107 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:28.107 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.937081+0000", 2024-09-11T09:52:28.107 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.9020999999999998e-05 2024-09-11T09:52:28.107 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:28.108 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:28.108 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:28.108 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.108 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.108 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4688.0:49 2.5 2:b6dd9fc9:::rbd_data.1250f5e25b80.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:28.108 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:22.938548+0000", 2024-09-11T09:52:28.108 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 65.101914422999997, 2024-09-11T09:52:28.108 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.022766950000000001, 2024-09-11T09:52:28.108 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:28.108 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:28.108 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:28.108 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:52:28.109 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4688", 2024-09-11T09:52:28.109 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:52:28.109 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 49 2024-09-11T09:52:28.109 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.109 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:28.109 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.109 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:28.109 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.938548+0000", 2024-09-11T09:52:28.109 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:28.109 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.109 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.109 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:28.109 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.938548+0000", 2024-09-11T09:52:28.109 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:28.110 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.110 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.110 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:28.110 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.938550+0000", 2024-09-11T09:52:28.110 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.7710000000000001e-06 2024-09-11T09:52:28.110 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.110 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.110 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:28.110 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.944237+0000", 2024-09-11T09:52:28.110 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0056879790000000001 2024-09-11T09:52:28.110 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.110 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.110 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:28.110 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.944262+0000", 2024-09-11T09:52:28.111 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.4247e-05 2024-09-11T09:52:28.111 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.111 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.111 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:28.111 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.944274+0000", 2024-09-11T09:52:28.111 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.2639999999999999e-05 2024-09-11T09:52:28.111 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.111 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.111 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:28.111 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.944344+0000", 2024-09-11T09:52:28.111 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.9574999999999997e-05 2024-09-11T09:52:28.111 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.111 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.111 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:28.112 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.944474+0000", 2024-09-11T09:52:28.112 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000130464 2024-09-11T09:52:28.112 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.112 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.112 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:52:28.112 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.944585+0000", 2024-09-11T09:52:28.112 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000110096 2024-09-11T09:52:28.112 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.112 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.112 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:52:28.112 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.952724+0000", 2024-09-11T09:52:28.112 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0081398900000000003 2024-09-11T09:52:28.112 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.112 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.113 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:28.113 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.961219+0000", 2024-09-11T09:52:28.113 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0084949789999999997 2024-09-11T09:52:28.113 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.113 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.113 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:28.113 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.961280+0000", 2024-09-11T09:52:28.113 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.0404e-05 2024-09-11T09:52:28.113 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.113 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.113 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:28.113 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.961315+0000", 2024-09-11T09:52:28.113 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.4904999999999999e-05 2024-09-11T09:52:28.114 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:28.114 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:28.114 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:28.114 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.114 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.114 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_repop(client.4688.0:44 2.6 e53/15 2:67246a22:::rbd_data.1250f5e25b80.0000000000000007:head v 53'4, mlcod=53'2)", 2024-09-11T09:52:28.114 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:22.951721+0000", 2024-09-11T09:52:28.114 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 65.088741150999994, 2024-09-11T09:52:28.114 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.01080981, 2024-09-11T09:52:28.114 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:28.114 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:28.114 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:28.114 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:28.114 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.115 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:28.115 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.951721+0000", 2024-09-11T09:52:28.115 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:28.115 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.115 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.115 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:28.115 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.951721+0000", 2024-09-11T09:52:28.115 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:28.115 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.115 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.115 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:28.115 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.951722+0000", 2024-09-11T09:52:28.116 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.0279999999999999e-06 2024-09-11T09:52:28.116 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.116 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.116 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:28.116 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.955612+0000", 2024-09-11T09:52:28.116 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0038896920000000001 2024-09-11T09:52:28.116 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.116 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.116 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:28.116 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.955628+0000", 2024-09-11T09:52:28.116 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.5945000000000001e-05 2024-09-11T09:52:28.116 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.116 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.116 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:28.117 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.955635+0000", 2024-09-11T09:52:28.117 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.8080000000000005e-06 2024-09-11T09:52:28.117 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.117 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.117 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:28.117 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.955700+0000", 2024-09-11T09:52:28.117 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.4114000000000003e-05 2024-09-11T09:52:28.117 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.117 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.117 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:28.117 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.955724+0000", 2024-09-11T09:52:28.117 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.3989999999999999e-05 2024-09-11T09:52:28.117 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.117 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.118 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:28.118 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.962490+0000", 2024-09-11T09:52:28.118 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0067662599999999996 2024-09-11T09:52:28.118 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.118 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.118 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:28.118 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.962531+0000", 2024-09-11T09:52:28.118 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.0973000000000002e-05 2024-09-11T09:52:28.118 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:28.118 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:28.118 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:28.118 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.118 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.119 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_repop(client.4688.0:51 2.6 e53/15 2:6b5894fc:::rbd_data.1250f5e25b80.000000000000000a:head v 53'5, mlcod=53'2)", 2024-09-11T09:52:28.119 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:22.959030+0000", 2024-09-11T09:52:28.119 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 65.081431976999994, 2024-09-11T09:52:28.119 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.011503638, 2024-09-11T09:52:28.119 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:28.119 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:28.119 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:28.119 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:28.119 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.119 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:28.119 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.959030+0000", 2024-09-11T09:52:28.119 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:28.119 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.120 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.120 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:28.120 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.959030+0000", 2024-09-11T09:52:28.120 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:28.120 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.120 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.120 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:28.120 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.959032+0000", 2024-09-11T09:52:28.120 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.356e-06 2024-09-11T09:52:28.120 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.120 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.120 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:28.120 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.962899+0000", 2024-09-11T09:52:28.121 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.003867639 2024-09-11T09:52:28.121 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.121 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.121 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:28.121 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.962909+0000", 2024-09-11T09:52:28.121 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.0105999999999999e-05 2024-09-11T09:52:28.121 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.121 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.121 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:28.121 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.962915+0000", 2024-09-11T09:52:28.121 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.8869999999999997e-06 2024-09-11T09:52:28.121 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.121 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.121 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:28.122 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.962962+0000", 2024-09-11T09:52:28.122 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.6511999999999998e-05 2024-09-11T09:52:28.122 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.122 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.122 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:28.122 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.962989+0000", 2024-09-11T09:52:28.122 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.7702999999999999e-05 2024-09-11T09:52:28.122 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.122 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.122 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:28.122 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.970495+0000", 2024-09-11T09:52:28.122 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0075052449999999998 2024-09-11T09:52:28.122 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.122 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.123 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:28.123 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.970534+0000", 2024-09-11T09:52:28.123 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.9190000000000003e-05 2024-09-11T09:52:28.123 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:28.123 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:28.123 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:28.123 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.123 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.123 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_repop(client.4688.0:58 2.6 e53/15 2:6277c644:::rbd_data.1250f5e25b80.000000000000000e:head v 53'6, mlcod=53'4)", 2024-09-11T09:52:28.123 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:22.996199+0000", 2024-09-11T09:52:28.123 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 65.044263056999995, 2024-09-11T09:52:28.123 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.010145573999999999, 2024-09-11T09:52:28.123 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:28.123 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:28.124 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:28.124 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:28.124 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.124 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:28.124 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.996199+0000", 2024-09-11T09:52:28.124 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:28.124 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.124 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.124 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:28.124 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.996199+0000", 2024-09-11T09:52:28.124 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:28.124 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.124 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.124 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:28.125 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.996200+0000", 2024-09-11T09:52:28.125 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.85e-07 2024-09-11T09:52:28.125 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.125 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.125 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:28.125 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.999914+0000", 2024-09-11T09:52:28.125 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.003714406 2024-09-11T09:52:28.125 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.125 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.125 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:28.125 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.999928+0000", 2024-09-11T09:52:28.125 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.3149000000000001e-05 2024-09-11T09:52:28.125 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.125 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.125 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:28.126 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.999951+0000", 2024-09-11T09:52:28.126 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.3116e-05 2024-09-11T09:52:28.126 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.126 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.126 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:28.126 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.000003+0000", 2024-09-11T09:52:28.126 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.2169000000000001e-05 2024-09-11T09:52:28.126 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.126 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.126 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:28.126 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.000018+0000", 2024-09-11T09:52:28.126 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.5500000000000001e-05 2024-09-11T09:52:28.126 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.126 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.126 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:28.127 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.006303+0000", 2024-09-11T09:52:28.127 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0062846710000000004 2024-09-11T09:52:28.127 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.127 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.127 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:28.127 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.006345+0000", 2024-09-11T09:52:28.127 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.1678000000000003e-05 2024-09-11T09:52:28.127 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:28.127 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:28.127 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:28.127 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.127 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.127 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4694.0:42 2.5 2:a4fc00ec:ns::rbd_data.1256a0c02879.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:28.127 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.111570+0000", 2024-09-11T09:52:28.128 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 64.928891950999997, 2024-09-11T09:52:28.128 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.038002197000000001, 2024-09-11T09:52:28.128 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:28.128 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:28.128 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:28.128 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:52:28.128 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4694", 2024-09-11T09:52:28.128 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:52:28.128 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 42 2024-09-11T09:52:28.128 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.128 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:28.128 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.128 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:28.128 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.111570+0000", 2024-09-11T09:52:28.129 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:28.129 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.129 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.129 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:28.129 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.111570+0000", 2024-09-11T09:52:28.129 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:28.129 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.129 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.129 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:28.129 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.111572+0000", 2024-09-11T09:52:28.129 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.3909999999999999e-06 2024-09-11T09:52:28.129 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.129 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.129 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:28.129 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.135929+0000", 2024-09-11T09:52:28.130 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.024357410999999999 2024-09-11T09:52:28.130 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.130 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.130 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:28.130 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.135951+0000", 2024-09-11T09:52:28.130 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.1775000000000001e-05 2024-09-11T09:52:28.130 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.130 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.130 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:28.130 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.135962+0000", 2024-09-11T09:52:28.130 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.0903e-05 2024-09-11T09:52:28.130 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.130 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.130 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:28.130 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.136045+0000", 2024-09-11T09:52:28.131 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.3128999999999996e-05 2024-09-11T09:52:28.131 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.131 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.131 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:28.131 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.136229+0000", 2024-09-11T09:52:28.131 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00018393700000000001 2024-09-11T09:52:28.131 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.131 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.131 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:52:28.131 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.136394+0000", 2024-09-11T09:52:28.131 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000165527 2024-09-11T09:52:28.131 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.131 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.131 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:28.131 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.149320+0000", 2024-09-11T09:52:28.132 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.01292568 2024-09-11T09:52:28.132 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.132 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.132 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:52:28.132 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.149493+0000", 2024-09-11T09:52:28.132 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00017285999999999999 2024-09-11T09:52:28.132 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.132 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.132 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:28.132 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.149539+0000", 2024-09-11T09:52:28.132 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.6628000000000001e-05 2024-09-11T09:52:28.132 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.132 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.132 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:28.133 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.149572+0000", 2024-09-11T09:52:28.133 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.2956000000000001e-05 2024-09-11T09:52:28.133 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:28.133 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:28.133 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:28.133 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.133 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.133 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_repop(client.4694.0:45 2.6 e53/15 2:671fb830:ns::rbd_data.1256a0c02879.0000000000000007:head v 53'9, mlcod=53'7)", 2024-09-11T09:52:28.133 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.121833+0000", 2024-09-11T09:52:28.133 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 64.918629593000006, 2024-09-11T09:52:28.133 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.023876192000000001, 2024-09-11T09:52:28.133 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:28.133 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:28.134 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:28.134 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:28.134 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.134 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:28.134 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.121833+0000", 2024-09-11T09:52:28.134 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:28.134 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.134 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.134 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:28.134 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.121833+0000", 2024-09-11T09:52:28.134 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:28.134 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.134 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.134 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:28.135 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.121834+0000", 2024-09-11T09:52:28.135 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.471e-06 2024-09-11T09:52:28.135 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.135 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.135 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:28.135 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.128990+0000", 2024-09-11T09:52:28.135 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0071557879999999997 2024-09-11T09:52:28.135 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.135 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.135 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:28.135 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.129011+0000", 2024-09-11T09:52:28.135 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.0760000000000001e-05 2024-09-11T09:52:28.135 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.135 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.136 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:28.136 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.129022+0000", 2024-09-11T09:52:28.136 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1056000000000001e-05 2024-09-11T09:52:28.136 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.136 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.136 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:28.136 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.129097+0000", 2024-09-11T09:52:28.136 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.5082999999999999e-05 2024-09-11T09:52:28.136 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.136 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.136 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:28.136 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.129130+0000", 2024-09-11T09:52:28.136 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.3345e-05 2024-09-11T09:52:28.137 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.137 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.137 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:28.137 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.145657+0000", 2024-09-11T09:52:28.137 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.016527224 2024-09-11T09:52:28.137 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.137 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.137 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:28.137 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.145709+0000", 2024-09-11T09:52:28.137 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.1465000000000003e-05 2024-09-11T09:52:28.137 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:28.137 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:28.138 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:28.138 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.138 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.138 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4694.0:43 2.0 2:16c8ae84:ns::rbd_data.1256a0c02879.0000000000000005:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:28.138 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.135999+0000", 2024-09-11T09:52:28.138 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 64.904463643, 2024-09-11T09:52:28.138 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.039468309, 2024-09-11T09:52:28.138 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:28.138 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:28.138 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:28.138 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:52:28.138 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4694", 2024-09-11T09:52:28.138 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:52:28.139 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 43 2024-09-11T09:52:28.139 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.139 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:28.139 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.139 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:28.139 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.135999+0000", 2024-09-11T09:52:28.139 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:28.139 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.139 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.139 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:28.139 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.135999+0000", 2024-09-11T09:52:28.139 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:28.139 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.139 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.139 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:28.140 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.136000+0000", 2024-09-11T09:52:28.140 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.6330000000000001e-06 2024-09-11T09:52:28.140 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.140 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.140 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:28.140 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.157961+0000", 2024-09-11T09:52:28.140 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0219606 2024-09-11T09:52:28.140 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.140 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.140 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:28.140 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.157995+0000", 2024-09-11T09:52:28.140 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.4391999999999999e-05 2024-09-11T09:52:28.140 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.141 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.141 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:28.141 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.158011+0000", 2024-09-11T09:52:28.141 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.5794000000000002e-05 2024-09-11T09:52:28.141 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.141 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.141 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:28.141 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.158083+0000", 2024-09-11T09:52:28.141 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.2273000000000004e-05 2024-09-11T09:52:28.141 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.141 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.141 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:28.141 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.158239+0000", 2024-09-11T09:52:28.142 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000155372 2024-09-11T09:52:28.142 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.142 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.142 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 1", 2024-09-11T09:52:28.142 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.158403+0000", 2024-09-11T09:52:28.142 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00016408199999999999 2024-09-11T09:52:28.142 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.142 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.142 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:52:28.142 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.173860+0000", 2024-09-11T09:52:28.142 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.015457347 2024-09-11T09:52:28.142 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.142 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.143 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:28.143 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.175400+0000", 2024-09-11T09:52:28.143 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001540006 2024-09-11T09:52:28.143 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.143 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.143 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:28.143 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.175438+0000", 2024-09-11T09:52:28.143 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.8135e-05 2024-09-11T09:52:28.143 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.143 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.143 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:28.143 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.175467+0000", 2024-09-11T09:52:28.143 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.8674999999999999e-05 2024-09-11T09:52:28.144 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:28.144 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:28.144 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:28.144 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.144 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.144 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4694.0:57 2.0 2:04b5549c:ns::rbd_data.1256a0c02879.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:28.144 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.158037+0000", 2024-09-11T09:52:28.144 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 64.882424904999993, 2024-09-11T09:52:28.144 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.027748790999999998, 2024-09-11T09:52:28.144 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:28.144 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:28.144 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:28.145 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:52:28.145 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4694", 2024-09-11T09:52:28.145 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:52:28.145 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 57 2024-09-11T09:52:28.145 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.145 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:28.145 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.145 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:28.145 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.158037+0000", 2024-09-11T09:52:28.145 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:28.145 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.145 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.145 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:28.146 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.158037+0000", 2024-09-11T09:52:28.146 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:28.146 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.146 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.146 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:28.146 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.158039+0000", 2024-09-11T09:52:28.146 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.356e-06 2024-09-11T09:52:28.146 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.146 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.146 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:28.146 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.176367+0000", 2024-09-11T09:52:28.146 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.018328739 2024-09-11T09:52:28.146 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.146 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.146 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:28.147 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.176378+0000", 2024-09-11T09:52:28.147 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.0713999999999999e-05 2024-09-11T09:52:28.147 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.147 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.147 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:28.147 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.176383+0000", 2024-09-11T09:52:28.147 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.0209999999999999e-06 2024-09-11T09:52:28.147 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.147 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.147 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:28.147 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.176426+0000", 2024-09-11T09:52:28.147 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.2623999999999997e-05 2024-09-11T09:52:28.147 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.147 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.148 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:28.148 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.176552+0000", 2024-09-11T09:52:28.148 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000126076 2024-09-11T09:52:28.148 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.148 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.148 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 1", 2024-09-11T09:52:28.148 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.176665+0000", 2024-09-11T09:52:28.148 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000112995 2024-09-11T09:52:28.148 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.148 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.148 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:52:28.148 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.183639+0000", 2024-09-11T09:52:28.148 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0069740039999999998 2024-09-11T09:52:28.149 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.149 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.149 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:28.149 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.185689+0000", 2024-09-11T09:52:28.149 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.002049852 2024-09-11T09:52:28.149 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.149 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.149 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:28.149 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.185725+0000", 2024-09-11T09:52:28.149 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.6804999999999997e-05 2024-09-11T09:52:28.149 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.149 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.149 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:28.149 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.185786+0000", 2024-09-11T09:52:28.150 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.0605e-05 2024-09-11T09:52:28.150 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:28.150 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:28.150 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:28.150 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.150 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.150 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4694.0:60 2.0 2:054f2af4:ns::rbd_data.1256a0c02879.000000000000000f:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:28.150 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.186855+0000", 2024-09-11T09:52:28.150 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 64.853606889000005, 2024-09-11T09:52:28.150 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.013682913, 2024-09-11T09:52:28.150 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:28.150 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:28.150 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:28.150 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:52:28.151 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4694", 2024-09-11T09:52:28.151 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:52:28.151 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 60 2024-09-11T09:52:28.151 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.151 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:28.151 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.151 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:28.151 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.186855+0000", 2024-09-11T09:52:28.151 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:28.151 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.151 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.151 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:28.151 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.186855+0000", 2024-09-11T09:52:28.152 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:28.152 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.152 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.152 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:28.152 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.186856+0000", 2024-09-11T09:52:28.152 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.1200000000000001e-07 2024-09-11T09:52:28.152 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.152 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.152 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:28.152 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.191484+0000", 2024-09-11T09:52:28.152 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0046274159999999996 2024-09-11T09:52:28.152 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.152 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.153 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:28.153 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.191496+0000", 2024-09-11T09:52:28.153 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.2149e-05 2024-09-11T09:52:28.153 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.153 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.153 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:28.153 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.191502+0000", 2024-09-11T09:52:28.153 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.1890000000000004e-06 2024-09-11T09:52:28.153 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.153 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.153 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:28.153 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.191544+0000", 2024-09-11T09:52:28.153 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.1724999999999998e-05 2024-09-11T09:52:28.154 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.154 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.154 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:28.154 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.191641+0000", 2024-09-11T09:52:28.154 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.6939000000000001e-05 2024-09-11T09:52:28.154 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.154 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.154 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 1", 2024-09-11T09:52:28.154 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.191731+0000", 2024-09-11T09:52:28.154 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.0651999999999999e-05 2024-09-11T09:52:28.154 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.154 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.154 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:52:28.154 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.199852+0000", 2024-09-11T09:52:28.155 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0081204529999999997 2024-09-11T09:52:28.155 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.155 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.155 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:28.155 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.200461+0000", 2024-09-11T09:52:28.155 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00060900899999999998 2024-09-11T09:52:28.155 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.155 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.155 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:28.155 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.200497+0000", 2024-09-11T09:52:28.155 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.5800000000000003e-05 2024-09-11T09:52:28.155 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.155 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.156 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:28.156 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.200538+0000", 2024-09-11T09:52:28.156 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.1668999999999997e-05 2024-09-11T09:52:28.156 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:28.156 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:28.156 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:28.156 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:28.156 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:28.156 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-11T09:52:28.158 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-11T09:52:28.158 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:52:28.158+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:52:28.258 INFO:tasks.ceph.osd.0.smithi017.stderr:2024-09-11T09:52:28.260+0000 7fb73c0f1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31665) UID: 0 2024-09-11T09:52:28.260 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-11T09:52:28.261 INFO:teuthology.orchestra.run.smithi096.stdout: "ops": [], 2024-09-11T09:52:28.261 INFO:teuthology.orchestra.run.smithi096.stdout: "num_ops": 0 2024-09-11T09:52:28.261 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-11T09:52:28.269 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-11T09:52:28.359 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:52:28.360+0000 7f3a7b90a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 42475) UID: 0 2024-09-11T09:52:28.371 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-11T09:52:28.371 INFO:teuthology.orchestra.run.smithi096.stdout: "ops": [], 2024-09-11T09:52:28.372 INFO:teuthology.orchestra.run.smithi096.stdout: "complaint_time": 30, 2024-09-11T09:52:28.372 INFO:teuthology.orchestra.run.smithi096.stdout: "num_blocked_ops": 0 2024-09-11T09:52:28.372 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-11T09:52:28.380 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-11T09:52:28.406 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T09:52:28.461 INFO:tasks.ceph.osd.2.smithi017.stderr:2024-09-11T09:52:28.462+0000 7fd20b170640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31672) UID: 0 2024-09-11T09:52:28.526 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-11T09:52:28.526 INFO:teuthology.orchestra.run.smithi096.stdout: "size": 20, 2024-09-11T09:52:28.526 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 600, 2024-09-11T09:52:28.526 INFO:teuthology.orchestra.run.smithi096.stdout: "ops": [ 2024-09-11T09:52:28.526 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.527 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4230.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-11T09:52:28.527 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:18.490736+0000", 2024-09-11T09:52:28.527 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 130.03530510900001, 2024-09-11T09:52:28.527 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0059097569999999999, 2024-09-11T09:52:28.527 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:28.527 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:28.527 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:28.527 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:52:28.527 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4230", 2024-09-11T09:52:28.527 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.17:0/4101109576", 2024-09-11T09:52:28.527 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 1 2024-09-11T09:52:28.527 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.528 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:28.528 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.528 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:28.528 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.490736+0000", 2024-09-11T09:52:28.528 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:28.528 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.528 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.528 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:28.528 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.490736+0000", 2024-09-11T09:52:28.528 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:28.528 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.528 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.529 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:28.529 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.490742+0000", 2024-09-11T09:52:28.529 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.48e-06 2024-09-11T09:52:28.529 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.529 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.529 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:28.529 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.490749+0000", 2024-09-11T09:52:28.529 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.4569999999999999e-06 2024-09-11T09:52:28.529 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.529 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.529 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:28.529 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.490762+0000", 2024-09-11T09:52:28.530 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.2721000000000001e-05 2024-09-11T09:52:28.530 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.530 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.530 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:28.530 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.490774+0000", 2024-09-11T09:52:28.530 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1823e-05 2024-09-11T09:52:28.530 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.530 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.530 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:28.530 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.490846+0000", 2024-09-11T09:52:28.530 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.1878000000000005e-05 2024-09-11T09:52:28.530 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.531 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.531 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for peered", 2024-09-11T09:52:28.531 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.490857+0000", 2024-09-11T09:52:28.531 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1017e-05 2024-09-11T09:52:28.531 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.531 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.531 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:28.531 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.493856+0000", 2024-09-11T09:52:28.531 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00299885 2024-09-11T09:52:28.531 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.531 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.531 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:28.532 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.494020+0000", 2024-09-11T09:52:28.532 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00016436599999999999 2024-09-11T09:52:28.532 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.532 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.532 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 2", 2024-09-11T09:52:28.532 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.494186+0000", 2024-09-11T09:52:28.532 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00016562799999999999 2024-09-11T09:52:28.532 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.532 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.532 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:28.532 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.495407+0000", 2024-09-11T09:52:28.533 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0012217420000000001 2024-09-11T09:52:28.533 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.533 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.533 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:52:28.533 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.496579+0000", 2024-09-11T09:52:28.533 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001171844 2024-09-11T09:52:28.533 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.533 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.533 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:28.533 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.496620+0000", 2024-09-11T09:52:28.533 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.0479000000000003e-05 2024-09-11T09:52:28.533 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.534 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.534 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:28.534 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.496646+0000", 2024-09-11T09:52:28.534 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.6472e-05 2024-09-11T09:52:28.534 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:28.534 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:28.534 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:28.534 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.534 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.534 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4230.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-09-11T09:52:28.534 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:20.498634+0000", 2024-09-11T09:52:28.534 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 128.02740708799999, 2024-09-11T09:52:28.535 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0050648539999999997, 2024-09-11T09:52:28.535 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:28.535 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:28.535 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:28.535 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:52:28.535 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4230", 2024-09-11T09:52:28.535 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.17:0/4101109576", 2024-09-11T09:52:28.535 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 7 2024-09-11T09:52:28.535 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.535 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:28.535 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.535 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:28.535 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.498634+0000", 2024-09-11T09:52:28.536 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:28.536 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.536 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.536 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:28.536 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.498634+0000", 2024-09-11T09:52:28.536 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:28.536 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.536 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.536 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:28.536 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.498637+0000", 2024-09-11T09:52:28.536 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.1919999999999999e-06 2024-09-11T09:52:28.536 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.536 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.536 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:28.537 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.498648+0000", 2024-09-11T09:52:28.537 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1116e-05 2024-09-11T09:52:28.537 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.537 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.537 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:28.537 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.498666+0000", 2024-09-11T09:52:28.537 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.8411999999999999e-05 2024-09-11T09:52:28.537 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.537 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.537 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:28.537 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.498675+0000", 2024-09-11T09:52:28.537 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.4059999999999994e-06 2024-09-11T09:52:28.537 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.538 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.538 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:28.538 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.498745+0000", 2024-09-11T09:52:28.538 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.9764000000000005e-05 2024-09-11T09:52:28.538 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.538 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.538 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op must wait for map", 2024-09-11T09:52:28.538 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.498754+0000", 2024-09-11T09:52:28.538 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.5969999999999992e-06 2024-09-11T09:52:28.538 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.538 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.538 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:28.538 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.500385+0000", 2024-09-11T09:52:28.539 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0016308360000000001 2024-09-11T09:52:28.539 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.539 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.539 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:28.539 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.500473+0000", 2024-09-11T09:52:28.539 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.7966000000000004e-05 2024-09-11T09:52:28.539 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.539 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.539 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 1", 2024-09-11T09:52:28.539 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.500591+0000", 2024-09-11T09:52:28.539 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000117895 2024-09-11T09:52:28.539 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.539 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.539 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:28.539 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.501685+0000", 2024-09-11T09:52:28.540 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001094886 2024-09-11T09:52:28.540 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.540 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.540 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:52:28.540 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.503627+0000", 2024-09-11T09:52:28.540 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001941199 2024-09-11T09:52:28.540 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.540 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.540 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:28.540 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.503667+0000", 2024-09-11T09:52:28.540 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.0645000000000001e-05 2024-09-11T09:52:28.540 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.540 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.540 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:28.541 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.503699+0000", 2024-09-11T09:52:28.541 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.1940000000000003e-05 2024-09-11T09:52:28.541 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:28.541 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:28.541 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:28.541 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.541 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.541 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4688.0:32 2.2 2:45941fd2:::rbd_data.1250f5e25b80.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:28.541 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:22.896607+0000", 2024-09-11T09:52:28.541 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 65.629434485000004, 2024-09-11T09:52:28.541 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.038655179999999997, 2024-09-11T09:52:28.541 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:28.541 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:28.542 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:28.542 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:52:28.542 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4688", 2024-09-11T09:52:28.542 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:52:28.542 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 32 2024-09-11T09:52:28.542 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.542 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:28.542 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.542 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:28.542 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.896607+0000", 2024-09-11T09:52:28.542 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:28.542 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.542 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.543 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:28.543 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.896607+0000", 2024-09-11T09:52:28.543 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:28.543 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.543 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.543 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:28.543 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.896608+0000", 2024-09-11T09:52:28.543 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.0690000000000001e-06 2024-09-11T09:52:28.543 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.543 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.543 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:28.543 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.914461+0000", 2024-09-11T09:52:28.544 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.017852903 2024-09-11T09:52:28.544 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.544 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.544 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:28.544 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.914495+0000", 2024-09-11T09:52:28.544 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.4301000000000002e-05 2024-09-11T09:52:28.544 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.544 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.544 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:28.544 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.914522+0000", 2024-09-11T09:52:28.544 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.7127e-05 2024-09-11T09:52:28.544 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.545 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.545 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:28.545 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.914583+0000", 2024-09-11T09:52:28.545 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.0730999999999998e-05 2024-09-11T09:52:28.545 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.545 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.545 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:28.545 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.914709+0000", 2024-09-11T09:52:28.545 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00012590400000000001 2024-09-11T09:52:28.545 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.545 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.545 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 1", 2024-09-11T09:52:28.545 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.914809+0000", 2024-09-11T09:52:28.546 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.9882000000000004e-05 2024-09-11T09:52:28.546 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.546 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.546 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:28.546 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.932551+0000", 2024-09-11T09:52:28.546 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.017742048999999999 2024-09-11T09:52:28.546 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.546 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.546 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:52:28.546 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.935144+0000", 2024-09-11T09:52:28.546 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0025927630000000001 2024-09-11T09:52:28.546 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.546 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.546 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:28.546 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.935205+0000", 2024-09-11T09:52:28.547 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.1634e-05 2024-09-11T09:52:28.547 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.547 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.547 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:28.547 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.935262+0000", 2024-09-11T09:52:28.547 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.6817000000000002e-05 2024-09-11T09:52:28.547 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:28.547 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:28.547 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:28.547 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.547 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.547 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_repop(client.4688.0:40 2.7 e53/38 2:fe41411c:::rbd_object_map.1250f5e25b80:head v 53'9, mlcod=53'7)", 2024-09-11T09:52:28.547 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:22.920457+0000", 2024-09-11T09:52:28.547 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 65.605584394000005, 2024-09-11T09:52:28.548 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.011132817, 2024-09-11T09:52:28.548 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:28.548 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:28.548 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:28.548 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:28.548 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.548 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:28.548 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.920457+0000", 2024-09-11T09:52:28.548 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:28.548 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.548 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.548 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:28.548 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.920457+0000", 2024-09-11T09:52:28.549 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:28.549 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.549 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.549 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:28.549 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.920459+0000", 2024-09-11T09:52:28.549 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.6729999999999999e-06 2024-09-11T09:52:28.549 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.549 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.549 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:28.549 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.920469+0000", 2024-09-11T09:52:28.549 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.944e-06 2024-09-11T09:52:28.549 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.549 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.549 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:28.549 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.920495+0000", 2024-09-11T09:52:28.550 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.6401999999999999e-05 2024-09-11T09:52:28.550 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.550 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.550 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:28.550 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.920506+0000", 2024-09-11T09:52:28.550 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.133e-05 2024-09-11T09:52:28.550 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.550 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.550 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:28.550 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.920581+0000", 2024-09-11T09:52:28.550 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.4932999999999996e-05 2024-09-11T09:52:28.550 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.550 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.551 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:28.551 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.920612+0000", 2024-09-11T09:52:28.551 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.0537999999999998e-05 2024-09-11T09:52:28.551 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.551 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.551 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:28.551 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.931537+0000", 2024-09-11T09:52:28.551 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.010925239 2024-09-11T09:52:28.551 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.551 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.551 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:28.551 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.931590+0000", 2024-09-11T09:52:28.551 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.2757999999999999e-05 2024-09-11T09:52:28.551 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:28.552 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:28.552 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:28.552 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.552 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.552 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_repop(client.4688.0:41 2.7 e53/38 2:fe41411c:::rbd_object_map.1250f5e25b80:head v 53'10, mlcod=53'7)", 2024-09-11T09:52:28.552 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:22.921422+0000", 2024-09-11T09:52:28.552 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 65.604618978000005, 2024-09-11T09:52:28.552 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.013734942, 2024-09-11T09:52:28.552 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:28.552 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:28.552 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:28.552 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:28.552 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.552 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:28.553 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921422+0000", 2024-09-11T09:52:28.553 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:28.553 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.553 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.553 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:28.553 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921422+0000", 2024-09-11T09:52:28.553 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:28.553 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.553 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.553 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:28.553 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921426+0000", 2024-09-11T09:52:28.553 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.6150000000000001e-06 2024-09-11T09:52:28.553 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.553 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.554 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:28.554 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921435+0000", 2024-09-11T09:52:28.554 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.9840000000000007e-06 2024-09-11T09:52:28.554 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.554 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.554 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:28.554 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921444+0000", 2024-09-11T09:52:28.554 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.6000000000000007e-06 2024-09-11T09:52:28.554 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.554 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.554 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:28.554 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921450+0000", 2024-09-11T09:52:28.554 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.4690000000000002e-06 2024-09-11T09:52:28.554 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.555 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.555 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:28.555 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921511+0000", 2024-09-11T09:52:28.555 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.0779999999999997e-05 2024-09-11T09:52:28.555 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.555 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.555 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:28.555 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921531+0000", 2024-09-11T09:52:28.555 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.0302999999999999e-05 2024-09-11T09:52:28.555 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.555 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.555 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:28.555 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.935115+0000", 2024-09-11T09:52:28.555 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.013583471 2024-09-11T09:52:28.556 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.556 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.556 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:28.556 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.935157+0000", 2024-09-11T09:52:28.556 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.2719999999999998e-05 2024-09-11T09:52:28.556 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:28.556 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:28.556 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:28.556 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.556 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.556 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4688.0:42 2.3 2:c54742d8:::rbd_data.1250f5e25b80.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:28.556 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:22.932560+0000", 2024-09-11T09:52:28.556 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 65.593481234999999, 2024-09-11T09:52:28.557 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.017798198000000001, 2024-09-11T09:52:28.557 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:28.557 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:28.557 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:28.557 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:52:28.557 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4688", 2024-09-11T09:52:28.557 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:52:28.557 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 42 2024-09-11T09:52:28.557 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.557 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:28.557 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.557 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:28.557 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.932560+0000", 2024-09-11T09:52:28.557 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:28.558 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.558 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.558 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:28.558 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.932560+0000", 2024-09-11T09:52:28.558 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:28.558 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.558 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.558 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:28.558 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.932562+0000", 2024-09-11T09:52:28.558 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.17e-06 2024-09-11T09:52:28.558 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.558 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.558 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:28.558 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.938851+0000", 2024-09-11T09:52:28.558 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.006288971 2024-09-11T09:52:28.559 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.559 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.559 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:28.559 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.938865+0000", 2024-09-11T09:52:28.559 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.365e-05 2024-09-11T09:52:28.559 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.559 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.559 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:28.559 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.938871+0000", 2024-09-11T09:52:28.559 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.8850000000000004e-06 2024-09-11T09:52:28.559 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.559 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.559 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:28.559 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.938926+0000", 2024-09-11T09:52:28.559 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.4691999999999997e-05 2024-09-11T09:52:28.560 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.560 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.560 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:28.560 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.939059+0000", 2024-09-11T09:52:28.560 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00013310899999999999 2024-09-11T09:52:28.560 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.560 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.560 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 2", 2024-09-11T09:52:28.560 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.939171+0000", 2024-09-11T09:52:28.560 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000112718 2024-09-11T09:52:28.560 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.560 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.560 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:28.561 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.950262+0000", 2024-09-11T09:52:28.561 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.011091063 2024-09-11T09:52:28.561 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.561 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.561 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:52:28.561 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.950274+0000", 2024-09-11T09:52:28.561 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1963e-05 2024-09-11T09:52:28.561 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.561 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.561 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:28.561 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.950318+0000", 2024-09-11T09:52:28.561 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.3519999999999997e-05 2024-09-11T09:52:28.561 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.561 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.562 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:28.562 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.950358+0000", 2024-09-11T09:52:28.562 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.0457000000000002e-05 2024-09-11T09:52:28.562 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:28.562 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:28.562 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:28.562 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.562 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.562 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_repop(client.4694.0:34 2.7 e53/38 2:f80bfc0c:ns::rbd_data.1256a0c02879.0000000000000002:head v 53'25, mlcod=53'23)", 2024-09-11T09:52:28.562 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.094944+0000", 2024-09-11T09:52:28.562 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 65.431097843000003, 2024-09-11T09:52:28.562 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.010896785000000001, 2024-09-11T09:52:28.563 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:28.563 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:28.563 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:28.563 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:28.563 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.563 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:28.563 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.094944+0000", 2024-09-11T09:52:28.563 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:28.563 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.563 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.563 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:28.563 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.094944+0000", 2024-09-11T09:52:28.563 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:28.564 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.564 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.564 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:28.564 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.094944+0000", 2024-09-11T09:52:28.564 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.0699999999999996e-07 2024-09-11T09:52:28.564 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.564 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.564 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:28.564 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.099060+0000", 2024-09-11T09:52:28.564 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0041155580000000001 2024-09-11T09:52:28.564 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.564 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.564 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:28.564 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.099082+0000", 2024-09-11T09:52:28.565 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.2323999999999998e-05 2024-09-11T09:52:28.565 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.565 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.565 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:28.565 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.099094+0000", 2024-09-11T09:52:28.565 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1535000000000001e-05 2024-09-11T09:52:28.565 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.565 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.565 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:28.565 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.099160+0000", 2024-09-11T09:52:28.565 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.6449e-05 2024-09-11T09:52:28.565 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.565 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.566 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:28.566 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.099183+0000", 2024-09-11T09:52:28.566 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.3108e-05 2024-09-11T09:52:28.566 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.566 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.566 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:28.566 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.105776+0000", 2024-09-11T09:52:28.566 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0065922760000000002 2024-09-11T09:52:28.566 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.566 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.566 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:28.566 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.105840+0000", 2024-09-11T09:52:28.566 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.4727999999999994e-05 2024-09-11T09:52:28.566 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:28.567 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:28.567 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:28.567 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.567 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.567 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_repop(client.4694.0:35 2.7 e53/38 2:e8766325:ns::rbd_object_map.1256a0c02879:head v 53'26, mlcod=53'23)", 2024-09-11T09:52:28.567 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.100964+0000", 2024-09-11T09:52:28.567 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 65.425077314999996, 2024-09-11T09:52:28.567 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0049649359999999997, 2024-09-11T09:52:28.567 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:28.567 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:28.567 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:28.567 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:28.567 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.568 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:28.568 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.100964+0000", 2024-09-11T09:52:28.568 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:28.568 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.568 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.568 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:28.568 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.100964+0000", 2024-09-11T09:52:28.568 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:28.568 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.568 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.568 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:28.568 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.100966+0000", 2024-09-11T09:52:28.568 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.6330000000000001e-06 2024-09-11T09:52:28.569 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.569 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.569 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:28.569 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.100979+0000", 2024-09-11T09:52:28.569 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.2813000000000001e-05 2024-09-11T09:52:28.569 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.569 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.569 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:28.569 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.100992+0000", 2024-09-11T09:52:28.569 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.3414999999999999e-05 2024-09-11T09:52:28.569 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.569 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.569 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:28.570 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.101002+0000", 2024-09-11T09:52:28.570 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.0159e-05 2024-09-11T09:52:28.570 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.570 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.570 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:28.570 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.102365+0000", 2024-09-11T09:52:28.570 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001362912 2024-09-11T09:52:28.570 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.570 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.570 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:28.570 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.102382+0000", 2024-09-11T09:52:28.570 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.7433000000000001e-05 2024-09-11T09:52:28.570 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.570 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.571 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:28.571 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.105901+0000", 2024-09-11T09:52:28.571 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.003518476 2024-09-11T09:52:28.571 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.571 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.571 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:28.571 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.105929+0000", 2024-09-11T09:52:28.571 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.8095000000000001e-05 2024-09-11T09:52:28.571 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:28.571 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:28.571 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:28.571 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.571 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.571 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4694.0:41 2.2 2:5e2505c4:ns::rbd_data.1256a0c02879.0000000000000003:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:28.571 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.107511+0000", 2024-09-11T09:52:28.572 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 65.418530161999996, 2024-09-11T09:52:28.572 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.018571066000000001, 2024-09-11T09:52:28.572 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:28.572 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:28.572 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:28.572 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:52:28.572 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4694", 2024-09-11T09:52:28.572 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:52:28.572 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 41 2024-09-11T09:52:28.572 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.572 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:28.572 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.572 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:28.573 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.107511+0000", 2024-09-11T09:52:28.573 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:28.573 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.573 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.573 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:28.573 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.107511+0000", 2024-09-11T09:52:28.573 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:28.573 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.573 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.573 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:28.573 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.107513+0000", 2024-09-11T09:52:28.573 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.3489999999999999e-06 2024-09-11T09:52:28.573 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.574 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.574 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:28.574 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.112704+0000", 2024-09-11T09:52:28.574 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0051918509999999999 2024-09-11T09:52:28.574 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.574 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.574 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:28.574 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.112718+0000", 2024-09-11T09:52:28.574 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.359e-05 2024-09-11T09:52:28.574 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.574 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.574 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:28.574 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.112725+0000", 2024-09-11T09:52:28.574 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.9940000000000003e-06 2024-09-11T09:52:28.574 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.575 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.575 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:28.575 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.112788+0000", 2024-09-11T09:52:28.575 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.2447999999999996e-05 2024-09-11T09:52:28.575 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.575 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.575 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:28.575 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.112937+0000", 2024-09-11T09:52:28.575 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00014921499999999999 2024-09-11T09:52:28.575 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.575 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.575 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 1", 2024-09-11T09:52:28.576 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.113085+0000", 2024-09-11T09:52:28.576 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000148085 2024-09-11T09:52:28.576 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.576 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.576 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:52:28.576 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.124318+0000", 2024-09-11T09:52:28.576 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.011233158 2024-09-11T09:52:28.576 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.576 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.576 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:28.576 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.125957+0000", 2024-09-11T09:52:28.576 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001638701 2024-09-11T09:52:28.576 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.577 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.577 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:28.577 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.126029+0000", 2024-09-11T09:52:28.577 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.2125000000000005e-05 2024-09-11T09:52:28.577 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.577 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.577 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:28.577 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.126082+0000", 2024-09-11T09:52:28.577 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.3550000000000001e-05 2024-09-11T09:52:28.577 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:28.577 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:28.577 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:28.578 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.578 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.578 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4694.0:44 2.3 2:cd1a7d1c:ns::rbd_data.1256a0c02879.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:28.578 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.112744+0000", 2024-09-11T09:52:28.578 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 65.413297595000003, 2024-09-11T09:52:28.578 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.021672212999999999, 2024-09-11T09:52:28.578 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:28.578 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:28.578 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:28.578 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:52:28.578 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4694", 2024-09-11T09:52:28.578 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:52:28.578 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 44 2024-09-11T09:52:28.578 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.579 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:28.579 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.579 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:28.579 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.112744+0000", 2024-09-11T09:52:28.579 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:28.579 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.579 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.579 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:28.579 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.112744+0000", 2024-09-11T09:52:28.579 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:28.579 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.579 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.579 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:28.580 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.112745+0000", 2024-09-11T09:52:28.580 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.2900000000000002e-07 2024-09-11T09:52:28.580 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.580 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.580 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:28.580 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.119929+0000", 2024-09-11T09:52:28.580 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0071846230000000002 2024-09-11T09:52:28.580 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.580 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.580 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:28.580 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.119949+0000", 2024-09-11T09:52:28.580 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.9264000000000001e-05 2024-09-11T09:52:28.580 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.581 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.581 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:28.581 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.119959+0000", 2024-09-11T09:52:28.581 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.0465999999999999e-05 2024-09-11T09:52:28.581 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.581 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.581 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:28.581 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.120028+0000", 2024-09-11T09:52:28.581 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.9318000000000003e-05 2024-09-11T09:52:28.581 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.581 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.581 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:28.581 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.120160+0000", 2024-09-11T09:52:28.581 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000132015 2024-09-11T09:52:28.582 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.582 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.582 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 2", 2024-09-11T09:52:28.582 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.120260+0000", 2024-09-11T09:52:28.582 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.9397999999999999e-05 2024-09-11T09:52:28.582 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.582 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.582 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:52:28.582 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.134246+0000", 2024-09-11T09:52:28.582 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.013986721000000001 2024-09-11T09:52:28.582 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.582 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.582 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:28.583 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.134295+0000", 2024-09-11T09:52:28.583 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.8649000000000001e-05 2024-09-11T09:52:28.583 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.583 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.583 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:28.583 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.134361+0000", 2024-09-11T09:52:28.583 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.6081999999999995e-05 2024-09-11T09:52:28.583 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.583 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.583 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:28.583 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.134416+0000", 2024-09-11T09:52:28.583 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.4848000000000001e-05 2024-09-11T09:52:28.583 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:28.583 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:28.584 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:28.584 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.584 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.584 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4694.0:46 2.3 2:ce0e1469:ns::rbd_data.1256a0c02879.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:28.584 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.119993+0000", 2024-09-11T09:52:28.584 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 65.406048119000005, 2024-09-11T09:52:28.584 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.025840610999999999, 2024-09-11T09:52:28.584 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:28.584 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:28.584 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:28.584 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:52:28.584 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4694", 2024-09-11T09:52:28.584 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:52:28.585 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 46 2024-09-11T09:52:28.585 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.585 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:28.585 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.585 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:28.585 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.119993+0000", 2024-09-11T09:52:28.585 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:28.585 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.585 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.585 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:28.585 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.119993+0000", 2024-09-11T09:52:28.585 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:28.585 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.586 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.586 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:28.586 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.119995+0000", 2024-09-11T09:52:28.586 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1930000000000001e-06 2024-09-11T09:52:28.586 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.586 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.586 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:28.586 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.128458+0000", 2024-09-11T09:52:28.586 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0084632630000000004 2024-09-11T09:52:28.586 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.586 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.586 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:28.586 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.128478+0000", 2024-09-11T09:52:28.586 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.0353e-05 2024-09-11T09:52:28.586 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.587 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.587 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:28.587 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.128491+0000", 2024-09-11T09:52:28.587 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.2954e-05 2024-09-11T09:52:28.587 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.587 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.587 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:28.587 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.128558+0000", 2024-09-11T09:52:28.587 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.7076999999999995e-05 2024-09-11T09:52:28.587 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.587 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.587 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:28.587 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.128708+0000", 2024-09-11T09:52:28.587 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00014939599999999999 2024-09-11T09:52:28.588 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.588 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.588 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 2", 2024-09-11T09:52:28.588 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.128866+0000", 2024-09-11T09:52:28.588 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000158145 2024-09-11T09:52:28.588 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.588 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.588 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:28.588 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.141045+0000", 2024-09-11T09:52:28.588 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.012179331999999999 2024-09-11T09:52:28.588 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.588 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.588 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:52:28.588 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.145742+0000", 2024-09-11T09:52:28.589 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0046972050000000003 2024-09-11T09:52:28.589 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.589 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.589 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:28.589 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.145779+0000", 2024-09-11T09:52:28.589 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.6544000000000001e-05 2024-09-11T09:52:28.589 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.589 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.589 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:28.589 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.145834+0000", 2024-09-11T09:52:28.589 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.5148999999999997e-05 2024-09-11T09:52:28.589 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:28.589 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:28.590 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:28.590 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.590 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.590 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4694.0:48 2.3 2:d2090b70:ns::rbd_data.1256a0c02879.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:28.590 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.130540+0000", 2024-09-11T09:52:28.590 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 65.395501852999999, 2024-09-11T09:52:28.590 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.017167789999999999, 2024-09-11T09:52:28.590 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:28.590 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:28.590 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:28.590 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:52:28.590 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4694", 2024-09-11T09:52:28.590 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:52:28.591 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 48 2024-09-11T09:52:28.591 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.591 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:28.591 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.591 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:28.591 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.130540+0000", 2024-09-11T09:52:28.591 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:28.591 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.591 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.591 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:28.591 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.130540+0000", 2024-09-11T09:52:28.591 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:28.591 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.591 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.591 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:28.592 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.130542+0000", 2024-09-11T09:52:28.592 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.9209999999999999e-06 2024-09-11T09:52:28.592 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.592 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.592 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:28.592 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.134691+0000", 2024-09-11T09:52:28.592 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0041498739999999996 2024-09-11T09:52:28.592 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.592 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.592 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:28.592 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.134705+0000", 2024-09-11T09:52:28.592 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.3669000000000001e-05 2024-09-11T09:52:28.592 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.593 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.593 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:28.593 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.134712+0000", 2024-09-11T09:52:28.593 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.8410000000000002e-06 2024-09-11T09:52:28.593 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.593 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.593 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:28.593 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.134754+0000", 2024-09-11T09:52:28.593 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.2524999999999997e-05 2024-09-11T09:52:28.593 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.593 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.593 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:28.593 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.134875+0000", 2024-09-11T09:52:28.594 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000120332 2024-09-11T09:52:28.594 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.594 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.594 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 2", 2024-09-11T09:52:28.594 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.135005+0000", 2024-09-11T09:52:28.594 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00013024900000000001 2024-09-11T09:52:28.594 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.594 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.594 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:28.594 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.145971+0000", 2024-09-11T09:52:28.594 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.010966203000000001 2024-09-11T09:52:28.594 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.594 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.595 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:52:28.595 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.147600+0000", 2024-09-11T09:52:28.595 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001628382 2024-09-11T09:52:28.595 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.595 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.595 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:28.595 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.147635+0000", 2024-09-11T09:52:28.595 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.5420000000000003e-05 2024-09-11T09:52:28.595 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.595 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.595 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:28.595 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.147707+0000", 2024-09-11T09:52:28.595 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.2373999999999995e-05 2024-09-11T09:52:28.595 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:28.596 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:28.596 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:28.596 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.596 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.596 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_repop(client.4694.0:49 2.7 e53/38 2:e8766325:ns::rbd_object_map.1256a0c02879:head v 53'33, mlcod=53'31)", 2024-09-11T09:52:28.596 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.135391+0000", 2024-09-11T09:52:28.596 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 65.390650288000003, 2024-09-11T09:52:28.596 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0051189859999999998, 2024-09-11T09:52:28.596 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:28.596 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:28.596 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:28.596 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:28.597 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.597 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:28.597 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.135391+0000", 2024-09-11T09:52:28.597 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:28.597 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.597 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.597 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:28.597 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.135391+0000", 2024-09-11T09:52:28.597 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:28.597 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.597 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.597 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:28.597 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.135393+0000", 2024-09-11T09:52:28.598 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.4020000000000001e-06 2024-09-11T09:52:28.598 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.598 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.598 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:28.598 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.135400+0000", 2024-09-11T09:52:28.598 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.5129999999999999e-06 2024-09-11T09:52:28.598 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.598 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.598 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:28.598 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.135411+0000", 2024-09-11T09:52:28.598 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1038e-05 2024-09-11T09:52:28.598 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.598 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.598 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:28.599 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.135418+0000", 2024-09-11T09:52:28.599 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.1489999999999997e-06 2024-09-11T09:52:28.599 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.599 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.599 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:28.599 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.135470+0000", 2024-09-11T09:52:28.599 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.2021999999999998e-05 2024-09-11T09:52:28.599 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.599 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.599 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:28.599 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.135501+0000", 2024-09-11T09:52:28.599 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.0349999999999999e-05 2024-09-11T09:52:28.599 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.600 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.600 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:28.600 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.140453+0000", 2024-09-11T09:52:28.600 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0049522409999999996 2024-09-11T09:52:28.600 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.600 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.600 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:28.600 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.140510+0000", 2024-09-11T09:52:28.600 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.7271000000000001e-05 2024-09-11T09:52:28.600 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:28.600 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:28.600 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:28.600 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.601 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.601 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4694.0:54 2.3 2:c44f0253:ns::rbd_data.1256a0c02879.000000000000000b:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:28.601 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.150151+0000", 2024-09-11T09:52:28.601 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 65.375890436999995, 2024-09-11T09:52:28.601 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.015103484, 2024-09-11T09:52:28.601 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:28.601 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:28.601 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:28.601 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:52:28.601 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4694", 2024-09-11T09:52:28.602 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:52:28.602 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 54 2024-09-11T09:52:28.602 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.602 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:28.602 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.602 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:28.602 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.150151+0000", 2024-09-11T09:52:28.602 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:28.602 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.602 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.602 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:28.602 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.150151+0000", 2024-09-11T09:52:28.602 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:28.603 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.603 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.603 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:28.603 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.150152+0000", 2024-09-11T09:52:28.603 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.2550000000000001e-06 2024-09-11T09:52:28.603 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.603 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.603 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:28.603 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.154528+0000", 2024-09-11T09:52:28.603 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0043758219999999997 2024-09-11T09:52:28.603 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.603 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.603 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:28.604 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.154541+0000", 2024-09-11T09:52:28.604 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.3414999999999999e-05 2024-09-11T09:52:28.604 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.604 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.604 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:28.604 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.154551+0000", 2024-09-11T09:52:28.604 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.4350000000000003e-06 2024-09-11T09:52:28.604 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.604 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.604 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:28.604 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.154615+0000", 2024-09-11T09:52:28.604 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.3627000000000006e-05 2024-09-11T09:52:28.604 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.604 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.605 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:28.605 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.154717+0000", 2024-09-11T09:52:28.605 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000102729 2024-09-11T09:52:28.605 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.605 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.605 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 2", 2024-09-11T09:52:28.605 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.154826+0000", 2024-09-11T09:52:28.605 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000108409 2024-09-11T09:52:28.605 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.605 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.605 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:52:28.605 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.165094+0000", 2024-09-11T09:52:28.605 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.010268210999999999 2024-09-11T09:52:28.606 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.606 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.606 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:28.606 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.165138+0000", 2024-09-11T09:52:28.606 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.4078e-05 2024-09-11T09:52:28.606 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.606 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.606 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:28.606 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.165204+0000", 2024-09-11T09:52:28.606 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.5716000000000006e-05 2024-09-11T09:52:28.606 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.606 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.606 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:28.606 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.165254+0000", 2024-09-11T09:52:28.607 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.0787e-05 2024-09-11T09:52:28.607 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:28.607 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:28.607 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:28.607 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.607 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.607 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_repop(client.4694.0:55 2.7 e53/38 2:e8766325:ns::rbd_object_map.1256a0c02879:head v 53'37, mlcod=53'34)", 2024-09-11T09:52:28.607 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.150221+0000", 2024-09-11T09:52:28.607 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 65.375820571999995, 2024-09-11T09:52:28.607 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00530015, 2024-09-11T09:52:28.607 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:28.607 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:28.607 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:28.607 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:28.608 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.608 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:28.608 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.150221+0000", 2024-09-11T09:52:28.608 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:28.608 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.608 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.608 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:28.608 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.150221+0000", 2024-09-11T09:52:28.608 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:28.608 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.608 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.608 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:28.608 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.150221+0000", 2024-09-11T09:52:28.609 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.9100000000000004e-07 2024-09-11T09:52:28.609 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.609 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.609 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:28.609 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.150226+0000", 2024-09-11T09:52:28.609 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.2250000000000002e-06 2024-09-11T09:52:28.609 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.609 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.609 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:28.609 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.150251+0000", 2024-09-11T09:52:28.609 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.4987e-05 2024-09-11T09:52:28.609 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.609 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.610 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:28.610 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.150258+0000", 2024-09-11T09:52:28.610 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.0790000000000003e-06 2024-09-11T09:52:28.610 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.610 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.610 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:28.610 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.150305+0000", 2024-09-11T09:52:28.610 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.6879000000000003e-05 2024-09-11T09:52:28.610 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.610 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.610 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:28.610 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.150320+0000", 2024-09-11T09:52:28.610 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.5268999999999999e-05 2024-09-11T09:52:28.611 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.611 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.611 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:28.611 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.155493+0000", 2024-09-11T09:52:28.611 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0051731390000000002 2024-09-11T09:52:28.611 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.611 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.611 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:28.611 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.155521+0000", 2024-09-11T09:52:28.611 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.8081000000000001e-05 2024-09-11T09:52:28.611 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:28.611 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:28.611 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:28.612 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.612 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.612 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4694.0:56 2.3 2:c462f029:ns::rbd_data.1256a0c02879.000000000000000c:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:28.612 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.154576+0000", 2024-09-11T09:52:28.612 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 65.371465595999993, 2024-09-11T09:52:28.612 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.020781621, 2024-09-11T09:52:28.612 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:28.612 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:28.612 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:28.612 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:52:28.612 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4694", 2024-09-11T09:52:28.612 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:52:28.612 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 56 2024-09-11T09:52:28.613 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.613 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:28.613 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.613 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:28.613 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.154576+0000", 2024-09-11T09:52:28.613 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:28.613 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.613 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.613 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:28.613 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.154576+0000", 2024-09-11T09:52:28.613 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:28.613 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.613 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.614 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:28.614 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.154577+0000", 2024-09-11T09:52:28.614 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.2249999999999999e-06 2024-09-11T09:52:28.614 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.614 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.614 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:28.614 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.158783+0000", 2024-09-11T09:52:28.614 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0042063889999999996 2024-09-11T09:52:28.614 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.614 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.614 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:28.614 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.158803+0000", 2024-09-11T09:52:28.614 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.9576999999999999e-05 2024-09-11T09:52:28.615 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.615 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.615 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:28.615 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.158815+0000", 2024-09-11T09:52:28.615 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.2221999999999999e-05 2024-09-11T09:52:28.615 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.615 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.615 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:28.615 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.158902+0000", 2024-09-11T09:52:28.615 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.6390000000000005e-05 2024-09-11T09:52:28.615 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.615 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.615 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:28.616 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.159141+0000", 2024-09-11T09:52:28.616 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000238886 2024-09-11T09:52:28.616 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.616 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.616 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 2", 2024-09-11T09:52:28.616 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.159279+0000", 2024-09-11T09:52:28.616 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000138073 2024-09-11T09:52:28.616 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.616 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.616 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:52:28.616 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.173297+0000", 2024-09-11T09:52:28.616 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.014018113 2024-09-11T09:52:28.616 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.617 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.617 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:28.617 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.175278+0000", 2024-09-11T09:52:28.617 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0019817390000000002 2024-09-11T09:52:28.617 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.617 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.617 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:28.617 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.175316+0000", 2024-09-11T09:52:28.617 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.7829e-05 2024-09-11T09:52:28.617 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.617 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.617 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:28.618 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.175357+0000", 2024-09-11T09:52:28.618 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.1177999999999998e-05 2024-09-11T09:52:28.618 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:28.618 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:28.618 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:28.618 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.618 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.618 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4694.0:59 2.2 2:41adefc9:ns::rbd_data.1256a0c02879.000000000000000e:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:28.618 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.158843+0000", 2024-09-11T09:52:28.618 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 65.367198029999997, 2024-09-11T09:52:28.618 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.017874207, 2024-09-11T09:52:28.618 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:28.618 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:28.619 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:28.619 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:52:28.619 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4694", 2024-09-11T09:52:28.619 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:52:28.619 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 59 2024-09-11T09:52:28.619 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.619 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:28.619 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.619 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:28.619 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.158843+0000", 2024-09-11T09:52:28.619 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:28.619 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.619 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.620 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:28.620 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.158843+0000", 2024-09-11T09:52:28.620 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:28.620 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.620 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.620 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:28.620 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.158845+0000", 2024-09-11T09:52:28.620 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.7439999999999999e-06 2024-09-11T09:52:28.620 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.620 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.620 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:28.620 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.162713+0000", 2024-09-11T09:52:28.620 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0038678829999999999 2024-09-11T09:52:28.621 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.621 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.621 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:28.621 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.162728+0000", 2024-09-11T09:52:28.621 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.5114e-05 2024-09-11T09:52:28.621 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.621 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.621 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:28.621 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.162737+0000", 2024-09-11T09:52:28.621 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.1099999999999992e-06 2024-09-11T09:52:28.621 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.621 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.621 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:28.621 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.162799+0000", 2024-09-11T09:52:28.621 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.1934999999999996e-05 2024-09-11T09:52:28.622 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.622 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.622 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:28.622 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.162911+0000", 2024-09-11T09:52:28.622 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000111867 2024-09-11T09:52:28.622 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.622 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.622 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 1", 2024-09-11T09:52:28.622 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.163030+0000", 2024-09-11T09:52:28.622 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000119108 2024-09-11T09:52:28.622 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.622 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.622 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:52:28.623 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.174086+0000", 2024-09-11T09:52:28.623 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.01105543 2024-09-11T09:52:28.623 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.623 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.623 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:28.623 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.176614+0000", 2024-09-11T09:52:28.623 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0025284940000000001 2024-09-11T09:52:28.623 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.623 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.623 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:28.623 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.176672+0000", 2024-09-11T09:52:28.623 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.7892000000000001e-05 2024-09-11T09:52:28.623 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.624 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.624 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:28.624 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.176718+0000", 2024-09-11T09:52:28.624 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.5630000000000002e-05 2024-09-11T09:52:28.624 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:28.624 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:28.624 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:28.624 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.624 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.624 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.5002.0:89 2.3 2:c54742d8:::rbd_data.1250f5e25b80.0000000000000006:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-09-11T09:52:28.624 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:38.546016+0000", 2024-09-11T09:52:28.624 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 49.980025484000002, 2024-09-11T09:52:28.624 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0049098550000000003, 2024-09-11T09:52:28.624 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:28.625 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:28.625 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:28.625 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:52:28.625 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.5002", 2024-09-11T09:52:28.625 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/2204440692", 2024-09-11T09:52:28.625 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 89 2024-09-11T09:52:28.625 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.625 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:28.625 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.625 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:28.625 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.546016+0000", 2024-09-11T09:52:28.625 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:28.625 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.625 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.626 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:28.626 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.546016+0000", 2024-09-11T09:52:28.626 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:28.626 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.626 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.626 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:28.626 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.546017+0000", 2024-09-11T09:52:28.626 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.3399999999999998e-07 2024-09-11T09:52:28.626 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.626 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.626 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:28.626 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.546839+0000", 2024-09-11T09:52:28.626 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00082222800000000004 2024-09-11T09:52:28.627 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.627 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.627 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:28.627 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.546851+0000", 2024-09-11T09:52:28.627 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.2075999999999999e-05 2024-09-11T09:52:28.627 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.627 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.627 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:28.627 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.546856+0000", 2024-09-11T09:52:28.627 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.4009999999999996e-06 2024-09-11T09:52:28.627 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.627 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.628 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:28.628 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.546897+0000", 2024-09-11T09:52:28.628 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.0113e-05 2024-09-11T09:52:28.628 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.628 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.628 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:28.628 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.546954+0000", 2024-09-11T09:52:28.628 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.7825000000000003e-05 2024-09-11T09:52:28.628 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.628 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.628 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 2", 2024-09-11T09:52:28.628 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.547055+0000", 2024-09-11T09:52:28.628 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000100269 2024-09-11T09:52:28.628 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.629 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.629 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:28.629 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.549675+0000", 2024-09-11T09:52:28.629 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0026206609999999998 2024-09-11T09:52:28.629 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.629 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.629 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:52:28.629 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.550858+0000", 2024-09-11T09:52:28.629 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0011825270000000001 2024-09-11T09:52:28.629 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.629 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.629 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:28.629 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.550894+0000", 2024-09-11T09:52:28.630 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.5896999999999999e-05 2024-09-11T09:52:28.630 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.630 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.630 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:28.630 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.550926+0000", 2024-09-11T09:52:28.630 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.2023999999999997e-05 2024-09-11T09:52:28.630 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:28.630 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:28.630 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:28.630 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.630 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.630 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.5002.0:91 2.3 2:c54742d8:::rbd_data.1250f5e25b80.0000000000000006:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-09-11T09:52:28.630 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:38.553743+0000", 2024-09-11T09:52:28.631 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 49.972298441, 2024-09-11T09:52:28.631 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0051633629999999998, 2024-09-11T09:52:28.631 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:28.631 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:28.631 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:28.631 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:52:28.631 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.5002", 2024-09-11T09:52:28.631 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/2204440692", 2024-09-11T09:52:28.631 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 91 2024-09-11T09:52:28.631 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.631 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:28.631 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.631 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:28.632 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.553743+0000", 2024-09-11T09:52:28.632 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:28.632 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.632 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.632 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:28.632 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.553743+0000", 2024-09-11T09:52:28.632 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:28.632 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.632 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.632 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:28.632 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.553745+0000", 2024-09-11T09:52:28.632 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.5120000000000001e-06 2024-09-11T09:52:28.633 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.633 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.633 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:28.633 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.554588+0000", 2024-09-11T09:52:28.633 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00084352299999999999 2024-09-11T09:52:28.633 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.633 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.633 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:28.633 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.554604+0000", 2024-09-11T09:52:28.633 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.643e-05 2024-09-11T09:52:28.633 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.634 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.634 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:28.634 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.554613+0000", 2024-09-11T09:52:28.634 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.5669999999999995e-06 2024-09-11T09:52:28.634 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.634 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.634 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:28.634 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.554662+0000", 2024-09-11T09:52:28.634 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.8510000000000001e-05 2024-09-11T09:52:28.634 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.634 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.635 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:28.635 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.554732+0000", 2024-09-11T09:52:28.635 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.0297999999999997e-05 2024-09-11T09:52:28.635 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.635 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.635 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 2", 2024-09-11T09:52:28.635 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.554870+0000", 2024-09-11T09:52:28.635 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00013856599999999999 2024-09-11T09:52:28.635 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.635 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.635 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:28.635 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.557875+0000", 2024-09-11T09:52:28.635 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0030042760000000002 2024-09-11T09:52:28.636 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.636 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.636 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:52:28.636 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.558835+0000", 2024-09-11T09:52:28.636 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00096057900000000003 2024-09-11T09:52:28.636 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.636 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.636 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:28.636 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.558874+0000", 2024-09-11T09:52:28.636 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.8992999999999997e-05 2024-09-11T09:52:28.636 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.636 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.637 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:28.637 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.558906+0000", 2024-09-11T09:52:28.637 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.2109e-05 2024-09-11T09:52:28.637 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:28.637 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:28.637 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:28.637 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.637 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.637 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.5002.0:93 2.3 2:c54742d8:::rbd_data.1250f5e25b80.0000000000000006:head [write 2228224~917504] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e61)", 2024-09-11T09:52:28.637 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:38.562067+0000", 2024-09-11T09:52:28.637 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 49.963974845000003, 2024-09-11T09:52:28.637 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0050551099999999998, 2024-09-11T09:52:28.638 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:28.638 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:28.638 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:28.638 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:52:28.638 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.5002", 2024-09-11T09:52:28.638 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/2204440692", 2024-09-11T09:52:28.638 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 93 2024-09-11T09:52:28.638 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.638 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:28.638 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.638 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:28.638 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.562067+0000", 2024-09-11T09:52:28.638 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:28.638 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.639 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.639 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:28.639 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.562067+0000", 2024-09-11T09:52:28.639 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:28.639 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.639 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.639 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:28.639 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.562068+0000", 2024-09-11T09:52:28.639 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.0669999999999999e-06 2024-09-11T09:52:28.639 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.639 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.639 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:28.639 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.563170+0000", 2024-09-11T09:52:28.639 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00110256 2024-09-11T09:52:28.639 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.640 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.640 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:28.640 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.563185+0000", 2024-09-11T09:52:28.640 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.4873e-05 2024-09-11T09:52:28.640 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.640 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.640 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:28.640 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.563193+0000", 2024-09-11T09:52:28.640 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.1319999999999994e-06 2024-09-11T09:52:28.640 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.640 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.640 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:28.640 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.563238+0000", 2024-09-11T09:52:28.641 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.4773e-05 2024-09-11T09:52:28.641 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.641 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.641 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:28.641 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.563290+0000", 2024-09-11T09:52:28.641 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.1690000000000001e-05 2024-09-11T09:52:28.641 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.641 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.641 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 2", 2024-09-11T09:52:28.641 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.563365+0000", 2024-09-11T09:52:28.641 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.5023000000000003e-05 2024-09-11T09:52:28.641 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.641 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.642 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:28.642 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.566552+0000", 2024-09-11T09:52:28.642 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0031869960000000001 2024-09-11T09:52:28.642 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.642 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.642 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:52:28.642 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.567048+0000", 2024-09-11T09:52:28.642 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00049605099999999996 2024-09-11T09:52:28.642 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.642 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.642 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:28.642 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.567095+0000", 2024-09-11T09:52:28.642 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.6823999999999998e-05 2024-09-11T09:52:28.643 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:28.643 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:28.643 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:28.643 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.567122+0000", 2024-09-11T09:52:28.643 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.7121e-05 2024-09-11T09:52:28.643 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:28.643 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:28.643 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:28.643 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:28.643 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:28.643 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-11T09:52:28.644 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-11T09:52:28.645 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T09:52:28.645 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-11T09:52:28.646 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T09:52:28.646 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-11T09:52:28.646 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-09-11T09:52:28.645+0000 7f7b6c7d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31904) UID: 0 2024-09-11T09:52:28.646 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-11T09:52:28.647 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T09:52:28.647 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-11T09:52:28.647 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T09:52:28.648 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-11T09:52:28.648 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T09:52:28.648 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-11T09:52:28.649 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T09:52:28.745 INFO:tasks.ceph.osd.0.smithi017.stderr:2024-09-11T09:52:28.746+0000 7fb73c0f1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31665) UID: 0 2024-09-11T09:52:28.788 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:52:28.788 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:52:28.797 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-11T09:52:28.845 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:52:28.846+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:52:28.900 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:52:28.900 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:52:28.909 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-11T09:52:28.947 INFO:tasks.ceph.osd.0.smithi017.stderr:2024-09-11T09:52:28.948+0000 7fb73c0f1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31665) UID: 0 2024-09-11T09:52:29.014 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:52:29.014 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:52:29.023 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-11T09:52:29.047 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:52:29.048+0000 7f3c96ec9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31909) UID: 0 2024-09-11T09:52:29.132 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:52:29.132 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:52:29.141 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-11T09:52:29.185 INFO:tasks.ceph.osd.2.smithi017.stderr:2024-09-11T09:52:29.186+0000 7fd20b170640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31672) UID: 0 2024-09-11T09:52:29.249 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:52:29.249 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:52:29.249 INFO:tasks.ceph.osd.2.smithi017.stderr:2024-09-11T09:52:29.250+0000 7fd20b170640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31672) UID: 0 2024-09-11T09:52:29.257 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-11T09:52:29.349 INFO:tasks.ceph.osd.2.smithi017.stderr:2024-09-11T09:52:29.350+0000 7fd20b170640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31672) UID: 0 2024-09-11T09:52:29.405 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:52:29.405 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:52:29.414 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-11T09:52:29.450 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:52:29.451+0000 7f3c96ec9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31909) UID: 0 2024-09-11T09:52:29.551 INFO:tasks.ceph.osd.2.smithi017.stderr:2024-09-11T09:52:29.552+0000 7fd20b170640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31672) UID: 0 2024-09-11T09:52:29.561 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:52:29.561 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:52:29.570 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-11T09:52:29.654 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:52:29.655+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:52:29.718 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:52:29.718 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:52:29.726 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-11T09:52:29.768 INFO:tasks.ceph.osd.0.smithi017.stderr:2024-09-11T09:52:29.769+0000 7fb73c0f1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31665) UID: 0 2024-09-11T09:52:29.853 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:52:29.854+0000 7f3c96ec9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31909) UID: 0 2024-09-11T09:52:29.874 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:52:29.874 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:52:29.883 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-11T09:52:29.932 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-11T09:52:29.977 INFO:tasks.ceph.osd.0.smithi017.stderr:2024-09-11T09:52:29.978+0000 7fb73c0f1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31665) UID: 0 2024-09-11T09:52:29.997 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:52:29.998 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:52:30.007 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-11T09:52:30.091 INFO:tasks.ceph.osd.2.smithi017.stderr:2024-09-11T09:52:30.092+0000 7fd20b170640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31672) UID: 0 2024-09-11T09:52:30.154 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:52:30.154 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:52:30.155 INFO:tasks.ceph.osd.0.smithi017.stderr:2024-09-11T09:52:30.156+0000 7fb73c0f1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31665) UID: 0 2024-09-11T09:52:30.164 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-11T09:52:30.257 INFO:tasks.ceph.osd.0.smithi017.stderr:2024-09-11T09:52:30.258+0000 7fb73c0f1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31665) UID: 0 2024-09-11T09:52:30.311 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:52:30.311 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:52:30.320 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-11T09:52:30.364 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:52:30.365+0000 7f3a7b90a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 42475) UID: 0 2024-09-11T09:52:30.435 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:52:30.435 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:52:30.445 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-11T09:52:30.458 INFO:tasks.ceph.osd.0.smithi017.stderr:2024-09-11T09:52:30.459+0000 7fb73c0f1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31665) UID: 0 2024-09-11T09:52:30.559 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:52:30.560+0000 7f3a7b90a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 42475) UID: 0 2024-09-11T09:52:30.589 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:52:30.589 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:52:30.599 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-11T09:52:30.650 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T09:52:30.683 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:52:30.684+0000 7f3a7b90a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 42475) UID: 0 2024-09-11T09:52:30.742 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:52:30.742 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:52:30.751 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-11T09:52:30.761 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:52:30.761+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:52:30.774 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-11T09:52:30.775 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T09:52:30.792 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-11T09:52:30.792 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T09:52:30.808 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-11T09:52:30.808 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T09:52:30.825 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-11T09:52:30.825 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T09:52:30.843 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-11T09:52:30.843 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T09:52:30.860 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-11T09:52:30.860 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T09:52:30.861 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:52:30.862+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:52:30.897 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-11T09:52:30.897 INFO:teuthology.orchestra.run.smithi096.stdout: "ops": [], 2024-09-11T09:52:30.897 INFO:teuthology.orchestra.run.smithi096.stdout: "num_ops": 0 2024-09-11T09:52:30.897 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-11T09:52:30.905 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-11T09:52:30.927 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 3, 4, 5] out_osds: [0] dead_osds: [] live_osds: [0, 1, 2, 3, 5, 4] 2024-09-11T09:52:30.927 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-11T09:52:30.927 INFO:tasks.thrashosds.thrasher:Reweighting osd 2 to 0.12773785021164794 2024-09-11T09:52:30.927 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 0.12773785021164794 2024-09-11T09:52:30.934 INFO:teuthology.orchestra.run.smithi017.stderr:nodeep-scrub is set 2024-09-11T09:52:30.938 INFO:tasks.workunit.client.0.smithi107.stderr: Removing all snapshots: 25% complete... Removing all snapshots: 50% complete... Removing all snapshots: 75% complete... 2024-09-11T09:52:30.938+0000 7f43e9408640 0 -- 172.21.15.107:0/61699540 >> [v2:172.21.15.96:6800/3374675329,v1:172.21.15.96:6801/3374675329] conn(0x7f43c4008c50 msgr2=0x7f43c400b0e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-11T09:52:30.938 INFO:tasks.workunit.client.0.smithi107.stderr:Removing all snapshots: 100% complete... Removing all snapshots: 100% complete...done. 2024-09-11T09:52:30.948 INFO:tasks.workunit.client.0.smithi107.stderr:+ rbd -p rbd --namespace '' remove testrbdnbd27121 2024-09-11T09:52:30.990 INFO:tasks.ceph.osd.0.smithi017.stderr:2024-09-11T09:52:30.991+0000 7fb73c0f1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31665) UID: 0 2024-09-11T09:52:31.012 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-11T09:52:31.012 INFO:teuthology.orchestra.run.smithi096.stdout: "ops": [], 2024-09-11T09:52:31.012 INFO:teuthology.orchestra.run.smithi096.stdout: "complaint_time": 30, 2024-09-11T09:52:31.012 INFO:teuthology.orchestra.run.smithi096.stdout: "num_blocked_ops": 0 2024-09-11T09:52:31.012 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-11T09:52:31.021 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-11T09:52:31.048 INFO:tasks.workunit.client.0.smithi107.stderr:2024-09-11T09:52:31.048+0000 7fab9f11c640 0 -- 172.21.15.107:0/1071190571 >> [v2:172.21.15.96:6800/3374675329,v1:172.21.15.96:6801/3374675329] conn(0x562faf0f8ae0 msgr2=0x562faf0faed0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-11T09:52:31.059 INFO:tasks.workunit.client.0.smithi107.stderr: Removing image: 6% complete... Removing image: 12% complete... Removing image: 18% complete... Removing image: 25% complete... Removing image: 31% complete... Removing image: 37% complete... Removing image: 43% complete... Removing image: 50% complete... Removing image: 56% complete... Removing image: 62% complete... Removing image: 68% complete... Removing image: 75% complete... Removing image: 81% complete... Removing image: 87% complete... Removing image: 93% complete...2024-09-11T09:52:31.060+0000 7fab9d6b9640 0 -- 172.21.15.107:0/1071190571 >> [v2:172.21.15.96:6816/332854807,v1:172.21.15.96:6817/332854807] conn(0x7fab78006060 msgr2=0x7fab78008f30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-11T09:52:31.063 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:52:31.063+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:52:31.070 INFO:tasks.workunit.client.0.smithi107.stderr: Removing image: 100% complete...done. 2024-09-11T09:52:31.073 INFO:tasks.workunit.client.0.smithi107.stderr:+ for ns in '' ${NS} 2024-09-11T09:52:31.073 INFO:tasks.workunit.client.0.smithi107.stderr:+ rbd -p rbd --namespace ns status testrbdnbd27121 2024-09-11T09:52:31.119 INFO:tasks.workunit.client.0.smithi107.stdout:Watchers: none 2024-09-11T09:52:31.122 INFO:tasks.workunit.client.0.smithi107.stderr:+ for s in 0.5 1 2 4 8 16 32 2024-09-11T09:52:31.123 INFO:tasks.workunit.client.0.smithi107.stderr:+ sleep 0.5 2024-09-11T09:52:31.132 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-11T09:52:31.132 INFO:teuthology.orchestra.run.smithi096.stdout: "size": 20, 2024-09-11T09:52:31.132 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 600, 2024-09-11T09:52:31.132 INFO:teuthology.orchestra.run.smithi096.stdout: "ops": [ 2024-09-11T09:52:31.132 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.132 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4230.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-11T09:52:31.132 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:18.490736+0000", 2024-09-11T09:52:31.133 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 132.64090519600001, 2024-09-11T09:52:31.133 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0059097569999999999, 2024-09-11T09:52:31.133 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:31.133 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:31.133 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:31.133 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:52:31.133 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4230", 2024-09-11T09:52:31.133 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.17:0/4101109576", 2024-09-11T09:52:31.133 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 1 2024-09-11T09:52:31.133 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.133 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:31.133 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.133 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:31.134 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.490736+0000", 2024-09-11T09:52:31.134 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:31.134 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.134 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.134 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:31.134 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.490736+0000", 2024-09-11T09:52:31.134 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:31.134 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.134 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.134 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:31.134 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.490742+0000", 2024-09-11T09:52:31.134 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.48e-06 2024-09-11T09:52:31.134 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.134 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.135 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:31.135 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.490749+0000", 2024-09-11T09:52:31.135 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.4569999999999999e-06 2024-09-11T09:52:31.135 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.135 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.135 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:31.135 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.490762+0000", 2024-09-11T09:52:31.135 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.2721000000000001e-05 2024-09-11T09:52:31.135 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.135 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.135 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:31.135 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.490774+0000", 2024-09-11T09:52:31.135 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1823e-05 2024-09-11T09:52:31.135 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.136 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.136 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:31.136 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.490846+0000", 2024-09-11T09:52:31.136 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.1878000000000005e-05 2024-09-11T09:52:31.136 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.136 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.136 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for peered", 2024-09-11T09:52:31.136 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.490857+0000", 2024-09-11T09:52:31.136 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1017e-05 2024-09-11T09:52:31.136 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.136 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.136 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:31.136 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.493856+0000", 2024-09-11T09:52:31.136 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00299885 2024-09-11T09:52:31.136 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.137 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.137 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:31.137 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.494020+0000", 2024-09-11T09:52:31.137 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00016436599999999999 2024-09-11T09:52:31.137 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.137 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.137 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 2", 2024-09-11T09:52:31.137 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.494186+0000", 2024-09-11T09:52:31.137 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00016562799999999999 2024-09-11T09:52:31.137 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.137 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.137 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:31.137 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.495407+0000", 2024-09-11T09:52:31.137 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0012217420000000001 2024-09-11T09:52:31.138 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.138 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.138 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:52:31.138 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.496579+0000", 2024-09-11T09:52:31.138 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001171844 2024-09-11T09:52:31.138 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.138 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.138 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:31.138 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.496620+0000", 2024-09-11T09:52:31.138 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.0479000000000003e-05 2024-09-11T09:52:31.138 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.138 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.138 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:31.138 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.496646+0000", 2024-09-11T09:52:31.138 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.6472e-05 2024-09-11T09:52:31.139 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:31.139 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:31.139 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:31.139 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.139 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.139 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4230.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-09-11T09:52:31.139 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:20.498634+0000", 2024-09-11T09:52:31.139 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 130.63300717499999, 2024-09-11T09:52:31.139 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0050648539999999997, 2024-09-11T09:52:31.139 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:31.139 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:31.139 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:31.139 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:52:31.139 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4230", 2024-09-11T09:52:31.140 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.17:0/4101109576", 2024-09-11T09:52:31.140 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 7 2024-09-11T09:52:31.140 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.140 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:31.140 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.140 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:31.140 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.498634+0000", 2024-09-11T09:52:31.140 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:31.140 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.140 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.140 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:31.140 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.498634+0000", 2024-09-11T09:52:31.140 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:31.140 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.141 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.141 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:31.141 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.498637+0000", 2024-09-11T09:52:31.141 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.1919999999999999e-06 2024-09-11T09:52:31.141 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.141 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.141 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:31.141 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.498648+0000", 2024-09-11T09:52:31.141 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1116e-05 2024-09-11T09:52:31.141 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.141 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.141 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:31.141 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.498666+0000", 2024-09-11T09:52:31.141 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.8411999999999999e-05 2024-09-11T09:52:31.141 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.142 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.142 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:31.142 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.498675+0000", 2024-09-11T09:52:31.142 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.4059999999999994e-06 2024-09-11T09:52:31.142 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.142 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.142 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:31.142 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.498745+0000", 2024-09-11T09:52:31.142 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.9764000000000005e-05 2024-09-11T09:52:31.142 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.142 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.142 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op must wait for map", 2024-09-11T09:52:31.142 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.498754+0000", 2024-09-11T09:52:31.143 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.5969999999999992e-06 2024-09-11T09:52:31.143 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.143 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.143 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:31.143 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.500385+0000", 2024-09-11T09:52:31.143 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0016308360000000001 2024-09-11T09:52:31.143 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.143 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.143 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:31.143 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.500473+0000", 2024-09-11T09:52:31.143 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.7966000000000004e-05 2024-09-11T09:52:31.143 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.143 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.144 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 1", 2024-09-11T09:52:31.144 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.500591+0000", 2024-09-11T09:52:31.144 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000117895 2024-09-11T09:52:31.144 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.144 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.144 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:31.144 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.501685+0000", 2024-09-11T09:52:31.144 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001094886 2024-09-11T09:52:31.144 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.144 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.144 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:52:31.144 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.503627+0000", 2024-09-11T09:52:31.144 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001941199 2024-09-11T09:52:31.144 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.145 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.145 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:31.145 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.503667+0000", 2024-09-11T09:52:31.145 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.0645000000000001e-05 2024-09-11T09:52:31.145 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.145 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.145 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:31.145 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.503699+0000", 2024-09-11T09:52:31.145 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.1940000000000003e-05 2024-09-11T09:52:31.145 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:31.145 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:31.145 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:31.145 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.145 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.146 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4688.0:32 2.2 2:45941fd2:::rbd_data.1250f5e25b80.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:31.146 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:22.896607+0000", 2024-09-11T09:52:31.146 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 68.235034572000004, 2024-09-11T09:52:31.146 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.038655179999999997, 2024-09-11T09:52:31.146 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:31.146 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:31.146 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:31.146 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:52:31.146 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4688", 2024-09-11T09:52:31.146 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:52:31.146 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 32 2024-09-11T09:52:31.146 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.146 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:31.146 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.146 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:31.147 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.896607+0000", 2024-09-11T09:52:31.147 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:31.147 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.147 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.147 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:31.147 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.896607+0000", 2024-09-11T09:52:31.147 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:31.147 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.147 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.147 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:31.147 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.896608+0000", 2024-09-11T09:52:31.147 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.0690000000000001e-06 2024-09-11T09:52:31.147 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.147 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.148 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:31.148 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.914461+0000", 2024-09-11T09:52:31.148 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.017852903 2024-09-11T09:52:31.148 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.148 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.148 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:31.148 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.914495+0000", 2024-09-11T09:52:31.148 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.4301000000000002e-05 2024-09-11T09:52:31.148 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.148 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.148 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:31.148 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.914522+0000", 2024-09-11T09:52:31.148 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.7127e-05 2024-09-11T09:52:31.148 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.148 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.149 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:31.149 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.914583+0000", 2024-09-11T09:52:31.149 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.0730999999999998e-05 2024-09-11T09:52:31.149 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.149 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.149 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:31.149 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.914709+0000", 2024-09-11T09:52:31.149 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00012590400000000001 2024-09-11T09:52:31.149 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.149 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.149 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 1", 2024-09-11T09:52:31.149 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.914809+0000", 2024-09-11T09:52:31.149 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.9882000000000004e-05 2024-09-11T09:52:31.149 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.150 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.150 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:31.150 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.932551+0000", 2024-09-11T09:52:31.150 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.017742048999999999 2024-09-11T09:52:31.150 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.150 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.150 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:52:31.150 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.935144+0000", 2024-09-11T09:52:31.150 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0025927630000000001 2024-09-11T09:52:31.150 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.150 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.150 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:31.150 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.935205+0000", 2024-09-11T09:52:31.150 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.1634e-05 2024-09-11T09:52:31.150 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.151 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.151 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:31.151 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.935262+0000", 2024-09-11T09:52:31.151 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.6817000000000002e-05 2024-09-11T09:52:31.151 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:31.151 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:31.151 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:31.151 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.151 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.151 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_repop(client.4688.0:40 2.7 e53/38 2:fe41411c:::rbd_object_map.1250f5e25b80:head v 53'9, mlcod=53'7)", 2024-09-11T09:52:31.151 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:22.920457+0000", 2024-09-11T09:52:31.151 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 68.211184481000004, 2024-09-11T09:52:31.151 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.011132817, 2024-09-11T09:52:31.151 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:31.151 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:31.152 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:31.152 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:31.152 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.152 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:31.152 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.920457+0000", 2024-09-11T09:52:31.152 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:31.152 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.152 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.152 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:31.152 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.920457+0000", 2024-09-11T09:52:31.152 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:31.152 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.152 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.152 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:31.153 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.920459+0000", 2024-09-11T09:52:31.153 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.6729999999999999e-06 2024-09-11T09:52:31.153 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.153 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.153 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:31.153 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.920469+0000", 2024-09-11T09:52:31.153 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.944e-06 2024-09-11T09:52:31.153 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.153 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.153 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:31.153 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.920495+0000", 2024-09-11T09:52:31.153 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.6401999999999999e-05 2024-09-11T09:52:31.153 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.153 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.154 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:31.154 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.920506+0000", 2024-09-11T09:52:31.154 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.133e-05 2024-09-11T09:52:31.154 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.154 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.154 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:31.154 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.920581+0000", 2024-09-11T09:52:31.154 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.4932999999999996e-05 2024-09-11T09:52:31.154 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.154 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.154 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:31.154 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.920612+0000", 2024-09-11T09:52:31.154 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.0537999999999998e-05 2024-09-11T09:52:31.154 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.154 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.154 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:31.155 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.931537+0000", 2024-09-11T09:52:31.155 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.010925239 2024-09-11T09:52:31.155 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.155 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.155 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:31.155 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.931590+0000", 2024-09-11T09:52:31.155 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.2757999999999999e-05 2024-09-11T09:52:31.155 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:31.155 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:31.155 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:31.155 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.155 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.155 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_repop(client.4688.0:41 2.7 e53/38 2:fe41411c:::rbd_object_map.1250f5e25b80:head v 53'10, mlcod=53'7)", 2024-09-11T09:52:31.156 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:22.921422+0000", 2024-09-11T09:52:31.156 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 68.210219065000004, 2024-09-11T09:52:31.156 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.013734942, 2024-09-11T09:52:31.156 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:31.156 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:31.156 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:31.156 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:31.156 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.156 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:31.156 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921422+0000", 2024-09-11T09:52:31.156 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:31.156 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.156 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.157 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:31.157 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921422+0000", 2024-09-11T09:52:31.157 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:31.157 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.157 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.157 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:31.157 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921426+0000", 2024-09-11T09:52:31.157 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.6150000000000001e-06 2024-09-11T09:52:31.157 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.157 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.157 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:31.157 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921435+0000", 2024-09-11T09:52:31.157 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.9840000000000007e-06 2024-09-11T09:52:31.157 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.158 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.158 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:31.158 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921444+0000", 2024-09-11T09:52:31.158 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.6000000000000007e-06 2024-09-11T09:52:31.158 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.158 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.158 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:31.158 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921450+0000", 2024-09-11T09:52:31.158 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.4690000000000002e-06 2024-09-11T09:52:31.158 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.158 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.158 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:31.158 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921511+0000", 2024-09-11T09:52:31.158 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.0779999999999997e-05 2024-09-11T09:52:31.158 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.159 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.159 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:31.159 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921531+0000", 2024-09-11T09:52:31.159 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.0302999999999999e-05 2024-09-11T09:52:31.159 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.159 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.159 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:31.159 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.935115+0000", 2024-09-11T09:52:31.159 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.013583471 2024-09-11T09:52:31.159 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.159 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.159 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:31.159 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.935157+0000", 2024-09-11T09:52:31.159 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.2719999999999998e-05 2024-09-11T09:52:31.159 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:31.160 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:31.160 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:31.160 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.160 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.160 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4688.0:42 2.3 2:c54742d8:::rbd_data.1250f5e25b80.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:31.160 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:22.932560+0000", 2024-09-11T09:52:31.160 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 68.199081321999998, 2024-09-11T09:52:31.160 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.017798198000000001, 2024-09-11T09:52:31.160 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:31.160 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:31.160 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:31.160 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:52:31.160 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4688", 2024-09-11T09:52:31.160 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:52:31.161 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 42 2024-09-11T09:52:31.161 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.161 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:31.161 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.161 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:31.161 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.932560+0000", 2024-09-11T09:52:31.161 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:31.161 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.161 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.161 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:31.161 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.932560+0000", 2024-09-11T09:52:31.161 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:31.161 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.162 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.162 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:31.162 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.932562+0000", 2024-09-11T09:52:31.162 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.17e-06 2024-09-11T09:52:31.162 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.162 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.162 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:31.162 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.938851+0000", 2024-09-11T09:52:31.162 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.006288971 2024-09-11T09:52:31.162 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.162 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.162 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:31.163 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.938865+0000", 2024-09-11T09:52:31.163 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.365e-05 2024-09-11T09:52:31.163 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.163 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.163 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:31.163 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.938871+0000", 2024-09-11T09:52:31.163 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.8850000000000004e-06 2024-09-11T09:52:31.163 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.163 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.163 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:31.163 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.938926+0000", 2024-09-11T09:52:31.163 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.4691999999999997e-05 2024-09-11T09:52:31.163 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.164 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.164 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:31.164 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.939059+0000", 2024-09-11T09:52:31.164 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00013310899999999999 2024-09-11T09:52:31.164 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.164 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.164 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 2", 2024-09-11T09:52:31.164 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.939171+0000", 2024-09-11T09:52:31.164 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000112718 2024-09-11T09:52:31.164 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.164 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.164 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:31.164 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.950262+0000", 2024-09-11T09:52:31.164 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.011091063 2024-09-11T09:52:31.165 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.165 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.165 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:52:31.165 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.950274+0000", 2024-09-11T09:52:31.165 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1963e-05 2024-09-11T09:52:31.165 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.165 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.165 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:31.165 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.950318+0000", 2024-09-11T09:52:31.165 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.3519999999999997e-05 2024-09-11T09:52:31.165 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.165 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.165 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:31.166 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.950358+0000", 2024-09-11T09:52:31.166 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.0457000000000002e-05 2024-09-11T09:52:31.166 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:31.166 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:31.166 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:31.166 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.166 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.166 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_repop(client.4694.0:34 2.7 e53/38 2:f80bfc0c:ns::rbd_data.1256a0c02879.0000000000000002:head v 53'25, mlcod=53'23)", 2024-09-11T09:52:31.166 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.094944+0000", 2024-09-11T09:52:31.166 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 68.036697930000003, 2024-09-11T09:52:31.166 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.010896785000000001, 2024-09-11T09:52:31.166 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:31.166 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:31.166 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:31.166 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:31.167 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.167 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:31.167 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.094944+0000", 2024-09-11T09:52:31.167 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:31.167 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.167 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.167 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:31.167 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.094944+0000", 2024-09-11T09:52:31.167 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:31.167 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.167 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.167 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:31.167 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.094944+0000", 2024-09-11T09:52:31.167 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.0699999999999996e-07 2024-09-11T09:52:31.167 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.168 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.168 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:31.168 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.099060+0000", 2024-09-11T09:52:31.168 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0041155580000000001 2024-09-11T09:52:31.168 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.168 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.168 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:31.168 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.099082+0000", 2024-09-11T09:52:31.168 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.2323999999999998e-05 2024-09-11T09:52:31.168 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.168 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.168 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:31.168 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.099094+0000", 2024-09-11T09:52:31.168 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1535000000000001e-05 2024-09-11T09:52:31.168 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.169 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.169 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:31.169 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.099160+0000", 2024-09-11T09:52:31.169 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.6449e-05 2024-09-11T09:52:31.169 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.169 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.169 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:31.169 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.099183+0000", 2024-09-11T09:52:31.169 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.3108e-05 2024-09-11T09:52:31.169 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.169 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.169 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:31.169 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.105776+0000", 2024-09-11T09:52:31.169 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0065922760000000002 2024-09-11T09:52:31.170 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.170 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.170 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:31.170 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.105840+0000", 2024-09-11T09:52:31.170 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.4727999999999994e-05 2024-09-11T09:52:31.170 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:31.170 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:31.170 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:31.170 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.170 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.170 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_repop(client.4694.0:35 2.7 e53/38 2:e8766325:ns::rbd_object_map.1256a0c02879:head v 53'26, mlcod=53'23)", 2024-09-11T09:52:31.170 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.100964+0000", 2024-09-11T09:52:31.170 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 68.030677401999995, 2024-09-11T09:52:31.170 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0049649359999999997, 2024-09-11T09:52:31.170 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:31.171 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:31.171 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:31.171 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:31.171 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.171 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:31.171 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.100964+0000", 2024-09-11T09:52:31.171 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:31.171 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.171 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.171 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:31.171 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.100964+0000", 2024-09-11T09:52:31.171 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:31.172 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.172 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.172 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:31.172 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.100966+0000", 2024-09-11T09:52:31.172 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.6330000000000001e-06 2024-09-11T09:52:31.172 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.172 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.172 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:31.172 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.100979+0000", 2024-09-11T09:52:31.172 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.2813000000000001e-05 2024-09-11T09:52:31.172 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.172 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.173 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:31.173 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.100992+0000", 2024-09-11T09:52:31.173 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.3414999999999999e-05 2024-09-11T09:52:31.173 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.173 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.173 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:31.173 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.101002+0000", 2024-09-11T09:52:31.173 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.0159e-05 2024-09-11T09:52:31.173 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.173 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.173 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:31.173 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.102365+0000", 2024-09-11T09:52:31.173 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001362912 2024-09-11T09:52:31.174 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.174 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.174 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:31.174 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.102382+0000", 2024-09-11T09:52:31.174 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.7433000000000001e-05 2024-09-11T09:52:31.174 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.174 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.174 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:31.174 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.105901+0000", 2024-09-11T09:52:31.174 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.003518476 2024-09-11T09:52:31.174 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.174 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.174 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:31.175 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.105929+0000", 2024-09-11T09:52:31.175 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.8095000000000001e-05 2024-09-11T09:52:31.175 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:31.175 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:31.175 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:31.175 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.175 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.175 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4694.0:41 2.2 2:5e2505c4:ns::rbd_data.1256a0c02879.0000000000000003:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:31.175 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.107511+0000", 2024-09-11T09:52:31.175 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 68.024130248999995, 2024-09-11T09:52:31.175 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.018571066000000001, 2024-09-11T09:52:31.175 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:31.175 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:31.175 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:31.176 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:52:31.176 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4694", 2024-09-11T09:52:31.176 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:52:31.176 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 41 2024-09-11T09:52:31.176 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.176 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:31.176 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.176 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:31.176 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.107511+0000", 2024-09-11T09:52:31.176 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:31.176 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.176 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.176 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:31.176 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.107511+0000", 2024-09-11T09:52:31.177 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:31.177 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.177 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.177 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:31.177 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.107513+0000", 2024-09-11T09:52:31.177 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.3489999999999999e-06 2024-09-11T09:52:31.177 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.177 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.177 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:31.177 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.112704+0000", 2024-09-11T09:52:31.177 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0051918509999999999 2024-09-11T09:52:31.177 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.178 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.178 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:31.178 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.112718+0000", 2024-09-11T09:52:31.178 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.359e-05 2024-09-11T09:52:31.178 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.178 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.178 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:31.178 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.112725+0000", 2024-09-11T09:52:31.178 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.9940000000000003e-06 2024-09-11T09:52:31.178 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.178 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.178 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:31.179 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.112788+0000", 2024-09-11T09:52:31.179 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.2447999999999996e-05 2024-09-11T09:52:31.179 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.179 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.179 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:31.179 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.112937+0000", 2024-09-11T09:52:31.179 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00014921499999999999 2024-09-11T09:52:31.179 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.179 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.179 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 1", 2024-09-11T09:52:31.179 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.113085+0000", 2024-09-11T09:52:31.179 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000148085 2024-09-11T09:52:31.179 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.180 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.180 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:52:31.180 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.124318+0000", 2024-09-11T09:52:31.180 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.011233158 2024-09-11T09:52:31.180 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.180 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.180 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:31.180 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.125957+0000", 2024-09-11T09:52:31.180 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001638701 2024-09-11T09:52:31.180 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.180 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.180 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:31.180 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.126029+0000", 2024-09-11T09:52:31.180 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.2125000000000005e-05 2024-09-11T09:52:31.181 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.181 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.181 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:31.181 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.126082+0000", 2024-09-11T09:52:31.181 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.3550000000000001e-05 2024-09-11T09:52:31.181 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:31.181 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:31.181 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:31.181 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.181 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.181 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4694.0:44 2.3 2:cd1a7d1c:ns::rbd_data.1256a0c02879.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:31.181 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.112744+0000", 2024-09-11T09:52:31.181 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 68.018897682000002, 2024-09-11T09:52:31.181 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.021672212999999999, 2024-09-11T09:52:31.182 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:31.182 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:31.182 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:31.182 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:52:31.182 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4694", 2024-09-11T09:52:31.182 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:52:31.182 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 44 2024-09-11T09:52:31.182 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.182 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:31.182 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.182 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:31.182 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.112744+0000", 2024-09-11T09:52:31.182 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:31.182 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.183 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.183 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:31.183 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.112744+0000", 2024-09-11T09:52:31.183 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:31.183 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.183 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.183 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:31.183 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.112745+0000", 2024-09-11T09:52:31.183 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.2900000000000002e-07 2024-09-11T09:52:31.183 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.183 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.183 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:31.183 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.119929+0000", 2024-09-11T09:52:31.183 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0071846230000000002 2024-09-11T09:52:31.184 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.184 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.184 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:31.184 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.119949+0000", 2024-09-11T09:52:31.184 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.9264000000000001e-05 2024-09-11T09:52:31.184 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.184 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.184 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:31.184 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.119959+0000", 2024-09-11T09:52:31.184 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.0465999999999999e-05 2024-09-11T09:52:31.184 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.184 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.184 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:31.184 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.120028+0000", 2024-09-11T09:52:31.185 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.9318000000000003e-05 2024-09-11T09:52:31.185 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.185 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.185 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:31.185 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.120160+0000", 2024-09-11T09:52:31.185 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000132015 2024-09-11T09:52:31.185 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.185 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.185 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 2", 2024-09-11T09:52:31.185 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.120260+0000", 2024-09-11T09:52:31.185 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.9397999999999999e-05 2024-09-11T09:52:31.185 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.186 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.186 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:52:31.186 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.134246+0000", 2024-09-11T09:52:31.186 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.013986721000000001 2024-09-11T09:52:31.186 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.186 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.186 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:31.186 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.134295+0000", 2024-09-11T09:52:31.186 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.8649000000000001e-05 2024-09-11T09:52:31.186 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.186 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.187 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:31.187 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.134361+0000", 2024-09-11T09:52:31.187 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.6081999999999995e-05 2024-09-11T09:52:31.187 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.187 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.187 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:31.187 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.134416+0000", 2024-09-11T09:52:31.187 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.4848000000000001e-05 2024-09-11T09:52:31.187 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:31.187 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:31.187 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:31.187 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.187 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.188 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4694.0:46 2.3 2:ce0e1469:ns::rbd_data.1256a0c02879.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:31.188 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.119993+0000", 2024-09-11T09:52:31.188 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 68.011648206000004, 2024-09-11T09:52:31.188 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.025840610999999999, 2024-09-11T09:52:31.188 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:31.188 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:31.188 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:31.188 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:52:31.188 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4694", 2024-09-11T09:52:31.188 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:52:31.188 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 46 2024-09-11T09:52:31.188 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.188 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:31.188 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.189 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:31.189 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.119993+0000", 2024-09-11T09:52:31.189 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:31.189 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.189 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.189 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:31.189 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.119993+0000", 2024-09-11T09:52:31.189 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:31.189 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.189 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.189 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:31.189 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.119995+0000", 2024-09-11T09:52:31.189 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1930000000000001e-06 2024-09-11T09:52:31.189 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.189 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.190 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:31.190 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.128458+0000", 2024-09-11T09:52:31.190 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0084632630000000004 2024-09-11T09:52:31.190 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.190 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.190 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:31.190 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.128478+0000", 2024-09-11T09:52:31.190 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.0353e-05 2024-09-11T09:52:31.190 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.190 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.190 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:31.190 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.128491+0000", 2024-09-11T09:52:31.190 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.2954e-05 2024-09-11T09:52:31.190 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.191 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.191 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:31.191 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.128558+0000", 2024-09-11T09:52:31.191 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.7076999999999995e-05 2024-09-11T09:52:31.191 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.191 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.191 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:31.191 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.128708+0000", 2024-09-11T09:52:31.191 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00014939599999999999 2024-09-11T09:52:31.191 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.191 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.191 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 2", 2024-09-11T09:52:31.191 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.128866+0000", 2024-09-11T09:52:31.191 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000158145 2024-09-11T09:52:31.192 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.192 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.192 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:31.192 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.141045+0000", 2024-09-11T09:52:31.192 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.012179331999999999 2024-09-11T09:52:31.192 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.192 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.192 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:52:31.192 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.145742+0000", 2024-09-11T09:52:31.192 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0046972050000000003 2024-09-11T09:52:31.192 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.192 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.193 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:31.193 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.145779+0000", 2024-09-11T09:52:31.193 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.6544000000000001e-05 2024-09-11T09:52:31.193 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.193 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.193 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:31.193 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.145834+0000", 2024-09-11T09:52:31.193 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.5148999999999997e-05 2024-09-11T09:52:31.193 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:31.193 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:31.193 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:31.193 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.193 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.194 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4694.0:48 2.3 2:d2090b70:ns::rbd_data.1256a0c02879.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:31.194 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.130540+0000", 2024-09-11T09:52:31.194 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 68.001101939999998, 2024-09-11T09:52:31.194 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.017167789999999999, 2024-09-11T09:52:31.194 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:31.194 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:31.194 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:31.194 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:52:31.194 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4694", 2024-09-11T09:52:31.194 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:52:31.194 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 48 2024-09-11T09:52:31.194 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.194 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:31.194 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.195 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:31.195 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.130540+0000", 2024-09-11T09:52:31.195 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:31.195 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.195 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.195 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:31.195 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.130540+0000", 2024-09-11T09:52:31.195 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:31.195 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.195 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.195 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:31.195 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.130542+0000", 2024-09-11T09:52:31.195 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.9209999999999999e-06 2024-09-11T09:52:31.195 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.196 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.196 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:31.196 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.134691+0000", 2024-09-11T09:52:31.196 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0041498739999999996 2024-09-11T09:52:31.196 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.196 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.196 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:31.196 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.134705+0000", 2024-09-11T09:52:31.196 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.3669000000000001e-05 2024-09-11T09:52:31.196 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.196 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.196 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:31.196 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.134712+0000", 2024-09-11T09:52:31.197 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.8410000000000002e-06 2024-09-11T09:52:31.197 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.197 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.197 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:31.197 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.134754+0000", 2024-09-11T09:52:31.197 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.2524999999999997e-05 2024-09-11T09:52:31.197 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.197 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.197 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:31.197 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.134875+0000", 2024-09-11T09:52:31.197 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000120332 2024-09-11T09:52:31.197 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.198 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.198 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 2", 2024-09-11T09:52:31.198 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.135005+0000", 2024-09-11T09:52:31.198 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00013024900000000001 2024-09-11T09:52:31.198 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.198 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.198 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:31.198 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.145971+0000", 2024-09-11T09:52:31.198 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.010966203000000001 2024-09-11T09:52:31.198 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.198 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.198 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:52:31.198 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.147600+0000", 2024-09-11T09:52:31.198 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001628382 2024-09-11T09:52:31.199 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.199 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.199 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:31.199 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.147635+0000", 2024-09-11T09:52:31.199 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.5420000000000003e-05 2024-09-11T09:52:31.199 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.199 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.199 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:31.199 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.147707+0000", 2024-09-11T09:52:31.199 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.2373999999999995e-05 2024-09-11T09:52:31.199 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:31.199 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:31.199 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:31.199 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.199 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.200 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_repop(client.4694.0:49 2.7 e53/38 2:e8766325:ns::rbd_object_map.1256a0c02879:head v 53'33, mlcod=53'31)", 2024-09-11T09:52:31.200 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.135391+0000", 2024-09-11T09:52:31.200 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 67.996250375000002, 2024-09-11T09:52:31.200 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0051189859999999998, 2024-09-11T09:52:31.200 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:31.200 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:31.200 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:31.200 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:31.200 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.200 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:31.200 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.135391+0000", 2024-09-11T09:52:31.200 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:31.200 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.201 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.201 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:31.201 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.135391+0000", 2024-09-11T09:52:31.201 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:31.201 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.201 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.201 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:31.201 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.135393+0000", 2024-09-11T09:52:31.201 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.4020000000000001e-06 2024-09-11T09:52:31.201 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.201 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.201 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:31.201 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.135400+0000", 2024-09-11T09:52:31.201 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.5129999999999999e-06 2024-09-11T09:52:31.202 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.202 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.202 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:31.202 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.135411+0000", 2024-09-11T09:52:31.202 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1038e-05 2024-09-11T09:52:31.202 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.202 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.202 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:31.202 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.135418+0000", 2024-09-11T09:52:31.202 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.1489999999999997e-06 2024-09-11T09:52:31.202 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.202 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.203 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:31.203 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.135470+0000", 2024-09-11T09:52:31.203 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.2021999999999998e-05 2024-09-11T09:52:31.203 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.203 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.203 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:31.203 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.135501+0000", 2024-09-11T09:52:31.203 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.0349999999999999e-05 2024-09-11T09:52:31.203 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.203 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.203 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:31.203 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.140453+0000", 2024-09-11T09:52:31.203 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0049522409999999996 2024-09-11T09:52:31.203 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.204 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.204 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:31.204 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.140510+0000", 2024-09-11T09:52:31.204 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.7271000000000001e-05 2024-09-11T09:52:31.204 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:31.204 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:31.204 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:31.204 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.204 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.204 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4694.0:54 2.3 2:c44f0253:ns::rbd_data.1256a0c02879.000000000000000b:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:31.204 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.150151+0000", 2024-09-11T09:52:31.204 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 67.981490523999994, 2024-09-11T09:52:31.204 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.015103484, 2024-09-11T09:52:31.204 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:31.205 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:31.205 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:31.205 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:52:31.205 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4694", 2024-09-11T09:52:31.205 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:52:31.205 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 54 2024-09-11T09:52:31.205 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.205 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:31.205 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.205 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:31.205 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.150151+0000", 2024-09-11T09:52:31.205 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:31.205 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.206 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.206 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:31.206 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.150151+0000", 2024-09-11T09:52:31.206 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:31.206 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.206 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.206 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:31.206 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.150152+0000", 2024-09-11T09:52:31.206 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.2550000000000001e-06 2024-09-11T09:52:31.206 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.206 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.206 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:31.206 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.154528+0000", 2024-09-11T09:52:31.207 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0043758219999999997 2024-09-11T09:52:31.207 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.207 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.207 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:31.207 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.154541+0000", 2024-09-11T09:52:31.207 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.3414999999999999e-05 2024-09-11T09:52:31.207 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.207 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.207 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:31.207 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.154551+0000", 2024-09-11T09:52:31.207 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.4350000000000003e-06 2024-09-11T09:52:31.207 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.207 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.208 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:31.208 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.154615+0000", 2024-09-11T09:52:31.208 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.3627000000000006e-05 2024-09-11T09:52:31.208 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.208 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.208 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:31.208 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.154717+0000", 2024-09-11T09:52:31.208 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000102729 2024-09-11T09:52:31.208 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.208 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.208 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 2", 2024-09-11T09:52:31.208 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.154826+0000", 2024-09-11T09:52:31.208 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000108409 2024-09-11T09:52:31.209 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.209 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.209 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:52:31.209 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.165094+0000", 2024-09-11T09:52:31.209 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.010268210999999999 2024-09-11T09:52:31.209 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.209 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.209 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:31.209 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.165138+0000", 2024-09-11T09:52:31.209 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.4078e-05 2024-09-11T09:52:31.209 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.209 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.209 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:31.209 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.165204+0000", 2024-09-11T09:52:31.210 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.5716000000000006e-05 2024-09-11T09:52:31.210 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.210 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.210 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:31.210 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.165254+0000", 2024-09-11T09:52:31.210 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.0787e-05 2024-09-11T09:52:31.210 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:31.210 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:31.210 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:31.210 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.210 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.210 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_repop(client.4694.0:55 2.7 e53/38 2:e8766325:ns::rbd_object_map.1256a0c02879:head v 53'37, mlcod=53'34)", 2024-09-11T09:52:31.210 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.150221+0000", 2024-09-11T09:52:31.211 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 67.981420658999994, 2024-09-11T09:52:31.211 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00530015, 2024-09-11T09:52:31.211 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:31.211 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:31.211 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:31.211 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:31.211 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.211 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:31.211 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.150221+0000", 2024-09-11T09:52:31.211 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:31.211 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.211 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.211 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:31.211 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.150221+0000", 2024-09-11T09:52:31.211 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:31.212 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.212 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.212 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:31.212 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.150221+0000", 2024-09-11T09:52:31.212 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.9100000000000004e-07 2024-09-11T09:52:31.212 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.212 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.212 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:31.212 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.150226+0000", 2024-09-11T09:52:31.212 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.2250000000000002e-06 2024-09-11T09:52:31.212 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.212 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.213 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:31.213 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.150251+0000", 2024-09-11T09:52:31.213 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.4987e-05 2024-09-11T09:52:31.213 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.213 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.213 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:31.213 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.150258+0000", 2024-09-11T09:52:31.213 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.0790000000000003e-06 2024-09-11T09:52:31.213 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.213 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.213 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:31.213 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.150305+0000", 2024-09-11T09:52:31.213 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.6879000000000003e-05 2024-09-11T09:52:31.213 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.214 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.214 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:31.214 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.150320+0000", 2024-09-11T09:52:31.214 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.5268999999999999e-05 2024-09-11T09:52:31.214 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.214 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.214 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:31.214 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.155493+0000", 2024-09-11T09:52:31.214 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0051731390000000002 2024-09-11T09:52:31.214 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.214 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.214 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:31.214 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.155521+0000", 2024-09-11T09:52:31.214 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.8081000000000001e-05 2024-09-11T09:52:31.215 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:31.215 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:31.215 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:31.215 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.215 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.215 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4694.0:56 2.3 2:c462f029:ns::rbd_data.1256a0c02879.000000000000000c:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:31.215 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.154576+0000", 2024-09-11T09:52:31.215 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 67.977065683000006, 2024-09-11T09:52:31.215 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.020781621, 2024-09-11T09:52:31.215 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:31.215 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:31.215 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:31.215 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:52:31.215 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4694", 2024-09-11T09:52:31.215 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:52:31.216 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 56 2024-09-11T09:52:31.216 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.216 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:31.216 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.216 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:31.216 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.154576+0000", 2024-09-11T09:52:31.216 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:31.216 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.216 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.216 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:31.216 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.154576+0000", 2024-09-11T09:52:31.216 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:31.217 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.217 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.217 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:31.217 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.154577+0000", 2024-09-11T09:52:31.217 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.2249999999999999e-06 2024-09-11T09:52:31.217 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.217 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.217 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:31.217 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.158783+0000", 2024-09-11T09:52:31.217 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0042063889999999996 2024-09-11T09:52:31.217 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.217 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.217 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:31.218 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.158803+0000", 2024-09-11T09:52:31.218 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.9576999999999999e-05 2024-09-11T09:52:31.218 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.218 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.218 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:31.218 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.158815+0000", 2024-09-11T09:52:31.218 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.2221999999999999e-05 2024-09-11T09:52:31.218 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.218 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.218 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:31.218 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.158902+0000", 2024-09-11T09:52:31.218 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.6390000000000005e-05 2024-09-11T09:52:31.218 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.218 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.218 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:31.219 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.159141+0000", 2024-09-11T09:52:31.219 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000238886 2024-09-11T09:52:31.219 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.219 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.219 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 2", 2024-09-11T09:52:31.219 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.159279+0000", 2024-09-11T09:52:31.219 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000138073 2024-09-11T09:52:31.219 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.219 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.219 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:52:31.219 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.173297+0000", 2024-09-11T09:52:31.219 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.014018113 2024-09-11T09:52:31.219 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.219 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.219 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:31.220 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.175278+0000", 2024-09-11T09:52:31.220 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0019817390000000002 2024-09-11T09:52:31.220 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.220 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.220 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:31.220 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.175316+0000", 2024-09-11T09:52:31.220 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.7829e-05 2024-09-11T09:52:31.220 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.220 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.220 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:31.220 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.175357+0000", 2024-09-11T09:52:31.220 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.1177999999999998e-05 2024-09-11T09:52:31.220 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:31.221 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:31.221 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:31.221 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.221 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.221 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4694.0:59 2.2 2:41adefc9:ns::rbd_data.1256a0c02879.000000000000000e:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:31.221 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.158843+0000", 2024-09-11T09:52:31.221 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 67.972798116999996, 2024-09-11T09:52:31.221 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.017874207, 2024-09-11T09:52:31.221 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:31.221 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:31.221 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:31.221 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:52:31.221 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4694", 2024-09-11T09:52:31.222 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:52:31.222 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 59 2024-09-11T09:52:31.222 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.222 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:31.222 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.222 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:31.222 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.158843+0000", 2024-09-11T09:52:31.222 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:31.222 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.222 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.222 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:31.222 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.158843+0000", 2024-09-11T09:52:31.222 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:31.222 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.223 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.223 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:31.223 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.158845+0000", 2024-09-11T09:52:31.223 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.7439999999999999e-06 2024-09-11T09:52:31.223 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.223 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.223 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:31.223 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.162713+0000", 2024-09-11T09:52:31.223 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0038678829999999999 2024-09-11T09:52:31.223 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.223 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.223 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:31.224 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.162728+0000", 2024-09-11T09:52:31.224 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.5114e-05 2024-09-11T09:52:31.224 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.224 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.224 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:31.224 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.162737+0000", 2024-09-11T09:52:31.224 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.1099999999999992e-06 2024-09-11T09:52:31.224 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.224 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.224 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:31.224 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.162799+0000", 2024-09-11T09:52:31.224 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.1934999999999996e-05 2024-09-11T09:52:31.224 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.224 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.225 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:31.225 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.162911+0000", 2024-09-11T09:52:31.225 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000111867 2024-09-11T09:52:31.225 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.225 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.225 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 1", 2024-09-11T09:52:31.225 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.163030+0000", 2024-09-11T09:52:31.225 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000119108 2024-09-11T09:52:31.225 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.225 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.225 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:52:31.225 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.174086+0000", 2024-09-11T09:52:31.225 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.01105543 2024-09-11T09:52:31.225 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.226 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.226 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:31.226 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.176614+0000", 2024-09-11T09:52:31.226 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0025284940000000001 2024-09-11T09:52:31.226 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.226 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.226 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:31.226 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.176672+0000", 2024-09-11T09:52:31.226 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.7892000000000001e-05 2024-09-11T09:52:31.226 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.226 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.226 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:31.226 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.176718+0000", 2024-09-11T09:52:31.227 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.5630000000000002e-05 2024-09-11T09:52:31.227 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:31.227 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:31.227 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:31.227 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.227 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.227 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.5002.0:89 2.3 2:c54742d8:::rbd_data.1250f5e25b80.0000000000000006:head [write 131072~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-09-11T09:52:31.227 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:38.546016+0000", 2024-09-11T09:52:31.227 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 52.585625571000001, 2024-09-11T09:52:31.227 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0049098550000000003, 2024-09-11T09:52:31.227 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:31.227 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:31.227 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:31.227 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:52:31.227 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.5002", 2024-09-11T09:52:31.228 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/2204440692", 2024-09-11T09:52:31.228 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 89 2024-09-11T09:52:31.228 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.228 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:31.228 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.228 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:31.228 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.546016+0000", 2024-09-11T09:52:31.228 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:31.228 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.228 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.228 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:31.228 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.546016+0000", 2024-09-11T09:52:31.228 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:31.229 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.229 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.229 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:31.229 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.546017+0000", 2024-09-11T09:52:31.229 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.3399999999999998e-07 2024-09-11T09:52:31.229 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.229 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.229 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:31.229 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.546839+0000", 2024-09-11T09:52:31.229 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00082222800000000004 2024-09-11T09:52:31.229 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.229 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.229 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:31.229 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.546851+0000", 2024-09-11T09:52:31.230 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.2075999999999999e-05 2024-09-11T09:52:31.230 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.230 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.230 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:31.230 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.546856+0000", 2024-09-11T09:52:31.230 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.4009999999999996e-06 2024-09-11T09:52:31.230 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.230 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.230 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:31.230 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.546897+0000", 2024-09-11T09:52:31.230 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.0113e-05 2024-09-11T09:52:31.230 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.230 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.231 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:31.231 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.546954+0000", 2024-09-11T09:52:31.231 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.7825000000000003e-05 2024-09-11T09:52:31.231 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.231 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.231 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 2", 2024-09-11T09:52:31.231 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.547055+0000", 2024-09-11T09:52:31.231 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000100269 2024-09-11T09:52:31.231 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.231 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.231 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:31.231 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.549675+0000", 2024-09-11T09:52:31.231 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0026206609999999998 2024-09-11T09:52:31.232 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.232 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.232 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:52:31.232 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.550858+0000", 2024-09-11T09:52:31.232 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0011825270000000001 2024-09-11T09:52:31.232 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.232 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.232 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:31.232 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.550894+0000", 2024-09-11T09:52:31.232 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.5896999999999999e-05 2024-09-11T09:52:31.232 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.232 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.233 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:31.233 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.550926+0000", 2024-09-11T09:52:31.233 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.2023999999999997e-05 2024-09-11T09:52:31.233 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:31.233 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:31.233 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:31.233 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.233 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.233 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.5002.0:91 2.3 2:c54742d8:::rbd_data.1250f5e25b80.0000000000000006:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-09-11T09:52:31.233 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:38.553743+0000", 2024-09-11T09:52:31.233 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 52.577898527999999, 2024-09-11T09:52:31.233 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0051633629999999998, 2024-09-11T09:52:31.234 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:31.234 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:31.234 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:31.234 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:52:31.234 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.5002", 2024-09-11T09:52:31.234 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/2204440692", 2024-09-11T09:52:31.234 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 91 2024-09-11T09:52:31.234 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.234 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:31.234 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.234 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:31.234 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.553743+0000", 2024-09-11T09:52:31.234 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:31.234 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.234 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.235 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:31.235 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.553743+0000", 2024-09-11T09:52:31.235 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:31.235 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.235 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.235 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:31.235 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.553745+0000", 2024-09-11T09:52:31.235 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.5120000000000001e-06 2024-09-11T09:52:31.235 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.235 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.235 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:31.235 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.554588+0000", 2024-09-11T09:52:31.235 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00084352299999999999 2024-09-11T09:52:31.235 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.236 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.236 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:31.236 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.554604+0000", 2024-09-11T09:52:31.236 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.643e-05 2024-09-11T09:52:31.236 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.236 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.236 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:31.236 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.554613+0000", 2024-09-11T09:52:31.236 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.5669999999999995e-06 2024-09-11T09:52:31.236 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.236 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.236 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:31.236 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.554662+0000", 2024-09-11T09:52:31.237 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.8510000000000001e-05 2024-09-11T09:52:31.237 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.237 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.237 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:31.237 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.554732+0000", 2024-09-11T09:52:31.237 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.0297999999999997e-05 2024-09-11T09:52:31.237 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.237 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.237 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 2", 2024-09-11T09:52:31.237 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.554870+0000", 2024-09-11T09:52:31.237 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00013856599999999999 2024-09-11T09:52:31.237 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.237 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.238 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:31.238 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.557875+0000", 2024-09-11T09:52:31.238 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0030042760000000002 2024-09-11T09:52:31.238 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.238 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.238 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:52:31.238 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.558835+0000", 2024-09-11T09:52:31.238 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00096057900000000003 2024-09-11T09:52:31.238 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.238 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.238 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:31.238 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.558874+0000", 2024-09-11T09:52:31.238 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.8992999999999997e-05 2024-09-11T09:52:31.238 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.239 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.239 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:31.239 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.558906+0000", 2024-09-11T09:52:31.239 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.2109e-05 2024-09-11T09:52:31.239 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:31.239 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:31.239 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:31.239 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.239 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.239 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.5002.0:93 2.3 2:c54742d8:::rbd_data.1250f5e25b80.0000000000000006:head [write 2228224~917504] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e61)", 2024-09-11T09:52:31.239 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:38.562067+0000", 2024-09-11T09:52:31.239 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 52.569574932000002, 2024-09-11T09:52:31.239 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0050551099999999998, 2024-09-11T09:52:31.240 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:31.240 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:31.240 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:31.240 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:52:31.240 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.5002", 2024-09-11T09:52:31.240 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/2204440692", 2024-09-11T09:52:31.240 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 93 2024-09-11T09:52:31.240 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.240 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:31.240 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.240 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:31.240 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.562067+0000", 2024-09-11T09:52:31.240 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:31.240 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.240 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.241 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:31.241 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.562067+0000", 2024-09-11T09:52:31.241 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:31.241 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.241 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.241 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:31.241 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.562068+0000", 2024-09-11T09:52:31.241 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.0669999999999999e-06 2024-09-11T09:52:31.241 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.241 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.241 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:31.241 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.563170+0000", 2024-09-11T09:52:31.241 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00110256 2024-09-11T09:52:31.241 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.242 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.242 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:31.242 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.563185+0000", 2024-09-11T09:52:31.242 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.4873e-05 2024-09-11T09:52:31.242 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.242 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.242 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:31.242 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.563193+0000", 2024-09-11T09:52:31.242 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.1319999999999994e-06 2024-09-11T09:52:31.242 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.242 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.242 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:31.242 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.563238+0000", 2024-09-11T09:52:31.242 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.4773e-05 2024-09-11T09:52:31.242 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.243 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.243 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:31.243 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.563290+0000", 2024-09-11T09:52:31.243 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.1690000000000001e-05 2024-09-11T09:52:31.243 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.243 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.243 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 2", 2024-09-11T09:52:31.243 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.563365+0000", 2024-09-11T09:52:31.243 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.5023000000000003e-05 2024-09-11T09:52:31.243 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.243 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.243 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:31.243 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.566552+0000", 2024-09-11T09:52:31.244 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0031869960000000001 2024-09-11T09:52:31.244 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.244 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.244 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:52:31.244 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.567048+0000", 2024-09-11T09:52:31.244 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00049605099999999996 2024-09-11T09:52:31.244 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.244 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.244 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:31.244 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.567095+0000", 2024-09-11T09:52:31.244 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.6823999999999998e-05 2024-09-11T09:52:31.244 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.244 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.244 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:31.245 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.567122+0000", 2024-09-11T09:52:31.245 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.7121e-05 2024-09-11T09:52:31.245 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:31.245 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:31.245 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:31.245 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:31.245 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:31.245 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-11T09:52:31.246 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-11T09:52:31.247 INFO:tasks.ceph.osd.2.smithi017.stderr:2024-09-11T09:52:31.247+0000 7fd20b170640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31672) UID: 0 2024-09-11T09:52:31.347 INFO:tasks.ceph.osd.2.smithi017.stderr:2024-09-11T09:52:31.348+0000 7fd20b170640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31672) UID: 0 2024-09-11T09:52:31.349 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-11T09:52:31.349 INFO:teuthology.orchestra.run.smithi096.stdout: "ops": [], 2024-09-11T09:52:31.349 INFO:teuthology.orchestra.run.smithi096.stdout: "num_ops": 0 2024-09-11T09:52:31.349 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-11T09:52:31.358 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-11T09:52:31.448 INFO:tasks.ceph.osd.2.smithi017.stderr:2024-09-11T09:52:31.449+0000 7fd20b170640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31672) UID: 0 2024-09-11T09:52:31.461 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-11T09:52:31.461 INFO:teuthology.orchestra.run.smithi096.stdout: "ops": [], 2024-09-11T09:52:31.461 INFO:teuthology.orchestra.run.smithi096.stdout: "complaint_time": 30, 2024-09-11T09:52:31.461 INFO:teuthology.orchestra.run.smithi096.stdout: "num_blocked_ops": 0 2024-09-11T09:52:31.461 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-11T09:52:31.470 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-11T09:52:31.549 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:52:31.549+0000 7f3a7b90a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 42475) UID: 0 2024-09-11T09:52:31.575 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-11T09:52:31.575 INFO:teuthology.orchestra.run.smithi096.stdout: "size": 20, 2024-09-11T09:52:31.576 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 600, 2024-09-11T09:52:31.576 INFO:teuthology.orchestra.run.smithi096.stdout: "ops": [ 2024-09-11T09:52:31.576 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.576 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "MOSDPGPush(2.4 82/81 [PushOp(2:3bfc818c:ns::rbd_data.1256a0c02879.0000000000000001:head, version: 53'14, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:3bfc818c:ns::rbd_data.1256a0c02879.0000000000000001:head@53'14, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-11T09:52:31.576 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:52:08.947926+0000", 2024-09-11T09:52:31.576 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 22.626838203999998, 2024-09-11T09:52:31.576 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0020846179999999999, 2024-09-11T09:52:31.576 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:31.576 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:31.576 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "started", 2024-09-11T09:52:31.576 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:31.576 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.577 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:31.577 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.947926+0000", 2024-09-11T09:52:31.577 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:31.577 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.577 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.577 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:31.577 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.947926+0000", 2024-09-11T09:52:31.577 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:31.577 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.577 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.577 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:31.577 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.947927+0000", 2024-09-11T09:52:31.578 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.9599999999999999e-07 2024-09-11T09:52:31.578 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.578 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.578 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:31.578 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.949094+0000", 2024-09-11T09:52:31.578 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001167944 2024-09-11T09:52:31.578 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.578 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.578 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:31.578 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.949110+0000", 2024-09-11T09:52:31.578 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.5650000000000001e-05 2024-09-11T09:52:31.578 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.579 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.579 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:31.579 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.949138+0000", 2024-09-11T09:52:31.579 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.7515e-05 2024-09-11T09:52:31.579 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.579 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.579 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:31.579 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.949196+0000", 2024-09-11T09:52:31.579 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.8272e-05 2024-09-11T09:52:31.580 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.580 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.580 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:31.580 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.949204+0000", 2024-09-11T09:52:31.580 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.2150000000000001e-06 2024-09-11T09:52:31.580 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.580 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.580 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:31.580 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.950011+0000", 2024-09-11T09:52:31.580 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00080652600000000001 2024-09-11T09:52:31.580 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:31.580 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:31.580 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:31.581 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.581 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.581 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "MOSDPGPush(2.4 82/81 [PushOp(2:3bfc818c:ns::rbd_data.1256a0c02879.0000000000000001:head, version: 53'14, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:3bfc818c:ns::rbd_data.1256a0c02879.0000000000000001:head@53'14, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3145728, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-11T09:52:31.581 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:52:08.955761+0000", 2024-09-11T09:52:31.581 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 22.619003529, 2024-09-11T09:52:31.581 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0019711659999999999, 2024-09-11T09:52:31.581 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:31.581 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:31.581 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "started", 2024-09-11T09:52:31.581 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:31.581 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.581 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:31.581 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.955761+0000", 2024-09-11T09:52:31.582 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:31.582 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.582 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.582 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:31.582 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.955761+0000", 2024-09-11T09:52:31.582 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:31.582 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.582 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.582 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:31.582 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.955761+0000", 2024-09-11T09:52:31.582 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.2399999999999997e-07 2024-09-11T09:52:31.582 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.582 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.583 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:31.583 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.956708+0000", 2024-09-11T09:52:31.583 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00094629899999999995 2024-09-11T09:52:31.583 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.583 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.583 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:31.583 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.956731+0000", 2024-09-11T09:52:31.583 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.3175000000000001e-05 2024-09-11T09:52:31.583 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.583 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.583 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:31.583 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.956750+0000", 2024-09-11T09:52:31.584 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.8859e-05 2024-09-11T09:52:31.584 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.584 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.584 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:31.584 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.956828+0000", 2024-09-11T09:52:31.584 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.8567999999999997e-05 2024-09-11T09:52:31.584 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.584 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.584 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:31.585 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.956841+0000", 2024-09-11T09:52:31.585 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.2489e-05 2024-09-11T09:52:31.585 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.585 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.585 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:31.585 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.957732+0000", 2024-09-11T09:52:31.585 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00089105200000000003 2024-09-11T09:52:31.585 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:31.585 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:31.585 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:31.585 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.585 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.585 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "MOSDPGPush(2.4 82/81 [PushOp(2:3bfc818c:ns::rbd_data.1256a0c02879.0000000000000001:head, version: 53'14, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:3bfc818c:ns::rbd_data.1256a0c02879.0000000000000001:head@53'14, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 4194304, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3145728, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-11T09:52:31.586 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:52:08.959483+0000", 2024-09-11T09:52:31.586 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 22.615280998999999, 2024-09-11T09:52:31.586 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0021404520000000002, 2024-09-11T09:52:31.586 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:31.586 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:31.586 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "started", 2024-09-11T09:52:31.586 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:31.586 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.586 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:31.586 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.959483+0000", 2024-09-11T09:52:31.587 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:31.587 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.587 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.587 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:31.587 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.959483+0000", 2024-09-11T09:52:31.587 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:31.587 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.587 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.587 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:31.587 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.959484+0000", 2024-09-11T09:52:31.588 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.4900000000000005e-07 2024-09-11T09:52:31.588 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.588 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.588 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:31.588 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.960450+0000", 2024-09-11T09:52:31.588 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000966141 2024-09-11T09:52:31.588 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.588 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.588 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:31.588 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.960466+0000", 2024-09-11T09:52:31.588 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.6302000000000001e-05 2024-09-11T09:52:31.588 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.588 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.589 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:31.589 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.960477+0000", 2024-09-11T09:52:31.589 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.0406e-05 2024-09-11T09:52:31.589 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.589 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.589 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:31.589 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.960555+0000", 2024-09-11T09:52:31.589 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.7837999999999996e-05 2024-09-11T09:52:31.589 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.589 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.589 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:31.589 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.960568+0000", 2024-09-11T09:52:31.589 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.2904e-05 2024-09-11T09:52:31.590 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.590 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.590 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:31.590 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.961624+0000", 2024-09-11T09:52:31.590 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0010561119999999999 2024-09-11T09:52:31.590 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:31.590 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:31.590 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:31.590 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.590 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.590 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "MOSDPGPush(2.4 82/81 [PushOp(2:2367385a:::rbd_data.1250f5e25b80.0000000000000002:6, version: 79'169, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 1, recovery_info: ObjectRecoveryInfo(2:2367385a:::rbd_data.1250f5e25b80.0000000000000002:6@79'169, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 6=[]:{6: [6, 5, 3]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-11T09:52:31.590 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:52:08.964132+0000", 2024-09-11T09:52:31.590 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 22.610632517999999, 2024-09-11T09:52:31.590 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0021074240000000001, 2024-09-11T09:52:31.591 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:31.591 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:31.591 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "started", 2024-09-11T09:52:31.591 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:31.591 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.591 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:31.591 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.964132+0000", 2024-09-11T09:52:31.591 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:31.591 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.591 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.591 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:31.591 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.964132+0000", 2024-09-11T09:52:31.591 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:31.591 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.592 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.592 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:31.592 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.964133+0000", 2024-09-11T09:52:31.592 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.02e-07 2024-09-11T09:52:31.592 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.592 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.592 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:31.592 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.965104+0000", 2024-09-11T09:52:31.592 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000971849 2024-09-11T09:52:31.592 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.592 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.592 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:31.592 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.965122+0000", 2024-09-11T09:52:31.593 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.7706999999999998e-05 2024-09-11T09:52:31.593 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.593 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.593 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:31.593 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.965134+0000", 2024-09-11T09:52:31.593 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1371e-05 2024-09-11T09:52:31.593 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.593 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.593 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:31.593 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.965221+0000", 2024-09-11T09:52:31.593 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.7037e-05 2024-09-11T09:52:31.593 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.594 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.594 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:31.594 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.965236+0000", 2024-09-11T09:52:31.594 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.5724999999999999e-05 2024-09-11T09:52:31.594 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.594 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.594 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:31.594 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.966239+0000", 2024-09-11T09:52:31.594 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0010028330000000001 2024-09-11T09:52:31.594 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:31.594 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:31.594 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:31.595 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.595 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.595 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "MOSDPGPush(2.4 82/81 [PushOp(2:2367385a:::rbd_data.1250f5e25b80.0000000000000002:6, version: 79'169, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:2367385a:::rbd_data.1250f5e25b80.0000000000000002:6@79'169, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 6=[]:{6: [6, 5, 3]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-11T09:52:31.595 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:52:08.968106+0000", 2024-09-11T09:52:31.595 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 22.606658051, 2024-09-11T09:52:31.595 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.002036936, 2024-09-11T09:52:31.595 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:31.595 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:31.595 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "started", 2024-09-11T09:52:31.595 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:31.595 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.595 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:31.595 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.968106+0000", 2024-09-11T09:52:31.595 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:31.596 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.596 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.596 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:31.596 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.968106+0000", 2024-09-11T09:52:31.596 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:31.596 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.596 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.596 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:31.596 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.968107+0000", 2024-09-11T09:52:31.596 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.0100000000000001e-06 2024-09-11T09:52:31.596 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.596 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.596 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:31.597 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.969086+0000", 2024-09-11T09:52:31.597 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00097889099999999996 2024-09-11T09:52:31.597 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.597 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.597 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:31.597 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.969104+0000", 2024-09-11T09:52:31.597 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.7940000000000001e-05 2024-09-11T09:52:31.597 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.597 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.597 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:31.597 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.969117+0000", 2024-09-11T09:52:31.597 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.3207000000000001e-05 2024-09-11T09:52:31.598 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.598 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.598 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:31.598 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.969192+0000", 2024-09-11T09:52:31.598 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.4419999999999996e-05 2024-09-11T09:52:31.598 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.598 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.598 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:31.598 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.969201+0000", 2024-09-11T09:52:31.598 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.4960000000000002e-06 2024-09-11T09:52:31.598 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.598 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.598 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:31.599 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.970143+0000", 2024-09-11T09:52:31.599 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00094197200000000001 2024-09-11T09:52:31.599 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:31.599 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:31.599 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:31.599 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.599 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.599 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "MOSDPGPush(2.4 82/81 [PushOp(2:2367385a:::rbd_data.1250f5e25b80.0000000000000002:6, version: 79'169, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:2367385a:::rbd_data.1250f5e25b80.0000000000000002:6@79'169, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 6=[]:{6: [6, 5, 3]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 4194304, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3145728, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-11T09:52:31.599 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:52:08.975206+0000", 2024-09-11T09:52:31.599 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 22.599558342999998, 2024-09-11T09:52:31.599 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0020632770000000001, 2024-09-11T09:52:31.599 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:31.600 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:31.600 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "started", 2024-09-11T09:52:31.600 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:31.600 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.600 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:31.600 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.975206+0000", 2024-09-11T09:52:31.600 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:31.600 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.600 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.600 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:31.600 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.975206+0000", 2024-09-11T09:52:31.600 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:31.600 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.600 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.601 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:31.601 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.975207+0000", 2024-09-11T09:52:31.601 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.2399999999999998e-07 2024-09-11T09:52:31.601 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.601 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.601 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:31.601 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.976173+0000", 2024-09-11T09:52:31.601 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00096649400000000003 2024-09-11T09:52:31.601 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.601 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.601 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:31.602 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.976189+0000", 2024-09-11T09:52:31.602 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.6223e-05 2024-09-11T09:52:31.602 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.602 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.602 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:31.602 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.976200+0000", 2024-09-11T09:52:31.602 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.0729e-05 2024-09-11T09:52:31.602 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.602 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.602 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:31.602 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.976265+0000", 2024-09-11T09:52:31.602 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.5326000000000004e-05 2024-09-11T09:52:31.602 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.603 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.603 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:31.603 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.976274+0000", 2024-09-11T09:52:31.603 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.8140000000000007e-06 2024-09-11T09:52:31.603 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.603 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.603 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:31.603 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.977269+0000", 2024-09-11T09:52:31.603 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00099506700000000009 2024-09-11T09:52:31.603 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:31.603 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:31.603 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:31.603 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.603 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.604 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "MOSDPGPush(2.4 82/81 [PushOp(2:26d5c2f0:::rbd_data.1250f5e25b80.000000000000000b:6, version: 79'171, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 1, recovery_info: ObjectRecoveryInfo(2:26d5c2f0:::rbd_data.1250f5e25b80.000000000000000b:6@79'171, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 6=[]:{6: [6, 5, 3]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-11T09:52:31.604 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:52:08.979123+0000", 2024-09-11T09:52:31.604 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 22.595640914000001, 2024-09-11T09:52:31.604 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0019749350000000001, 2024-09-11T09:52:31.604 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:31.604 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:31.604 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "started", 2024-09-11T09:52:31.604 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:31.604 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.604 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:31.604 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.979123+0000", 2024-09-11T09:52:31.605 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:31.605 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.605 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.605 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:31.605 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.979123+0000", 2024-09-11T09:52:31.605 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:31.605 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.605 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.605 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:31.605 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.979124+0000", 2024-09-11T09:52:31.606 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.5799999999999999e-07 2024-09-11T09:52:31.606 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.606 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.606 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:31.606 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.980088+0000", 2024-09-11T09:52:31.606 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00096377699999999997 2024-09-11T09:52:31.606 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.606 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.606 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:31.606 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.980106+0000", 2024-09-11T09:52:31.606 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.7881e-05 2024-09-11T09:52:31.606 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.607 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.607 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:31.607 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.980117+0000", 2024-09-11T09:52:31.607 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1681e-05 2024-09-11T09:52:31.607 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.607 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.608 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:31.608 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.980189+0000", 2024-09-11T09:52:31.608 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.1365999999999994e-05 2024-09-11T09:52:31.608 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.608 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.608 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:31.608 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.980198+0000", 2024-09-11T09:52:31.608 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.3300000000000005e-06 2024-09-11T09:52:31.608 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.608 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.608 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:31.608 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.981098+0000", 2024-09-11T09:52:31.609 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00090024199999999999 2024-09-11T09:52:31.609 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:31.609 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:31.609 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:31.609 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.609 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.609 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "MOSDPGPush(2.4 82/81 [PushOp(2:26d5c2f0:::rbd_data.1250f5e25b80.000000000000000b:6, version: 79'171, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:26d5c2f0:::rbd_data.1250f5e25b80.000000000000000b:6@79'171, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 6=[]:{6: [6, 5, 3]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-11T09:52:31.609 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:52:08.982550+0000", 2024-09-11T09:52:31.609 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 22.592214468999998, 2024-09-11T09:52:31.609 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0020071540000000001, 2024-09-11T09:52:31.609 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:31.609 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:31.609 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "started", 2024-09-11T09:52:31.609 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:31.610 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.610 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:31.610 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.982550+0000", 2024-09-11T09:52:31.610 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:31.610 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.610 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.610 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:31.610 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.982550+0000", 2024-09-11T09:52:31.610 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:31.610 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.610 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.610 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:31.610 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.982550+0000", 2024-09-11T09:52:31.611 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.89e-07 2024-09-11T09:52:31.611 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.611 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.611 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:31.611 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.983523+0000", 2024-09-11T09:52:31.611 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00097246400000000003 2024-09-11T09:52:31.611 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.611 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.611 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:31.611 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.983540+0000", 2024-09-11T09:52:31.611 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.7388999999999999e-05 2024-09-11T09:52:31.611 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.611 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.611 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:31.612 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.983553+0000", 2024-09-11T09:52:31.612 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.2876e-05 2024-09-11T09:52:31.612 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.612 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.612 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:31.612 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.983623+0000", 2024-09-11T09:52:31.612 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.0199999999999999e-05 2024-09-11T09:52:31.612 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.612 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.612 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:31.612 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.983632+0000", 2024-09-11T09:52:31.612 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.8859999999999993e-06 2024-09-11T09:52:31.612 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.612 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.612 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:31.613 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.984557+0000", 2024-09-11T09:52:31.613 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00092484999999999996 2024-09-11T09:52:31.613 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:31.613 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:31.613 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:31.613 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.613 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.613 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "MOSDPGPush(2.4 82/81 [PushOp(2:26d5c2f0:::rbd_data.1250f5e25b80.000000000000000b:6, version: 79'171, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:26d5c2f0:::rbd_data.1250f5e25b80.000000000000000b:6@79'171, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 6=[]:{6: [6, 5, 3]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 4194304, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3145728, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-11T09:52:31.613 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:52:08.989143+0000", 2024-09-11T09:52:31.613 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 22.585620777999999, 2024-09-11T09:52:31.613 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0022141589999999998, 2024-09-11T09:52:31.613 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:31.614 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:31.614 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "started", 2024-09-11T09:52:31.614 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:31.614 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.614 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:31.614 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.989143+0000", 2024-09-11T09:52:31.614 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:31.614 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.614 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.614 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:31.614 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.989143+0000", 2024-09-11T09:52:31.614 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:31.614 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.614 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.615 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:31.615 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.989144+0000", 2024-09-11T09:52:31.615 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.1699999999999997e-07 2024-09-11T09:52:31.615 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.615 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.615 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:31.615 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.990124+0000", 2024-09-11T09:52:31.615 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00097941400000000007 2024-09-11T09:52:31.615 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.615 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.615 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:31.615 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.990140+0000", 2024-09-11T09:52:31.615 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.6583e-05 2024-09-11T09:52:31.615 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.615 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.616 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:31.616 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.990152+0000", 2024-09-11T09:52:31.616 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1331999999999999e-05 2024-09-11T09:52:31.616 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.616 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.616 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:31.616 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.990220+0000", 2024-09-11T09:52:31.616 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.8777999999999998e-05 2024-09-11T09:52:31.616 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.616 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.616 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:31.616 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.990229+0000", 2024-09-11T09:52:31.616 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.9590000000000001e-06 2024-09-11T09:52:31.616 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.617 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.617 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:31.617 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.991358+0000", 2024-09-11T09:52:31.617 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0011283759999999999 2024-09-11T09:52:31.617 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:31.617 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:31.617 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:31.617 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.617 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.617 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "MOSDPGPush(2.4 82/81 [PushOp(2:239e915b:::rbd_data.1250f5e25b80.000000000000000c:6, version: 79'173, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 1, recovery_info: ObjectRecoveryInfo(2:239e915b:::rbd_data.1250f5e25b80.000000000000000c:6@79'173, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 6=[]:{6: [6, 5, 3]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-11T09:52:31.617 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:52:08.994080+0000", 2024-09-11T09:52:31.617 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 22.580684641000001, 2024-09-11T09:52:31.617 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0019554189999999999, 2024-09-11T09:52:31.617 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:31.618 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:31.618 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "started", 2024-09-11T09:52:31.618 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:31.618 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.618 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:31.618 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.994080+0000", 2024-09-11T09:52:31.618 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:31.618 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.618 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.618 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:31.618 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.994080+0000", 2024-09-11T09:52:31.618 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:31.618 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.618 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.618 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:31.619 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.994080+0000", 2024-09-11T09:52:31.619 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.7599999999999997e-07 2024-09-11T09:52:31.619 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.619 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.619 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:31.619 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.995065+0000", 2024-09-11T09:52:31.619 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00098436199999999991 2024-09-11T09:52:31.619 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.619 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.619 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:31.619 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.995080+0000", 2024-09-11T09:52:31.619 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.5119e-05 2024-09-11T09:52:31.619 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.619 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.619 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:31.620 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.995090+0000", 2024-09-11T09:52:31.620 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.0715e-05 2024-09-11T09:52:31.620 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.620 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.620 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:31.620 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.995152+0000", 2024-09-11T09:52:31.620 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.2089999999999997e-05 2024-09-11T09:52:31.620 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.620 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.620 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:31.620 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.995159+0000", 2024-09-11T09:52:31.620 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.9210000000000003e-06 2024-09-11T09:52:31.620 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.620 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.621 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:31.621 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.996035+0000", 2024-09-11T09:52:31.621 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00087563599999999995 2024-09-11T09:52:31.621 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:31.621 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:31.621 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:31.621 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.621 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.621 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "MOSDPGPush(2.4 82/81 [PushOp(2:239e915b:::rbd_data.1250f5e25b80.000000000000000c:6, version: 79'173, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:239e915b:::rbd_data.1250f5e25b80.000000000000000c:6@79'173, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 6=[]:{6: [6, 5, 3]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 4194304, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3145728, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-11T09:52:31.621 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:52:09.005094+0000", 2024-09-11T09:52:31.621 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 22.569669780000002, 2024-09-11T09:52:31.621 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0024127419999999998, 2024-09-11T09:52:31.621 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:31.621 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:31.622 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "started", 2024-09-11T09:52:31.622 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:31.622 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.622 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:31.622 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.005094+0000", 2024-09-11T09:52:31.622 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:31.622 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.622 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.622 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:31.622 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.005094+0000", 2024-09-11T09:52:31.622 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:31.622 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.622 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.623 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:31.623 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.005095+0000", 2024-09-11T09:52:31.623 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.8999999999999997e-07 2024-09-11T09:52:31.623 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.623 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.623 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:31.623 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.006059+0000", 2024-09-11T09:52:31.623 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00096429100000000004 2024-09-11T09:52:31.623 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.623 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.623 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:31.623 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.006075+0000", 2024-09-11T09:52:31.623 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.6036e-05 2024-09-11T09:52:31.623 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.623 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.624 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:31.624 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.006088+0000", 2024-09-11T09:52:31.624 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.2557e-05 2024-09-11T09:52:31.624 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.624 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.624 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:31.624 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.006167+0000", 2024-09-11T09:52:31.624 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.8843000000000004e-05 2024-09-11T09:52:31.624 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.624 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.624 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:31.624 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.006182+0000", 2024-09-11T09:52:31.624 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.4968e-05 2024-09-11T09:52:31.624 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.625 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.625 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:31.625 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.007507+0000", 2024-09-11T09:52:31.625 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001325557 2024-09-11T09:52:31.625 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:31.625 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:31.625 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:31.625 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.625 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.625 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "MOSDPGPush(2.4 82/81 [PushOp(2:2167012f:::rbd_data.1250f5e25b80.000000000000000d:6, version: 79'175, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:2167012f:::rbd_data.1250f5e25b80.000000000000000d:6@79'175, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 6=[]:{6: [6, 5, 3]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-11T09:52:31.625 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:52:09.013431+0000", 2024-09-11T09:52:31.625 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 22.561333445999999, 2024-09-11T09:52:31.625 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.002012247, 2024-09-11T09:52:31.626 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:31.626 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:31.626 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "started", 2024-09-11T09:52:31.626 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:31.626 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.626 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:31.626 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.013431+0000", 2024-09-11T09:52:31.626 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:31.626 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.626 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.626 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:31.626 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.013431+0000", 2024-09-11T09:52:31.626 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:31.626 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.627 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.627 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:31.627 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.013432+0000", 2024-09-11T09:52:31.627 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.1900000000000001e-07 2024-09-11T09:52:31.627 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.627 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.627 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:31.627 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.014359+0000", 2024-09-11T09:52:31.627 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00092687900000000003 2024-09-11T09:52:31.627 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.627 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.627 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:31.627 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.014377+0000", 2024-09-11T09:52:31.628 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.8085e-05 2024-09-11T09:52:31.628 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.628 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.628 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:31.628 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.014391+0000", 2024-09-11T09:52:31.628 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.4099e-05 2024-09-11T09:52:31.628 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.628 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.628 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:31.628 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.014476+0000", 2024-09-11T09:52:31.628 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.4827999999999995e-05 2024-09-11T09:52:31.628 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.628 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.628 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:31.629 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.014489+0000", 2024-09-11T09:52:31.629 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.3169e-05 2024-09-11T09:52:31.629 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.629 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.629 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:31.629 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.015443+0000", 2024-09-11T09:52:31.629 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00095436800000000003 2024-09-11T09:52:31.629 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:31.629 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:31.629 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:31.629 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.629 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.629 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "MOSDPGPush(2.4 82/81 [PushOp(2:2167012f:::rbd_data.1250f5e25b80.000000000000000d:6, version: 79'175, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:2167012f:::rbd_data.1250f5e25b80.000000000000000d:6@79'175, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 6=[]:{6: [6, 5, 3]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3145728, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-11T09:52:31.629 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:52:09.017158+0000", 2024-09-11T09:52:31.630 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 22.557606457999999, 2024-09-11T09:52:31.630 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0020239889999999999, 2024-09-11T09:52:31.630 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:31.630 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:31.630 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "started", 2024-09-11T09:52:31.630 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:31.630 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.630 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:31.630 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.017158+0000", 2024-09-11T09:52:31.630 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:31.630 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.630 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.630 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:31.630 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.017158+0000", 2024-09-11T09:52:31.631 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:31.631 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.631 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.631 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:31.631 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.017158+0000", 2024-09-11T09:52:31.631 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.7000000000000004e-07 2024-09-11T09:52:31.631 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.631 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.631 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:31.631 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.018136+0000", 2024-09-11T09:52:31.631 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00097795 2024-09-11T09:52:31.631 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.631 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.631 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:31.632 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.018152+0000", 2024-09-11T09:52:31.632 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.5208000000000001e-05 2024-09-11T09:52:31.632 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.632 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.632 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:31.632 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.018164+0000", 2024-09-11T09:52:31.632 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.2157000000000001e-05 2024-09-11T09:52:31.632 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.632 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.632 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:31.632 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.018245+0000", 2024-09-11T09:52:31.632 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.1284000000000003e-05 2024-09-11T09:52:31.632 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.632 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.633 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:31.633 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.018257+0000", 2024-09-11T09:52:31.633 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.2160999999999999e-05 2024-09-11T09:52:31.633 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.633 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.633 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:31.633 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.019182+0000", 2024-09-11T09:52:31.633 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00092455900000000004 2024-09-11T09:52:31.633 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:31.633 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:31.633 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:31.633 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.633 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.633 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "MOSDPGPush(2.4 82/81 [PushOp(2:2167012f:::rbd_data.1250f5e25b80.000000000000000d:6, version: 79'175, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:2167012f:::rbd_data.1250f5e25b80.000000000000000d:6@79'175, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 6=[]:{6: [6, 5, 3]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 4194304, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3145728, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-11T09:52:31.634 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:52:09.020895+0000", 2024-09-11T09:52:31.634 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 22.553869405, 2024-09-11T09:52:31.634 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.002075234, 2024-09-11T09:52:31.634 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:31.634 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:31.634 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "started", 2024-09-11T09:52:31.634 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:31.634 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.634 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:31.634 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.020895+0000", 2024-09-11T09:52:31.634 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:31.634 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.634 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.634 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:31.634 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.020895+0000", 2024-09-11T09:52:31.635 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:31.635 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.635 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.635 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:31.635 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.020895+0000", 2024-09-11T09:52:31.635 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.0999999999999999e-07 2024-09-11T09:52:31.635 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.635 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.635 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:31.635 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.021855+0000", 2024-09-11T09:52:31.635 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00095982700000000001 2024-09-11T09:52:31.635 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.635 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.635 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:31.635 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.021870+0000", 2024-09-11T09:52:31.636 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.4609e-05 2024-09-11T09:52:31.636 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.636 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.636 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:31.636 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.021881+0000", 2024-09-11T09:52:31.636 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.083e-05 2024-09-11T09:52:31.636 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.636 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.636 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:31.636 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.021943+0000", 2024-09-11T09:52:31.636 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.2780999999999994e-05 2024-09-11T09:52:31.636 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.636 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.636 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:31.637 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.021952+0000", 2024-09-11T09:52:31.637 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.8629999999999997e-06 2024-09-11T09:52:31.637 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.637 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.637 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:31.637 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.022970+0000", 2024-09-11T09:52:31.637 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0010178139999999999 2024-09-11T09:52:31.637 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:31.637 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:31.637 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:31.637 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.637 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.637 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.6054.0:3 2.4 2:2906e6a7:::rbd_header.1250f5e25b80:head [call rbd.get_size in=11b,call,call rbd.get_features in=15b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e95)", 2024-09-11T09:52:31.637 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:52:26.497840+0000", 2024-09-11T09:52:31.637 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 5.0769246849999998, 2024-09-11T09:52:31.638 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.421058402, 2024-09-11T09:52:31.638 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:31.638 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:31.638 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "started", 2024-09-11T09:52:31.638 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:52:31.638 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.6054", 2024-09-11T09:52:31.638 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/3379198229", 2024-09-11T09:52:31.638 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 3 2024-09-11T09:52:31.638 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.638 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:31.638 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.638 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:31.638 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:26.497840+0000", 2024-09-11T09:52:31.638 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:31.638 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.639 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.639 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:31.639 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:26.497840+0000", 2024-09-11T09:52:31.639 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:31.639 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.639 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.639 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:31.639 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:26.497841+0000", 2024-09-11T09:52:31.639 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.254e-06 2024-09-11T09:52:31.639 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.639 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.639 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:31.639 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:26.497846+0000", 2024-09-11T09:52:31.639 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.6710000000000004e-06 2024-09-11T09:52:31.640 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.640 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.640 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:31.640 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:26.497856+0000", 2024-09-11T09:52:31.640 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.3109999999999995e-06 2024-09-11T09:52:31.640 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.640 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.640 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:31.640 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:26.497863+0000", 2024-09-11T09:52:31.640 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.4919999999999996e-06 2024-09-11T09:52:31.640 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.640 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.640 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:31.640 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:26.497941+0000", 2024-09-11T09:52:31.641 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.7412999999999999e-05 2024-09-11T09:52:31.641 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.641 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.641 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for peered", 2024-09-11T09:52:31.641 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:26.497959+0000", 2024-09-11T09:52:31.641 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.8343e-05 2024-09-11T09:52:31.641 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.641 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.641 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:31.641 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:26.918496+0000", 2024-09-11T09:52:31.641 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.42053728299999998 2024-09-11T09:52:31.641 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.641 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.642 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:31.642 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:26.918642+0000", 2024-09-11T09:52:31.642 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000145422 2024-09-11T09:52:31.642 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.642 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.642 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:31.642 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:26.918898+0000", 2024-09-11T09:52:31.642 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00025621299999999998 2024-09-11T09:52:31.642 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:31.642 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:31.642 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:31.642 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.642 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.642 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.6072.0:26 2.4 2:2906e6a7:::rbd_header.1250f5e25b80:head [call rbd.snapshot_remove in=18b] snapc 0=[] ondisk+write+known_if_redirected+full_try+supports_pool_eio e97)", 2024-09-11T09:52:31.642 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:52:27.917078+0000", 2024-09-11T09:52:31.643 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 3.657685995, 2024-09-11T09:52:31.643 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.010482035000000001, 2024-09-11T09:52:31.643 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:31.643 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:31.643 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:31.643 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:52:31.643 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.6072", 2024-09-11T09:52:31.643 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/61699540", 2024-09-11T09:52:31.643 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 26 2024-09-11T09:52:31.643 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.643 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:31.643 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.643 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:31.643 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:27.917078+0000", 2024-09-11T09:52:31.644 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:31.644 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.644 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.644 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:31.644 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:27.917078+0000", 2024-09-11T09:52:31.644 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:31.644 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.644 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.644 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:31.644 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:27.917080+0000", 2024-09-11T09:52:31.644 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.1720000000000001e-06 2024-09-11T09:52:31.644 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.644 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.644 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:31.645 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:27.917090+0000", 2024-09-11T09:52:31.645 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.0759999999999992e-06 2024-09-11T09:52:31.645 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.645 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.645 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:31.645 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:27.917110+0000", 2024-09-11T09:52:31.645 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.0231e-05 2024-09-11T09:52:31.645 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.645 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.645 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:31.645 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:27.917122+0000", 2024-09-11T09:52:31.645 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.2106000000000001e-05 2024-09-11T09:52:31.646 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.646 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.646 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:31.646 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:27.917198+0000", 2024-09-11T09:52:31.646 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.6259000000000003e-05 2024-09-11T09:52:31.646 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.646 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.646 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op must wait for map", 2024-09-11T09:52:31.646 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:27.917207+0000", 2024-09-11T09:52:31.646 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.4659999999999999e-06 2024-09-11T09:52:31.646 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.646 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.646 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:31.647 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:27.923132+0000", 2024-09-11T09:52:31.647 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0059250730000000003 2024-09-11T09:52:31.647 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.647 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.647 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:31.647 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:27.923193+0000", 2024-09-11T09:52:31.647 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.1445999999999994e-05 2024-09-11T09:52:31.647 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.647 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.647 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 1", 2024-09-11T09:52:31.647 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:27.923411+0000", 2024-09-11T09:52:31.647 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00021771799999999999 2024-09-11T09:52:31.647 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.647 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.648 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:31.648 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:27.925324+0000", 2024-09-11T09:52:31.648 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0019128579999999999 2024-09-11T09:52:31.648 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.648 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.648 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:52:31.648 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:27.927463+0000", 2024-09-11T09:52:31.648 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.002139564 2024-09-11T09:52:31.648 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.648 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.648 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:31.648 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:27.927518+0000", 2024-09-11T09:52:31.648 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.4591e-05 2024-09-11T09:52:31.649 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.649 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.649 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:31.649 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:27.927560+0000", 2024-09-11T09:52:31.649 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.2475000000000002e-05 2024-09-11T09:52:31.649 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:31.649 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:31.649 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:31.649 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.649 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.649 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_repop(osd.3.0:238 2.5 e97/80 2:b6dd9fc9:::rbd_data.1250f5e25b80.0000000000000009:6 v 97'50, mlcod=79'45)", 2024-09-11T09:52:31.649 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:52:27.918868+0000", 2024-09-11T09:52:31.649 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 3.655896104, 2024-09-11T09:52:31.650 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0037081330000000002, 2024-09-11T09:52:31.650 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:31.650 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:31.650 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:31.650 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:31.650 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.650 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:31.650 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:27.918868+0000", 2024-09-11T09:52:31.650 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:31.650 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.650 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.650 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:31.650 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:27.918868+0000", 2024-09-11T09:52:31.651 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:31.651 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.651 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.651 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:31.651 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:27.918869+0000", 2024-09-11T09:52:31.651 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.9200000000000001e-07 2024-09-11T09:52:31.651 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.651 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.651 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:31.651 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:27.918873+0000", 2024-09-11T09:52:31.651 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.036e-06 2024-09-11T09:52:31.651 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.651 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.651 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:31.652 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:27.918880+0000", 2024-09-11T09:52:31.652 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.3590000000000002e-06 2024-09-11T09:52:31.652 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.652 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.652 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:31.652 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:27.918887+0000", 2024-09-11T09:52:31.652 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.9410000000000001e-06 2024-09-11T09:52:31.652 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.652 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.652 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:31.652 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:27.919121+0000", 2024-09-11T09:52:31.652 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000233935 2024-09-11T09:52:31.652 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.652 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.653 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:31.653 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:27.919135+0000", 2024-09-11T09:52:31.653 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.394e-05 2024-09-11T09:52:31.653 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.653 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.653 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:31.653 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:27.922468+0000", 2024-09-11T09:52:31.653 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.003333094 2024-09-11T09:52:31.653 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.653 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.653 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:31.653 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:27.922576+0000", 2024-09-11T09:52:31.653 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000108336 2024-09-11T09:52:31.654 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:31.654 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:31.654 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:31.654 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.654 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.654 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.6072.0:71 2.4 2:2906e6a7:::rbd_header.1250f5e25b80:head [call rbd.snapshot_remove in=18b] snapc 0=[] ondisk+write+known_if_redirected+full_try+supports_pool_eio e100)", 2024-09-11T09:52:31.654 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:52:30.936137+0000", 2024-09-11T09:52:31.654 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 0.63862745300000001, 2024-09-11T09:52:31.654 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.002867779, 2024-09-11T09:52:31.654 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:31.654 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:31.654 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:31.654 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:52:31.654 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.6072", 2024-09-11T09:52:31.654 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/61699540", 2024-09-11T09:52:31.655 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 71 2024-09-11T09:52:31.655 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.655 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:31.655 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.655 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:31.655 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:30.936137+0000", 2024-09-11T09:52:31.655 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:31.655 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.655 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.655 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:31.655 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:30.936137+0000", 2024-09-11T09:52:31.655 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:31.655 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.655 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.655 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:31.656 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:30.936139+0000", 2024-09-11T09:52:31.656 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.2029999999999999e-06 2024-09-11T09:52:31.656 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.656 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.656 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:31.656 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:30.936151+0000", 2024-09-11T09:52:31.656 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1677e-05 2024-09-11T09:52:31.656 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.656 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.656 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:31.656 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:30.936170+0000", 2024-09-11T09:52:31.656 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.9193000000000001e-05 2024-09-11T09:52:31.656 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.656 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.657 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:31.657 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:30.936180+0000", 2024-09-11T09:52:31.657 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.8749999999999995e-06 2024-09-11T09:52:31.657 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.657 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.657 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:31.657 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:30.936246+0000", 2024-09-11T09:52:31.657 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.6398999999999999e-05 2024-09-11T09:52:31.657 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.657 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.657 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op must wait for map", 2024-09-11T09:52:31.657 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:30.936252+0000", 2024-09-11T09:52:31.657 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.6169999999999998e-06 2024-09-11T09:52:31.657 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.657 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.658 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:31.658 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:30.937633+0000", 2024-09-11T09:52:31.658 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0013812989999999999 2024-09-11T09:52:31.658 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.658 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.658 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:31.658 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:30.937701+0000", 2024-09-11T09:52:31.658 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.7895e-05 2024-09-11T09:52:31.658 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.658 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.658 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 1", 2024-09-11T09:52:31.658 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:30.937937+0000", 2024-09-11T09:52:31.658 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00023624299999999999 2024-09-11T09:52:31.658 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.659 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.659 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:52:31.659 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:30.938479+0000", 2024-09-11T09:52:31.659 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00054196300000000002 2024-09-11T09:52:31.659 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.659 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.659 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:31.659 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:30.938937+0000", 2024-09-11T09:52:31.659 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00045802100000000002 2024-09-11T09:52:31.659 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.659 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.659 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:31.659 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:30.938971+0000", 2024-09-11T09:52:31.660 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.3420999999999997e-05 2024-09-11T09:52:31.660 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.660 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.660 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:31.660 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:30.939005+0000", 2024-09-11T09:52:31.660 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.3973000000000002e-05 2024-09-11T09:52:31.660 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:31.660 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:31.660 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:31.660 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.660 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.660 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.5648.0:84 2.4 2:2367385a:::rbd_data.1250f5e25b80.0000000000000002:head [delete] snapc 7=[] ondisk+write+known_if_redirected+full_try+supports_pool_eio e100)", 2024-09-11T09:52:31.660 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:52:31.053615+0000", 2024-09-11T09:52:31.660 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 0.52114968299999997, 2024-09-11T09:52:31.660 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0023006210000000001, 2024-09-11T09:52:31.661 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:31.661 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:31.661 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:31.661 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:52:31.661 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.5648", 2024-09-11T09:52:31.661 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/1071190571", 2024-09-11T09:52:31.661 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 84 2024-09-11T09:52:31.661 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.661 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:31.661 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.661 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:31.661 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:31.053615+0000", 2024-09-11T09:52:31.661 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:31.661 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.662 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.662 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:31.662 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:31.053615+0000", 2024-09-11T09:52:31.662 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:31.662 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.662 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.662 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:31.662 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:31.053615+0000", 2024-09-11T09:52:31.662 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.9599999999999998e-07 2024-09-11T09:52:31.662 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.662 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.662 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:31.662 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:31.053619+0000", 2024-09-11T09:52:31.662 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.946e-06 2024-09-11T09:52:31.663 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.663 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.663 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:31.663 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:31.053629+0000", 2024-09-11T09:52:31.663 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.5980000000000003e-06 2024-09-11T09:52:31.663 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.663 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.663 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:31.663 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:31.053634+0000", 2024-09-11T09:52:31.663 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.0640000000000001e-06 2024-09-11T09:52:31.663 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.663 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.663 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:31.663 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:31.053671+0000", 2024-09-11T09:52:31.663 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.7426999999999999e-05 2024-09-11T09:52:31.664 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.664 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.664 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:31.664 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:31.053730+0000", 2024-09-11T09:52:31.664 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.8597000000000002e-05 2024-09-11T09:52:31.664 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.664 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.664 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 1", 2024-09-11T09:52:31.664 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:31.053846+0000", 2024-09-11T09:52:31.664 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000115524 2024-09-11T09:52:31.664 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.664 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.664 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:52:31.664 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:31.054812+0000", 2024-09-11T09:52:31.665 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00096634300000000004 2024-09-11T09:52:31.665 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.665 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.665 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:31.665 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:31.055823+0000", 2024-09-11T09:52:31.665 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001011444 2024-09-11T09:52:31.665 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.665 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.665 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:31.665 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:31.055873+0000", 2024-09-11T09:52:31.665 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.9740000000000001e-05 2024-09-11T09:52:31.665 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.665 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.665 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:31.666 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:31.055915+0000", 2024-09-11T09:52:31.666 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.2141999999999997e-05 2024-09-11T09:52:31.666 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:31.666 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:31.666 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:31.666 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.666 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.666 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.5648.0:89 2.4 2:239e915b:::rbd_data.1250f5e25b80.000000000000000c:head [delete] snapc 7=[] ondisk+write+known_if_redirected+full_try+supports_pool_eio e100)", 2024-09-11T09:52:31.666 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:52:31.053720+0000", 2024-09-11T09:52:31.666 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 0.52104431799999995, 2024-09-11T09:52:31.666 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.002622194, 2024-09-11T09:52:31.666 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:31.666 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:31.666 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:31.667 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:52:31.667 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.5648", 2024-09-11T09:52:31.667 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/1071190571", 2024-09-11T09:52:31.667 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 89 2024-09-11T09:52:31.667 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.667 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:31.667 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.667 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:31.667 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:31.053720+0000", 2024-09-11T09:52:31.667 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:31.667 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.667 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.667 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:31.667 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:31.053720+0000", 2024-09-11T09:52:31.668 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:31.668 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.668 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.668 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:31.668 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:31.053721+0000", 2024-09-11T09:52:31.668 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.8299999999999995e-07 2024-09-11T09:52:31.668 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.668 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.668 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:31.668 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:31.053726+0000", 2024-09-11T09:52:31.668 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.8770000000000001e-06 2024-09-11T09:52:31.668 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.668 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.668 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:31.669 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:31.053745+0000", 2024-09-11T09:52:31.669 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.9262999999999998e-05 2024-09-11T09:52:31.669 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.669 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.669 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:31.669 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:31.053752+0000", 2024-09-11T09:52:31.669 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.294e-06 2024-09-11T09:52:31.669 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.669 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.669 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:31.669 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:31.054261+0000", 2024-09-11T09:52:31.669 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00050814800000000004 2024-09-11T09:52:31.669 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.669 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.669 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:31.670 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:31.054337+0000", 2024-09-11T09:52:31.670 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.682e-05 2024-09-11T09:52:31.670 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.670 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.670 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 1", 2024-09-11T09:52:31.670 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:31.054420+0000", 2024-09-11T09:52:31.670 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.2576000000000004e-05 2024-09-11T09:52:31.670 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.670 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.670 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:31.670 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:31.056204+0000", 2024-09-11T09:52:31.670 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0017841949999999999 2024-09-11T09:52:31.670 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.671 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.671 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:52:31.671 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:31.056218+0000", 2024-09-11T09:52:31.671 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.3417999999999999e-05 2024-09-11T09:52:31.671 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.671 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.671 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:31.671 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:31.056290+0000", 2024-09-11T09:52:31.671 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.2675000000000005e-05 2024-09-11T09:52:31.671 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:31.671 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:31.671 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:31.671 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:31.056342+0000", 2024-09-11T09:52:31.672 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.1944999999999998e-05 2024-09-11T09:52:31.672 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:31.672 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:31.672 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:31.672 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:31.672 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:31.672 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-11T09:52:31.673 INFO:tasks.workunit.client.0.smithi107.stderr:+ rbd -p rbd --namespace ns status testrbdnbd27121 2024-09-11T09:52:31.673 INFO:tasks.workunit.client.0.smithi107.stderr:+ grep 'Watchers: none' 2024-09-11T09:52:31.674 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:52:31.674+0000 7f3c96ec9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31909) UID: 0 2024-09-11T09:52:31.674 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-11T09:52:31.678 INFO:tasks.workunit.client.0.smithi107.stdout:Watchers: none 2024-09-11T09:52:31.679 INFO:tasks.workunit.client.0.smithi107.stderr:+ break 2024-09-11T09:52:31.679 INFO:tasks.workunit.client.0.smithi107.stderr:+ rbd -p rbd --namespace ns snap purge testrbdnbd27121 2024-09-11T09:52:31.774 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-09-11T09:52:31.775+0000 7f7b6c7d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31904) UID: 0 2024-09-11T09:52:31.789 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:52:31.789 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [], 2024-09-11T09:52:31.789 INFO:teuthology.orchestra.run.smithi017.stdout: "num_ops": 0 2024-09-11T09:52:31.789 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:52:31.798 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-11T09:52:31.875 INFO:tasks.ceph.osd.2.smithi017.stderr:2024-09-11T09:52:31.876+0000 7fd20b170640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31672) UID: 0 2024-09-11T09:52:31.905 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:52:31.906 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [], 2024-09-11T09:52:31.906 INFO:teuthology.orchestra.run.smithi017.stdout: "complaint_time": 30, 2024-09-11T09:52:31.906 INFO:teuthology.orchestra.run.smithi017.stdout: "num_blocked_ops": 0 2024-09-11T09:52:31.906 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:52:31.914 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-11T09:52:31.935 INFO:teuthology.orchestra.run.smithi017.stderr:reweighted osd.2 to 0.127738 (20b3) 2024-09-11T09:52:31.992 INFO:tasks.ceph.osd.2.smithi017.stderr:2024-09-11T09:52:31.993+0000 7fd20b170640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31672) UID: 0 2024-09-11T09:52:32.045 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:52:32.045 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 20, 2024-09-11T09:52:32.045 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 600, 2024-09-11T09:52:32.045 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [ 2024-09-11T09:52:32.046 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.046 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4688.0:34 2.4 e53/32 2:2367385a:::rbd_data.1250f5e25b80.0000000000000002:head v 53'6, mlcod=53'4)", 2024-09-11T09:52:32.046 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.907415+0000", 2024-09-11T09:52:32.046 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 69.138068701999998, 2024-09-11T09:52:32.046 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.011285778999999999, 2024-09-11T09:52:32.046 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:32.046 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:32.046 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:32.046 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:32.046 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.046 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:32.046 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.907415+0000", 2024-09-11T09:52:32.046 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:32.046 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.047 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.047 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:32.047 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.907415+0000", 2024-09-11T09:52:32.047 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:32.047 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.047 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.047 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:32.047 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.907417+0000", 2024-09-11T09:52:32.047 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.2910000000000002e-06 2024-09-11T09:52:32.047 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.047 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.047 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:32.047 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.910477+0000", 2024-09-11T09:52:32.047 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0030597829999999999 2024-09-11T09:52:32.047 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.048 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.048 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:32.048 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.910501+0000", 2024-09-11T09:52:32.048 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.3332e-05 2024-09-11T09:52:32.048 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.048 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.048 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:32.048 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.910514+0000", 2024-09-11T09:52:32.048 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3957e-05 2024-09-11T09:52:32.048 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.048 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.048 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:32.048 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.910612+0000", 2024-09-11T09:52:32.049 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.7075000000000001e-05 2024-09-11T09:52:32.049 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.049 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.049 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:32.049 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.910640+0000", 2024-09-11T09:52:32.049 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.8768999999999998e-05 2024-09-11T09:52:32.049 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.049 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.049 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:32.049 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.918640+0000", 2024-09-11T09:52:32.049 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0079991899999999998 2024-09-11T09:52:32.049 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.049 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.049 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:32.049 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.918701+0000", 2024-09-11T09:52:32.050 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.1382000000000003e-05 2024-09-11T09:52:32.050 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:32.050 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:32.050 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:32.050 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.050 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.050 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4688.0:38 2.5 e53/32 2:abde2694:::rbd_data.1250f5e25b80.0000000000000004:head v 53'1, mlcod=0'0)", 2024-09-11T09:52:32.050 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.915432+0000", 2024-09-11T09:52:32.050 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 69.130052321999997, 2024-09-11T09:52:32.050 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.015849652999999998, 2024-09-11T09:52:32.050 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:32.050 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:32.050 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:32.050 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:32.051 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.051 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:32.051 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.915432+0000", 2024-09-11T09:52:32.051 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:32.051 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.051 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.051 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:32.051 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.915432+0000", 2024-09-11T09:52:32.051 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:32.051 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.051 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.051 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:32.051 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.915432+0000", 2024-09-11T09:52:32.051 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.91e-07 2024-09-11T09:52:32.051 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.052 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.052 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:32.052 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.924943+0000", 2024-09-11T09:52:32.052 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0095110630000000002 2024-09-11T09:52:32.052 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.052 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.052 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:32.052 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.924960+0000", 2024-09-11T09:52:32.052 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6915999999999999e-05 2024-09-11T09:52:32.052 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.052 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.052 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:32.052 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.924968+0000", 2024-09-11T09:52:32.052 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.6999999999999997e-06 2024-09-11T09:52:32.052 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.053 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.053 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:32.053 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.925025+0000", 2024-09-11T09:52:32.053 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.6949e-05 2024-09-11T09:52:32.053 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.053 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.053 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:32.053 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.925043+0000", 2024-09-11T09:52:32.053 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7784000000000001e-05 2024-09-11T09:52:32.053 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.053 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.053 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:32.053 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.931237+0000", 2024-09-11T09:52:32.053 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0061935389999999996 2024-09-11T09:52:32.053 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.054 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.054 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:32.054 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.931281+0000", 2024-09-11T09:52:32.054 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.4511000000000001e-05 2024-09-11T09:52:32.054 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:32.054 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:32.054 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:32.054 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.054 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.054 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:40 2.7 2:fe41411c:::rbd_object_map.1250f5e25b80:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:32.054 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.919652+0000", 2024-09-11T09:52:32.054 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 69.125831654999999, 2024-09-11T09:52:32.055 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.012187073, 2024-09-11T09:52:32.055 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:32.055 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:32.055 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:32.055 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:32.055 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:52:32.055 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:52:32.055 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 40 2024-09-11T09:52:32.055 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.055 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:32.055 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.055 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:32.055 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919652+0000", 2024-09-11T09:52:32.055 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:32.055 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.056 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.056 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:32.056 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919652+0000", 2024-09-11T09:52:32.056 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:32.056 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.056 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.056 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:32.056 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919654+0000", 2024-09-11T09:52:32.056 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.88e-06 2024-09-11T09:52:32.056 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.056 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.056 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:32.056 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919669+0000", 2024-09-11T09:52:32.056 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.5271e-05 2024-09-11T09:52:32.057 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.057 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.057 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:32.057 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919687+0000", 2024-09-11T09:52:32.057 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7448999999999998e-05 2024-09-11T09:52:32.057 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.057 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.057 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:32.057 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919695+0000", 2024-09-11T09:52:32.057 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.1149999999999994e-06 2024-09-11T09:52:32.057 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.057 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.057 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:32.057 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919757+0000", 2024-09-11T09:52:32.057 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.2348999999999995e-05 2024-09-11T09:52:32.058 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.058 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.058 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:32.058 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919875+0000", 2024-09-11T09:52:32.058 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00011750499999999999 2024-09-11T09:52:32.058 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.058 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.058 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 5", 2024-09-11T09:52:32.058 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.920201+0000", 2024-09-11T09:52:32.058 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00032628099999999999 2024-09-11T09:52:32.058 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.058 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.058 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:32.058 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.922089+0000", 2024-09-11T09:52:32.058 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.001887978 2024-09-11T09:52:32.059 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.059 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.059 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:32.059 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.931752+0000", 2024-09-11T09:52:32.059 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0096626309999999997 2024-09-11T09:52:32.059 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.059 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.059 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:32.059 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.931794+0000", 2024-09-11T09:52:32.059 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.2849999999999998e-05 2024-09-11T09:52:32.059 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.059 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.059 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:32.059 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.931839+0000", 2024-09-11T09:52:32.060 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.4764e-05 2024-09-11T09:52:32.060 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:32.060 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:32.060 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:32.060 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.060 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.060 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:41 2.7 2:fe41411c:::rbd_object_map.1250f5e25b80:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:32.060 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.919716+0000", 2024-09-11T09:52:32.060 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 69.125767811000003, 2024-09-11T09:52:32.060 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.015641605999999999, 2024-09-11T09:52:32.060 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:32.060 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:32.060 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:32.060 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:32.061 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:52:32.061 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:52:32.061 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 41 2024-09-11T09:52:32.061 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.061 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:32.061 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.061 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:32.061 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919716+0000", 2024-09-11T09:52:32.061 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:32.061 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.061 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.061 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:32.061 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919716+0000", 2024-09-11T09:52:32.061 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:32.062 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.062 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.062 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:32.062 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919716+0000", 2024-09-11T09:52:32.062 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.9999999999999998e-07 2024-09-11T09:52:32.062 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.062 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.062 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:32.062 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919720+0000", 2024-09-11T09:52:32.062 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.6440000000000002e-06 2024-09-11T09:52:32.062 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.062 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.062 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:32.062 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919729+0000", 2024-09-11T09:52:32.062 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.1759999999999999e-06 2024-09-11T09:52:32.063 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.063 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.063 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:32.063 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919736+0000", 2024-09-11T09:52:32.063 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.5390000000000004e-06 2024-09-11T09:52:32.063 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.063 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.063 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:32.063 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.920669+0000", 2024-09-11T09:52:32.063 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00093275599999999997 2024-09-11T09:52:32.063 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.063 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.063 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:32.063 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.920797+0000", 2024-09-11T09:52:32.064 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00012893400000000001 2024-09-11T09:52:32.064 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.064 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.064 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 5", 2024-09-11T09:52:32.064 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.921269+0000", 2024-09-11T09:52:32.064 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00047128600000000001 2024-09-11T09:52:32.064 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.064 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.064 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:32.064 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.923198+0000", 2024-09-11T09:52:32.064 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0019289369999999999 2024-09-11T09:52:32.064 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.064 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.064 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:32.065 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.935300+0000", 2024-09-11T09:52:32.065 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.012102392 2024-09-11T09:52:32.065 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.065 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.065 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:32.065 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.935332+0000", 2024-09-11T09:52:32.065 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.2292000000000002e-05 2024-09-11T09:52:32.065 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.065 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.065 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:32.065 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.935358+0000", 2024-09-11T09:52:32.065 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.525e-05 2024-09-11T09:52:32.065 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:32.065 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:32.066 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:32.066 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.066 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.066 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:48 2.7 2:fe41411c:::rbd_object_map.1250f5e25b80:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:32.066 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.938799+0000", 2024-09-11T09:52:32.066 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 69.106684701999995, 2024-09-11T09:52:32.066 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0041642340000000002, 2024-09-11T09:52:32.066 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:32.066 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:32.066 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:32.066 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:32.066 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:52:32.066 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:52:32.066 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 48 2024-09-11T09:52:32.067 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.067 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:32.067 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.067 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:32.067 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.938799+0000", 2024-09-11T09:52:32.067 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:32.067 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.067 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.067 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:32.067 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.938799+0000", 2024-09-11T09:52:32.067 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:32.067 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.067 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.067 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:32.068 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.938800+0000", 2024-09-11T09:52:32.068 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.71e-07 2024-09-11T09:52:32.068 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.068 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.068 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:32.068 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.938805+0000", 2024-09-11T09:52:32.068 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.3140000000000003e-06 2024-09-11T09:52:32.068 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.068 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.068 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:32.068 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.938816+0000", 2024-09-11T09:52:32.068 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0231000000000001e-05 2024-09-11T09:52:32.068 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.069 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.069 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:32.069 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.938822+0000", 2024-09-11T09:52:32.069 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.9550000000000003e-06 2024-09-11T09:52:32.069 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.069 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.069 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:32.069 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.938868+0000", 2024-09-11T09:52:32.069 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.5562999999999997e-05 2024-09-11T09:52:32.069 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.069 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.069 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:32.069 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.938918+0000", 2024-09-11T09:52:32.070 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.0317e-05 2024-09-11T09:52:32.070 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.070 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.070 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 5", 2024-09-11T09:52:32.070 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.939114+0000", 2024-09-11T09:52:32.070 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00019538300000000001 2024-09-11T09:52:32.070 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.070 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.070 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:32.070 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.939663+0000", 2024-09-11T09:52:32.070 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00054932199999999998 2024-09-11T09:52:32.070 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.070 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.070 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:32.070 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.942910+0000", 2024-09-11T09:52:32.071 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0032473839999999999 2024-09-11T09:52:32.071 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.071 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.071 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:32.071 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.942940+0000", 2024-09-11T09:52:32.071 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.9306999999999999e-05 2024-09-11T09:52:32.071 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.071 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.071 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:32.071 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.942963+0000", 2024-09-11T09:52:32.071 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.3587e-05 2024-09-11T09:52:32.071 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:32.071 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:32.072 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:32.072 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.072 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.072 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4688.0:49 2.5 e53/32 2:b6dd9fc9:::rbd_data.1250f5e25b80.0000000000000009:head v 53'2, mlcod=0'0)", 2024-09-11T09:52:32.072 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.944712+0000", 2024-09-11T09:52:32.072 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 69.100771863000006, 2024-09-11T09:52:32.072 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.016185069999999999, 2024-09-11T09:52:32.072 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:32.072 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:32.072 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:32.072 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:32.072 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.072 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:32.072 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.944712+0000", 2024-09-11T09:52:32.073 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:32.073 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.073 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.073 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:32.073 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.944712+0000", 2024-09-11T09:52:32.073 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:32.073 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.073 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.073 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:32.073 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.944713+0000", 2024-09-11T09:52:32.073 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0139999999999999e-06 2024-09-11T09:52:32.073 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.073 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.074 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:32.074 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.955518+0000", 2024-09-11T09:52:32.074 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.01080542 2024-09-11T09:52:32.074 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.074 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.074 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:32.074 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.955536+0000", 2024-09-11T09:52:32.074 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7818000000000001e-05 2024-09-11T09:52:32.074 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.074 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.074 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:32.074 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.955544+0000", 2024-09-11T09:52:32.074 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.1699999999999997e-06 2024-09-11T09:52:32.074 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.075 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.075 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:32.075 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.955607+0000", 2024-09-11T09:52:32.075 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.2451000000000002e-05 2024-09-11T09:52:32.075 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.075 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.075 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:32.075 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.955632+0000", 2024-09-11T09:52:32.075 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.5335000000000001e-05 2024-09-11T09:52:32.075 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.075 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.075 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:32.075 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.960826+0000", 2024-09-11T09:52:32.075 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0051938560000000002 2024-09-11T09:52:32.076 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.076 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.076 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:32.076 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.960897+0000", 2024-09-11T09:52:32.076 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.1006000000000004e-05 2024-09-11T09:52:32.076 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:32.076 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:32.076 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:32.076 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.076 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.076 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:53 2.7 2:fe41411c:::rbd_object_map.1250f5e25b80:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:32.076 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.951291+0000", 2024-09-11T09:52:32.076 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 69.094192872999997, 2024-09-11T09:52:32.076 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0044964999999999996, 2024-09-11T09:52:32.077 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:32.077 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:32.077 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:32.077 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:32.077 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:52:32.077 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:52:32.077 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 53 2024-09-11T09:52:32.077 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.077 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:32.077 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.077 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:32.077 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951291+0000", 2024-09-11T09:52:32.077 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:32.077 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.078 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.078 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:32.078 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951291+0000", 2024-09-11T09:52:32.078 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:32.078 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.078 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.078 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:32.078 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951293+0000", 2024-09-11T09:52:32.078 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7889999999999999e-06 2024-09-11T09:52:32.078 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.078 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.078 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:32.078 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951302+0000", 2024-09-11T09:52:32.078 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.9819999999999997e-06 2024-09-11T09:52:32.078 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.079 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.079 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:32.079 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951324+0000", 2024-09-11T09:52:32.079 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.2141e-05 2024-09-11T09:52:32.079 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.079 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.079 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:32.079 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951338+0000", 2024-09-11T09:52:32.079 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3750999999999999e-05 2024-09-11T09:52:32.079 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.079 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.079 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:32.079 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951409+0000", 2024-09-11T09:52:32.079 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.1658999999999999e-05 2024-09-11T09:52:32.080 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.080 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.080 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:32.080 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951517+0000", 2024-09-11T09:52:32.080 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00010746799999999999 2024-09-11T09:52:32.080 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.080 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.080 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 5", 2024-09-11T09:52:32.080 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951942+0000", 2024-09-11T09:52:32.080 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00042524300000000001 2024-09-11T09:52:32.080 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.080 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.080 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:32.080 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.952872+0000", 2024-09-11T09:52:32.081 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00092976300000000001 2024-09-11T09:52:32.081 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.081 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.081 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:32.081 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.955682+0000", 2024-09-11T09:52:32.081 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0028099459999999998 2024-09-11T09:52:32.081 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.081 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.081 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:32.081 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.955746+0000", 2024-09-11T09:52:32.081 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.4084999999999994e-05 2024-09-11T09:52:32.081 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.081 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.081 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:32.082 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.955787+0000", 2024-09-11T09:52:32.082 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.1672999999999999e-05 2024-09-11T09:52:32.082 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:32.082 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:32.082 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:32.082 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.082 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.082 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4688.0:52 2.4 e53/32 2:26d5c2f0:::rbd_data.1250f5e25b80.000000000000000b:head v 53'8, mlcod=53'6)", 2024-09-11T09:52:32.082 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.963677+0000", 2024-09-11T09:52:32.082 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 69.081807318000003, 2024-09-11T09:52:32.082 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0089437969999999999, 2024-09-11T09:52:32.082 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:32.082 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:32.082 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:32.083 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:32.083 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.083 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:32.083 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.963677+0000", 2024-09-11T09:52:32.083 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:32.083 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.083 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.083 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:32.083 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.963677+0000", 2024-09-11T09:52:32.083 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:32.083 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.083 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.083 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:32.083 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.963677+0000", 2024-09-11T09:52:32.083 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.7199999999999998e-07 2024-09-11T09:52:32.084 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.084 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.084 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:32.084 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.965506+0000", 2024-09-11T09:52:32.084 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.001829064 2024-09-11T09:52:32.084 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.084 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.084 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:32.084 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.965538+0000", 2024-09-11T09:52:32.084 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.1306999999999997e-05 2024-09-11T09:52:32.084 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.084 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.084 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:32.084 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.965553+0000", 2024-09-11T09:52:32.084 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.4924e-05 2024-09-11T09:52:32.085 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.085 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.085 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:32.085 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.965663+0000", 2024-09-11T09:52:32.085 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000110315 2024-09-11T09:52:32.085 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.085 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.085 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:32.085 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.965692+0000", 2024-09-11T09:52:32.085 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.934e-05 2024-09-11T09:52:32.085 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.085 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.085 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:32.085 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.972537+0000", 2024-09-11T09:52:32.086 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0068444969999999997 2024-09-11T09:52:32.086 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.086 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.086 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:32.086 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.972620+0000", 2024-09-11T09:52:32.086 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.3578000000000005e-05 2024-09-11T09:52:32.086 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:32.086 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:32.086 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:32.086 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.086 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.086 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4688.0:54 2.4 e53/32 2:239e915b:::rbd_data.1250f5e25b80.000000000000000c:head v 53'9, mlcod=53'7)", 2024-09-11T09:52:32.086 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.987028+0000", 2024-09-11T09:52:32.086 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 69.058455401000003, 2024-09-11T09:52:32.087 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0094155650000000007, 2024-09-11T09:52:32.087 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:32.087 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:32.087 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:32.087 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:32.087 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.087 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:32.087 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.987028+0000", 2024-09-11T09:52:32.087 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:32.087 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.087 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.087 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:32.087 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.987028+0000", 2024-09-11T09:52:32.087 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:32.088 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.088 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.088 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:32.088 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.987030+0000", 2024-09-11T09:52:32.088 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6130000000000001e-06 2024-09-11T09:52:32.088 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.088 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.088 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:32.088 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.988575+0000", 2024-09-11T09:52:32.088 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.001545193 2024-09-11T09:52:32.088 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.088 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.088 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:32.088 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.988604+0000", 2024-09-11T09:52:32.089 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.9173e-05 2024-09-11T09:52:32.089 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.089 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.089 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:32.089 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.988620+0000", 2024-09-11T09:52:32.089 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.5517999999999999e-05 2024-09-11T09:52:32.089 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.089 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.089 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:32.089 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.988704+0000", 2024-09-11T09:52:32.089 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.3588e-05 2024-09-11T09:52:32.089 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.089 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.089 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:32.090 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.988735+0000", 2024-09-11T09:52:32.090 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.1643000000000002e-05 2024-09-11T09:52:32.090 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.090 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.090 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:32.090 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.996383+0000", 2024-09-11T09:52:32.090 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0076478099999999997 2024-09-11T09:52:32.090 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.090 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.090 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:32.090 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.996444+0000", 2024-09-11T09:52:32.090 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.1026999999999997e-05 2024-09-11T09:52:32.090 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:32.091 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:32.091 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:32.091 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.091 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.091 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4688.0:56 2.4 e53/32 2:2167012f:::rbd_data.1250f5e25b80.000000000000000d:head v 53'10, mlcod=53'7)", 2024-09-11T09:52:32.091 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.993894+0000", 2024-09-11T09:52:32.091 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 69.051589859000003, 2024-09-11T09:52:32.091 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0099268559999999995, 2024-09-11T09:52:32.091 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:32.091 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:32.091 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:32.091 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:32.091 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.092 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:32.092 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.993894+0000", 2024-09-11T09:52:32.092 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:32.092 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.092 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.092 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:32.092 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.993894+0000", 2024-09-11T09:52:32.092 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:32.092 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.092 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.092 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:32.092 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.993896+0000", 2024-09-11T09:52:32.092 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6249999999999999e-06 2024-09-11T09:52:32.093 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.093 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.093 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:32.093 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.997606+0000", 2024-09-11T09:52:32.093 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.003710737 2024-09-11T09:52:32.093 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.093 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.093 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:32.093 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.997628+0000", 2024-09-11T09:52:32.093 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.1971e-05 2024-09-11T09:52:32.093 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.093 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.093 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:32.093 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.997640+0000", 2024-09-11T09:52:32.094 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1481000000000001e-05 2024-09-11T09:52:32.094 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.094 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.094 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:32.094 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.997712+0000", 2024-09-11T09:52:32.094 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.2255000000000005e-05 2024-09-11T09:52:32.094 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.094 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.094 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:32.094 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.997738+0000", 2024-09-11T09:52:32.094 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.6050999999999998e-05 2024-09-11T09:52:32.094 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.094 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.094 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:32.094 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.003779+0000", 2024-09-11T09:52:32.095 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0060410960000000001 2024-09-11T09:52:32.095 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.095 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.095 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:32.095 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.003821+0000", 2024-09-11T09:52:32.095 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.1640000000000001e-05 2024-09-11T09:52:32.095 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:32.095 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:32.095 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:32.095 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.095 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.095 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "pg_update_log_missing(2.4 epoch 53/32 rep_tid 16 entries 53'11 (0'0) error 2:2906e6a7:::rbd_header.1250f5e25b80:head by client.4688.0:59 0.000000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 53'7)", 2024-09-11T09:52:32.095 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.998956+0000", 2024-09-11T09:52:32.096 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 69.046527690000005, 2024-09-11T09:52:32.096 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.004934127, 2024-09-11T09:52:32.096 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:32.096 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:32.096 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "reached pg", 2024-09-11T09:52:32.096 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:32.096 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.096 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:32.096 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.998956+0000", 2024-09-11T09:52:32.096 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:32.096 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.096 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.096 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:32.097 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.998956+0000", 2024-09-11T09:52:32.097 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:32.097 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.097 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.097 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:32.097 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.998957+0000", 2024-09-11T09:52:32.097 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.139e-06 2024-09-11T09:52:32.097 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.097 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.097 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:32.097 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.998985+0000", 2024-09-11T09:52:32.097 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.7912e-05 2024-09-11T09:52:32.098 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.098 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.098 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:32.098 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.999011+0000", 2024-09-11T09:52:32.098 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.6061e-05 2024-09-11T09:52:32.098 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.098 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.098 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:32.098 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.999028+0000", 2024-09-11T09:52:32.098 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7065999999999999e-05 2024-09-11T09:52:32.098 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.098 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.098 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:32.098 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.001919+0000", 2024-09-11T09:52:32.099 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.002891167 2024-09-11T09:52:32.099 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.099 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.099 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:32.099 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.003890+0000", 2024-09-11T09:52:32.099 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0019707819999999999 2024-09-11T09:52:32.099 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:32.099 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:32.099 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:32.099 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.099 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.099 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4694.0:34 2.7 2:f80bfc0c:ns::rbd_data.1256a0c02879.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:32.099 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.085183+0000", 2024-09-11T09:52:32.099 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 68.960301177000005, 2024-09-11T09:52:32.100 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.020908599, 2024-09-11T09:52:32.100 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:32.100 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:32.100 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:32.100 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:32.100 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4694", 2024-09-11T09:52:32.100 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:52:32.100 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 34 2024-09-11T09:52:32.100 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.100 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:32.100 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.100 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:32.100 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.085183+0000", 2024-09-11T09:52:32.101 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:32.101 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.101 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.101 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:32.101 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.085183+0000", 2024-09-11T09:52:32.101 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:32.101 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.101 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.101 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:32.101 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.085183+0000", 2024-09-11T09:52:32.101 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.4300000000000002e-07 2024-09-11T09:52:32.101 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.101 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.101 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:32.102 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094337+0000", 2024-09-11T09:52:32.102 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0091532769999999996 2024-09-11T09:52:32.102 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.102 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.102 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:32.102 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094362+0000", 2024-09-11T09:52:32.102 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.5287999999999999e-05 2024-09-11T09:52:32.102 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.102 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.102 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:32.102 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094375+0000", 2024-09-11T09:52:32.102 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2446000000000001e-05 2024-09-11T09:52:32.102 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.103 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.103 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:32.103 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094455+0000", 2024-09-11T09:52:32.103 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.0092999999999994e-05 2024-09-11T09:52:32.103 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.103 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.103 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:32.103 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094618+0000", 2024-09-11T09:52:32.103 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000163747 2024-09-11T09:52:32.103 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.103 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.103 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 5", 2024-09-11T09:52:32.103 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094783+0000", 2024-09-11T09:52:32.104 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000165038 2024-09-11T09:52:32.104 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.104 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.104 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:32.104 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.105853+0000", 2024-09-11T09:52:32.104 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.011069294 2024-09-11T09:52:32.104 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.104 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.104 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:32.104 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.106004+0000", 2024-09-11T09:52:32.104 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00015123900000000001 2024-09-11T09:52:32.105 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.105 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.105 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:32.105 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.106062+0000", 2024-09-11T09:52:32.105 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.8205000000000003e-05 2024-09-11T09:52:32.105 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.105 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.105 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:32.105 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.106091+0000", 2024-09-11T09:52:32.105 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.9128999999999998e-05 2024-09-11T09:52:32.105 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:32.105 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:32.105 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:32.105 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.106 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.106 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4694.0:35 2.7 2:e8766325:ns::rbd_object_map.1256a0c02879:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:32.106 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.094405+0000", 2024-09-11T09:52:32.106 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 68.951078541000001, 2024-09-11T09:52:32.106 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.012623564, 2024-09-11T09:52:32.106 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:32.106 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:32.106 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:32.106 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:32.106 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4694", 2024-09-11T09:52:32.106 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:52:32.106 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 35 2024-09-11T09:52:32.106 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.107 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:32.107 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.107 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:32.107 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094405+0000", 2024-09-11T09:52:32.107 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:32.107 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.107 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.107 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:32.107 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094405+0000", 2024-09-11T09:52:32.107 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:32.107 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.107 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.107 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:32.107 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094407+0000", 2024-09-11T09:52:32.108 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.739e-06 2024-09-11T09:52:32.108 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.108 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.108 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:32.108 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094411+0000", 2024-09-11T09:52:32.108 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.3340000000000003e-06 2024-09-11T09:52:32.108 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.108 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.108 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:32.108 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094419+0000", 2024-09-11T09:52:32.108 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.8909999999999995e-06 2024-09-11T09:52:32.108 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.108 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.108 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:32.109 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094424+0000", 2024-09-11T09:52:32.109 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.7639999999999996e-06 2024-09-11T09:52:32.109 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.109 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.109 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:32.109 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.100209+0000", 2024-09-11T09:52:32.109 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0057850489999999996 2024-09-11T09:52:32.109 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.109 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.110 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:32.110 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.100341+0000", 2024-09-11T09:52:32.110 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000131438 2024-09-11T09:52:32.110 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.110 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.110 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 5", 2024-09-11T09:52:32.110 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.100805+0000", 2024-09-11T09:52:32.110 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00046471499999999998 2024-09-11T09:52:32.110 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.110 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.110 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:32.110 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.106148+0000", 2024-09-11T09:52:32.110 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0053426460000000004 2024-09-11T09:52:32.110 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.111 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.111 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:32.111 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.106929+0000", 2024-09-11T09:52:32.111 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00078161300000000001 2024-09-11T09:52:32.111 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.111 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.111 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:32.111 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.106985+0000", 2024-09-11T09:52:32.111 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.5223999999999998e-05 2024-09-11T09:52:32.111 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.111 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.111 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:32.111 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.107029+0000", 2024-09-11T09:52:32.111 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.4150999999999998e-05 2024-09-11T09:52:32.112 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:32.112 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:32.112 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:32.112 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.112 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.112 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4694.0:36 2.7 2:e8766325:ns::rbd_object_map.1256a0c02879:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:32.112 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.094452+0000", 2024-09-11T09:52:32.112 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 68.951031530999998, 2024-09-11T09:52:32.112 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.012680383, 2024-09-11T09:52:32.112 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:32.112 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:32.112 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:32.112 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:32.112 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4694", 2024-09-11T09:52:32.113 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:52:32.113 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 36 2024-09-11T09:52:32.113 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.113 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:32.113 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.113 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:32.113 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094452+0000", 2024-09-11T09:52:32.113 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:32.113 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.113 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.113 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:32.113 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094452+0000", 2024-09-11T09:52:32.113 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:32.113 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.114 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.114 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:32.114 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094453+0000", 2024-09-11T09:52:32.114 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.1800000000000002e-07 2024-09-11T09:52:32.114 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.114 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.114 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:32.114 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094455+0000", 2024-09-11T09:52:32.114 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.7020000000000001e-06 2024-09-11T09:52:32.114 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.114 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.114 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:32.114 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094463+0000", 2024-09-11T09:52:32.114 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.1999999999999997e-06 2024-09-11T09:52:32.115 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.115 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.115 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:32.115 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094467+0000", 2024-09-11T09:52:32.115 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.729e-06 2024-09-11T09:52:32.115 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.115 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.115 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:32.115 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.101269+0000", 2024-09-11T09:52:32.115 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0068018799999999997 2024-09-11T09:52:32.115 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.115 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.115 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:32.116 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.101353+0000", 2024-09-11T09:52:32.116 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.3827999999999998e-05 2024-09-11T09:52:32.116 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.116 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.116 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 5", 2024-09-11T09:52:32.116 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.101588+0000", 2024-09-11T09:52:32.116 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00023467099999999999 2024-09-11T09:52:32.116 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.116 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.116 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:32.116 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.106182+0000", 2024-09-11T09:52:32.116 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0045939489999999999 2024-09-11T09:52:32.116 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.116 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.117 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:32.117 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.107041+0000", 2024-09-11T09:52:32.117 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00085965800000000002 2024-09-11T09:52:32.117 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.117 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.117 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:32.117 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.107084+0000", 2024-09-11T09:52:32.117 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.261e-05 2024-09-11T09:52:32.117 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.117 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.117 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:32.117 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.107133+0000", 2024-09-11T09:52:32.117 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.8838000000000002e-05 2024-09-11T09:52:32.117 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:32.118 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:32.118 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:32.118 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.118 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.118 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4694.0:37 2.7 2:e8766325:ns::rbd_object_map.1256a0c02879:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:32.118 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.094473+0000", 2024-09-11T09:52:32.118 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 68.95101038, 2024-09-11T09:52:32.118 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.012778484999999999, 2024-09-11T09:52:32.118 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:32.118 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:32.118 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:32.118 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:32.118 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4694", 2024-09-11T09:52:32.119 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:52:32.119 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 37 2024-09-11T09:52:32.119 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.119 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:32.119 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.119 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:32.119 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094473+0000", 2024-09-11T09:52:32.119 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:32.119 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.119 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.119 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:32.119 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094473+0000", 2024-09-11T09:52:32.119 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:32.119 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.120 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.120 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:32.120 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094474+0000", 2024-09-11T09:52:32.120 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.9699999999999996e-07 2024-09-11T09:52:32.120 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.120 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.120 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:32.120 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094477+0000", 2024-09-11T09:52:32.120 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.151e-06 2024-09-11T09:52:32.120 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.120 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.120 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:32.120 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094483+0000", 2024-09-11T09:52:32.120 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.6330000000000001e-06 2024-09-11T09:52:32.121 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.121 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.121 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:32.121 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.094487+0000", 2024-09-11T09:52:32.121 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.5399999999999997e-06 2024-09-11T09:52:32.121 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.121 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.121 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:32.121 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.101866+0000", 2024-09-11T09:52:32.121 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.007378768 2024-09-11T09:52:32.121 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.121 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.121 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:32.121 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.101960+0000", 2024-09-11T09:52:32.122 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.3861000000000001e-05 2024-09-11T09:52:32.122 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.122 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.122 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 5", 2024-09-11T09:52:32.122 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.102197+0000", 2024-09-11T09:52:32.122 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00023723200000000001 2024-09-11T09:52:32.122 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.122 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.122 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:32.122 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.106222+0000", 2024-09-11T09:52:32.122 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0040249689999999998 2024-09-11T09:52:32.122 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.122 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.122 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:32.123 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.107141+0000", 2024-09-11T09:52:32.123 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00091913900000000002 2024-09-11T09:52:32.123 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.123 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.123 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:32.123 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.107198+0000", 2024-09-11T09:52:32.123 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.6564999999999998e-05 2024-09-11T09:52:32.123 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.123 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.123 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:32.123 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.107252+0000", 2024-09-11T09:52:32.123 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.4030000000000003e-05 2024-09-11T09:52:32.123 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:32.123 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:32.124 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:32.124 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.124 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.124 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4694.0:33 2.4 e53/32 2:3bfc818c:ns::rbd_data.1256a0c02879.0000000000000001:head v 53'14, mlcod=53'12)", 2024-09-11T09:52:32.124 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.096496+0000", 2024-09-11T09:52:32.124 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 68.948987806999995, 2024-09-11T09:52:32.124 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0099139310000000008, 2024-09-11T09:52:32.124 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:32.124 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:32.124 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:32.124 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:32.124 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.124 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:32.124 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.096496+0000", 2024-09-11T09:52:32.125 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:32.125 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.125 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.125 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:32.125 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.096496+0000", 2024-09-11T09:52:32.125 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:32.125 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.125 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.125 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:32.125 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.096498+0000", 2024-09-11T09:52:32.125 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6309999999999999e-06 2024-09-11T09:52:32.125 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.125 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.125 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:32.126 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.098087+0000", 2024-09-11T09:52:32.126 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0015889019999999999 2024-09-11T09:52:32.126 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.126 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.126 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:32.126 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.098112+0000", 2024-09-11T09:52:32.126 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.5369000000000001e-05 2024-09-11T09:52:32.126 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.126 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.126 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:32.126 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.098127+0000", 2024-09-11T09:52:32.126 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.5218999999999999e-05 2024-09-11T09:52:32.126 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.126 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.126 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:32.127 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.098209+0000", 2024-09-11T09:52:32.127 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.2120999999999995e-05 2024-09-11T09:52:32.127 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.127 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.127 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:32.127 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.098237+0000", 2024-09-11T09:52:32.127 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.7657999999999998e-05 2024-09-11T09:52:32.127 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.127 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.127 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:32.127 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.106376+0000", 2024-09-11T09:52:32.127 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0081386749999999997 2024-09-11T09:52:32.127 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.128 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.128 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:32.128 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.106410+0000", 2024-09-11T09:52:32.128 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.4356000000000001e-05 2024-09-11T09:52:32.128 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:32.128 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:32.128 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:32.128 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.128 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.128 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4694.0:49 2.7 2:e8766325:ns::rbd_object_map.1256a0c02879:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:32.128 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.134806+0000", 2024-09-11T09:52:32.128 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 68.910677738000004, 2024-09-11T09:52:32.128 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0061629049999999998, 2024-09-11T09:52:32.129 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:32.129 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:32.129 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:32.129 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:32.129 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4694", 2024-09-11T09:52:32.129 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:52:32.129 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 49 2024-09-11T09:52:32.129 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.129 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:32.129 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.129 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:32.129 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.134806+0000", 2024-09-11T09:52:32.129 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:32.129 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.130 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.130 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:32.130 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.134806+0000", 2024-09-11T09:52:32.130 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:32.130 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.130 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.130 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:32.130 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.134808+0000", 2024-09-11T09:52:32.130 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.9870000000000002e-06 2024-09-11T09:52:32.130 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.130 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.130 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:32.130 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.134818+0000", 2024-09-11T09:52:32.131 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.5599999999999999e-06 2024-09-11T09:52:32.131 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.131 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.131 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:32.131 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.134835+0000", 2024-09-11T09:52:32.131 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7399000000000001e-05 2024-09-11T09:52:32.131 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.131 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.131 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:32.131 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.134845+0000", 2024-09-11T09:52:32.131 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0285000000000001e-05 2024-09-11T09:52:32.131 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.131 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.132 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:32.132 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.134911+0000", 2024-09-11T09:52:32.132 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.5654999999999994e-05 2024-09-11T09:52:32.132 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.132 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.132 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:32.132 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.135023+0000", 2024-09-11T09:52:32.132 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00011186799999999999 2024-09-11T09:52:32.132 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.132 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.132 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 5", 2024-09-11T09:52:32.132 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.135290+0000", 2024-09-11T09:52:32.132 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00026725300000000001 2024-09-11T09:52:32.132 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.133 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.133 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:32.133 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.136506+0000", 2024-09-11T09:52:32.133 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0012161139999999999 2024-09-11T09:52:32.133 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.133 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.133 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:32.133 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.140884+0000", 2024-09-11T09:52:32.133 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0043781990000000002 2024-09-11T09:52:32.133 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.133 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.133 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:32.133 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.140923+0000", 2024-09-11T09:52:32.133 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.8813999999999998e-05 2024-09-11T09:52:32.133 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.134 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.134 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:32.134 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.140969+0000", 2024-09-11T09:52:32.134 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.5770999999999999e-05 2024-09-11T09:52:32.134 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:32.134 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:32.134 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:32.134 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.134 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.134 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4694.0:42 2.5 e53/32 2:a4fc00ec:ns::rbd_data.1256a0c02879.0000000000000004:head v 53'3, mlcod=53'1)", 2024-09-11T09:52:32.134 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.136646+0000", 2024-09-11T09:52:32.134 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 68.908837431999999, 2024-09-11T09:52:32.134 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.012545501000000001, 2024-09-11T09:52:32.135 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:32.135 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:32.135 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:32.135 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:32.135 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.135 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:32.135 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.136646+0000", 2024-09-11T09:52:32.135 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:32.135 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.135 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.135 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:32.135 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.136646+0000", 2024-09-11T09:52:32.135 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:32.135 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.135 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.136 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:32.136 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.136647+0000", 2024-09-11T09:52:32.136 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.2900000000000002e-07 2024-09-11T09:52:32.136 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.136 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.136 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:32.136 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.143620+0000", 2024-09-11T09:52:32.136 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0069727440000000003 2024-09-11T09:52:32.136 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.136 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.136 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:32.136 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.143642+0000", 2024-09-11T09:52:32.136 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.1846000000000001e-05 2024-09-11T09:52:32.137 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.137 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.137 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:32.137 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.143654+0000", 2024-09-11T09:52:32.137 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2081e-05 2024-09-11T09:52:32.137 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.137 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.137 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:32.137 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.143717+0000", 2024-09-11T09:52:32.137 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.3443999999999998e-05 2024-09-11T09:52:32.137 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.137 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.137 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:32.137 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.143740+0000", 2024-09-11T09:52:32.138 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.2626999999999999e-05 2024-09-11T09:52:32.138 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.138 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.138 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:32.138 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.149142+0000", 2024-09-11T09:52:32.138 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0054017630000000004 2024-09-11T09:52:32.138 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.138 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.138 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:32.138 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.149192+0000", 2024-09-11T09:52:32.138 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.0067e-05 2024-09-11T09:52:32.138 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:32.138 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:32.138 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:32.139 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.139 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.139 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4694.0:53 2.7 2:e8766325:ns::rbd_object_map.1256a0c02879:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:32.139 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.147978+0000", 2024-09-11T09:52:32.139 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 68.897505948000003, 2024-09-11T09:52:32.139 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0044549769999999997, 2024-09-11T09:52:32.139 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:32.139 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:32.139 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:32.139 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:32.139 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4694", 2024-09-11T09:52:32.139 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:52:32.139 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 53 2024-09-11T09:52:32.139 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.140 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:32.140 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.140 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:32.140 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.147978+0000", 2024-09-11T09:52:32.140 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:32.140 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.140 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.140 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:32.140 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.147978+0000", 2024-09-11T09:52:32.140 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:32.140 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.140 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.140 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:32.140 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.147980+0000", 2024-09-11T09:52:32.141 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.8309999999999999e-06 2024-09-11T09:52:32.141 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.141 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.141 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:32.141 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.147989+0000", 2024-09-11T09:52:32.141 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.9639999999999992e-06 2024-09-11T09:52:32.141 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.141 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.141 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:32.141 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.148007+0000", 2024-09-11T09:52:32.141 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7884999999999999e-05 2024-09-11T09:52:32.141 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.141 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.141 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:32.142 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.148017+0000", 2024-09-11T09:52:32.142 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0895e-05 2024-09-11T09:52:32.142 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.142 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.142 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:32.142 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.148091+0000", 2024-09-11T09:52:32.142 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.3709999999999997e-05 2024-09-11T09:52:32.142 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.142 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.142 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:32.142 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.148189+0000", 2024-09-11T09:52:32.142 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.7403000000000002e-05 2024-09-11T09:52:32.142 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.142 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.143 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 5", 2024-09-11T09:52:32.143 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.148504+0000", 2024-09-11T09:52:32.143 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00031589900000000002 2024-09-11T09:52:32.143 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.143 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.143 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:32.143 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.149514+0000", 2024-09-11T09:52:32.143 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0010096 2024-09-11T09:52:32.143 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.143 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.143 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:32.143 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.152367+0000", 2024-09-11T09:52:32.143 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0028532309999999999 2024-09-11T09:52:32.143 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.144 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.144 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:32.144 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.152402+0000", 2024-09-11T09:52:32.144 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.4232e-05 2024-09-11T09:52:32.144 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.144 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.144 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:32.144 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.152433+0000", 2024-09-11T09:52:32.144 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.1327e-05 2024-09-11T09:52:32.144 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:32.144 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:32.144 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:32.144 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.144 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.145 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4694.0:55 2.7 2:e8766325:ns::rbd_object_map.1256a0c02879:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:32.145 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.149838+0000", 2024-09-11T09:52:32.145 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 68.895646214999999, 2024-09-11T09:52:32.145 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0060695159999999996, 2024-09-11T09:52:32.145 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:32.145 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:32.145 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:32.145 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:32.145 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4694", 2024-09-11T09:52:32.145 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:52:32.145 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 55 2024-09-11T09:52:32.145 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.145 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:32.145 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.146 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:32.146 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.149838+0000", 2024-09-11T09:52:32.146 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:32.146 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.146 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.146 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:32.146 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.149838+0000", 2024-09-11T09:52:32.146 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:32.146 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.146 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.146 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:32.146 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.149839+0000", 2024-09-11T09:52:32.146 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.119e-06 2024-09-11T09:52:32.147 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.147 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.147 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:32.147 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.149845+0000", 2024-09-11T09:52:32.147 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.7810000000000002e-06 2024-09-11T09:52:32.147 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.147 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.147 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:32.147 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.149856+0000", 2024-09-11T09:52:32.147 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1199999999999999e-05 2024-09-11T09:52:32.147 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.147 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.147 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:32.147 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.149863+0000", 2024-09-11T09:52:32.148 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.4109999999999999e-06 2024-09-11T09:52:32.148 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.148 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.148 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:32.148 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.149908+0000", 2024-09-11T09:52:32.148 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.5184e-05 2024-09-11T09:52:32.148 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.148 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.148 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:32.148 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.149964+0000", 2024-09-11T09:52:32.148 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.5395000000000001e-05 2024-09-11T09:52:32.148 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.148 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.148 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 5", 2024-09-11T09:52:32.149 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.150157+0000", 2024-09-11T09:52:32.149 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000193489 2024-09-11T09:52:32.149 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.149 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.149 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:32.149 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.152238+0000", 2024-09-11T09:52:32.149 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0020803480000000001 2024-09-11T09:52:32.149 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.149 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.149 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:32.149 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.155828+0000", 2024-09-11T09:52:32.149 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0035902210000000002 2024-09-11T09:52:32.149 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.150 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.150 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:32.150 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.155873+0000", 2024-09-11T09:52:32.150 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.5145999999999998e-05 2024-09-11T09:52:32.150 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.150 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.150 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:32.150 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.155907+0000", 2024-09-11T09:52:32.150 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.4221999999999999e-05 2024-09-11T09:52:32.150 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:32.150 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:32.150 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:32.150 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:32.150 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:32.150 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:52:32.152 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-11T09:52:32.152 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:52:32.152+0000 7f3c96ec9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31909) UID: 0 2024-09-11T09:52:32.253 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:52:32.253+0000 7f3a7b90a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 42475) UID: 0 2024-09-11T09:52:32.300 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:52:32.300 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [], 2024-09-11T09:52:32.300 INFO:teuthology.orchestra.run.smithi017.stdout: "num_ops": 0 2024-09-11T09:52:32.300 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:52:32.309 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-11T09:52:32.353 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-09-11T09:52:32.354+0000 7f7b6c7d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31904) UID: 0 2024-09-11T09:52:32.416 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:52:32.416 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [], 2024-09-11T09:52:32.417 INFO:teuthology.orchestra.run.smithi017.stdout: "complaint_time": 30, 2024-09-11T09:52:32.417 INFO:teuthology.orchestra.run.smithi017.stdout: "num_blocked_ops": 0 2024-09-11T09:52:32.417 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:52:32.425 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-11T09:52:32.470 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:52:32.471+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:52:32.534 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:52:32.534 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 20, 2024-09-11T09:52:32.534 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 600, 2024-09-11T09:52:32.534 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [ 2024-09-11T09:52:32.534 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.534 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:34 2.4 2:2367385a:::rbd_data.1250f5e25b80.0000000000000002:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:32.534 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.897717+0000", 2024-09-11T09:52:32.535 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 69.636537292, 2024-09-11T09:52:32.535 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.021323661000000001, 2024-09-11T09:52:32.535 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:32.535 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:32.535 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:32.535 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:32.535 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:52:32.535 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:52:32.535 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 34 2024-09-11T09:52:32.535 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.535 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:32.535 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.535 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:32.535 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.897717+0000", 2024-09-11T09:52:32.535 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:32.536 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.536 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.536 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:32.536 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.897717+0000", 2024-09-11T09:52:32.536 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:32.536 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.536 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.536 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:32.536 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.897718+0000", 2024-09-11T09:52:32.536 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.1800000000000004e-07 2024-09-11T09:52:32.536 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.536 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.536 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:32.536 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.905339+0000", 2024-09-11T09:52:32.537 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0076210239999999997 2024-09-11T09:52:32.537 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.537 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.537 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:32.537 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.905366+0000", 2024-09-11T09:52:32.537 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.6959000000000001e-05 2024-09-11T09:52:32.537 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.537 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.537 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:32.537 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.905375+0000", 2024-09-11T09:52:32.537 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.516e-06 2024-09-11T09:52:32.537 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.537 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.537 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:32.537 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.905475+0000", 2024-09-11T09:52:32.538 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.9738e-05 2024-09-11T09:52:32.538 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.538 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.538 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:32.538 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.905660+0000", 2024-09-11T09:52:32.538 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00018554199999999999 2024-09-11T09:52:32.538 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.538 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.538 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 0", 2024-09-11T09:52:32.538 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.905808+0000", 2024-09-11T09:52:32.538 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000147386 2024-09-11T09:52:32.538 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.538 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.538 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:32.538 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.918230+0000", 2024-09-11T09:52:32.539 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.012421690000000001 2024-09-11T09:52:32.539 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.539 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.539 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:32.539 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.918932+0000", 2024-09-11T09:52:32.539 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00070267300000000001 2024-09-11T09:52:32.539 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.539 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.539 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:32.539 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.918989+0000", 2024-09-11T09:52:32.539 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.7268999999999997e-05 2024-09-11T09:52:32.539 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.539 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.539 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:32.539 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919040+0000", 2024-09-11T09:52:32.540 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.0946000000000003e-05 2024-09-11T09:52:32.540 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:32.540 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:32.540 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:32.540 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.540 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.540 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4688.0:31 2.1 e53/15 2:8654721b:::rbd_data.1250f5e25b80.0000000000000000:head v 53'2, mlcod=0'0)", 2024-09-11T09:52:32.540 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.904661+0000", 2024-09-11T09:52:32.540 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 69.629592802999994, 2024-09-11T09:52:32.540 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.014152804999999999, 2024-09-11T09:52:32.540 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:32.540 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:32.540 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:32.540 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:32.540 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.541 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:32.541 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.904661+0000", 2024-09-11T09:52:32.541 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:32.541 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.541 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.541 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:32.541 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.904661+0000", 2024-09-11T09:52:32.541 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:32.541 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.541 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.541 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:32.541 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.904662+0000", 2024-09-11T09:52:32.541 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0759999999999999e-06 2024-09-11T09:52:32.541 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.541 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.542 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:32.542 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.907298+0000", 2024-09-11T09:52:32.542 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0026361240000000001 2024-09-11T09:52:32.542 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.542 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.542 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:32.542 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.907320+0000", 2024-09-11T09:52:32.542 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.1268999999999999e-05 2024-09-11T09:52:32.542 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.542 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.542 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:32.542 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.907331+0000", 2024-09-11T09:52:32.542 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1031e-05 2024-09-11T09:52:32.542 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.542 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.543 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:32.543 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.907403+0000", 2024-09-11T09:52:32.543 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.2571999999999996e-05 2024-09-11T09:52:32.543 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.543 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.543 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:32.543 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.907428+0000", 2024-09-11T09:52:32.543 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.4380999999999999e-05 2024-09-11T09:52:32.543 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.543 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.543 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:32.543 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.918750+0000", 2024-09-11T09:52:32.543 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.011322211 2024-09-11T09:52:32.543 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.543 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.544 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:32.544 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.918814+0000", 2024-09-11T09:52:32.544 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.4140999999999995e-05 2024-09-11T09:52:32.544 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:32.544 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:32.544 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:32.544 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.544 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.544 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:36 2.6 2:61e1b442:::rbd_data.1250f5e25b80.0000000000000003:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:32.544 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.905404+0000", 2024-09-11T09:52:32.544 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 69.628849931999994, 2024-09-11T09:52:32.544 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.030912479999999999, 2024-09-11T09:52:32.544 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:32.545 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:32.545 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:32.545 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:32.545 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:52:32.545 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:52:32.545 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 36 2024-09-11T09:52:32.545 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.545 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:32.545 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.545 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:32.545 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.905404+0000", 2024-09-11T09:52:32.545 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:32.545 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.545 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.545 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:32.545 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.905404+0000", 2024-09-11T09:52:32.546 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:32.546 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.546 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.546 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:32.546 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.905405+0000", 2024-09-11T09:52:32.546 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2270000000000001e-06 2024-09-11T09:52:32.546 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.546 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.546 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:32.546 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.914859+0000", 2024-09-11T09:52:32.546 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0094541190000000004 2024-09-11T09:52:32.546 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.546 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.546 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:32.546 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.914879+0000", 2024-09-11T09:52:32.547 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.0052999999999999e-05 2024-09-11T09:52:32.547 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.547 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.547 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:32.547 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.914897+0000", 2024-09-11T09:52:32.547 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7331000000000001e-05 2024-09-11T09:52:32.547 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.547 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.547 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:32.547 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.915019+0000", 2024-09-11T09:52:32.547 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00012168 2024-09-11T09:52:32.547 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.547 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.547 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:32.547 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.915269+0000", 2024-09-11T09:52:32.548 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00025054899999999998 2024-09-11T09:52:32.548 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.548 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.548 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 3", 2024-09-11T09:52:32.548 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.915479+0000", 2024-09-11T09:52:32.548 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00021017799999999999 2024-09-11T09:52:32.548 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.548 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.548 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:32.548 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.926070+0000", 2024-09-11T09:52:32.548 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.010590624999999999 2024-09-11T09:52:32.548 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.548 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.548 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:32.549 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.936198+0000", 2024-09-11T09:52:32.549 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.010128194 2024-09-11T09:52:32.549 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.549 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.549 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:32.549 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.936263+0000", 2024-09-11T09:52:32.549 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.5366999999999999e-05 2024-09-11T09:52:32.549 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.549 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.549 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:32.549 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.936317+0000", 2024-09-11T09:52:32.549 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.3156999999999999e-05 2024-09-11T09:52:32.549 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:32.550 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:32.550 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:32.550 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.550 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.550 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:39 2.6 2:743e7616:::rbd_data.1250f5e25b80.0000000000000005:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:32.550 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.914960+0000", 2024-09-11T09:52:32.550 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 69.619293630000001, 2024-09-11T09:52:32.550 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.022371518, 2024-09-11T09:52:32.550 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:32.550 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:32.550 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:32.550 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:32.550 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:52:32.551 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:52:32.551 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 39 2024-09-11T09:52:32.551 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.551 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:32.551 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.551 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:32.551 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.914960+0000", 2024-09-11T09:52:32.551 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:32.551 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.551 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.551 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:32.551 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.914960+0000", 2024-09-11T09:52:32.551 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:32.551 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.552 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.552 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:32.552 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.914963+0000", 2024-09-11T09:52:32.552 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.1119999999999999e-06 2024-09-11T09:52:32.552 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.552 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.552 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:32.552 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.920417+0000", 2024-09-11T09:52:32.552 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0054543969999999997 2024-09-11T09:52:32.552 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.552 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.552 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:32.552 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.920430+0000", 2024-09-11T09:52:32.553 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3414000000000001e-05 2024-09-11T09:52:32.553 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.553 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.553 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:32.553 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.920437+0000", 2024-09-11T09:52:32.553 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.6159999999999998e-06 2024-09-11T09:52:32.553 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.553 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.553 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:32.553 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.921325+0000", 2024-09-11T09:52:32.553 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000887731 2024-09-11T09:52:32.553 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.553 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.553 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:32.554 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.921422+0000", 2024-09-11T09:52:32.554 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.7201999999999995e-05 2024-09-11T09:52:32.554 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.554 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.554 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 3", 2024-09-11T09:52:32.554 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.921511+0000", 2024-09-11T09:52:32.554 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.8880000000000003e-05 2024-09-11T09:52:32.554 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.554 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.554 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:32.554 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.928092+0000", 2024-09-11T09:52:32.554 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0065816290000000003 2024-09-11T09:52:32.554 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.554 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.554 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:32.555 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.937240+0000", 2024-09-11T09:52:32.555 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0091481070000000008 2024-09-11T09:52:32.555 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.555 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.555 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:32.555 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.937294+0000", 2024-09-11T09:52:32.555 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.3100000000000003e-05 2024-09-11T09:52:32.555 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.555 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.555 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:32.555 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.937332+0000", 2024-09-11T09:52:32.555 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.8330000000000001e-05 2024-09-11T09:52:32.555 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:32.555 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:32.555 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:32.556 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.556 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.556 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4688.0:32 2.2 e53/15 2:45941fd2:::rbd_data.1250f5e25b80.0000000000000001:head v 53'3, mlcod=17'1)", 2024-09-11T09:52:32.556 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.916509+0000", 2024-09-11T09:52:32.556 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 69.617744548000005, 2024-09-11T09:52:32.556 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.015885318999999998, 2024-09-11T09:52:32.556 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:32.556 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:32.556 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:32.556 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:32.556 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.556 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:32.556 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.916509+0000", 2024-09-11T09:52:32.556 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:32.557 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.557 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.557 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:32.557 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.916509+0000", 2024-09-11T09:52:32.557 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:32.557 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.557 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.557 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:32.557 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.916511+0000", 2024-09-11T09:52:32.557 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.052e-06 2024-09-11T09:52:32.557 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.557 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.557 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:32.557 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.927019+0000", 2024-09-11T09:52:32.557 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.010508224 2024-09-11T09:52:32.558 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.558 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.558 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:32.558 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.927038+0000", 2024-09-11T09:52:32.558 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.9590999999999999e-05 2024-09-11T09:52:32.558 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.558 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.558 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:32.558 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.927050+0000", 2024-09-11T09:52:32.558 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1439e-05 2024-09-11T09:52:32.558 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.558 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.558 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:32.558 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.927115+0000", 2024-09-11T09:52:32.559 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.5301000000000004e-05 2024-09-11T09:52:32.559 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.559 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.559 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:32.559 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.927132+0000", 2024-09-11T09:52:32.559 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7297000000000001e-05 2024-09-11T09:52:32.559 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.559 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.559 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:32.559 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.932354+0000", 2024-09-11T09:52:32.559 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0052211870000000004 2024-09-11T09:52:32.559 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.559 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.559 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:32.559 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.932395+0000", 2024-09-11T09:52:32.560 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.1227999999999999e-05 2024-09-11T09:52:32.560 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:32.560 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:32.560 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:32.560 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.560 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.560 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:44 2.6 2:67246a22:::rbd_data.1250f5e25b80.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:32.560 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.943181+0000", 2024-09-11T09:52:32.560 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 69.591073336999997, 2024-09-11T09:52:32.560 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.022629371999999998, 2024-09-11T09:52:32.560 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:32.560 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:32.560 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:32.560 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:32.561 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:52:32.561 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:52:32.561 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 44 2024-09-11T09:52:32.561 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.561 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:32.561 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.561 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:32.561 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.943181+0000", 2024-09-11T09:52:32.561 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:32.561 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.561 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.561 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:32.561 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.943181+0000", 2024-09-11T09:52:32.561 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:32.561 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.562 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.562 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:32.562 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.943182+0000", 2024-09-11T09:52:32.562 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.0500000000000002e-07 2024-09-11T09:52:32.562 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.562 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.562 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:32.562 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951240+0000", 2024-09-11T09:52:32.562 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.008058529 2024-09-11T09:52:32.562 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.562 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.562 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:32.562 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951260+0000", 2024-09-11T09:52:32.562 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.9570000000000001e-05 2024-09-11T09:52:32.563 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.563 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.563 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:32.563 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951270+0000", 2024-09-11T09:52:32.563 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0018e-05 2024-09-11T09:52:32.563 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.563 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.563 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:32.563 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951349+0000", 2024-09-11T09:52:32.563 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.8801000000000007e-05 2024-09-11T09:52:32.563 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.563 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.563 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:32.563 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951483+0000", 2024-09-11T09:52:32.563 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00013427 2024-09-11T09:52:32.564 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.564 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.564 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 3", 2024-09-11T09:52:32.564 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951593+0000", 2024-09-11T09:52:32.564 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00011031000000000001 2024-09-11T09:52:32.564 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.564 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.564 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:32.564 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.957303+0000", 2024-09-11T09:52:32.564 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0057097959999999996 2024-09-11T09:52:32.564 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.564 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.564 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:32.564 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.965690+0000", 2024-09-11T09:52:32.564 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0083867420000000008 2024-09-11T09:52:32.565 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.565 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.565 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:32.565 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.965749+0000", 2024-09-11T09:52:32.565 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.8981999999999999e-05 2024-09-11T09:52:32.565 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.565 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.565 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:32.565 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.965810+0000", 2024-09-11T09:52:32.565 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.1449000000000001e-05 2024-09-11T09:52:32.565 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:32.565 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:32.565 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:32.565 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.565 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.566 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:51 2.6 2:6b5894fc:::rbd_data.1250f5e25b80.000000000000000a:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:32.566 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.951324+0000", 2024-09-11T09:52:32.566 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 69.582930263999998, 2024-09-11T09:52:32.566 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.021375273, 2024-09-11T09:52:32.566 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:32.566 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:32.566 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:32.566 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:32.566 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:52:32.566 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:52:32.566 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 51 2024-09-11T09:52:32.566 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.566 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:32.566 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.566 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:32.567 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951324+0000", 2024-09-11T09:52:32.567 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:32.567 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.567 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.567 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:32.567 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951324+0000", 2024-09-11T09:52:32.567 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:32.567 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.567 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.567 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:32.567 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.951324+0000", 2024-09-11T09:52:32.567 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.39e-07 2024-09-11T09:52:32.567 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.567 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.568 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:32.568 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.958537+0000", 2024-09-11T09:52:32.568 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0072126769999999998 2024-09-11T09:52:32.568 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.568 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.568 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:32.568 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.958554+0000", 2024-09-11T09:52:32.568 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7232e-05 2024-09-11T09:52:32.568 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.568 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.568 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:32.568 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.958570+0000", 2024-09-11T09:52:32.568 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.5747999999999999e-05 2024-09-11T09:52:32.569 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.569 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.569 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:32.569 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.958638+0000", 2024-09-11T09:52:32.569 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.7810000000000003e-05 2024-09-11T09:52:32.569 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.569 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.569 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:32.569 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.958758+0000", 2024-09-11T09:52:32.569 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000120689 2024-09-11T09:52:32.569 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.569 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.569 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 3", 2024-09-11T09:52:32.569 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.958865+0000", 2024-09-11T09:52:32.570 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00010692 2024-09-11T09:52:32.570 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.570 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.570 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:32.570 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.970712+0000", 2024-09-11T09:52:32.570 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.011846327 2024-09-11T09:52:32.570 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.570 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.570 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:32.570 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.972603+0000", 2024-09-11T09:52:32.570 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0018918610000000001 2024-09-11T09:52:32.570 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.570 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.570 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:32.570 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.972653+0000", 2024-09-11T09:52:32.571 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.9762999999999998e-05 2024-09-11T09:52:32.571 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.571 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.571 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:32.571 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.972699+0000", 2024-09-11T09:52:32.571 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.5806999999999997e-05 2024-09-11T09:52:32.571 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:32.571 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:32.571 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:32.571 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.571 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.571 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:52 2.4 2:26d5c2f0:::rbd_data.1250f5e25b80.000000000000000b:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:32.571 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.958596+0000", 2024-09-11T09:52:32.571 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 69.575658068999999, 2024-09-11T09:52:32.572 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.016477849999999999, 2024-09-11T09:52:32.572 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:32.572 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:32.572 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:32.572 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:32.572 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:52:32.572 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:52:32.572 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 52 2024-09-11T09:52:32.572 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.572 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:32.572 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.572 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:32.572 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.958596+0000", 2024-09-11T09:52:32.573 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:32.573 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.573 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.573 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:32.573 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.958596+0000", 2024-09-11T09:52:32.573 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:32.573 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.573 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.573 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:32.573 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.958597+0000", 2024-09-11T09:52:32.573 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1960000000000001e-06 2024-09-11T09:52:32.573 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.573 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.573 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:32.574 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.963157+0000", 2024-09-11T09:52:32.574 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0045603029999999999 2024-09-11T09:52:32.574 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.574 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.574 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:32.574 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.963171+0000", 2024-09-11T09:52:32.574 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3777999999999999e-05 2024-09-11T09:52:32.574 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.574 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.574 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:32.574 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.963181+0000", 2024-09-11T09:52:32.574 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.9010000000000006e-06 2024-09-11T09:52:32.574 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.575 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.575 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:32.575 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.963246+0000", 2024-09-11T09:52:32.575 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.5351000000000005e-05 2024-09-11T09:52:32.575 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.575 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.575 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:32.575 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.963419+0000", 2024-09-11T09:52:32.575 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00017299700000000001 2024-09-11T09:52:32.575 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.575 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.575 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 0", 2024-09-11T09:52:32.576 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.963570+0000", 2024-09-11T09:52:32.576 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00015066999999999999 2024-09-11T09:52:32.576 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.576 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.576 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:32.576 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.973881+0000", 2024-09-11T09:52:32.576 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.010311268 2024-09-11T09:52:32.576 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.576 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.576 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:32.576 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.974963+0000", 2024-09-11T09:52:32.576 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.001081493 2024-09-11T09:52:32.576 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.577 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.577 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:32.577 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.975012+0000", 2024-09-11T09:52:32.577 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.9471000000000001e-05 2024-09-11T09:52:32.577 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.577 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.577 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:32.577 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.975074+0000", 2024-09-11T09:52:32.577 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.1421999999999996e-05 2024-09-11T09:52:32.577 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:32.578 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:32.578 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:32.578 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.578 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.578 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4688.0:47 2.1 e53/15 2:95f53b3e:::rbd_data.1250f5e25b80.0000000000000008:head v 53'3, mlcod=53'1)", 2024-09-11T09:52:32.578 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.960861+0000", 2024-09-11T09:52:32.578 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 69.573392651000006, 2024-09-11T09:52:32.578 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.012346107, 2024-09-11T09:52:32.578 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:32.578 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:32.578 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:32.578 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:32.578 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.578 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:32.579 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.960861+0000", 2024-09-11T09:52:32.579 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:32.579 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.579 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.579 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:32.579 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.960861+0000", 2024-09-11T09:52:32.579 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:32.579 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.579 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.579 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:32.579 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.960863+0000", 2024-09-11T09:52:32.579 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7150000000000001e-06 2024-09-11T09:52:32.579 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.580 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.580 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:32.580 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.962428+0000", 2024-09-11T09:52:32.580 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.001564668 2024-09-11T09:52:32.580 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.580 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.580 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:32.580 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.962446+0000", 2024-09-11T09:52:32.580 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.8476999999999999e-05 2024-09-11T09:52:32.580 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.580 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.580 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:32.580 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.962458+0000", 2024-09-11T09:52:32.580 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1263999999999999e-05 2024-09-11T09:52:32.581 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.581 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.581 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:32.581 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.962575+0000", 2024-09-11T09:52:32.581 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000117025 2024-09-11T09:52:32.581 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.581 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.581 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:32.581 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.962601+0000", 2024-09-11T09:52:32.581 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.5984999999999999e-05 2024-09-11T09:52:32.581 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.581 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.581 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:32.582 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.973065+0000", 2024-09-11T09:52:32.582 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.010464753 2024-09-11T09:52:32.582 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.582 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.582 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:32.582 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.973207+0000", 2024-09-11T09:52:32.582 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00014222 2024-09-11T09:52:32.582 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:32.582 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:32.582 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:32.582 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.582 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.582 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:54 2.4 2.da8979c4 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:32.582 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.963207+0000", 2024-09-11T09:52:32.583 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 69.571047213, 2024-09-11T09:52:32.583 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.010641575, 2024-09-11T09:52:32.583 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:32.583 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:32.583 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "reached pg", 2024-09-11T09:52:32.583 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:32.583 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:52:32.583 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:52:32.583 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 54 2024-09-11T09:52:32.583 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.583 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:32.583 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.583 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:32.584 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.963207+0000", 2024-09-11T09:52:32.584 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:32.584 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.584 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.584 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:32.584 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.963207+0000", 2024-09-11T09:52:32.584 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:32.584 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.584 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.584 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:32.584 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.963208+0000", 2024-09-11T09:52:32.584 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2979999999999999e-06 2024-09-11T09:52:32.584 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.585 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.585 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:32.585 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.967419+0000", 2024-09-11T09:52:32.585 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0042111750000000002 2024-09-11T09:52:32.585 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.585 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.585 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:32.585 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.967439+0000", 2024-09-11T09:52:32.585 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.0129999999999999e-05 2024-09-11T09:52:32.585 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.585 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.585 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:32.586 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.967453+0000", 2024-09-11T09:52:32.586 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3781000000000001e-05 2024-09-11T09:52:32.586 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.586 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.586 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:32.586 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.973840+0000", 2024-09-11T09:52:32.586 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0063872269999999997 2024-09-11T09:52:32.586 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.586 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.586 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:32.586 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.973848+0000", 2024-09-11T09:52:32.586 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.9640000000000003e-06 2024-09-11T09:52:32.586 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:32.586 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:32.587 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:32.587 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.587 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.587 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:54 2.4 2:239e915b:::rbd_data.1250f5e25b80.000000000000000c:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:32.587 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.982401+0000", 2024-09-11T09:52:32.587 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 69.551853285999997, 2024-09-11T09:52:32.587 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.016817347, 2024-09-11T09:52:32.587 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:32.587 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:32.587 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:32.587 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:32.587 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:52:32.587 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:52:32.587 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 54 2024-09-11T09:52:32.588 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.588 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:32.588 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.588 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:32.588 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.982401+0000", 2024-09-11T09:52:32.588 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:32.588 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.588 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.588 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:32.588 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.982401+0000", 2024-09-11T09:52:32.588 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:32.588 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.588 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.588 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:32.588 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.982402+0000", 2024-09-11T09:52:32.589 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.8999999999999995e-07 2024-09-11T09:52:32.589 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.589 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.589 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:32.589 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.986535+0000", 2024-09-11T09:52:32.589 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0041333689999999996 2024-09-11T09:52:32.589 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.589 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.589 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:32.589 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.986555+0000", 2024-09-11T09:52:32.589 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.9851e-05 2024-09-11T09:52:32.589 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.589 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.590 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:32.590 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.986569+0000", 2024-09-11T09:52:32.590 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3784000000000001e-05 2024-09-11T09:52:32.590 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.590 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.590 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:32.590 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.986683+0000", 2024-09-11T09:52:32.590 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000114474 2024-09-11T09:52:32.590 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.590 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.590 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:32.590 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.986815+0000", 2024-09-11T09:52:32.590 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00013217799999999999 2024-09-11T09:52:32.591 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.591 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.591 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 0", 2024-09-11T09:52:32.591 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.986918+0000", 2024-09-11T09:52:32.591 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00010235599999999999 2024-09-11T09:52:32.591 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.591 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.591 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:32.591 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.998983+0000", 2024-09-11T09:52:32.591 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.012065352 2024-09-11T09:52:32.591 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.591 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.591 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:32.591 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.999081+0000", 2024-09-11T09:52:32.592 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.8091000000000007e-05 2024-09-11T09:52:32.592 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.592 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.592 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:32.592 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.999159+0000", 2024-09-11T09:52:32.592 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.7823000000000004e-05 2024-09-11T09:52:32.592 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.592 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.592 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:32.592 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.999218+0000", 2024-09-11T09:52:32.592 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.9178999999999997e-05 2024-09-11T09:52:32.592 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:32.592 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:32.593 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:32.593 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.593 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.593 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:56 2.4 2:2167012f:::rbd_data.1250f5e25b80.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:32.593 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.986661+0000", 2024-09-11T09:52:32.593 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 69.547592832999996, 2024-09-11T09:52:32.593 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.017353539000000001, 2024-09-11T09:52:32.593 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:32.593 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:32.593 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:32.593 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:32.593 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:52:32.593 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:52:32.594 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 56 2024-09-11T09:52:32.594 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.594 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:32.594 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.594 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:32.594 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.986661+0000", 2024-09-11T09:52:32.594 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:32.594 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.594 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.594 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:32.594 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.986661+0000", 2024-09-11T09:52:32.594 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:32.594 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.594 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.595 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:32.595 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.986663+0000", 2024-09-11T09:52:32.595 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.8989999999999999e-06 2024-09-11T09:52:32.595 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.595 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.595 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:32.595 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.990444+0000", 2024-09-11T09:52:32.595 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0037808569999999999 2024-09-11T09:52:32.595 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.595 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.595 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:32.595 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.990462+0000", 2024-09-11T09:52:32.595 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.8450000000000001e-05 2024-09-11T09:52:32.595 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.596 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.596 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:32.596 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.990476+0000", 2024-09-11T09:52:32.596 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3565e-05 2024-09-11T09:52:32.596 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.596 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.596 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:32.596 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.993582+0000", 2024-09-11T09:52:32.596 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0031059109999999998 2024-09-11T09:52:32.596 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.596 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.596 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:32.596 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.993686+0000", 2024-09-11T09:52:32.596 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000104424 2024-09-11T09:52:32.597 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.597 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.597 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 0", 2024-09-11T09:52:32.597 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.993799+0000", 2024-09-11T09:52:32.597 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000112985 2024-09-11T09:52:32.597 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.597 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.597 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:32.597 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.002449+0000", 2024-09-11T09:52:32.597 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.008650151 2024-09-11T09:52:32.597 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.597 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.597 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:32.598 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.003948+0000", 2024-09-11T09:52:32.598 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.001498636 2024-09-11T09:52:32.598 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.598 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.598 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:32.598 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.003985+0000", 2024-09-11T09:52:32.598 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.7054999999999997e-05 2024-09-11T09:52:32.598 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.598 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.598 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:32.598 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.004015+0000", 2024-09-11T09:52:32.598 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.9606000000000001e-05 2024-09-11T09:52:32.598 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:32.598 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:32.599 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:32.599 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.599 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.599 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:58 2.6 2:6277c644:::rbd_data.1250f5e25b80.000000000000000e:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:32.599 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.990540+0000", 2024-09-11T09:52:32.599 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 69.543714038000005, 2024-09-11T09:52:32.599 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.016232068999999998, 2024-09-11T09:52:32.599 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:32.599 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:32.599 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:32.599 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:32.599 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:52:32.599 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:52:32.599 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 58 2024-09-11T09:52:32.600 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.600 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:32.600 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.600 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:32.600 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.990540+0000", 2024-09-11T09:52:32.600 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:32.600 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.600 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.600 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:32.600 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.990540+0000", 2024-09-11T09:52:32.600 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:32.600 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.600 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.600 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:32.601 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.990541+0000", 2024-09-11T09:52:32.601 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2359999999999999e-06 2024-09-11T09:52:32.601 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.601 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.601 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:32.601 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.994626+0000", 2024-09-11T09:52:32.601 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.004084865 2024-09-11T09:52:32.601 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.601 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.601 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:32.601 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.994644+0000", 2024-09-11T09:52:32.601 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7866000000000001e-05 2024-09-11T09:52:32.601 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.602 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.602 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:32.602 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.994653+0000", 2024-09-11T09:52:32.602 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.2159999999999995e-06 2024-09-11T09:52:32.602 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.602 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.602 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:32.602 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.994732+0000", 2024-09-11T09:52:32.602 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.8954000000000003e-05 2024-09-11T09:52:32.602 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.602 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.602 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:32.602 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.994906+0000", 2024-09-11T09:52:32.602 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0001741 2024-09-11T09:52:32.603 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.603 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.603 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 3", 2024-09-11T09:52:32.603 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.995083+0000", 2024-09-11T09:52:32.603 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00017637100000000001 2024-09-11T09:52:32.603 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.603 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.603 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:32.603 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.003066+0000", 2024-09-11T09:52:32.603 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0079833360000000006 2024-09-11T09:52:32.603 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.603 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.603 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:32.603 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.006634+0000", 2024-09-11T09:52:32.604 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.003568309 2024-09-11T09:52:32.604 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.604 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.604 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:32.604 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.006716+0000", 2024-09-11T09:52:32.604 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.1377000000000004e-05 2024-09-11T09:52:32.604 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.604 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.604 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:32.604 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.006772+0000", 2024-09-11T09:52:32.604 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.6439e-05 2024-09-11T09:52:32.604 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:32.605 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:32.605 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:32.605 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.605 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.605 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:59 2.4 2:2906e6a7:::rbd_header.1250f5e25b80:head [watch reconnect cookie 140424358411408 gen 2] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:32.605 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.994684+0000", 2024-09-11T09:52:32.605 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 69.539570406999999, 2024-09-11T09:52:32.605 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0094134349999999995, 2024-09-11T09:52:32.605 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:32.605 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:32.605 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "started", 2024-09-11T09:52:32.605 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:32.605 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:52:32.606 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:52:32.606 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 59 2024-09-11T09:52:32.606 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.606 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:32.606 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.606 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:32.606 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.994684+0000", 2024-09-11T09:52:32.606 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:32.606 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.606 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.606 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:32.606 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.994684+0000", 2024-09-11T09:52:32.606 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:32.606 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.606 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.607 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:32.607 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.994686+0000", 2024-09-11T09:52:32.607 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.8159999999999998e-06 2024-09-11T09:52:32.607 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.607 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.607 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:32.607 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.994691+0000", 2024-09-11T09:52:32.607 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.5009999999999998e-06 2024-09-11T09:52:32.607 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.607 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.607 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:32.607 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.994698+0000", 2024-09-11T09:52:32.607 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.3039999999999999e-06 2024-09-11T09:52:32.608 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.608 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.608 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:32.608 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.994703+0000", 2024-09-11T09:52:32.608 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.5190000000000003e-06 2024-09-11T09:52:32.608 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.608 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.608 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:32.608 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.998678+0000", 2024-09-11T09:52:32.608 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0039752340000000002 2024-09-11T09:52:32.608 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.608 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.608 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:32.609 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.998748+0000", 2024-09-11T09:52:32.609 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.0114e-05 2024-09-11T09:52:32.609 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.609 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.609 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:32.609 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.004097+0000", 2024-09-11T09:52:32.609 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0053489469999999997 2024-09-11T09:52:32.609 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:32.609 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:32.609 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:32.609 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.609 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.609 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4694.0:33 2.4 2:3bfc818c:ns::rbd_data.1256a0c02879.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:32.609 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.084537+0000", 2024-09-11T09:52:32.610 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 69.449716979000002, 2024-09-11T09:52:32.610 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.022075773999999999, 2024-09-11T09:52:32.610 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:32.610 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:32.610 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:32.610 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:32.610 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4694", 2024-09-11T09:52:32.610 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:52:32.610 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 33 2024-09-11T09:52:32.610 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.610 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:32.610 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.610 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:32.611 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.084537+0000", 2024-09-11T09:52:32.611 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:32.611 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.611 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.611 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:32.611 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.084537+0000", 2024-09-11T09:52:32.611 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:32.611 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.611 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.611 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:32.611 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.084538+0000", 2024-09-11T09:52:32.611 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.5000000000000001e-07 2024-09-11T09:52:32.611 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.612 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.612 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:32.612 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.095984+0000", 2024-09-11T09:52:32.612 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.011446543 2024-09-11T09:52:32.612 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.612 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.612 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:32.612 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.096014+0000", 2024-09-11T09:52:32.612 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.9745000000000001e-05 2024-09-11T09:52:32.612 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.612 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.612 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:32.612 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.096030+0000", 2024-09-11T09:52:32.613 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6152999999999999e-05 2024-09-11T09:52:32.613 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.613 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.613 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:32.613 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.096113+0000", 2024-09-11T09:52:32.613 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.2731999999999993e-05 2024-09-11T09:52:32.613 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.613 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.613 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:32.613 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.096273+0000", 2024-09-11T09:52:32.613 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00015973699999999999 2024-09-11T09:52:32.613 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.613 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.613 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 0", 2024-09-11T09:52:32.614 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.096394+0000", 2024-09-11T09:52:32.614 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000120883 2024-09-11T09:52:32.614 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.614 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.614 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:32.614 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.106048+0000", 2024-09-11T09:52:32.614 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0096548020000000005 2024-09-11T09:52:32.614 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.614 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.614 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:32.614 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.106531+0000", 2024-09-11T09:52:32.614 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00048264299999999999 2024-09-11T09:52:32.614 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.614 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.615 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:32.615 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.106565+0000", 2024-09-11T09:52:32.615 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.4343e-05 2024-09-11T09:52:32.615 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.615 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.615 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:32.615 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.106613+0000", 2024-09-11T09:52:32.615 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.7342999999999997e-05 2024-09-11T09:52:32.615 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:32.615 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:32.615 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:32.615 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.615 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.616 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4694.0:32 2.1 e53/15 2:8a862b48:ns::rbd_data.1256a0c02879.0000000000000000:head v 53'8, mlcod=53'6)", 2024-09-11T09:52:32.616 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.091418+0000", 2024-09-11T09:52:32.616 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 69.442835810000005, 2024-09-11T09:52:32.616 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.013027372000000001, 2024-09-11T09:52:32.616 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:32.616 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:32.616 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:32.616 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:32.616 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.616 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:32.616 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.091418+0000", 2024-09-11T09:52:32.616 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:32.616 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.617 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.617 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:32.617 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.091418+0000", 2024-09-11T09:52:32.617 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:32.617 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.617 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.617 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:32.617 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.091420+0000", 2024-09-11T09:52:32.617 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6899999999999999e-06 2024-09-11T09:52:32.617 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.617 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.617 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:32.617 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.093086+0000", 2024-09-11T09:52:32.618 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0016662510000000001 2024-09-11T09:52:32.618 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.618 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.618 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:32.618 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.093107+0000", 2024-09-11T09:52:32.618 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.0999e-05 2024-09-11T09:52:32.618 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.618 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.618 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:32.618 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.093120+0000", 2024-09-11T09:52:32.618 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2544999999999999e-05 2024-09-11T09:52:32.618 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.618 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.618 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:32.618 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.093186+0000", 2024-09-11T09:52:32.619 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.6595999999999997e-05 2024-09-11T09:52:32.619 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.619 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.619 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:32.619 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.093208+0000", 2024-09-11T09:52:32.619 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.1624000000000002e-05 2024-09-11T09:52:32.619 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.619 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.619 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:32.619 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.104392+0000", 2024-09-11T09:52:32.619 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.011184265000000001 2024-09-11T09:52:32.619 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.619 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.619 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:32.620 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.104446+0000", 2024-09-11T09:52:32.620 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.3402000000000001e-05 2024-09-11T09:52:32.620 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:32.620 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:32.620 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:32.620 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.620 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.620 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4694.0:45 2.6 2:671fb830:ns::rbd_data.1256a0c02879.0000000000000007:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:32.620 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.109629+0000", 2024-09-11T09:52:32.620 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 69.424624969000007, 2024-09-11T09:52:32.620 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.036403143999999998, 2024-09-11T09:52:32.620 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:32.620 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:32.620 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:32.621 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:32.621 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4694", 2024-09-11T09:52:32.621 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:52:32.621 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 45 2024-09-11T09:52:32.621 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.621 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:32.621 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.621 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:32.621 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.109629+0000", 2024-09-11T09:52:32.621 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:32.621 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.621 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.621 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:32.622 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.109629+0000", 2024-09-11T09:52:32.622 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:32.622 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.622 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.622 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:32.622 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.109630+0000", 2024-09-11T09:52:32.622 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1200000000000001e-06 2024-09-11T09:52:32.622 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.622 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.622 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:32.622 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.121370+0000", 2024-09-11T09:52:32.622 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.011739571000000001 2024-09-11T09:52:32.622 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.622 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.623 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:32.623 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.121391+0000", 2024-09-11T09:52:32.623 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.1175e-05 2024-09-11T09:52:32.623 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.623 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.623 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:32.623 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.121404+0000", 2024-09-11T09:52:32.623 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2807e-05 2024-09-11T09:52:32.623 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.623 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.623 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:32.623 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.121498+0000", 2024-09-11T09:52:32.623 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.3770999999999994e-05 2024-09-11T09:52:32.623 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.624 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.624 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:32.624 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.121612+0000", 2024-09-11T09:52:32.624 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000114666 2024-09-11T09:52:32.624 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.624 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.624 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 3", 2024-09-11T09:52:32.624 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.121711+0000", 2024-09-11T09:52:32.624 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.8647999999999995e-05 2024-09-11T09:52:32.624 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.624 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.624 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:32.624 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.126558+0000", 2024-09-11T09:52:32.625 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0048473769999999999 2024-09-11T09:52:32.625 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.625 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.625 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:32.625 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.145912+0000", 2024-09-11T09:52:32.625 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.019354281000000001 2024-09-11T09:52:32.625 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.625 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.625 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:32.625 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.145977+0000", 2024-09-11T09:52:32.625 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.4135999999999997e-05 2024-09-11T09:52:32.625 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.625 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.625 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:32.626 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.146032+0000", 2024-09-11T09:52:32.626 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.5591999999999999e-05 2024-09-11T09:52:32.626 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:32.626 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:32.626 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:32.626 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.626 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.626 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4694.0:41 2.2 e53/15 2:5e2505c4:ns::rbd_data.1256a0c02879.0000000000000003:head v 53'4, mlcod=18'2)", 2024-09-11T09:52:32.626 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.113199+0000", 2024-09-11T09:52:32.626 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 69.421054558999998, 2024-09-11T09:52:32.626 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.011404512, 2024-09-11T09:52:32.626 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:32.626 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:32.626 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:32.626 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:32.627 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.627 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:32.627 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.113199+0000", 2024-09-11T09:52:32.627 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:32.627 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.627 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.627 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:32.627 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.113199+0000", 2024-09-11T09:52:32.627 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:32.627 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.627 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.627 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:32.627 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.113200+0000", 2024-09-11T09:52:32.627 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.2799999999999995e-07 2024-09-11T09:52:32.628 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.628 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.628 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:32.628 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.118872+0000", 2024-09-11T09:52:32.628 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0056718089999999999 2024-09-11T09:52:32.628 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.628 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.628 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:32.628 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.118889+0000", 2024-09-11T09:52:32.628 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7428999999999999e-05 2024-09-11T09:52:32.628 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.628 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.628 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:32.628 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.118900+0000", 2024-09-11T09:52:32.629 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0243e-05 2024-09-11T09:52:32.629 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.629 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.629 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:32.629 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.118959+0000", 2024-09-11T09:52:32.629 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.9364000000000003e-05 2024-09-11T09:52:32.629 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.629 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.629 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:32.629 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.118976+0000", 2024-09-11T09:52:32.629 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7124000000000001e-05 2024-09-11T09:52:32.629 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.629 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.629 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:32.630 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.124557+0000", 2024-09-11T09:52:32.630 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0055812659999999997 2024-09-11T09:52:32.630 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.630 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.630 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:32.630 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.124604+0000", 2024-09-11T09:52:32.630 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.6548999999999998e-05 2024-09-11T09:52:32.630 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:32.630 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:32.630 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:32.630 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.630 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.630 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4694.0:43 2.0 e53/15 2:16c8ae84:ns::rbd_data.1256a0c02879.0000000000000005:head v 53'2, mlcod=0'0)", 2024-09-11T09:52:32.630 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.158559+0000", 2024-09-11T09:52:32.631 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 69.375695233000002, 2024-09-11T09:52:32.631 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.016713677999999999, 2024-09-11T09:52:32.631 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:32.631 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:32.631 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:32.631 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:32.631 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.631 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:32.631 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.158559+0000", 2024-09-11T09:52:32.631 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:32.631 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.631 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.631 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:32.632 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.158559+0000", 2024-09-11T09:52:32.632 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:32.632 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.632 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.632 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:32.632 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.158560+0000", 2024-09-11T09:52:32.632 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1459999999999999e-06 2024-09-11T09:52:32.632 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.632 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.632 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:32.632 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.166465+0000", 2024-09-11T09:52:32.632 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.007904767 2024-09-11T09:52:32.632 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.632 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.633 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:32.633 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.166491+0000", 2024-09-11T09:52:32.633 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.6308e-05 2024-09-11T09:52:32.633 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.633 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.633 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:32.633 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.166502+0000", 2024-09-11T09:52:32.633 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0736e-05 2024-09-11T09:52:32.633 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.633 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.633 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:32.633 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.166564+0000", 2024-09-11T09:52:32.633 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.2578999999999998e-05 2024-09-11T09:52:32.634 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.634 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.634 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:32.634 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.166585+0000", 2024-09-11T09:52:32.634 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.0395999999999999e-05 2024-09-11T09:52:32.634 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.634 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.634 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:32.634 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.175227+0000", 2024-09-11T09:52:32.634 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0086426420000000007 2024-09-11T09:52:32.634 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.634 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.634 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:32.634 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.175272+0000", 2024-09-11T09:52:32.635 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.5104e-05 2024-09-11T09:52:32.635 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:32.635 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:32.635 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:32.635 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.635 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.635 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4694.0:59 2.2 e53/15 2:41adefc9:ns::rbd_data.1256a0c02879.000000000000000e:head v 53'5, mlcod=53'3)", 2024-09-11T09:52:32.635 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.163389+0000", 2024-09-11T09:52:32.635 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 69.370865257000005, 2024-09-11T09:52:32.635 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.013095719, 2024-09-11T09:52:32.635 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:32.635 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:32.635 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:32.635 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:32.635 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.636 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:32.636 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.163389+0000", 2024-09-11T09:52:32.636 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:32.636 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.636 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.636 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:32.636 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.163389+0000", 2024-09-11T09:52:32.636 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:32.636 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.636 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.636 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:32.636 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.163390+0000", 2024-09-11T09:52:32.636 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.336e-06 2024-09-11T09:52:32.637 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.637 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.637 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:32.637 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.170335+0000", 2024-09-11T09:52:32.637 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0069451080000000002 2024-09-11T09:52:32.637 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.637 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.637 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:32.637 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.170354+0000", 2024-09-11T09:52:32.637 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.8423e-05 2024-09-11T09:52:32.637 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.637 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.637 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:32.637 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.170366+0000", 2024-09-11T09:52:32.638 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2235999999999999e-05 2024-09-11T09:52:32.638 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.638 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.638 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:32.638 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.170439+0000", 2024-09-11T09:52:32.638 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.2989000000000002e-05 2024-09-11T09:52:32.638 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.638 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.638 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:32.638 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.170460+0000", 2024-09-11T09:52:32.638 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.1211000000000001e-05 2024-09-11T09:52:32.638 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.638 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.638 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:32.638 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.176442+0000", 2024-09-11T09:52:32.639 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0059821969999999999 2024-09-11T09:52:32.639 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:32.639 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:32.639 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:32.639 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.176484+0000", 2024-09-11T09:52:32.639 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.2218999999999997e-05 2024-09-11T09:52:32.639 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:32.639 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:32.639 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:32.639 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:32.639 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:32.639 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:52:32.641 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-11T09:52:32.641 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:52:32.641+0000 7f3a7b90a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 42475) UID: 0 2024-09-11T09:52:32.741 INFO:tasks.ceph.osd.2.smithi017.stderr:2024-09-11T09:52:32.742+0000 7fd20b170640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31672) UID: 0 2024-09-11T09:52:32.789 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:52:32.789 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [], 2024-09-11T09:52:32.789 INFO:teuthology.orchestra.run.smithi017.stdout: "num_ops": 0 2024-09-11T09:52:32.789 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:52:32.797 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-11T09:52:32.871 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-11T09:52:32.883 INFO:tasks.ceph.osd.0.smithi017.stderr:2024-09-11T09:52:32.884+0000 7fb73c0f1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31665) UID: 0 2024-09-11T09:52:32.943 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:52:32.944+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:52:32.949 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-11T09:52:32.953 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:52:32.953 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [], 2024-09-11T09:52:32.953 INFO:teuthology.orchestra.run.smithi017.stdout: "complaint_time": 30, 2024-09-11T09:52:32.953 INFO:teuthology.orchestra.run.smithi017.stdout: "num_blocked_ops": 0 2024-09-11T09:52:32.953 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:52:32.964 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-11T09:52:33.018 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-11T09:52:33.018 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T09:52:33.040 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-11T09:52:33.040 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T09:52:33.044 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:52:33.045+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:52:33.058 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-11T09:52:33.058 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T09:52:33.076 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-11T09:52:33.077 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T09:52:33.094 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-11T09:52:33.094 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T09:52:33.111 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-11T09:52:33.112 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T09:52:33.133 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:52:33.133 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:52:33.142 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-11T09:52:33.183 INFO:tasks.ceph.osd.2.smithi017.stderr:2024-09-11T09:52:33.184+0000 7fd20b170640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31672) UID: 0 2024-09-11T09:52:33.246 INFO:tasks.ceph.osd.0.smithi017.stderr:2024-09-11T09:52:33.247+0000 7fb73c0f1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31665) UID: 0 2024-09-11T09:52:33.248 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:52:33.248 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:52:33.257 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-11T09:52:33.347 INFO:tasks.ceph.osd.2.smithi017.stderr:2024-09-11T09:52:33.348+0000 7fd20b170640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31672) UID: 0 2024-09-11T09:52:33.359 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:52:33.360 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:52:33.368 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-11T09:52:33.448 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:52:33.448+0000 7f3a7b90a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 42475) UID: 0 2024-09-11T09:52:33.472 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:52:33.472 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:52:33.481 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-11T09:52:33.549 INFO:tasks.ceph.osd.0.smithi017.stderr:2024-09-11T09:52:33.550+0000 7fb73c0f1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31665) UID: 0 2024-09-11T09:52:33.624 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:52:33.624 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:52:33.632 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-11T09:52:33.676 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-09-11T09:52:33.677+0000 7f7b6c7d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31904) UID: 0 2024-09-11T09:52:33.735 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:52:33.735 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:52:33.744 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-11T09:52:33.750 INFO:tasks.ceph.osd.2.smithi017.stderr:2024-09-11T09:52:33.751+0000 7fd20b170640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31672) UID: 0 2024-09-11T09:52:33.852 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:52:33.853+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:52:33.887 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:52:33.887 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:52:33.896 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-11T09:52:33.939 INFO:teuthology.orchestra.run.smithi017.stderr:noscrub is unset 2024-09-11T09:52:33.953 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:52:33.954+0000 7f3a7b90a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 42475) UID: 0 2024-09-11T09:52:33.999 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:52:33.999 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:52:34.008 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-11T09:52:34.054 INFO:tasks.ceph.osd.0.smithi017.stderr:2024-09-11T09:52:34.055+0000 7fb73c0f1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31665) UID: 0 2024-09-11T09:52:34.151 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:52:34.151 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:52:34.155 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:52:34.156+0000 7f3a7b90a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 42475) UID: 0 2024-09-11T09:52:34.159 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-11T09:52:34.243 INFO:tasks.workunit.client.0.smithi107.stderr: Removing all snapshots: 100% complete... Removing all snapshots: 100% complete...done. 2024-09-11T09:52:34.256 INFO:tasks.workunit.client.0.smithi107.stderr:+ rbd -p rbd --namespace ns remove testrbdnbd27121 2024-09-11T09:52:34.256 INFO:tasks.ceph.osd.0.smithi017.stderr:2024-09-11T09:52:34.257+0000 7fb73c0f1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31665) UID: 0 2024-09-11T09:52:34.310 INFO:teuthology.orchestra.run.smithi096.stdout:ERROR: (22) Invalid argument 2024-09-11T09:52:34.310 INFO:teuthology.orchestra.run.smithi096.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:52:34.320 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-11T09:52:34.362 INFO:tasks.workunit.client.0.smithi107.stderr:2024-09-11T09:52:34.362+0000 7fd7c98d7640 0 -- 172.21.15.107:0/486547629 >> [v2:172.21.15.96:6816/332854807,v1:172.21.15.96:6817/332854807] conn(0x55ca90a89ac0 msgr2=0x55ca90a8beb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-11T09:52:34.364 INFO:tasks.ceph.osd.2.smithi017.stderr:2024-09-11T09:52:34.365+0000 7fd20b170640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 31672) UID: 0 2024-09-11T09:52:34.374 INFO:tasks.workunit.client.0.smithi107.stderr: Removing image: 6% complete... Removing image: 12% complete... Removing image: 18% complete... Removing image: 25% complete... Removing image: 31% complete... Removing image: 37% complete... Removing image: 43% complete... Removing image: 50% complete... Removing image: 56% complete... Removing image: 62% complete... Removing image: 68% complete... Removing image: 75% complete... Removing image: 81% complete... Removing image: 87% complete... Removing image: 93% complete...2024-09-11T09:52:34.375+0000 7fd7ca0d8640 0 -- 172.21.15.107:0/486547629 >> [v2:172.21.15.96:6808/509935370,v1:172.21.15.96:6809/509935370] conn(0x55ca90adce60 msgr2=0x55ca90adf2f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-11T09:52:34.389 INFO:tasks.workunit.client.0.smithi107.stderr: Removing image: 100% complete...done. 2024-09-11T09:52:34.392 INFO:tasks.workunit.client.0.smithi107.stderr:+ rbd namespace remove rbd/ns 2024-09-11T09:52:34.438 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:52:34.438 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:52:34.439 INFO:tasks.workunit.client.0.smithi107.stderr:+ rbd config global rm global rbd_default_pool 2024-09-11T09:52:34.447 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-11T09:52:34.458 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:52:34.459+0000 7f3c96ec9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31909) UID: 0 2024-09-11T09:52:34.474 INFO:tasks.workunit.client.0.smithi107.stderr:+ ceph osd pool delete new_default_pool27121 new_default_pool27121 --yes-i-really-really-mean-it 2024-09-11T09:52:34.559 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-09-11T09:52:34.560+0000 7f7b6c7d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31904) UID: 0 2024-09-11T09:52:34.595 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:52:34.595 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:52:34.604 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-11T09:52:34.659 INFO:tasks.ceph.osd.0.smithi017.stderr:2024-09-11T09:52:34.660+0000 7fb73c0f1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31665) UID: 0 2024-09-11T09:52:34.711 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:52:34.711 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:52:34.720 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-11T09:52:34.761 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:52:34.761+0000 7f3a7b90a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 42475) UID: 0 2024-09-11T09:52:34.861 INFO:tasks.ceph.osd.1.smithi017.stderr:2024-09-11T09:52:34.862+0000 7f61ad460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 31670) UID: 0 2024-09-11T09:52:34.868 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:52:34.868 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:52:34.877 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-11T09:52:34.962 INFO:tasks.ceph.osd.3.smithi096.stderr:2024-09-11T09:52:34.963+0000 7f7b6c7d0640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 31904) UID: 0 2024-09-11T09:52:35.029 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:52:35.029 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:52:35.037 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-11T09:52:35.048 INFO:tasks.workunit.client.0.smithi107.stderr:pool 'new_default_pool27121' does not exist 2024-09-11T09:52:35.060 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-11T09:52:35.061 DEBUG:teuthology.orchestra.run.smithi107:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-11T09:52:35.063 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:52:35.064+0000 7f3c96ec9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31909) UID: 0 2024-09-11T09:52:35.124 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-11T09:52:35.126 INFO:tasks.workunit:Stopping ['rbd/rbd-nbd.sh'] on client.0... 2024-09-11T09:52:35.126 DEBUG:teuthology.orchestra.run.smithi107:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-09-11T09:52:35.145 INFO:teuthology.orchestra.run.smithi017.stdout:ERROR: (22) Invalid argument 2024-09-11T09:52:35.146 INFO:teuthology.orchestra.run.smithi017.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-11T09:52:35.155 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-11T09:52:35.164 INFO:tasks.ceph.osd.5.smithi096.stderr:2024-09-11T09:52:35.164+0000 7f3c96ec9640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 31909) UID: 0 2024-09-11T09:52:35.248 INFO:teuthology.orchestra.run.smithi017.stdout:osd.0: {} 2024-09-11T09:52:35.249 INFO:teuthology.orchestra.run.smithi017.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-11T09:52:35.264 INFO:tasks.ceph.osd.0.smithi017.stderr:2024-09-11T09:52:35.265+0000 7fb73c0f1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31665) UID: 0 2024-09-11T09:52:35.266 INFO:teuthology.orchestra.run.smithi017.stdout:osd.1: {} 2024-09-11T09:52:35.266 INFO:teuthology.orchestra.run.smithi017.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-11T09:52:35.282 INFO:teuthology.orchestra.run.smithi017.stdout:osd.2: {} 2024-09-11T09:52:35.283 INFO:teuthology.orchestra.run.smithi017.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-11T09:52:35.299 INFO:teuthology.orchestra.run.smithi017.stdout:osd.3: {} 2024-09-11T09:52:35.300 INFO:teuthology.orchestra.run.smithi017.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-11T09:52:35.305 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:52:35.305 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [], 2024-09-11T09:52:35.305 INFO:teuthology.orchestra.run.smithi017.stdout: "num_ops": 0 2024-09-11T09:52:35.305 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:52:35.314 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-11T09:52:35.317 INFO:teuthology.orchestra.run.smithi017.stdout:osd.4: {} 2024-09-11T09:52:35.317 INFO:teuthology.orchestra.run.smithi017.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-11T09:52:35.333 INFO:teuthology.orchestra.run.smithi017.stdout:osd.5: {} 2024-09-11T09:52:35.334 INFO:teuthology.orchestra.run.smithi017.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-11T09:52:35.365 INFO:tasks.ceph.osd.4.smithi096.stderr:2024-09-11T09:52:35.366+0000 7f3a7b90a640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 42475) UID: 0 2024-09-11T09:52:35.424 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:52:35.424 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [], 2024-09-11T09:52:35.424 INFO:teuthology.orchestra.run.smithi017.stdout: "complaint_time": 30, 2024-09-11T09:52:35.424 INFO:teuthology.orchestra.run.smithi017.stdout: "num_blocked_ops": 0 2024-09-11T09:52:35.425 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:52:35.433 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-11T09:52:35.447 DEBUG:teuthology.parallel:result is None 2024-09-11T09:52:35.448 DEBUG:teuthology.orchestra.run.smithi107:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-09-11T09:52:35.471 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-09-11T09:52:35.472 DEBUG:teuthology.orchestra.run.smithi107:> rmdir -- /home/ubuntu/cephtest/mnt.0 2024-09-11T09:52:35.477 INFO:tasks.ceph.osd.0.smithi017.stderr:2024-09-11T09:52:35.478+0000 7fb73c0f1640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 31665) UID: 0 2024-09-11T09:52:35.536 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.0/client.0 2024-09-11T09:52:35.536 DEBUG:teuthology.run_tasks:Unwinding manager thrashosds 2024-09-11T09:52:35.550 INFO:tasks.thrashosds:joining thrashosds 2024-09-11T09:52:35.551 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:52:35.551 INFO:teuthology.orchestra.run.smithi017.stdout: "size": 20, 2024-09-11T09:52:35.552 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 600, 2024-09-11T09:52:35.552 INFO:teuthology.orchestra.run.smithi017.stdout: "ops": [ 2024-09-11T09:52:35.552 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.552 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:31 2.1 2:8654721b:::rbd_data.1250f5e25b80.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:35.552 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.896024+0000", 2024-09-11T09:52:35.552 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 72.647009886000006, 2024-09-11T09:52:35.552 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.023148401999999998, 2024-09-11T09:52:35.552 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:35.552 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:35.552 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:35.552 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:35.552 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:52:35.552 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:52:35.552 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 31 2024-09-11T09:52:35.553 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.553 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:35.553 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.553 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:35.553 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.896024+0000", 2024-09-11T09:52:35.553 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:35.553 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.553 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.553 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:35.553 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.896024+0000", 2024-09-11T09:52:35.553 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:35.553 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.553 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.554 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:35.554 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.896025+0000", 2024-09-11T09:52:35.554 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.54e-07 2024-09-11T09:52:35.554 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.554 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.554 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:35.554 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.904287+0000", 2024-09-11T09:52:35.554 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0082624729999999993 2024-09-11T09:52:35.554 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.554 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.554 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:35.554 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.904305+0000", 2024-09-11T09:52:35.554 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.7912999999999999e-05 2024-09-11T09:52:35.554 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.555 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.555 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:35.555 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.904312+0000", 2024-09-11T09:52:35.555 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.0230000000000004e-06 2024-09-11T09:52:35.555 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.555 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.555 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:35.555 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.904395+0000", 2024-09-11T09:52:35.555 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.2355999999999996e-05 2024-09-11T09:52:35.555 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.555 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.555 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:35.555 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.904499+0000", 2024-09-11T09:52:35.555 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000104672 2024-09-11T09:52:35.556 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.556 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.556 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1", 2024-09-11T09:52:35.556 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.904594+0000", 2024-09-11T09:52:35.556 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.5129000000000003e-05 2024-09-11T09:52:35.556 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.556 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.556 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:35.556 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.918183+0000", 2024-09-11T09:52:35.556 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.013588355999999999 2024-09-11T09:52:35.556 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.556 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.556 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:35.556 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919053+0000", 2024-09-11T09:52:35.557 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00087066800000000001 2024-09-11T09:52:35.557 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.557 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.557 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:35.557 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919125+0000", 2024-09-11T09:52:35.557 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.1333000000000003e-05 2024-09-11T09:52:35.557 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.557 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.557 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:35.557 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.919172+0000", 2024-09-11T09:52:35.557 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.7725000000000001e-05 2024-09-11T09:52:35.557 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:35.557 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:35.558 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:35.558 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.558 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.558 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4688.0:42 2.3 e53/15 2:c54742d8:::rbd_data.1250f5e25b80.0000000000000006:head v 53'2, mlcod=0'0)", 2024-09-11T09:52:35.558 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.939272+0000", 2024-09-11T09:52:35.558 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 72.603761902000002, 2024-09-11T09:52:35.558 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.010821958, 2024-09-11T09:52:35.558 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:35.558 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:35.558 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:35.558 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:35.558 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.558 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:35.558 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.939272+0000", 2024-09-11T09:52:35.559 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:35.559 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.559 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.559 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:35.559 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.939272+0000", 2024-09-11T09:52:35.559 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:35.559 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.559 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.559 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:35.559 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.939273+0000", 2024-09-11T09:52:35.559 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.7899999999999997e-07 2024-09-11T09:52:35.559 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.559 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.560 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:35.560 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.944403+0000", 2024-09-11T09:52:35.560 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0051302129999999998 2024-09-11T09:52:35.560 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.560 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.560 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:35.560 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.944413+0000", 2024-09-11T09:52:35.560 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0182e-05 2024-09-11T09:52:35.560 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.560 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.560 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:35.560 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.944420+0000", 2024-09-11T09:52:35.560 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.7000000000000002e-06 2024-09-11T09:52:35.560 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.561 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.561 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:35.561 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.944482+0000", 2024-09-11T09:52:35.561 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.1911999999999999e-05 2024-09-11T09:52:35.561 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.561 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.561 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:35.561 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.944508+0000", 2024-09-11T09:52:35.561 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.6582000000000001e-05 2024-09-11T09:52:35.561 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.561 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.561 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:35.561 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.950050+0000", 2024-09-11T09:52:35.561 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0055414080000000003 2024-09-11T09:52:35.562 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.562 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.562 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:35.562 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.950094+0000", 2024-09-11T09:52:35.562 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.4082000000000002e-05 2024-09-11T09:52:35.562 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:35.562 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:35.562 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:35.562 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.562 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.562 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:47 2.1 2:95f53b3e:::rbd_data.1250f5e25b80.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:35.562 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.948330+0000", 2024-09-11T09:52:35.562 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 72.594703558000006, 2024-09-11T09:52:35.562 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.025156603999999999, 2024-09-11T09:52:35.563 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:35.563 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:35.563 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:35.563 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:35.563 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:52:35.563 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:52:35.563 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 47 2024-09-11T09:52:35.563 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.563 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:35.563 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.563 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:35.563 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.948330+0000", 2024-09-11T09:52:35.563 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:35.563 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.564 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.564 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:35.564 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.948330+0000", 2024-09-11T09:52:35.564 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:35.564 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.564 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.564 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:35.564 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.948333+0000", 2024-09-11T09:52:35.564 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.481e-06 2024-09-11T09:52:35.564 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.564 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.564 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:35.564 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.960263+0000", 2024-09-11T09:52:35.565 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.011930671 2024-09-11T09:52:35.565 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.565 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.565 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:35.565 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.960289+0000", 2024-09-11T09:52:35.565 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.5573000000000001e-05 2024-09-11T09:52:35.565 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.565 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.565 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:35.565 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.960304+0000", 2024-09-11T09:52:35.565 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.4647000000000001e-05 2024-09-11T09:52:35.565 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.565 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.566 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:35.566 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.960390+0000", 2024-09-11T09:52:35.566 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.6088999999999995e-05 2024-09-11T09:52:35.566 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.566 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.566 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:35.566 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.960562+0000", 2024-09-11T09:52:35.566 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00017237400000000001 2024-09-11T09:52:35.566 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.566 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.566 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1", 2024-09-11T09:52:35.566 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.960725+0000", 2024-09-11T09:52:35.566 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00016321599999999999 2024-09-11T09:52:35.566 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.567 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.567 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:35.567 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.969465+0000", 2024-09-11T09:52:35.567 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0087396899999999996 2024-09-11T09:52:35.567 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.567 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.567 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:35.567 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.973393+0000", 2024-09-11T09:52:35.567 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0039274669999999996 2024-09-11T09:52:35.567 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.567 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.567 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:35.567 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.973443+0000", 2024-09-11T09:52:35.567 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.0340000000000003e-05 2024-09-11T09:52:35.568 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.568 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.568 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:35.568 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.973487+0000", 2024-09-11T09:52:35.568 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.4056e-05 2024-09-11T09:52:35.568 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:35.568 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:35.568 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:35.568 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.568 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.568 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4688.0:61 2.1 2:9af038b1:::rbd_data.1250f5e25b80.000000000000000f:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:35.568 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:22.975764+0000", 2024-09-11T09:52:35.568 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 72.567270034000003, 2024-09-11T09:52:35.569 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.016158891000000002, 2024-09-11T09:52:35.569 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:35.569 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:35.569 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:35.569 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:35.569 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4688", 2024-09-11T09:52:35.569 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:52:35.569 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 61 2024-09-11T09:52:35.569 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.569 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:35.569 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.569 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:35.569 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.975764+0000", 2024-09-11T09:52:35.569 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:35.570 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.570 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.570 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:35.570 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.975764+0000", 2024-09-11T09:52:35.570 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:35.570 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.570 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.570 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:35.570 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.975766+0000", 2024-09-11T09:52:35.570 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.767e-06 2024-09-11T09:52:35.570 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.570 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.570 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:35.570 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.983204+0000", 2024-09-11T09:52:35.571 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0074388400000000004 2024-09-11T09:52:35.571 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.571 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.571 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:35.571 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.983226+0000", 2024-09-11T09:52:35.571 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.1279000000000001e-05 2024-09-11T09:52:35.571 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.571 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.571 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:35.571 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.983238+0000", 2024-09-11T09:52:35.571 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2563e-05 2024-09-11T09:52:35.571 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.571 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.571 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:35.572 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.983318+0000", 2024-09-11T09:52:35.572 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.0075999999999997e-05 2024-09-11T09:52:35.572 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.572 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.572 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:35.572 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.983429+0000", 2024-09-11T09:52:35.572 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000110618 2024-09-11T09:52:35.572 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.572 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.572 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1", 2024-09-11T09:52:35.572 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.983525+0000", 2024-09-11T09:52:35.572 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.603e-05 2024-09-11T09:52:35.572 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.573 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.573 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:35.573 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.989004+0000", 2024-09-11T09:52:35.573 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0054785429999999998 2024-09-11T09:52:35.573 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.573 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.573 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:35.573 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.991788+0000", 2024-09-11T09:52:35.573 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0027845729999999998 2024-09-11T09:52:35.573 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.573 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.573 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:35.573 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.991870+0000", 2024-09-11T09:52:35.573 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.1567e-05 2024-09-11T09:52:35.574 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.574 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.574 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:35.574 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:22.991923+0000", 2024-09-11T09:52:35.574 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.3035000000000003e-05 2024-09-11T09:52:35.574 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:35.574 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:35.574 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:35.574 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.574 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.574 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4694.0:32 2.1 2:8a862b48:ns::rbd_data.1256a0c02879.0000000000000000:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:35.574 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.084089+0000", 2024-09-11T09:52:35.574 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 72.458944560000006, 2024-09-11T09:52:35.574 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.020630553999999999, 2024-09-11T09:52:35.575 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:35.575 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:35.575 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:35.575 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:35.575 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4694", 2024-09-11T09:52:35.575 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:52:35.575 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 32 2024-09-11T09:52:35.575 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.575 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:35.575 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.575 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:35.575 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.084089+0000", 2024-09-11T09:52:35.575 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:35.575 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.576 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.576 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:35.576 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.084089+0000", 2024-09-11T09:52:35.576 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:35.576 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.576 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.576 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:35.576 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.084090+0000", 2024-09-11T09:52:35.576 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.2900000000000003e-07 2024-09-11T09:52:35.576 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.576 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.576 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:35.576 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.090866+0000", 2024-09-11T09:52:35.577 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0067756609999999997 2024-09-11T09:52:35.577 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.577 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.577 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:35.577 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.090891+0000", 2024-09-11T09:52:35.577 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.5119000000000001e-05 2024-09-11T09:52:35.577 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.577 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.577 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:35.577 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.090900+0000", 2024-09-11T09:52:35.577 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.4110000000000002e-06 2024-09-11T09:52:35.577 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.577 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.577 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:35.578 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.090968+0000", 2024-09-11T09:52:35.578 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.8041999999999997e-05 2024-09-11T09:52:35.578 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.578 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.578 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:35.578 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.091115+0000", 2024-09-11T09:52:35.578 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00014633499999999999 2024-09-11T09:52:35.578 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.578 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.578 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1", 2024-09-11T09:52:35.578 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.091284+0000", 2024-09-11T09:52:35.578 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000169836 2024-09-11T09:52:35.578 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.579 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.579 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:35.579 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.100896+0000", 2024-09-11T09:52:35.579 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0096118639999999995 2024-09-11T09:52:35.579 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.579 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.579 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:35.579 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.104604+0000", 2024-09-11T09:52:35.579 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0037074619999999999 2024-09-11T09:52:35.579 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.579 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.579 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:35.579 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.104669+0000", 2024-09-11T09:52:35.580 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.4836000000000001e-05 2024-09-11T09:52:35.580 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.580 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.580 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:35.580 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.104720+0000", 2024-09-11T09:52:35.580 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.1258999999999998e-05 2024-09-11T09:52:35.580 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:35.580 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:35.580 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:35.580 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.580 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.580 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4694.0:44 2.3 e53/15 2:cd1a7d1c:ns::rbd_data.1256a0c02879.0000000000000006:head v 53'3, mlcod=16'1)", 2024-09-11T09:52:35.580 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.120387+0000", 2024-09-11T09:52:35.581 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 72.422646907000001, 2024-09-11T09:52:35.581 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.011988468, 2024-09-11T09:52:35.581 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:35.581 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:35.581 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:35.581 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:35.581 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.581 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:35.581 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.120387+0000", 2024-09-11T09:52:35.581 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:35.581 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.581 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.581 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:35.582 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.120387+0000", 2024-09-11T09:52:35.582 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:35.582 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.582 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.582 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:35.582 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.120389+0000", 2024-09-11T09:52:35.582 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.018e-06 2024-09-11T09:52:35.582 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.582 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.582 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:35.582 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.127403+0000", 2024-09-11T09:52:35.582 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0070144769999999999 2024-09-11T09:52:35.582 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.583 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.583 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:35.583 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.127414+0000", 2024-09-11T09:52:35.583 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0991e-05 2024-09-11T09:52:35.583 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.583 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.583 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:35.583 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.127423+0000", 2024-09-11T09:52:35.583 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.8580000000000006e-06 2024-09-11T09:52:35.583 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.583 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.583 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:35.584 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.127468+0000", 2024-09-11T09:52:35.584 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.4830999999999998e-05 2024-09-11T09:52:35.584 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.584 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.584 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:35.584 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.127485+0000", 2024-09-11T09:52:35.584 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6838999999999999e-05 2024-09-11T09:52:35.584 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.584 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.584 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:35.584 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.132341+0000", 2024-09-11T09:52:35.584 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0048557959999999999 2024-09-11T09:52:35.584 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.584 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.585 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:35.585 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.132375+0000", 2024-09-11T09:52:35.585 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.4657999999999999e-05 2024-09-11T09:52:35.585 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:35.585 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:35.585 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:35.585 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.585 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.585 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4694.0:46 2.3 e53/15 2:ce0e1469:ns::rbd_data.1256a0c02879.0000000000000008:head v 53'4, mlcod=16'1)", 2024-09-11T09:52:35.585 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.129008+0000", 2024-09-11T09:52:35.585 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 72.414026320999994, 2024-09-11T09:52:35.585 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0087151220000000005, 2024-09-11T09:52:35.585 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:35.585 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:35.586 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:35.586 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:35.586 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.586 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:35.586 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.129008+0000", 2024-09-11T09:52:35.586 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:35.586 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.586 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.586 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:35.586 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.129008+0000", 2024-09-11T09:52:35.586 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:35.586 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.587 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.587 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:35.587 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.129009+0000", 2024-09-11T09:52:35.587 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.367e-06 2024-09-11T09:52:35.587 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.587 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.587 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:35.587 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.132839+0000", 2024-09-11T09:52:35.587 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0038297610000000001 2024-09-11T09:52:35.587 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.587 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.587 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:35.587 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.132849+0000", 2024-09-11T09:52:35.588 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0672e-05 2024-09-11T09:52:35.588 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.588 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.588 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:35.588 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.132856+0000", 2024-09-11T09:52:35.588 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.1380000000000004e-06 2024-09-11T09:52:35.588 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.588 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.588 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:35.588 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.132891+0000", 2024-09-11T09:52:35.588 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.5712e-05 2024-09-11T09:52:35.588 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.588 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.589 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:35.589 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.132907+0000", 2024-09-11T09:52:35.589 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.6166000000000001e-05 2024-09-11T09:52:35.589 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.589 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.589 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:35.589 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.137681+0000", 2024-09-11T09:52:35.589 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0047735299999999998 2024-09-11T09:52:35.589 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.589 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.589 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:35.589 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.137723+0000", 2024-09-11T09:52:35.589 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.1776000000000001e-05 2024-09-11T09:52:35.590 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:35.590 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:35.590 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:35.590 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.590 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.590 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4694.0:48 2.3 e53/15 2:d2090b70:ns::rbd_data.1256a0c02879.0000000000000009:head v 53'5, mlcod=53'2)", 2024-09-11T09:52:35.590 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.135108+0000", 2024-09-11T09:52:35.590 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 72.407925423999998, 2024-09-11T09:52:35.590 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.010710679000000001, 2024-09-11T09:52:35.590 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:35.590 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:35.590 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:35.590 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:35.590 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.591 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:35.591 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.135108+0000", 2024-09-11T09:52:35.591 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:35.591 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.591 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.591 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:35.591 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.135108+0000", 2024-09-11T09:52:35.591 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:35.591 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.591 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.591 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:35.591 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.135110+0000", 2024-09-11T09:52:35.591 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.122e-06 2024-09-11T09:52:35.592 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.592 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.592 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:35.592 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.140876+0000", 2024-09-11T09:52:35.592 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.005766162 2024-09-11T09:52:35.592 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.592 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.592 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:35.592 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.140888+0000", 2024-09-11T09:52:35.592 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.201e-05 2024-09-11T09:52:35.592 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.592 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.592 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:35.592 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.140894+0000", 2024-09-11T09:52:35.593 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.1739999999999997e-06 2024-09-11T09:52:35.593 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.593 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.593 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:35.593 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.140971+0000", 2024-09-11T09:52:35.593 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.7495000000000003e-05 2024-09-11T09:52:35.593 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.593 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.593 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:35.593 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.140995+0000", 2024-09-11T09:52:35.593 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.3204999999999999e-05 2024-09-11T09:52:35.593 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.593 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.594 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:35.594 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.145757+0000", 2024-09-11T09:52:35.594 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0047627190000000003 2024-09-11T09:52:35.594 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.594 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.594 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:35.594 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.145819+0000", 2024-09-11T09:52:35.594 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.1791999999999994e-05 2024-09-11T09:52:35.594 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:35.594 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:35.594 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:35.594 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.594 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.595 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.4694.0:50 2.1 2:83078786:ns::rbd_data.1256a0c02879.000000000000000a:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:35.595 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.141467+0000", 2024-09-11T09:52:35.595 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 72.401566700000004, 2024-09-11T09:52:35.595 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.012785889, 2024-09-11T09:52:35.595 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:35.595 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:35.595 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:35.595 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:35.595 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.4694", 2024-09-11T09:52:35.595 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:52:35.595 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 50 2024-09-11T09:52:35.595 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.595 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:35.595 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.596 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:35.596 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.141467+0000", 2024-09-11T09:52:35.596 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:35.596 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.596 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.596 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:35.596 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.141467+0000", 2024-09-11T09:52:35.596 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:35.596 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.596 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.596 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:35.596 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.141468+0000", 2024-09-11T09:52:35.596 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1009999999999999e-06 2024-09-11T09:52:35.597 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.597 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.597 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:35.597 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.146432+0000", 2024-09-11T09:52:35.597 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0049640539999999999 2024-09-11T09:52:35.597 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.597 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.597 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:35.597 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.146446+0000", 2024-09-11T09:52:35.597 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3254e-05 2024-09-11T09:52:35.597 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.597 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.597 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:35.598 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.146452+0000", 2024-09-11T09:52:35.598 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.207e-06 2024-09-11T09:52:35.598 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.598 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.598 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:35.598 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.146510+0000", 2024-09-11T09:52:35.598 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.8322999999999997e-05 2024-09-11T09:52:35.598 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.598 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.598 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:35.598 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.146650+0000", 2024-09-11T09:52:35.598 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00013947100000000001 2024-09-11T09:52:35.598 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.598 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.599 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1", 2024-09-11T09:52:35.599 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.146779+0000", 2024-09-11T09:52:35.599 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000129059 2024-09-11T09:52:35.599 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.599 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.599 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:35.599 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.153778+0000", 2024-09-11T09:52:35.599 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0069997130000000003 2024-09-11T09:52:35.599 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.599 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.599 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:35.599 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.154180+0000", 2024-09-11T09:52:35.599 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00040159800000000003 2024-09-11T09:52:35.599 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.600 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.600 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:35.600 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.154218+0000", 2024-09-11T09:52:35.600 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.7795e-05 2024-09-11T09:52:35.600 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.600 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.600 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:35.600 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.154253+0000", 2024-09-11T09:52:35.600 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.5314000000000001e-05 2024-09-11T09:52:35.600 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:35.600 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:35.600 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:35.601 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.601 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.601 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4694.0:54 2.3 e53/15 2:c44f0253:ns::rbd_data.1256a0c02879.000000000000000b:head v 53'6, mlcod=53'4)", 2024-09-11T09:52:35.601 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.154926+0000", 2024-09-11T09:52:35.601 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 72.388108325000005, 2024-09-11T09:52:35.601 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0089497099999999996, 2024-09-11T09:52:35.601 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:35.601 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:35.601 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:35.601 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:35.601 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.602 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:35.602 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.154926+0000", 2024-09-11T09:52:35.602 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:35.602 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.602 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.602 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:35.602 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.154926+0000", 2024-09-11T09:52:35.602 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:35.602 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.602 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.602 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:35.602 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.154927+0000", 2024-09-11T09:52:35.603 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.7000000000000003e-07 2024-09-11T09:52:35.603 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.603 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.603 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:35.603 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.158712+0000", 2024-09-11T09:52:35.603 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0037855469999999998 2024-09-11T09:52:35.603 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.603 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.603 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:35.603 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.158726+0000", 2024-09-11T09:52:35.603 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3621000000000001e-05 2024-09-11T09:52:35.603 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.604 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.604 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:35.604 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.158732+0000", 2024-09-11T09:52:35.604 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.9089999999999997e-06 2024-09-11T09:52:35.604 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.604 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.604 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:35.604 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.158781+0000", 2024-09-11T09:52:35.604 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.8900000000000003e-05 2024-09-11T09:52:35.604 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.604 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.604 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:35.604 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.158805+0000", 2024-09-11T09:52:35.605 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.44e-05 2024-09-11T09:52:35.605 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.605 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.605 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:35.605 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.163826+0000", 2024-09-11T09:52:35.605 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0050206399999999998 2024-09-11T09:52:35.605 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.605 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.605 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:35.605 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.163875+0000", 2024-09-11T09:52:35.605 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.9722999999999998e-05 2024-09-11T09:52:35.605 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:35.606 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:35.606 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:35.606 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.606 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.606 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_repop(client.4694.0:56 2.3 e53/15 2:c462f029:ns::rbd_data.1256a0c02879.000000000000000c:head v 53'7, mlcod=53'4)", 2024-09-11T09:52:35.606 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:23.159549+0000", 2024-09-11T09:52:35.606 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 72.383484619000001, 2024-09-11T09:52:35.606 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.015591666000000001, 2024-09-11T09:52:35.606 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:35.606 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:35.606 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:35.606 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:35.606 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.606 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:35.607 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.159549+0000", 2024-09-11T09:52:35.607 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:35.607 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.607 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.607 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:35.607 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.159549+0000", 2024-09-11T09:52:35.607 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:35.607 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.607 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.607 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:35.607 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.159550+0000", 2024-09-11T09:52:35.607 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.1500000000000004e-07 2024-09-11T09:52:35.607 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.607 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.608 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:35.608 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.167834+0000", 2024-09-11T09:52:35.608 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0082840220000000003 2024-09-11T09:52:35.608 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.608 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.608 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:35.608 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.167857+0000", 2024-09-11T09:52:35.608 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.3337e-05 2024-09-11T09:52:35.608 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.608 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.608 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:35.608 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.167869+0000", 2024-09-11T09:52:35.608 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.1556000000000001e-05 2024-09-11T09:52:35.608 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.609 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.609 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:35.609 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.167939+0000", 2024-09-11T09:52:35.609 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.0674999999999997e-05 2024-09-11T09:52:35.609 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.609 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.609 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:35.609 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.167967+0000", 2024-09-11T09:52:35.609 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.7322999999999999e-05 2024-09-11T09:52:35.609 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.609 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.609 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:35.609 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.175079+0000", 2024-09-11T09:52:35.610 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0071126150000000001 2024-09-11T09:52:35.610 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.610 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.610 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:35.610 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:23.175141+0000", 2024-09-11T09:52:35.610 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.1523e-05 2024-09-11T09:52:35.610 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:35.610 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:35.610 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:35.610 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.610 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.610 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.5314.0:31 2.1 2:8654721b:::rbd_data.1250f5e25b80.0000000000000000:head [write 393216~131072] snapc 3=[3] ondisk+write+known_if_redirected+supports_pool_eio e68)", 2024-09-11T09:52:35.610 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:50.230850+0000", 2024-09-11T09:52:35.611 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 45.312183857999997, 2024-09-11T09:52:35.611 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0073772539999999998, 2024-09-11T09:52:35.611 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:35.611 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:35.611 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:35.611 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:35.611 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.5314", 2024-09-11T09:52:35.611 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3853995732", 2024-09-11T09:52:35.611 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 31 2024-09-11T09:52:35.611 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.611 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:35.611 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.611 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:35.612 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.230850+0000", 2024-09-11T09:52:35.612 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:35.612 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.612 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.612 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:35.612 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.230850+0000", 2024-09-11T09:52:35.612 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:35.612 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.612 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.612 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:35.612 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.230850+0000", 2024-09-11T09:52:35.612 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.32e-07 2024-09-11T09:52:35.613 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.613 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.613 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:35.613 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.230908+0000", 2024-09-11T09:52:35.613 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.7698000000000002e-05 2024-09-11T09:52:35.613 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.613 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.613 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:35.613 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.230915+0000", 2024-09-11T09:52:35.613 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.81e-06 2024-09-11T09:52:35.613 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.613 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.614 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:35.614 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.230920+0000", 2024-09-11T09:52:35.614 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.9660000000000004e-06 2024-09-11T09:52:35.614 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.614 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.614 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:35.614 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.235055+0000", 2024-09-11T09:52:35.614 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0041354629999999998 2024-09-11T09:52:35.614 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.614 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.614 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:35.614 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.235145+0000", 2024-09-11T09:52:35.615 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.9362000000000003e-05 2024-09-11T09:52:35.615 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.615 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.615 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1", 2024-09-11T09:52:35.615 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.235282+0000", 2024-09-11T09:52:35.615 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00013710999999999999 2024-09-11T09:52:35.615 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.615 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.615 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:35.615 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.237335+0000", 2024-09-11T09:52:35.615 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.002053165 2024-09-11T09:52:35.615 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.615 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.616 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:35.616 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.238106+0000", 2024-09-11T09:52:35.616 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000771015 2024-09-11T09:52:35.616 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.616 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.616 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:35.616 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.238165+0000", 2024-09-11T09:52:35.616 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.8588000000000002e-05 2024-09-11T09:52:35.616 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.616 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.617 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:35.617 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.238227+0000", 2024-09-11T09:52:35.617 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.2644999999999994e-05 2024-09-11T09:52:35.617 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:35.617 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:35.617 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:35.617 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.617 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.617 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.5314.0:32 2.1 2:8654721b:::rbd_data.1250f5e25b80.0000000000000000:head [write 524288~131072] snapc 3=[3] ondisk+write+known_if_redirected+supports_pool_eio e68)", 2024-09-11T09:52:35.617 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:50.230932+0000", 2024-09-11T09:52:35.617 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 45.312102232999997, 2024-09-11T09:52:35.617 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0074882129999999996, 2024-09-11T09:52:35.618 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:35.618 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:35.618 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:35.618 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:35.618 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.5314", 2024-09-11T09:52:35.618 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3853995732", 2024-09-11T09:52:35.618 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 32 2024-09-11T09:52:35.618 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.618 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:35.618 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.618 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:35.618 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.230932+0000", 2024-09-11T09:52:35.619 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:35.619 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.619 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.619 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:35.619 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.230932+0000", 2024-09-11T09:52:35.619 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:35.619 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.619 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.619 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:35.619 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.230932+0000", 2024-09-11T09:52:35.619 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.9999999999999999e-07 2024-09-11T09:52:35.619 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.619 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.619 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:35.619 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.231022+0000", 2024-09-11T09:52:35.620 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.0439999999999995e-05 2024-09-11T09:52:35.620 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.620 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.620 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:35.620 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.231030+0000", 2024-09-11T09:52:35.620 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.3350000000000001e-06 2024-09-11T09:52:35.620 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.620 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.620 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:35.620 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.231034+0000", 2024-09-11T09:52:35.620 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.9759999999999997e-06 2024-09-11T09:52:35.620 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.620 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.621 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:35.621 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.235837+0000", 2024-09-11T09:52:35.621 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0048031910000000001 2024-09-11T09:52:35.621 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.621 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.621 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:35.621 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.235904+0000", 2024-09-11T09:52:35.621 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.7181999999999995e-05 2024-09-11T09:52:35.621 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.621 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.621 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1", 2024-09-11T09:52:35.621 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.236040+0000", 2024-09-11T09:52:35.621 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00013545199999999999 2024-09-11T09:52:35.622 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.622 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.622 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:35.622 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.238242+0000", 2024-09-11T09:52:35.622 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.002202809 2024-09-11T09:52:35.622 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.622 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.622 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:35.622 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.238301+0000", 2024-09-11T09:52:35.622 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.8573999999999999e-05 2024-09-11T09:52:35.622 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.622 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.622 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:35.622 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.238378+0000", 2024-09-11T09:52:35.623 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.7323999999999994e-05 2024-09-11T09:52:35.623 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.623 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.623 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:35.623 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.238420+0000", 2024-09-11T09:52:35.623 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.163e-05 2024-09-11T09:52:35.623 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:35.623 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:35.623 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:35.623 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.623 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.623 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.5314.0:33 2.1 2:8654721b:::rbd_data.1250f5e25b80.0000000000000000:head [write 655360~131072] snapc 3=[3] ondisk+write+known_if_redirected+supports_pool_eio e68)", 2024-09-11T09:52:35.623 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:50.231041+0000", 2024-09-11T09:52:35.623 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 45.311993074, 2024-09-11T09:52:35.624 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0075627139999999999, 2024-09-11T09:52:35.624 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:35.624 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:35.624 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:35.624 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:35.624 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.5314", 2024-09-11T09:52:35.624 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3853995732", 2024-09-11T09:52:35.624 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 33 2024-09-11T09:52:35.624 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.624 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:35.624 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.624 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:35.624 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.231041+0000", 2024-09-11T09:52:35.625 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:35.625 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.625 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.625 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:35.625 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.231041+0000", 2024-09-11T09:52:35.625 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:35.625 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.625 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.625 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:35.625 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.231041+0000", 2024-09-11T09:52:35.625 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.9499999999999998e-07 2024-09-11T09:52:35.625 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.625 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.626 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:35.626 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.231111+0000", 2024-09-11T09:52:35.626 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.0166999999999994e-05 2024-09-11T09:52:35.626 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.626 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.626 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:35.626 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.231122+0000", 2024-09-11T09:52:35.626 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.0518e-05 2024-09-11T09:52:35.626 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.626 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.626 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:35.626 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.231128+0000", 2024-09-11T09:52:35.626 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.3509999999999998e-06 2024-09-11T09:52:35.626 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.627 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.627 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:35.627 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.236622+0000", 2024-09-11T09:52:35.627 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0054934989999999998 2024-09-11T09:52:35.627 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.627 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.627 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:35.627 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.236719+0000", 2024-09-11T09:52:35.627 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.7497999999999994e-05 2024-09-11T09:52:35.627 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.627 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.627 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1", 2024-09-11T09:52:35.627 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.236854+0000", 2024-09-11T09:52:35.627 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000134535 2024-09-11T09:52:35.628 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.628 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.628 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:35.628 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.238491+0000", 2024-09-11T09:52:35.628 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0016368610000000001 2024-09-11T09:52:35.628 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.628 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.628 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:35.628 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.238504+0000", 2024-09-11T09:52:35.628 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3927999999999999e-05 2024-09-11T09:52:35.628 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.628 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.628 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:35.629 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.238561+0000", 2024-09-11T09:52:35.629 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.6029999999999997e-05 2024-09-11T09:52:35.629 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.629 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.629 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:35.629 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.238604+0000", 2024-09-11T09:52:35.629 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.3031999999999997e-05 2024-09-11T09:52:35.629 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:35.629 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:35.629 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:35.629 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.629 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.629 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.5314.0:34 2.1 2:8654721b:::rbd_data.1250f5e25b80.0000000000000000:head [write 786432~131072] snapc 3=[3] ondisk+write+known_if_redirected+supports_pool_eio e68)", 2024-09-11T09:52:35.629 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:50.231139+0000", 2024-09-11T09:52:35.630 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 45.311894471000002, 2024-09-11T09:52:35.630 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0083689690000000004, 2024-09-11T09:52:35.630 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:35.630 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:35.630 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:35.630 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:35.630 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.5314", 2024-09-11T09:52:35.630 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3853995732", 2024-09-11T09:52:35.630 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 34 2024-09-11T09:52:35.630 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.630 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:35.630 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.630 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:35.630 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.231139+0000", 2024-09-11T09:52:35.631 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:35.631 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.631 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.631 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:35.631 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.231139+0000", 2024-09-11T09:52:35.631 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:35.631 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.631 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.631 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:35.631 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.231140+0000", 2024-09-11T09:52:35.631 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.1699999999999998e-07 2024-09-11T09:52:35.631 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.631 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.631 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:35.632 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.231247+0000", 2024-09-11T09:52:35.632 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000107088 2024-09-11T09:52:35.632 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.632 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.632 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:35.632 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.231260+0000", 2024-09-11T09:52:35.632 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.289e-05 2024-09-11T09:52:35.632 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.632 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.632 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:35.632 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.231272+0000", 2024-09-11T09:52:35.632 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2371999999999999e-05 2024-09-11T09:52:35.632 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.633 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.633 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:35.633 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.237380+0000", 2024-09-11T09:52:35.633 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.006107683 2024-09-11T09:52:35.633 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.633 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.633 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:35.633 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.237434+0000", 2024-09-11T09:52:35.633 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.4143e-05 2024-09-11T09:52:35.633 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.633 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.633 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1", 2024-09-11T09:52:35.633 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.237541+0000", 2024-09-11T09:52:35.633 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00010676800000000001 2024-09-11T09:52:35.634 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.634 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.634 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:35.634 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.239390+0000", 2024-09-11T09:52:35.634 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.001848861 2024-09-11T09:52:35.634 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.634 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.634 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:35.634 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.239409+0000", 2024-09-11T09:52:35.634 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.9505e-05 2024-09-11T09:52:35.634 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.634 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.634 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:35.634 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.239463+0000", 2024-09-11T09:52:35.634 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.3838000000000002e-05 2024-09-11T09:52:35.635 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.635 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.635 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:35.635 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.239508+0000", 2024-09-11T09:52:35.635 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.5303999999999999e-05 2024-09-11T09:52:35.635 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:35.635 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:35.635 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:35.635 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.635 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.635 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.5314.0:35 2.1 2:8654721b:::rbd_data.1250f5e25b80.0000000000000000:head [write 917504~131072] snapc 3=[3] ondisk+write+known_if_redirected+supports_pool_eio e68)", 2024-09-11T09:52:35.635 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:50.231287+0000", 2024-09-11T09:52:35.635 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 45.311747136999998, 2024-09-11T09:52:35.635 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0086930730000000008, 2024-09-11T09:52:35.635 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:35.636 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:35.636 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:35.636 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:35.636 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.5314", 2024-09-11T09:52:35.636 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3853995732", 2024-09-11T09:52:35.636 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 35 2024-09-11T09:52:35.636 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.636 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:35.636 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.636 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:35.636 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.231287+0000", 2024-09-11T09:52:35.636 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:35.636 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.636 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.637 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:35.637 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.231287+0000", 2024-09-11T09:52:35.637 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:35.637 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.637 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.637 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:35.637 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.231287+0000", 2024-09-11T09:52:35.637 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.7700000000000004e-07 2024-09-11T09:52:35.637 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.637 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.637 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:35.637 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.231354+0000", 2024-09-11T09:52:35.637 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.6304999999999996e-05 2024-09-11T09:52:35.638 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.638 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.638 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:35.638 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.231367+0000", 2024-09-11T09:52:35.638 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.2952999999999999e-05 2024-09-11T09:52:35.638 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.638 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.638 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:35.638 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.231374+0000", 2024-09-11T09:52:35.638 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.0469999999999996e-06 2024-09-11T09:52:35.638 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.638 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.638 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:35.638 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.238650+0000", 2024-09-11T09:52:35.638 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0072760280000000004 2024-09-11T09:52:35.639 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.639 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.639 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:35.639 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.238732+0000", 2024-09-11T09:52:35.639 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.2194000000000006e-05 2024-09-11T09:52:35.639 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.639 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.639 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1", 2024-09-11T09:52:35.639 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.238876+0000", 2024-09-11T09:52:35.639 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00014458 2024-09-11T09:52:35.639 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.639 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.639 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:35.640 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.239739+0000", 2024-09-11T09:52:35.640 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00086244499999999999 2024-09-11T09:52:35.640 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.640 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.640 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:35.640 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.239917+0000", 2024-09-11T09:52:35.640 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00017777299999999999 2024-09-11T09:52:35.640 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.640 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.640 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:35.640 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.239954+0000", 2024-09-11T09:52:35.640 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.7379999999999998e-05 2024-09-11T09:52:35.640 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.640 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.641 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:35.641 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.239980+0000", 2024-09-11T09:52:35.641 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.5791000000000001e-05 2024-09-11T09:52:35.641 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:35.641 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:35.641 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:35.641 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.641 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.641 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.5314.0:70 2.1 2:8654721b:::rbd_data.1250f5e25b80.0000000000000000:head [write 524288~131072] snapc 5=[5,3] ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-09-11T09:52:35.641 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:50.824556+0000", 2024-09-11T09:52:35.641 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 44.718478335999997, 2024-09-11T09:52:35.641 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0072794469999999997, 2024-09-11T09:52:35.641 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:35.642 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:35.642 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:35.642 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:35.642 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.5314", 2024-09-11T09:52:35.642 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3853995732", 2024-09-11T09:52:35.642 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 70 2024-09-11T09:52:35.642 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.642 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:35.642 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.642 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:35.642 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.824556+0000", 2024-09-11T09:52:35.642 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:35.642 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.642 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.643 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:35.643 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.824556+0000", 2024-09-11T09:52:35.643 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:35.643 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.643 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.643 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:35.643 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.824564+0000", 2024-09-11T09:52:35.643 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.2409999999999995e-06 2024-09-11T09:52:35.643 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.643 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.643 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:35.643 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.824613+0000", 2024-09-11T09:52:35.643 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.9579000000000001e-05 2024-09-11T09:52:35.643 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.644 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.644 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:35.644 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.824638+0000", 2024-09-11T09:52:35.644 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.4510000000000001e-05 2024-09-11T09:52:35.644 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.644 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.644 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:35.644 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.824653+0000", 2024-09-11T09:52:35.644 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.4831000000000001e-05 2024-09-11T09:52:35.644 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.644 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.644 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:35.644 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.829072+0000", 2024-09-11T09:52:35.644 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0044195700000000003 2024-09-11T09:52:35.645 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.645 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.645 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:35.645 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.829166+0000", 2024-09-11T09:52:35.645 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 9.3212000000000001e-05 2024-09-11T09:52:35.645 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.645 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.645 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1", 2024-09-11T09:52:35.645 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.829309+0000", 2024-09-11T09:52:35.645 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00014301900000000001 2024-09-11T09:52:35.645 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.645 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.645 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:35.645 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.831074+0000", 2024-09-11T09:52:35.646 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.001765305 2024-09-11T09:52:35.646 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.646 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.646 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:35.646 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.831751+0000", 2024-09-11T09:52:35.646 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00067694000000000001 2024-09-11T09:52:35.646 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.646 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.646 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:35.646 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.831803+0000", 2024-09-11T09:52:35.646 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.2441000000000002e-05 2024-09-11T09:52:35.646 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.646 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.646 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:35.647 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.831835+0000", 2024-09-11T09:52:35.647 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.1798999999999998e-05 2024-09-11T09:52:35.647 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:35.647 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:35.647 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:35.647 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.647 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.647 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.5314.0:71 2.1 2:8654721b:::rbd_data.1250f5e25b80.0000000000000000:head [write 655360~131072] snapc 5=[5,3] ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-09-11T09:52:35.647 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:50.824662+0000", 2024-09-11T09:52:35.647 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 44.718372219999999, 2024-09-11T09:52:35.647 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0082172219999999997, 2024-09-11T09:52:35.647 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:35.647 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:35.647 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:35.647 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:35.648 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.5314", 2024-09-11T09:52:35.648 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3853995732", 2024-09-11T09:52:35.648 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 71 2024-09-11T09:52:35.648 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.648 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:35.648 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.648 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:35.648 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.824662+0000", 2024-09-11T09:52:35.648 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:35.648 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.648 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.648 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:35.648 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.824662+0000", 2024-09-11T09:52:35.648 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:35.649 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.649 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.649 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:35.649 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.824662+0000", 2024-09-11T09:52:35.649 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.0699999999999998e-07 2024-09-11T09:52:35.649 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.649 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.649 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:35.649 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.824859+0000", 2024-09-11T09:52:35.649 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00019725799999999999 2024-09-11T09:52:35.649 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.649 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.649 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:35.650 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.824906+0000", 2024-09-11T09:52:35.650 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.7162e-05 2024-09-11T09:52:35.650 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.650 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.650 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:35.650 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.824926+0000", 2024-09-11T09:52:35.650 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 2.0038999999999999e-05 2024-09-11T09:52:35.650 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.650 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.650 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:35.650 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.830313+0000", 2024-09-11T09:52:35.650 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0053868409999999999 2024-09-11T09:52:35.650 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.650 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.651 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:35.651 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.830359+0000", 2024-09-11T09:52:35.651 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.5370000000000001e-05 2024-09-11T09:52:35.651 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.651 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.651 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1", 2024-09-11T09:52:35.651 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.830460+0000", 2024-09-11T09:52:35.651 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00010171299999999999 2024-09-11T09:52:35.651 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.651 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.651 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:35.651 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.831843+0000", 2024-09-11T09:52:35.651 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.001382501 2024-09-11T09:52:35.651 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.652 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.652 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:35.652 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.832769+0000", 2024-09-11T09:52:35.652 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00092638099999999999 2024-09-11T09:52:35.652 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.652 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.652 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:35.652 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.832835+0000", 2024-09-11T09:52:35.652 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.6106999999999996e-05 2024-09-11T09:52:35.652 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.652 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.652 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:35.652 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.832879+0000", 2024-09-11T09:52:35.653 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 4.3543e-05 2024-09-11T09:52:35.653 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:35.653 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:35.653 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:35.653 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.653 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.653 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.5314.0:72 2.1 2:8654721b:::rbd_data.1250f5e25b80.0000000000000000:head [write 786432~131072] snapc 5=[5,3] ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-09-11T09:52:35.653 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:50.824938+0000", 2024-09-11T09:52:35.653 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 44.718095918000003, 2024-09-11T09:52:35.653 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0082389799999999999, 2024-09-11T09:52:35.653 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:35.653 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:35.653 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:35.653 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:35.654 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.5314", 2024-09-11T09:52:35.654 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3853995732", 2024-09-11T09:52:35.654 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 72 2024-09-11T09:52:35.654 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.654 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:35.654 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.654 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:35.654 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.824938+0000", 2024-09-11T09:52:35.654 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:35.654 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.654 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.654 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:35.654 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.824938+0000", 2024-09-11T09:52:35.654 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:35.654 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.655 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.655 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:35.655 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.824939+0000", 2024-09-11T09:52:35.655 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.6199999999999997e-07 2024-09-11T09:52:35.655 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.655 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.655 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:35.655 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.825028+0000", 2024-09-11T09:52:35.655 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.9105999999999997e-05 2024-09-11T09:52:35.655 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.655 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.655 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:35.655 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.825042+0000", 2024-09-11T09:52:35.655 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.3760999999999999e-05 2024-09-11T09:52:35.655 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.656 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.656 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:35.656 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.825061+0000", 2024-09-11T09:52:35.656 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 1.9088000000000001e-05 2024-09-11T09:52:35.656 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.656 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.656 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:35.656 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.831122+0000", 2024-09-11T09:52:35.656 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.006061299 2024-09-11T09:52:35.656 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.656 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.656 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:35.656 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.831195+0000", 2024-09-11T09:52:35.656 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 7.2957e-05 2024-09-11T09:52:35.657 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.657 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.657 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1", 2024-09-11T09:52:35.657 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.831301+0000", 2024-09-11T09:52:35.657 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000106049 2024-09-11T09:52:35.657 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.657 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.657 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:35.657 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.832893+0000", 2024-09-11T09:52:35.657 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0015923770000000001 2024-09-11T09:52:35.657 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.657 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.657 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:35.657 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.833076+0000", 2024-09-11T09:52:35.658 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000182855 2024-09-11T09:52:35.658 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.658 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.658 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:35.658 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.833137+0000", 2024-09-11T09:52:35.658 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 6.1193999999999997e-05 2024-09-11T09:52:35.658 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.658 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.658 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:35.658 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.833177+0000", 2024-09-11T09:52:35.658 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.9632000000000003e-05 2024-09-11T09:52:35.658 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:35.658 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:35.658 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:35.659 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.659 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.659 INFO:teuthology.orchestra.run.smithi017.stdout: "description": "osd_op(client.5314.0:73 2.1 2:8654721b:::rbd_data.1250f5e25b80.0000000000000000:head [write 917504~131072] snapc 5=[5,3] ondisk+write+known_if_redirected+supports_pool_eio e69)", 2024-09-11T09:52:35.659 INFO:teuthology.orchestra.run.smithi017.stdout: "initiated_at": "2024-09-11T09:51:50.825071+0000", 2024-09-11T09:52:35.659 INFO:teuthology.orchestra.run.smithi017.stdout: "age": 44.717962411000002, 2024-09-11T09:52:35.659 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0084039160000000009, 2024-09-11T09:52:35.659 INFO:teuthology.orchestra.run.smithi017.stdout: "continuous": false, 2024-09-11T09:52:35.659 INFO:teuthology.orchestra.run.smithi017.stdout: "type_data": { 2024-09-11T09:52:35.659 INFO:teuthology.orchestra.run.smithi017.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:35.659 INFO:teuthology.orchestra.run.smithi017.stdout: "client_info": { 2024-09-11T09:52:35.659 INFO:teuthology.orchestra.run.smithi017.stdout: "client": "client.5314", 2024-09-11T09:52:35.659 INFO:teuthology.orchestra.run.smithi017.stdout: "client_addr": "172.21.15.107:0/3853995732", 2024-09-11T09:52:35.659 INFO:teuthology.orchestra.run.smithi017.stdout: "tid": 73 2024-09-11T09:52:35.659 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.660 INFO:teuthology.orchestra.run.smithi017.stdout: "events": [ 2024-09-11T09:52:35.660 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.660 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "initiated", 2024-09-11T09:52:35.660 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.825071+0000", 2024-09-11T09:52:35.660 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:35.660 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.660 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.660 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "header_read", 2024-09-11T09:52:35.660 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.825071+0000", 2024-09-11T09:52:35.660 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0 2024-09-11T09:52:35.660 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.660 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.660 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "throttled", 2024-09-11T09:52:35.661 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.825072+0000", 2024-09-11T09:52:35.661 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.5499999999999998e-07 2024-09-11T09:52:35.661 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.661 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.661 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "all_read", 2024-09-11T09:52:35.661 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.825124+0000", 2024-09-11T09:52:35.661 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.2197000000000002e-05 2024-09-11T09:52:35.661 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.661 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.661 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "dispatched", 2024-09-11T09:52:35.661 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.825132+0000", 2024-09-11T09:52:35.661 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.0290000000000005e-06 2024-09-11T09:52:35.661 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.661 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.662 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "queued_for_pg", 2024-09-11T09:52:35.662 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.825136+0000", 2024-09-11T09:52:35.662 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.9380000000000002e-06 2024-09-11T09:52:35.662 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.662 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.662 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "reached_pg", 2024-09-11T09:52:35.662 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.832001+0000", 2024-09-11T09:52:35.662 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.0068650949999999999 2024-09-11T09:52:35.662 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.662 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.662 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "started", 2024-09-11T09:52:35.662 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.832091+0000", 2024-09-11T09:52:35.662 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 8.9524999999999994e-05 2024-09-11T09:52:35.662 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.663 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.663 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "waiting for subops from 1", 2024-09-11T09:52:35.663 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.832234+0000", 2024-09-11T09:52:35.663 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.000142783 2024-09-11T09:52:35.663 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.663 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.663 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "op_commit", 2024-09-11T09:52:35.663 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.833198+0000", 2024-09-11T09:52:35.663 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00096476899999999998 2024-09-11T09:52:35.663 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.663 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.663 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:35.663 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.833386+0000", 2024-09-11T09:52:35.663 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 0.00018772 2024-09-11T09:52:35.663 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.664 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.664 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "commit_sent", 2024-09-11T09:52:35.664 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.833440+0000", 2024-09-11T09:52:35.664 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 5.4228999999999999e-05 2024-09-11T09:52:35.664 INFO:teuthology.orchestra.run.smithi017.stdout: }, 2024-09-11T09:52:35.664 INFO:teuthology.orchestra.run.smithi017.stdout: { 2024-09-11T09:52:35.664 INFO:teuthology.orchestra.run.smithi017.stdout: "event": "done", 2024-09-11T09:52:35.664 INFO:teuthology.orchestra.run.smithi017.stdout: "time": "2024-09-11T09:51:50.833475+0000", 2024-09-11T09:52:35.664 INFO:teuthology.orchestra.run.smithi017.stdout: "duration": 3.5076000000000001e-05 2024-09-11T09:52:35.664 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:35.664 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:35.664 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:35.664 INFO:teuthology.orchestra.run.smithi017.stdout: } 2024-09-11T09:52:35.664 INFO:teuthology.orchestra.run.smithi017.stdout: ] 2024-09-11T09:52:35.664 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:52:35.665 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-11T09:52:35.771 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-11T09:52:35.771 INFO:teuthology.orchestra.run.smithi096.stdout: "ops": [], 2024-09-11T09:52:35.771 INFO:teuthology.orchestra.run.smithi096.stdout: "num_ops": 0 2024-09-11T09:52:35.771 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-11T09:52:35.780 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-11T09:52:35.883 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-11T09:52:35.883 INFO:teuthology.orchestra.run.smithi096.stdout: "ops": [], 2024-09-11T09:52:35.883 INFO:teuthology.orchestra.run.smithi096.stdout: "complaint_time": 30, 2024-09-11T09:52:35.883 INFO:teuthology.orchestra.run.smithi096.stdout: "num_blocked_ops": 0 2024-09-11T09:52:35.883 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-11T09:52:35.892 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-11T09:52:35.958 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-11T09:52:35.997 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-11T09:52:35.997 INFO:teuthology.orchestra.run.smithi096.stdout: "size": 20, 2024-09-11T09:52:35.997 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 600, 2024-09-11T09:52:35.997 INFO:teuthology.orchestra.run.smithi096.stdout: "ops": [ 2024-09-11T09:52:35.997 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:35.997 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:111 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:52:35.997 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.711668+0000", 2024-09-11T09:52:35.997 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 144.28487779899999, 2024-09-11T09:52:35.997 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0077145570000000004, 2024-09-11T09:52:35.998 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:35.998 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:35.998 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:35.998 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:52:35.998 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:52:35.998 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:52:35.998 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 111 2024-09-11T09:52:35.998 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:35.998 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:35.998 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:35.998 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:35.998 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711668+0000", 2024-09-11T09:52:35.998 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:35.998 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:35.998 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:35.999 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:35.999 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711668+0000", 2024-09-11T09:52:35.999 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:35.999 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:35.999 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:35.999 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:35.999 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711669+0000", 2024-09-11T09:52:35.999 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.99e-07 2024-09-11T09:52:35.999 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:35.999 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:35.999 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:35.999 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711673+0000", 2024-09-11T09:52:35.999 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.7440000000000001e-06 2024-09-11T09:52:35.999 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.000 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.000 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:36.000 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711686+0000", 2024-09-11T09:52:36.000 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.3317e-05 2024-09-11T09:52:36.000 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.000 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.000 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:36.000 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711697+0000", 2024-09-11T09:52:36.000 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1022000000000001e-05 2024-09-11T09:52:36.000 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.000 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.000 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:36.000 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.717065+0000", 2024-09-11T09:52:36.000 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.005367531 2024-09-11T09:52:36.000 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.001 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.001 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:36.001 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.717114+0000", 2024-09-11T09:52:36.001 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.9707999999999999e-05 2024-09-11T09:52:36.001 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.001 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.001 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:52:36.001 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.717190+0000", 2024-09-11T09:52:36.001 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.5394000000000004e-05 2024-09-11T09:52:36.001 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.001 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.001 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:36.001 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718793+0000", 2024-09-11T09:52:36.002 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001603653 2024-09-11T09:52:36.002 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.002 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.002 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:52:36.002 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719329+0000", 2024-09-11T09:52:36.002 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00053543799999999995 2024-09-11T09:52:36.002 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.002 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.002 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:36.002 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719359+0000", 2024-09-11T09:52:36.002 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.0420999999999999e-05 2024-09-11T09:52:36.002 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.003 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.003 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:36.003 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719383+0000", 2024-09-11T09:52:36.003 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.3730000000000001e-05 2024-09-11T09:52:36.003 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:36.003 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:36.003 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:36.003 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.003 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.003 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:112 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:52:36.003 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.711712+0000", 2024-09-11T09:52:36.003 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 144.28483423899999, 2024-09-11T09:52:36.003 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.008162084, 2024-09-11T09:52:36.004 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:36.004 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:36.004 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:36.004 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:52:36.004 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:52:36.004 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:52:36.004 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 112 2024-09-11T09:52:36.004 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.004 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:36.004 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.004 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:36.004 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711712+0000", 2024-09-11T09:52:36.004 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:36.004 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.004 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.005 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:36.005 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711712+0000", 2024-09-11T09:52:36.005 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:36.005 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.005 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.005 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:36.005 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711712+0000", 2024-09-11T09:52:36.005 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.6000000000000004e-07 2024-09-11T09:52:36.005 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.005 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.005 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:36.005 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711808+0000", 2024-09-11T09:52:36.005 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.5977999999999995e-05 2024-09-11T09:52:36.005 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.006 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.006 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:36.006 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711819+0000", 2024-09-11T09:52:36.006 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.0755e-05 2024-09-11T09:52:36.006 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.006 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.006 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:36.006 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711826+0000", 2024-09-11T09:52:36.006 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.7970000000000003e-06 2024-09-11T09:52:36.006 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.006 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.006 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:36.006 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.717435+0000", 2024-09-11T09:52:36.006 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0056090239999999998 2024-09-11T09:52:36.006 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.007 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.007 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:36.007 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.717496+0000", 2024-09-11T09:52:36.007 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.0543e-05 2024-09-11T09:52:36.007 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.007 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.007 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:52:36.007 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.717591+0000", 2024-09-11T09:52:36.007 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.5667000000000004e-05 2024-09-11T09:52:36.007 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.007 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.007 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:36.007 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718831+0000", 2024-09-11T09:52:36.008 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001240204 2024-09-11T09:52:36.008 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.008 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.008 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:52:36.008 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719819+0000", 2024-09-11T09:52:36.008 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00098751500000000001 2024-09-11T09:52:36.008 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.008 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.008 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:36.008 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719850+0000", 2024-09-11T09:52:36.008 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.1535999999999998e-05 2024-09-11T09:52:36.008 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.008 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.008 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:36.009 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719874+0000", 2024-09-11T09:52:36.009 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.3504999999999999e-05 2024-09-11T09:52:36.009 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:36.009 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:36.009 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:36.009 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.009 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.009 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:113 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:52:36.009 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.711842+0000", 2024-09-11T09:52:36.009 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 144.28470374, 2024-09-11T09:52:36.009 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0081160099999999999, 2024-09-11T09:52:36.009 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:36.009 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:36.009 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:36.009 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:52:36.010 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:52:36.010 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:52:36.010 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 113 2024-09-11T09:52:36.010 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.010 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:36.010 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.010 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:36.010 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711842+0000", 2024-09-11T09:52:36.010 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:36.010 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.010 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.010 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:36.010 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711842+0000", 2024-09-11T09:52:36.010 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:36.011 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.011 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.011 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:36.011 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711843+0000", 2024-09-11T09:52:36.011 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.4399999999999999e-07 2024-09-11T09:52:36.011 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.011 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.011 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:36.011 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711848+0000", 2024-09-11T09:52:36.011 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.4159999999999997e-06 2024-09-11T09:52:36.011 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.011 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.012 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:36.012 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711856+0000", 2024-09-11T09:52:36.012 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.7870000000000007e-06 2024-09-11T09:52:36.012 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.012 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.012 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:36.012 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711862+0000", 2024-09-11T09:52:36.012 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.3689999999999998e-06 2024-09-11T09:52:36.012 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.012 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.012 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:36.012 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.717965+0000", 2024-09-11T09:52:36.012 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0061028899999999997 2024-09-11T09:52:36.013 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.013 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.013 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:36.013 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718023+0000", 2024-09-11T09:52:36.013 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.8313000000000002e-05 2024-09-11T09:52:36.013 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.013 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.013 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:52:36.013 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718109+0000", 2024-09-11T09:52:36.013 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.6205000000000006e-05 2024-09-11T09:52:36.013 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.013 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.013 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:36.014 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719808+0000", 2024-09-11T09:52:36.014 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0016990060000000001 2024-09-11T09:52:36.014 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.014 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.014 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:52:36.014 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719883+0000", 2024-09-11T09:52:36.014 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.4624999999999998e-05 2024-09-11T09:52:36.014 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.014 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.014 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:36.014 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719910+0000", 2024-09-11T09:52:36.014 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.7512e-05 2024-09-11T09:52:36.014 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.014 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.015 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:36.015 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719958+0000", 2024-09-11T09:52:36.015 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.8143000000000003e-05 2024-09-11T09:52:36.015 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:36.015 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:36.015 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:36.015 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.015 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.015 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:114 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:52:36.015 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.711887+0000", 2024-09-11T09:52:36.015 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 144.28465912600001, 2024-09-11T09:52:36.015 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0086063589999999992, 2024-09-11T09:52:36.015 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:36.015 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:36.016 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:36.016 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:52:36.016 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:52:36.016 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:52:36.016 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 114 2024-09-11T09:52:36.016 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.016 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:36.016 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.016 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:36.016 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711887+0000", 2024-09-11T09:52:36.016 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:36.016 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.016 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.017 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:36.017 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711887+0000", 2024-09-11T09:52:36.017 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:36.017 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.017 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.017 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:36.017 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711887+0000", 2024-09-11T09:52:36.017 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.1500000000000005e-07 2024-09-11T09:52:36.017 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.017 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.017 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:36.017 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711897+0000", 2024-09-11T09:52:36.017 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.1900000000000001e-06 2024-09-11T09:52:36.017 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.018 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.018 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:36.018 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711902+0000", 2024-09-11T09:52:36.018 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.2349999999999999e-06 2024-09-11T09:52:36.018 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.018 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.018 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:36.018 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711906+0000", 2024-09-11T09:52:36.018 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.7129999999999999e-06 2024-09-11T09:52:36.018 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.018 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.018 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:36.018 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718429+0000", 2024-09-11T09:52:36.019 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0065231960000000002 2024-09-11T09:52:36.019 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.019 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.019 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:36.019 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718480+0000", 2024-09-11T09:52:36.019 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.1161e-05 2024-09-11T09:52:36.019 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.019 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.019 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:52:36.019 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718558+0000", 2024-09-11T09:52:36.019 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.7519000000000001e-05 2024-09-11T09:52:36.019 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.019 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.019 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:36.020 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720038+0000", 2024-09-11T09:52:36.020 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001480016 2024-09-11T09:52:36.020 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.020 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.020 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:52:36.020 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720417+0000", 2024-09-11T09:52:36.020 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00037943699999999999 2024-09-11T09:52:36.020 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.020 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.020 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:36.020 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720452+0000", 2024-09-11T09:52:36.020 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.4595000000000003e-05 2024-09-11T09:52:36.020 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.021 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.021 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:36.021 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720493+0000", 2024-09-11T09:52:36.021 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.1782000000000001e-05 2024-09-11T09:52:36.021 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:36.021 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:36.021 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:36.021 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.021 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.021 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:115 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:52:36.021 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.711912+0000", 2024-09-11T09:52:36.021 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 144.28463445700001, 2024-09-11T09:52:36.021 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0086864750000000008, 2024-09-11T09:52:36.021 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:36.022 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:36.022 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:36.022 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:52:36.022 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:52:36.022 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:52:36.022 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 115 2024-09-11T09:52:36.022 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.022 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:36.022 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.022 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:36.022 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711912+0000", 2024-09-11T09:52:36.022 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:36.022 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.023 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.023 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:36.023 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711912+0000", 2024-09-11T09:52:36.023 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:36.023 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.023 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.023 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:36.023 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711912+0000", 2024-09-11T09:52:36.023 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.3900000000000001e-07 2024-09-11T09:52:36.023 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.023 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.023 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:36.023 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711969+0000", 2024-09-11T09:52:36.023 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.7501999999999999e-05 2024-09-11T09:52:36.024 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.024 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.024 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:36.024 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711986+0000", 2024-09-11T09:52:36.024 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.6402999999999999e-05 2024-09-11T09:52:36.024 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.024 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.024 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:36.024 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711990+0000", 2024-09-11T09:52:36.024 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.929e-06 2024-09-11T09:52:36.024 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.024 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.024 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:36.025 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718861+0000", 2024-09-11T09:52:36.025 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0068710530000000002 2024-09-11T09:52:36.025 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.025 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.025 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:36.025 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718908+0000", 2024-09-11T09:52:36.025 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.7431e-05 2024-09-11T09:52:36.025 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.025 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.025 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:52:36.025 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.718984+0000", 2024-09-11T09:52:36.025 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.5500999999999994e-05 2024-09-11T09:52:36.026 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.026 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.026 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:52:36.026 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720500+0000", 2024-09-11T09:52:36.026 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0015158820000000001 2024-09-11T09:52:36.026 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.026 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.026 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:36.026 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720536+0000", 2024-09-11T09:52:36.026 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.6118000000000002e-05 2024-09-11T09:52:36.026 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.026 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.026 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:36.027 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720570+0000", 2024-09-11T09:52:36.027 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.4387999999999997e-05 2024-09-11T09:52:36.027 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.027 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.027 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:36.027 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720598+0000", 2024-09-11T09:52:36.027 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.8028999999999999e-05 2024-09-11T09:52:36.027 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:36.027 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:36.027 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:36.027 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.027 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.027 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:116 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:52:36.028 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.711998+0000", 2024-09-11T09:52:36.028 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 144.28454827799999, 2024-09-11T09:52:36.028 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.010785813999999999, 2024-09-11T09:52:36.028 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:36.028 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:36.028 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:36.028 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:52:36.028 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:52:36.028 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:52:36.028 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 116 2024-09-11T09:52:36.028 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.028 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:36.028 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.028 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:36.028 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711998+0000", 2024-09-11T09:52:36.029 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:36.029 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.029 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.029 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:36.029 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711998+0000", 2024-09-11T09:52:36.029 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:36.029 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.029 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.029 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:36.029 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.711998+0000", 2024-09-11T09:52:36.029 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.3300000000000003e-07 2024-09-11T09:52:36.029 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.029 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.030 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:36.030 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712001+0000", 2024-09-11T09:52:36.030 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.9349999999999999e-06 2024-09-11T09:52:36.030 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.030 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.030 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:36.030 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712006+0000", 2024-09-11T09:52:36.030 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.1630000000000004e-06 2024-09-11T09:52:36.030 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.030 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.030 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:36.030 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712010+0000", 2024-09-11T09:52:36.030 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.574e-06 2024-09-11T09:52:36.030 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.031 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.031 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:36.031 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719415+0000", 2024-09-11T09:52:36.031 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0074049099999999998 2024-09-11T09:52:36.031 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.031 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.031 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:36.031 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719465+0000", 2024-09-11T09:52:36.031 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.0129000000000001e-05 2024-09-11T09:52:36.031 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.031 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.031 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:52:36.031 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.719545+0000", 2024-09-11T09:52:36.031 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.0253999999999994e-05 2024-09-11T09:52:36.032 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.032 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.032 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:36.032 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720636+0000", 2024-09-11T09:52:36.032 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0010909330000000001 2024-09-11T09:52:36.032 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.032 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.032 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:52:36.032 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.722707+0000", 2024-09-11T09:52:36.032 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0020713300000000001 2024-09-11T09:52:36.032 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.032 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.032 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:36.033 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.722743+0000", 2024-09-11T09:52:36.033 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.5351000000000001e-05 2024-09-11T09:52:36.033 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.033 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.033 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:36.033 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.722784+0000", 2024-09-11T09:52:36.033 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.0802e-05 2024-09-11T09:52:36.033 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:36.033 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:36.033 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:36.033 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.033 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.033 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:52:36.033 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:11.712014+0000", 2024-09-11T09:52:36.034 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 144.284532238, 2024-09-11T09:52:36.034 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.013420938, 2024-09-11T09:52:36.034 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:36.034 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:36.034 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:36.034 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:52:36.034 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:52:36.034 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:52:36.034 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 117 2024-09-11T09:52:36.034 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.034 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:36.034 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.034 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:36.034 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712014+0000", 2024-09-11T09:52:36.035 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:36.035 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.035 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.035 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:36.035 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712014+0000", 2024-09-11T09:52:36.035 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:36.035 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.035 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.035 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:36.035 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712014+0000", 2024-09-11T09:52:36.035 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.2000000000000001e-07 2024-09-11T09:52:36.035 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.035 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.036 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:36.036 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712016+0000", 2024-09-11T09:52:36.036 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.2309999999999999e-06 2024-09-11T09:52:36.036 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.036 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.036 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:36.036 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712020+0000", 2024-09-11T09:52:36.036 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.3830000000000001e-06 2024-09-11T09:52:36.036 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.036 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.036 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:36.036 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.712023+0000", 2024-09-11T09:52:36.037 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.3340000000000002e-06 2024-09-11T09:52:36.037 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.037 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.037 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:36.037 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720083+0000", 2024-09-11T09:52:36.037 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0080601189999999993 2024-09-11T09:52:36.037 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.037 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.037 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:36.037 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720125+0000", 2024-09-11T09:52:36.037 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.2029e-05 2024-09-11T09:52:36.037 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.037 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.037 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:52:36.038 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720205+0000", 2024-09-11T09:52:36.038 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.0010000000000001e-05 2024-09-11T09:52:36.038 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.038 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.038 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:36.038 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.720849+0000", 2024-09-11T09:52:36.038 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00064369899999999996 2024-09-11T09:52:36.038 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.038 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.038 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:52:36.038 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.725357+0000", 2024-09-11T09:52:36.038 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0045077379999999998 2024-09-11T09:52:36.038 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.038 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.038 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:36.039 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.725392+0000", 2024-09-11T09:52:36.039 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.5862999999999999e-05 2024-09-11T09:52:36.039 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.039 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.039 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:36.039 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:11.725435+0000", 2024-09-11T09:52:36.039 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.2311999999999997e-05 2024-09-11T09:52:36.039 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:36.039 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:36.039 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:36.039 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.039 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.039 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4183.0:159 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e13)", 2024-09-11T09:52:36.040 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:12.452099+0000", 2024-09-11T09:52:36.040 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 143.54444665, 2024-09-11T09:52:36.040 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0069885950000000002, 2024-09-11T09:52:36.040 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:36.040 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:36.040 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:36.040 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:52:36.040 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4183", 2024-09-11T09:52:36.040 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.96:0/1227819331", 2024-09-11T09:52:36.040 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 159 2024-09-11T09:52:36.040 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.040 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:36.040 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.040 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:36.041 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452099+0000", 2024-09-11T09:52:36.041 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:36.041 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.041 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.041 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:36.041 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452099+0000", 2024-09-11T09:52:36.041 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:36.041 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.041 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.041 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:36.041 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452100+0000", 2024-09-11T09:52:36.041 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.2800000000000003e-07 2024-09-11T09:52:36.041 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.041 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.042 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:36.042 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452102+0000", 2024-09-11T09:52:36.042 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.824e-06 2024-09-11T09:52:36.042 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.042 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.042 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:36.042 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452105+0000", 2024-09-11T09:52:36.042 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.0460000000000001e-06 2024-09-11T09:52:36.042 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.042 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.042 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:36.042 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.452107+0000", 2024-09-11T09:52:36.042 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.543e-06 2024-09-11T09:52:36.043 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.043 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.043 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:36.043 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.455484+0000", 2024-09-11T09:52:36.043 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0033769759999999998 2024-09-11T09:52:36.043 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.043 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.043 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:36.043 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.455527+0000", 2024-09-11T09:52:36.043 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.2589999999999997e-05 2024-09-11T09:52:36.043 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.043 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.043 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:52:36.043 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.455594+0000", 2024-09-11T09:52:36.044 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.6775999999999999e-05 2024-09-11T09:52:36.044 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.044 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.044 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:36.044 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.456312+0000", 2024-09-11T09:52:36.044 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00071874700000000005 2024-09-11T09:52:36.044 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.044 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.044 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:52:36.044 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.459015+0000", 2024-09-11T09:52:36.044 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0027028540000000002 2024-09-11T09:52:36.044 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.044 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.044 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:36.045 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.459054+0000", 2024-09-11T09:52:36.045 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.8840000000000001e-05 2024-09-11T09:52:36.045 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.045 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.045 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:36.045 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:12.459088+0000", 2024-09-11T09:52:36.045 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.4071e-05 2024-09-11T09:52:36.045 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:36.045 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:36.045 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:36.045 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.045 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.045 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4688.0:38 2.5 2:abde2694:::rbd_data.1250f5e25b80.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:36.046 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:22.903926+0000", 2024-09-11T09:52:36.046 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 73.092619974000002, 2024-09-11T09:52:36.046 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.028303676, 2024-09-11T09:52:36.046 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:36.046 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:36.046 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:36.046 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:52:36.046 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4688", 2024-09-11T09:52:36.046 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:52:36.046 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 38 2024-09-11T09:52:36.046 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.046 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:36.046 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.046 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:36.046 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.903926+0000", 2024-09-11T09:52:36.047 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:36.047 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.047 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.047 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:36.047 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.903926+0000", 2024-09-11T09:52:36.047 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:36.047 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.047 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.047 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:36.047 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.903928+0000", 2024-09-11T09:52:36.047 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.4249999999999999e-06 2024-09-11T09:52:36.047 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.047 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.047 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:36.048 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.914774+0000", 2024-09-11T09:52:36.048 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.010846613 2024-09-11T09:52:36.048 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.048 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.048 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:36.048 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.914793+0000", 2024-09-11T09:52:36.048 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.8879999999999999e-05 2024-09-11T09:52:36.048 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.048 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.048 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:36.048 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.914801+0000", 2024-09-11T09:52:36.048 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.2230000000000007e-06 2024-09-11T09:52:36.048 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.048 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.048 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:36.049 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.914854+0000", 2024-09-11T09:52:36.049 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.2883999999999997e-05 2024-09-11T09:52:36.049 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.049 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.049 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:36.049 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.914951+0000", 2024-09-11T09:52:36.049 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.6731000000000006e-05 2024-09-11T09:52:36.049 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.049 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.049 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:52:36.049 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.915080+0000", 2024-09-11T09:52:36.049 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00012926999999999999 2024-09-11T09:52:36.049 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.050 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.050 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:36.050 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.931469+0000", 2024-09-11T09:52:36.050 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.016389107 2024-09-11T09:52:36.050 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.050 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.050 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:52:36.050 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.932118+0000", 2024-09-11T09:52:36.050 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00064915600000000002 2024-09-11T09:52:36.050 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.050 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.050 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:36.050 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.932177+0000", 2024-09-11T09:52:36.051 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.9032e-05 2024-09-11T09:52:36.051 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.051 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.051 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:36.051 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.932230+0000", 2024-09-11T09:52:36.051 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.2355000000000003e-05 2024-09-11T09:52:36.051 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:36.051 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:36.051 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:36.051 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.051 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.051 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_repop(client.4688.0:36 2.6 e53/15 2:61e1b442:::rbd_data.1250f5e25b80.0000000000000003:head v 53'2, mlcod=0'0)", 2024-09-11T09:52:36.051 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:22.915597+0000", 2024-09-11T09:52:36.051 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 73.080949423999996, 2024-09-11T09:52:36.051 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.020430798, 2024-09-11T09:52:36.051 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:36.052 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:36.052 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:36.052 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:36.052 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.052 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:36.052 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.915597+0000", 2024-09-11T09:52:36.052 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:36.052 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.052 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.052 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:36.052 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.915597+0000", 2024-09-11T09:52:36.052 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:36.052 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.052 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.053 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:36.053 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.915598+0000", 2024-09-11T09:52:36.053 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.7150000000000001e-06 2024-09-11T09:52:36.053 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.053 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.053 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:36.053 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921741+0000", 2024-09-11T09:52:36.053 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0061424339999999996 2024-09-11T09:52:36.053 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.053 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.053 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:36.053 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921757+0000", 2024-09-11T09:52:36.053 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.6144e-05 2024-09-11T09:52:36.054 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.054 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.054 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:36.054 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921765+0000", 2024-09-11T09:52:36.054 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.0809999999999994e-06 2024-09-11T09:52:36.054 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.054 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.054 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:36.054 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921826+0000", 2024-09-11T09:52:36.054 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.0878999999999998e-05 2024-09-11T09:52:36.054 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.054 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.054 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:36.055 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921849+0000", 2024-09-11T09:52:36.055 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.2966999999999999e-05 2024-09-11T09:52:36.055 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.055 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.055 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:36.055 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.935968+0000", 2024-09-11T09:52:36.055 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.014119191 2024-09-11T09:52:36.055 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.055 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.055 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:36.055 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.936027+0000", 2024-09-11T09:52:36.055 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.9386999999999999e-05 2024-09-11T09:52:36.055 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:36.055 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:36.055 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:36.056 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.056 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.056 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_repop(client.4688.0:39 2.6 e53/15 2:743e7616:::rbd_data.1250f5e25b80.0000000000000005:head v 53'3, mlcod=0'0)", 2024-09-11T09:52:36.056 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:22.921806+0000", 2024-09-11T09:52:36.056 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 73.074740374000001, 2024-09-11T09:52:36.056 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.015275018, 2024-09-11T09:52:36.056 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:36.056 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:36.057 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:36.057 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:36.057 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.057 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:36.057 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921806+0000", 2024-09-11T09:52:36.057 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:36.057 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.057 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.057 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:36.057 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921806+0000", 2024-09-11T09:52:36.057 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:36.057 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.057 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.057 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:36.058 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921806+0000", 2024-09-11T09:52:36.058 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.2200000000000003e-07 2024-09-11T09:52:36.058 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.058 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.058 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:36.058 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.925590+0000", 2024-09-11T09:52:36.058 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0037838949999999998 2024-09-11T09:52:36.058 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.058 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.058 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:36.058 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.925605+0000", 2024-09-11T09:52:36.058 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.4321999999999999e-05 2024-09-11T09:52:36.058 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.058 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.059 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:36.059 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.925613+0000", 2024-09-11T09:52:36.059 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.1980000000000001e-06 2024-09-11T09:52:36.059 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.059 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.059 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:36.059 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.925857+0000", 2024-09-11T09:52:36.059 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000244064 2024-09-11T09:52:36.059 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.059 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.059 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:36.059 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.925875+0000", 2024-09-11T09:52:36.059 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.838e-05 2024-09-11T09:52:36.060 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.060 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.060 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:36.060 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.937042+0000", 2024-09-11T09:52:36.060 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.011166416 2024-09-11T09:52:36.060 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.060 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.060 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:36.060 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.937081+0000", 2024-09-11T09:52:36.060 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.9020999999999998e-05 2024-09-11T09:52:36.060 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:36.060 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:36.060 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:36.060 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.061 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.061 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4688.0:49 2.5 2:b6dd9fc9:::rbd_data.1250f5e25b80.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:36.061 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:22.938548+0000", 2024-09-11T09:52:36.061 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 73.057998370999996, 2024-09-11T09:52:36.061 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.022766950000000001, 2024-09-11T09:52:36.061 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:36.061 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:36.061 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:36.061 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:52:36.061 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4688", 2024-09-11T09:52:36.061 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:52:36.061 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 49 2024-09-11T09:52:36.061 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.062 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:36.062 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.062 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:36.062 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.938548+0000", 2024-09-11T09:52:36.062 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:36.062 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.062 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.062 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:36.062 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.938548+0000", 2024-09-11T09:52:36.062 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:36.062 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.062 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.062 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:36.062 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.938550+0000", 2024-09-11T09:52:36.063 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.7710000000000001e-06 2024-09-11T09:52:36.063 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.063 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.063 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:36.063 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.944237+0000", 2024-09-11T09:52:36.063 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0056879790000000001 2024-09-11T09:52:36.063 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.063 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.063 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:36.063 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.944262+0000", 2024-09-11T09:52:36.063 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.4247e-05 2024-09-11T09:52:36.063 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.063 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.064 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:36.064 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.944274+0000", 2024-09-11T09:52:36.064 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.2639999999999999e-05 2024-09-11T09:52:36.064 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.064 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.064 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:36.064 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.944344+0000", 2024-09-11T09:52:36.064 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.9574999999999997e-05 2024-09-11T09:52:36.064 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.064 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.064 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:36.064 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.944474+0000", 2024-09-11T09:52:36.064 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000130464 2024-09-11T09:52:36.064 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.064 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.065 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:52:36.065 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.944585+0000", 2024-09-11T09:52:36.065 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000110096 2024-09-11T09:52:36.065 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.065 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.065 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:52:36.065 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.952724+0000", 2024-09-11T09:52:36.065 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0081398900000000003 2024-09-11T09:52:36.065 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.065 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.065 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:36.065 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.961219+0000", 2024-09-11T09:52:36.065 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0084949789999999997 2024-09-11T09:52:36.066 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.066 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.066 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:36.066 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.961280+0000", 2024-09-11T09:52:36.066 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.0404e-05 2024-09-11T09:52:36.066 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.066 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.066 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:36.066 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.961315+0000", 2024-09-11T09:52:36.066 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.4904999999999999e-05 2024-09-11T09:52:36.066 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:36.066 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:36.066 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:36.066 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.066 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.067 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_repop(client.4688.0:44 2.6 e53/15 2:67246a22:::rbd_data.1250f5e25b80.0000000000000007:head v 53'4, mlcod=53'2)", 2024-09-11T09:52:36.067 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:22.951721+0000", 2024-09-11T09:52:36.067 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 73.044825098999993, 2024-09-11T09:52:36.067 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.01080981, 2024-09-11T09:52:36.067 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:36.067 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:36.067 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:36.067 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:36.067 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.067 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:36.067 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.951721+0000", 2024-09-11T09:52:36.067 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:36.067 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.067 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.067 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:36.068 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.951721+0000", 2024-09-11T09:52:36.068 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:36.068 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.068 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.068 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:36.068 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.951722+0000", 2024-09-11T09:52:36.068 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.0279999999999999e-06 2024-09-11T09:52:36.068 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.068 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.068 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:36.068 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.955612+0000", 2024-09-11T09:52:36.068 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0038896920000000001 2024-09-11T09:52:36.068 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.069 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.069 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:36.069 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.955628+0000", 2024-09-11T09:52:36.069 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.5945000000000001e-05 2024-09-11T09:52:36.069 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.069 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.069 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:36.069 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.955635+0000", 2024-09-11T09:52:36.069 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.8080000000000005e-06 2024-09-11T09:52:36.069 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.069 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.069 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:36.069 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.955700+0000", 2024-09-11T09:52:36.069 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.4114000000000003e-05 2024-09-11T09:52:36.069 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.070 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.070 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:36.070 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.955724+0000", 2024-09-11T09:52:36.070 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.3989999999999999e-05 2024-09-11T09:52:36.070 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.070 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.070 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:36.070 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.962490+0000", 2024-09-11T09:52:36.070 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0067662599999999996 2024-09-11T09:52:36.070 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.070 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.070 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:36.070 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.962531+0000", 2024-09-11T09:52:36.070 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.0973000000000002e-05 2024-09-11T09:52:36.071 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:36.071 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:36.071 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:36.071 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.071 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.071 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_repop(client.4688.0:51 2.6 e53/15 2:6b5894fc:::rbd_data.1250f5e25b80.000000000000000a:head v 53'5, mlcod=53'2)", 2024-09-11T09:52:36.071 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:22.959030+0000", 2024-09-11T09:52:36.071 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 73.037515924999994, 2024-09-11T09:52:36.071 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.011503638, 2024-09-11T09:52:36.071 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:36.071 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:36.071 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:36.071 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:36.071 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.071 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:36.072 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.959030+0000", 2024-09-11T09:52:36.072 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:36.072 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.072 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.072 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:36.072 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.959030+0000", 2024-09-11T09:52:36.072 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:36.072 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.072 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.072 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:36.072 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.959032+0000", 2024-09-11T09:52:36.072 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.356e-06 2024-09-11T09:52:36.072 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.072 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.073 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:36.073 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.962899+0000", 2024-09-11T09:52:36.073 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.003867639 2024-09-11T09:52:36.073 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.073 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.073 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:36.073 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.962909+0000", 2024-09-11T09:52:36.073 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.0105999999999999e-05 2024-09-11T09:52:36.073 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.073 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.073 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:36.073 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.962915+0000", 2024-09-11T09:52:36.073 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.8869999999999997e-06 2024-09-11T09:52:36.073 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.074 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.074 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:36.074 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.962962+0000", 2024-09-11T09:52:36.074 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.6511999999999998e-05 2024-09-11T09:52:36.074 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.074 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.074 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:36.074 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.962989+0000", 2024-09-11T09:52:36.074 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.7702999999999999e-05 2024-09-11T09:52:36.074 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.074 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.074 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:36.074 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.970495+0000", 2024-09-11T09:52:36.075 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0075052449999999998 2024-09-11T09:52:36.075 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.075 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.075 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:36.075 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.970534+0000", 2024-09-11T09:52:36.075 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.9190000000000003e-05 2024-09-11T09:52:36.075 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:36.075 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:36.075 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:36.075 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.075 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.075 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_repop(client.4688.0:58 2.6 e53/15 2:6277c644:::rbd_data.1250f5e25b80.000000000000000e:head v 53'6, mlcod=53'4)", 2024-09-11T09:52:36.075 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:22.996199+0000", 2024-09-11T09:52:36.076 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 73.000347004999995, 2024-09-11T09:52:36.076 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.010145573999999999, 2024-09-11T09:52:36.076 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:36.076 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:36.076 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:36.076 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:36.076 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.076 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:36.076 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.996199+0000", 2024-09-11T09:52:36.076 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:36.076 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.076 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.076 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:36.076 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.996199+0000", 2024-09-11T09:52:36.076 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:36.077 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.077 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.077 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:36.077 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.996200+0000", 2024-09-11T09:52:36.077 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.85e-07 2024-09-11T09:52:36.077 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.077 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.077 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:36.077 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.999914+0000", 2024-09-11T09:52:36.077 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.003714406 2024-09-11T09:52:36.077 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.077 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.077 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:36.078 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.999928+0000", 2024-09-11T09:52:36.078 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.3149000000000001e-05 2024-09-11T09:52:36.078 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.078 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.078 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:36.078 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.999951+0000", 2024-09-11T09:52:36.078 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.3116e-05 2024-09-11T09:52:36.078 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.078 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.078 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:36.078 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.000003+0000", 2024-09-11T09:52:36.078 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.2169000000000001e-05 2024-09-11T09:52:36.078 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.079 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.079 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:36.079 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.000018+0000", 2024-09-11T09:52:36.079 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.5500000000000001e-05 2024-09-11T09:52:36.079 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.079 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.079 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:36.079 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.006303+0000", 2024-09-11T09:52:36.079 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0062846710000000004 2024-09-11T09:52:36.079 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.079 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.079 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:36.079 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.006345+0000", 2024-09-11T09:52:36.079 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.1678000000000003e-05 2024-09-11T09:52:36.079 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:36.080 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:36.080 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:36.080 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.080 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.080 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4694.0:42 2.5 2:a4fc00ec:ns::rbd_data.1256a0c02879.0000000000000004:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:36.080 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.111570+0000", 2024-09-11T09:52:36.080 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 72.884975898999997, 2024-09-11T09:52:36.080 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.038002197000000001, 2024-09-11T09:52:36.080 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:36.080 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:36.080 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:36.080 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:52:36.080 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4694", 2024-09-11T09:52:36.080 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:52:36.081 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 42 2024-09-11T09:52:36.081 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.081 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:36.081 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.081 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:36.081 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.111570+0000", 2024-09-11T09:52:36.081 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:36.081 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.081 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.081 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:36.081 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.111570+0000", 2024-09-11T09:52:36.081 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:36.081 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.082 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.082 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:36.082 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.111572+0000", 2024-09-11T09:52:36.082 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.3909999999999999e-06 2024-09-11T09:52:36.082 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.082 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.082 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:36.082 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.135929+0000", 2024-09-11T09:52:36.082 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.024357410999999999 2024-09-11T09:52:36.082 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.082 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.082 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:36.082 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.135951+0000", 2024-09-11T09:52:36.082 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.1775000000000001e-05 2024-09-11T09:52:36.083 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.083 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.083 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:36.083 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.135962+0000", 2024-09-11T09:52:36.083 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.0903e-05 2024-09-11T09:52:36.083 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.083 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.083 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:36.083 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.136045+0000", 2024-09-11T09:52:36.083 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.3128999999999996e-05 2024-09-11T09:52:36.083 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.083 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.083 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:36.083 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.136229+0000", 2024-09-11T09:52:36.084 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00018393700000000001 2024-09-11T09:52:36.084 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.084 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.084 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 0", 2024-09-11T09:52:36.084 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.136394+0000", 2024-09-11T09:52:36.084 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000165527 2024-09-11T09:52:36.084 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.084 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.084 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:36.084 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.149320+0000", 2024-09-11T09:52:36.084 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.01292568 2024-09-11T09:52:36.084 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.084 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.084 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:52:36.085 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.149493+0000", 2024-09-11T09:52:36.085 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00017285999999999999 2024-09-11T09:52:36.085 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.085 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.085 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:36.085 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.149539+0000", 2024-09-11T09:52:36.085 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.6628000000000001e-05 2024-09-11T09:52:36.085 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.085 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.085 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:36.085 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.149572+0000", 2024-09-11T09:52:36.085 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.2956000000000001e-05 2024-09-11T09:52:36.085 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:36.086 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:36.086 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:36.086 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.086 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.086 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_repop(client.4694.0:45 2.6 e53/15 2:671fb830:ns::rbd_data.1256a0c02879.0000000000000007:head v 53'9, mlcod=53'7)", 2024-09-11T09:52:36.086 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.121833+0000", 2024-09-11T09:52:36.086 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 72.874713541000006, 2024-09-11T09:52:36.086 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.023876192000000001, 2024-09-11T09:52:36.086 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:36.086 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:36.086 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:36.086 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:36.086 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.086 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:36.087 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.121833+0000", 2024-09-11T09:52:36.087 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:36.087 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.087 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.087 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:36.087 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.121833+0000", 2024-09-11T09:52:36.087 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:36.087 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.087 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.087 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:36.087 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.121834+0000", 2024-09-11T09:52:36.087 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.471e-06 2024-09-11T09:52:36.087 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.088 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.088 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:36.088 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.128990+0000", 2024-09-11T09:52:36.088 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0071557879999999997 2024-09-11T09:52:36.088 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.088 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.088 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:36.088 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.129011+0000", 2024-09-11T09:52:36.088 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.0760000000000001e-05 2024-09-11T09:52:36.088 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.088 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.088 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:36.088 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.129022+0000", 2024-09-11T09:52:36.088 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1056000000000001e-05 2024-09-11T09:52:36.089 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.089 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.089 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:36.089 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.129097+0000", 2024-09-11T09:52:36.089 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.5082999999999999e-05 2024-09-11T09:52:36.089 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.089 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.089 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:36.089 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.129130+0000", 2024-09-11T09:52:36.089 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.3345e-05 2024-09-11T09:52:36.089 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.089 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.089 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:36.089 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.145657+0000", 2024-09-11T09:52:36.090 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.016527224 2024-09-11T09:52:36.090 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.090 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.090 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:36.090 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.145709+0000", 2024-09-11T09:52:36.090 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.1465000000000003e-05 2024-09-11T09:52:36.090 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:36.090 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:36.090 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:36.090 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.090 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.090 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4694.0:43 2.0 2:16c8ae84:ns::rbd_data.1256a0c02879.0000000000000005:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:36.090 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.135999+0000", 2024-09-11T09:52:36.090 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 72.860547591, 2024-09-11T09:52:36.090 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.039468309, 2024-09-11T09:52:36.091 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:36.091 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:36.091 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:36.091 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:52:36.091 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4694", 2024-09-11T09:52:36.091 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:52:36.091 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 43 2024-09-11T09:52:36.091 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.091 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:36.091 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.091 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:36.091 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.135999+0000", 2024-09-11T09:52:36.091 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:36.091 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.092 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.092 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:36.092 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.135999+0000", 2024-09-11T09:52:36.092 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:36.092 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.092 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.092 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:36.092 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.136000+0000", 2024-09-11T09:52:36.092 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.6330000000000001e-06 2024-09-11T09:52:36.092 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.092 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.093 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:36.093 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.157961+0000", 2024-09-11T09:52:36.093 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0219606 2024-09-11T09:52:36.093 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.093 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.093 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:36.093 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.157995+0000", 2024-09-11T09:52:36.093 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.4391999999999999e-05 2024-09-11T09:52:36.093 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.093 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.093 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:36.093 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.158011+0000", 2024-09-11T09:52:36.093 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.5794000000000002e-05 2024-09-11T09:52:36.093 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.093 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.094 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:36.094 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.158083+0000", 2024-09-11T09:52:36.094 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.2273000000000004e-05 2024-09-11T09:52:36.094 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.094 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.094 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:36.094 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.158239+0000", 2024-09-11T09:52:36.094 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000155372 2024-09-11T09:52:36.094 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.094 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.094 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 1", 2024-09-11T09:52:36.094 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.158403+0000", 2024-09-11T09:52:36.094 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00016408199999999999 2024-09-11T09:52:36.094 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.094 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.095 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:52:36.095 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.173860+0000", 2024-09-11T09:52:36.095 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.015457347 2024-09-11T09:52:36.095 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.095 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.095 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:36.095 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.175400+0000", 2024-09-11T09:52:36.095 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001540006 2024-09-11T09:52:36.095 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.095 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.095 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:36.095 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.175438+0000", 2024-09-11T09:52:36.095 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.8135e-05 2024-09-11T09:52:36.095 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.096 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.096 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:36.096 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.175467+0000", 2024-09-11T09:52:36.096 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.8674999999999999e-05 2024-09-11T09:52:36.096 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:36.096 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:36.096 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:36.096 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.096 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.096 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4694.0:57 2.0 2:04b5549c:ns::rbd_data.1256a0c02879.000000000000000d:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:36.097 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.158037+0000", 2024-09-11T09:52:36.097 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 72.838508852999993, 2024-09-11T09:52:36.097 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.027748790999999998, 2024-09-11T09:52:36.097 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:36.097 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:36.097 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:36.097 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:52:36.097 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4694", 2024-09-11T09:52:36.097 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:52:36.097 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 57 2024-09-11T09:52:36.097 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.097 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:36.097 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.097 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:36.098 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.158037+0000", 2024-09-11T09:52:36.098 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:36.098 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.098 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.098 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:36.098 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.158037+0000", 2024-09-11T09:52:36.098 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:36.098 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.098 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.098 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:36.098 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.158039+0000", 2024-09-11T09:52:36.098 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.356e-06 2024-09-11T09:52:36.098 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.098 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.098 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:36.099 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.176367+0000", 2024-09-11T09:52:36.099 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.018328739 2024-09-11T09:52:36.099 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.099 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.099 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:36.099 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.176378+0000", 2024-09-11T09:52:36.099 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.0713999999999999e-05 2024-09-11T09:52:36.099 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.099 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.099 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:36.099 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.176383+0000", 2024-09-11T09:52:36.099 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.0209999999999999e-06 2024-09-11T09:52:36.099 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.099 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.100 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:36.100 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.176426+0000", 2024-09-11T09:52:36.100 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.2623999999999997e-05 2024-09-11T09:52:36.100 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.100 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.100 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:36.100 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.176552+0000", 2024-09-11T09:52:36.100 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000126076 2024-09-11T09:52:36.100 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.100 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.100 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 1", 2024-09-11T09:52:36.100 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.176665+0000", 2024-09-11T09:52:36.100 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000112995 2024-09-11T09:52:36.100 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.101 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.101 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:52:36.101 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.183639+0000", 2024-09-11T09:52:36.101 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0069740039999999998 2024-09-11T09:52:36.101 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.101 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.101 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:36.101 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.185689+0000", 2024-09-11T09:52:36.101 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.002049852 2024-09-11T09:52:36.101 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.101 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.101 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:36.101 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.185725+0000", 2024-09-11T09:52:36.102 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.6804999999999997e-05 2024-09-11T09:52:36.102 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.102 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.102 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:36.102 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.185786+0000", 2024-09-11T09:52:36.102 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.0605e-05 2024-09-11T09:52:36.102 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:36.102 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:36.102 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:36.102 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.102 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.102 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4694.0:60 2.0 2:054f2af4:ns::rbd_data.1256a0c02879.000000000000000f:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:36.102 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.186855+0000", 2024-09-11T09:52:36.103 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 72.809690837000005, 2024-09-11T09:52:36.103 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.013682913, 2024-09-11T09:52:36.103 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:36.103 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:36.103 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:36.103 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:52:36.103 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4694", 2024-09-11T09:52:36.103 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:52:36.103 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 60 2024-09-11T09:52:36.103 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.104 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:36.104 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.104 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:36.104 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.186855+0000", 2024-09-11T09:52:36.104 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:36.104 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.104 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.104 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:36.104 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.186855+0000", 2024-09-11T09:52:36.104 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:36.104 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.104 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.104 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:36.105 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.186856+0000", 2024-09-11T09:52:36.105 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.1200000000000001e-07 2024-09-11T09:52:36.105 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.105 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.105 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:36.105 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.191484+0000", 2024-09-11T09:52:36.105 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0046274159999999996 2024-09-11T09:52:36.105 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.105 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.105 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:36.105 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.191496+0000", 2024-09-11T09:52:36.105 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.2149e-05 2024-09-11T09:52:36.105 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.105 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.105 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:36.106 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.191502+0000", 2024-09-11T09:52:36.106 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.1890000000000004e-06 2024-09-11T09:52:36.106 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.106 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.106 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:36.106 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.191544+0000", 2024-09-11T09:52:36.106 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.1724999999999998e-05 2024-09-11T09:52:36.106 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.106 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.106 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:36.106 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.191641+0000", 2024-09-11T09:52:36.106 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.6939000000000001e-05 2024-09-11T09:52:36.106 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.106 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.107 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 1", 2024-09-11T09:52:36.107 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.191731+0000", 2024-09-11T09:52:36.107 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.0651999999999999e-05 2024-09-11T09:52:36.107 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.107 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.107 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:52:36.107 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.199852+0000", 2024-09-11T09:52:36.107 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0081204529999999997 2024-09-11T09:52:36.107 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.107 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.107 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:36.107 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.200461+0000", 2024-09-11T09:52:36.107 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00060900899999999998 2024-09-11T09:52:36.107 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.108 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.108 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:36.108 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.200497+0000", 2024-09-11T09:52:36.108 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.5800000000000003e-05 2024-09-11T09:52:36.108 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.108 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.108 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:36.108 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.200538+0000", 2024-09-11T09:52:36.108 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.1668999999999997e-05 2024-09-11T09:52:36.108 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:36.108 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:36.108 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:36.108 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:36.108 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:36.109 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-11T09:52:36.110 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-11T09:52:36.212 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-11T09:52:36.212 INFO:teuthology.orchestra.run.smithi096.stdout: "ops": [], 2024-09-11T09:52:36.213 INFO:teuthology.orchestra.run.smithi096.stdout: "num_ops": 0 2024-09-11T09:52:36.213 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-11T09:52:36.221 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-11T09:52:36.323 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-11T09:52:36.323 INFO:teuthology.orchestra.run.smithi096.stdout: "ops": [], 2024-09-11T09:52:36.323 INFO:teuthology.orchestra.run.smithi096.stdout: "complaint_time": 30, 2024-09-11T09:52:36.323 INFO:teuthology.orchestra.run.smithi096.stdout: "num_blocked_ops": 0 2024-09-11T09:52:36.324 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-11T09:52:36.332 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-11T09:52:36.436 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-11T09:52:36.436 INFO:teuthology.orchestra.run.smithi096.stdout: "size": 20, 2024-09-11T09:52:36.436 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 600, 2024-09-11T09:52:36.436 INFO:teuthology.orchestra.run.smithi096.stdout: "ops": [ 2024-09-11T09:52:36.436 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.437 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4230.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-11T09:52:36.437 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:18.490736+0000", 2024-09-11T09:52:36.437 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 137.94532995200001, 2024-09-11T09:52:36.437 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0059097569999999999, 2024-09-11T09:52:36.437 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:36.437 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:36.437 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:36.437 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:52:36.437 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4230", 2024-09-11T09:52:36.437 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.17:0/4101109576", 2024-09-11T09:52:36.437 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 1 2024-09-11T09:52:36.437 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.437 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:36.438 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.438 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:36.438 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.490736+0000", 2024-09-11T09:52:36.438 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:36.438 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.438 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.438 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:36.438 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.490736+0000", 2024-09-11T09:52:36.438 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:36.438 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.438 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.438 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:36.438 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.490742+0000", 2024-09-11T09:52:36.438 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.48e-06 2024-09-11T09:52:36.438 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.439 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.439 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:36.439 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.490749+0000", 2024-09-11T09:52:36.439 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.4569999999999999e-06 2024-09-11T09:52:36.439 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.439 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.439 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:36.439 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.490762+0000", 2024-09-11T09:52:36.439 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.2721000000000001e-05 2024-09-11T09:52:36.439 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.439 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.439 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:36.439 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.490774+0000", 2024-09-11T09:52:36.440 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1823e-05 2024-09-11T09:52:36.440 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.440 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.440 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:36.440 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.490846+0000", 2024-09-11T09:52:36.440 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.1878000000000005e-05 2024-09-11T09:52:36.440 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.440 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.440 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for peered", 2024-09-11T09:52:36.440 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.490857+0000", 2024-09-11T09:52:36.440 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1017e-05 2024-09-11T09:52:36.440 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.440 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.440 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:36.441 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.493856+0000", 2024-09-11T09:52:36.441 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00299885 2024-09-11T09:52:36.441 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.441 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.441 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:36.441 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.494020+0000", 2024-09-11T09:52:36.441 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00016436599999999999 2024-09-11T09:52:36.441 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.441 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.441 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 2", 2024-09-11T09:52:36.441 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.494186+0000", 2024-09-11T09:52:36.441 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00016562799999999999 2024-09-11T09:52:36.441 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.442 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.442 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:36.442 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.495407+0000", 2024-09-11T09:52:36.442 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0012217420000000001 2024-09-11T09:52:36.442 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.442 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.442 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:52:36.442 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.496579+0000", 2024-09-11T09:52:36.442 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001171844 2024-09-11T09:52:36.442 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.442 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.442 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:36.442 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.496620+0000", 2024-09-11T09:52:36.442 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.0479000000000003e-05 2024-09-11T09:52:36.442 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.443 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.443 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:36.443 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:18.496646+0000", 2024-09-11T09:52:36.443 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.6472e-05 2024-09-11T09:52:36.443 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:36.443 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:36.443 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:36.443 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.443 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.443 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4230.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18)", 2024-09-11T09:52:36.443 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:50:20.498634+0000", 2024-09-11T09:52:36.443 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 135.93743193099999, 2024-09-11T09:52:36.443 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0050648539999999997, 2024-09-11T09:52:36.444 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:36.444 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:36.444 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:36.444 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:52:36.444 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4230", 2024-09-11T09:52:36.444 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.17:0/4101109576", 2024-09-11T09:52:36.444 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 7 2024-09-11T09:52:36.444 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.444 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:36.444 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.444 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:36.444 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.498634+0000", 2024-09-11T09:52:36.444 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:36.444 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.445 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.445 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:36.445 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.498634+0000", 2024-09-11T09:52:36.445 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:36.445 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.445 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.445 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:36.445 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.498637+0000", 2024-09-11T09:52:36.445 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.1919999999999999e-06 2024-09-11T09:52:36.445 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.445 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.445 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:36.445 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.498648+0000", 2024-09-11T09:52:36.445 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1116e-05 2024-09-11T09:52:36.446 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.446 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.446 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:36.446 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.498666+0000", 2024-09-11T09:52:36.446 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.8411999999999999e-05 2024-09-11T09:52:36.446 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.446 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.446 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:36.446 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.498675+0000", 2024-09-11T09:52:36.446 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.4059999999999994e-06 2024-09-11T09:52:36.446 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.446 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.446 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:36.446 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.498745+0000", 2024-09-11T09:52:36.446 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.9764000000000005e-05 2024-09-11T09:52:36.446 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.447 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.447 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op must wait for map", 2024-09-11T09:52:36.447 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.498754+0000", 2024-09-11T09:52:36.447 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.5969999999999992e-06 2024-09-11T09:52:36.447 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.447 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.447 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:36.447 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.500385+0000", 2024-09-11T09:52:36.447 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0016308360000000001 2024-09-11T09:52:36.447 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.447 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.447 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:36.447 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.500473+0000", 2024-09-11T09:52:36.448 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.7966000000000004e-05 2024-09-11T09:52:36.448 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.448 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.448 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 1", 2024-09-11T09:52:36.448 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.500591+0000", 2024-09-11T09:52:36.448 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000117895 2024-09-11T09:52:36.448 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.448 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.448 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:36.448 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.501685+0000", 2024-09-11T09:52:36.448 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001094886 2024-09-11T09:52:36.448 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.449 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.449 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:52:36.449 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.503627+0000", 2024-09-11T09:52:36.449 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001941199 2024-09-11T09:52:36.449 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.449 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.449 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:36.449 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.503667+0000", 2024-09-11T09:52:36.449 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.0645000000000001e-05 2024-09-11T09:52:36.449 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.449 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.449 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:36.449 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:50:20.503699+0000", 2024-09-11T09:52:36.449 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.1940000000000003e-05 2024-09-11T09:52:36.449 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:36.450 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:36.450 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:36.450 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.450 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.450 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4688.0:32 2.2 2:45941fd2:::rbd_data.1250f5e25b80.0000000000000001:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:36.450 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:22.896607+0000", 2024-09-11T09:52:36.450 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 73.539459328000007, 2024-09-11T09:52:36.450 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.038655179999999997, 2024-09-11T09:52:36.450 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:36.450 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:36.450 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:36.450 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:52:36.450 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4688", 2024-09-11T09:52:36.450 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:52:36.450 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 32 2024-09-11T09:52:36.451 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.451 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:36.451 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.451 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:36.451 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.896607+0000", 2024-09-11T09:52:36.451 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:36.451 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.451 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.451 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:36.451 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.896607+0000", 2024-09-11T09:52:36.451 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:36.451 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.451 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.451 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:36.451 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.896608+0000", 2024-09-11T09:52:36.452 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.0690000000000001e-06 2024-09-11T09:52:36.452 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.452 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.452 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:36.452 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.914461+0000", 2024-09-11T09:52:36.452 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.017852903 2024-09-11T09:52:36.452 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.452 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.452 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:36.452 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.914495+0000", 2024-09-11T09:52:36.452 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.4301000000000002e-05 2024-09-11T09:52:36.452 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.452 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.452 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:36.453 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.914522+0000", 2024-09-11T09:52:36.453 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.7127e-05 2024-09-11T09:52:36.453 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.453 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.453 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:36.453 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.914583+0000", 2024-09-11T09:52:36.453 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.0730999999999998e-05 2024-09-11T09:52:36.453 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.453 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.453 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:36.453 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.914709+0000", 2024-09-11T09:52:36.453 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00012590400000000001 2024-09-11T09:52:36.453 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.453 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.453 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 1", 2024-09-11T09:52:36.454 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.914809+0000", 2024-09-11T09:52:36.454 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.9882000000000004e-05 2024-09-11T09:52:36.454 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.454 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.454 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:36.454 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.932551+0000", 2024-09-11T09:52:36.454 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.017742048999999999 2024-09-11T09:52:36.454 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.454 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.454 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:52:36.454 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.935144+0000", 2024-09-11T09:52:36.454 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0025927630000000001 2024-09-11T09:52:36.454 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.454 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.455 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:36.455 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.935205+0000", 2024-09-11T09:52:36.455 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.1634e-05 2024-09-11T09:52:36.455 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.455 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.455 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:36.455 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.935262+0000", 2024-09-11T09:52:36.455 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.6817000000000002e-05 2024-09-11T09:52:36.455 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:36.455 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:36.455 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:36.455 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.455 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.455 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_repop(client.4688.0:40 2.7 e53/38 2:fe41411c:::rbd_object_map.1250f5e25b80:head v 53'9, mlcod=53'7)", 2024-09-11T09:52:36.456 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:22.920457+0000", 2024-09-11T09:52:36.456 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 73.515609237000007, 2024-09-11T09:52:36.456 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.011132817, 2024-09-11T09:52:36.456 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:36.456 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:36.456 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:36.456 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:36.456 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.456 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:36.456 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.920457+0000", 2024-09-11T09:52:36.456 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:36.456 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.456 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.456 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:36.456 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.920457+0000", 2024-09-11T09:52:36.457 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:36.457 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.457 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.457 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:36.457 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.920459+0000", 2024-09-11T09:52:36.457 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.6729999999999999e-06 2024-09-11T09:52:36.457 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.457 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.457 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:36.457 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.920469+0000", 2024-09-11T09:52:36.457 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.944e-06 2024-09-11T09:52:36.457 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.457 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.457 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:36.458 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.920495+0000", 2024-09-11T09:52:36.458 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.6401999999999999e-05 2024-09-11T09:52:36.458 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.458 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.458 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:36.458 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.920506+0000", 2024-09-11T09:52:36.458 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.133e-05 2024-09-11T09:52:36.458 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.458 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.458 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:36.458 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.920581+0000", 2024-09-11T09:52:36.458 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.4932999999999996e-05 2024-09-11T09:52:36.458 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.458 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.459 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:36.459 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.920612+0000", 2024-09-11T09:52:36.459 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.0537999999999998e-05 2024-09-11T09:52:36.459 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.459 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.459 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:36.459 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.931537+0000", 2024-09-11T09:52:36.459 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.010925239 2024-09-11T09:52:36.459 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.459 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.459 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:36.459 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.931590+0000", 2024-09-11T09:52:36.459 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.2757999999999999e-05 2024-09-11T09:52:36.459 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:36.459 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:36.460 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:36.460 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.460 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.460 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_repop(client.4688.0:41 2.7 e53/38 2:fe41411c:::rbd_object_map.1250f5e25b80:head v 53'10, mlcod=53'7)", 2024-09-11T09:52:36.460 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:22.921422+0000", 2024-09-11T09:52:36.460 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 73.514643820999993, 2024-09-11T09:52:36.460 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.013734942, 2024-09-11T09:52:36.460 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:36.460 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:36.460 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:36.460 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:36.460 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.460 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:36.460 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921422+0000", 2024-09-11T09:52:36.460 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:36.461 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.461 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.461 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:36.461 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921422+0000", 2024-09-11T09:52:36.461 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:36.461 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.461 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.461 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:36.461 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921426+0000", 2024-09-11T09:52:36.461 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.6150000000000001e-06 2024-09-11T09:52:36.464 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.464 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.464 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:36.464 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921435+0000", 2024-09-11T09:52:36.464 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.9840000000000007e-06 2024-09-11T09:52:36.465 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.465 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.465 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:36.465 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921444+0000", 2024-09-11T09:52:36.465 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.6000000000000007e-06 2024-09-11T09:52:36.465 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.465 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.465 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:36.465 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921450+0000", 2024-09-11T09:52:36.465 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.4690000000000002e-06 2024-09-11T09:52:36.465 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.465 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.465 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:36.465 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921511+0000", 2024-09-11T09:52:36.466 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.0779999999999997e-05 2024-09-11T09:52:36.466 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.466 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.466 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:36.466 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.921531+0000", 2024-09-11T09:52:36.466 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.0302999999999999e-05 2024-09-11T09:52:36.466 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.466 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.466 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:36.466 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.935115+0000", 2024-09-11T09:52:36.466 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.013583471 2024-09-11T09:52:36.466 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.466 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.467 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:36.467 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.935157+0000", 2024-09-11T09:52:36.467 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.2719999999999998e-05 2024-09-11T09:52:36.467 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:36.467 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:36.467 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:36.467 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.467 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.467 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4688.0:42 2.3 2:c54742d8:::rbd_data.1250f5e25b80.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:36.467 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:22.932560+0000", 2024-09-11T09:52:36.467 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 73.503506078000001, 2024-09-11T09:52:36.467 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.017798198000000001, 2024-09-11T09:52:36.467 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:36.467 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:36.467 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:36.468 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:52:36.468 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4688", 2024-09-11T09:52:36.468 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/3325461587", 2024-09-11T09:52:36.468 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 42 2024-09-11T09:52:36.468 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.468 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:36.468 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.468 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:36.468 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.932560+0000", 2024-09-11T09:52:36.468 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:36.468 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.468 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.468 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:36.468 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.932560+0000", 2024-09-11T09:52:36.469 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:36.469 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.469 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.469 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:36.469 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.932562+0000", 2024-09-11T09:52:36.469 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.17e-06 2024-09-11T09:52:36.469 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.469 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.469 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:36.469 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.938851+0000", 2024-09-11T09:52:36.469 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.006288971 2024-09-11T09:52:36.469 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.469 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.470 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:36.470 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.938865+0000", 2024-09-11T09:52:36.470 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.365e-05 2024-09-11T09:52:36.470 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.470 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.470 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:36.470 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.938871+0000", 2024-09-11T09:52:36.470 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.8850000000000004e-06 2024-09-11T09:52:36.470 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.470 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.470 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:36.470 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.938926+0000", 2024-09-11T09:52:36.470 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.4691999999999997e-05 2024-09-11T09:52:36.470 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.470 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.471 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:36.471 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.939059+0000", 2024-09-11T09:52:36.471 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00013310899999999999 2024-09-11T09:52:36.471 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.471 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.471 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 2", 2024-09-11T09:52:36.471 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.939171+0000", 2024-09-11T09:52:36.471 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000112718 2024-09-11T09:52:36.471 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.471 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.471 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:36.471 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.950262+0000", 2024-09-11T09:52:36.471 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.011091063 2024-09-11T09:52:36.471 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.471 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.472 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:52:36.472 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.950274+0000", 2024-09-11T09:52:36.472 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1963e-05 2024-09-11T09:52:36.472 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.472 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.472 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:36.472 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.950318+0000", 2024-09-11T09:52:36.472 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.3519999999999997e-05 2024-09-11T09:52:36.472 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.472 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.472 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:36.472 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:22.950358+0000", 2024-09-11T09:52:36.472 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.0457000000000002e-05 2024-09-11T09:52:36.473 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:36.473 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:36.473 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:36.473 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.473 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.473 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_repop(client.4694.0:34 2.7 e53/38 2:f80bfc0c:ns::rbd_data.1256a0c02879.0000000000000002:head v 53'25, mlcod=53'23)", 2024-09-11T09:52:36.473 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.094944+0000", 2024-09-11T09:52:36.473 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 73.341122686000006, 2024-09-11T09:52:36.473 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.010896785000000001, 2024-09-11T09:52:36.473 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:36.473 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:36.473 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:36.473 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:36.473 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.474 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:36.474 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.094944+0000", 2024-09-11T09:52:36.474 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:36.474 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.474 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.474 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:36.474 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.094944+0000", 2024-09-11T09:52:36.474 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:36.474 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.474 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.474 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:36.474 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.094944+0000", 2024-09-11T09:52:36.474 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.0699999999999996e-07 2024-09-11T09:52:36.475 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.475 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.475 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:36.475 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.099060+0000", 2024-09-11T09:52:36.475 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0041155580000000001 2024-09-11T09:52:36.475 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.475 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.475 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:36.475 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.099082+0000", 2024-09-11T09:52:36.475 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.2323999999999998e-05 2024-09-11T09:52:36.475 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.475 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.475 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:36.476 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.099094+0000", 2024-09-11T09:52:36.476 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1535000000000001e-05 2024-09-11T09:52:36.476 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.476 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.476 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:36.476 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.099160+0000", 2024-09-11T09:52:36.476 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.6449e-05 2024-09-11T09:52:36.476 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.476 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.476 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:36.476 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.099183+0000", 2024-09-11T09:52:36.476 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.3108e-05 2024-09-11T09:52:36.476 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.477 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.477 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:36.477 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.105776+0000", 2024-09-11T09:52:36.477 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0065922760000000002 2024-09-11T09:52:36.477 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.477 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.477 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:36.477 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.105840+0000", 2024-09-11T09:52:36.477 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.4727999999999994e-05 2024-09-11T09:52:36.477 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:36.477 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:36.477 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:36.477 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.478 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.478 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4694.0:41 2.2 2:5e2505c4:ns::rbd_data.1256a0c02879.0000000000000003:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:36.478 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.107511+0000", 2024-09-11T09:52:36.478 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 73.328555004999998, 2024-09-11T09:52:36.478 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.018571066000000001, 2024-09-11T09:52:36.478 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:36.478 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:36.478 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:36.478 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:52:36.478 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4694", 2024-09-11T09:52:36.478 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:52:36.478 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 41 2024-09-11T09:52:36.479 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.479 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:36.479 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.479 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:36.479 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.107511+0000", 2024-09-11T09:52:36.479 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:36.479 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.479 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.479 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:36.479 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.107511+0000", 2024-09-11T09:52:36.479 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:36.479 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.479 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.479 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:36.480 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.107513+0000", 2024-09-11T09:52:36.480 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.3489999999999999e-06 2024-09-11T09:52:36.480 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.480 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.480 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:36.480 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.112704+0000", 2024-09-11T09:52:36.480 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0051918509999999999 2024-09-11T09:52:36.480 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.480 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.480 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:36.480 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.112718+0000", 2024-09-11T09:52:36.480 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.359e-05 2024-09-11T09:52:36.480 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.480 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.481 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:36.481 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.112725+0000", 2024-09-11T09:52:36.481 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.9940000000000003e-06 2024-09-11T09:52:36.481 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.481 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.481 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:36.481 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.112788+0000", 2024-09-11T09:52:36.481 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.2447999999999996e-05 2024-09-11T09:52:36.481 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.481 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.481 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:36.481 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.112937+0000", 2024-09-11T09:52:36.481 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00014921499999999999 2024-09-11T09:52:36.481 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.482 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.482 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 1", 2024-09-11T09:52:36.482 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.113085+0000", 2024-09-11T09:52:36.482 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000148085 2024-09-11T09:52:36.482 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.482 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.482 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:52:36.482 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.124318+0000", 2024-09-11T09:52:36.482 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.011233158 2024-09-11T09:52:36.482 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.482 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.482 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:36.482 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.125957+0000", 2024-09-11T09:52:36.482 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001638701 2024-09-11T09:52:36.483 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.483 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.483 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:36.483 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.126029+0000", 2024-09-11T09:52:36.483 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.2125000000000005e-05 2024-09-11T09:52:36.483 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.483 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.483 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:36.483 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.126082+0000", 2024-09-11T09:52:36.483 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.3550000000000001e-05 2024-09-11T09:52:36.483 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:36.483 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:36.483 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:36.483 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.484 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.484 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4694.0:44 2.3 2:cd1a7d1c:ns::rbd_data.1256a0c02879.0000000000000006:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:36.484 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.112744+0000", 2024-09-11T09:52:36.484 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 73.323322438000005, 2024-09-11T09:52:36.484 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.021672212999999999, 2024-09-11T09:52:36.484 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:36.484 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:36.484 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:36.484 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:52:36.484 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4694", 2024-09-11T09:52:36.484 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:52:36.484 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 44 2024-09-11T09:52:36.484 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.485 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:36.485 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.485 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:36.485 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.112744+0000", 2024-09-11T09:52:36.485 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:36.485 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.485 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.485 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:36.485 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.112744+0000", 2024-09-11T09:52:36.485 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:36.485 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.485 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.485 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:36.486 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.112745+0000", 2024-09-11T09:52:36.486 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.2900000000000002e-07 2024-09-11T09:52:36.486 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.486 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.486 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:36.486 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.119929+0000", 2024-09-11T09:52:36.486 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0071846230000000002 2024-09-11T09:52:36.486 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.486 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.486 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:36.486 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.119949+0000", 2024-09-11T09:52:36.486 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.9264000000000001e-05 2024-09-11T09:52:36.486 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.486 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.486 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:36.487 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.119959+0000", 2024-09-11T09:52:36.487 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.0465999999999999e-05 2024-09-11T09:52:36.487 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.487 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.487 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:36.487 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.120028+0000", 2024-09-11T09:52:36.487 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.9318000000000003e-05 2024-09-11T09:52:36.487 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.487 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.487 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:36.487 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.120160+0000", 2024-09-11T09:52:36.487 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000132015 2024-09-11T09:52:36.487 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.487 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.487 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 2", 2024-09-11T09:52:36.488 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.120260+0000", 2024-09-11T09:52:36.488 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.9397999999999999e-05 2024-09-11T09:52:36.488 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.488 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.488 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:52:36.488 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.134246+0000", 2024-09-11T09:52:36.488 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.013986721000000001 2024-09-11T09:52:36.488 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.488 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.488 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:36.488 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.134295+0000", 2024-09-11T09:52:36.488 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.8649000000000001e-05 2024-09-11T09:52:36.488 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.489 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.489 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:36.489 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.134361+0000", 2024-09-11T09:52:36.489 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.6081999999999995e-05 2024-09-11T09:52:36.489 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.489 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.489 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:36.489 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.134416+0000", 2024-09-11T09:52:36.489 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.4848000000000001e-05 2024-09-11T09:52:36.489 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:36.489 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:36.489 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:36.489 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.489 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.490 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4694.0:46 2.3 2:ce0e1469:ns::rbd_data.1256a0c02879.0000000000000008:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:36.490 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.119993+0000", 2024-09-11T09:52:36.490 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 73.316072962000007, 2024-09-11T09:52:36.490 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.025840610999999999, 2024-09-11T09:52:36.490 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:36.490 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:36.490 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:36.490 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:52:36.490 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4694", 2024-09-11T09:52:36.490 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:52:36.490 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 46 2024-09-11T09:52:36.490 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.490 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:36.490 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.491 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:36.491 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.119993+0000", 2024-09-11T09:52:36.491 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:36.491 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.491 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.491 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:36.491 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.119993+0000", 2024-09-11T09:52:36.491 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:36.491 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.491 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.491 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:36.491 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.119995+0000", 2024-09-11T09:52:36.491 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1930000000000001e-06 2024-09-11T09:52:36.491 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.492 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.492 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:36.492 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.128458+0000", 2024-09-11T09:52:36.492 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0084632630000000004 2024-09-11T09:52:36.492 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.492 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.492 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:36.492 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.128478+0000", 2024-09-11T09:52:36.492 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.0353e-05 2024-09-11T09:52:36.492 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.492 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.492 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:36.492 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.128491+0000", 2024-09-11T09:52:36.492 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.2954e-05 2024-09-11T09:52:36.492 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.493 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.493 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:36.493 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.128558+0000", 2024-09-11T09:52:36.493 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.7076999999999995e-05 2024-09-11T09:52:36.493 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.493 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.493 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:36.493 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.128708+0000", 2024-09-11T09:52:36.493 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00014939599999999999 2024-09-11T09:52:36.493 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.493 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.493 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 2", 2024-09-11T09:52:36.493 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.128866+0000", 2024-09-11T09:52:36.494 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000158145 2024-09-11T09:52:36.494 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.494 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.494 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:36.494 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.141045+0000", 2024-09-11T09:52:36.494 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.012179331999999999 2024-09-11T09:52:36.494 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.494 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.494 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:52:36.494 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.145742+0000", 2024-09-11T09:52:36.494 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0046972050000000003 2024-09-11T09:52:36.494 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.494 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.495 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:36.495 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.145779+0000", 2024-09-11T09:52:36.495 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.6544000000000001e-05 2024-09-11T09:52:36.495 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.495 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.495 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:36.495 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.145834+0000", 2024-09-11T09:52:36.495 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.5148999999999997e-05 2024-09-11T09:52:36.495 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:36.495 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:36.495 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:36.495 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.495 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.495 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4694.0:48 2.3 2:d2090b70:ns::rbd_data.1256a0c02879.0000000000000009:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:36.496 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.130540+0000", 2024-09-11T09:52:36.496 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 73.305526696000001, 2024-09-11T09:52:36.496 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.017167789999999999, 2024-09-11T09:52:36.496 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:36.496 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:36.496 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:36.496 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:52:36.496 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4694", 2024-09-11T09:52:36.496 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:52:36.496 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 48 2024-09-11T09:52:36.496 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.496 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:36.496 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.497 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:36.497 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.130540+0000", 2024-09-11T09:52:36.497 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:36.497 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.497 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.497 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:36.497 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.130540+0000", 2024-09-11T09:52:36.497 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:36.497 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.497 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.498 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:36.498 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.130542+0000", 2024-09-11T09:52:36.498 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.9209999999999999e-06 2024-09-11T09:52:36.498 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.498 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.498 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:36.498 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.134691+0000", 2024-09-11T09:52:36.498 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0041498739999999996 2024-09-11T09:52:36.498 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.498 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.498 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:36.498 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.134705+0000", 2024-09-11T09:52:36.498 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.3669000000000001e-05 2024-09-11T09:52:36.499 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.499 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.499 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:36.499 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.134712+0000", 2024-09-11T09:52:36.499 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.8410000000000002e-06 2024-09-11T09:52:36.499 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.499 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.499 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:36.499 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.134754+0000", 2024-09-11T09:52:36.499 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.2524999999999997e-05 2024-09-11T09:52:36.499 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.499 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.499 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:36.499 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.134875+0000", 2024-09-11T09:52:36.500 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000120332 2024-09-11T09:52:36.500 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.500 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.500 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 2", 2024-09-11T09:52:36.500 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.135005+0000", 2024-09-11T09:52:36.500 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00013024900000000001 2024-09-11T09:52:36.500 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.500 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.500 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:36.500 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.145971+0000", 2024-09-11T09:52:36.500 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.010966203000000001 2024-09-11T09:52:36.500 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.500 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.500 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:52:36.501 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.147600+0000", 2024-09-11T09:52:36.501 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001628382 2024-09-11T09:52:36.501 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.501 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.501 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:36.501 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.147635+0000", 2024-09-11T09:52:36.501 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.5420000000000003e-05 2024-09-11T09:52:36.501 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.501 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.501 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:36.501 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.147707+0000", 2024-09-11T09:52:36.501 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.2373999999999995e-05 2024-09-11T09:52:36.501 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:36.501 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:36.502 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:36.502 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.502 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.502 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_repop(client.4694.0:49 2.7 e53/38 2:e8766325:ns::rbd_object_map.1256a0c02879:head v 53'33, mlcod=53'31)", 2024-09-11T09:52:36.502 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.135391+0000", 2024-09-11T09:52:36.502 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 73.300675131000006, 2024-09-11T09:52:36.502 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0051189859999999998, 2024-09-11T09:52:36.502 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:36.502 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:36.502 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:36.502 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:36.502 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.502 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:36.502 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.135391+0000", 2024-09-11T09:52:36.502 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:36.503 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.503 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.503 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:36.503 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.135391+0000", 2024-09-11T09:52:36.503 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:36.503 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.503 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.503 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:36.503 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.135393+0000", 2024-09-11T09:52:36.503 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.4020000000000001e-06 2024-09-11T09:52:36.503 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.503 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.503 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:36.503 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.135400+0000", 2024-09-11T09:52:36.503 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.5129999999999999e-06 2024-09-11T09:52:36.504 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.504 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.504 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:36.504 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.135411+0000", 2024-09-11T09:52:36.504 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1038e-05 2024-09-11T09:52:36.504 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.504 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.504 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:36.505 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.135418+0000", 2024-09-11T09:52:36.505 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.1489999999999997e-06 2024-09-11T09:52:36.505 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.505 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.505 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:36.505 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.135470+0000", 2024-09-11T09:52:36.505 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.2021999999999998e-05 2024-09-11T09:52:36.505 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.505 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.505 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:36.505 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.135501+0000", 2024-09-11T09:52:36.505 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.0349999999999999e-05 2024-09-11T09:52:36.506 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.506 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.506 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:36.506 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.140453+0000", 2024-09-11T09:52:36.506 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0049522409999999996 2024-09-11T09:52:36.506 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.506 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.506 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:36.506 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.140510+0000", 2024-09-11T09:52:36.506 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.7271000000000001e-05 2024-09-11T09:52:36.506 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:36.506 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:36.506 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:36.506 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.506 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.507 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4694.0:54 2.3 2:c44f0253:ns::rbd_data.1256a0c02879.000000000000000b:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:36.507 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.150151+0000", 2024-09-11T09:52:36.507 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 73.285915279999998, 2024-09-11T09:52:36.507 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.015103484, 2024-09-11T09:52:36.507 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:36.507 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:36.507 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:36.507 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:52:36.507 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4694", 2024-09-11T09:52:36.507 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:52:36.507 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 54 2024-09-11T09:52:36.507 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.507 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:36.508 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.508 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:36.508 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.150151+0000", 2024-09-11T09:52:36.508 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:36.508 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.508 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.508 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:36.508 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.150151+0000", 2024-09-11T09:52:36.508 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:36.508 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.508 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.508 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:36.509 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.150152+0000", 2024-09-11T09:52:36.509 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.2550000000000001e-06 2024-09-11T09:52:36.509 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.509 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.509 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:36.509 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.154528+0000", 2024-09-11T09:52:36.509 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0043758219999999997 2024-09-11T09:52:36.509 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.509 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.509 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:36.509 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.154541+0000", 2024-09-11T09:52:36.509 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.3414999999999999e-05 2024-09-11T09:52:36.509 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.510 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.510 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:36.510 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.154551+0000", 2024-09-11T09:52:36.510 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.4350000000000003e-06 2024-09-11T09:52:36.510 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.510 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.510 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:36.510 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.154615+0000", 2024-09-11T09:52:36.510 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.3627000000000006e-05 2024-09-11T09:52:36.510 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.510 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.510 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:36.510 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.154717+0000", 2024-09-11T09:52:36.510 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000102729 2024-09-11T09:52:36.510 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.511 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.511 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 2", 2024-09-11T09:52:36.511 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.154826+0000", 2024-09-11T09:52:36.511 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000108409 2024-09-11T09:52:36.511 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.511 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.511 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:52:36.511 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.165094+0000", 2024-09-11T09:52:36.511 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.010268210999999999 2024-09-11T09:52:36.511 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.511 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.511 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:36.511 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.165138+0000", 2024-09-11T09:52:36.511 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.4078e-05 2024-09-11T09:52:36.512 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.512 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.512 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:36.512 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.165204+0000", 2024-09-11T09:52:36.512 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.5716000000000006e-05 2024-09-11T09:52:36.512 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.512 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.512 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:36.512 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.165254+0000", 2024-09-11T09:52:36.512 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.0787e-05 2024-09-11T09:52:36.512 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:36.512 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:36.512 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:36.512 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.513 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.513 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_repop(client.4694.0:55 2.7 e53/38 2:e8766325:ns::rbd_object_map.1256a0c02879:head v 53'37, mlcod=53'34)", 2024-09-11T09:52:36.513 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.150221+0000", 2024-09-11T09:52:36.513 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 73.285845414999997, 2024-09-11T09:52:36.513 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00530015, 2024-09-11T09:52:36.513 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:36.513 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:36.513 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:36.513 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:36.513 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.513 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:36.513 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.150221+0000", 2024-09-11T09:52:36.513 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:36.513 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.513 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.514 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:36.514 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.150221+0000", 2024-09-11T09:52:36.514 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:36.514 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.514 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.514 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:36.514 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.150221+0000", 2024-09-11T09:52:36.514 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.9100000000000004e-07 2024-09-11T09:52:36.514 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.514 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.514 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:36.514 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.150226+0000", 2024-09-11T09:52:36.514 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.2250000000000002e-06 2024-09-11T09:52:36.514 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.514 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.515 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:36.515 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.150251+0000", 2024-09-11T09:52:36.515 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.4987e-05 2024-09-11T09:52:36.515 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.515 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.515 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:36.515 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.150258+0000", 2024-09-11T09:52:36.515 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.0790000000000003e-06 2024-09-11T09:52:36.515 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.515 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.515 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:36.515 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.150305+0000", 2024-09-11T09:52:36.515 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.6879000000000003e-05 2024-09-11T09:52:36.515 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.516 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.516 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:36.516 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.150320+0000", 2024-09-11T09:52:36.516 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.5268999999999999e-05 2024-09-11T09:52:36.516 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.516 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.516 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:36.516 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.155493+0000", 2024-09-11T09:52:36.516 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0051731390000000002 2024-09-11T09:52:36.516 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.516 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.516 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:36.516 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.155521+0000", 2024-09-11T09:52:36.516 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.8081000000000001e-05 2024-09-11T09:52:36.517 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:36.517 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:36.517 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:36.517 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.517 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.517 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4694.0:56 2.3 2:c462f029:ns::rbd_data.1256a0c02879.000000000000000c:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:36.517 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.154576+0000", 2024-09-11T09:52:36.517 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 73.281490438999995, 2024-09-11T09:52:36.517 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.020781621, 2024-09-11T09:52:36.517 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:36.517 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:36.517 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:36.517 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:52:36.517 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4694", 2024-09-11T09:52:36.518 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:52:36.518 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 56 2024-09-11T09:52:36.518 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.518 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:36.518 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.518 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:36.518 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.154576+0000", 2024-09-11T09:52:36.518 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:36.518 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.518 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.518 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:36.518 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.154576+0000", 2024-09-11T09:52:36.518 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:36.518 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.518 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.519 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:36.519 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.154577+0000", 2024-09-11T09:52:36.519 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.2249999999999999e-06 2024-09-11T09:52:36.519 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.519 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.519 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:36.519 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.158783+0000", 2024-09-11T09:52:36.519 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0042063889999999996 2024-09-11T09:52:36.519 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.519 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.519 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:36.519 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.158803+0000", 2024-09-11T09:52:36.519 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.9576999999999999e-05 2024-09-11T09:52:36.519 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.519 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.520 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:36.520 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.158815+0000", 2024-09-11T09:52:36.520 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.2221999999999999e-05 2024-09-11T09:52:36.520 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.520 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.520 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:36.520 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.158902+0000", 2024-09-11T09:52:36.520 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.6390000000000005e-05 2024-09-11T09:52:36.520 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.520 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.520 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:36.520 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.159141+0000", 2024-09-11T09:52:36.520 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000238886 2024-09-11T09:52:36.520 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.520 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.521 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 2", 2024-09-11T09:52:36.521 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.159279+0000", 2024-09-11T09:52:36.521 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000138073 2024-09-11T09:52:36.521 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.521 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.521 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:52:36.521 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.173297+0000", 2024-09-11T09:52:36.521 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.014018113 2024-09-11T09:52:36.521 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.521 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.521 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:36.521 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.175278+0000", 2024-09-11T09:52:36.521 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0019817390000000002 2024-09-11T09:52:36.522 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.522 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.522 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:36.522 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.175316+0000", 2024-09-11T09:52:36.522 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.7829e-05 2024-09-11T09:52:36.522 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.522 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.522 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:36.522 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.175357+0000", 2024-09-11T09:52:36.522 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.1177999999999998e-05 2024-09-11T09:52:36.522 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:36.522 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:36.522 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:36.522 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.523 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.523 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.4694.0:59 2.2 2:41adefc9:ns::rbd_data.1256a0c02879.000000000000000e:head [set-alloc-hint object_size 4194304 write_size 4194304,writefull 0~4194304 [fadvise_nocache]] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53)", 2024-09-11T09:52:36.523 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:23.158843+0000", 2024-09-11T09:52:36.523 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 73.277222872999999, 2024-09-11T09:52:36.523 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.017874207, 2024-09-11T09:52:36.523 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:36.523 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:36.523 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:36.523 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:52:36.523 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.4694", 2024-09-11T09:52:36.523 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/1790305122", 2024-09-11T09:52:36.523 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 59 2024-09-11T09:52:36.523 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.523 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:36.523 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.524 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:36.524 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.158843+0000", 2024-09-11T09:52:36.524 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:36.524 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.524 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.524 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:36.524 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.158843+0000", 2024-09-11T09:52:36.524 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:36.524 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.524 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.524 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:36.524 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.158845+0000", 2024-09-11T09:52:36.524 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.7439999999999999e-06 2024-09-11T09:52:36.525 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.525 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.525 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:36.525 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.162713+0000", 2024-09-11T09:52:36.525 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0038678829999999999 2024-09-11T09:52:36.525 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.525 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.525 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:36.525 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.162728+0000", 2024-09-11T09:52:36.526 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.5114e-05 2024-09-11T09:52:36.526 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.526 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.526 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:36.526 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.162737+0000", 2024-09-11T09:52:36.526 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.1099999999999992e-06 2024-09-11T09:52:36.526 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.526 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.526 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:36.526 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.162799+0000", 2024-09-11T09:52:36.526 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.1934999999999996e-05 2024-09-11T09:52:36.526 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.526 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.526 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:36.526 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.162911+0000", 2024-09-11T09:52:36.527 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000111867 2024-09-11T09:52:36.527 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.527 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.527 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 1", 2024-09-11T09:52:36.527 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.163030+0000", 2024-09-11T09:52:36.527 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000119108 2024-09-11T09:52:36.527 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.527 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.527 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:52:36.527 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.174086+0000", 2024-09-11T09:52:36.527 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.01105543 2024-09-11T09:52:36.527 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.527 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.527 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:36.527 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.176614+0000", 2024-09-11T09:52:36.528 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0025284940000000001 2024-09-11T09:52:36.528 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.528 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.528 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:36.528 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.176672+0000", 2024-09-11T09:52:36.528 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.7892000000000001e-05 2024-09-11T09:52:36.528 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.528 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.528 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:36.528 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:23.176718+0000", 2024-09-11T09:52:36.528 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.5630000000000002e-05 2024-09-11T09:52:36.528 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:36.528 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:36.529 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:36.529 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.529 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.529 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.5002.0:91 2.3 2:c54742d8:::rbd_data.1250f5e25b80.0000000000000006:head [write 1179648~917504] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e61)", 2024-09-11T09:52:36.529 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:38.553743+0000", 2024-09-11T09:52:36.529 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 57.882323284000002, 2024-09-11T09:52:36.529 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0051633629999999998, 2024-09-11T09:52:36.529 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:36.529 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:36.529 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:36.529 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:52:36.529 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.5002", 2024-09-11T09:52:36.529 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/2204440692", 2024-09-11T09:52:36.530 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 91 2024-09-11T09:52:36.530 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.530 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:36.530 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.530 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:36.530 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.553743+0000", 2024-09-11T09:52:36.530 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:36.530 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.530 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.530 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:36.530 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.553743+0000", 2024-09-11T09:52:36.530 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:36.530 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.531 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.531 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:36.531 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.553745+0000", 2024-09-11T09:52:36.531 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.5120000000000001e-06 2024-09-11T09:52:36.531 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.531 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.531 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:36.531 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.554588+0000", 2024-09-11T09:52:36.531 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00084352299999999999 2024-09-11T09:52:36.531 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.531 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.531 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:36.531 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.554604+0000", 2024-09-11T09:52:36.531 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.643e-05 2024-09-11T09:52:36.532 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.532 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.532 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:36.532 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.554613+0000", 2024-09-11T09:52:36.532 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.5669999999999995e-06 2024-09-11T09:52:36.532 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.532 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.532 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:36.532 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.554662+0000", 2024-09-11T09:52:36.532 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.8510000000000001e-05 2024-09-11T09:52:36.532 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.532 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.532 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:36.533 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.554732+0000", 2024-09-11T09:52:36.533 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.0297999999999997e-05 2024-09-11T09:52:36.533 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.533 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.533 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 2", 2024-09-11T09:52:36.533 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.554870+0000", 2024-09-11T09:52:36.533 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00013856599999999999 2024-09-11T09:52:36.533 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.533 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.533 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:36.533 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.557875+0000", 2024-09-11T09:52:36.533 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0030042760000000002 2024-09-11T09:52:36.533 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.533 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.534 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:52:36.534 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.558835+0000", 2024-09-11T09:52:36.534 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00096057900000000003 2024-09-11T09:52:36.534 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.534 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.534 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:36.534 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.558874+0000", 2024-09-11T09:52:36.534 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.8992999999999997e-05 2024-09-11T09:52:36.534 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.534 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.534 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:36.534 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.558906+0000", 2024-09-11T09:52:36.534 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.2109e-05 2024-09-11T09:52:36.534 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:36.535 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:36.535 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:36.535 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.535 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.535 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.5002.0:93 2.3 2:c54742d8:::rbd_data.1250f5e25b80.0000000000000006:head [write 2228224~917504] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e61)", 2024-09-11T09:52:36.535 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:51:38.562067+0000", 2024-09-11T09:52:36.535 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 57.873999687999998, 2024-09-11T09:52:36.535 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0050551099999999998, 2024-09-11T09:52:36.535 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:36.535 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:36.535 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:36.535 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:52:36.535 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.5002", 2024-09-11T09:52:36.535 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/2204440692", 2024-09-11T09:52:36.535 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 93 2024-09-11T09:52:36.536 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.536 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:36.536 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.536 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:36.536 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.562067+0000", 2024-09-11T09:52:36.536 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:36.536 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.536 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.536 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:36.536 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.562067+0000", 2024-09-11T09:52:36.536 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:36.536 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.536 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.536 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:36.537 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.562068+0000", 2024-09-11T09:52:36.537 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.0669999999999999e-06 2024-09-11T09:52:36.537 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.537 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.537 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:36.537 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.563170+0000", 2024-09-11T09:52:36.537 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00110256 2024-09-11T09:52:36.537 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.537 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.537 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:36.537 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.563185+0000", 2024-09-11T09:52:36.537 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.4873e-05 2024-09-11T09:52:36.537 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.537 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.537 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:36.538 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.563193+0000", 2024-09-11T09:52:36.538 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.1319999999999994e-06 2024-09-11T09:52:36.538 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.538 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.538 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:36.538 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.563238+0000", 2024-09-11T09:52:36.538 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.4773e-05 2024-09-11T09:52:36.538 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.538 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.538 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:36.538 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.563290+0000", 2024-09-11T09:52:36.538 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.1690000000000001e-05 2024-09-11T09:52:36.538 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.538 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.539 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 2", 2024-09-11T09:52:36.539 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.563365+0000", 2024-09-11T09:52:36.539 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.5023000000000003e-05 2024-09-11T09:52:36.539 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.539 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.539 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:36.539 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.566552+0000", 2024-09-11T09:52:36.539 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0031869960000000001 2024-09-11T09:52:36.539 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.539 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.539 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:52:36.539 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.567048+0000", 2024-09-11T09:52:36.539 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00049605099999999996 2024-09-11T09:52:36.539 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.539 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.540 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:36.540 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.567095+0000", 2024-09-11T09:52:36.540 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.6823999999999998e-05 2024-09-11T09:52:36.540 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.540 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.540 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:36.540 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:51:38.567122+0000", 2024-09-11T09:52:36.540 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.7121e-05 2024-09-11T09:52:36.540 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:36.540 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:36.540 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:36.540 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.540 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.541 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.5669.0:27 2.1 2.5f6219c1 (undecoded) ondisk+write+known_if_redirected+full_try+supports_pool_eio e101)", 2024-09-11T09:52:36.541 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:52:31.937241+0000", 2024-09-11T09:52:36.541 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 4.4988248000000004, 2024-09-11T09:52:36.541 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.0042738, 2024-09-11T09:52:36.541 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:36.541 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:36.541 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "reached pg", 2024-09-11T09:52:36.541 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:52:36.541 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.5669", 2024-09-11T09:52:36.541 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/3950035931", 2024-09-11T09:52:36.541 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 27 2024-09-11T09:52:36.541 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.541 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:36.541 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.542 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:36.542 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:31.937241+0000", 2024-09-11T09:52:36.542 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:36.542 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.542 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.542 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:36.542 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:31.937241+0000", 2024-09-11T09:52:36.542 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:36.542 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.542 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.542 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:36.542 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:31.937243+0000", 2024-09-11T09:52:36.542 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.77e-06 2024-09-11T09:52:36.542 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.542 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.543 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:36.543 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:31.937249+0000", 2024-09-11T09:52:36.543 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.4609999999999999e-06 2024-09-11T09:52:36.543 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.543 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.543 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:36.543 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:31.937260+0000", 2024-09-11T09:52:36.543 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1739000000000001e-05 2024-09-11T09:52:36.543 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.543 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.543 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:36.543 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:31.937269+0000", 2024-09-11T09:52:36.543 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.7050000000000005e-06 2024-09-11T09:52:36.543 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.543 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.543 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:36.544 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:31.938384+0000", 2024-09-11T09:52:36.544 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0011151500000000001 2024-09-11T09:52:36.544 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.544 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.544 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for peered", 2024-09-11T09:52:36.544 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:31.938394+0000", 2024-09-11T09:52:36.544 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.5889999999999992e-06 2024-09-11T09:52:36.544 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.544 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.544 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:36.544 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:32.941501+0000", 2024-09-11T09:52:36.544 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.003107194 2024-09-11T09:52:36.544 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.544 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.545 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:36.545 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:32.941515+0000", 2024-09-11T09:52:36.545 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.4192000000000001e-05 2024-09-11T09:52:36.545 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:36.545 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:36.545 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:36.545 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.545 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.545 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.5669.0:28 2.1 2.5f6219c1 (undecoded) ondisk+write+known_if_redirected+full_try+supports_pool_eio e101)", 2024-09-11T09:52:36.545 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:52:31.938430+0000", 2024-09-11T09:52:36.545 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 4.497636623, 2024-09-11T09:52:36.545 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.0031132519999999, 2024-09-11T09:52:36.545 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:36.545 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:36.546 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "reached pg", 2024-09-11T09:52:36.546 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:52:36.546 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.5669", 2024-09-11T09:52:36.546 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/3950035931", 2024-09-11T09:52:36.546 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 28 2024-09-11T09:52:36.546 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.546 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:36.546 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.546 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:36.546 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:31.938430+0000", 2024-09-11T09:52:36.546 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:36.546 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.546 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.547 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:36.547 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:31.938430+0000", 2024-09-11T09:52:36.547 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:36.547 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.547 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.547 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:36.547 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:31.938430+0000", 2024-09-11T09:52:36.547 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.2900000000000002e-07 2024-09-11T09:52:36.547 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.547 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.547 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:36.547 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:31.938436+0000", 2024-09-11T09:52:36.547 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.3240000000000002e-06 2024-09-11T09:52:36.547 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.548 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.548 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:36.548 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:31.938446+0000", 2024-09-11T09:52:36.548 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.0676000000000001e-05 2024-09-11T09:52:36.548 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.548 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.548 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:36.548 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:31.938454+0000", 2024-09-11T09:52:36.548 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.3050000000000004e-06 2024-09-11T09:52:36.548 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.548 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.548 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:36.548 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:31.938496+0000", 2024-09-11T09:52:36.548 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.2302000000000003e-05 2024-09-11T09:52:36.549 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.549 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.549 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for peered", 2024-09-11T09:52:36.549 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:31.938503+0000", 2024-09-11T09:52:36.549 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.1380000000000001e-06 2024-09-11T09:52:36.549 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.549 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.549 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:36.549 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:32.941538+0000", 2024-09-11T09:52:36.549 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.0030346400000001 2024-09-11T09:52:36.549 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.549 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.549 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:36.550 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:32.941543+0000", 2024-09-11T09:52:36.550 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.0379999999999999e-06 2024-09-11T09:52:36.550 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:36.550 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:36.550 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:36.550 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:36.550 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:36.550 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-11T09:52:36.551 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-11T09:52:36.654 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-11T09:52:36.655 INFO:teuthology.orchestra.run.smithi096.stdout: "ops": [], 2024-09-11T09:52:36.655 INFO:teuthology.orchestra.run.smithi096.stdout: "num_ops": 0 2024-09-11T09:52:36.655 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-11T09:52:36.663 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-11T09:52:36.765 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-11T09:52:36.765 INFO:teuthology.orchestra.run.smithi096.stdout: "ops": [], 2024-09-11T09:52:36.765 INFO:teuthology.orchestra.run.smithi096.stdout: "complaint_time": 30, 2024-09-11T09:52:36.765 INFO:teuthology.orchestra.run.smithi096.stdout: "num_blocked_ops": 0 2024-09-11T09:52:36.766 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-11T09:52:36.774 DEBUG:teuthology.orchestra.run.smithi096:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-11T09:52:36.878 INFO:teuthology.orchestra.run.smithi096.stdout:{ 2024-09-11T09:52:36.878 INFO:teuthology.orchestra.run.smithi096.stdout: "size": 20, 2024-09-11T09:52:36.878 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 600, 2024-09-11T09:52:36.878 INFO:teuthology.orchestra.run.smithi096.stdout: "ops": [ 2024-09-11T09:52:36.878 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.878 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "MOSDPGPush(2.4 82/81 [PushOp(2:3bfc818c:ns::rbd_data.1256a0c02879.0000000000000001:head, version: 53'14, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:3bfc818c:ns::rbd_data.1256a0c02879.0000000000000001:head@53'14, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-11T09:52:36.878 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:52:08.947926+0000", 2024-09-11T09:52:36.879 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 27.930116855000001, 2024-09-11T09:52:36.879 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0020846179999999999, 2024-09-11T09:52:36.879 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:36.879 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:36.879 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "started", 2024-09-11T09:52:36.879 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:36.879 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.879 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:36.879 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.947926+0000", 2024-09-11T09:52:36.879 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:36.879 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.879 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.879 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:36.880 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.947926+0000", 2024-09-11T09:52:36.880 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:36.880 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.880 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.880 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:36.880 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.947927+0000", 2024-09-11T09:52:36.880 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.9599999999999999e-07 2024-09-11T09:52:36.880 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.880 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.880 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:36.880 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.949094+0000", 2024-09-11T09:52:36.880 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001167944 2024-09-11T09:52:36.880 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.880 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.881 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:36.881 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.949110+0000", 2024-09-11T09:52:36.881 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.5650000000000001e-05 2024-09-11T09:52:36.881 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.881 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.881 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:36.881 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.949138+0000", 2024-09-11T09:52:36.881 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.7515e-05 2024-09-11T09:52:36.881 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.881 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.881 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:36.881 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.949196+0000", 2024-09-11T09:52:36.881 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.8272e-05 2024-09-11T09:52:36.882 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.882 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.882 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:36.882 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.949204+0000", 2024-09-11T09:52:36.882 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.2150000000000001e-06 2024-09-11T09:52:36.882 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.882 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.882 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:36.882 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.950011+0000", 2024-09-11T09:52:36.882 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00080652600000000001 2024-09-11T09:52:36.882 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:36.882 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:36.882 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:36.883 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.883 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.883 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "MOSDPGPush(2.4 82/81 [PushOp(2:3bfc818c:ns::rbd_data.1256a0c02879.0000000000000001:head, version: 53'14, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:3bfc818c:ns::rbd_data.1256a0c02879.0000000000000001:head@53'14, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3145728, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-11T09:52:36.883 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:52:08.955761+0000", 2024-09-11T09:52:36.883 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 27.92228218, 2024-09-11T09:52:36.883 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0019711659999999999, 2024-09-11T09:52:36.883 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:36.883 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:36.883 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "started", 2024-09-11T09:52:36.883 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:36.883 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.883 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:36.883 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.955761+0000", 2024-09-11T09:52:36.883 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:36.884 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.884 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.884 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:36.884 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.955761+0000", 2024-09-11T09:52:36.884 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:36.884 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.884 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.884 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:36.884 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.955761+0000", 2024-09-11T09:52:36.884 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.2399999999999997e-07 2024-09-11T09:52:36.884 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.884 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.884 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:36.884 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.956708+0000", 2024-09-11T09:52:36.885 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00094629899999999995 2024-09-11T09:52:36.885 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.885 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.885 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:36.885 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.956731+0000", 2024-09-11T09:52:36.885 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.3175000000000001e-05 2024-09-11T09:52:36.885 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.885 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.885 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:36.885 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.956750+0000", 2024-09-11T09:52:36.885 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.8859e-05 2024-09-11T09:52:36.885 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.885 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.886 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:36.886 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.956828+0000", 2024-09-11T09:52:36.886 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.8567999999999997e-05 2024-09-11T09:52:36.886 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.886 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.886 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:36.886 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.956841+0000", 2024-09-11T09:52:36.886 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.2489e-05 2024-09-11T09:52:36.886 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.886 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.886 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:36.886 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.957732+0000", 2024-09-11T09:52:36.886 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00089105200000000003 2024-09-11T09:52:36.886 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:36.886 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:36.887 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:36.887 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.887 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.887 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "MOSDPGPush(2.4 82/81 [PushOp(2:3bfc818c:ns::rbd_data.1256a0c02879.0000000000000001:head, version: 53'14, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:3bfc818c:ns::rbd_data.1256a0c02879.0000000000000001:head@53'14, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 4194304, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3145728, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-11T09:52:36.887 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:52:08.959483+0000", 2024-09-11T09:52:36.887 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 27.918559649999999, 2024-09-11T09:52:36.887 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0021404520000000002, 2024-09-11T09:52:36.887 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:36.887 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:36.887 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "started", 2024-09-11T09:52:36.887 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:36.887 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.887 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:36.887 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.959483+0000", 2024-09-11T09:52:36.888 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:36.888 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.888 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.888 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:36.888 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.959483+0000", 2024-09-11T09:52:36.888 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:36.888 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.888 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.888 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:36.888 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.959484+0000", 2024-09-11T09:52:36.888 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.4900000000000005e-07 2024-09-11T09:52:36.888 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.888 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.888 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:36.889 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.960450+0000", 2024-09-11T09:52:36.889 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000966141 2024-09-11T09:52:36.889 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.889 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.889 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:36.889 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.960466+0000", 2024-09-11T09:52:36.889 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.6302000000000001e-05 2024-09-11T09:52:36.889 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.889 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.889 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:36.889 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.960477+0000", 2024-09-11T09:52:36.889 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.0406e-05 2024-09-11T09:52:36.889 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.890 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.890 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:36.890 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.960555+0000", 2024-09-11T09:52:36.890 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.7837999999999996e-05 2024-09-11T09:52:36.890 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.890 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.890 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:36.890 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.960568+0000", 2024-09-11T09:52:36.890 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.2904e-05 2024-09-11T09:52:36.890 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.890 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.890 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:36.890 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.961624+0000", 2024-09-11T09:52:36.890 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0010561119999999999 2024-09-11T09:52:36.891 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:36.891 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:36.891 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:36.891 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.891 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.891 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "MOSDPGPush(2.4 82/81 [PushOp(2:2367385a:::rbd_data.1250f5e25b80.0000000000000002:6, version: 79'169, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 1, recovery_info: ObjectRecoveryInfo(2:2367385a:::rbd_data.1250f5e25b80.0000000000000002:6@79'169, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 6=[]:{6: [6, 5, 3]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-11T09:52:36.891 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:52:08.964132+0000", 2024-09-11T09:52:36.891 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 27.913911168999999, 2024-09-11T09:52:36.891 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0021074240000000001, 2024-09-11T09:52:36.891 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:36.891 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:36.891 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "started", 2024-09-11T09:52:36.891 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:36.891 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.892 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:36.892 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.964132+0000", 2024-09-11T09:52:36.892 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:36.892 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.892 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.892 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:36.892 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.964132+0000", 2024-09-11T09:52:36.892 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:36.892 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.892 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.892 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:36.892 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.964133+0000", 2024-09-11T09:52:36.892 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.02e-07 2024-09-11T09:52:36.893 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.893 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.893 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:36.893 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.965104+0000", 2024-09-11T09:52:36.893 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000971849 2024-09-11T09:52:36.893 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.893 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.893 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:36.893 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.965122+0000", 2024-09-11T09:52:36.893 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.7706999999999998e-05 2024-09-11T09:52:36.893 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.893 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.893 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:36.893 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.965134+0000", 2024-09-11T09:52:36.894 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1371e-05 2024-09-11T09:52:36.894 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.894 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.894 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:36.894 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.965221+0000", 2024-09-11T09:52:36.894 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.7037e-05 2024-09-11T09:52:36.894 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.894 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.894 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:36.894 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.965236+0000", 2024-09-11T09:52:36.894 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.5724999999999999e-05 2024-09-11T09:52:36.894 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.894 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.894 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:36.895 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.966239+0000", 2024-09-11T09:52:36.895 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0010028330000000001 2024-09-11T09:52:36.895 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:36.895 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:36.895 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:36.895 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.895 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.895 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "MOSDPGPush(2.4 82/81 [PushOp(2:2367385a:::rbd_data.1250f5e25b80.0000000000000002:6, version: 79'169, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:2367385a:::rbd_data.1250f5e25b80.0000000000000002:6@79'169, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 6=[]:{6: [6, 5, 3]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-11T09:52:36.895 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:52:08.968106+0000", 2024-09-11T09:52:36.895 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 27.909936702, 2024-09-11T09:52:36.895 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.002036936, 2024-09-11T09:52:36.895 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:36.895 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:36.896 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "started", 2024-09-11T09:52:36.896 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:36.896 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.896 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:36.896 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.968106+0000", 2024-09-11T09:52:36.896 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:36.896 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.896 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.896 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:36.896 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.968106+0000", 2024-09-11T09:52:36.896 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:36.896 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.897 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.897 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:36.897 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.968107+0000", 2024-09-11T09:52:36.897 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.0100000000000001e-06 2024-09-11T09:52:36.897 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.897 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.897 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:36.897 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.969086+0000", 2024-09-11T09:52:36.897 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00097889099999999996 2024-09-11T09:52:36.897 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.897 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.897 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:36.897 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.969104+0000", 2024-09-11T09:52:36.898 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.7940000000000001e-05 2024-09-11T09:52:36.898 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.898 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.898 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:36.898 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.969117+0000", 2024-09-11T09:52:36.898 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.3207000000000001e-05 2024-09-11T09:52:36.898 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.898 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.898 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:36.898 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.969192+0000", 2024-09-11T09:52:36.899 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.4419999999999996e-05 2024-09-11T09:52:36.899 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.899 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.899 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:36.899 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.969201+0000", 2024-09-11T09:52:36.899 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.4960000000000002e-06 2024-09-11T09:52:36.899 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.899 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.899 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:36.899 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.970143+0000", 2024-09-11T09:52:36.899 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00094197200000000001 2024-09-11T09:52:36.899 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:36.900 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:36.900 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:36.900 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.900 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.900 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "MOSDPGPush(2.4 82/81 [PushOp(2:2367385a:::rbd_data.1250f5e25b80.0000000000000002:6, version: 79'169, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:2367385a:::rbd_data.1250f5e25b80.0000000000000002:6@79'169, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 6=[]:{6: [6, 5, 3]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 4194304, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3145728, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-11T09:52:36.900 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:52:08.975206+0000", 2024-09-11T09:52:36.900 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 27.902836994000001, 2024-09-11T09:52:36.900 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0020632770000000001, 2024-09-11T09:52:36.900 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:36.900 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:36.900 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "started", 2024-09-11T09:52:36.900 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:36.901 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.901 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:36.901 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.975206+0000", 2024-09-11T09:52:36.901 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:36.901 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.901 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.901 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:36.901 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.975206+0000", 2024-09-11T09:52:36.901 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:36.901 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.901 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.901 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:36.901 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.975207+0000", 2024-09-11T09:52:36.901 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.2399999999999998e-07 2024-09-11T09:52:36.902 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.902 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.902 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:36.902 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.976173+0000", 2024-09-11T09:52:36.902 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00096649400000000003 2024-09-11T09:52:36.902 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.902 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.902 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:36.902 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.976189+0000", 2024-09-11T09:52:36.902 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.6223e-05 2024-09-11T09:52:36.902 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.902 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.902 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:36.903 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.976200+0000", 2024-09-11T09:52:36.903 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.0729e-05 2024-09-11T09:52:36.903 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.903 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.903 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:36.903 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.976265+0000", 2024-09-11T09:52:36.903 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.5326000000000004e-05 2024-09-11T09:52:36.903 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.903 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.903 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:36.903 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.976274+0000", 2024-09-11T09:52:36.903 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.8140000000000007e-06 2024-09-11T09:52:36.903 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.903 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.904 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:36.904 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.977269+0000", 2024-09-11T09:52:36.904 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00099506700000000009 2024-09-11T09:52:36.904 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:36.904 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:36.904 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:36.904 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.904 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.904 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "MOSDPGPush(2.4 82/81 [PushOp(2:26d5c2f0:::rbd_data.1250f5e25b80.000000000000000b:6, version: 79'171, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 1, recovery_info: ObjectRecoveryInfo(2:26d5c2f0:::rbd_data.1250f5e25b80.000000000000000b:6@79'171, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 6=[]:{6: [6, 5, 3]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-11T09:52:36.904 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:52:08.979123+0000", 2024-09-11T09:52:36.904 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 27.898919565, 2024-09-11T09:52:36.904 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0019749350000000001, 2024-09-11T09:52:36.904 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:36.904 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:36.904 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "started", 2024-09-11T09:52:36.905 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:36.905 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.905 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:36.905 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.979123+0000", 2024-09-11T09:52:36.905 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:36.905 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.905 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.905 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:36.905 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.979123+0000", 2024-09-11T09:52:36.905 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:36.905 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.905 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.905 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:36.906 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.979124+0000", 2024-09-11T09:52:36.906 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.5799999999999999e-07 2024-09-11T09:52:36.906 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.906 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.906 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:36.906 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.980088+0000", 2024-09-11T09:52:36.906 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00096377699999999997 2024-09-11T09:52:36.906 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.906 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.906 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:36.906 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.980106+0000", 2024-09-11T09:52:36.907 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.7881e-05 2024-09-11T09:52:36.907 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.907 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.907 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:36.907 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.980117+0000", 2024-09-11T09:52:36.907 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1681e-05 2024-09-11T09:52:36.907 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.907 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.907 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:36.907 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.980189+0000", 2024-09-11T09:52:36.907 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.1365999999999994e-05 2024-09-11T09:52:36.907 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.907 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.907 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:36.908 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.980198+0000", 2024-09-11T09:52:36.908 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.3300000000000005e-06 2024-09-11T09:52:36.908 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.908 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.908 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:36.908 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.981098+0000", 2024-09-11T09:52:36.908 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00090024199999999999 2024-09-11T09:52:36.908 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:36.908 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:36.908 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:36.908 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.908 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.908 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "MOSDPGPush(2.4 82/81 [PushOp(2:26d5c2f0:::rbd_data.1250f5e25b80.000000000000000b:6, version: 79'171, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:26d5c2f0:::rbd_data.1250f5e25b80.000000000000000b:6@79'171, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 6=[]:{6: [6, 5, 3]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-11T09:52:36.908 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:52:08.982550+0000", 2024-09-11T09:52:36.909 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 27.895493120000001, 2024-09-11T09:52:36.909 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0020071540000000001, 2024-09-11T09:52:36.909 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:36.909 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:36.909 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "started", 2024-09-11T09:52:36.909 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:36.909 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.909 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:36.909 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.982550+0000", 2024-09-11T09:52:36.909 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:36.909 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.909 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.909 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:36.910 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.982550+0000", 2024-09-11T09:52:36.910 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:36.910 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.910 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.910 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:36.910 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.982550+0000", 2024-09-11T09:52:36.910 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.89e-07 2024-09-11T09:52:36.910 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.910 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.910 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:36.910 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.983523+0000", 2024-09-11T09:52:36.910 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00097246400000000003 2024-09-11T09:52:36.910 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.910 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.910 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:36.911 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.983540+0000", 2024-09-11T09:52:36.911 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.7388999999999999e-05 2024-09-11T09:52:36.911 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.911 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.911 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:36.911 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.983553+0000", 2024-09-11T09:52:36.911 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.2876e-05 2024-09-11T09:52:36.911 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.911 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.911 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:36.911 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.983623+0000", 2024-09-11T09:52:36.911 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.0199999999999999e-05 2024-09-11T09:52:36.911 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.911 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.911 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:36.912 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.983632+0000", 2024-09-11T09:52:36.912 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.8859999999999993e-06 2024-09-11T09:52:36.912 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.912 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.912 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:36.912 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.984557+0000", 2024-09-11T09:52:36.912 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00092484999999999996 2024-09-11T09:52:36.912 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:36.912 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:36.912 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:36.912 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.912 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.912 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "MOSDPGPush(2.4 82/81 [PushOp(2:26d5c2f0:::rbd_data.1250f5e25b80.000000000000000b:6, version: 79'171, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:26d5c2f0:::rbd_data.1250f5e25b80.000000000000000b:6@79'171, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 6=[]:{6: [6, 5, 3]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 4194304, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3145728, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-11T09:52:36.912 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:52:08.989143+0000", 2024-09-11T09:52:36.913 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 27.888899428999999, 2024-09-11T09:52:36.913 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0022141589999999998, 2024-09-11T09:52:36.913 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:36.913 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:36.913 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "started", 2024-09-11T09:52:36.913 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:36.913 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.913 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:36.913 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.989143+0000", 2024-09-11T09:52:36.913 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:36.913 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.913 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.913 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:36.913 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.989143+0000", 2024-09-11T09:52:36.914 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:36.914 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.914 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.914 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:36.914 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.989144+0000", 2024-09-11T09:52:36.914 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.1699999999999997e-07 2024-09-11T09:52:36.914 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.914 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.914 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:36.914 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.990124+0000", 2024-09-11T09:52:36.914 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00097941400000000007 2024-09-11T09:52:36.914 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.914 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.914 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:36.914 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.990140+0000", 2024-09-11T09:52:36.914 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.6583e-05 2024-09-11T09:52:36.915 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.915 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.915 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:36.915 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.990152+0000", 2024-09-11T09:52:36.915 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1331999999999999e-05 2024-09-11T09:52:36.915 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.915 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.915 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:36.915 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.990220+0000", 2024-09-11T09:52:36.915 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.8777999999999998e-05 2024-09-11T09:52:36.915 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.915 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.915 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:36.915 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.990229+0000", 2024-09-11T09:52:36.915 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.9590000000000001e-06 2024-09-11T09:52:36.916 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.916 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.916 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:36.916 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.991358+0000", 2024-09-11T09:52:36.916 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0011283759999999999 2024-09-11T09:52:36.916 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:36.916 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:36.916 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:36.916 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.916 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.916 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "MOSDPGPush(2.4 82/81 [PushOp(2:239e915b:::rbd_data.1250f5e25b80.000000000000000c:6, version: 79'173, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 1, recovery_info: ObjectRecoveryInfo(2:239e915b:::rbd_data.1250f5e25b80.000000000000000c:6@79'173, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 6=[]:{6: [6, 5, 3]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-11T09:52:36.916 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:52:08.994080+0000", 2024-09-11T09:52:36.916 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 27.883963292000001, 2024-09-11T09:52:36.917 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0019554189999999999, 2024-09-11T09:52:36.917 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:36.917 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:36.917 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "started", 2024-09-11T09:52:36.917 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:36.917 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.917 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:36.917 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.994080+0000", 2024-09-11T09:52:36.917 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:36.917 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.917 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.917 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:36.917 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.994080+0000", 2024-09-11T09:52:36.917 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:36.918 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.918 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.918 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:36.918 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.994080+0000", 2024-09-11T09:52:36.918 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.7599999999999997e-07 2024-09-11T09:52:36.918 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.918 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.918 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:36.918 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.995065+0000", 2024-09-11T09:52:36.918 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00098436199999999991 2024-09-11T09:52:36.918 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.918 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.918 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:36.918 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.995080+0000", 2024-09-11T09:52:36.919 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.5119e-05 2024-09-11T09:52:36.919 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.919 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.919 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:36.919 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.995090+0000", 2024-09-11T09:52:36.919 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.0715e-05 2024-09-11T09:52:36.919 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.919 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.919 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:36.919 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.995152+0000", 2024-09-11T09:52:36.919 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.2089999999999997e-05 2024-09-11T09:52:36.919 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.919 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.919 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:36.919 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.995159+0000", 2024-09-11T09:52:36.919 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.9210000000000003e-06 2024-09-11T09:52:36.920 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.920 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.920 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:36.920 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:08.996035+0000", 2024-09-11T09:52:36.920 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00087563599999999995 2024-09-11T09:52:36.920 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:36.920 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:36.920 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:36.920 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.920 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.920 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "MOSDPGPush(2.4 82/81 [PushOp(2:239e915b:::rbd_data.1250f5e25b80.000000000000000c:6, version: 79'173, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:239e915b:::rbd_data.1250f5e25b80.000000000000000c:6@79'173, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 6=[]:{6: [6, 5, 3]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 4194304, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3145728, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-11T09:52:36.920 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:52:09.005094+0000", 2024-09-11T09:52:36.920 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 27.872948431000001, 2024-09-11T09:52:36.921 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0024127419999999998, 2024-09-11T09:52:36.921 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:36.921 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:36.921 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "started", 2024-09-11T09:52:36.921 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:36.921 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.921 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:36.921 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.005094+0000", 2024-09-11T09:52:36.921 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:36.921 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.921 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.921 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:36.921 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.005094+0000", 2024-09-11T09:52:36.921 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:36.921 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.922 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.922 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:36.922 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.005095+0000", 2024-09-11T09:52:36.922 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.8999999999999997e-07 2024-09-11T09:52:36.922 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.922 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.922 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:36.922 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.006059+0000", 2024-09-11T09:52:36.922 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00096429100000000004 2024-09-11T09:52:36.922 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.922 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.922 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:36.922 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.006075+0000", 2024-09-11T09:52:36.922 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.6036e-05 2024-09-11T09:52:36.922 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.923 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.923 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:36.923 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.006088+0000", 2024-09-11T09:52:36.923 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.2557e-05 2024-09-11T09:52:36.923 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.923 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.923 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:36.923 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.006167+0000", 2024-09-11T09:52:36.923 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.8843000000000004e-05 2024-09-11T09:52:36.923 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.923 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.923 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:36.923 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.006182+0000", 2024-09-11T09:52:36.923 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.4968e-05 2024-09-11T09:52:36.923 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.924 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.924 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:36.924 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.007507+0000", 2024-09-11T09:52:36.924 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001325557 2024-09-11T09:52:36.924 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:36.924 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:36.924 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:36.924 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.924 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.924 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "MOSDPGPush(2.4 82/81 [PushOp(2:2167012f:::rbd_data.1250f5e25b80.000000000000000d:6, version: 79'175, data_included: [1048576~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:2167012f:::rbd_data.1250f5e25b80.000000000000000d:6@79'175, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 6=[]:{6: [6, 5, 3]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-11T09:52:36.924 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:52:09.013431+0000", 2024-09-11T09:52:36.924 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 27.864612096999998, 2024-09-11T09:52:36.925 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.002012247, 2024-09-11T09:52:36.925 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:36.925 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:36.925 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "started", 2024-09-11T09:52:36.925 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:36.925 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.925 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:36.925 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.013431+0000", 2024-09-11T09:52:36.925 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:36.925 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.925 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.925 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:36.925 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.013431+0000", 2024-09-11T09:52:36.925 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:36.926 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.926 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.926 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:36.926 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.013432+0000", 2024-09-11T09:52:36.926 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.1900000000000001e-07 2024-09-11T09:52:36.926 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.926 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.926 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:36.926 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.014359+0000", 2024-09-11T09:52:36.926 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00092687900000000003 2024-09-11T09:52:36.926 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.926 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.926 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:36.927 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.014377+0000", 2024-09-11T09:52:36.927 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.8085e-05 2024-09-11T09:52:36.927 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.927 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.927 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:36.927 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.014391+0000", 2024-09-11T09:52:36.927 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.4099e-05 2024-09-11T09:52:36.927 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.927 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.928 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:36.928 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.014476+0000", 2024-09-11T09:52:36.928 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.4827999999999995e-05 2024-09-11T09:52:36.928 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.928 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.928 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:36.928 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.014489+0000", 2024-09-11T09:52:36.928 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.3169e-05 2024-09-11T09:52:36.928 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.928 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.928 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:36.928 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.015443+0000", 2024-09-11T09:52:36.928 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00095436800000000003 2024-09-11T09:52:36.928 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:36.928 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:36.929 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:36.929 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.929 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.929 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "MOSDPGPush(2.4 82/81 [PushOp(2:2167012f:::rbd_data.1250f5e25b80.000000000000000d:6, version: 79'175, data_included: [2097152~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:2167012f:::rbd_data.1250f5e25b80.000000000000000d:6@79'175, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 6=[]:{6: [6, 5, 3]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3145728, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-11T09:52:36.929 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:52:09.017158+0000", 2024-09-11T09:52:36.929 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 27.860885109000002, 2024-09-11T09:52:36.929 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0020239889999999999, 2024-09-11T09:52:36.929 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:36.929 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:36.929 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "started", 2024-09-11T09:52:36.929 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:36.929 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.929 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:36.929 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.017158+0000", 2024-09-11T09:52:36.930 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:36.930 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.930 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.930 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:36.930 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.017158+0000", 2024-09-11T09:52:36.930 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:36.930 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.930 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.930 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:36.930 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.017158+0000", 2024-09-11T09:52:36.930 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.7000000000000004e-07 2024-09-11T09:52:36.930 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.930 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.930 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:36.931 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.018136+0000", 2024-09-11T09:52:36.931 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00097795 2024-09-11T09:52:36.931 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.931 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.931 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:36.931 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.018152+0000", 2024-09-11T09:52:36.931 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.5208000000000001e-05 2024-09-11T09:52:36.931 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.931 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.931 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:36.931 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.018164+0000", 2024-09-11T09:52:36.931 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.2157000000000001e-05 2024-09-11T09:52:36.931 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.931 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.931 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:36.932 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.018245+0000", 2024-09-11T09:52:36.932 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.1284000000000003e-05 2024-09-11T09:52:36.932 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.932 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.932 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:36.932 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.018257+0000", 2024-09-11T09:52:36.932 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.2160999999999999e-05 2024-09-11T09:52:36.932 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.932 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.932 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:36.932 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.019182+0000", 2024-09-11T09:52:36.932 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00092455900000000004 2024-09-11T09:52:36.932 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:36.932 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:36.933 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:36.933 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.933 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.933 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "MOSDPGPush(2.4 82/81 [PushOp(2:2167012f:::rbd_data.1250f5e25b80.000000000000000d:6, version: 79'175, data_included: [3145728~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(2:2167012f:::rbd_data.1250f5e25b80.000000000000000d:6@79'175, size: 4194304, copy_subset: [(0, 4194304)], clone_subset: {}, snapset: 6=[]:{6: [6, 5, 3]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 4194304, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3145728, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-11T09:52:36.933 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:52:09.020895+0000", 2024-09-11T09:52:36.933 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 27.857148056, 2024-09-11T09:52:36.933 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.002075234, 2024-09-11T09:52:36.933 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:36.933 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:36.933 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "started", 2024-09-11T09:52:36.933 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:36.933 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.933 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:36.933 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.020895+0000", 2024-09-11T09:52:36.934 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:36.934 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.934 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.934 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:36.934 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.020895+0000", 2024-09-11T09:52:36.934 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:36.934 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.934 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.934 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:36.934 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.020895+0000", 2024-09-11T09:52:36.934 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.0999999999999999e-07 2024-09-11T09:52:36.934 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.934 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.934 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:36.935 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.021855+0000", 2024-09-11T09:52:36.935 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00095982700000000001 2024-09-11T09:52:36.935 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.935 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.935 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:36.935 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.021870+0000", 2024-09-11T09:52:36.935 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.4609e-05 2024-09-11T09:52:36.935 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.935 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.935 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:36.935 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.021881+0000", 2024-09-11T09:52:36.935 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.083e-05 2024-09-11T09:52:36.935 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.935 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.935 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:36.936 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.021943+0000", 2024-09-11T09:52:36.936 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.2780999999999994e-05 2024-09-11T09:52:36.936 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.936 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.936 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:36.936 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.021952+0000", 2024-09-11T09:52:36.936 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.8629999999999997e-06 2024-09-11T09:52:36.936 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.936 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.936 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:36.936 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:09.022970+0000", 2024-09-11T09:52:36.936 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0010178139999999999 2024-09-11T09:52:36.936 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:36.936 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:36.936 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:36.937 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.937 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.937 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.6054.0:3 2.4 2:2906e6a7:::rbd_header.1250f5e25b80:head [call rbd.get_size in=11b,call,call rbd.get_features in=15b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e95)", 2024-09-11T09:52:36.937 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:52:26.497840+0000", 2024-09-11T09:52:36.937 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 10.380203335999999, 2024-09-11T09:52:36.937 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.421058402, 2024-09-11T09:52:36.937 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:36.937 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:36.937 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "started", 2024-09-11T09:52:36.937 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:52:36.937 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.6054", 2024-09-11T09:52:36.937 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/3379198229", 2024-09-11T09:52:36.937 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 3 2024-09-11T09:52:36.937 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.937 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:36.938 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.938 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:36.938 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:26.497840+0000", 2024-09-11T09:52:36.938 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:36.938 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.938 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.938 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:36.938 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:26.497840+0000", 2024-09-11T09:52:36.938 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:36.938 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.938 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.938 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:36.938 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:26.497841+0000", 2024-09-11T09:52:36.938 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.254e-06 2024-09-11T09:52:36.938 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.939 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.939 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:36.939 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:26.497846+0000", 2024-09-11T09:52:36.939 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.6710000000000004e-06 2024-09-11T09:52:36.939 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.939 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.939 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:36.939 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:26.497856+0000", 2024-09-11T09:52:36.939 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.3109999999999995e-06 2024-09-11T09:52:36.939 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.939 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.939 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:36.939 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:26.497863+0000", 2024-09-11T09:52:36.940 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.4919999999999996e-06 2024-09-11T09:52:36.940 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.940 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.940 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:36.940 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:26.497941+0000", 2024-09-11T09:52:36.940 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.7412999999999999e-05 2024-09-11T09:52:36.940 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.940 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.940 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for peered", 2024-09-11T09:52:36.940 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:26.497959+0000", 2024-09-11T09:52:36.940 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.8343e-05 2024-09-11T09:52:36.940 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.940 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.940 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:36.940 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:26.918496+0000", 2024-09-11T09:52:36.941 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.42053728299999998 2024-09-11T09:52:36.941 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.941 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.941 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:36.941 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:26.918642+0000", 2024-09-11T09:52:36.941 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000145422 2024-09-11T09:52:36.941 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.941 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.941 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:36.941 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:26.918898+0000", 2024-09-11T09:52:36.941 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00025621299999999998 2024-09-11T09:52:36.941 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:36.941 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:36.941 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:36.942 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.942 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.942 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.6072.0:26 2.4 2:2906e6a7:::rbd_header.1250f5e25b80:head [call rbd.snapshot_remove in=18b] snapc 0=[] ondisk+write+known_if_redirected+full_try+supports_pool_eio e97)", 2024-09-11T09:52:36.942 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:52:27.917078+0000", 2024-09-11T09:52:36.942 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 8.9609646460000008, 2024-09-11T09:52:36.942 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.010482035000000001, 2024-09-11T09:52:36.942 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:36.942 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:36.942 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:36.942 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:52:36.942 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.6072", 2024-09-11T09:52:36.942 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/61699540", 2024-09-11T09:52:36.942 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 26 2024-09-11T09:52:36.942 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.942 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:36.943 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.943 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:36.943 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:27.917078+0000", 2024-09-11T09:52:36.943 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:36.943 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.943 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.943 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:36.943 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:27.917078+0000", 2024-09-11T09:52:36.943 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:36.943 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.943 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.943 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:36.943 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:27.917080+0000", 2024-09-11T09:52:36.943 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.1720000000000001e-06 2024-09-11T09:52:36.943 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.943 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.944 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:36.944 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:27.917090+0000", 2024-09-11T09:52:36.944 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.0759999999999992e-06 2024-09-11T09:52:36.944 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.944 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.944 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:36.944 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:27.917110+0000", 2024-09-11T09:52:36.944 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.0231e-05 2024-09-11T09:52:36.944 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.944 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.944 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:36.944 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:27.917122+0000", 2024-09-11T09:52:36.944 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.2106000000000001e-05 2024-09-11T09:52:36.944 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.945 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.945 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:36.945 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:27.917198+0000", 2024-09-11T09:52:36.945 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.6259000000000003e-05 2024-09-11T09:52:36.945 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.945 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.945 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op must wait for map", 2024-09-11T09:52:36.945 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:27.917207+0000", 2024-09-11T09:52:36.945 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.4659999999999999e-06 2024-09-11T09:52:36.945 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.945 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.945 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:36.945 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:27.923132+0000", 2024-09-11T09:52:36.945 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0059250730000000003 2024-09-11T09:52:36.945 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.946 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.946 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:36.946 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:27.923193+0000", 2024-09-11T09:52:36.946 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.1445999999999994e-05 2024-09-11T09:52:36.946 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.946 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.946 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 1", 2024-09-11T09:52:36.946 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:27.923411+0000", 2024-09-11T09:52:36.946 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00021771799999999999 2024-09-11T09:52:36.946 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.946 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.946 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:36.946 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:27.925324+0000", 2024-09-11T09:52:36.946 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0019128579999999999 2024-09-11T09:52:36.947 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.947 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.947 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:52:36.947 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:27.927463+0000", 2024-09-11T09:52:36.947 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.002139564 2024-09-11T09:52:36.947 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.947 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.947 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:36.947 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:27.927518+0000", 2024-09-11T09:52:36.947 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.4591e-05 2024-09-11T09:52:36.947 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.947 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.947 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:36.947 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:27.927560+0000", 2024-09-11T09:52:36.948 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.2475000000000002e-05 2024-09-11T09:52:36.948 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:36.948 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:36.948 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:36.948 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.948 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.948 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_repop(osd.3.0:238 2.5 e97/80 2:b6dd9fc9:::rbd_data.1250f5e25b80.0000000000000009:6 v 97'50, mlcod=79'45)", 2024-09-11T09:52:36.948 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:52:27.918868+0000", 2024-09-11T09:52:36.948 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 8.9591747549999994, 2024-09-11T09:52:36.948 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0037081330000000002, 2024-09-11T09:52:36.948 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:36.948 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:36.948 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:36.948 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:36.949 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.949 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:36.949 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:27.918868+0000", 2024-09-11T09:52:36.949 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:36.949 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.949 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.949 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:36.949 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:27.918868+0000", 2024-09-11T09:52:36.949 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:36.949 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.949 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.949 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:36.949 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:27.918869+0000", 2024-09-11T09:52:36.949 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.9200000000000001e-07 2024-09-11T09:52:36.950 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.950 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.950 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:36.950 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:27.918873+0000", 2024-09-11T09:52:36.950 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.036e-06 2024-09-11T09:52:36.950 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.950 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.950 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:36.950 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:27.918880+0000", 2024-09-11T09:52:36.950 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.3590000000000002e-06 2024-09-11T09:52:36.950 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.950 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.950 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:36.950 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:27.918887+0000", 2024-09-11T09:52:36.950 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.9410000000000001e-06 2024-09-11T09:52:36.951 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.951 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.951 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:36.951 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:27.919121+0000", 2024-09-11T09:52:36.951 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000233935 2024-09-11T09:52:36.951 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.951 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.951 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:36.951 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:27.919135+0000", 2024-09-11T09:52:36.951 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.394e-05 2024-09-11T09:52:36.951 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.951 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.951 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:36.952 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:27.922468+0000", 2024-09-11T09:52:36.952 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.003333094 2024-09-11T09:52:36.952 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.952 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.952 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:36.952 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:27.922576+0000", 2024-09-11T09:52:36.952 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000108336 2024-09-11T09:52:36.952 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:36.952 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:36.952 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:36.952 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.952 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.952 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.6072.0:71 2.4 2:2906e6a7:::rbd_header.1250f5e25b80:head [call rbd.snapshot_remove in=18b] snapc 0=[] ondisk+write+known_if_redirected+full_try+supports_pool_eio e100)", 2024-09-11T09:52:36.952 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:52:30.936137+0000", 2024-09-11T09:52:36.953 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 5.9419061040000001, 2024-09-11T09:52:36.953 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.002867779, 2024-09-11T09:52:36.953 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:36.953 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:36.953 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:36.953 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:52:36.953 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.6072", 2024-09-11T09:52:36.953 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/61699540", 2024-09-11T09:52:36.953 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 71 2024-09-11T09:52:36.953 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.953 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:36.953 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.953 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:36.953 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:30.936137+0000", 2024-09-11T09:52:36.954 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:36.954 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.954 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.954 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:36.954 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:30.936137+0000", 2024-09-11T09:52:36.954 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:36.954 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.954 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.954 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:36.954 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:30.936139+0000", 2024-09-11T09:52:36.954 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 2.2029999999999999e-06 2024-09-11T09:52:36.954 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.954 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.954 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:36.954 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:30.936151+0000", 2024-09-11T09:52:36.955 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.1677e-05 2024-09-11T09:52:36.955 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.955 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.955 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:36.955 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:30.936170+0000", 2024-09-11T09:52:36.955 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.9193000000000001e-05 2024-09-11T09:52:36.955 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.955 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.955 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:36.955 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:30.936180+0000", 2024-09-11T09:52:36.955 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.8749999999999995e-06 2024-09-11T09:52:36.955 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.955 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.955 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:36.956 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:30.936246+0000", 2024-09-11T09:52:36.956 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.6398999999999999e-05 2024-09-11T09:52:36.956 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.956 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.956 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op must wait for map", 2024-09-11T09:52:36.956 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:30.936252+0000", 2024-09-11T09:52:36.956 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.6169999999999998e-06 2024-09-11T09:52:36.956 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.956 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.956 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:36.956 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:30.937633+0000", 2024-09-11T09:52:36.956 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0013812989999999999 2024-09-11T09:52:36.956 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.956 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.956 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:36.957 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:30.937701+0000", 2024-09-11T09:52:36.957 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 6.7895e-05 2024-09-11T09:52:36.957 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.957 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.957 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 1", 2024-09-11T09:52:36.957 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:30.937937+0000", 2024-09-11T09:52:36.957 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00023624299999999999 2024-09-11T09:52:36.957 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.957 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.957 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:52:36.957 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:30.938479+0000", 2024-09-11T09:52:36.957 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00054196300000000002 2024-09-11T09:52:36.957 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.957 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.958 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:36.958 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:30.938937+0000", 2024-09-11T09:52:36.958 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00045802100000000002 2024-09-11T09:52:36.958 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.958 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.958 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:36.958 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:30.938971+0000", 2024-09-11T09:52:36.958 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.3420999999999997e-05 2024-09-11T09:52:36.958 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.958 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.958 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:36.958 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:30.939005+0000", 2024-09-11T09:52:36.958 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.3973000000000002e-05 2024-09-11T09:52:36.958 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:36.958 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:36.959 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:36.959 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.959 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.959 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.5648.0:84 2.4 2:2367385a:::rbd_data.1250f5e25b80.0000000000000002:head [delete] snapc 7=[] ondisk+write+known_if_redirected+full_try+supports_pool_eio e100)", 2024-09-11T09:52:36.959 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:52:31.053615+0000", 2024-09-11T09:52:36.959 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 5.8244283340000003, 2024-09-11T09:52:36.959 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0023006210000000001, 2024-09-11T09:52:36.959 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:36.959 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:36.959 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:36.959 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:52:36.959 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.5648", 2024-09-11T09:52:36.959 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/1071190571", 2024-09-11T09:52:36.959 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 84 2024-09-11T09:52:36.959 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.960 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:36.960 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.960 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:36.960 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:31.053615+0000", 2024-09-11T09:52:36.960 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:36.960 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.960 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.960 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:36.960 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:31.053615+0000", 2024-09-11T09:52:36.960 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:36.960 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.960 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.960 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:36.960 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:31.053615+0000", 2024-09-11T09:52:36.960 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.9599999999999998e-07 2024-09-11T09:52:36.961 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.961 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.961 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:36.961 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:31.053619+0000", 2024-09-11T09:52:36.961 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.946e-06 2024-09-11T09:52:36.961 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.961 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.961 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:36.961 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:31.053629+0000", 2024-09-11T09:52:36.961 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.5980000000000003e-06 2024-09-11T09:52:36.961 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.961 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.961 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:36.961 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:31.053634+0000", 2024-09-11T09:52:36.961 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.0640000000000001e-06 2024-09-11T09:52:36.961 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.962 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.962 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:36.962 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:31.053671+0000", 2024-09-11T09:52:36.962 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 3.7426999999999999e-05 2024-09-11T09:52:36.962 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.962 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.962 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:36.962 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:31.053730+0000", 2024-09-11T09:52:36.962 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.8597000000000002e-05 2024-09-11T09:52:36.962 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.962 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.962 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 1", 2024-09-11T09:52:36.962 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:31.053846+0000", 2024-09-11T09:52:36.962 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.000115524 2024-09-11T09:52:36.962 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.963 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.963 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:52:36.963 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:31.054812+0000", 2024-09-11T09:52:36.963 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00096634300000000004 2024-09-11T09:52:36.963 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.963 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.963 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:36.963 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:31.055823+0000", 2024-09-11T09:52:36.963 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.001011444 2024-09-11T09:52:36.963 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.963 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.963 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:36.963 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:31.055873+0000", 2024-09-11T09:52:36.963 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.9740000000000001e-05 2024-09-11T09:52:36.964 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.964 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.964 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:36.964 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:31.055915+0000", 2024-09-11T09:52:36.964 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.2141999999999997e-05 2024-09-11T09:52:36.964 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:36.964 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:36.964 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:36.964 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.964 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.964 INFO:teuthology.orchestra.run.smithi096.stdout: "description": "osd_op(client.5648.0:89 2.4 2:239e915b:::rbd_data.1250f5e25b80.000000000000000c:head [delete] snapc 7=[] ondisk+write+known_if_redirected+full_try+supports_pool_eio e100)", 2024-09-11T09:52:36.964 INFO:teuthology.orchestra.run.smithi096.stdout: "initiated_at": "2024-09-11T09:52:31.053720+0000", 2024-09-11T09:52:36.964 INFO:teuthology.orchestra.run.smithi096.stdout: "age": 5.8243229689999998, 2024-09-11T09:52:36.964 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.002622194, 2024-09-11T09:52:36.964 INFO:teuthology.orchestra.run.smithi096.stdout: "continuous": false, 2024-09-11T09:52:36.964 INFO:teuthology.orchestra.run.smithi096.stdout: "type_data": { 2024-09-11T09:52:36.965 INFO:teuthology.orchestra.run.smithi096.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-11T09:52:36.965 INFO:teuthology.orchestra.run.smithi096.stdout: "client_info": { 2024-09-11T09:52:36.965 INFO:teuthology.orchestra.run.smithi096.stdout: "client": "client.5648", 2024-09-11T09:52:36.965 INFO:teuthology.orchestra.run.smithi096.stdout: "client_addr": "172.21.15.107:0/1071190571", 2024-09-11T09:52:36.965 INFO:teuthology.orchestra.run.smithi096.stdout: "tid": 89 2024-09-11T09:52:36.965 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.965 INFO:teuthology.orchestra.run.smithi096.stdout: "events": [ 2024-09-11T09:52:36.965 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.965 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "initiated", 2024-09-11T09:52:36.965 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:31.053720+0000", 2024-09-11T09:52:36.965 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:36.965 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.965 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.965 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "header_read", 2024-09-11T09:52:36.966 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:31.053720+0000", 2024-09-11T09:52:36.966 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0 2024-09-11T09:52:36.966 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.966 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.966 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "throttled", 2024-09-11T09:52:36.966 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:31.053721+0000", 2024-09-11T09:52:36.966 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 9.8299999999999995e-07 2024-09-11T09:52:36.966 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.966 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.966 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "all_read", 2024-09-11T09:52:36.966 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:31.053726+0000", 2024-09-11T09:52:36.966 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 4.8770000000000001e-06 2024-09-11T09:52:36.966 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.966 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.967 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "dispatched", 2024-09-11T09:52:36.967 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:31.053745+0000", 2024-09-11T09:52:36.967 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.9262999999999998e-05 2024-09-11T09:52:36.967 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.967 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.967 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "queued_for_pg", 2024-09-11T09:52:36.967 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:31.053752+0000", 2024-09-11T09:52:36.967 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.294e-06 2024-09-11T09:52:36.967 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.967 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.967 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "reached_pg", 2024-09-11T09:52:36.967 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:31.054261+0000", 2024-09-11T09:52:36.967 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.00050814800000000004 2024-09-11T09:52:36.968 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.968 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.968 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "started", 2024-09-11T09:52:36.968 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:31.054337+0000", 2024-09-11T09:52:36.968 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.682e-05 2024-09-11T09:52:36.968 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.968 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.968 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "waiting for subops from 1", 2024-09-11T09:52:36.968 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:31.054420+0000", 2024-09-11T09:52:36.968 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 8.2576000000000004e-05 2024-09-11T09:52:36.968 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.968 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.968 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "sub_op_commit_rec", 2024-09-11T09:52:36.968 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:31.056204+0000", 2024-09-11T09:52:36.968 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 0.0017841949999999999 2024-09-11T09:52:36.969 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.969 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.969 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "op_commit", 2024-09-11T09:52:36.969 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:31.056218+0000", 2024-09-11T09:52:36.969 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 1.3417999999999999e-05 2024-09-11T09:52:36.969 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.969 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.969 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "commit_sent", 2024-09-11T09:52:36.969 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:31.056290+0000", 2024-09-11T09:52:36.969 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 7.2675000000000005e-05 2024-09-11T09:52:36.969 INFO:teuthology.orchestra.run.smithi096.stdout: }, 2024-09-11T09:52:36.969 INFO:teuthology.orchestra.run.smithi096.stdout: { 2024-09-11T09:52:36.969 INFO:teuthology.orchestra.run.smithi096.stdout: "event": "done", 2024-09-11T09:52:36.969 INFO:teuthology.orchestra.run.smithi096.stdout: "time": "2024-09-11T09:52:31.056342+0000", 2024-09-11T09:52:36.970 INFO:teuthology.orchestra.run.smithi096.stdout: "duration": 5.1944999999999998e-05 2024-09-11T09:52:36.970 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:36.970 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:36.970 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:36.970 INFO:teuthology.orchestra.run.smithi096.stdout: } 2024-09-11T09:52:36.970 INFO:teuthology.orchestra.run.smithi096.stdout: ] 2024-09-11T09:52:36.970 INFO:teuthology.orchestra.run.smithi096.stdout:} 2024-09-11T09:52:36.970 INFO:tasks.thrashosds.thrasher:inning osd 2024-09-11T09:52:36.970 INFO:tasks.thrashosds.thrasher:Adding osd 0 2024-09-11T09:52:36.970 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd in 0 2024-09-11T09:52:36.972 INFO:teuthology.orchestra.run.smithi017.stderr:noscrub is unset 2024-09-11T09:52:36.973 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-11T09:52:37.327 INFO:tasks.ceph.mon.a.smithi017.stderr:2024-09-11T09:52:37.328+0000 7ff44db6e640 -1 mon.a@0(leader).osd e106 definitely_dead 0 2024-09-11T09:52:37.959 INFO:teuthology.orchestra.run.smithi017.stderr:nodeep-scrub is unset 2024-09-11T09:52:37.962 INFO:teuthology.orchestra.run.smithi017.stderr:marked in osd.0. 2024-09-11T09:52:37.984 INFO:tasks.thrashosds.thrasher:Added osd 0 2024-09-11T09:52:37.984 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=600 2024-09-11T09:52:38.147 INFO:teuthology.orchestra.run.smithi017.stdout:mon.a: {} 2024-09-11T09:52:38.147 INFO:teuthology.orchestra.run.smithi017.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-11T09:52:38.157 INFO:teuthology.orchestra.run.smithi017.stdout:mon.b: {} 2024-09-11T09:52:38.157 INFO:teuthology.orchestra.run.smithi017.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-11T09:52:38.166 INFO:teuthology.orchestra.run.smithi017.stdout:mon.c: {} 2024-09-11T09:52:38.166 INFO:teuthology.orchestra.run.smithi017.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-11T09:52:38.177 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-09-11T09:52:38.962 INFO:teuthology.orchestra.run.smithi017.stderr:reweighted osd.0 to 1 (10000) 2024-09-11T09:52:38.980 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 0 1 2024-09-11T09:52:39.958 INFO:teuthology.orchestra.run.smithi017.stderr:set osd.0 primary-affinity to 1 (8655362) 2024-09-11T09:52:39.978 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-09-11T09:52:40.971 INFO:teuthology.orchestra.run.smithi017.stderr:reweighted osd.1 to 1 (10000) 2024-09-11T09:52:40.989 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 1 1 2024-09-11T09:52:41.975 INFO:teuthology.orchestra.run.smithi017.stderr:set osd.1 primary-affinity to 1 (8655362) 2024-09-11T09:52:41.993 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-09-11T09:52:42.981 INFO:teuthology.orchestra.run.smithi017.stderr:reweighted osd.2 to 1 (10000) 2024-09-11T09:52:42.998 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 2 1 2024-09-11T09:52:43.979 INFO:teuthology.orchestra.run.smithi017.stderr:set osd.2 primary-affinity to 1 (8655362) 2024-09-11T09:52:43.999 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-09-11T09:52:44.987 INFO:teuthology.orchestra.run.smithi017.stderr:reweighted osd.3 to 1 (10000) 2024-09-11T09:52:45.006 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 3 1 2024-09-11T09:52:45.988 INFO:teuthology.orchestra.run.smithi017.stderr:set osd.3 primary-affinity to 1 (8655362) 2024-09-11T09:52:46.005 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 5 1 2024-09-11T09:52:46.990 INFO:teuthology.orchestra.run.smithi017.stderr:reweighted osd.5 to 1 (10000) 2024-09-11T09:52:47.009 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 5 1 2024-09-11T09:52:48.000 INFO:teuthology.orchestra.run.smithi017.stderr:set osd.5 primary-affinity to 1 (8655362) 2024-09-11T09:52:48.017 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 4 1 2024-09-11T09:52:49.005 INFO:teuthology.orchestra.run.smithi017.stderr:reweighted osd.4 to 1 (10000) 2024-09-11T09:52:49.022 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 4 1 2024-09-11T09:52:50.012 INFO:teuthology.orchestra.run.smithi017.stderr:set osd.4 primary-affinity to 1 (8655362) 2024-09-11T09:52:50.028 INFO:tasks.thrashosds.thrasher:joining the do_sighup greenlet 2024-09-11T09:52:50.028 INFO:tasks.thrashosds.thrasher:joining the do_optrack_toggle greenlet 2024-09-11T09:52:50.028 INFO:tasks.thrashosds.thrasher:joining the do_dump_ops greenlet 2024-09-11T09:52:50.028 INFO:tasks.thrashosds.thrasher:joining the do_noscrub_toggle greenlet 2024-09-11T09:52:50.028 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-09-11T09:52:50.028 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-11T09:52:50.325 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-11T09:52:50.325 INFO:teuthology.orchestra.run.smithi017.stdout:{"epoch":119,"fsid":"a2f1daec-d3b0-4bf5-82c8-72ecbb93d4e4","created":"2024-09-11T09:50:00.165804+0000","modified":"2024-09-11T09:52:50.010309+0000","last_up_change":"2024-09-11T09:51:44.764990+0000","last_in_change":"2024-09-11T09:52:37.955638+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-11T09:50:10.457184+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-11T09:50:16.993202+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"101","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":7,"snap_epoch":101,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4829b7a1-d607-4a56-b6d2-55036cb8acf8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6808","nonce":4154449929},{"type":"v1","addr":"172.21.15.17:6809","nonce":4154449929}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6810","nonce":4154449929},{"type":"v1","addr":"172.21.15.17:6811","nonce":4154449929}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6814","nonce":4154449929},{"type":"v1","addr":"172.21.15.17:6815","nonce":4154449929}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6812","nonce":4154449929},{"type":"v1","addr":"172.21.15.17:6813","nonce":4154449929}]},"public_addr":"172.21.15.17:6809/4154449929","cluster_addr":"172.21.15.17:6811/4154449929","heartbeat_back_addr":"172.21.15.17:6815/4154449929","heartbeat_front_addr":"172.21.15.17:6813/4154449929","state":["exists","up"]},{"osd":1,"uuid":"ab1d5df0-3c0e-44cd-88a0-96b07964148b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6800","nonce":4139511644},{"type":"v1","addr":"172.21.15.17:6801","nonce":4139511644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6802","nonce":4139511644},{"type":"v1","addr":"172.21.15.17:6803","nonce":4139511644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6806","nonce":4139511644},{"type":"v1","addr":"172.21.15.17:6807","nonce":4139511644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6804","nonce":4139511644},{"type":"v1","addr":"172.21.15.17:6805","nonce":4139511644}]},"public_addr":"172.21.15.17:6801/4139511644","cluster_addr":"172.21.15.17:6803/4139511644","heartbeat_back_addr":"172.21.15.17:6807/4139511644","heartbeat_front_addr":"172.21.15.17:6805/4139511644","state":["exists","up"]},{"osd":2,"uuid":"4c8e50db-b49c-4d44-a913-47b5b30f8699","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6816","nonce":1151282506},{"type":"v1","addr":"172.21.15.17:6817","nonce":1151282506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6818","nonce":1151282506},{"type":"v1","addr":"172.21.15.17:6819","nonce":1151282506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6822","nonce":1151282506},{"type":"v1","addr":"172.21.15.17:6823","nonce":1151282506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6820","nonce":1151282506},{"type":"v1","addr":"172.21.15.17:6821","nonce":1151282506}]},"public_addr":"172.21.15.17:6817/1151282506","cluster_addr":"172.21.15.17:6819/1151282506","heartbeat_back_addr":"172.21.15.17:6823/1151282506","heartbeat_front_addr":"172.21.15.17:6821/1151282506","state":["exists","up"]},{"osd":3,"uuid":"37abc55b-a1eb-49d5-b1ff-b507ee2cb5a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":332854807},{"type":"v1","addr":"172.21.15.96:6817","nonce":332854807}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6818","nonce":332854807},{"type":"v1","addr":"172.21.15.96:6819","nonce":332854807}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":332854807},{"type":"v1","addr":"172.21.15.96:6823","nonce":332854807}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":332854807},{"type":"v1","addr":"172.21.15.96:6821","nonce":332854807}]},"public_addr":"172.21.15.96:6817/332854807","cluster_addr":"172.21.15.96:6819/332854807","heartbeat_back_addr":"172.21.15.96:6823/332854807","heartbeat_front_addr":"172.21.15.96:6821/332854807","state":["exists","up"]},{"osd":4,"uuid":"067d9586-4f5a-4d17-96c5-86a7b8bcd114","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":11,"last_clean_end":54,"up_from":66,"up_thru":101,"down_at":55,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":3374675329},{"type":"v1","addr":"172.21.15.96:6801","nonce":3374675329}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":3374675329},{"type":"v1","addr":"172.21.15.96:6803","nonce":3374675329}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":3374675329},{"type":"v1","addr":"172.21.15.96:6807","nonce":3374675329}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":3374675329},{"type":"v1","addr":"172.21.15.96:6805","nonce":3374675329}]},"public_addr":"172.21.15.96:6801/3374675329","cluster_addr":"172.21.15.96:6803/3374675329","heartbeat_back_addr":"172.21.15.96:6807/3374675329","heartbeat_front_addr":"172.21.15.96:6805/3374675329","state":["exists","up"]},{"osd":5,"uuid":"8ff7a724-0a3a-4aa2-baf1-fd52f88d476c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":509935370},{"type":"v1","addr":"172.21.15.96:6809","nonce":509935370}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6810","nonce":509935370},{"type":"v1","addr":"172.21.15.96:6811","nonce":509935370}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6814","nonce":509935370},{"type":"v1","addr":"172.21.15.96:6815","nonce":509935370}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":509935370},{"type":"v1","addr":"172.21.15.96:6813","nonce":509935370}]},"public_addr":"172.21.15.96:6809/509935370","cluster_addr":"172.21.15.96:6811/509935370","heartbeat_back_addr":"172.21.15.96:6815/509935370","heartbeat_front_addr":"172.21.15.96:6813/509935370","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T09:50:08.513270+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T09:50:07.522863+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T09:50:08.134364+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T09:50:08.622366+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T09:50:08.156142+0000","dead_epoch":55},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T09:50:08.108116+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-11T09:52:50.338 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-09-11T09:52:50.338 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-11T09:52:50.630 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-11T09:52:50.630 INFO:teuthology.orchestra.run.smithi017.stdout:{"epoch":119,"fsid":"a2f1daec-d3b0-4bf5-82c8-72ecbb93d4e4","created":"2024-09-11T09:50:00.165804+0000","modified":"2024-09-11T09:52:50.010309+0000","last_up_change":"2024-09-11T09:51:44.764990+0000","last_in_change":"2024-09-11T09:52:37.955638+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-11T09:50:10.457184+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-11T09:50:16.993202+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"101","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":7,"snap_epoch":101,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4829b7a1-d607-4a56-b6d2-55036cb8acf8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6808","nonce":4154449929},{"type":"v1","addr":"172.21.15.17:6809","nonce":4154449929}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6810","nonce":4154449929},{"type":"v1","addr":"172.21.15.17:6811","nonce":4154449929}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6814","nonce":4154449929},{"type":"v1","addr":"172.21.15.17:6815","nonce":4154449929}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6812","nonce":4154449929},{"type":"v1","addr":"172.21.15.17:6813","nonce":4154449929}]},"public_addr":"172.21.15.17:6809/4154449929","cluster_addr":"172.21.15.17:6811/4154449929","heartbeat_back_addr":"172.21.15.17:6815/4154449929","heartbeat_front_addr":"172.21.15.17:6813/4154449929","state":["exists","up"]},{"osd":1,"uuid":"ab1d5df0-3c0e-44cd-88a0-96b07964148b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6800","nonce":4139511644},{"type":"v1","addr":"172.21.15.17:6801","nonce":4139511644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6802","nonce":4139511644},{"type":"v1","addr":"172.21.15.17:6803","nonce":4139511644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6806","nonce":4139511644},{"type":"v1","addr":"172.21.15.17:6807","nonce":4139511644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6804","nonce":4139511644},{"type":"v1","addr":"172.21.15.17:6805","nonce":4139511644}]},"public_addr":"172.21.15.17:6801/4139511644","cluster_addr":"172.21.15.17:6803/4139511644","heartbeat_back_addr":"172.21.15.17:6807/4139511644","heartbeat_front_addr":"172.21.15.17:6805/4139511644","state":["exists","up"]},{"osd":2,"uuid":"4c8e50db-b49c-4d44-a913-47b5b30f8699","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6816","nonce":1151282506},{"type":"v1","addr":"172.21.15.17:6817","nonce":1151282506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6818","nonce":1151282506},{"type":"v1","addr":"172.21.15.17:6819","nonce":1151282506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6822","nonce":1151282506},{"type":"v1","addr":"172.21.15.17:6823","nonce":1151282506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6820","nonce":1151282506},{"type":"v1","addr":"172.21.15.17:6821","nonce":1151282506}]},"public_addr":"172.21.15.17:6817/1151282506","cluster_addr":"172.21.15.17:6819/1151282506","heartbeat_back_addr":"172.21.15.17:6823/1151282506","heartbeat_front_addr":"172.21.15.17:6821/1151282506","state":["exists","up"]},{"osd":3,"uuid":"37abc55b-a1eb-49d5-b1ff-b507ee2cb5a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":332854807},{"type":"v1","addr":"172.21.15.96:6817","nonce":332854807}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6818","nonce":332854807},{"type":"v1","addr":"172.21.15.96:6819","nonce":332854807}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":332854807},{"type":"v1","addr":"172.21.15.96:6823","nonce":332854807}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":332854807},{"type":"v1","addr":"172.21.15.96:6821","nonce":332854807}]},"public_addr":"172.21.15.96:6817/332854807","cluster_addr":"172.21.15.96:6819/332854807","heartbeat_back_addr":"172.21.15.96:6823/332854807","heartbeat_front_addr":"172.21.15.96:6821/332854807","state":["exists","up"]},{"osd":4,"uuid":"067d9586-4f5a-4d17-96c5-86a7b8bcd114","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":11,"last_clean_end":54,"up_from":66,"up_thru":101,"down_at":55,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":3374675329},{"type":"v1","addr":"172.21.15.96:6801","nonce":3374675329}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":3374675329},{"type":"v1","addr":"172.21.15.96:6803","nonce":3374675329}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":3374675329},{"type":"v1","addr":"172.21.15.96:6807","nonce":3374675329}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":3374675329},{"type":"v1","addr":"172.21.15.96:6805","nonce":3374675329}]},"public_addr":"172.21.15.96:6801/3374675329","cluster_addr":"172.21.15.96:6803/3374675329","heartbeat_back_addr":"172.21.15.96:6807/3374675329","heartbeat_front_addr":"172.21.15.96:6805/3374675329","state":["exists","up"]},{"osd":5,"uuid":"8ff7a724-0a3a-4aa2-baf1-fd52f88d476c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":509935370},{"type":"v1","addr":"172.21.15.96:6809","nonce":509935370}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6810","nonce":509935370},{"type":"v1","addr":"172.21.15.96:6811","nonce":509935370}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6814","nonce":509935370},{"type":"v1","addr":"172.21.15.96:6815","nonce":509935370}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":509935370},{"type":"v1","addr":"172.21.15.96:6813","nonce":509935370}]},"public_addr":"172.21.15.96:6809/509935370","cluster_addr":"172.21.15.96:6811/509935370","heartbeat_back_addr":"172.21.15.96:6815/509935370","heartbeat_front_addr":"172.21.15.96:6813/509935370","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T09:50:08.513270+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T09:50:07.522863+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T09:50:08.134364+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T09:50:08.622366+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T09:50:08.156142+0000","dead_epoch":55},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T09:50:08.108116+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-11T09:52:50.644 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-09-11T09:52:50.645 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-09-11T09:52:50.645 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-09-11T09:52:50.645 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-09-11T09:52:50.646 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-09-11T09:52:50.646 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-09-11T09:52:50.824 INFO:teuthology.orchestra.run.smithi017.stdout:47244640292 2024-09-11T09:52:50.824 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-11T09:52:50.828 INFO:teuthology.orchestra.run.smithi017.stdout:47244640293 2024-09-11T09:52:50.828 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-09-11T09:52:50.856 INFO:teuthology.orchestra.run.smithi017.stdout:51539607587 2024-09-11T09:52:50.857 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-11T09:52:50.863 INFO:teuthology.orchestra.run.smithi017.stdout:283467841551 2024-09-11T09:52:50.863 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-09-11T09:52:50.869 INFO:teuthology.orchestra.run.smithi017.stdout:47244640292 2024-09-11T09:52:50.869 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-11T09:52:50.873 INFO:teuthology.orchestra.run.smithi017.stdout:51539607588 2024-09-11T09:52:50.874 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-11T09:52:51.201 INFO:teuthology.orchestra.run.smithi017.stdout:47244640291 2024-09-11T09:52:51.205 INFO:teuthology.orchestra.run.smithi017.stdout:47244640290 2024-09-11T09:52:51.216 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640293 got 47244640291 for osd.5 2024-09-11T09:52:51.219 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640292 got 47244640290 for osd.2 2024-09-11T09:52:51.319 INFO:teuthology.orchestra.run.smithi017.stdout:51539607586 2024-09-11T09:52:51.327 INFO:teuthology.orchestra.run.smithi017.stdout:47244640290 2024-09-11T09:52:51.333 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607587 got 51539607586 for osd.0 2024-09-11T09:52:51.343 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640292 got 47244640290 for osd.1 2024-09-11T09:52:51.351 INFO:teuthology.orchestra.run.smithi017.stdout:283467841550 2024-09-11T09:52:51.364 INFO:tasks.ceph.ceph_manager.ceph:need seq 283467841551 got 283467841550 for osd.4 2024-09-11T09:52:51.364 INFO:teuthology.orchestra.run.smithi017.stdout:51539607587 2024-09-11T09:52:51.377 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607588 got 51539607587 for osd.3 2024-09-11T09:52:52.217 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-09-11T09:52:52.220 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-11T09:52:52.334 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-11T09:52:52.344 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-11T09:52:52.365 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-09-11T09:52:52.378 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-11T09:52:52.597 INFO:teuthology.orchestra.run.smithi017.stdout:47244640290 2024-09-11T09:52:52.597 INFO:teuthology.orchestra.run.smithi017.stdout:47244640291 2024-09-11T09:52:52.611 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640292 got 47244640290 for osd.2 2024-09-11T09:52:52.619 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640293 got 47244640291 for osd.5 2024-09-11T09:52:52.728 INFO:teuthology.orchestra.run.smithi017.stdout:283467841550 2024-09-11T09:52:52.739 INFO:teuthology.orchestra.run.smithi017.stdout:51539607586 2024-09-11T09:52:52.742 INFO:tasks.ceph.ceph_manager.ceph:need seq 283467841551 got 283467841550 for osd.4 2024-09-11T09:52:52.752 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607587 got 51539607586 for osd.0 2024-09-11T09:52:52.821 INFO:teuthology.orchestra.run.smithi017.stdout:47244640290 2024-09-11T09:52:52.824 INFO:teuthology.orchestra.run.smithi017.stdout:51539607587 2024-09-11T09:52:52.838 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640292 got 47244640290 for osd.1 2024-09-11T09:52:52.840 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607588 got 51539607587 for osd.3 2024-09-11T09:52:53.613 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-11T09:52:53.641 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-09-11T09:52:53.742 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-09-11T09:52:53.753 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-11T09:52:53.839 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-11T09:52:53.841 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-11T09:52:54.056 INFO:teuthology.orchestra.run.smithi017.stdout:47244640292 2024-09-11T09:52:54.057 INFO:teuthology.orchestra.run.smithi017.stdout:47244640293 2024-09-11T09:52:54.076 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640292 got 47244640292 for osd.2 2024-09-11T09:52:54.076 DEBUG:teuthology.parallel:result is None 2024-09-11T09:52:54.078 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640293 got 47244640293 for osd.5 2024-09-11T09:52:54.078 DEBUG:teuthology.parallel:result is None 2024-09-11T09:52:54.124 INFO:teuthology.orchestra.run.smithi017.stdout:283467841551 2024-09-11T09:52:54.137 INFO:tasks.ceph.ceph_manager.ceph:need seq 283467841551 got 283467841551 for osd.4 2024-09-11T09:52:54.138 DEBUG:teuthology.parallel:result is None 2024-09-11T09:52:54.150 INFO:teuthology.orchestra.run.smithi017.stdout:51539607588 2024-09-11T09:52:54.163 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607587 got 51539607588 for osd.0 2024-09-11T09:52:54.164 DEBUG:teuthology.parallel:result is None 2024-09-11T09:52:54.224 INFO:teuthology.orchestra.run.smithi017.stdout:47244640292 2024-09-11T09:52:54.237 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640292 got 47244640292 for osd.1 2024-09-11T09:52:54.238 DEBUG:teuthology.parallel:result is None 2024-09-11T09:52:54.239 INFO:teuthology.orchestra.run.smithi017.stdout:51539607589 2024-09-11T09:52:54.252 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607588 got 51539607589 for osd.3 2024-09-11T09:52:54.252 DEBUG:teuthology.parallel:result is None 2024-09-11T09:52:54.253 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-09-11T09:52:54.253 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T09:52:54.581 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-11T09:52:54.582 INFO:teuthology.orchestra.run.smithi017.stderr:dumped all 2024-09-11T09:52:54.593 INFO:teuthology.orchestra.run.smithi017.stdout:{"pg_ready":true,"pg_map":{"version":196,"stamp":"2024-09-11T09:52:54.443268+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":7668,"num_read_kb":171458,"num_write":1172,"num_write_kb":275599,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":113,"num_bytes_recovered":71057650,"num_keys_recovered":22,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1267,"ondisk_log_size":1267,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":19,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":204964,"kb_used_data":8412,"kb_used_omap":12,"kb_used_meta":196403,"kb_avail":566026076,"statfs":{"total":579820584960,"available":579610701824,"internally_reserved":0,"allocated":8613888,"data_stored":4472698,"data_compressed":66628,"data_compressed_allocated":90112,"data_compressed_original":1163264,"omap_allocated":12403,"internal_metadata":201117581},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.440981"},"pg_stats":[{"pgid":"1.0","version":"13'117","reported_seq":421,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-11T09:52:50.186293+0000","last_change":"2024-09-11T09:52:41.980919+0000","last_active":"2024-09-11T09:52:50.186293+0000","last_peered":"2024-09-11T09:52:50.186293+0000","last_clean":"2024-09-11T09:52:50.186293+0000","last_became_active":"2024-09-11T09:52:41.980573+0000","last_became_peered":"2024-09-11T09:52:41.980573+0000","last_unstale":"2024-09-11T09:52:50.186293+0000","last_undegraded":"2024-09-11T09:52:50.186293+0000","last_fullsized":"2024-09-11T09:52:50.186293+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:50:10.457262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:50:10.457262+0000","last_clean_scrub_stamp":"2024-09-11T09:50:10.457262+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:59:08.639668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"103'48","reported_seq":392,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-11T09:52:50.014080+0000","last_change":"2024-09-11T09:52:44.000267+0000","last_active":"2024-09-11T09:52:50.014080+0000","last_peered":"2024-09-11T09:52:50.014080+0000","last_clean":"2024-09-11T09:52:50.014080+0000","last_became_active":"2024-09-11T09:52:43.990093+0000","last_became_peered":"2024-09-11T09:52:43.990093+0000","last_unstale":"2024-09-11T09:52:50.014080+0000","last_undegraded":"2024-09-11T09:52:50.014080+0000","last_fullsized":"2024-09-11T09:52:50.014080+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_clean_scrub_stamp":"2024-09-11T09:50:17.477890+0000","objects_scrubbed":0,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:57:25.381839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000328833,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":53,"num_read_kb":12312,"num_write":40,"num_write_kb":33291,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":20971520,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"103'18","reported_seq":430,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-11T09:52:50.186272+0000","last_change":"2024-09-11T09:52:31.939410+0000","last_active":"2024-09-11T09:52:50.186272+0000","last_peered":"2024-09-11T09:52:50.186272+0000","last_clean":"2024-09-11T09:52:50.186272+0000","last_became_active":"2024-09-11T09:50:18.493210+0000","last_became_peered":"2024-09-11T09:50:18.493210+0000","last_unstale":"2024-09-11T09:52:50.186272+0000","last_undegraded":"2024-09-11T09:52:50.186272+0000","last_fullsized":"2024-09-11T09:52:50.186272+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_clean_scrub_stamp":"2024-09-11T09:50:17.477890+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:47:27.009441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043766200000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":174,"num_read_kb":2862,"num_write":16,"num_write_kb":12294,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"103'304","reported_seq":2240,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-11T09:52:50.013413+0000","last_change":"2024-09-11T09:52:46.995900+0000","last_active":"2024-09-11T09:52:50.013413+0000","last_peered":"2024-09-11T09:52:50.013413+0000","last_clean":"2024-09-11T09:52:50.013413+0000","last_became_active":"2024-09-11T09:52:46.995086+0000","last_became_peered":"2024-09-11T09:52:46.995086+0000","last_unstale":"2024-09-11T09:52:50.013413+0000","last_undegraded":"2024-09-11T09:52:50.013413+0000","last_fullsized":"2024-09-11T09:52:50.013413+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_clean_scrub_stamp":"2024-09-11T09:50:17.477890+0000","objects_scrubbed":0,"log_size":304,"log_dups_size":0,"ondisk_log_size":304,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:18:59.319613+0000","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.0022869520000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2124,"num_read_kb":45623,"num_write":237,"num_write_kb":52658,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":51,"num_bytes_recovered":8388608,"num_keys_recovered":8,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"103'35","reported_seq":484,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-11T09:52:50.014091+0000","last_change":"2024-09-11T09:52:31.938934+0000","last_active":"2024-09-11T09:52:50.014091+0000","last_peered":"2024-09-11T09:52:50.014091+0000","last_clean":"2024-09-11T09:52:50.014091+0000","last_became_active":"2024-09-11T09:50:18.493165+0000","last_became_peered":"2024-09-11T09:50:18.493165+0000","last_unstale":"2024-09-11T09:52:50.014091+0000","last_undegraded":"2024-09-11T09:52:50.014091+0000","last_fullsized":"2024-09-11T09:52:50.014091+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_clean_scrub_stamp":"2024-09-11T09:50:17.477890+0000","objects_scrubbed":0,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T21:11:40.038981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000392338,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":202,"num_read_kb":8394,"num_write":27,"num_write_kb":20483,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"103'314","reported_seq":1828,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-11T09:52:50.013370+0000","last_change":"2024-09-11T09:52:42.985519+0000","last_active":"2024-09-11T09:52:50.013370+0000","last_peered":"2024-09-11T09:52:50.013370+0000","last_clean":"2024-09-11T09:52:50.013370+0000","last_became_active":"2024-09-11T09:52:42.985012+0000","last_became_peered":"2024-09-11T09:52:42.985012+0000","last_unstale":"2024-09-11T09:52:50.013370+0000","last_undegraded":"2024-09-11T09:52:50.013370+0000","last_fullsized":"2024-09-11T09:52:50.013370+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_clean_scrub_stamp":"2024-09-11T09:50:17.477890+0000","objects_scrubbed":0,"log_size":314,"log_dups_size":0,"ondisk_log_size":314,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:35:33.948046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026174,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3711,"num_read_kb":35790,"num_write":188,"num_write_kb":54320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":22031360,"num_keys_recovered":14,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"103'62","reported_seq":422,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-11T09:52:50.186313+0000","last_change":"2024-09-11T09:52:38.977632+0000","last_active":"2024-09-11T09:52:50.186313+0000","last_peered":"2024-09-11T09:52:50.186313+0000","last_clean":"2024-09-11T09:52:50.186313+0000","last_became_active":"2024-09-11T09:52:38.970665+0000","last_became_peered":"2024-09-11T09:52:38.970665+0000","last_unstale":"2024-09-11T09:52:50.186313+0000","last_undegraded":"2024-09-11T09:52:50.186313+0000","last_fullsized":"2024-09-11T09:52:50.186313+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_clean_scrub_stamp":"2024-09-11T09:50:17.477890+0000","objects_scrubbed":0,"log_size":62,"log_dups_size":0,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T14:28:24.105715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000633111,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":73,"num_read_kb":24582,"num_write":46,"num_write_kb":29188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":13110272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"103'157","reported_seq":609,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-11T09:52:50.013311+0000","last_change":"2024-09-11T09:52:42.983192+0000","last_active":"2024-09-11T09:52:50.013311+0000","last_peered":"2024-09-11T09:52:50.013311+0000","last_clean":"2024-09-11T09:52:50.013311+0000","last_became_active":"2024-09-11T09:52:42.982623+0000","last_became_peered":"2024-09-11T09:52:42.982623+0000","last_unstale":"2024-09-11T09:52:50.013311+0000","last_undegraded":"2024-09-11T09:52:50.013311+0000","last_fullsized":"2024-09-11T09:52:50.013311+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_clean_scrub_stamp":"2024-09-11T09:50:17.477890+0000","objects_scrubbed":0,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T17:46:27.402621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040558,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":244,"num_read_kb":41018,"num_write":121,"num_write_kb":66581,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"103'212","reported_seq":477,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-11T09:52:50.186301+0000","last_change":"2024-09-11T09:52:31.939671+0000","last_active":"2024-09-11T09:52:50.186301+0000","last_peered":"2024-09-11T09:52:50.186301+0000","last_clean":"2024-09-11T09:52:50.186301+0000","last_became_active":"2024-09-11T09:52:04.840466+0000","last_became_peered":"2024-09-11T09:52:04.840466+0000","last_unstale":"2024-09-11T09:52:50.186301+0000","last_undegraded":"2024-09-11T09:52:50.186301+0000","last_fullsized":"2024-09-11T09:52:50.186301+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_clean_scrub_stamp":"2024-09-11T09:50:17.477890+0000","objects_scrubbed":0,"log_size":212,"log_dups_size":0,"ondisk_log_size":212,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:52:38.106407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045445899999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":971,"num_read_kb":777,"num_write":338,"num_write_kb":4368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":4194418,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7552,"num_read_kb":171358,"num_write":1013,"num_write_kb":273183,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":105,"num_bytes_recovered":68696178,"num_keys_recovered":22,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1150,"ondisk_log_size":1150,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1771104,"data_compressed":100760,"data_compressed_allocated":135168,"data_compressed_original":1753088,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":12,"seq":51539607589,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32644,"kb_used_data":1448,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339196,"statfs":{"total":96636764160,"available":96603336704,"internally_reserved":0,"allocated":1482752,"data_stored":1139785,"data_compressed":32496,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":12,"seq":51539607588,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36784,"kb_used_data":1432,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335056,"statfs":{"total":96636764160,"available":96599097344,"internally_reserved":0,"allocated":1466368,"data_stored":1139785,"data_compressed":34132,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":11,"seq":47244640293,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32576,"kb_used_data":1388,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339264,"statfs":{"total":96636764160,"available":96603406336,"internally_reserved":0,"allocated":1421312,"data_stored":549436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":66,"seq":283467841552,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29500,"kb_used_data":1384,"kb_used_omap":4,"kb_used_meta":28091,"kb_avail":94342340,"statfs":{"total":96636764160,"available":96606556160,"internally_reserved":0,"allocated":1417216,"data_stored":549417,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":28765872},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640292,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36724,"kb_used_data":1372,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335116,"statfs":{"total":96636764160,"available":96599158784,"internally_reserved":0,"allocated":1404928,"data_stored":544839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640292,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36736,"kb_used_data":1388,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335104,"statfs":{"total":96636764160,"available":96599146496,"internally_reserved":0,"allocated":1421312,"data_stored":549436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":34132,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":34132,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32496,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T09:52:54.594 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T09:52:54.880 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-11T09:52:54.880 INFO:teuthology.orchestra.run.smithi017.stderr:dumped all 2024-09-11T09:52:54.893 INFO:teuthology.orchestra.run.smithi017.stdout:{"pg_ready":true,"pg_map":{"version":196,"stamp":"2024-09-11T09:52:54.443268+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":7668,"num_read_kb":171458,"num_write":1172,"num_write_kb":275599,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":113,"num_bytes_recovered":71057650,"num_keys_recovered":22,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1267,"ondisk_log_size":1267,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":19,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":204964,"kb_used_data":8412,"kb_used_omap":12,"kb_used_meta":196403,"kb_avail":566026076,"statfs":{"total":579820584960,"available":579610701824,"internally_reserved":0,"allocated":8613888,"data_stored":4472698,"data_compressed":66628,"data_compressed_allocated":90112,"data_compressed_original":1163264,"omap_allocated":12403,"internal_metadata":201117581},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.440981"},"pg_stats":[{"pgid":"1.0","version":"13'117","reported_seq":421,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-11T09:52:50.186293+0000","last_change":"2024-09-11T09:52:41.980919+0000","last_active":"2024-09-11T09:52:50.186293+0000","last_peered":"2024-09-11T09:52:50.186293+0000","last_clean":"2024-09-11T09:52:50.186293+0000","last_became_active":"2024-09-11T09:52:41.980573+0000","last_became_peered":"2024-09-11T09:52:41.980573+0000","last_unstale":"2024-09-11T09:52:50.186293+0000","last_undegraded":"2024-09-11T09:52:50.186293+0000","last_fullsized":"2024-09-11T09:52:50.186293+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:50:10.457262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:50:10.457262+0000","last_clean_scrub_stamp":"2024-09-11T09:50:10.457262+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:59:08.639668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"103'48","reported_seq":392,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-11T09:52:50.014080+0000","last_change":"2024-09-11T09:52:44.000267+0000","last_active":"2024-09-11T09:52:50.014080+0000","last_peered":"2024-09-11T09:52:50.014080+0000","last_clean":"2024-09-11T09:52:50.014080+0000","last_became_active":"2024-09-11T09:52:43.990093+0000","last_became_peered":"2024-09-11T09:52:43.990093+0000","last_unstale":"2024-09-11T09:52:50.014080+0000","last_undegraded":"2024-09-11T09:52:50.014080+0000","last_fullsized":"2024-09-11T09:52:50.014080+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_clean_scrub_stamp":"2024-09-11T09:50:17.477890+0000","objects_scrubbed":0,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:57:25.381839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000328833,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":53,"num_read_kb":12312,"num_write":40,"num_write_kb":33291,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":20971520,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"103'18","reported_seq":430,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-11T09:52:50.186272+0000","last_change":"2024-09-11T09:52:31.939410+0000","last_active":"2024-09-11T09:52:50.186272+0000","last_peered":"2024-09-11T09:52:50.186272+0000","last_clean":"2024-09-11T09:52:50.186272+0000","last_became_active":"2024-09-11T09:50:18.493210+0000","last_became_peered":"2024-09-11T09:50:18.493210+0000","last_unstale":"2024-09-11T09:52:50.186272+0000","last_undegraded":"2024-09-11T09:52:50.186272+0000","last_fullsized":"2024-09-11T09:52:50.186272+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_clean_scrub_stamp":"2024-09-11T09:50:17.477890+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:47:27.009441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043766200000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":174,"num_read_kb":2862,"num_write":16,"num_write_kb":12294,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"103'304","reported_seq":2240,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-11T09:52:50.013413+0000","last_change":"2024-09-11T09:52:46.995900+0000","last_active":"2024-09-11T09:52:50.013413+0000","last_peered":"2024-09-11T09:52:50.013413+0000","last_clean":"2024-09-11T09:52:50.013413+0000","last_became_active":"2024-09-11T09:52:46.995086+0000","last_became_peered":"2024-09-11T09:52:46.995086+0000","last_unstale":"2024-09-11T09:52:50.013413+0000","last_undegraded":"2024-09-11T09:52:50.013413+0000","last_fullsized":"2024-09-11T09:52:50.013413+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_clean_scrub_stamp":"2024-09-11T09:50:17.477890+0000","objects_scrubbed":0,"log_size":304,"log_dups_size":0,"ondisk_log_size":304,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:18:59.319613+0000","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.0022869520000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2124,"num_read_kb":45623,"num_write":237,"num_write_kb":52658,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":51,"num_bytes_recovered":8388608,"num_keys_recovered":8,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"103'35","reported_seq":484,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-11T09:52:50.014091+0000","last_change":"2024-09-11T09:52:31.938934+0000","last_active":"2024-09-11T09:52:50.014091+0000","last_peered":"2024-09-11T09:52:50.014091+0000","last_clean":"2024-09-11T09:52:50.014091+0000","last_became_active":"2024-09-11T09:50:18.493165+0000","last_became_peered":"2024-09-11T09:50:18.493165+0000","last_unstale":"2024-09-11T09:52:50.014091+0000","last_undegraded":"2024-09-11T09:52:50.014091+0000","last_fullsized":"2024-09-11T09:52:50.014091+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_clean_scrub_stamp":"2024-09-11T09:50:17.477890+0000","objects_scrubbed":0,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T21:11:40.038981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000392338,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":202,"num_read_kb":8394,"num_write":27,"num_write_kb":20483,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"103'314","reported_seq":1828,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-11T09:52:50.013370+0000","last_change":"2024-09-11T09:52:42.985519+0000","last_active":"2024-09-11T09:52:50.013370+0000","last_peered":"2024-09-11T09:52:50.013370+0000","last_clean":"2024-09-11T09:52:50.013370+0000","last_became_active":"2024-09-11T09:52:42.985012+0000","last_became_peered":"2024-09-11T09:52:42.985012+0000","last_unstale":"2024-09-11T09:52:50.013370+0000","last_undegraded":"2024-09-11T09:52:50.013370+0000","last_fullsized":"2024-09-11T09:52:50.013370+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_clean_scrub_stamp":"2024-09-11T09:50:17.477890+0000","objects_scrubbed":0,"log_size":314,"log_dups_size":0,"ondisk_log_size":314,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:35:33.948046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026174,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3711,"num_read_kb":35790,"num_write":188,"num_write_kb":54320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":22031360,"num_keys_recovered":14,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"103'62","reported_seq":422,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-11T09:52:50.186313+0000","last_change":"2024-09-11T09:52:38.977632+0000","last_active":"2024-09-11T09:52:50.186313+0000","last_peered":"2024-09-11T09:52:50.186313+0000","last_clean":"2024-09-11T09:52:50.186313+0000","last_became_active":"2024-09-11T09:52:38.970665+0000","last_became_peered":"2024-09-11T09:52:38.970665+0000","last_unstale":"2024-09-11T09:52:50.186313+0000","last_undegraded":"2024-09-11T09:52:50.186313+0000","last_fullsized":"2024-09-11T09:52:50.186313+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_clean_scrub_stamp":"2024-09-11T09:50:17.477890+0000","objects_scrubbed":0,"log_size":62,"log_dups_size":0,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T14:28:24.105715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000633111,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":73,"num_read_kb":24582,"num_write":46,"num_write_kb":29188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":13110272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"103'157","reported_seq":609,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-11T09:52:50.013311+0000","last_change":"2024-09-11T09:52:42.983192+0000","last_active":"2024-09-11T09:52:50.013311+0000","last_peered":"2024-09-11T09:52:50.013311+0000","last_clean":"2024-09-11T09:52:50.013311+0000","last_became_active":"2024-09-11T09:52:42.982623+0000","last_became_peered":"2024-09-11T09:52:42.982623+0000","last_unstale":"2024-09-11T09:52:50.013311+0000","last_undegraded":"2024-09-11T09:52:50.013311+0000","last_fullsized":"2024-09-11T09:52:50.013311+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_clean_scrub_stamp":"2024-09-11T09:50:17.477890+0000","objects_scrubbed":0,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T17:46:27.402621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040558,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":244,"num_read_kb":41018,"num_write":121,"num_write_kb":66581,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"103'212","reported_seq":477,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-11T09:52:50.186301+0000","last_change":"2024-09-11T09:52:31.939671+0000","last_active":"2024-09-11T09:52:50.186301+0000","last_peered":"2024-09-11T09:52:50.186301+0000","last_clean":"2024-09-11T09:52:50.186301+0000","last_became_active":"2024-09-11T09:52:04.840466+0000","last_became_peered":"2024-09-11T09:52:04.840466+0000","last_unstale":"2024-09-11T09:52:50.186301+0000","last_undegraded":"2024-09-11T09:52:50.186301+0000","last_fullsized":"2024-09-11T09:52:50.186301+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_clean_scrub_stamp":"2024-09-11T09:50:17.477890+0000","objects_scrubbed":0,"log_size":212,"log_dups_size":0,"ondisk_log_size":212,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:52:38.106407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045445899999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":971,"num_read_kb":777,"num_write":338,"num_write_kb":4368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":4194418,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7552,"num_read_kb":171358,"num_write":1013,"num_write_kb":273183,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":105,"num_bytes_recovered":68696178,"num_keys_recovered":22,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1150,"ondisk_log_size":1150,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1771104,"data_compressed":100760,"data_compressed_allocated":135168,"data_compressed_original":1753088,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":12,"seq":51539607589,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32644,"kb_used_data":1448,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339196,"statfs":{"total":96636764160,"available":96603336704,"internally_reserved":0,"allocated":1482752,"data_stored":1139785,"data_compressed":32496,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":12,"seq":51539607588,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36784,"kb_used_data":1432,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335056,"statfs":{"total":96636764160,"available":96599097344,"internally_reserved":0,"allocated":1466368,"data_stored":1139785,"data_compressed":34132,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":11,"seq":47244640293,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32576,"kb_used_data":1388,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339264,"statfs":{"total":96636764160,"available":96603406336,"internally_reserved":0,"allocated":1421312,"data_stored":549436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":66,"seq":283467841552,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29500,"kb_used_data":1384,"kb_used_omap":4,"kb_used_meta":28091,"kb_avail":94342340,"statfs":{"total":96636764160,"available":96606556160,"internally_reserved":0,"allocated":1417216,"data_stored":549417,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":28765872},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640292,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36724,"kb_used_data":1372,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335116,"statfs":{"total":96636764160,"available":96599158784,"internally_reserved":0,"allocated":1404928,"data_stored":544839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640292,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36736,"kb_used_data":1388,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335104,"statfs":{"total":96636764160,"available":96599146496,"internally_reserved":0,"allocated":1421312,"data_stored":549436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":34132,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":34132,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32496,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T09:52:54.894 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-09-11T09:52:54.894 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr_debug_aggressive_pg_num_changes 2024-09-11T09:52:55.202 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-09-11T09:52:55.212 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-11T09:52:55.212 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T09:52:55.499 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-11T09:52:55.499 INFO:teuthology.orchestra.run.smithi017.stderr:dumped all 2024-09-11T09:52:55.511 INFO:teuthology.orchestra.run.smithi017.stdout:{"pg_ready":true,"pg_map":{"version":196,"stamp":"2024-09-11T09:52:54.443268+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":7668,"num_read_kb":171458,"num_write":1172,"num_write_kb":275599,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":113,"num_bytes_recovered":71057650,"num_keys_recovered":22,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1267,"ondisk_log_size":1267,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":19,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":204964,"kb_used_data":8412,"kb_used_omap":12,"kb_used_meta":196403,"kb_avail":566026076,"statfs":{"total":579820584960,"available":579610701824,"internally_reserved":0,"allocated":8613888,"data_stored":4472698,"data_compressed":66628,"data_compressed_allocated":90112,"data_compressed_original":1163264,"omap_allocated":12403,"internal_metadata":201117581},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.440981"},"pg_stats":[{"pgid":"1.0","version":"13'117","reported_seq":421,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-11T09:52:50.186293+0000","last_change":"2024-09-11T09:52:41.980919+0000","last_active":"2024-09-11T09:52:50.186293+0000","last_peered":"2024-09-11T09:52:50.186293+0000","last_clean":"2024-09-11T09:52:50.186293+0000","last_became_active":"2024-09-11T09:52:41.980573+0000","last_became_peered":"2024-09-11T09:52:41.980573+0000","last_unstale":"2024-09-11T09:52:50.186293+0000","last_undegraded":"2024-09-11T09:52:50.186293+0000","last_fullsized":"2024-09-11T09:52:50.186293+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:50:10.457262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:50:10.457262+0000","last_clean_scrub_stamp":"2024-09-11T09:50:10.457262+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:59:08.639668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"103'48","reported_seq":392,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-11T09:52:50.014080+0000","last_change":"2024-09-11T09:52:44.000267+0000","last_active":"2024-09-11T09:52:50.014080+0000","last_peered":"2024-09-11T09:52:50.014080+0000","last_clean":"2024-09-11T09:52:50.014080+0000","last_became_active":"2024-09-11T09:52:43.990093+0000","last_became_peered":"2024-09-11T09:52:43.990093+0000","last_unstale":"2024-09-11T09:52:50.014080+0000","last_undegraded":"2024-09-11T09:52:50.014080+0000","last_fullsized":"2024-09-11T09:52:50.014080+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_clean_scrub_stamp":"2024-09-11T09:50:17.477890+0000","objects_scrubbed":0,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:57:25.381839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000328833,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":53,"num_read_kb":12312,"num_write":40,"num_write_kb":33291,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":20971520,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"103'18","reported_seq":430,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-11T09:52:50.186272+0000","last_change":"2024-09-11T09:52:31.939410+0000","last_active":"2024-09-11T09:52:50.186272+0000","last_peered":"2024-09-11T09:52:50.186272+0000","last_clean":"2024-09-11T09:52:50.186272+0000","last_became_active":"2024-09-11T09:50:18.493210+0000","last_became_peered":"2024-09-11T09:50:18.493210+0000","last_unstale":"2024-09-11T09:52:50.186272+0000","last_undegraded":"2024-09-11T09:52:50.186272+0000","last_fullsized":"2024-09-11T09:52:50.186272+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_clean_scrub_stamp":"2024-09-11T09:50:17.477890+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:47:27.009441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043766200000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":174,"num_read_kb":2862,"num_write":16,"num_write_kb":12294,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"103'304","reported_seq":2240,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-11T09:52:50.013413+0000","last_change":"2024-09-11T09:52:46.995900+0000","last_active":"2024-09-11T09:52:50.013413+0000","last_peered":"2024-09-11T09:52:50.013413+0000","last_clean":"2024-09-11T09:52:50.013413+0000","last_became_active":"2024-09-11T09:52:46.995086+0000","last_became_peered":"2024-09-11T09:52:46.995086+0000","last_unstale":"2024-09-11T09:52:50.013413+0000","last_undegraded":"2024-09-11T09:52:50.013413+0000","last_fullsized":"2024-09-11T09:52:50.013413+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_clean_scrub_stamp":"2024-09-11T09:50:17.477890+0000","objects_scrubbed":0,"log_size":304,"log_dups_size":0,"ondisk_log_size":304,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:18:59.319613+0000","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.0022869520000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2124,"num_read_kb":45623,"num_write":237,"num_write_kb":52658,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":51,"num_bytes_recovered":8388608,"num_keys_recovered":8,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"103'35","reported_seq":484,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-11T09:52:50.014091+0000","last_change":"2024-09-11T09:52:31.938934+0000","last_active":"2024-09-11T09:52:50.014091+0000","last_peered":"2024-09-11T09:52:50.014091+0000","last_clean":"2024-09-11T09:52:50.014091+0000","last_became_active":"2024-09-11T09:50:18.493165+0000","last_became_peered":"2024-09-11T09:50:18.493165+0000","last_unstale":"2024-09-11T09:52:50.014091+0000","last_undegraded":"2024-09-11T09:52:50.014091+0000","last_fullsized":"2024-09-11T09:52:50.014091+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_clean_scrub_stamp":"2024-09-11T09:50:17.477890+0000","objects_scrubbed":0,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T21:11:40.038981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000392338,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":202,"num_read_kb":8394,"num_write":27,"num_write_kb":20483,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"103'314","reported_seq":1828,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-11T09:52:50.013370+0000","last_change":"2024-09-11T09:52:42.985519+0000","last_active":"2024-09-11T09:52:50.013370+0000","last_peered":"2024-09-11T09:52:50.013370+0000","last_clean":"2024-09-11T09:52:50.013370+0000","last_became_active":"2024-09-11T09:52:42.985012+0000","last_became_peered":"2024-09-11T09:52:42.985012+0000","last_unstale":"2024-09-11T09:52:50.013370+0000","last_undegraded":"2024-09-11T09:52:50.013370+0000","last_fullsized":"2024-09-11T09:52:50.013370+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_clean_scrub_stamp":"2024-09-11T09:50:17.477890+0000","objects_scrubbed":0,"log_size":314,"log_dups_size":0,"ondisk_log_size":314,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:35:33.948046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026174,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3711,"num_read_kb":35790,"num_write":188,"num_write_kb":54320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":22031360,"num_keys_recovered":14,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"103'62","reported_seq":422,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-11T09:52:50.186313+0000","last_change":"2024-09-11T09:52:38.977632+0000","last_active":"2024-09-11T09:52:50.186313+0000","last_peered":"2024-09-11T09:52:50.186313+0000","last_clean":"2024-09-11T09:52:50.186313+0000","last_became_active":"2024-09-11T09:52:38.970665+0000","last_became_peered":"2024-09-11T09:52:38.970665+0000","last_unstale":"2024-09-11T09:52:50.186313+0000","last_undegraded":"2024-09-11T09:52:50.186313+0000","last_fullsized":"2024-09-11T09:52:50.186313+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_clean_scrub_stamp":"2024-09-11T09:50:17.477890+0000","objects_scrubbed":0,"log_size":62,"log_dups_size":0,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T14:28:24.105715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000633111,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":73,"num_read_kb":24582,"num_write":46,"num_write_kb":29188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":13110272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"103'157","reported_seq":609,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-11T09:52:50.013311+0000","last_change":"2024-09-11T09:52:42.983192+0000","last_active":"2024-09-11T09:52:50.013311+0000","last_peered":"2024-09-11T09:52:50.013311+0000","last_clean":"2024-09-11T09:52:50.013311+0000","last_became_active":"2024-09-11T09:52:42.982623+0000","last_became_peered":"2024-09-11T09:52:42.982623+0000","last_unstale":"2024-09-11T09:52:50.013311+0000","last_undegraded":"2024-09-11T09:52:50.013311+0000","last_fullsized":"2024-09-11T09:52:50.013311+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_clean_scrub_stamp":"2024-09-11T09:50:17.477890+0000","objects_scrubbed":0,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T17:46:27.402621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040558,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":244,"num_read_kb":41018,"num_write":121,"num_write_kb":66581,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"103'212","reported_seq":477,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-11T09:52:50.186301+0000","last_change":"2024-09-11T09:52:31.939671+0000","last_active":"2024-09-11T09:52:50.186301+0000","last_peered":"2024-09-11T09:52:50.186301+0000","last_clean":"2024-09-11T09:52:50.186301+0000","last_became_active":"2024-09-11T09:52:04.840466+0000","last_became_peered":"2024-09-11T09:52:04.840466+0000","last_unstale":"2024-09-11T09:52:50.186301+0000","last_undegraded":"2024-09-11T09:52:50.186301+0000","last_fullsized":"2024-09-11T09:52:50.186301+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_clean_scrub_stamp":"2024-09-11T09:50:17.477890+0000","objects_scrubbed":0,"log_size":212,"log_dups_size":0,"ondisk_log_size":212,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:52:38.106407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045445899999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":971,"num_read_kb":777,"num_write":338,"num_write_kb":4368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":4194418,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7552,"num_read_kb":171358,"num_write":1013,"num_write_kb":273183,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":105,"num_bytes_recovered":68696178,"num_keys_recovered":22,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1150,"ondisk_log_size":1150,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1771104,"data_compressed":100760,"data_compressed_allocated":135168,"data_compressed_original":1753088,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":12,"seq":51539607589,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32644,"kb_used_data":1448,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339196,"statfs":{"total":96636764160,"available":96603336704,"internally_reserved":0,"allocated":1482752,"data_stored":1139785,"data_compressed":32496,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":12,"seq":51539607588,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36784,"kb_used_data":1432,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335056,"statfs":{"total":96636764160,"available":96599097344,"internally_reserved":0,"allocated":1466368,"data_stored":1139785,"data_compressed":34132,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":11,"seq":47244640293,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32576,"kb_used_data":1388,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339264,"statfs":{"total":96636764160,"available":96603406336,"internally_reserved":0,"allocated":1421312,"data_stored":549436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":66,"seq":283467841552,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29500,"kb_used_data":1384,"kb_used_omap":4,"kb_used_meta":28091,"kb_avail":94342340,"statfs":{"total":96636764160,"available":96606556160,"internally_reserved":0,"allocated":1417216,"data_stored":549417,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":28765872},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640292,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36724,"kb_used_data":1372,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335116,"statfs":{"total":96636764160,"available":96599158784,"internally_reserved":0,"allocated":1404928,"data_stored":544839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640292,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36736,"kb_used_data":1388,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335104,"statfs":{"total":96636764160,"available":96599146496,"internally_reserved":0,"allocated":1421312,"data_stored":549436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":34132,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":34132,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32496,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T09:52:55.512 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T09:52:55.800 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-11T09:52:55.800 INFO:teuthology.orchestra.run.smithi017.stderr:dumped all 2024-09-11T09:52:55.812 INFO:teuthology.orchestra.run.smithi017.stdout:{"pg_ready":true,"pg_map":{"version":196,"stamp":"2024-09-11T09:52:54.443268+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":7668,"num_read_kb":171458,"num_write":1172,"num_write_kb":275599,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":113,"num_bytes_recovered":71057650,"num_keys_recovered":22,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1267,"ondisk_log_size":1267,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":19,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":204964,"kb_used_data":8412,"kb_used_omap":12,"kb_used_meta":196403,"kb_avail":566026076,"statfs":{"total":579820584960,"available":579610701824,"internally_reserved":0,"allocated":8613888,"data_stored":4472698,"data_compressed":66628,"data_compressed_allocated":90112,"data_compressed_original":1163264,"omap_allocated":12403,"internal_metadata":201117581},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.440981"},"pg_stats":[{"pgid":"1.0","version":"13'117","reported_seq":421,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-11T09:52:50.186293+0000","last_change":"2024-09-11T09:52:41.980919+0000","last_active":"2024-09-11T09:52:50.186293+0000","last_peered":"2024-09-11T09:52:50.186293+0000","last_clean":"2024-09-11T09:52:50.186293+0000","last_became_active":"2024-09-11T09:52:41.980573+0000","last_became_peered":"2024-09-11T09:52:41.980573+0000","last_unstale":"2024-09-11T09:52:50.186293+0000","last_undegraded":"2024-09-11T09:52:50.186293+0000","last_fullsized":"2024-09-11T09:52:50.186293+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:50:10.457262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:50:10.457262+0000","last_clean_scrub_stamp":"2024-09-11T09:50:10.457262+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:59:08.639668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"103'48","reported_seq":392,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-11T09:52:50.014080+0000","last_change":"2024-09-11T09:52:44.000267+0000","last_active":"2024-09-11T09:52:50.014080+0000","last_peered":"2024-09-11T09:52:50.014080+0000","last_clean":"2024-09-11T09:52:50.014080+0000","last_became_active":"2024-09-11T09:52:43.990093+0000","last_became_peered":"2024-09-11T09:52:43.990093+0000","last_unstale":"2024-09-11T09:52:50.014080+0000","last_undegraded":"2024-09-11T09:52:50.014080+0000","last_fullsized":"2024-09-11T09:52:50.014080+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_clean_scrub_stamp":"2024-09-11T09:50:17.477890+0000","objects_scrubbed":0,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:57:25.381839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000328833,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":53,"num_read_kb":12312,"num_write":40,"num_write_kb":33291,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":20971520,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"103'18","reported_seq":430,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-11T09:52:50.186272+0000","last_change":"2024-09-11T09:52:31.939410+0000","last_active":"2024-09-11T09:52:50.186272+0000","last_peered":"2024-09-11T09:52:50.186272+0000","last_clean":"2024-09-11T09:52:50.186272+0000","last_became_active":"2024-09-11T09:50:18.493210+0000","last_became_peered":"2024-09-11T09:50:18.493210+0000","last_unstale":"2024-09-11T09:52:50.186272+0000","last_undegraded":"2024-09-11T09:52:50.186272+0000","last_fullsized":"2024-09-11T09:52:50.186272+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_clean_scrub_stamp":"2024-09-11T09:50:17.477890+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:47:27.009441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043766200000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":174,"num_read_kb":2862,"num_write":16,"num_write_kb":12294,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"103'304","reported_seq":2240,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-11T09:52:50.013413+0000","last_change":"2024-09-11T09:52:46.995900+0000","last_active":"2024-09-11T09:52:50.013413+0000","last_peered":"2024-09-11T09:52:50.013413+0000","last_clean":"2024-09-11T09:52:50.013413+0000","last_became_active":"2024-09-11T09:52:46.995086+0000","last_became_peered":"2024-09-11T09:52:46.995086+0000","last_unstale":"2024-09-11T09:52:50.013413+0000","last_undegraded":"2024-09-11T09:52:50.013413+0000","last_fullsized":"2024-09-11T09:52:50.013413+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_clean_scrub_stamp":"2024-09-11T09:50:17.477890+0000","objects_scrubbed":0,"log_size":304,"log_dups_size":0,"ondisk_log_size":304,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:18:59.319613+0000","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.0022869520000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2124,"num_read_kb":45623,"num_write":237,"num_write_kb":52658,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":51,"num_bytes_recovered":8388608,"num_keys_recovered":8,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"103'35","reported_seq":484,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-11T09:52:50.014091+0000","last_change":"2024-09-11T09:52:31.938934+0000","last_active":"2024-09-11T09:52:50.014091+0000","last_peered":"2024-09-11T09:52:50.014091+0000","last_clean":"2024-09-11T09:52:50.014091+0000","last_became_active":"2024-09-11T09:50:18.493165+0000","last_became_peered":"2024-09-11T09:50:18.493165+0000","last_unstale":"2024-09-11T09:52:50.014091+0000","last_undegraded":"2024-09-11T09:52:50.014091+0000","last_fullsized":"2024-09-11T09:52:50.014091+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_clean_scrub_stamp":"2024-09-11T09:50:17.477890+0000","objects_scrubbed":0,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T21:11:40.038981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000392338,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":202,"num_read_kb":8394,"num_write":27,"num_write_kb":20483,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"103'314","reported_seq":1828,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-11T09:52:50.013370+0000","last_change":"2024-09-11T09:52:42.985519+0000","last_active":"2024-09-11T09:52:50.013370+0000","last_peered":"2024-09-11T09:52:50.013370+0000","last_clean":"2024-09-11T09:52:50.013370+0000","last_became_active":"2024-09-11T09:52:42.985012+0000","last_became_peered":"2024-09-11T09:52:42.985012+0000","last_unstale":"2024-09-11T09:52:50.013370+0000","last_undegraded":"2024-09-11T09:52:50.013370+0000","last_fullsized":"2024-09-11T09:52:50.013370+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_clean_scrub_stamp":"2024-09-11T09:50:17.477890+0000","objects_scrubbed":0,"log_size":314,"log_dups_size":0,"ondisk_log_size":314,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:35:33.948046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026174,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3711,"num_read_kb":35790,"num_write":188,"num_write_kb":54320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":22031360,"num_keys_recovered":14,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"103'62","reported_seq":422,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-11T09:52:50.186313+0000","last_change":"2024-09-11T09:52:38.977632+0000","last_active":"2024-09-11T09:52:50.186313+0000","last_peered":"2024-09-11T09:52:50.186313+0000","last_clean":"2024-09-11T09:52:50.186313+0000","last_became_active":"2024-09-11T09:52:38.970665+0000","last_became_peered":"2024-09-11T09:52:38.970665+0000","last_unstale":"2024-09-11T09:52:50.186313+0000","last_undegraded":"2024-09-11T09:52:50.186313+0000","last_fullsized":"2024-09-11T09:52:50.186313+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_clean_scrub_stamp":"2024-09-11T09:50:17.477890+0000","objects_scrubbed":0,"log_size":62,"log_dups_size":0,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T14:28:24.105715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000633111,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":73,"num_read_kb":24582,"num_write":46,"num_write_kb":29188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":13110272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"103'157","reported_seq":609,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-11T09:52:50.013311+0000","last_change":"2024-09-11T09:52:42.983192+0000","last_active":"2024-09-11T09:52:50.013311+0000","last_peered":"2024-09-11T09:52:50.013311+0000","last_clean":"2024-09-11T09:52:50.013311+0000","last_became_active":"2024-09-11T09:52:42.982623+0000","last_became_peered":"2024-09-11T09:52:42.982623+0000","last_unstale":"2024-09-11T09:52:50.013311+0000","last_undegraded":"2024-09-11T09:52:50.013311+0000","last_fullsized":"2024-09-11T09:52:50.013311+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_clean_scrub_stamp":"2024-09-11T09:50:17.477890+0000","objects_scrubbed":0,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T17:46:27.402621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040558,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":244,"num_read_kb":41018,"num_write":121,"num_write_kb":66581,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"103'212","reported_seq":477,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-11T09:52:50.186301+0000","last_change":"2024-09-11T09:52:31.939671+0000","last_active":"2024-09-11T09:52:50.186301+0000","last_peered":"2024-09-11T09:52:50.186301+0000","last_clean":"2024-09-11T09:52:50.186301+0000","last_became_active":"2024-09-11T09:52:04.840466+0000","last_became_peered":"2024-09-11T09:52:04.840466+0000","last_unstale":"2024-09-11T09:52:50.186301+0000","last_undegraded":"2024-09-11T09:52:50.186301+0000","last_fullsized":"2024-09-11T09:52:50.186301+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_clean_scrub_stamp":"2024-09-11T09:50:17.477890+0000","objects_scrubbed":0,"log_size":212,"log_dups_size":0,"ondisk_log_size":212,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:52:38.106407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045445899999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":971,"num_read_kb":777,"num_write":338,"num_write_kb":4368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":4194418,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7552,"num_read_kb":171358,"num_write":1013,"num_write_kb":273183,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":105,"num_bytes_recovered":68696178,"num_keys_recovered":22,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1150,"ondisk_log_size":1150,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1771104,"data_compressed":100760,"data_compressed_allocated":135168,"data_compressed_original":1753088,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":12,"seq":51539607589,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32644,"kb_used_data":1448,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339196,"statfs":{"total":96636764160,"available":96603336704,"internally_reserved":0,"allocated":1482752,"data_stored":1139785,"data_compressed":32496,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":12,"seq":51539607588,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36784,"kb_used_data":1432,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335056,"statfs":{"total":96636764160,"available":96599097344,"internally_reserved":0,"allocated":1466368,"data_stored":1139785,"data_compressed":34132,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":11,"seq":47244640293,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32576,"kb_used_data":1388,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339264,"statfs":{"total":96636764160,"available":96603406336,"internally_reserved":0,"allocated":1421312,"data_stored":549436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":66,"seq":283467841552,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29500,"kb_used_data":1384,"kb_used_omap":4,"kb_used_meta":28091,"kb_avail":94342340,"statfs":{"total":96636764160,"available":96606556160,"internally_reserved":0,"allocated":1417216,"data_stored":549417,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":28765872},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640292,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36724,"kb_used_data":1372,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335116,"statfs":{"total":96636764160,"available":96599158784,"internally_reserved":0,"allocated":1404928,"data_stored":544839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640292,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36736,"kb_used_data":1388,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335104,"statfs":{"total":96636764160,"available":96599146496,"internally_reserved":0,"allocated":1421312,"data_stored":549436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":34132,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":34132,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32496,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T09:52:55.813 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-11T09:52:55.813 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T09:52:56.102 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-11T09:52:56.102 INFO:teuthology.orchestra.run.smithi017.stderr:dumped all 2024-09-11T09:52:56.114 INFO:teuthology.orchestra.run.smithi017.stdout:{"pg_ready":true,"pg_map":{"version":196,"stamp":"2024-09-11T09:52:54.443268+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":7668,"num_read_kb":171458,"num_write":1172,"num_write_kb":275599,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":113,"num_bytes_recovered":71057650,"num_keys_recovered":22,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1267,"ondisk_log_size":1267,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":19,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":204964,"kb_used_data":8412,"kb_used_omap":12,"kb_used_meta":196403,"kb_avail":566026076,"statfs":{"total":579820584960,"available":579610701824,"internally_reserved":0,"allocated":8613888,"data_stored":4472698,"data_compressed":66628,"data_compressed_allocated":90112,"data_compressed_original":1163264,"omap_allocated":12403,"internal_metadata":201117581},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.440981"},"pg_stats":[{"pgid":"1.0","version":"13'117","reported_seq":421,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-11T09:52:50.186293+0000","last_change":"2024-09-11T09:52:41.980919+0000","last_active":"2024-09-11T09:52:50.186293+0000","last_peered":"2024-09-11T09:52:50.186293+0000","last_clean":"2024-09-11T09:52:50.186293+0000","last_became_active":"2024-09-11T09:52:41.980573+0000","last_became_peered":"2024-09-11T09:52:41.980573+0000","last_unstale":"2024-09-11T09:52:50.186293+0000","last_undegraded":"2024-09-11T09:52:50.186293+0000","last_fullsized":"2024-09-11T09:52:50.186293+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:50:10.457262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:50:10.457262+0000","last_clean_scrub_stamp":"2024-09-11T09:50:10.457262+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:59:08.639668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"103'48","reported_seq":392,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-11T09:52:50.014080+0000","last_change":"2024-09-11T09:52:44.000267+0000","last_active":"2024-09-11T09:52:50.014080+0000","last_peered":"2024-09-11T09:52:50.014080+0000","last_clean":"2024-09-11T09:52:50.014080+0000","last_became_active":"2024-09-11T09:52:43.990093+0000","last_became_peered":"2024-09-11T09:52:43.990093+0000","last_unstale":"2024-09-11T09:52:50.014080+0000","last_undegraded":"2024-09-11T09:52:50.014080+0000","last_fullsized":"2024-09-11T09:52:50.014080+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_clean_scrub_stamp":"2024-09-11T09:50:17.477890+0000","objects_scrubbed":0,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:57:25.381839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000328833,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":53,"num_read_kb":12312,"num_write":40,"num_write_kb":33291,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":20971520,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"103'18","reported_seq":430,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-11T09:52:50.186272+0000","last_change":"2024-09-11T09:52:31.939410+0000","last_active":"2024-09-11T09:52:50.186272+0000","last_peered":"2024-09-11T09:52:50.186272+0000","last_clean":"2024-09-11T09:52:50.186272+0000","last_became_active":"2024-09-11T09:50:18.493210+0000","last_became_peered":"2024-09-11T09:50:18.493210+0000","last_unstale":"2024-09-11T09:52:50.186272+0000","last_undegraded":"2024-09-11T09:52:50.186272+0000","last_fullsized":"2024-09-11T09:52:50.186272+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_clean_scrub_stamp":"2024-09-11T09:50:17.477890+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:47:27.009441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043766200000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":174,"num_read_kb":2862,"num_write":16,"num_write_kb":12294,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"103'304","reported_seq":2240,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-11T09:52:50.013413+0000","last_change":"2024-09-11T09:52:46.995900+0000","last_active":"2024-09-11T09:52:50.013413+0000","last_peered":"2024-09-11T09:52:50.013413+0000","last_clean":"2024-09-11T09:52:50.013413+0000","last_became_active":"2024-09-11T09:52:46.995086+0000","last_became_peered":"2024-09-11T09:52:46.995086+0000","last_unstale":"2024-09-11T09:52:50.013413+0000","last_undegraded":"2024-09-11T09:52:50.013413+0000","last_fullsized":"2024-09-11T09:52:50.013413+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_clean_scrub_stamp":"2024-09-11T09:50:17.477890+0000","objects_scrubbed":0,"log_size":304,"log_dups_size":0,"ondisk_log_size":304,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:18:59.319613+0000","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.0022869520000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2124,"num_read_kb":45623,"num_write":237,"num_write_kb":52658,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":51,"num_bytes_recovered":8388608,"num_keys_recovered":8,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"103'35","reported_seq":484,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-11T09:52:50.014091+0000","last_change":"2024-09-11T09:52:31.938934+0000","last_active":"2024-09-11T09:52:50.014091+0000","last_peered":"2024-09-11T09:52:50.014091+0000","last_clean":"2024-09-11T09:52:50.014091+0000","last_became_active":"2024-09-11T09:50:18.493165+0000","last_became_peered":"2024-09-11T09:50:18.493165+0000","last_unstale":"2024-09-11T09:52:50.014091+0000","last_undegraded":"2024-09-11T09:52:50.014091+0000","last_fullsized":"2024-09-11T09:52:50.014091+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_clean_scrub_stamp":"2024-09-11T09:50:17.477890+0000","objects_scrubbed":0,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T21:11:40.038981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000392338,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":202,"num_read_kb":8394,"num_write":27,"num_write_kb":20483,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"103'314","reported_seq":1828,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-11T09:52:50.013370+0000","last_change":"2024-09-11T09:52:42.985519+0000","last_active":"2024-09-11T09:52:50.013370+0000","last_peered":"2024-09-11T09:52:50.013370+0000","last_clean":"2024-09-11T09:52:50.013370+0000","last_became_active":"2024-09-11T09:52:42.985012+0000","last_became_peered":"2024-09-11T09:52:42.985012+0000","last_unstale":"2024-09-11T09:52:50.013370+0000","last_undegraded":"2024-09-11T09:52:50.013370+0000","last_fullsized":"2024-09-11T09:52:50.013370+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_clean_scrub_stamp":"2024-09-11T09:50:17.477890+0000","objects_scrubbed":0,"log_size":314,"log_dups_size":0,"ondisk_log_size":314,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:35:33.948046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026174,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3711,"num_read_kb":35790,"num_write":188,"num_write_kb":54320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":22031360,"num_keys_recovered":14,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"103'62","reported_seq":422,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-11T09:52:50.186313+0000","last_change":"2024-09-11T09:52:38.977632+0000","last_active":"2024-09-11T09:52:50.186313+0000","last_peered":"2024-09-11T09:52:50.186313+0000","last_clean":"2024-09-11T09:52:50.186313+0000","last_became_active":"2024-09-11T09:52:38.970665+0000","last_became_peered":"2024-09-11T09:52:38.970665+0000","last_unstale":"2024-09-11T09:52:50.186313+0000","last_undegraded":"2024-09-11T09:52:50.186313+0000","last_fullsized":"2024-09-11T09:52:50.186313+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_clean_scrub_stamp":"2024-09-11T09:50:17.477890+0000","objects_scrubbed":0,"log_size":62,"log_dups_size":0,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T14:28:24.105715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000633111,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":73,"num_read_kb":24582,"num_write":46,"num_write_kb":29188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":13110272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"103'157","reported_seq":609,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-11T09:52:50.013311+0000","last_change":"2024-09-11T09:52:42.983192+0000","last_active":"2024-09-11T09:52:50.013311+0000","last_peered":"2024-09-11T09:52:50.013311+0000","last_clean":"2024-09-11T09:52:50.013311+0000","last_became_active":"2024-09-11T09:52:42.982623+0000","last_became_peered":"2024-09-11T09:52:42.982623+0000","last_unstale":"2024-09-11T09:52:50.013311+0000","last_undegraded":"2024-09-11T09:52:50.013311+0000","last_fullsized":"2024-09-11T09:52:50.013311+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_clean_scrub_stamp":"2024-09-11T09:50:17.477890+0000","objects_scrubbed":0,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T17:46:27.402621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040558,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":244,"num_read_kb":41018,"num_write":121,"num_write_kb":66581,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"103'212","reported_seq":477,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-11T09:52:50.186301+0000","last_change":"2024-09-11T09:52:31.939671+0000","last_active":"2024-09-11T09:52:50.186301+0000","last_peered":"2024-09-11T09:52:50.186301+0000","last_clean":"2024-09-11T09:52:50.186301+0000","last_became_active":"2024-09-11T09:52:04.840466+0000","last_became_peered":"2024-09-11T09:52:04.840466+0000","last_unstale":"2024-09-11T09:52:50.186301+0000","last_undegraded":"2024-09-11T09:52:50.186301+0000","last_fullsized":"2024-09-11T09:52:50.186301+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_clean_scrub_stamp":"2024-09-11T09:50:17.477890+0000","objects_scrubbed":0,"log_size":212,"log_dups_size":0,"ondisk_log_size":212,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:52:38.106407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045445899999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":971,"num_read_kb":777,"num_write":338,"num_write_kb":4368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":4194418,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7552,"num_read_kb":171358,"num_write":1013,"num_write_kb":273183,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":105,"num_bytes_recovered":68696178,"num_keys_recovered":22,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1150,"ondisk_log_size":1150,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1771104,"data_compressed":100760,"data_compressed_allocated":135168,"data_compressed_original":1753088,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":12,"seq":51539607589,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32644,"kb_used_data":1448,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339196,"statfs":{"total":96636764160,"available":96603336704,"internally_reserved":0,"allocated":1482752,"data_stored":1139785,"data_compressed":32496,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":12,"seq":51539607588,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36784,"kb_used_data":1432,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335056,"statfs":{"total":96636764160,"available":96599097344,"internally_reserved":0,"allocated":1466368,"data_stored":1139785,"data_compressed":34132,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":11,"seq":47244640293,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32576,"kb_used_data":1388,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339264,"statfs":{"total":96636764160,"available":96603406336,"internally_reserved":0,"allocated":1421312,"data_stored":549436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":66,"seq":283467841552,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29500,"kb_used_data":1384,"kb_used_omap":4,"kb_used_meta":28091,"kb_avail":94342340,"statfs":{"total":96636764160,"available":96606556160,"internally_reserved":0,"allocated":1417216,"data_stored":549417,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":28765872},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640292,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36724,"kb_used_data":1372,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335116,"statfs":{"total":96636764160,"available":96599158784,"internally_reserved":0,"allocated":1404928,"data_stored":544839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640292,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36736,"kb_used_data":1388,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335104,"statfs":{"total":96636764160,"available":96599146496,"internally_reserved":0,"allocated":1421312,"data_stored":549436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":34132,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":34132,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32496,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T09:52:56.115 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-11T09:52:56.403 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-11T09:52:56.403 INFO:teuthology.orchestra.run.smithi017.stdout:{"epoch":119,"fsid":"a2f1daec-d3b0-4bf5-82c8-72ecbb93d4e4","created":"2024-09-11T09:50:00.165804+0000","modified":"2024-09-11T09:52:50.010309+0000","last_up_change":"2024-09-11T09:51:44.764990+0000","last_in_change":"2024-09-11T09:52:37.955638+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-11T09:50:10.457184+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-11T09:50:16.993202+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"101","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":7,"snap_epoch":101,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4829b7a1-d607-4a56-b6d2-55036cb8acf8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":114,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6808","nonce":4154449929},{"type":"v1","addr":"172.21.15.17:6809","nonce":4154449929}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6810","nonce":4154449929},{"type":"v1","addr":"172.21.15.17:6811","nonce":4154449929}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6814","nonce":4154449929},{"type":"v1","addr":"172.21.15.17:6815","nonce":4154449929}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6812","nonce":4154449929},{"type":"v1","addr":"172.21.15.17:6813","nonce":4154449929}]},"public_addr":"172.21.15.17:6809/4154449929","cluster_addr":"172.21.15.17:6811/4154449929","heartbeat_back_addr":"172.21.15.17:6815/4154449929","heartbeat_front_addr":"172.21.15.17:6813/4154449929","state":["exists","up"]},{"osd":1,"uuid":"ab1d5df0-3c0e-44cd-88a0-96b07964148b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6800","nonce":4139511644},{"type":"v1","addr":"172.21.15.17:6801","nonce":4139511644}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6802","nonce":4139511644},{"type":"v1","addr":"172.21.15.17:6803","nonce":4139511644}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6806","nonce":4139511644},{"type":"v1","addr":"172.21.15.17:6807","nonce":4139511644}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6804","nonce":4139511644},{"type":"v1","addr":"172.21.15.17:6805","nonce":4139511644}]},"public_addr":"172.21.15.17:6801/4139511644","cluster_addr":"172.21.15.17:6803/4139511644","heartbeat_back_addr":"172.21.15.17:6807/4139511644","heartbeat_front_addr":"172.21.15.17:6805/4139511644","state":["exists","up"]},{"osd":2,"uuid":"4c8e50db-b49c-4d44-a913-47b5b30f8699","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6816","nonce":1151282506},{"type":"v1","addr":"172.21.15.17:6817","nonce":1151282506}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6818","nonce":1151282506},{"type":"v1","addr":"172.21.15.17:6819","nonce":1151282506}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6822","nonce":1151282506},{"type":"v1","addr":"172.21.15.17:6823","nonce":1151282506}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.17:6820","nonce":1151282506},{"type":"v1","addr":"172.21.15.17:6821","nonce":1151282506}]},"public_addr":"172.21.15.17:6817/1151282506","cluster_addr":"172.21.15.17:6819/1151282506","heartbeat_back_addr":"172.21.15.17:6823/1151282506","heartbeat_front_addr":"172.21.15.17:6821/1151282506","state":["exists","up"]},{"osd":3,"uuid":"37abc55b-a1eb-49d5-b1ff-b507ee2cb5a6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":110,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6816","nonce":332854807},{"type":"v1","addr":"172.21.15.96:6817","nonce":332854807}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6818","nonce":332854807},{"type":"v1","addr":"172.21.15.96:6819","nonce":332854807}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6822","nonce":332854807},{"type":"v1","addr":"172.21.15.96:6823","nonce":332854807}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6820","nonce":332854807},{"type":"v1","addr":"172.21.15.96:6821","nonce":332854807}]},"public_addr":"172.21.15.96:6817/332854807","cluster_addr":"172.21.15.96:6819/332854807","heartbeat_back_addr":"172.21.15.96:6823/332854807","heartbeat_front_addr":"172.21.15.96:6821/332854807","state":["exists","up"]},{"osd":4,"uuid":"067d9586-4f5a-4d17-96c5-86a7b8bcd114","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":11,"last_clean_end":54,"up_from":66,"up_thru":101,"down_at":55,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6800","nonce":3374675329},{"type":"v1","addr":"172.21.15.96:6801","nonce":3374675329}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6802","nonce":3374675329},{"type":"v1","addr":"172.21.15.96:6803","nonce":3374675329}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6806","nonce":3374675329},{"type":"v1","addr":"172.21.15.96:6807","nonce":3374675329}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6804","nonce":3374675329},{"type":"v1","addr":"172.21.15.96:6805","nonce":3374675329}]},"public_addr":"172.21.15.96:6801/3374675329","cluster_addr":"172.21.15.96:6803/3374675329","heartbeat_back_addr":"172.21.15.96:6807/3374675329","heartbeat_front_addr":"172.21.15.96:6805/3374675329","state":["exists","up"]},{"osd":5,"uuid":"8ff7a724-0a3a-4aa2-baf1-fd52f88d476c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6808","nonce":509935370},{"type":"v1","addr":"172.21.15.96:6809","nonce":509935370}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6810","nonce":509935370},{"type":"v1","addr":"172.21.15.96:6811","nonce":509935370}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6814","nonce":509935370},{"type":"v1","addr":"172.21.15.96:6815","nonce":509935370}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.96:6812","nonce":509935370},{"type":"v1","addr":"172.21.15.96:6813","nonce":509935370}]},"public_addr":"172.21.15.96:6809/509935370","cluster_addr":"172.21.15.96:6811/509935370","heartbeat_back_addr":"172.21.15.96:6815/509935370","heartbeat_front_addr":"172.21.15.96:6813/509935370","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T09:50:08.513270+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T09:50:07.522863+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T09:50:08.134364+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T09:50:08.622366+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T09:50:08.156142+0000","dead_epoch":55},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T09:50:08.108116+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-11T09:52:57.417 INFO:tasks.ceph:Scrubbing osd.0 2024-09-11T09:52:57.417 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-09-11T09:52:57.556 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:52:57.556 INFO:teuthology.orchestra.run.smithi017.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-11T09:52:57.557 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:52:57.568 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-09-11T09:52:57.856 INFO:teuthology.orchestra.run.smithi017.stderr:instructed osd(s) 0 to deep-scrub 2024-09-11T09:52:57.869 INFO:tasks.ceph:Scrubbing osd.1 2024-09-11T09:52:57.870 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-09-11T09:52:57.994 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:52:57.994 INFO:teuthology.orchestra.run.smithi017.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-11T09:52:57.994 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:52:58.005 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-09-11T09:52:58.297 INFO:teuthology.orchestra.run.smithi017.stderr:instructed osd(s) 1 to deep-scrub 2024-09-11T09:52:58.310 INFO:tasks.ceph:Scrubbing osd.2 2024-09-11T09:52:58.311 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-09-11T09:52:58.437 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:52:58.437 INFO:teuthology.orchestra.run.smithi017.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-11T09:52:58.437 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:52:58.448 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-09-11T09:52:58.737 INFO:teuthology.orchestra.run.smithi017.stderr:instructed osd(s) 2 to deep-scrub 2024-09-11T09:52:58.750 INFO:tasks.ceph:Scrubbing osd.3 2024-09-11T09:52:58.750 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-09-11T09:52:58.876 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:52:58.876 INFO:teuthology.orchestra.run.smithi017.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-11T09:52:58.876 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:52:58.887 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-09-11T09:52:59.180 INFO:teuthology.orchestra.run.smithi017.stderr:instructed osd(s) 3 to deep-scrub 2024-09-11T09:52:59.192 INFO:tasks.ceph:Scrubbing osd.4 2024-09-11T09:52:59.193 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2024-09-11T09:52:59.318 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:52:59.318 INFO:teuthology.orchestra.run.smithi017.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-11T09:52:59.318 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:52:59.328 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2024-09-11T09:52:59.617 INFO:teuthology.orchestra.run.smithi017.stderr:instructed osd(s) 4 to deep-scrub 2024-09-11T09:52:59.629 INFO:tasks.ceph:Scrubbing osd.5 2024-09-11T09:52:59.629 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2024-09-11T09:52:59.755 INFO:teuthology.orchestra.run.smithi017.stdout:{ 2024-09-11T09:52:59.756 INFO:teuthology.orchestra.run.smithi017.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-11T09:52:59.756 INFO:teuthology.orchestra.run.smithi017.stdout:} 2024-09-11T09:52:59.766 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2024-09-11T09:53:00.055 INFO:teuthology.orchestra.run.smithi017.stderr:instructed osd(s) 5 to deep-scrub 2024-09-11T09:53:00.068 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T09:53:00.355 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-11T09:53:00.356 INFO:teuthology.orchestra.run.smithi017.stderr:dumped all 2024-09-11T09:53:00.369 INFO:teuthology.orchestra.run.smithi017.stdout:{"pg_ready":true,"pg_map":{"version":198,"stamp":"2024-09-11T09:52:58.444113+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":7668,"num_read_kb":171458,"num_write":1172,"num_write_kb":275599,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":113,"num_bytes_recovered":71057650,"num_keys_recovered":22,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1267,"ondisk_log_size":1267,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":204980,"kb_used_data":8424,"kb_used_omap":12,"kb_used_meta":196403,"kb_avail":566026060,"statfs":{"total":579820584960,"available":579610685440,"internally_reserved":0,"allocated":8626176,"data_stored":4477276,"data_compressed":66628,"data_compressed_allocated":90112,"data_compressed_original":1163264,"omap_allocated":12403,"internal_metadata":201117581},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.436439"},"pg_stats":[{"pgid":"1.0","version":"13'117","reported_seq":421,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-11T09:52:50.186293+0000","last_change":"2024-09-11T09:52:41.980919+0000","last_active":"2024-09-11T09:52:50.186293+0000","last_peered":"2024-09-11T09:52:50.186293+0000","last_clean":"2024-09-11T09:52:50.186293+0000","last_became_active":"2024-09-11T09:52:41.980573+0000","last_became_peered":"2024-09-11T09:52:41.980573+0000","last_unstale":"2024-09-11T09:52:50.186293+0000","last_undegraded":"2024-09-11T09:52:50.186293+0000","last_fullsized":"2024-09-11T09:52:50.186293+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:50:10.457262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:50:10.457262+0000","last_clean_scrub_stamp":"2024-09-11T09:50:10.457262+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:59:08.639668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"103'48","reported_seq":392,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-11T09:52:50.014080+0000","last_change":"2024-09-11T09:52:44.000267+0000","last_active":"2024-09-11T09:52:50.014080+0000","last_peered":"2024-09-11T09:52:50.014080+0000","last_clean":"2024-09-11T09:52:50.014080+0000","last_became_active":"2024-09-11T09:52:43.990093+0000","last_became_peered":"2024-09-11T09:52:43.990093+0000","last_unstale":"2024-09-11T09:52:50.014080+0000","last_undegraded":"2024-09-11T09:52:50.014080+0000","last_fullsized":"2024-09-11T09:52:50.014080+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_clean_scrub_stamp":"2024-09-11T09:50:17.477890+0000","objects_scrubbed":0,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:57:25.381839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000328833,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":53,"num_read_kb":12312,"num_write":40,"num_write_kb":33291,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":20971520,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"103'18","reported_seq":430,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-11T09:52:50.186272+0000","last_change":"2024-09-11T09:52:31.939410+0000","last_active":"2024-09-11T09:52:50.186272+0000","last_peered":"2024-09-11T09:52:50.186272+0000","last_clean":"2024-09-11T09:52:50.186272+0000","last_became_active":"2024-09-11T09:50:18.493210+0000","last_became_peered":"2024-09-11T09:50:18.493210+0000","last_unstale":"2024-09-11T09:52:50.186272+0000","last_undegraded":"2024-09-11T09:52:50.186272+0000","last_fullsized":"2024-09-11T09:52:50.186272+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_clean_scrub_stamp":"2024-09-11T09:50:17.477890+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:47:27.009441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043766200000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":174,"num_read_kb":2862,"num_write":16,"num_write_kb":12294,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"103'304","reported_seq":2240,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-11T09:52:50.013413+0000","last_change":"2024-09-11T09:52:46.995900+0000","last_active":"2024-09-11T09:52:50.013413+0000","last_peered":"2024-09-11T09:52:50.013413+0000","last_clean":"2024-09-11T09:52:50.013413+0000","last_became_active":"2024-09-11T09:52:46.995086+0000","last_became_peered":"2024-09-11T09:52:46.995086+0000","last_unstale":"2024-09-11T09:52:50.013413+0000","last_undegraded":"2024-09-11T09:52:50.013413+0000","last_fullsized":"2024-09-11T09:52:50.013413+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_clean_scrub_stamp":"2024-09-11T09:50:17.477890+0000","objects_scrubbed":0,"log_size":304,"log_dups_size":0,"ondisk_log_size":304,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:18:59.319613+0000","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.0022869520000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2124,"num_read_kb":45623,"num_write":237,"num_write_kb":52658,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":51,"num_bytes_recovered":8388608,"num_keys_recovered":8,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"103'35","reported_seq":484,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-11T09:52:50.014091+0000","last_change":"2024-09-11T09:52:31.938934+0000","last_active":"2024-09-11T09:52:50.014091+0000","last_peered":"2024-09-11T09:52:50.014091+0000","last_clean":"2024-09-11T09:52:50.014091+0000","last_became_active":"2024-09-11T09:50:18.493165+0000","last_became_peered":"2024-09-11T09:50:18.493165+0000","last_unstale":"2024-09-11T09:52:50.014091+0000","last_undegraded":"2024-09-11T09:52:50.014091+0000","last_fullsized":"2024-09-11T09:52:50.014091+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_clean_scrub_stamp":"2024-09-11T09:50:17.477890+0000","objects_scrubbed":0,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T21:11:40.038981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000392338,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":202,"num_read_kb":8394,"num_write":27,"num_write_kb":20483,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"103'314","reported_seq":1828,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-11T09:52:50.013370+0000","last_change":"2024-09-11T09:52:42.985519+0000","last_active":"2024-09-11T09:52:50.013370+0000","last_peered":"2024-09-11T09:52:50.013370+0000","last_clean":"2024-09-11T09:52:50.013370+0000","last_became_active":"2024-09-11T09:52:42.985012+0000","last_became_peered":"2024-09-11T09:52:42.985012+0000","last_unstale":"2024-09-11T09:52:50.013370+0000","last_undegraded":"2024-09-11T09:52:50.013370+0000","last_fullsized":"2024-09-11T09:52:50.013370+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_clean_scrub_stamp":"2024-09-11T09:50:17.477890+0000","objects_scrubbed":0,"log_size":314,"log_dups_size":0,"ondisk_log_size":314,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:35:33.948046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026174,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3711,"num_read_kb":35790,"num_write":188,"num_write_kb":54320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":22031360,"num_keys_recovered":14,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"103'62","reported_seq":422,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-11T09:52:50.186313+0000","last_change":"2024-09-11T09:52:38.977632+0000","last_active":"2024-09-11T09:52:50.186313+0000","last_peered":"2024-09-11T09:52:50.186313+0000","last_clean":"2024-09-11T09:52:50.186313+0000","last_became_active":"2024-09-11T09:52:38.970665+0000","last_became_peered":"2024-09-11T09:52:38.970665+0000","last_unstale":"2024-09-11T09:52:50.186313+0000","last_undegraded":"2024-09-11T09:52:50.186313+0000","last_fullsized":"2024-09-11T09:52:50.186313+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_clean_scrub_stamp":"2024-09-11T09:50:17.477890+0000","objects_scrubbed":0,"log_size":62,"log_dups_size":0,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T14:28:24.105715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000633111,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":73,"num_read_kb":24582,"num_write":46,"num_write_kb":29188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":13110272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"103'157","reported_seq":609,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-11T09:52:50.013311+0000","last_change":"2024-09-11T09:52:42.983192+0000","last_active":"2024-09-11T09:52:50.013311+0000","last_peered":"2024-09-11T09:52:50.013311+0000","last_clean":"2024-09-11T09:52:50.013311+0000","last_became_active":"2024-09-11T09:52:42.982623+0000","last_became_peered":"2024-09-11T09:52:42.982623+0000","last_unstale":"2024-09-11T09:52:50.013311+0000","last_undegraded":"2024-09-11T09:52:50.013311+0000","last_fullsized":"2024-09-11T09:52:50.013311+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_clean_scrub_stamp":"2024-09-11T09:50:17.477890+0000","objects_scrubbed":0,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T17:46:27.402621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040558,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":244,"num_read_kb":41018,"num_write":121,"num_write_kb":66581,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"103'212","reported_seq":477,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-11T09:52:50.186301+0000","last_change":"2024-09-11T09:52:31.939671+0000","last_active":"2024-09-11T09:52:50.186301+0000","last_peered":"2024-09-11T09:52:50.186301+0000","last_clean":"2024-09-11T09:52:50.186301+0000","last_became_active":"2024-09-11T09:52:04.840466+0000","last_became_peered":"2024-09-11T09:52:04.840466+0000","last_unstale":"2024-09-11T09:52:50.186301+0000","last_undegraded":"2024-09-11T09:52:50.186301+0000","last_fullsized":"2024-09-11T09:52:50.186301+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:50:17.477890+0000","last_clean_scrub_stamp":"2024-09-11T09:50:17.477890+0000","objects_scrubbed":0,"log_size":212,"log_dups_size":0,"ondisk_log_size":212,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:52:38.106407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045445899999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":971,"num_read_kb":777,"num_write":338,"num_write_kb":4368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":4194418,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7552,"num_read_kb":171358,"num_write":1013,"num_write_kb":273183,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":105,"num_bytes_recovered":68696178,"num_keys_recovered":22,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1150,"ondisk_log_size":1150,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1771104,"data_compressed":100760,"data_compressed_allocated":135168,"data_compressed_original":1753088,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":12,"seq":51539607590,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32644,"kb_used_data":1448,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339196,"statfs":{"total":96636764160,"available":96603336704,"internally_reserved":0,"allocated":1482752,"data_stored":1139785,"data_compressed":32496,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":12,"seq":51539607589,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36784,"kb_used_data":1432,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335056,"statfs":{"total":96636764160,"available":96599097344,"internally_reserved":0,"allocated":1466368,"data_stored":1139785,"data_compressed":34132,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":11,"seq":47244640294,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32576,"kb_used_data":1388,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339264,"statfs":{"total":96636764160,"available":96603406336,"internally_reserved":0,"allocated":1421312,"data_stored":549436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":66,"seq":283467841552,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29500,"kb_used_data":1384,"kb_used_omap":4,"kb_used_meta":28091,"kb_avail":94342340,"statfs":{"total":96636764160,"available":96606556160,"internally_reserved":0,"allocated":1417216,"data_stored":549417,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":28765872},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640293,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36740,"kb_used_data":1384,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335100,"statfs":{"total":96636764160,"available":96599142400,"internally_reserved":0,"allocated":1417216,"data_stored":549417,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640293,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36736,"kb_used_data":1388,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335104,"statfs":{"total":96636764160,"available":96599146496,"internally_reserved":0,"allocated":1421312,"data_stored":549436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":34132,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":34132,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32496,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T09:53:00.372 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-09-11T09:50:10.457262+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=9, tm_min=50, tm_sec=10, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=9, tm_min=52, tm_sec=56, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T09:53:00.373 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-09-11T09:50:17.477890+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=9, tm_min=50, tm_sec=17, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=9, tm_min=52, tm_sec=56, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T09:53:00.373 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-09-11T09:50:17.477890+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=9, tm_min=50, tm_sec=17, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=9, tm_min=52, tm_sec=56, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T09:53:00.373 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2024-09-11T09:50:17.477890+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=9, tm_min=50, tm_sec=17, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=9, tm_min=52, tm_sec=56, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T09:53:00.373 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-09-11T09:50:17.477890+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=9, tm_min=50, tm_sec=17, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=9, tm_min=52, tm_sec=56, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T09:53:00.373 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2024-09-11T09:50:17.477890+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=9, tm_min=50, tm_sec=17, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=9, tm_min=52, tm_sec=56, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T09:53:00.373 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-09-11T09:50:17.477890+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=9, tm_min=50, tm_sec=17, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=9, tm_min=52, tm_sec=56, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T09:53:00.373 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-09-11T09:50:17.477890+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=9, tm_min=50, tm_sec=17, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=9, tm_min=52, tm_sec=56, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T09:53:00.374 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-09-11T09:50:17.477890+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=9, tm_min=50, tm_sec=17, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=9, tm_min=52, tm_sec=56, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T09:53:00.374 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-09-11T09:53:20.374 DEBUG:teuthology.orchestra.run.smithi017:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-11T09:53:20.677 INFO:teuthology.orchestra.run.smithi017.stdout: 2024-09-11T09:53:20.677 INFO:teuthology.orchestra.run.smithi017.stderr:dumped all 2024-09-11T09:53:20.689 INFO:teuthology.orchestra.run.smithi017.stdout:{"pg_ready":true,"pg_map":{"version":209,"stamp":"2024-09-11T09:53:20.448584+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":7668,"num_read_kb":171458,"num_write":1172,"num_write_kb":275599,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":113,"num_bytes_recovered":71057650,"num_keys_recovered":22,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1271,"ondisk_log_size":1271,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":204940,"kb_used_data":8424,"kb_used_omap":12,"kb_used_meta":196403,"kb_avail":566026100,"statfs":{"total":579820584960,"available":579610726400,"internally_reserved":0,"allocated":8626176,"data_stored":4477276,"data_compressed":66628,"data_compressed_allocated":90112,"data_compressed_original":1163264,"omap_allocated":12403,"internal_metadata":201117581},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002615"},"pg_stats":[{"pgid":"1.0","version":"119'119","reported_seq":432,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-11T09:53:03.120396+0000","last_change":"2024-09-11T09:53:03.120436+0000","last_active":"2024-09-11T09:53:03.120396+0000","last_peered":"2024-09-11T09:53:03.120396+0000","last_clean":"2024-09-11T09:53:03.120396+0000","last_became_active":"2024-09-11T09:52:41.980573+0000","last_became_peered":"2024-09-11T09:52:41.980573+0000","last_unstale":"2024-09-11T09:53:03.120396+0000","last_undegraded":"2024-09-11T09:53:03.120396+0000","last_fullsized":"2024-09-11T09:53:03.120396+0000","mapping_epoch":110,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":111,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'119","last_scrub_stamp":"2024-09-11T09:53:03.120307+0000","last_deep_scrub":"119'119","last_deep_scrub_stamp":"2024-09-11T09:53:03.120307+0000","last_clean_scrub_stamp":"2024-09-11T09:53:03.120307+0000","objects_scrubbed":2,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:53:19.004929+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"119'49","reported_seq":402,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-11T09:53:01.253221+0000","last_change":"2024-09-11T09:53:01.253284+0000","last_active":"2024-09-11T09:53:01.253221+0000","last_peered":"2024-09-11T09:53:01.253221+0000","last_clean":"2024-09-11T09:53:01.253221+0000","last_became_active":"2024-09-11T09:52:43.990093+0000","last_became_peered":"2024-09-11T09:52:43.990093+0000","last_unstale":"2024-09-11T09:53:01.253221+0000","last_undegraded":"2024-09-11T09:53:01.253221+0000","last_fullsized":"2024-09-11T09:53:01.253221+0000","mapping_epoch":112,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":113,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'49","last_scrub_stamp":"2024-09-11T09:53:01.253117+0000","last_deep_scrub":"119'49","last_deep_scrub_stamp":"2024-09-11T09:53:01.253117+0000","last_clean_scrub_stamp":"2024-09-11T09:53:01.253117+0000","objects_scrubbed":1,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:58:57.072842+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.000328833,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":53,"num_read_kb":12312,"num_write":40,"num_write_kb":33291,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":20971520,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"103'18","reported_seq":439,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-11T09:53:00.056560+0000","last_change":"2024-09-11T09:53:00.056590+0000","last_active":"2024-09-11T09:53:00.056560+0000","last_peered":"2024-09-11T09:53:00.056560+0000","last_clean":"2024-09-11T09:53:00.056560+0000","last_became_active":"2024-09-11T09:50:18.493210+0000","last_became_peered":"2024-09-11T09:50:18.493210+0000","last_unstale":"2024-09-11T09:53:00.056560+0000","last_undegraded":"2024-09-11T09:53:00.056560+0000","last_fullsized":"2024-09-11T09:53:00.056560+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"103'18","last_scrub_stamp":"2024-09-11T09:53:00.056467+0000","last_deep_scrub":"103'18","last_deep_scrub_stamp":"2024-09-11T09:53:00.056467+0000","last_clean_scrub_stamp":"2024-09-11T09:53:00.056467+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:59:18.589411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043766200000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":174,"num_read_kb":2862,"num_write":16,"num_write_kb":12294,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"103'304","reported_seq":2249,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-11T09:52:58.903710+0000","last_change":"2024-09-11T09:52:58.903731+0000","last_active":"2024-09-11T09:52:58.903710+0000","last_peered":"2024-09-11T09:52:58.903710+0000","last_clean":"2024-09-11T09:52:58.903710+0000","last_became_active":"2024-09-11T09:52:46.995086+0000","last_became_peered":"2024-09-11T09:52:46.995086+0000","last_unstale":"2024-09-11T09:52:58.903710+0000","last_undegraded":"2024-09-11T09:52:58.903710+0000","last_fullsized":"2024-09-11T09:52:58.903710+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"103'304","last_scrub_stamp":"2024-09-11T09:52:58.903628+0000","last_deep_scrub":"103'304","last_deep_scrub_stamp":"2024-09-11T09:52:58.903628+0000","last_clean_scrub_stamp":"2024-09-11T09:52:58.903628+0000","objects_scrubbed":0,"log_size":304,"log_dups_size":0,"ondisk_log_size":304,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:15:40.726595+0000","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.0022869520000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2124,"num_read_kb":45623,"num_write":237,"num_write_kb":52658,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":51,"num_bytes_recovered":8388608,"num_keys_recovered":8,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"119'36","reported_seq":494,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-11T09:53:00.230032+0000","last_change":"2024-09-11T09:53:00.230065+0000","last_active":"2024-09-11T09:53:00.230032+0000","last_peered":"2024-09-11T09:53:00.230032+0000","last_clean":"2024-09-11T09:53:00.230032+0000","last_became_active":"2024-09-11T09:50:18.493165+0000","last_became_peered":"2024-09-11T09:50:18.493165+0000","last_unstale":"2024-09-11T09:53:00.230032+0000","last_undegraded":"2024-09-11T09:53:00.230032+0000","last_fullsized":"2024-09-11T09:53:00.230032+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"119'36","last_scrub_stamp":"2024-09-11T09:53:00.229934+0000","last_deep_scrub":"119'36","last_deep_scrub_stamp":"2024-09-11T09:53:00.229934+0000","last_clean_scrub_stamp":"2024-09-11T09:53:00.229934+0000","objects_scrubbed":1,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:17:02.063657+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.000392338,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":202,"num_read_kb":8394,"num_write":27,"num_write_kb":20483,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"103'314","reported_seq":1837,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-11T09:52:59.375630+0000","last_change":"2024-09-11T09:52:59.375679+0000","last_active":"2024-09-11T09:52:59.375630+0000","last_peered":"2024-09-11T09:52:59.375630+0000","last_clean":"2024-09-11T09:52:59.375630+0000","last_became_active":"2024-09-11T09:52:42.985012+0000","last_became_peered":"2024-09-11T09:52:42.985012+0000","last_unstale":"2024-09-11T09:52:59.375630+0000","last_undegraded":"2024-09-11T09:52:59.375630+0000","last_fullsized":"2024-09-11T09:52:59.375630+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"103'314","last_scrub_stamp":"2024-09-11T09:52:59.375528+0000","last_deep_scrub":"103'314","last_deep_scrub_stamp":"2024-09-11T09:52:59.375528+0000","last_clean_scrub_stamp":"2024-09-11T09:52:59.375528+0000","objects_scrubbed":2,"log_size":314,"log_dups_size":0,"ondisk_log_size":314,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T20:49:52.093776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026174,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3711,"num_read_kb":35790,"num_write":188,"num_write_kb":54320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":22031360,"num_keys_recovered":14,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"103'62","reported_seq":431,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-11T09:53:02.151312+0000","last_change":"2024-09-11T09:53:02.151341+0000","last_active":"2024-09-11T09:53:02.151312+0000","last_peered":"2024-09-11T09:53:02.151312+0000","last_clean":"2024-09-11T09:53:02.151312+0000","last_became_active":"2024-09-11T09:52:38.970665+0000","last_became_peered":"2024-09-11T09:52:38.970665+0000","last_unstale":"2024-09-11T09:53:02.151312+0000","last_undegraded":"2024-09-11T09:53:02.151312+0000","last_fullsized":"2024-09-11T09:53:02.151312+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"103'62","last_scrub_stamp":"2024-09-11T09:53:02.151180+0000","last_deep_scrub":"103'62","last_deep_scrub_stamp":"2024-09-11T09:53:02.151180+0000","last_clean_scrub_stamp":"2024-09-11T09:53:02.151180+0000","objects_scrubbed":0,"log_size":62,"log_dups_size":0,"ondisk_log_size":62,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T19:19:00.498789+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.000633111,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":73,"num_read_kb":24582,"num_write":46,"num_write_kb":29188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":13110272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"103'157","reported_seq":618,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-11T09:52:58.397159+0000","last_change":"2024-09-11T09:52:58.397189+0000","last_active":"2024-09-11T09:52:58.397159+0000","last_peered":"2024-09-11T09:52:58.397159+0000","last_clean":"2024-09-11T09:52:58.397159+0000","last_became_active":"2024-09-11T09:52:42.982623+0000","last_became_peered":"2024-09-11T09:52:42.982623+0000","last_unstale":"2024-09-11T09:52:58.397159+0000","last_undegraded":"2024-09-11T09:52:58.397159+0000","last_fullsized":"2024-09-11T09:52:58.397159+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"103'157","last_scrub_stamp":"2024-09-11T09:52:58.397054+0000","last_deep_scrub":"103'157","last_deep_scrub_stamp":"2024-09-11T09:52:58.397054+0000","last_clean_scrub_stamp":"2024-09-11T09:52:58.397054+0000","objects_scrubbed":0,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T14:11:00.900712+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00040558,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":244,"num_read_kb":41018,"num_write":121,"num_write_kb":66581,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"103'212","reported_seq":486,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-09-11T09:53:01.105018+0000","last_change":"2024-09-11T09:53:01.105046+0000","last_active":"2024-09-11T09:53:01.105018+0000","last_peered":"2024-09-11T09:53:01.105018+0000","last_clean":"2024-09-11T09:53:01.105018+0000","last_became_active":"2024-09-11T09:52:04.840466+0000","last_became_peered":"2024-09-11T09:52:04.840466+0000","last_unstale":"2024-09-11T09:53:01.105018+0000","last_undegraded":"2024-09-11T09:53:01.105018+0000","last_fullsized":"2024-09-11T09:53:01.105018+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"103'212","last_scrub_stamp":"2024-09-11T09:53:01.104898+0000","last_deep_scrub":"103'212","last_deep_scrub_stamp":"2024-09-11T09:53:01.104898+0000","last_clean_scrub_stamp":"2024-09-11T09:53:01.104898+0000","objects_scrubbed":0,"log_size":212,"log_dups_size":0,"ondisk_log_size":212,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T20:08:23.824802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045445899999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":971,"num_read_kb":777,"num_write":338,"num_write_kb":4368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":4194418,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":7552,"num_read_kb":171358,"num_write":1013,"num_write_kb":273183,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":105,"num_bytes_recovered":68696178,"num_keys_recovered":22,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1152,"ondisk_log_size":1152,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":1771104,"data_compressed":100760,"data_compressed_allocated":135168,"data_compressed_original":1753088,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":3}],"osd_stats":[{"osd":3,"up_from":12,"seq":51539607594,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32632,"kb_used_data":1448,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339208,"statfs":{"total":96636764160,"available":96603348992,"internally_reserved":0,"allocated":1482752,"data_stored":1139785,"data_compressed":32496,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":12,"seq":51539607593,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36776,"kb_used_data":1432,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335064,"statfs":{"total":96636764160,"available":96599105536,"internally_reserved":0,"allocated":1466368,"data_stored":1139785,"data_compressed":34132,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":11,"seq":47244640299,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32572,"kb_used_data":1388,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339268,"statfs":{"total":96636764160,"available":96603410432,"internally_reserved":0,"allocated":1421312,"data_stored":549436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":66,"seq":283467841557,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29500,"kb_used_data":1384,"kb_used_omap":4,"kb_used_meta":28091,"kb_avail":94342340,"statfs":{"total":96636764160,"available":96606556160,"internally_reserved":0,"allocated":1417216,"data_stored":549417,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4432,"internal_metadata":28765872},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":11,"seq":47244640298,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36728,"kb_used_data":1384,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335112,"statfs":{"total":96636764160,"available":96599154688,"internally_reserved":0,"allocated":1417216,"data_stored":549417,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640298,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36732,"kb_used_data":1388,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335108,"statfs":{"total":96636764160,"available":96599150592,"internally_reserved":0,"allocated":1421312,"data_stored":549436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":34132,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":34132,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":32496,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T09:53:20.691 DEBUG:teuthology.orchestra.run.smithi017:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-09-11T09:53:20.995 INFO:teuthology.misc:Shutting down mds daemons... 2024-09-11T09:53:20.995 INFO:teuthology.misc:Shutting down osd daemons... 2024-09-11T09:53:20.995 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-09-11T09:53:20.996 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T09:53:21.032 INFO:tasks.ceph.osd.0:Stopped 2024-09-11T09:53:21.032 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-09-11T09:53:21.033 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T09:53:21.098 INFO:tasks.ceph.osd.1:Stopped 2024-09-11T09:53:21.098 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-09-11T09:53:21.098 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T09:53:21.167 INFO:tasks.ceph.osd.2:Stopped 2024-09-11T09:53:21.168 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-09-11T09:53:21.168 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T09:53:21.203 INFO:tasks.ceph.osd.3:Stopped 2024-09-11T09:53:21.203 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-09-11T09:53:21.203 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T09:53:21.274 INFO:tasks.ceph.osd.4:Stopped 2024-09-11T09:53:21.275 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-09-11T09:53:21.275 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T09:53:21.348 INFO:tasks.ceph.osd.5:Stopped 2024-09-11T09:53:21.349 INFO:teuthology.misc:Shutting down mgr daemons... 2024-09-11T09:53:21.349 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-09-11T09:53:21.349 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T09:53:21.419 INFO:tasks.ceph.mgr.x:Stopped 2024-09-11T09:53:21.419 INFO:teuthology.misc:Shutting down mon daemons... 2024-09-11T09:53:21.420 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-11T09:53:21.420 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T09:53:21.431 INFO:tasks.ceph.mon.a:Stopped 2024-09-11T09:53:21.431 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-11T09:53:21.431 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T09:53:21.482 INFO:tasks.ceph.mon.c:Stopped 2024-09-11T09:53:21.483 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-11T09:53:21.483 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T09:53:21.495 INFO:tasks.ceph.mon.b:Stopped 2024-09-11T09:53:21.496 INFO:tasks.ceph:Checking cluster log for badness... 2024-09-11T09:53:21.496 DEBUG:teuthology.orchestra.run.smithi017:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'timeout on replica' | egrep -v 'late reservation from' | egrep -v MON_DOWN | egrep -v OSDMAP_FLAGS | egrep -v OSD_DOWN | egrep -v PG_DEGRADED | egrep -v PG_AVAILABILITY | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'mons down' | egrep -v 'mon down' | egrep -v 'out of quorum' | egrep -v noscrub | egrep -v nodeep-scrub | head -n 1 2024-09-11T09:53:21.563 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi017.front.sepia.ceph.com 2024-09-11T09:53:21.563 DEBUG:teuthology.orchestra.run.smithi017:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-09-11T09:53:21.732 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi017.front.sepia.ceph.com 2024-09-11T09:53:21.732 DEBUG:teuthology.orchestra.run.smithi017:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-09-11T09:53:21.819 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi017.front.sepia.ceph.com 2024-09-11T09:53:21.820 DEBUG:teuthology.orchestra.run.smithi017:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-09-11T09:53:21.869 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi096.front.sepia.ceph.com 2024-09-11T09:53:21.870 DEBUG:teuthology.orchestra.run.smithi096:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-09-11T09:53:21.982 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi096.front.sepia.ceph.com 2024-09-11T09:53:21.982 DEBUG:teuthology.orchestra.run.smithi096:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-09-11T09:53:22.078 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi096.front.sepia.ceph.com 2024-09-11T09:53:22.079 DEBUG:teuthology.orchestra.run.smithi096:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-09-11T09:53:22.138 INFO:tasks.ceph:Archiving mon data... 2024-09-11T09:53:22.139 DEBUG:teuthology.misc:Transferring archived files from smithi017:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/teuthology-2024-09-10_21:16:03-rbd-squid-distro-default-smithi/7900734/data/mon.a.tgz 2024-09-11T09:53:22.139 DEBUG:teuthology.orchestra.run.smithi017:> mktemp 2024-09-11T09:53:22.156 INFO:teuthology.orchestra.run.smithi017.stdout:/tmp/tmp.MtS23PKpz5 2024-09-11T09:53:22.156 DEBUG:teuthology.orchestra.run.smithi017:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.MtS23PKpz5 2024-09-11T09:53:22.292 DEBUG:teuthology.orchestra.run.smithi017:> sudo chmod 0666 /tmp/tmp.MtS23PKpz5 2024-09-11T09:53:22.376 DEBUG:teuthology.orchestra.remote:smithi017:/tmp/tmp.MtS23PKpz5 is 279KB 2024-09-11T09:53:22.472 DEBUG:teuthology.orchestra.run.smithi017:> rm -fr /tmp/tmp.MtS23PKpz5 2024-09-11T09:53:22.487 DEBUG:teuthology.misc:Transferring archived files from smithi017:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/teuthology-2024-09-10_21:16:03-rbd-squid-distro-default-smithi/7900734/data/mon.c.tgz 2024-09-11T09:53:22.488 DEBUG:teuthology.orchestra.run.smithi017:> mktemp 2024-09-11T09:53:22.543 INFO:teuthology.orchestra.run.smithi017.stdout:/tmp/tmp.zjwGSWtXCG 2024-09-11T09:53:22.543 DEBUG:teuthology.orchestra.run.smithi017:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.zjwGSWtXCG 2024-09-11T09:53:22.687 DEBUG:teuthology.orchestra.run.smithi017:> sudo chmod 0666 /tmp/tmp.zjwGSWtXCG 2024-09-11T09:53:22.772 DEBUG:teuthology.orchestra.remote:smithi017:/tmp/tmp.zjwGSWtXCG is 378KB 2024-09-11T09:53:22.840 DEBUG:teuthology.orchestra.run.smithi017:> rm -fr /tmp/tmp.zjwGSWtXCG 2024-09-11T09:53:22.855 DEBUG:teuthology.misc:Transferring archived files from smithi096:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/teuthology-2024-09-10_21:16:03-rbd-squid-distro-default-smithi/7900734/data/mon.b.tgz 2024-09-11T09:53:22.855 DEBUG:teuthology.orchestra.run.smithi096:> mktemp 2024-09-11T09:53:22.874 INFO:teuthology.orchestra.run.smithi096.stdout:/tmp/tmp.jJY3lx7QnY 2024-09-11T09:53:22.874 DEBUG:teuthology.orchestra.run.smithi096:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.jJY3lx7QnY 2024-09-11T09:53:23.051 DEBUG:teuthology.orchestra.run.smithi096:> sudo chmod 0666 /tmp/tmp.jJY3lx7QnY 2024-09-11T09:53:23.136 DEBUG:teuthology.orchestra.remote:smithi096:/tmp/tmp.jJY3lx7QnY is 378KB 2024-09-11T09:53:23.204 DEBUG:teuthology.orchestra.run.smithi096:> rm -fr /tmp/tmp.jJY3lx7QnY 2024-09-11T09:53:23.220 INFO:tasks.ceph:Cleaning ceph cluster... 2024-09-11T09:53:23.220 DEBUG:teuthology.orchestra.run.smithi017:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-11T09:53:23.222 DEBUG:teuthology.orchestra.run.smithi096:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-11T09:53:23.261 DEBUG:teuthology.orchestra.run.smithi107:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-11T09:53:23.309 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-09-11T09:53:23.339 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-09-11T09:53:23.357 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-09-11T09:53:23.358 INFO:tasks.ceph:Archiving crash dumps... 2024-09-11T09:53:23.359 DEBUG:teuthology.misc:Transferring archived files from smithi017:/var/lib/ceph/crash to /home/teuthworker/archive/teuthology-2024-09-10_21:16:03-rbd-squid-distro-default-smithi/7900734/remote/smithi017/crash 2024-09-11T09:53:23.360 DEBUG:teuthology.orchestra.run.smithi017:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-11T09:53:23.386 DEBUG:teuthology.misc:Transferring archived files from smithi096:/var/lib/ceph/crash to /home/teuthworker/archive/teuthology-2024-09-10_21:16:03-rbd-squid-distro-default-smithi/7900734/remote/smithi096/crash 2024-09-11T09:53:23.387 DEBUG:teuthology.orchestra.run.smithi096:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-11T09:53:23.411 DEBUG:teuthology.misc:Transferring archived files from smithi107:/var/lib/ceph/crash to /home/teuthworker/archive/teuthology-2024-09-10_21:16:03-rbd-squid-distro-default-smithi/7900734/remote/smithi107/crash 2024-09-11T09:53:23.412 DEBUG:teuthology.orchestra.run.smithi107:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-11T09:53:23.480 INFO:tasks.ceph:Compressing logs... 2024-09-11T09:53:23.480 DEBUG:teuthology.orchestra.run.smithi017:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-11T09:53:23.483 DEBUG:teuthology.orchestra.run.smithi096:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-11T09:53:23.486 DEBUG:teuthology.orchestra.run.smithi107:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-11T09:53:23.506 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-09-11T09:53:23.506 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52249.log 2024-09-11T09:53:23.524 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/objectstore_tool.39553.log 2024-09-11T09:53:23.524 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31828.log 2024-09-11T09:53:23.524 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose/var/log/ceph/objectstore_tool.39553.log: -- /var/log/ceph/ceph-client.admin.31668.log 2024-09-11T09:53:23.524 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.31828.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-09-11T09:53:23.524 INFO:teuthology.orchestra.run.smithi096.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31828.log.gz 2024-09-11T09:53:23.524 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.31668.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31668.log.gz 2024-09-11T09:53:23.524 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-09-11T09:53:23.524 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/objectstore_tool.38673.log 2024-09-11T09:53:23.524 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31788.log 2024-09-11T09:53:23.524 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/objectstore_tool.38673.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31628.log 2024-09-11T09:53:23.524 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.31788.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31708.log 2024-09-11T09:53:23.525 INFO:teuthology.orchestra.run.smithi096.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31788.log.gz 2024-09-11T09:53:23.525 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.31628.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-09-11T09:53:23.525 INFO:teuthology.orchestra.run.smithi096.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31628.log.gz 2024-09-11T09:53:23.525 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.31708.log: gzip -5 --verbose -- 89.0% /var/log/ceph/ceph.log 2024-09-11T09:53:23.525 INFO:teuthology.orchestra.run.smithi096.stderr: -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-09-11T09:53:23.525 INFO:teuthology.orchestra.run.smithi096.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31708.log.gz 2024-09-11T09:53:23.525 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-09-11T09:53:23.525 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/objectstore_tool.40433.log 2024-09-11T09:53:23.525 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-09-11T09:53:23.525 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/objectstore_tool.40433.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31748.log 2024-09-11T09:53:23.525 INFO:teuthology.orchestra.run.smithi096.stderr: 85.0% -- replaced with /var/log/ceph/ceph.log.gz 2024-09-11T09:53:23.525 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-mon.b.log: gzip -5 --verbose -- /var/log/ceph/objectstore_tool.41313.log 2024-09-11T09:53:23.525 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-client.admin.31748.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31748.log.gz 2024-09-11T09:53:23.525 INFO:teuthology.orchestra.run.smithi096.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-09-11T09:53:23.526 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/objectstore_tool.41313.log: 95.1% -- replaced with /var/log/ceph/objectstore_tool.39553.log.gz 2024-09-11T09:53:23.526 INFO:teuthology.orchestra.run.smithi096.stderr:/var/log/ceph/ceph-osd.3.log: 95.1% -- replaced with /var/log/ceph/objectstore_tool.38673.log.gz 2024-09-11T09:53:23.526 INFO:teuthology.orchestra.run.smithi096.stderr: 95.1% -- replaced with /var/log/ceph/objectstore_tool.40433.log.gz 2024-09-11T09:53:23.526 INFO:teuthology.orchestra.run.smithi096.stderr: 95.1% -- replaced with /var/log/ceph/objectstore_tool.41313.log.gz 2024-09-11T09:53:23.586 INFO:teuthology.orchestra.run.smithi096.stderr: 91.5% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-09-11T09:53:23.643 INFO:teuthology.orchestra.run.smithi096.stderr: 93.2% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-09-11T09:53:23.655 INFO:teuthology.orchestra.run.smithi096.stderr: 91.1% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-09-11T09:53:23.681 INFO:teuthology.orchestra.run.smithi096.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-09-11T09:53:23.703 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29700.log 2024-09-11T09:53:23.703 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29394.log 2024-09-11T09:53:23.704 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.29700.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29358.log 2024-09-11T09:53:23.704 INFO:teuthology.orchestra.run.smithi107.stderr: 57.8% -- replaced with /var/log/ceph/ceph-client.admin.29700.log.gz 2024-09-11T09:53:23.704 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.29394.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29767.log 2024-09-11T09:53:23.704 INFO:teuthology.orchestra.run.smithi107.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29394.log.gz 2024-09-11T09:53:23.704 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.29358.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28323.log 2024-09-11T09:53:23.704 INFO:teuthology.orchestra.run.smithi107.stderr: 18.8% -- replaced with /var/log/ceph/ceph-client.admin.29358.log.gz 2024-09-11T09:53:23.704 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.29767.log: gzip -5 --verbose 0.0% -- /var/log/ceph/ceph-client.admin.28529.log -- replaced with /var/log/ceph/ceph-client.admin.29767.log.gz 2024-09-11T09:53:23.704 INFO:teuthology.orchestra.run.smithi107.stderr: 2024-09-11T09:53:23.704 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.28323.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30691.log 2024-09-11T09:53:23.704 INFO:teuthology.orchestra.run.smithi107.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28323.log.gz 2024-09-11T09:53:23.704 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.28529.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29367.log 2024-09-11T09:53:23.704 INFO:teuthology.orchestra.run.smithi107.stderr: 18.8% -- replaced with /var/log/ceph/ceph-client.admin.28529.log.gz 2024-09-11T09:53:23.705 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29689.log 2024-09-11T09:53:23.705 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.30691.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30691.log.gz 2024-09-11T09:53:23.705 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27498.log 2024-09-11T09:53:23.705 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.29367.log: 62.4% -- replaced with /var/log/ceph/ceph-client.admin.29367.log.gz 2024-09-11T09:53:23.705 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.29689.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28850.log 2024-09-11T09:53:23.705 INFO:teuthology.orchestra.run.smithi107.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29689.log.gz 2024-09-11T09:53:23.705 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.27498.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30125.log 2024-09-11T09:53:23.705 INFO:teuthology.orchestra.run.smithi107.stderr: 20.0% -- replaced with /var/log/ceph/ceph-client.admin.27498.log.gz 2024-09-11T09:53:23.705 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.28850.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28782.log 2024-09-11T09:53:23.705 INFO:teuthology.orchestra.run.smithi107.stderr: 18.2% -- replaced with /var/log/ceph/ceph-client.admin.28850.log.gz 2024-09-11T09:53:23.705 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.30125.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27449.log 2024-09-11T09:53:23.705 INFO:teuthology.orchestra.run.smithi107.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30125.log.gz 2024-09-11T09:53:23.705 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.28782.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30036.log 2024-09-11T09:53:23.706 INFO:teuthology.orchestra.run.smithi107.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28782.log.gz 2024-09-11T09:53:23.706 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.27449.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29137.log 2024-09-11T09:53:23.706 INFO:teuthology.orchestra.run.smithi107.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27449.log.gz 2024-09-11T09:53:23.706 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.30036.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29759.log 2024-09-11T09:53:23.706 INFO:teuthology.orchestra.run.smithi107.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30036.log.gz 2024-09-11T09:53:23.706 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.29137.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30492.log 2024-09-11T09:53:23.706 INFO:teuthology.orchestra.run.smithi107.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29137.log.gz 2024-09-11T09:53:23.706 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.29759.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29726.log 2024-09-11T09:53:23.706 INFO:teuthology.orchestra.run.smithi107.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29759.log.gz 2024-09-11T09:53:23.706 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.30492.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27928.log 2024-09-11T09:53:23.706 INFO:teuthology.orchestra.run.smithi107.stderr: 3.6% -- replaced with /var/log/ceph/ceph-client.admin.30492.log.gz 2024-09-11T09:53:23.706 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.29726.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29866.log 2024-09-11T09:53:23.706 INFO:teuthology.orchestra.run.smithi107.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29726.log.gz 2024-09-11T09:53:23.707 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.27928.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27174.log 2024-09-11T09:53:23.707 INFO:teuthology.orchestra.run.smithi107.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27928.log.gz 2024-09-11T09:53:23.707 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.29866.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29745.log 2024-09-11T09:53:23.707 INFO:teuthology.orchestra.run.smithi107.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29866.log.gz 2024-09-11T09:53:23.707 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.27174.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28899.log 2024-09-11T09:53:23.707 INFO:teuthology.orchestra.run.smithi107.stderr: 24.0% -- replaced with /var/log/ceph/ceph-client.admin.27174.log.gz 2024-09-11T09:53:23.707 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.29745.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28472.log 2024-09-11T09:53:23.707 INFO:teuthology.orchestra.run.smithi107.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29745.log.gz 2024-09-11T09:53:23.707 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.28899.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30810.log 2024-09-11T09:53:23.707 INFO:teuthology.orchestra.run.smithi107.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28899.log.gz 2024-09-11T09:53:23.708 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.28472.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27264.log 2024-09-11T09:53:23.708 INFO:teuthology.orchestra.run.smithi107.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28472.log.gz 2024-09-11T09:53:23.708 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.30810.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30693.log 2024-09-11T09:53:23.708 INFO:teuthology.orchestra.run.smithi107.stderr: 19.4% -- replaced with /var/log/ceph/ceph-client.admin.30810.log.gz 2024-09-11T09:53:23.708 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.27264.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27218.log 2024-09-11T09:53:23.708 INFO:teuthology.orchestra.run.smithi107.stderr: 25.7% -- replaced with /var/log/ceph/ceph-client.admin.27264.log.gz 2024-09-11T09:53:23.708 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.30693.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28833.log 2024-09-11T09:53:23.708 INFO:teuthology.orchestra.run.smithi107.stderr: 20.0% -- replaced with /var/log/ceph/ceph-client.admin.30693.log.gz 2024-09-11T09:53:23.708 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.27218.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27833.log 2024-09-11T09:53:23.708 INFO:teuthology.orchestra.run.smithi107.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27218.log.gz 2024-09-11T09:53:23.708 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.28833.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28527.log 2024-09-11T09:53:23.708 INFO:teuthology.orchestra.run.smithi107.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28833.log.gz 2024-09-11T09:53:23.709 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.27833.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28032.log 2024-09-11T09:53:23.709 INFO:teuthology.orchestra.run.smithi107.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27833.log.gz 2024-09-11T09:53:23.709 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.28527.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.28527.log.gz 2024-09-11T09:53:23.709 INFO:teuthology.orchestra.run.smithi107.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.27997.log 2024-09-11T09:53:23.709 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.28032.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29229.log 2024-09-11T09:53:23.709 INFO:teuthology.orchestra.run.smithi107.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28032.log.gz 2024-09-11T09:53:23.709 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.27997.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28593.log 2024-09-11T09:53:23.709 INFO:teuthology.orchestra.run.smithi107.stderr: 19.4% -- replaced with /var/log/ceph/ceph-client.admin.27997.log.gz 2024-09-11T09:53:23.709 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.29229.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29229.log.gz 2024-09-11T09:53:23.709 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30071.log 2024-09-11T09:53:23.709 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.28593.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27656.log 2024-09-11T09:53:23.709 INFO:teuthology.orchestra.run.smithi107.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28593.log.gz 2024-09-11T09:53:23.709 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.30071.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27673.log 2024-09-11T09:53:23.709 INFO:teuthology.orchestra.run.smithi107.stderr: 20.0% -- replaced with /var/log/ceph/ceph-client.admin.30071.log.gz 2024-09-11T09:53:23.710 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.27656.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27656.log.gz 2024-09-11T09:53:23.710 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27766.log 2024-09-11T09:53:23.710 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.27673.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27593.log 2024-09-11T09:53:23.710 INFO:teuthology.orchestra.run.smithi107.stderr: 19.4% -- replaced with /var/log/ceph/ceph-client.admin.27673.log.gz 2024-09-11T09:53:23.710 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.27766.log: gzip -5 --verbose 0.0% -- -- replaced with /var/log/ceph/ceph-client.admin.27766.log.gz /var/log/ceph/ceph-client.admin.29819.log 2024-09-11T09:53:23.710 INFO:teuthology.orchestra.run.smithi107.stderr: 2024-09-11T09:53:23.710 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.27593.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29073.log 2024-09-11T09:53:23.710 INFO:teuthology.orchestra.run.smithi107.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27593.log.gz 2024-09-11T09:53:23.710 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.29819.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28859.log 2024-09-11T09:53:23.710 INFO:teuthology.orchestra.run.smithi107.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29819.log.gz 2024-09-11T09:53:23.711 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.29073.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27671.log 2024-09-11T09:53:23.711 INFO:teuthology.orchestra.run.smithi107.stderr: 24.1% -- replaced with /var/log/ceph/ceph-client.admin.29073.log.gz 2024-09-11T09:53:23.711 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.28859.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30425.log 2024-09-11T09:53:23.711 INFO:teuthology.orchestra.run.smithi107.stderr: 34.7% -- replaced with /var/log/ceph/ceph-client.admin.28859.log.gz 2024-09-11T09:53:23.711 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.27671.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28424.log 2024-09-11T09:53:23.711 INFO:teuthology.orchestra.run.smithi107.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27671.log.gz 2024-09-11T09:53:23.711 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.30425.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28389.log 2024-09-11T09:53:23.711 INFO:teuthology.orchestra.run.smithi107.stderr: 4.5% -- replaced with /var/log/ceph/ceph-client.admin.30425.log.gz 2024-09-11T09:53:23.711 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.28424.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28284.log 2024-09-11T09:53:23.711 INFO:teuthology.orchestra.run.smithi107.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28424.log.gz 2024-09-11T09:53:23.711 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.28389.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30971.log 2024-09-11T09:53:23.711 INFO:teuthology.orchestra.run.smithi107.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28389.log.gz 2024-09-11T09:53:23.711 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.28284.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28284.log.gz 2024-09-11T09:53:23.711 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27435.log 2024-09-11T09:53:23.712 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.30971.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29292.log 2024-09-11T09:53:23.712 INFO:teuthology.orchestra.run.smithi107.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30971.log.gz 2024-09-11T09:53:23.712 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.27435.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28162.log 2024-09-11T09:53:23.712 INFO:teuthology.orchestra.run.smithi107.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27435.log.gz 2024-09-11T09:53:23.712 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.29292.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30791.log 2024-09-11T09:53:23.712 INFO:teuthology.orchestra.run.smithi107.stderr: 19.4% -- replaced with /var/log/ceph/ceph-client.admin.29292.log.gz 2024-09-11T09:53:23.712 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.28162.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30862.log 2024-09-11T09:53:23.712 INFO:teuthology.orchestra.run.smithi107.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28162.log.gz 2024-09-11T09:53:23.712 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.30791.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30210.log 2024-09-11T09:53:23.712 INFO:teuthology.orchestra.run.smithi107.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30791.log.gz 2024-09-11T09:53:23.712 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.30862.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27844.log 2024-09-11T09:53:23.712 INFO:teuthology.orchestra.run.smithi107.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30862.log.gz 2024-09-11T09:53:23.712 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.30210.log: gzip -5 0.0% --verbose -- replaced with /var/log/ceph/ceph-client.admin.30210.log.gz -- 2024-09-11T09:53:23.713 INFO:teuthology.orchestra.run.smithi107.stderr: /var/log/ceph/ceph-client.admin.30644.log 2024-09-11T09:53:23.713 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.27844.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27154.log 2024-09-11T09:53:23.713 INFO:teuthology.orchestra.run.smithi107.stderr: 52.5% -- replaced with /var/log/ceph/ceph-client.admin.27844.log.gz 2024-09-11T09:53:23.713 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.30644.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.30644.log.gz 2024-09-11T09:53:23.713 INFO:teuthology.orchestra.run.smithi107.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.29110.log 2024-09-11T09:53:23.713 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.27154.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30392.log 2024-09-11T09:53:23.713 INFO:teuthology.orchestra.run.smithi107.stderr: 25.0% -- replaced with /var/log/ceph/ceph-client.admin.27154.log.gz 2024-09-11T09:53:23.713 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.29110.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28718.log 2024-09-11T09:53:23.713 INFO:teuthology.orchestra.run.smithi107.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29110.log.gz 2024-09-11T09:53:23.713 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.30392.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28039.log 2024-09-11T09:53:23.713 INFO:teuthology.orchestra.run.smithi107.stderr: 4.5% -- replaced with /var/log/ceph/ceph-client.admin.30392.log.gz 2024-09-11T09:53:23.713 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.28718.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29154.log 2024-09-11T09:53:23.713 INFO:teuthology.orchestra.run.smithi107.stderr: 19.4% -- replaced with /var/log/ceph/ceph-client.admin.28718.log.gz 2024-09-11T09:53:23.713 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.28039.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30358.log 2024-09-11T09:53:23.714 INFO:teuthology.orchestra.run.smithi107.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28039.log.gz 2024-09-11T09:53:23.714 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.29154.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31095.log 2024-09-11T09:53:23.714 INFO:teuthology.orchestra.run.smithi107.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29154.log.gz 2024-09-11T09:53:23.714 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.30358.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29815.log 2024-09-11T09:53:23.714 INFO:teuthology.orchestra.run.smithi107.stderr: 4.5% -- replaced with /var/log/ceph/ceph-client.admin.30358.log.gz 2024-09-11T09:53:23.714 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.31095.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30117.log 2024-09-11T09:53:23.714 INFO:teuthology.orchestra.run.smithi107.stderr: 54.4% -- replaced with /var/log/ceph/ceph-client.admin.31095.log.gz 2024-09-11T09:53:23.714 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.29815.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30808.log 2024-09-11T09:53:23.714 INFO:teuthology.orchestra.run.smithi107.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29815.log.gz 2024-09-11T09:53:23.714 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.30117.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28325.log 2024-09-11T09:53:23.714 INFO:teuthology.orchestra.run.smithi107.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30117.log.gz 2024-09-11T09:53:23.714 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.30808.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30788.log 2024-09-11T09:53:23.714 INFO:teuthology.orchestra.run.smithi107.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30808.log.gz 2024-09-11T09:53:23.714 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.28325.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28216.log 2024-09-11T09:53:23.715 INFO:teuthology.orchestra.run.smithi107.stderr: 19.4% -- replaced with /var/log/ceph/ceph-client.admin.28325.log.gz 2024-09-11T09:53:23.715 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.30788.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27137.log 2024-09-11T09:53:23.715 INFO:teuthology.orchestra.run.smithi107.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30788.log.gz 2024-09-11T09:53:23.715 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.28216.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27255.log 2024-09-11T09:53:23.715 INFO:teuthology.orchestra.run.smithi107.stderr: 54.0% -- replaced with /var/log/ceph/ceph-client.admin.28216.log.gz 2024-09-11T09:53:23.715 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.27137.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30888.log 2024-09-11T09:53:23.715 INFO:teuthology.orchestra.run.smithi107.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27137.log.gz 2024-09-11T09:53:23.715 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.27255.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30728.log 2024-09-11T09:53:23.715 INFO:teuthology.orchestra.run.smithi107.stderr: 19.4% -- replaced with /var/log/ceph/ceph-client.admin.27255.log.gz 2024-09-11T09:53:23.715 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.30888.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30069.log 2024-09-11T09:53:23.715 INFO:teuthology.orchestra.run.smithi107.stderr: 25.3% -- replaced with /var/log/ceph/ceph-client.admin.30888.log.gz 2024-09-11T09:53:23.715 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.30728.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30106.log 2024-09-11T09:53:23.715 INFO:teuthology.orchestra.run.smithi107.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30728.log.gz 2024-09-11T09:53:23.715 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.30069.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29029.log 2024-09-11T09:53:23.716 INFO:teuthology.orchestra.run.smithi107.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30069.log.gz 2024-09-11T09:53:23.716 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.30106.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30106.log.gz 2024-09-11T09:53:23.716 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30593.log 2024-09-11T09:53:23.716 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.29029.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28896.log 2024-09-11T09:53:23.716 INFO:teuthology.orchestra.run.smithi107.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29029.log.gz 2024-09-11T09:53:23.716 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.30593.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30183.log 2024-09-11T09:53:23.716 INFO:teuthology.orchestra.run.smithi107.stderr: 5.5% -- replaced with /var/log/ceph/ceph-client.admin.30593.log.gz 2024-09-11T09:53:23.716 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.28896.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29327.log 2024-09-11T09:53:23.716 INFO:teuthology.orchestra.run.smithi107.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28896.log.gz 2024-09-11T09:53:23.716 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.30183.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27714.log 2024-09-11T09:53:23.716 INFO:teuthology.orchestra.run.smithi107.stderr: 56.6% -- replaced with /var/log/ceph/ceph-client.admin.30183.log.gz 2024-09-11T09:53:23.716 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.29327.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30879.log 2024-09-11T09:53:23.716 INFO:teuthology.orchestra.run.smithi107.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29327.log.gz 2024-09-11T09:53:23.716 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.27714.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28435.log 2024-09-11T09:53:23.716 INFO:teuthology.orchestra.run.smithi107.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27714.log.gz 2024-09-11T09:53:23.717 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.30879.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27460.log 2024-09-11T09:53:23.717 INFO:teuthology.orchestra.run.smithi107.stderr: 19.4% -- replaced with /var/log/ceph/ceph-client.admin.30879.log.gz 2024-09-11T09:53:23.717 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.28435.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27835.log 2024-09-11T09:53:23.717 INFO:teuthology.orchestra.run.smithi107.stderr: 35.3% -- replaced with /var/log/ceph/ceph-client.admin.28435.log.gz 2024-09-11T09:53:23.717 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.27460.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27412.log 2024-09-11T09:53:23.717 INFO:teuthology.orchestra.run.smithi107.stderr: 25.3% -- replaced with /var/log/ceph/ceph-client.admin.27460.log.gz 2024-09-11T09:53:23.717 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.27835.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29014.log 2024-09-11T09:53:23.717 INFO:teuthology.orchestra.run.smithi107.stderr: 19.4% -- replaced with /var/log/ceph/ceph-client.admin.27835.log.gz 2024-09-11T09:53:23.717 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.27412.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27876.log 2024-09-11T09:53:23.717 INFO:teuthology.orchestra.run.smithi107.stderr: 26.3% -- replaced with /var/log/ceph/ceph-client.admin.27412.log.gz 2024-09-11T09:53:23.717 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.29014.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28830.log 2024-09-11T09:53:23.717 INFO:teuthology.orchestra.run.smithi107.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29014.log.gz 2024-09-11T09:53:23.717 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.27876.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28973.log 2024-09-11T09:53:23.717 INFO:teuthology.orchestra.run.smithi107.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27876.log.gz 2024-09-11T09:53:23.718 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.28830.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29959.log 2024-09-11T09:53:23.718 INFO:teuthology.orchestra.run.smithi107.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28830.log.gz 2024-09-11T09:53:23.718 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.28973.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28753.log 2024-09-11T09:53:23.718 INFO:teuthology.orchestra.run.smithi107.stderr: 35.0% -- replaced with /var/log/ceph/ceph-client.admin.28973.log.gz 2024-09-11T09:53:23.718 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.29959.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30733.log 2024-09-11T09:53:23.718 INFO:teuthology.orchestra.run.smithi107.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29959.log.gz 2024-09-11T09:53:23.718 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.28753.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28885.log 2024-09-11T09:53:23.718 INFO:teuthology.orchestra.run.smithi107.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28753.log.gz 2024-09-11T09:53:23.718 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.30733.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28764.log 2024-09-11T09:53:23.718 INFO:teuthology.orchestra.run.smithi107.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30733.log.gz 2024-09-11T09:53:23.718 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.28885.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28916.log 2024-09-11T09:53:23.718 INFO:teuthology.orchestra.run.smithi107.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28885.log.gz 2024-09-11T09:53:23.718 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.28764.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27898.log 2024-09-11T09:53:23.719 INFO:teuthology.orchestra.run.smithi107.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28764.log.gz 2024-09-11T09:53:23.719 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.28916.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29911.log 2024-09-11T09:53:23.719 INFO:teuthology.orchestra.run.smithi107.stderr: 20.0% -- replaced with /var/log/ceph/ceph-client.admin.28916.log.gz 2024-09-11T09:53:23.719 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.27898.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28334.log 2024-09-11T09:53:23.719 INFO:teuthology.orchestra.run.smithi107.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27898.log.gz 2024-09-11T09:53:23.719 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.29911.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30702.log 2024-09-11T09:53:23.719 INFO:teuthology.orchestra.run.smithi107.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29911.log.gz 2024-09-11T09:53:23.719 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.28334.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29064.log 2024-09-11T09:53:23.719 INFO:teuthology.orchestra.run.smithi107.stderr: 24.1% -- replaced with /var/log/ceph/ceph-client.admin.28334.log.gz 2024-09-11T09:53:23.719 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.30702.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29829.log 2024-09-11T09:53:23.719 INFO:teuthology.orchestra.run.smithi107.stderr: 56.5% -- replaced with /var/log/ceph/ceph-client.admin.30702.log.gz 2024-09-11T09:53:23.719 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.29064.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27440.log 2024-09-11T09:53:23.719 INFO:teuthology.orchestra.run.smithi107.stderr: 20.0% -- replaced with /var/log/ceph/ceph-client.admin.29064.log.gz 2024-09-11T09:53:23.719 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.29829.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28962.log 2024-09-11T09:53:23.720 INFO:teuthology.orchestra.run.smithi107.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29829.log.gz 2024-09-11T09:53:23.720 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.27440.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29047.log 2024-09-11T09:53:23.720 INFO:teuthology.orchestra.run.smithi107.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27440.log.gz 2024-09-11T09:53:23.720 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.28962.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29532.log 2024-09-11T09:53:23.720 INFO:teuthology.orchestra.run.smithi107.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28962.log.gz 2024-09-11T09:53:23.720 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.29047.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30121.log 2024-09-11T09:53:23.720 INFO:teuthology.orchestra.run.smithi107.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29047.log.gz 2024-09-11T09:53:23.720 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.29532.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27487.log 2024-09-11T09:53:23.720 INFO:teuthology.orchestra.run.smithi107.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29532.log.gz 2024-09-11T09:53:23.720 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.30121.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27588.log 2024-09-11T09:53:23.720 INFO:teuthology.orchestra.run.smithi107.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30121.log.gz 2024-09-11T09:53:23.720 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.27487.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28964.log 2024-09-11T09:53:23.720 INFO:teuthology.orchestra.run.smithi107.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27487.log.gz 2024-09-11T09:53:23.720 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.27588.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27588.log.gz 2024-09-11T09:53:23.721 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27736.log 2024-09-11T09:53:23.721 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.28964.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29128.log 2024-09-11T09:53:23.721 INFO:teuthology.orchestra.run.smithi107.stderr: 20.0% -- replaced with /var/log/ceph/ceph-client.admin.28964.log.gz 2024-09-11T09:53:23.721 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.27736.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30047.log 2024-09-11T09:53:23.721 INFO:teuthology.orchestra.run.smithi107.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27736.log.gz 2024-09-11T09:53:23.721 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.29128.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29972.log 2024-09-11T09:53:23.721 INFO:teuthology.orchestra.run.smithi107.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29128.log.gz 2024-09-11T09:53:23.721 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.30047.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29340.log 2024-09-11T09:53:23.721 INFO:teuthology.orchestra.run.smithi107.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30047.log.gz 2024-09-11T09:53:23.721 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.29972.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29356.log 2024-09-11T09:53:23.721 INFO:teuthology.orchestra.run.smithi107.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29972.log.gz 2024-09-11T09:53:23.721 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.29340.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30145.log 2024-09-11T09:53:23.721 INFO:teuthology.orchestra.run.smithi107.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29340.log.gz 2024-09-11T09:53:23.721 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.29356.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27368.log 2024-09-11T09:53:23.722 INFO:teuthology.orchestra.run.smithi107.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29356.log.gz 2024-09-11T09:53:23.722 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.30145.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30145.log.gz 2024-09-11T09:53:23.722 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29011.log 2024-09-11T09:53:23.722 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.27368.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30665.log 2024-09-11T09:53:23.722 INFO:teuthology.orchestra.run.smithi107.stderr: 20.0% -- replaced with /var/log/ceph/ceph-client.admin.27368.log.gz 2024-09-11T09:53:23.722 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.29011.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30762.log 2024-09-11T09:53:23.722 INFO:teuthology.orchestra.run.smithi107.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29011.log.gz 2024-09-11T09:53:23.722 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.30665.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30051.log 2024-09-11T09:53:23.722 INFO:teuthology.orchestra.run.smithi107.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30665.log.gz 2024-09-11T09:53:23.722 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.30762.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30938.log 2024-09-11T09:53:23.722 INFO:teuthology.orchestra.run.smithi107.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30762.log.gz 2024-09-11T09:53:23.722 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.30051.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29177.log 2024-09-11T09:53:23.723 INFO:teuthology.orchestra.run.smithi107.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30051.log.gz 2024-09-11T09:53:23.723 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.30938.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28538.log 2024-09-11T09:53:23.723 INFO:teuthology.orchestra.run.smithi107.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30938.log.gz 2024-09-11T09:53:23.723 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.29177.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28257.log 2024-09-11T09:53:23.723 INFO:teuthology.orchestra.run.smithi107.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29177.log.gz 2024-09-11T09:53:23.723 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.28538.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27437.log 2024-09-11T09:53:23.723 INFO:teuthology.orchestra.run.smithi107.stderr: 35.7% -- replaced with /var/log/ceph/ceph-client.admin.28538.log.gz 2024-09-11T09:53:23.723 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.28257.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28426.log 2024-09-11T09:53:23.723 INFO:teuthology.orchestra.run.smithi107.stderr: 41.5% -- replaced with /var/log/ceph/ceph-client.admin.28257.log.gz 2024-09-11T09:53:23.723 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.27437.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29194.log 2024-09-11T09:53:23.723 INFO:teuthology.orchestra.run.smithi107.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27437.log.gz 2024-09-11T09:53:23.723 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.28426.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29621.log 2024-09-11T09:53:23.723 INFO:teuthology.orchestra.run.smithi107.stderr: 18.2% -- replaced with /var/log/ceph/ceph-client.admin.28426.log.gz 2024-09-11T09:53:23.723 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.29194.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29778.log 2024-09-11T09:53:23.724 INFO:teuthology.orchestra.run.smithi107.stderr: 20.0% -- replaced with /var/log/ceph/ceph-client.admin.29194.log.gz 2024-09-11T09:53:23.724 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.29621.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28190.log 2024-09-11T09:53:23.724 INFO:teuthology.orchestra.run.smithi107.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29621.log.gz 2024-09-11T09:53:23.724 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.29778.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29672.log 2024-09-11T09:53:23.724 INFO:teuthology.orchestra.run.smithi107.stderr: 45.5% -- replaced with /var/log/ceph/ceph-client.admin.29778.log.gz 2024-09-11T09:53:23.724 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.28190.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29840.log 2024-09-11T09:53:23.724 INFO:teuthology.orchestra.run.smithi107.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28190.log.gz 2024-09-11T09:53:23.724 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.29672.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28205.log 2024-09-11T09:53:23.724 INFO:teuthology.orchestra.run.smithi107.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29672.log.gz 2024-09-11T09:53:23.724 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.29840.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29301.log 2024-09-11T09:53:23.724 INFO:teuthology.orchestra.run.smithi107.stderr: 24.7% -- replaced with /var/log/ceph/ceph-client.admin.29840.log.gz 2024-09-11T09:53:23.724 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.28205.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28360.log 2024-09-11T09:53:23.724 INFO:teuthology.orchestra.run.smithi107.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28205.log.gz 2024-09-11T09:53:23.725 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.29301.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29893.log 2024-09-11T09:53:23.725 INFO:teuthology.orchestra.run.smithi107.stderr: 47.9% -- replaced with /var/log/ceph/ceph-client.admin.29301.log.gz 2024-09-11T09:53:23.725 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.28360.log: gzip -5 --verbose -- 0.0% /var/log/ceph/ceph-client.admin.27563.log 2024-09-11T09:53:23.725 INFO:teuthology.orchestra.run.smithi107.stderr: -- replaced with /var/log/ceph/ceph-client.admin.28360.log.gz 2024-09-11T09:53:23.725 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.29893.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27451.log 2024-09-11T09:53:23.725 INFO:teuthology.orchestra.run.smithi107.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29893.log.gz 2024-09-11T09:53:23.725 INFO:teuthology.orchestra.run.smithi107.stderr:gzip/var/log/ceph/ceph-client.admin.27563.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.30559.log 2024-09-11T09:53:23.725 INFO:teuthology.orchestra.run.smithi107.stderr: 56.5% -- replaced with /var/log/ceph/ceph-client.admin.27563.log.gz 2024-09-11T09:53:23.725 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.27451.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27238.log 2024-09-11T09:53:23.725 INFO:teuthology.orchestra.run.smithi107.stderr: 20.0% -- replaced with /var/log/ceph/ceph-client.admin.27451.log.gz 2024-09-11T09:53:23.725 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.30559.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27653.log 2024-09-11T09:53:23.726 INFO:teuthology.orchestra.run.smithi107.stderr: 3.6% -- replaced with /var/log/ceph/ceph-client.admin.30559.log.gz 2024-09-11T09:53:23.726 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.27238.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30001.log 2024-09-11T09:53:23.726 INFO:teuthology.orchestra.run.smithi107.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27238.log.gz 2024-09-11T09:53:23.726 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29099.log 2024-09-11T09:53:23.726 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.27653.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27653.log.gz 2024-09-11T09:53:23.726 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.30001.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30325.log 2024-09-11T09:53:23.726 INFO:teuthology.orchestra.run.smithi107.stderr: 18.2% -- replaced with /var/log/ceph/ceph-client.admin.30001.log.gz 2024-09-11T09:53:23.726 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.29099.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28296.log 2024-09-11T09:53:23.726 INFO:teuthology.orchestra.run.smithi107.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29099.log.gz 2024-09-11T09:53:23.726 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.30325.log: gzip 5.5% -5 -- replaced with /var/log/ceph/ceph-client.admin.30325.log.gz --verbose 2024-09-11T09:53:23.726 INFO:teuthology.orchestra.run.smithi107.stderr: -- /var/log/ceph/ceph-client.admin.28848.log 2024-09-11T09:53:23.726 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.28296.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30935.log 2024-09-11T09:53:23.726 INFO:teuthology.orchestra.run.smithi107.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28296.log.gz 2024-09-11T09:53:23.726 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.28848.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.28848.log.gz 2024-09-11T09:53:23.727 INFO:teuthology.orchestra.run.smithi107.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.30526.log 2024-09-11T09:53:23.727 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.30935.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28628.log 2024-09-11T09:53:23.727 INFO:teuthology.orchestra.run.smithi107.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30935.log.gz 2024-09-11T09:53:23.727 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.30526.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27788.log 2024-09-11T09:53:23.727 INFO:teuthology.orchestra.run.smithi107.stderr: 5.5% -- replaced with /var/log/ceph/ceph-client.admin.30526.log.gz 2024-09-11T09:53:23.727 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.28628.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28575.log 2024-09-11T09:53:23.727 INFO:teuthology.orchestra.run.smithi107.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28628.log.gz 2024-09-11T09:53:23.727 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.27788.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29465.log 2024-09-11T09:53:23.727 INFO:teuthology.orchestra.run.smithi107.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27788.log.gz 2024-09-11T09:53:23.727 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.28575.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29399.log 2024-09-11T09:53:23.727 INFO:teuthology.orchestra.run.smithi107.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28575.log.gz 2024-09-11T09:53:23.727 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.29465.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27403.log 2024-09-11T09:53:23.727 INFO:teuthology.orchestra.run.smithi107.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29465.log.gz 2024-09-11T09:53:23.728 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.29399.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27194.log 2024-09-11T09:53:23.728 INFO:teuthology.orchestra.run.smithi107.stderr: 45.5% -- replaced with /var/log/ceph/ceph-client.admin.29399.log.gz 2024-09-11T09:53:23.728 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.27403.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30010.log 2024-09-11T09:53:23.728 INFO:teuthology.orchestra.run.smithi107.stderr: 19.4% -- replaced with /var/log/ceph/ceph-client.admin.27403.log.gz 2024-09-11T09:53:23.728 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.27194.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28951.log 2024-09-11T09:53:23.728 INFO:teuthology.orchestra.run.smithi107.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27194.log.gz 2024-09-11T09:53:23.728 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.30010.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30950.log 2024-09-11T09:53:23.728 INFO:teuthology.orchestra.run.smithi107.stderr: 45.0% -- replaced with /var/log/ceph/ceph-client.admin.30010.log.gz 2024-09-11T09:53:23.728 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.28951.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28136.log 2024-09-11T09:53:23.728 INFO:teuthology.orchestra.run.smithi107.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28951.log.gz 2024-09-11T09:53:23.728 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.30950.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31133.log 2024-09-11T09:53:23.728 INFO:teuthology.orchestra.run.smithi107.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30950.log.gz 2024-09-11T09:53:23.728 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.28136.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28793.log 2024-09-11T09:53:23.728 INFO:teuthology.orchestra.run.smithi107.stderr: 52.3% -- replaced with /var/log/ceph/ceph-client.admin.28136.log.gz 2024-09-11T09:53:23.729 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.31133.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29032.log 2024-09-11T09:53:23.729 INFO:teuthology.orchestra.run.smithi107.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31133.log.gz 2024-09-11T09:53:23.729 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.28793.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28080.log 2024-09-11T09:53:23.729 INFO:teuthology.orchestra.run.smithi107.stderr: 35.0% -- replaced with /var/log/ceph/ceph-client.admin.28793.log.gz 2024-09-11T09:53:23.729 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.29032.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31150.log 2024-09-11T09:53:23.729 INFO:teuthology.orchestra.run.smithi107.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29032.log.gz 2024-09-11T09:53:23.729 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.28080.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29258.log 2024-09-11T09:53:23.729 INFO:teuthology.orchestra.run.smithi107.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28080.log.gz 2024-09-11T09:53:23.729 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.31150.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28371.log 2024-09-11T09:53:23.729 INFO:teuthology.orchestra.run.smithi107.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31150.log.gz 2024-09-11T09:53:23.729 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.29258.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31116.log 2024-09-11T09:53:23.730 INFO:teuthology.orchestra.run.smithi107.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29258.log.gz 2024-09-11T09:53:23.730 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.28371.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27682.log 2024-09-11T09:53:23.730 INFO:teuthology.orchestra.run.smithi107.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28371.log.gz 2024-09-11T09:53:23.730 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.31116.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31116.log.gz 2024-09-11T09:53:23.730 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28187.log 2024-09-11T09:53:23.730 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.27682.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30992.log 2024-09-11T09:53:23.730 INFO:teuthology.orchestra.run.smithi107.stderr: 56.5% -- replaced with /var/log/ceph/ceph-client.admin.27682.log.gz 2024-09-11T09:53:23.730 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.28187.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29290.log 2024-09-11T09:53:23.730 INFO:teuthology.orchestra.run.smithi107.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28187.log.gz 2024-09-11T09:53:23.730 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.30992.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28767.log 2024-09-11T09:53:23.730 INFO:teuthology.orchestra.run.smithi107.stderr: 24.7% -- replaced with /var/log/ceph/ceph-client.admin.30992.log.gz 2024-09-11T09:53:23.730 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.29290.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29999.log 2024-09-11T09:53:23.730 INFO:teuthology.orchestra.run.smithi107.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29290.log.gz 2024-09-11T09:53:23.731 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.28767.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30819.log 2024-09-11T09:53:23.731 INFO:teuthology.orchestra.run.smithi107.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28767.log.gz 2024-09-11T09:53:23.731 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.29999.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29203.log 2024-09-11T09:53:23.731 INFO:teuthology.orchestra.run.smithi107.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29999.log.gz 2024-09-11T09:53:23.731 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.30819.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29240.log 2024-09-11T09:53:23.731 INFO:teuthology.orchestra.run.smithi107.stderr: 25.3% -- replaced with /var/log/ceph/ceph-client.admin.30819.log.gz 2024-09-11T09:53:23.731 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.29203.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30459.log 2024-09-11T09:53:23.731 INFO:teuthology.orchestra.run.smithi107.stderr: 24.7% -- replaced with /var/log/ceph/ceph-client.admin.29203.log.gz 2024-09-11T09:53:23.731 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.29240.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29240.log.gz 2024-09-11T09:53:23.731 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29431.log 2024-09-11T09:53:23.731 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.30459.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29896.log 2024-09-11T09:53:23.731 INFO:teuthology.orchestra.run.smithi107.stderr: 5.5% -- replaced with /var/log/ceph/ceph-client.admin.30459.log.gz 2024-09-11T09:53:23.731 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.29431.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28398.log 2024-09-11T09:53:23.731 INFO:teuthology.orchestra.run.smithi107.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29431.log.gz 2024-09-11T09:53:23.732 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.29896.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28107.log 2024-09-11T09:53:23.732 INFO:teuthology.orchestra.run.smithi107.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29896.log.gz 2024-09-11T09:53:23.732 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.28398.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29922.log 2024-09-11T09:53:23.732 INFO:teuthology.orchestra.run.smithi107.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28398.log.gz 2024-09-11T09:53:23.732 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.28107.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28107.log.gz 2024-09-11T09:53:23.732 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28490.log 2024-09-11T09:53:23.732 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.29922.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28242.log 2024-09-11T09:53:23.732 INFO:teuthology.orchestra.run.smithi107.stderr: 24.7% -- replaced with /var/log/ceph/ceph-client.admin.29922.log.gz 2024-09-11T09:53:23.732 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.28490.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30258.log 2024-09-11T09:53:23.732 INFO:teuthology.orchestra.run.smithi107.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28490.log.gz 2024-09-11T09:53:23.732 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.28242.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29913.log 2024-09-11T09:53:23.732 INFO:teuthology.orchestra.run.smithi107.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28242.log.gz 2024-09-11T09:53:23.733 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.30258.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31075.log 2024-09-11T09:53:23.733 INFO:teuthology.orchestra.run.smithi107.stderr: 5.5% -- replaced with /var/log/ceph/ceph-client.admin.30258.log.gz 2024-09-11T09:53:23.733 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.29913.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28630.log 2024-09-11T09:53:23.733 INFO:teuthology.orchestra.run.smithi107.stderr: 18.8% -- replaced with /var/log/ceph/ceph-client.admin.29913.log.gz 2024-09-11T09:53:23.733 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.31075.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28125.log 2024-09-11T09:53:23.733 INFO:teuthology.orchestra.run.smithi107.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31075.log.gz 2024-09-11T09:53:23.733 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.28630.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28006.log 2024-09-11T09:53:23.733 INFO:teuthology.orchestra.run.smithi107.stderr: 19.4% -- replaced with /var/log/ceph/ceph-client.admin.28630.log.gz 2024-09-11T09:53:23.733 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.28125.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31054.log 2024-09-11T09:53:23.733 INFO:teuthology.orchestra.run.smithi107.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28125.log.gz 2024-09-11T09:53:23.733 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.28006.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29831.log 2024-09-11T09:53:23.733 INFO:teuthology.orchestra.run.smithi107.stderr: 51.7% -- replaced with /var/log/ceph/ceph-client.admin.28006.log.gz 2024-09-11T09:53:23.733 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.31054.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27977.log 2024-09-11T09:53:23.733 INFO:teuthology.orchestra.run.smithi107.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31054.log.gz 2024-09-11T09:53:23.734 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.29831.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28461.log 2024-09-11T09:53:23.734 INFO:teuthology.orchestra.run.smithi107.stderr: 18.8% -- replaced with /var/log/ceph/ceph-client.admin.29831.log.gz 2024-09-11T09:53:23.734 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.27977.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29804.log 2024-09-11T09:53:23.734 INFO:teuthology.orchestra.run.smithi107.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27977.log.gz 2024-09-11T09:53:23.734 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.28461.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29345.log 2024-09-11T09:53:23.734 INFO:teuthology.orchestra.run.smithi107.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28461.log.gz 2024-09-11T09:53:23.734 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.29804.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29600.log 2024-09-11T09:53:23.734 INFO:teuthology.orchestra.run.smithi107.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29804.log.gz 2024-09-11T09:53:23.734 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.29345.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29000.log 2024-09-11T09:53:23.734 INFO:teuthology.orchestra.run.smithi107.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29345.log.gz 2024-09-11T09:53:23.734 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.29600.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28784.log 2024-09-11T09:53:23.734 INFO:teuthology.orchestra.run.smithi107.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29600.log.gz 2024-09-11T09:53:23.734 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.29000.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30632.log 2024-09-11T09:53:23.734 INFO:teuthology.orchestra.run.smithi107.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29000.log.gz 2024-09-11T09:53:23.735 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.28784.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30291.log 2024-09-11T09:53:23.735 INFO:teuthology.orchestra.run.smithi107.stderr: 19.4% -- replaced with /var/log/ceph/ceph-client.admin.28784.log.gz 2024-09-11T09:53:23.735 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.30632.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29669.log 2024-09-11T09:53:23.735 INFO:teuthology.orchestra.run.smithi107.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30632.log.gz 2024-09-11T09:53:23.735 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.30291.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28207.log 2024-09-11T09:53:23.735 INFO:teuthology.orchestra.run.smithi107.stderr: 4.5% -- replaced with /var/log/ceph/ceph-client.admin.30291.log.gz 2024-09-11T09:53:23.735 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.29669.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28925.log 2024-09-11T09:53:23.735 INFO:teuthology.orchestra.run.smithi107.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29669.log.gz 2024-09-11T09:53:23.735 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.28207.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28914.log 2024-09-11T09:53:23.735 INFO:teuthology.orchestra.run.smithi107.stderr: 18.8% -- replaced with /var/log/ceph/ceph-client.admin.28207.log.gz 2024-09-11T09:53:23.735 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.28925.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28287.log 2024-09-11T09:53:23.735 INFO:teuthology.orchestra.run.smithi107.stderr: 24.1% -- replaced with /var/log/ceph/ceph-client.admin.28925.log.gz 2024-09-11T09:53:23.735 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.28914.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27377.log 2024-09-11T09:53:23.736 INFO:teuthology.orchestra.run.smithi107.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28914.log.gz 2024-09-11T09:53:23.736 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.28287.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29691.log 2024-09-11T09:53:23.736 INFO:teuthology.orchestra.run.smithi107.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28287.log.gz 2024-09-11T09:53:23.736 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.27377.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27995.log 2024-09-11T09:53:23.736 INFO:teuthology.orchestra.run.smithi107.stderr: 4.8% -- replaced with /var/log/ceph/ceph-client.admin.27377.log.gz 2024-09-11T09:53:23.736 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.29691.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28665.log 2024-09-11T09:53:23.736 INFO:teuthology.orchestra.run.smithi107.stderr: 20.0% -- replaced with /var/log/ceph/ceph-client.admin.29691.log.gz 2024-09-11T09:53:23.736 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27626.log 2024-09-11T09:53:23.736 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.27995.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27995.log.gz 2024-09-11T09:53:23.736 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.28665.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30845.log 2024-09-11T09:53:23.736 INFO:teuthology.orchestra.run.smithi107.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28665.log.gz 2024-09-11T09:53:23.736 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.27626.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30172.log 2024-09-11T09:53:23.736 INFO:teuthology.orchestra.run.smithi107.stderr: 54.9% -- replaced with /var/log/ceph/ceph-client.admin.27626.log.gz 2024-09-11T09:53:23.737 INFO:teuthology.orchestra.run.smithi107.stderr:gzip/var/log/ceph/ceph-client.admin.30845.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.28727.log 2024-09-11T09:53:23.737 INFO:teuthology.orchestra.run.smithi107.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30845.log.gz 2024-09-11T09:53:23.737 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.30172.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31033.log 2024-09-11T09:53:23.737 INFO:teuthology.orchestra.run.smithi107.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30172.log.gz 2024-09-11T09:53:23.737 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.28727.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29962.log 2024-09-11T09:53:23.737 INFO:teuthology.orchestra.run.smithi107.stderr: 35.3% -- replaced with /var/log/ceph/ceph-client.admin.28727.log.gz 2024-09-11T09:53:23.737 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30859.log 2024-09-11T09:53:23.737 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.31033.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31033.log.gz 2024-09-11T09:53:23.737 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.29962.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28819.log 2024-09-11T09:53:23.737 INFO:teuthology.orchestra.run.smithi107.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29962.log.gz 2024-09-11T09:53:23.737 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.30859.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29769.log 2024-09-11T09:53:23.737 INFO:teuthology.orchestra.run.smithi107.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30859.log.gz 2024-09-11T09:53:23.737 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.28819.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29547.log 2024-09-11T09:53:23.737 INFO:teuthology.orchestra.run.smithi107.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28819.log.gz 2024-09-11T09:53:23.738 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.29769.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27870.log 2024-09-11T09:53:23.738 INFO:teuthology.orchestra.run.smithi107.stderr: 20.0% -- replaced with /var/log/ceph/ceph-client.admin.29769.log.gz 2024-09-11T09:53:23.738 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.29547.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29749.log 2024-09-11T09:53:23.738 INFO:teuthology.orchestra.run.smithi107.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29547.log.gz 2024-09-11T09:53:23.738 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.27870.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29948.log 2024-09-11T09:53:23.738 INFO:teuthology.orchestra.run.smithi107.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27870.log.gz 2024-09-11T09:53:23.738 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.29749.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28602.log 2024-09-11T09:53:23.738 INFO:teuthology.orchestra.run.smithi107.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29749.log.gz 2024-09-11T09:53:23.738 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.29948.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27351.log 2024-09-11T09:53:23.738 INFO:teuthology.orchestra.run.smithi107.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29948.log.gz 2024-09-11T09:53:23.738 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.28602.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28564.log 2024-09-11T09:53:23.738 INFO:teuthology.orchestra.run.smithi107.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28602.log.gz 2024-09-11T09:53:23.738 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.27351.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28059.log 2024-09-11T09:53:23.739 INFO:teuthology.orchestra.run.smithi107.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27351.log.gz 2024-09-11T09:53:23.739 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.28564.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30080.log 2024-09-11T09:53:23.739 INFO:teuthology.orchestra.run.smithi107.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28564.log.gz 2024-09-11T09:53:23.739 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.28059.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27536.log 2024-09-11T09:53:23.739 INFO:teuthology.orchestra.run.smithi107.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28059.log.gz 2024-09-11T09:53:23.739 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.30080.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28110.log 2024-09-11T09:53:23.739 INFO:teuthology.orchestra.run.smithi107.stderr: 45.0% -- replaced with /var/log/ceph/ceph-client.admin.30080.log.gz 2024-09-11T09:53:23.739 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.27536.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27818.log 2024-09-11T09:53:23.739 INFO:teuthology.orchestra.run.smithi107.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27536.log.gz 2024-09-11T09:53:23.739 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.28110.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30061.log 2024-09-11T09:53:23.739 INFO:teuthology.orchestra.run.smithi107.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28110.log.gz 2024-09-11T09:53:23.739 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.27818.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27950.log 2024-09-11T09:53:23.740 INFO:teuthology.orchestra.run.smithi107.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27818.log.gz 2024-09-11T09:53:23.740 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.30061.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28639.log 2024-09-11T09:53:23.740 INFO:teuthology.orchestra.run.smithi107.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30061.log.gz 2024-09-11T09:53:23.740 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.27950.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27539.log 2024-09-11T09:53:23.740 INFO:teuthology.orchestra.run.smithi107.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27950.log.gz 2024-09-11T09:53:23.740 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.28639.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28689.log 2024-09-11T09:53:23.740 INFO:teuthology.orchestra.run.smithi107.stderr: 35.2% -- replaced with /var/log/ceph/ceph-client.admin.28639.log.gz 2024-09-11T09:53:23.740 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.27539.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28716.log 2024-09-11T09:53:23.740 INFO:teuthology.orchestra.run.smithi107.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27539.log.gz 2024-09-11T09:53:23.740 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.28689.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28689.log.gz 2024-09-11T09:53:23.740 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27507.log 2024-09-11T09:53:23.740 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.28716.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30877.log 2024-09-11T09:53:23.741 INFO:teuthology.orchestra.run.smithi107.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28716.log.gz 2024-09-11T09:53:23.741 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.27507.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28676.log 2024-09-11T09:53:23.741 INFO:teuthology.orchestra.run.smithi107.stderr: 26.3% -- replaced with /var/log/ceph/ceph-client.admin.27507.log.gz 2024-09-11T09:53:23.741 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.30877.log: gzip -5 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30877.log.gz --verbose 2024-09-11T09:53:23.741 INFO:teuthology.orchestra.run.smithi107.stderr: -- /var/log/ceph/ceph-client.admin.28679.log 2024-09-11T09:53:23.741 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.28676.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27554.log 2024-09-11T09:53:23.741 INFO:teuthology.orchestra.run.smithi107.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28676.log.gz 2024-09-11T09:53:23.741 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.28679.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28127.log 2024-09-11T09:53:23.741 INFO:teuthology.orchestra.run.smithi107.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28679.log.gz 2024-09-11T09:53:23.741 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.27554.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28500.log 2024-09-11T09:53:23.741 INFO:teuthology.orchestra.run.smithi107.stderr: 19.4% -- replaced with /var/log/ceph/ceph-client.admin.27554.log.gz 2024-09-11T09:53:23.741 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.28127.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30921.log 2024-09-11T09:53:23.741 INFO:teuthology.orchestra.run.smithi107.stderr: 18.8% -- replaced with /var/log/ceph/ceph-client.admin.28127.log.gz 2024-09-11T09:53:23.741 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.28500.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27980.log 2024-09-11T09:53:23.742 INFO:teuthology.orchestra.run.smithi107.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28500.log.gz 2024-09-11T09:53:23.742 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.30921.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30635.log 2024-09-11T09:53:23.742 INFO:teuthology.orchestra.run.smithi107.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30921.log.gz 2024-09-11T09:53:23.742 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.27980.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29495.log 2024-09-11T09:53:23.742 INFO:teuthology.orchestra.run.smithi107.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27980.log.gz 2024-09-11T09:53:23.742 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.30635.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31012.log 2024-09-11T09:53:23.742 INFO:teuthology.orchestra.run.smithi107.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30635.log.gz 2024-09-11T09:53:23.742 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.29495.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29495.log.gz 2024-09-11T09:53:23.742 INFO:teuthology.orchestra.run.smithi107.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27708.log 2024-09-11T09:53:23.742 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.31012.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30174.log 2024-09-11T09:53:23.742 INFO:teuthology.orchestra.run.smithi107.stderr: 53.0% -- replaced with /var/log/ceph/ceph-client.admin.31012.log.gz 2024-09-11T09:53:23.742 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.27708.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29267.log 2024-09-11T09:53:23.742 INFO:teuthology.orchestra.run.smithi107.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27708.log.gz 2024-09-11T09:53:23.742 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.30174.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27815.log 2024-09-11T09:53:23.742 INFO:teuthology.orchestra.run.smithi107.stderr: 19.4% -- replaced with /var/log/ceph/ceph-client.admin.30174.log.gz 2024-09-11T09:53:23.743 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.29267.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29267.log.gz 2024-09-11T09:53:23.743 INFO:teuthology.orchestra.run.smithi107.stderr:/var/log/ceph/ceph-client.admin.27815.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27815.log.gz 2024-09-11T09:53:23.743 INFO:teuthology.orchestra.run.smithi107.stderr: 2024-09-11T09:53:23.743 INFO:teuthology.orchestra.run.smithi107.stderr:real 0m0.087s 2024-09-11T09:53:23.743 INFO:teuthology.orchestra.run.smithi107.stderr:user 0m0.092s 2024-09-11T09:53:23.743 INFO:teuthology.orchestra.run.smithi107.stderr:sys 0m0.111s 2024-09-11T09:53:23.746 INFO:teuthology.orchestra.run.smithi096.stderr: 93.5% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-09-11T09:53:23.746 INFO:teuthology.orchestra.run.smithi096.stderr: 2024-09-11T09:53:23.746 INFO:teuthology.orchestra.run.smithi096.stderr:real 0m0.231s 2024-09-11T09:53:23.746 INFO:teuthology.orchestra.run.smithi096.stderr:user 0m0.730s 2024-09-11T09:53:23.746 INFO:teuthology.orchestra.run.smithi096.stderr:sys 0m0.044s 2024-09-11T09:53:23.747 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51042.log 2024-09-11T09:53:23.747 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.52249.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39227.log 2024-09-11T09:53:23.748 INFO:teuthology.orchestra.run.smithi017.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52249.log.gz 2024-09-11T09:53:23.748 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.51042.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48194.log 2024-09-11T09:53:23.748 INFO:teuthology.orchestra.run.smithi017.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51042.log.gz 2024-09-11T09:53:23.748 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.39227.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50410.log 2024-09-11T09:53:23.748 INFO:teuthology.orchestra.run.smithi017.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39227.log.gz 2024-09-11T09:53:23.748 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.48194.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35642.log 2024-09-11T09:53:23.748 INFO:teuthology.orchestra.run.smithi017.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48194.log.gz 2024-09-11T09:53:23.748 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.50410.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36889.log 2024-09-11T09:53:23.748 INFO:teuthology.orchestra.run.smithi017.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50410.log.gz 2024-09-11T09:53:23.748 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.35642.log: gzip -5 --verbose -- 0.0% /var/log/ceph/ceph-client.admin.38511.log 2024-09-11T09:53:23.748 INFO:teuthology.orchestra.run.smithi017.stderr: -- replaced with /var/log/ceph/ceph-client.admin.35642.log.gz 2024-09-11T09:53:23.748 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.36889.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51941.log 2024-09-11T09:53:23.748 INFO:teuthology.orchestra.run.smithi017.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36889.log.gz 2024-09-11T09:53:23.748 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.38511.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52716.log 2024-09-11T09:53:23.749 INFO:teuthology.orchestra.run.smithi017.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38511.log.gz 2024-09-11T09:53:23.749 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.51941.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40258.log 2024-09-11T09:53:23.749 INFO:teuthology.orchestra.run.smithi017.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51941.log.gz 2024-09-11T09:53:23.749 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.52716.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45964.log 2024-09-11T09:53:23.749 INFO:teuthology.orchestra.run.smithi017.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52716.log.gz 2024-09-11T09:53:23.749 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.40258.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35251.log 2024-09-11T09:53:23.749 INFO:teuthology.orchestra.run.smithi017.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40258.log.gz 2024-09-11T09:53:23.749 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.45964.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52655.log 2024-09-11T09:53:23.749 INFO:teuthology.orchestra.run.smithi017.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45964.log.gz 2024-09-11T09:53:23.749 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.35251.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34988.log 2024-09-11T09:53:23.749 INFO:teuthology.orchestra.run.smithi017.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35251.log.gz 2024-09-11T09:53:23.749 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.52655.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53873.log 2024-09-11T09:53:23.749 INFO:teuthology.orchestra.run.smithi017.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52655.log.gz 2024-09-11T09:53:23.749 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.34988.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39717.log 2024-09-11T09:53:23.750 INFO:teuthology.orchestra.run.smithi017.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34988.log.gz 2024-09-11T09:53:23.750 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.53873.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34432.log 2024-09-11T09:53:23.750 INFO:teuthology.orchestra.run.smithi017.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53873.log.gz 2024-09-11T09:53:23.750 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.39717.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48552.log 2024-09-11T09:53:23.750 INFO:teuthology.orchestra.run.smithi017.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39717.log.gz 2024-09-11T09:53:23.750 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.34432.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43030.log 2024-09-11T09:53:23.750 INFO:teuthology.orchestra.run.smithi017.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34432.log.gz 2024-09-11T09:53:23.750 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.48552.log: gzip -5 --verbose -- 0.0% /var/log/ceph/ceph-client.admin.49794.log -- replaced with /var/log/ceph/ceph-client.admin.48552.log.gz 2024-09-11T09:53:23.750 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-11T09:53:23.750 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.43030.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50512.log 2024-09-11T09:53:23.750 INFO:teuthology.orchestra.run.smithi017.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43030.log.gz 2024-09-11T09:53:23.750 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.49794.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45361.log 2024-09-11T09:53:23.750 INFO:teuthology.orchestra.run.smithi017.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49794.log.gz 2024-09-11T09:53:23.751 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.50512.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-09-11T09:53:23.751 INFO:teuthology.orchestra.run.smithi017.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50512.log.gz 2024-09-11T09:53:23.751 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.45361.log: gzip -5 0.0% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.45361.log.gz /var/log/ceph/ceph-client.admin.37658.log 2024-09-11T09:53:23.751 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-11T09:53:23.751 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39971.log 2024-09-11T09:53:23.751 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.37658.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35824.log 2024-09-11T09:53:23.751 INFO:teuthology.orchestra.run.smithi017.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37658.log.gz 2024-09-11T09:53:23.751 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.39971.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53201.log 2024-09-11T09:53:23.751 INFO:teuthology.orchestra.run.smithi017.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39971.log.gz 2024-09-11T09:53:23.751 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.35824.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35824.log.gz 2024-09-11T09:53:23.751 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53915.log 2024-09-11T09:53:23.751 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.53201.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49752.log 2024-09-11T09:53:23.751 INFO:teuthology.orchestra.run.smithi017.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53201.log.gz 2024-09-11T09:53:23.751 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.53915.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42253.log 2024-09-11T09:53:23.752 INFO:teuthology.orchestra.run.smithi017.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53915.log.gz 2024-09-11T09:53:23.752 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.49752.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49752.log.gz 2024-09-11T09:53:23.752 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36591.log 2024-09-11T09:53:23.752 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.42253.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37997.log 2024-09-11T09:53:23.752 INFO:teuthology.orchestra.run.smithi017.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42253.log.gz 2024-09-11T09:53:23.752 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.36591.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51563.log 2024-09-11T09:53:23.752 INFO:teuthology.orchestra.run.smithi017.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36591.log.gz 2024-09-11T09:53:23.752 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.37997.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39084.log 2024-09-11T09:53:23.752 INFO:teuthology.orchestra.run.smithi017.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37997.log.gz 2024-09-11T09:53:23.752 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.51563.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53327.log 2024-09-11T09:53:23.752 INFO:teuthology.orchestra.run.smithi017.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51563.log.gz 2024-09-11T09:53:23.752 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.39084.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48347.log 2024-09-11T09:53:23.752 INFO:teuthology.orchestra.run.smithi017.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39084.log.gz 2024-09-11T09:53:23.752 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.53327.log: gzip -5 --verbose 0.0% -- -- replaced with /var/log/ceph/ceph-client.admin.53327.log.gz 2024-09-11T09:53:23.753 INFO:teuthology.orchestra.run.smithi017.stderr: /var/log/ceph/ceph-client.admin.47088.log 2024-09-11T09:53:23.753 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.48347.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47151.log 2024-09-11T09:53:23.753 INFO:teuthology.orchestra.run.smithi017.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48347.log.gz 2024-09-11T09:53:23.753 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.47088.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53411.log 2024-09-11T09:53:23.753 INFO:teuthology.orchestra.run.smithi017.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47088.log.gz 2024-09-11T09:53:23.753 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.47151.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51689.log 2024-09-11T09:53:23.753 INFO:teuthology.orchestra.run.smithi017.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47151.log.gz 2024-09-11T09:53:23.753 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.53411.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53411.log.gzgzip 2024-09-11T09:53:23.753 INFO:teuthology.orchestra.run.smithi017.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.46894.log 2024-09-11T09:53:23.753 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.51689.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52431.log 2024-09-11T09:53:23.753 INFO:teuthology.orchestra.run.smithi017.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51689.log.gz 2024-09-11T09:53:23.753 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.46894.log: gzip 0.0% -5 -- replaced with /var/log/ceph/ceph-client.admin.46894.log.gz --verbose 2024-09-11T09:53:23.753 INFO:teuthology.orchestra.run.smithi017.stderr: -- /var/log/ceph/ceph-client.admin.42570.log 2024-09-11T09:53:23.754 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.52431.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50257.log 2024-09-11T09:53:23.754 INFO:teuthology.orchestra.run.smithi017.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52431.log.gz 2024-09-11T09:53:23.754 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.42570.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42570.log.gz 2024-09-11T09:53:23.754 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52987.log 2024-09-11T09:53:23.754 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.50257.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40968.log 2024-09-11T09:53:23.754 INFO:teuthology.orchestra.run.smithi017.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50257.log.gz 2024-09-11T09:53:23.754 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.52987.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52987.log.gz -5 2024-09-11T09:53:23.754 INFO:teuthology.orchestra.run.smithi017.stderr: --verbose -- /var/log/ceph/ceph-client.admin.52793.log 2024-09-11T09:53:23.754 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.40968.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38100.log 2024-09-11T09:53:23.754 INFO:teuthology.orchestra.run.smithi017.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40968.log.gz 2024-09-11T09:53:23.754 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.52793.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51773.log 2024-09-11T09:53:23.754 INFO:teuthology.orchestra.run.smithi017.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52793.log.gz 2024-09-11T09:53:23.754 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.38100.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34733.log 2024-09-11T09:53:23.754 INFO:teuthology.orchestra.run.smithi017.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38100.log.gz 2024-09-11T09:53:23.755 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.51773.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51773.log.gz -5 2024-09-11T09:53:23.755 INFO:teuthology.orchestra.run.smithi017.stderr: --verbose -- /var/log/ceph/ceph-client.admin.34734.log 2024-09-11T09:53:23.755 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.34733.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37401.log 2024-09-11T09:53:23.755 INFO:teuthology.orchestra.run.smithi017.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34733.log.gz 2024-09-11T09:53:23.755 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.34734.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35138.log 2024-09-11T09:53:23.755 INFO:teuthology.orchestra.run.smithi017.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34734.log.gz 2024-09-11T09:53:23.755 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.37401.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52109.log 2024-09-11T09:53:23.755 INFO:teuthology.orchestra.run.smithi017.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37401.log.gz 2024-09-11T09:53:23.755 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.35138.log: gzip -5 --verbose -- 0.0% /var/log/ceph/ceph-client.admin.44040.log 2024-09-11T09:53:23.755 INFO:teuthology.orchestra.run.smithi017.stderr: -- replaced with /var/log/ceph/ceph-client.admin.35138.log.gz 2024-09-11T09:53:23.755 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.52109.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36962.log 2024-09-11T09:53:23.755 INFO:teuthology.orchestra.run.smithi017.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52109.log.gz 2024-09-11T09:53:23.755 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.44040.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44040.log.gz 2024-09-11T09:53:23.755 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35558.log 2024-09-11T09:53:23.755 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.36962.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47028.log 2024-09-11T09:53:23.756 INFO:teuthology.orchestra.run.smithi017.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36962.log.gz 2024-09-11T09:53:23.756 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.35558.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35558.log.gz 2024-09-11T09:53:23.756 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52482.log 2024-09-11T09:53:23.756 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.47028.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45050.log 2024-09-11T09:53:23.756 INFO:teuthology.orchestra.run.smithi017.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47028.log.gz 2024-09-11T09:53:23.756 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.52482.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52482.log.gz 2024-09-11T09:53:23.756 INFO:teuthology.orchestra.run.smithi017.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.52926.log 2024-09-11T09:53:23.756 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.45050.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34902.log 2024-09-11T09:53:23.756 INFO:teuthology.orchestra.run.smithi017.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45050.log.gz 2024-09-11T09:53:23.756 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.52926.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52926.log.gz 2024-09-11T09:53:23.756 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41143.log 2024-09-11T09:53:23.756 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.34902.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38398.log 2024-09-11T09:53:23.756 INFO:teuthology.orchestra.run.smithi017.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34902.log.gz 2024-09-11T09:53:23.756 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.41143.log: gzip 0.0% -5 -- replaced with /var/log/ceph/ceph-client.admin.41143.log.gz --verbose 2024-09-11T09:53:23.757 INFO:teuthology.orchestra.run.smithi017.stderr: -- /var/log/ceph/ceph-client.admin.49956.log 2024-09-11T09:53:23.757 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.38398.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40500.log 2024-09-11T09:53:23.757 INFO:teuthology.orchestra.run.smithi017.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38398.log.gz 2024-09-11T09:53:23.757 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.49956.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.49956.log.gz 2024-09-11T09:53:23.757 INFO:teuthology.orchestra.run.smithi017.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.34265.log 2024-09-11T09:53:23.757 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.40500.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34731.log 2024-09-11T09:53:23.757 INFO:teuthology.orchestra.run.smithi017.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40500.log.gz 2024-09-11T09:53:23.757 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.34265.log: gzip -5 --verbose -- 0.0% /var/log/ceph/ceph-client.admin.45556.log -- replaced with /var/log/ceph/ceph-client.admin.34265.log.gz 2024-09-11T09:53:23.757 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-11T09:53:23.757 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.34731.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52794.log 2024-09-11T09:53:23.758 INFO:teuthology.orchestra.run.smithi017.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34731.log.gz 2024-09-11T09:53:23.758 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.45556.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45556.log.gzgzip 2024-09-11T09:53:23.758 INFO:teuthology.orchestra.run.smithi017.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.48450.log 2024-09-11T09:53:23.758 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.52794.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49915.log 2024-09-11T09:53:23.758 INFO:teuthology.orchestra.run.smithi017.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52794.log.gz 2024-09-11T09:53:23.758 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.48450.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48450.log.gz 2024-09-11T09:53:23.758 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37341.log 2024-09-11T09:53:23.759 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.49915.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34998.log 2024-09-11T09:53:23.759 INFO:teuthology.orchestra.run.smithi017.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49915.log.gz 2024-09-11T09:53:23.759 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.37341.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.37341.log.gz 2024-09-11T09:53:23.759 INFO:teuthology.orchestra.run.smithi017.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.31677.log 2024-09-11T09:53:23.759 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.34998.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53159.log 2024-09-11T09:53:23.759 INFO:teuthology.orchestra.run.smithi017.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34998.log.gz 2024-09-11T09:53:23.759 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.31677.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44988.log 2024-09-11T09:53:23.759 INFO:teuthology.orchestra.run.smithi017.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31677.log.gz 2024-09-11T09:53:23.759 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.53159.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51605.log 2024-09-11T09:53:23.759 INFO:teuthology.orchestra.run.smithi017.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53159.log.gz 2024-09-11T09:53:23.759 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.44988.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44988.log.gz 2024-09-11T09:53:23.759 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51983.log 2024-09-11T09:53:23.759 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.51605.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47741.log 2024-09-11T09:53:23.759 INFO:teuthology.orchestra.run.smithi017.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51605.log.gz 2024-09-11T09:53:23.760 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.51983.log: gzip 0.0% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.51983.log.gz 2024-09-11T09:53:23.760 INFO:teuthology.orchestra.run.smithi017.stderr: -- /var/log/ceph/ceph-client.admin.34926.log 2024-09-11T09:53:23.760 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.47741.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43322.log 2024-09-11T09:53:23.760 INFO:teuthology.orchestra.run.smithi017.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47741.log.gz 2024-09-11T09:53:23.760 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.34926.log: gzip -5 --verbose -- 0.0% /var/log/ceph/ceph.audit.log -- replaced with /var/log/ceph/ceph-client.admin.34926.log.gz 2024-09-11T09:53:23.760 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-11T09:53:23.760 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.43322.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38039.log 2024-09-11T09:53:23.760 INFO:teuthology.orchestra.run.smithi017.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43322.log.gz 2024-09-11T09:53:23.760 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45629.log 2024-09-11T09:53:23.760 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.38039.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.38039.log.gz -5 2024-09-11T09:53:23.760 INFO:teuthology.orchestra.run.smithi017.stderr: --verbose -- /var/log/ceph/ceph-client.admin.50174.log 2024-09-11T09:53:23.760 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52025.log 2024-09-11T09:53:23.761 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.45629.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45629.log.gz 2024-09-11T09:53:23.761 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.50174.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50174.log.gz 2024-09-11T09:53:23.761 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46640.log 2024-09-11T09:53:23.761 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35704.log 2024-09-11T09:53:23.761 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.52025.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52025.log.gz 2024-09-11T09:53:23.761 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.46640.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46640.log.gz 2024-09-11T09:53:23.761 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35390.log 2024-09-11T09:53:23.761 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34727.log 2024-09-11T09:53:23.761 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.35704.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35704.log.gz 2024-09-11T09:53:23.761 INFO:teuthology.orchestra.run.smithi017.stderr: 93.7% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-09-11T09:53:23.761 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.35390.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35390.log.gz 2024-09-11T09:53:23.761 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51516.log 2024-09-11T09:53:23.761 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49955.log 2024-09-11T09:53:23.761 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.34727.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34727.log.gz 2024-09-11T09:53:23.762 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.51516.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53495.log 2024-09-11T09:53:23.762 INFO:teuthology.orchestra.run.smithi017.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51516.log.gz 2024-09-11T09:53:23.762 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53537.log 2024-09-11T09:53:23.762 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.49955.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49955.log.gz 2024-09-11T09:53:23.762 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.53495.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53495.log.gz 2024-09-11T09:53:23.762 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35474.log 2024-09-11T09:53:23.762 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51451.log 2024-09-11T09:53:23.762 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.53537.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53537.log.gz 2024-09-11T09:53:23.762 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.35474.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53957.log 2024-09-11T09:53:23.762 INFO:teuthology.orchestra.run.smithi017.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35474.log.gz 2024-09-11T09:53:23.762 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.51451.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49393.log 2024-09-11T09:53:23.762 INFO:teuthology.orchestra.run.smithi017.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51451.log.gz 2024-09-11T09:53:23.762 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.53957.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53705.log 0.0% 2024-09-11T09:53:23.762 INFO:teuthology.orchestra.run.smithi017.stderr: -- replaced with /var/log/ceph/ceph-client.admin.53957.log.gz 2024-09-11T09:53:23.763 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.49393.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34592.log 2024-09-11T09:53:23.763 INFO:teuthology.orchestra.run.smithi017.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49393.log.gz 2024-09-11T09:53:23.763 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.53705.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53705.log.gz -5 2024-09-11T09:53:23.763 INFO:teuthology.orchestra.run.smithi017.stderr: --verbose -- /var/log/ceph/ceph-client.admin.35827.log 2024-09-11T09:53:23.763 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37740.log 2024-09-11T09:53:23.763 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.34592.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34592.log.gz 2024-09-11T09:53:23.763 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.35827.log: gzip -5 0.0% --verbose -- replaced with /var/log/ceph/ceph-client.admin.35827.log.gz -- 2024-09-11T09:53:23.763 INFO:teuthology.orchestra.run.smithi017.stderr: /var/log/ceph/ceph-client.admin.48551.log 2024-09-11T09:53:23.763 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.37740.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40735.log 2024-09-11T09:53:23.763 INFO:teuthology.orchestra.run.smithi017.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37740.log.gz 2024-09-11T09:53:23.763 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.48551.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42115.log 2024-09-11T09:53:23.763 INFO:teuthology.orchestra.run.smithi017.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48551.log.gz 2024-09-11T09:53:23.763 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.40735.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42792.log 2024-09-11T09:53:23.763 INFO:teuthology.orchestra.run.smithi017.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40735.log.gz 2024-09-11T09:53:23.764 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.42115.log: gzip -5 0.0% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.42115.log.gz 2024-09-11T09:53:23.764 INFO:teuthology.orchestra.run.smithi017.stderr: /var/log/ceph/ceph-client.admin.47194.log 2024-09-11T09:53:23.764 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.42792.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37256.log 2024-09-11T09:53:23.764 INFO:teuthology.orchestra.run.smithi017.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42792.log.gz 2024-09-11T09:53:23.764 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.47194.log: gzip -5 0.0% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.47194.log.gz 2024-09-11T09:53:23.764 INFO:teuthology.orchestra.run.smithi017.stderr: /var/log/ceph/ceph-client.admin.43534.log 2024-09-11T09:53:23.764 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51731.log 2024-09-11T09:53:23.764 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.37256.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37256.log.gz 2024-09-11T09:53:23.764 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.43534.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41276.log 2024-09-11T09:53:23.764 INFO:teuthology.orchestra.run.smithi017.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43534.log.gz 2024-09-11T09:53:23.764 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.51731.log: /var/log/ceph/ceph-client.admin.47702.log 2024-09-11T09:53:23.764 INFO:teuthology.orchestra.run.smithi017.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51731.log.gz 2024-09-11T09:53:23.764 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.41276.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38865.log 2024-09-11T09:53:23.764 INFO:teuthology.orchestra.run.smithi017.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41276.log.gz 2024-09-11T09:53:23.764 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.47702.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37638.log 2024-09-11T09:53:23.765 INFO:teuthology.orchestra.run.smithi017.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47702.log.gz 2024-09-11T09:53:23.765 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.38865.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44522.log 2024-09-11T09:53:23.765 INFO:teuthology.orchestra.run.smithi017.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38865.log.gz 2024-09-11T09:53:23.765 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.37638.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37638.log.gz 2024-09-11T09:53:23.765 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49115.log 2024-09-11T09:53:23.765 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.44522.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42116.log 2024-09-11T09:53:23.765 INFO:teuthology.orchestra.run.smithi017.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44522.log.gz 2024-09-11T09:53:23.765 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.49115.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51815.log 2024-09-11T09:53:23.765 INFO:teuthology.orchestra.run.smithi017.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49115.log.gz 2024-09-11T09:53:23.765 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.42116.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-09-11T09:53:23.765 INFO:teuthology.orchestra.run.smithi017.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42116.log.gz 2024-09-11T09:53:23.765 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.51815.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51815.log.gz 2024-09-11T09:53:23.765 INFO:teuthology.orchestra.run.smithi017.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.36084.log 2024-09-11T09:53:23.765 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42178.log 2024-09-11T09:53:23.766 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.36084.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35600.log 2024-09-11T09:53:23.766 INFO:teuthology.orchestra.run.smithi017.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36084.log.gz 2024-09-11T09:53:23.766 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.42178.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53243.log 2024-09-11T09:53:23.766 INFO:teuthology.orchestra.run.smithi017.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42178.log.gz 2024-09-11T09:53:23.766 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50092.log 2024-09-11T09:53:23.766 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.35600.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35600.log.gz 2024-09-11T09:53:23.766 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.53243.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52251.log 2024-09-11T09:53:23.766 INFO:teuthology.orchestra.run.smithi017.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53243.log.gz 2024-09-11T09:53:23.766 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.50092.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53453.log 2024-09-11T09:53:23.766 INFO:teuthology.orchestra.run.smithi017.stderr: -- replaced with /var/log/ceph/ceph-client.admin.50092.log.gz 2024-09-11T09:53:23.766 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46239.log 2024-09-11T09:53:23.766 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.52251.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52251.log.gz 2024-09-11T09:53:23.766 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.53453.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43958.log 2024-09-11T09:53:23.766 INFO:teuthology.orchestra.run.smithi017.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53453.log.gz 2024-09-11T09:53:23.767 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34989.log 2024-09-11T09:53:23.767 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.46239.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46239.log.gz 2024-09-11T09:53:23.767 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.43958.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52675.log 2024-09-11T09:53:23.767 INFO:teuthology.orchestra.run.smithi017.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43958.log.gz 2024-09-11T09:53:23.767 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.34989.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34729.log 2024-09-11T09:53:23.767 INFO:teuthology.orchestra.run.smithi017.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34989.log.gz 2024-09-11T09:53:23.767 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38866.log 2024-09-11T09:53:23.767 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.52675.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52675.log.gz 2024-09-11T09:53:23.767 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.34729.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35198.log 2024-09-11T09:53:23.767 INFO:teuthology.orchestra.run.smithi017.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34729.log.gz 2024-09-11T09:53:23.767 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.38866.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38866.log.gz -5 2024-09-11T09:53:23.767 INFO:teuthology.orchestra.run.smithi017.stderr: --verbose -- /var/log/ceph/ceph-client.admin.43807.log 2024-09-11T09:53:23.767 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52067.log 2024-09-11T09:53:23.768 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.35198.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35198.log.gz 2024-09-11T09:53:23.768 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.43807.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43807.log.gz 2024-09-11T09:53:23.768 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-09-11T09:53:23.768 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42517.log 2024-09-11T09:53:23.768 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.52067.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52067.log.gz 2024-09-11T09:53:23.768 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50786.log 2024-09-11T09:53:23.768 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.42517.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52238.log 2024-09-11T09:53:23.768 INFO:teuthology.orchestra.run.smithi017.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42517.log.gz 2024-09-11T09:53:23.769 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.50786.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41801.log 2024-09-11T09:53:23.769 INFO:teuthology.orchestra.run.smithi017.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50786.log.gz 2024-09-11T09:53:23.769 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.52238.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38989.log 2024-09-11T09:53:23.769 INFO:teuthology.orchestra.run.smithi017.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52238.log.gz 2024-09-11T09:53:23.769 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53663.log 2024-09-11T09:53:23.769 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.41801.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41801.log.gz 2024-09-11T09:53:23.769 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.38989.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46680.log 2024-09-11T09:53:23.769 INFO:teuthology.orchestra.run.smithi017.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38989.log.gz 2024-09-11T09:53:23.769 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.53663.log: gzip -5 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53663.log.gz 2024-09-11T09:53:23.769 INFO:teuthology.orchestra.run.smithi017.stderr: --verbose -- /var/log/ceph/ceph-client.admin.47426.log 2024-09-11T09:53:23.769 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.46680.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46680.log.gz 2024-09-11T09:53:23.770 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52250.log 2024-09-11T09:53:23.770 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.47426.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38988.log 2024-09-11T09:53:23.770 INFO:teuthology.orchestra.run.smithi017.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47426.log.gz 2024-09-11T09:53:23.770 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.52250.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53999.log 2024-09-11T09:53:23.770 INFO:teuthology.orchestra.run.smithi017.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52250.log.gz 2024-09-11T09:53:23.770 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39635.log 2024-09-11T09:53:23.770 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.38988.log: /var/log/ceph/ceph-client.admin.53999.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38988.log.gz 2024-09-11T09:53:23.770 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-09-11T09:53:23.771 INFO:teuthology.orchestra.run.smithi017.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53999.log.gz 2024-09-11T09:53:23.771 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.39635.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35158.log 2024-09-11T09:53:23.771 INFO:teuthology.orchestra.run.smithi017.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39635.log.gz 2024-09-11T09:53:23.771 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41974.log 2024-09-11T09:53:23.771 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.35158.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52419.log 2024-09-11T09:53:23.771 INFO:teuthology.orchestra.run.smithi017.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35158.log.gz 2024-09-11T09:53:23.771 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.41974.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48748.log 2024-09-11T09:53:23.771 INFO:teuthology.orchestra.run.smithi017.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41974.log.gz 2024-09-11T09:53:23.771 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51062.log 2024-09-11T09:53:23.771 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.52419.log: /var/log/ceph/ceph-client.admin.48748.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48748.log.gz 2024-09-11T09:53:23.771 INFO:teuthology.orchestra.run.smithi017.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.53831.log 2024-09-11T09:53:23.771 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.51062.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52419.log.gz 2024-09-11T09:53:23.771 INFO:teuthology.orchestra.run.smithi017.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51062.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52505.log 2024-09-11T09:53:23.771 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-11T09:53:23.772 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.53831.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53831.log.gz 2024-09-11T09:53:23.772 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51647.log 2024-09-11T09:53:23.772 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.52505.log: gzip -5 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52505.log.gz 2024-09-11T09:53:23.772 INFO:teuthology.orchestra.run.smithi017.stderr: --verbose -- /var/log/ceph/ceph-client.admin.53067.log 2024-09-11T09:53:23.772 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.51647.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51647.log.gz 2024-09-11T09:53:23.772 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40205.log 2024-09-11T09:53:23.772 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.53067.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49677.log 2024-09-11T09:53:23.772 INFO:teuthology.orchestra.run.smithi017.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53067.log.gz 2024-09-11T09:53:23.772 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.40205.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48072.log 2024-09-11T09:53:23.772 INFO:teuthology.orchestra.run.smithi017.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40205.log.gz 2024-09-11T09:53:23.772 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44312.log 2024-09-11T09:53:23.772 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.49677.log: /var/log/ceph/ceph-client.admin.48072.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52239.log 2024-09-11T09:53:23.772 INFO:teuthology.orchestra.run.smithi017.stderr: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49677.log.gz 2024-09-11T09:53:23.772 INFO:teuthology.orchestra.run.smithi017.stderr: -- replaced with /var/log/ceph/ceph-client.admin.48072.log.gz 2024-09-11T09:53:23.773 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.44312.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44312.log.gz 2024-09-11T09:53:23.773 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39482.log 2024-09-11T09:53:23.773 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.52239.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53747.log 2024-09-11T09:53:23.773 INFO:teuthology.orchestra.run.smithi017.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52239.log.gz 2024-09-11T09:53:23.773 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40907.log 2024-09-11T09:53:23.773 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.39482.log: /var/log/ceph/ceph-client.admin.53747.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39482.log.gz 2024-09-11T09:53:23.773 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51899.log 2024-09-11T09:53:23.773 INFO:teuthology.orchestra.run.smithi017.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53747.log.gz 2024-09-11T09:53:23.773 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47742.log 2024-09-11T09:53:23.773 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.40907.log: /var/log/ceph/ceph-client.admin.51899.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49515.log 2024-09-11T09:53:23.773 INFO:teuthology.orchestra.run.smithi017.stderr: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51899.log.gz 2024-09-11T09:53:23.773 INFO:teuthology.orchestra.run.smithi017.stderr: -- replaced with /var/log/ceph/ceph-client.admin.40907.log.gz 2024-09-11T09:53:23.773 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.47742.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47742.log.gz 2024-09-11T09:53:23.773 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46106.log 2024-09-11T09:53:23.774 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.49515.log: 0.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51277.log 2024-09-11T09:53:23.774 INFO:teuthology.orchestra.run.smithi017.stderr: -- replaced with /var/log/ceph/ceph-client.admin.49515.log.gz 2024-09-11T09:53:23.774 INFO:teuthology.orchestra.run.smithi017.stderr:gzip/var/log/ceph/ceph-client.admin.46106.log: -5 --verbose 0.0% -- /var/log/ceph/ceph-client.admin.51452.log 2024-09-11T09:53:23.774 INFO:teuthology.orchestra.run.smithi017.stderr: -- replaced with /var/log/ceph/ceph-client.admin.46106.log.gz 2024-09-11T09:53:23.774 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.51277.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51277.log.gz 2024-09-11T09:53:23.774 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39360.log 2024-09-11T09:53:23.774 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46464.log 2024-09-11T09:53:23.774 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.51452.log: /var/log/ceph/ceph-client.admin.39360.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51452.log.gz 2024-09-11T09:53:23.774 INFO:teuthology.orchestra.run.smithi017.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39360.log.gz 2024-09-11T09:53:23.774 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52795.log 2024-09-11T09:53:23.774 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.46464.log: -- /var/log/ceph/ceph.log 2024-09-11T09:53:23.774 INFO:teuthology.orchestra.run.smithi017.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46464.log.gz 2024-09-11T09:53:23.775 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.52795.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52795.log.gz 2024-09-11T09:53:23.775 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35298.log 2024-09-11T09:53:23.775 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53621.log 2024-09-11T09:53:23.775 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.35298.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41779.log 2024-09-11T09:53:23.775 INFO:teuthology.orchestra.run.smithi017.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35298.log.gz 2024-09-11T09:53:23.775 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.53621.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46259.log 2024-09-11T09:53:23.775 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.41779.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53621.log.gz 2024-09-11T09:53:23.775 INFO:teuthology.orchestra.run.smithi017.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41779.log.gz 2024-09-11T09:53:23.775 INFO:teuthology.orchestra.run.smithi017.stderr: 91.7% -- replaced with /var/log/ceph/ceph.log.gz 2024-09-11T09:53:23.775 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54039.log 2024-09-11T09:53:23.775 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.46259.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46259.log.gz 2024-09-11T09:53:23.775 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43625.log 2024-09-11T09:53:23.776 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.54039.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.54039.log.gz 2024-09-11T09:53:23.776 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34376.log 2024-09-11T09:53:23.776 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.43625.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43625.log.gz 2024-09-11T09:53:23.776 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46484.log 2024-09-11T09:53:23.776 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.34376.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34376.log.gz 2024-09-11T09:53:23.776 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45831.log 2024-09-11T09:53:23.776 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.46484.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46484.log.gz 2024-09-11T09:53:23.776 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44231.log 2024-09-11T09:53:23.776 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.45831.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45831.log.gz 2024-09-11T09:53:23.776 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35432.log 2024-09-11T09:53:23.776 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.44231.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44231.log.gz 2024-09-11T09:53:23.776 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50840.log 2024-09-11T09:53:23.777 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.35432.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35432.log.gz 2024-09-11T09:53:23.777 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36359.log 2024-09-11T09:53:23.777 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.50840.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50840.log.gz 2024-09-11T09:53:23.777 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31447.log 2024-09-11T09:53:23.777 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.36359.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36359.log.gz 2024-09-11T09:53:23.777 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42908.log 2024-09-11T09:53:23.777 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.31447.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31447.log.gz 2024-09-11T09:53:23.777 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36216.log 2024-09-11T09:53:23.777 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.42908.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42908.log.gz 2024-09-11T09:53:23.777 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48019.log 2024-09-11T09:53:23.778 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.36216.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36216.log.gz 2024-09-11T09:53:23.778 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48367.log 2024-09-11T09:53:23.778 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.48019.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48019.log.gz 2024-09-11T09:53:23.778 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53369.log 2024-09-11T09:53:23.778 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.48367.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48367.log.gz 2024-09-11T09:53:23.778 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.27077.log 2024-09-11T09:53:23.778 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.53369.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53369.log.gzgzip 2024-09-11T09:53:23.778 INFO:teuthology.orchestra.run.smithi017.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.34457.log 2024-09-11T09:53:23.778 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph.tmp-client.admin.27077.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.27077.log.gz 2024-09-11T09:53:23.778 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36674.log 2024-09-11T09:53:23.778 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.34457.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34457.log.gz 2024-09-11T09:53:23.778 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41715.log 2024-09-11T09:53:23.778 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.36674.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36674.log.gz 2024-09-11T09:53:23.778 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49495.log 2024-09-11T09:53:23.779 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.41715.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41715.log.gz 2024-09-11T09:53:23.779 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35516.log 2024-09-11T09:53:23.779 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.49495.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49495.log.gz 2024-09-11T09:53:23.779 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53285.log 2024-09-11T09:53:23.779 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.35516.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35516.log.gz 2024-09-11T09:53:23.779 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41523.log 2024-09-11T09:53:23.779 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.53285.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53285.log.gz 2024-09-11T09:53:23.779 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38175.log 2024-09-11T09:53:23.779 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.41523.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41523.log.gz 2024-09-11T09:53:23.779 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53066.log 2024-09-11T09:53:23.779 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.38175.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38175.log.gz 2024-09-11T09:53:23.779 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43241.log 2024-09-11T09:53:23.779 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.53066.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53066.log.gz 2024-09-11T09:53:23.780 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42335.log 2024-09-11T09:53:23.780 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.43241.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43241.log.gz 2024-09-11T09:53:23.780 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35253.log 2024-09-11T09:53:23.780 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.42335.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42335.log.gz 2024-09-11T09:53:23.780 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38101.log 2024-09-11T09:53:23.780 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.35253.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35253.log.gz 2024-09-11T09:53:23.780 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52247.log 2024-09-11T09:53:23.780 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.38101.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38101.log.gz 2024-09-11T09:53:23.780 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41714.log 2024-09-11T09:53:23.781 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.52247.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52247.log.gz 2024-09-11T09:53:23.781 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46382.log 2024-09-11T09:53:23.781 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.41714.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41714.log.gz 2024-09-11T09:53:23.781 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53007.log 2024-09-11T09:53:23.781 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.46382.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46382.log.gz 2024-09-11T09:53:23.781 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37576.log 2024-09-11T09:53:23.781 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.53007.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53007.log.gz 2024-09-11T09:53:23.781 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34550.log 2024-09-11T09:53:23.781 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.37576.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37576.log.gz 2024-09-11T09:53:23.781 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52481.log 2024-09-11T09:53:23.781 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.34550.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34550.log.gz 2024-09-11T09:53:23.781 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45281.log 2024-09-11T09:53:23.781 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.52481.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52481.log.gz 2024-09-11T09:53:23.782 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31749.log 2024-09-11T09:53:23.782 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.45281.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45281.log.gz 2024-09-11T09:53:23.782 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37875.log 2024-09-11T09:53:23.782 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.31749.log: -- /var/log/ceph/ceph-client.admin.52483.log 2024-09-11T09:53:23.782 INFO:teuthology.orchestra.run.smithi017.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31749.log.gz 2024-09-11T09:53:23.782 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.37875.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37875.log.gz 2024-09-11T09:53:23.782 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41378.log 2024-09-11T09:53:23.782 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.52483.log: -- /var/log/ceph/ceph-client.admin.34906.log 2024-09-11T09:53:23.782 INFO:teuthology.orchestra.run.smithi017.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52483.log.gz 2024-09-11T09:53:23.782 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.41378.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41378.log.gz 2024-09-11T09:53:23.782 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34730.log 2024-09-11T09:53:23.782 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.34906.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34906.log.gz 2024-09-11T09:53:23.783 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51857.log 2024-09-11T09:53:23.783 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.34730.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49023.log 2024-09-11T09:53:23.783 INFO:teuthology.orchestra.run.smithi017.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34730.log.gz 2024-09-11T09:53:23.783 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.51857.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50705.log 2024-09-11T09:53:23.783 INFO:teuthology.orchestra.run.smithi017.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51857.log.gz 2024-09-11T09:53:23.783 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41439.log 2024-09-11T09:53:23.783 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.49023.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49023.log.gz 2024-09-11T09:53:23.783 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40573.log 2024-09-11T09:53:23.783 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.50705.log: /var/log/ceph/ceph-client.admin.41439.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50705.log.gz 2024-09-11T09:53:23.783 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38673.log 2024-09-11T09:53:23.783 INFO:teuthology.orchestra.run.smithi017.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41439.log.gz 2024-09-11T09:53:23.783 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.40573.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40573.log.gz 2024-09-11T09:53:23.784 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49339.log 2024-09-11T09:53:23.784 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.38673.log: gzip -5 --verbose 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38673.log.gz 2024-09-11T09:53:23.784 INFO:teuthology.orchestra.run.smithi017.stderr: -- /var/log/ceph/ceph-client.admin.51390.log 2024-09-11T09:53:23.784 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.49339.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49339.log.gz 2024-09-11T09:53:23.784 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44817.log 2024-09-11T09:53:23.784 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.51390.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51390.log.gz 2024-09-11T09:53:23.784 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36592.log 2024-09-11T09:53:23.784 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.44817.log: gzip -5 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44817.log.gz 2024-09-11T09:53:23.784 INFO:teuthology.orchestra.run.smithi017.stderr: --verbose -- /var/log/ceph/ceph-client.admin.47468.log 2024-09-11T09:53:23.784 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.36592.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36592.log.gz 2024-09-11T09:53:23.784 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39930.log 2024-09-11T09:53:23.784 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.47468.log: gzip -5 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47468.log.gz 2024-09-11T09:53:23.785 INFO:teuthology.orchestra.run.smithi017.stderr: --verbose -- /var/log/ceph/ceph-client.admin.53579.log 2024-09-11T09:53:23.785 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.39930.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39930.log.gz 2024-09-11T09:53:23.785 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37257.log 2024-09-11T09:53:23.785 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.53579.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44615.log 2024-09-11T09:53:23.785 INFO:teuthology.orchestra.run.smithi017.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53579.log.gz 2024-09-11T09:53:23.785 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.37257.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37124.log 2024-09-11T09:53:23.785 INFO:teuthology.orchestra.run.smithi017.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37257.log.gz 2024-09-11T09:53:23.785 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52927.log 2024-09-11T09:53:23.785 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.44615.log: /var/log/ceph/ceph-client.admin.37124.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48768.log 2024-09-11T09:53:23.785 INFO:teuthology.orchestra.run.smithi017.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37124.log.gz 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44615.log.gz 2024-09-11T09:53:23.786 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-11T09:53:23.786 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.52927.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52927.log.gz 2024-09-11T09:53:23.786 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49258.log 2024-09-11T09:53:23.786 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.48768.log: /var/log/ceph/ceph-client.admin.53789.log 2024-09-11T09:53:23.786 INFO:teuthology.orchestra.run.smithi017.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48768.log.gz 2024-09-11T09:53:23.786 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.49258.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49258.log.gz 2024-09-11T09:53:23.786 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35825.log 2024-09-11T09:53:23.786 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.53789.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53789.log.gz 2024-09-11T09:53:23.786 INFO:teuthology.orchestra.run.smithi017.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34508.log 2024-09-11T09:53:23.786 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.35825.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35825.log.gz 2024-09-11T09:53:23.786 INFO:teuthology.orchestra.run.smithi017.stderr:/var/log/ceph/ceph-client.admin.34508.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34508.log.gz 2024-09-11T09:53:23.786 INFO:teuthology.orchestra.run.smithi017.stderr: 91.3% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-09-11T09:53:23.786 INFO:teuthology.orchestra.run.smithi017.stderr: 93.1% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-09-11T09:53:23.787 INFO:teuthology.orchestra.run.smithi017.stderr: 90.5% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-09-11T09:53:23.882 INFO:teuthology.orchestra.run.smithi017.stderr: 94.7% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-09-11T09:53:23.964 INFO:teuthology.orchestra.run.smithi017.stderr: 94.4% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-09-11T09:53:23.965 INFO:teuthology.orchestra.run.smithi017.stderr: 2024-09-11T09:53:23.966 INFO:teuthology.orchestra.run.smithi017.stderr:real 0m0.470s 2024-09-11T09:53:23.966 INFO:teuthology.orchestra.run.smithi017.stderr:user 0m1.323s 2024-09-11T09:53:23.966 INFO:teuthology.orchestra.run.smithi017.stderr:sys 0m0.186s 2024-09-11T09:53:23.966 INFO:tasks.ceph:Archiving logs... 2024-09-11T09:53:23.966 DEBUG:teuthology.misc:Transferring archived files from smithi017:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-09-10_21:16:03-rbd-squid-distro-default-smithi/7900734/remote/smithi017/log 2024-09-11T09:53:23.967 DEBUG:teuthology.orchestra.run.smithi017:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-11T09:53:24.377 DEBUG:teuthology.misc:Transferring archived files from smithi096:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-09-10_21:16:03-rbd-squid-distro-default-smithi/7900734/remote/smithi096/log 2024-09-11T09:53:24.377 DEBUG:teuthology.orchestra.run.smithi096:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-11T09:53:24.515 DEBUG:teuthology.misc:Transferring archived files from smithi107:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-09-10_21:16:03-rbd-squid-distro-default-smithi/7900734/remote/smithi107/log 2024-09-11T09:53:24.516 DEBUG:teuthology.orchestra.run.smithi107:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-11T09:53:24.744 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-09-11T09:53:24.757 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-09-11T09:53:24.757 DEBUG:teuthology.orchestra.run.smithi017:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-11T09:53:24.760 DEBUG:teuthology.orchestra.run.smithi096:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-11T09:53:24.762 DEBUG:teuthology.orchestra.run.smithi107:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-11T09:53:24.862 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-11T09:53:24.871 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-11T09:53:24.871 DEBUG:teuthology.orchestra.run.smithi017:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-11T09:53:24.874 DEBUG:teuthology.orchestra.run.smithi096:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-11T09:53:24.876 DEBUG:teuthology.orchestra.run.smithi107:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-11T09:53:24.888 INFO:teuthology.orchestra.run.smithi017.stderr:bash: line 1: ntpq: command not found 2024-09-11T09:53:24.890 INFO:teuthology.orchestra.run.smithi096.stderr:bash: line 1: ntpq: command not found 2024-09-11T09:53:24.893 INFO:teuthology.orchestra.run.smithi107.stderr:bash: line 1: ntpq: command not found 2024-09-11T09:53:24.893 INFO:teuthology.orchestra.run.smithi017.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-11T09:53:24.893 INFO:teuthology.orchestra.run.smithi017.stdout:=============================================================================== 2024-09-11T09:53:24.893 INFO:teuthology.orchestra.run.smithi017.stdout:^* hv01.front.sepia.ceph.com 3 6 377 61 -2905us[-2909us] +/- 64ms 2024-09-11T09:53:24.893 INFO:teuthology.orchestra.run.smithi017.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 60 +1790us[+1790us] +/- 73ms 2024-09-11T09:53:24.893 INFO:teuthology.orchestra.run.smithi017.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 60 +1580us[+1580us] +/- 73ms 2024-09-11T09:53:24.894 INFO:teuthology.orchestra.run.smithi017.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-09-11T09:53:24.894 INFO:teuthology.orchestra.run.smithi096.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-11T09:53:24.894 INFO:teuthology.orchestra.run.smithi096.stdout:=============================================================================== 2024-09-11T09:53:24.895 INFO:teuthology.orchestra.run.smithi096.stdout:^* hv01.front.sepia.ceph.com 3 6 377 60 -2907us[-2912us] +/- 64ms 2024-09-11T09:53:24.895 INFO:teuthology.orchestra.run.smithi096.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 61 +1780us[+1774us] +/- 73ms 2024-09-11T09:53:24.895 INFO:teuthology.orchestra.run.smithi096.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 61 +1587us[+1581us] +/- 73ms 2024-09-11T09:53:24.895 INFO:teuthology.orchestra.run.smithi096.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-09-11T09:53:24.897 INFO:teuthology.orchestra.run.smithi107.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-11T09:53:24.897 INFO:teuthology.orchestra.run.smithi107.stdout:=============================================================================== 2024-09-11T09:53:24.897 INFO:teuthology.orchestra.run.smithi107.stdout:^* hv01.front.sepia.ceph.com 3 6 377 60 -2891us[-2885us] +/- 64ms 2024-09-11T09:53:24.897 INFO:teuthology.orchestra.run.smithi107.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 59 +1825us[+1825us] +/- 73ms 2024-09-11T09:53:24.898 INFO:teuthology.orchestra.run.smithi107.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 59 +1637us[+1637us] +/- 73ms 2024-09-11T09:53:24.898 INFO:teuthology.orchestra.run.smithi107.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-09-11T09:53:24.898 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-11T09:53:24.907 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-11T09:53:24.907 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-11T09:53:24.915 DEBUG:teuthology.orchestra.run.smithi017:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-11T09:53:25.016 DEBUG:teuthology.orchestra.run.smithi096:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-11T09:53:25.090 DEBUG:teuthology.orchestra.run.smithi107:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-11T09:53:25.158 DEBUG:teuthology.orchestra.run.smithi017:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-09-11T09:53:25.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T09:53:25.189 DEBUG:teuthology.orchestra.run.smithi096:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-09-11T09:53:25.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T09:53:25.220 DEBUG:teuthology.orchestra.run.smithi107:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-09-11T09:53:25.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T09:53:25.251 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-11T09:53:25.260 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-11T09:53:25.268 INFO:teuthology.task.internal:Duration was 621.644193 seconds 2024-09-11T09:53:25.268 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-11T09:53:25.277 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-11T09:53:25.277 DEBUG:teuthology.orchestra.run.smithi017:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-11T09:53:25.280 DEBUG:teuthology.orchestra.run.smithi096:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-11T09:53:25.282 DEBUG:teuthology.orchestra.run.smithi107:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-11T09:53:25.320 INFO:teuthology.orchestra.run.smithi017.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-11T09:53:25.320 INFO:teuthology.orchestra.run.smithi096.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-11T09:53:25.332 INFO:teuthology.orchestra.run.smithi107.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-11T09:53:25.615 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-11T09:53:25.616 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi017.front.sepia.ceph.com 2024-09-11T09:53:25.616 DEBUG:teuthology.orchestra.run.smithi017:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-11T09:53:25.640 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi096.front.sepia.ceph.com 2024-09-11T09:53:25.640 DEBUG:teuthology.orchestra.run.smithi096:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-11T09:53:25.684 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi107.front.sepia.ceph.com 2024-09-11T09:53:25.684 DEBUG:teuthology.orchestra.run.smithi107:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-11T09:53:25.711 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-11T09:53:25.711 DEBUG:teuthology.orchestra.run.smithi017:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-11T09:53:25.713 DEBUG:teuthology.orchestra.run.smithi096:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-11T09:53:25.727 DEBUG:teuthology.orchestra.run.smithi107:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-11T09:53:25.778 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-11T09:53:25.778 DEBUG:teuthology.orchestra.run.smithi017:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-11T09:53:25.782 DEBUG:teuthology.orchestra.run.smithi096:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-11T09:53:25.795 DEBUG:teuthology.orchestra.run.smithi107:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-11T09:53:25.919 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-11T09:53:25.929 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-11T09:53:25.929 DEBUG:teuthology.orchestra.run.smithi017:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-11T09:53:25.954 DEBUG:teuthology.orchestra.run.smithi096:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-11T09:53:25.979 DEBUG:teuthology.orchestra.run.smithi107:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-11T09:53:26.004 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-11T09:53:26.013 DEBUG:teuthology.orchestra.run.smithi017:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-11T09:53:26.016 DEBUG:teuthology.orchestra.run.smithi096:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-11T09:53:26.021 DEBUG:teuthology.orchestra.run.smithi107:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-11T09:53:26.037 INFO:teuthology.orchestra.run.smithi017.stdout:kernel.core_pattern = core 2024-09-11T09:53:26.042 INFO:teuthology.orchestra.run.smithi096.stdout:kernel.core_pattern = core 2024-09-11T09:53:26.067 INFO:teuthology.orchestra.run.smithi107.stdout:kernel.core_pattern = core 2024-09-11T09:53:26.080 DEBUG:teuthology.orchestra.run.smithi017:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-11T09:53:26.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T09:53:26.114 DEBUG:teuthology.orchestra.run.smithi096:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-11T09:53:26.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T09:53:26.130 DEBUG:teuthology.orchestra.run.smithi107:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-11T09:53:26.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T09:53:26.144 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-11T09:53:26.153 INFO:teuthology.task.internal:Transferring archived files... 2024-09-11T09:53:26.154 DEBUG:teuthology.misc:Transferring archived files from smithi017:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-09-10_21:16:03-rbd-squid-distro-default-smithi/7900734/remote/smithi017 2024-09-11T09:53:26.154 DEBUG:teuthology.orchestra.run.smithi017:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-11T09:53:26.190 DEBUG:teuthology.misc:Transferring archived files from smithi096:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-09-10_21:16:03-rbd-squid-distro-default-smithi/7900734/remote/smithi096 2024-09-11T09:53:26.191 DEBUG:teuthology.orchestra.run.smithi096:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-11T09:53:26.224 DEBUG:teuthology.misc:Transferring archived files from smithi107:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-09-10_21:16:03-rbd-squid-distro-default-smithi/7900734/remote/smithi107 2024-09-11T09:53:26.224 DEBUG:teuthology.orchestra.run.smithi107:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-11T09:53:26.257 INFO:teuthology.task.internal:Removing archive directory... 2024-09-11T09:53:26.257 DEBUG:teuthology.orchestra.run.smithi017:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-11T09:53:26.259 DEBUG:teuthology.orchestra.run.smithi096:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-11T09:53:26.261 DEBUG:teuthology.orchestra.run.smithi107:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-11T09:53:26.307 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-11T09:53:26.319 INFO:teuthology.task.internal:Not uploading archives. 2024-09-11T09:53:26.319 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-11T09:53:26.336 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-11T09:53:26.336 DEBUG:teuthology.orchestra.run.smithi017:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-11T09:53:26.340 DEBUG:teuthology.orchestra.run.smithi096:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-11T09:53:26.343 DEBUG:teuthology.orchestra.run.smithi107:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-11T09:53:26.353 INFO:teuthology.orchestra.run.smithi017.stdout: 262169 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 11 09:53 /home/ubuntu/cephtest 2024-09-11T09:53:26.355 INFO:teuthology.orchestra.run.smithi096.stdout: 262169 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 11 09:53 /home/ubuntu/cephtest 2024-09-11T09:53:26.362 INFO:teuthology.orchestra.run.smithi107.stdout: 262169 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 11 09:53 /home/ubuntu/cephtest 2024-09-11T09:53:26.363 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-11T09:53:26.373 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-11T09:53:26.396 INFO:teuthology.run:Summary data: description: rbd/device/{base/install cluster/{fixed-3 openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-comp-snappy supported-random-distro$/{centos_latest} thrashers/default thrashosds-health workloads/rbd_nbd} duration: 621.6441926956177 flavor: default owner: scheduled_teuthology@teuthology success: true 2024-09-11T09:53:26.396 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-11T09:53:26.505 INFO:teuthology.run:pass